Method for detecting objects from point cloud data using a transformer including an attention model

Abstract

The present invention relates to a method for detecting multiple objects (O1, O2) from point cloud data using a transformer including an attention model, where the states of the tracked objects (O1, O2) are stored in a feature space, comprising the following steps: a. Calculating a feature vector from the point cloud data by a backbone (2), where the feature vector is a key vector (k) for the transformer; j ) and the value vector (v i ) from the point cloud data by the sampling method (4), a first anchor position (p i (0) ) using the encoding (5); c. i (0) ) for the first layer (s0) of the transformer, i (0) ) and the first layer (s0) of the transformer decoder (6) converts the object query (y i (0) ) and key vector (k i ) and the value vector (v j ) to obtain the result feature vector (z i (0) ) and e. Calculating the result feature vector (z i (0) ) with respect to the box parameter (b zi (0) ) (7) Step f. Calculate the box parameter (z i (0) ) is the first anchor position (p i (0) ) to determine the anchor position (p i (s) g. Encoding (50, 150) to find the adapted anchor position (pi (s) ) for at least one further layer(s) of the transformer. i (s) ) and step h. i (l) ) the first layer result feature vector (z i (0) ) to obtain the transformed result feature vector (z i (0) ) serves as an object query for at least one further layer(s) of the transformer; the transformed result feature vector (y i (s) ), the calculated object query of the current layer(s), as well as the key vector (k i ) and the value vector (v j ) to the resulting feature vector (z i (s) ) is calculated.

Description

[Technical Field]

[0001] The present invention relates to a method for detecting multiple objects from point cloud data using a transformer that includes an attention model. [Background technology]

[0002] Today, object detection is performed in imaging sensors. Typically, multiple objects are present in the recorded environment, and therefore, the detection of multiple objects is performed. For example, object detection by sensors is used with respect to vehicles to capture other vehicles, other road users, and infrastructure. This data can be used for (partially) automated or autonomous driving.

[0003] In recent years, the concept of using transformers for object detection has been explored. Transformers were initially described in the context of natural language processing in the paper "Attention is all you need" by Ashish Vaswani et al. (arXiv preprint arXiv:1706.03762, 2017). In object detection, measurements of each object are used to calculate a bounding box describing the object and its box parameters, such as the object's position, size, orientation, velocity, and / or class identifier. Transformers can also be used for downstream applications such as object tracking, prediction, or (path) planning. When using transformers for object detection, the suppression of duplicate detection, which was traditionally applied in post-processing, does not need to be a major consideration. To date, such transformers have been applied, for example, to image data. In contrast, their use with large point clouds, such as those arising in the context of autonomous and automated driving, is not well known. [Overview of the Initiative]

[0004] The present invention relates to a method for detecting multiple objects from point cloud data using a transformer including an attention model. The point cloud data is captured by, for example, LiDAR. However, this method is not limited to LiDAR, and other sensor types can also be used. Preferably, the sensor or sensor system is placed in a vehicle so that point cloud data is recorded from the vehicle.

[0005] This method involves the following steps: First, feature vectors are calculated from the point cloud data. This is done by a backbone rather than by the transformer's encoder as is typical. The backbone is a neural network used to extract features from the measurement data or to transform the input into a specific feature representation that can be further processed later. Thus, the transformer's encoder can be eliminated. Preferably, the output of the backbone is reformatted to obtain a sequence of feature vectors of a predetermined length. By calculating feature vectors using a backbone, the input sequence in self-attention is less restricted than when using the transformer's encoder, and in grid-based backbones such as PointPillars, a sufficiently small cell size can be selected. The feature vectors thus calculated are then fed to the transformer to serve as key and value vectors for calculating cross-attention.

[0006] Furthermore, a first anchor position for the first layer of the transformer is calculated from the point cloud data using a sampling method such as farthest point sampling (FPS). From the first anchor position, a feature vector is calculated using encoding, such as Fourier encoding. Encoding can be completed, in particular, by a feedforward network. The feature vector thus calculated functions as an object query for the first layer of the transformer's decoder. The object query for the anchor position serves as a starting point for object retrieval. However, the search is not limited to these anchor positions, and objects far from these anchor positions are also detected. The anchor positions do not correspond to anchor boxes used in other detection methods. Therefore, the object query for the transformer is data-dependent rather than being learned as usual. This is particularly advantageous in sparse point clouds, where a lot of computational resources are usually consumed to find the positions that actually contain the data. Such sparse point clouds occur especially in measurements using LiDAR. The object query calculated from the anchor position serves as a slot for possible objects.

[0007] The transformer's decoder, in the first layer, calculates a result feature vector (also called the decoder output vector) from the object query (i.e., the feature vector mentioned above) and the key vector and value vector (i.e., the feature vector mentioned at the beginning).

[0008] From the resulting feature vectors, box parameters relating to the bounding box describing the object, namely, for example, the object's position or position difference relative to the anchor position, size, direction, velocity, and / or class identifier, are calculated using a feedforward network. Preferably, a different feedforward network is used to calculate the object query, and this feedforward network has different weightings.

[0009] Next, the anchor positions are fitted using the calculated box parameters to process at least one further layer of the transformer. When fitting the anchor positions, the position difference of the box parameters calculated from the result feature vector of the first layer of the transformer is added to the first anchor position. Generally, the box parameters can be obtained with respect to the result feature vector of the first layer, and the box parameters are far from the first anchor position and therefore have a large position difference. The fitting of the anchor positions yields fitted anchor positions, which are closer to the actual objects. From the fitted anchor positions, feature vectors are calculated using the encoding described above, which serve as object queries for at least one further layer of the transformer.

[0010] In addition to propagating information about the fitted anchor locations, the information about the high-dimensional result feature vectors of the first layer is also propagated, so a transformation of the result feature vectors of the first layer with respect to the fitted anchor locations is performed. Here, the result feature vectors are aligned to the fitted anchor locations. Advantageously, this is achieved by a feedforward network consisting of two layers, including ReLU activation (Rectified Linear Unit). Since only a feedforward network with two layers is used here, this incurs only a small additional cost.

[0011] The steps described above—fitting anchor locations, calculating feature vectors from the fitted anchor locations, and transforming the resulting feature vectors—are referred to as object query refinement in this specification.

[0012] Here, the transformed result feature vector and the computed object query, particularly their vector sum, are supplied to the transformer's decoder as input for at least one further layer, where they function as slots for possible objects. The transformer's decoder computes a result feature vector for at least one further layer from the transformed result feature vector computed for the previous layer, from the computed object query (calculated from the fitted anchor locations as described above), and from the key vector and value vector mentioned at the beginning.

[0013] As a result, the feature vectors, and therefore the bounding boxes, of at least one further layer, and ultimately the objects calculated in at least one further layer, are also calculated depending on the refined object query of the fitted anchor position of the first layer. Here, the position of the refined object query is usually closer to the actual object than the position of the original object query. The distance between the position of the object query from which the capture is performed and the actual object will affect the accuracy of the capture in each layer. By fitting the position of the refined object query to the previous box parameter, the distance between the position of the (refined) object query and the actual object is reduced, and therefore more accurate capture is achieved.

[0014] By transforming the resulting feature vector with respect to the fitted anchor locations, it can still be used as an object query for evaluation in subsequent layers. Here, the shape of the resulting feature vector remains unchanged, and therefore known types of encoding can be used. In particular, when the aforementioned feedforward network, which has only two layers in this regard, is used, the transformation can be performed at a small additional cost. Furthermore, the same encoding of the anchor locations as for the first layer is used, and therefore no additional parameters are required.

[0015] Furthermore, the resulting feature vector is a position-based vector, and when processed by the decoder and adapted to the anchor position, more information about the object is sequentially obtained. The object information is encoded not only in the low-dimensional box parameters as in the prior art, but also in the potential feature space. In a further step, such vectors can then be propagated temporally and used, for example, for object tracking and prediction. As a result, the transformer can be used for downstream applications that assume object recognition and handle large point clouds.

[0016] Particularly, a significant reduction in distance is achieved during the first refinement of the object query. Thus, even between the first layer and the second layer of the transformer, the refinement of the object query already achieves a large effect. Preferably, the steps of calculating box parameters for the resulting feature vector, adapting the anchor position, and calculating the feature vector for the adapted anchor position by encoding and transforming the resulting feature vector from the adapted anchor position are performed for at least one further layer of the transformer other than the first layer, and in the above steps, a further layer is used instead of the first layer.

[0017] The term "first layer" should be understood in this specification as the first layer of the transformer to which this method is applied. Although it is advantageous to apply this method immediately to the first layer of the transformer, it is also conceivable to utilize this method from subsequent layers. In this case, this subsequent layer is interpreted as the "first" layer.

[0018] To train a transformer or a model of a transformer, preferably the following steps are performed. For the decoder output of each layer, a plurality of box parameter sets (preferably, the same number of box parameter sets as the number of object queries provided at the input of the decoder) are calculated. Further, the ground truth box parameters to be assigned to the closest estimated box parameters are provided. For this, preferably, the Hungarian method is applied. Unsuitable box parameters are assigned to the "non-object" class and discarded. Median regression, also called l1 loss, is applied to the deviation between the ground truth box parameters and the assigned estimated box parameters. Finally, the transformer is trained using median regression.

[0019] The training of the transformation of the resulting feature vectors for the adapted anchor positions, especially using a feed-forward network as described above, is trained independently of the transformer or model and can then be used with fixed weights. For the input data for the transformation and to calculate the ground truth, a trained transformer with fixed weights is used, which calculates the resulting feature vectors from the point cloud data as described above. These are then fed into the transformation, thereby obtaining the transformed resulting feature vectors. To obtain the ground truth, the estimation of the box parameters is applied to both the calculated resulting feature vectors and the transformed resulting feature vectors. Here, all box parameters other than the position difference are not changed. Finally, the transformed resulting feature vectors are adapted until the position difference of the box parameters for the new anchor positions becomes zero after the transformation, and thus the box parameters overlap with each other.

[0020] A computer program is designed to perform each step of the method, especially when executed on a computing device or a control device. This makes it possible to implement the method in a conventional electronic control device without the need for structural changes. For implementation, the computer program is stored in a machine-readable storage medium.

[0021] By loading a computer program into a conventional electronic control device, an electronic control unit designed to perform detection of a plurality of objects from point cloud data is obtained. Exemplary embodiments of the present invention are shown in the drawings and will be described in more detail in the following description.

Brief Description of the Drawings

[0022] [Figure 1a] It is a bird's-eye view visualizing the calculation of a bounding box according to the prior art. [Figure 1b] It is a bird's-eye view visualizing the calculation of a bounding box according to an embodiment of the method of the present invention. [Figure 2] It is a flowchart of an embodiment of the method according to the present invention. [Figure 3] It is a flowchart of converting a result feature vector with respect to an adapted anchor position according to the method of the present invention.

Mode for Carrying Out the Invention

[0023] FIG. 1a and FIG. 1b respectively show, in a bird's-eye view, a bounding box B gt according to ground truth, and an estimated bounding box B e calculated by an object detection method using a transformer, as well as the positions P y,0 and P y,1 of the object queries that are the starting points for the respective calculations. In FIG. 1a, the estimated bounding box B e is always calculated starting from the same position P y,0 of the object query. Due to the distance between the position P y,0 of the object query and the position of the object, that is, the position where the bounding box B gt [[ID=4B]]is arranged according to ground truth, inaccuracies occur in the calculation by the decoder of the transformer, and the estimated bounding box B e and the bounding box B gtThe two are significantly different. Figure 1b shows the results of the method according to the present invention. Estimated bounding box B in the first layer of the transformer. e The calculation is performed using the original position P of the object query. y,0 Starting from there, the process is carried out similarly to Figure 1a, which is not shown in Figure 1b. Next, as described below, the object query is refined, which is the bounding box B calculated in the first layer. e It is adapted to a new anchor position that depends on the second layer of the transformer, bounding box B shown here. e The calculation of the new position P of the refined object query y,1 It starts from there. This new position P of the refined object query. y,1 This is bounded by the actual object, i.e., the ground truth, according to the bounding box B. gt It is close to the location where it is located, and therefore, the estimated boundary box B e This allows for better determination and therefore more accurate detection of objects.

[0024] Figure 2 shows a flowchart of the method according to the present invention relating to two layers of a transformer. The same steps are indicated by the same reference numerals and will be described in detail only once. Hereinafter, s represents the layer number of the decoder of the transformer. i is used as a control variable for the feature vectors, and M feature vectors are provided.

[0025] First, the LiDAR sensor on vehicle F captures the environment. A visual representation of this received point cloud data is shown in 1. From the point cloud data, backbone 2 calculates feature vectors, which are then amplified by position encoding 3 using sine and cosine to finally produce key vectors k i and value vector v i It is supplied to the transformer's decoder 6 as such.

[0026] Simultaneously, the first anchor position is obtained from the point cloud data using a sampling method 4, such as sampling the farthest point.

[0027]

number

[0028] The result is calculated, and then Fourier encoding 5 is performed.

[0029]

number

[0030] Here, B is a matrix with entries in a normal distribution, and FFN represents a feedforward network consisting of two layers, including ReLU activation (Rectified Linear Unit).

[0031]

number

[0032] This is the computed feature vector, which is supplied to the transformer's decoder 6 as an object query. First anchor position

[0033]

number

[0034] The first set of feature vectors directly calculated from are represented by Y0, and the object query

[0035]

number

[0036] It consists of each object query.

[0037]

number

[0038] These function as slots for possible objects (shown by individual boxes in Figure 2). The transformer's decoder 6 consists of six layers s, each layer having eight attention heads. In the first layer s0 (s=0), the decoder 6 performs object queries.

[0039]

number

[0040] , as well as key vector k i and value vector v i From this, the resulting feature vector

[0041]

number

[0042] Calculate the object query.

[0043]

number

[0044] , key vector k i , and value vector v i The dimension is, for example, 256. This allows the detection of two objects, O1 and O2. The feedforward network 7 then analyzes the resulting feature vector of the first layer s0.

[0045]

number

[0046] From there, box parameters relating to objects O1 and O2

[0047]

number

[0048] Calculate the following, where Δx, Δy, and Δz are the anchor positions in 3D.

[0049]

number

[0050] This represents the difference in position relative to, where w, l, and h represent the sizes of objects O1 and O2 in three dimensions, and γ is the orientation of objects O1 and O2, and v x , v y represents the velocities of objects O1 and O2 in the horizontal plane, and cls represents the class identifier. Objects O1 and O2 have been detected and are shown here in the visual representation indicated by 8.

[0051] According to the present invention, refined VQ is performed for object queries. For this purpose, on the one hand, anchor position

[0052]

number

[0053] Adaptation 40 is performed, and the adapted anchor position for further layers s of decoder 6 is determined.

[0054]

number

[0055] Obtain the box parameters calculated in the first layer s0 of decoder 6.

[0056]

number

[0057] The positional differences Δx, Δy, and Δz are the first anchor position.

[0058]

number

[0059] This is added to the compliant anchor position.

[0060]

number

[0061] You can obtain this.

[0062]

number

[0063] Resulting Feature Vector

[0064]

number

[0065] From box parameters

[0066]

number

[0067] This can be obtained, which is the first anchor position

[0068]

number

[0069] It is located far away from and therefore has a high position difference Δx, Δy, Δz. By adapting the anchor position 40, the adapted anchor position

[0070]

number

[0071] This is obtained, and it is closer to the object. These are the positions of the suitable anchors

[0072]

number

[0073] Starting from there, encoding 50 is performed (this corresponds to encoding 5 above, please refer to that). This then allows object queries for further layers of the transformer's decoder 6.

[0074]

number

[0075] The feature vectors supplied as are obtained. On the other hand, the anchor alignment module (AAM) is used to obtain the resulting feature vector of the first layer s0.

[0076]

number

[0077] Transformation 90 is performed, which will be described in more detail with reference to Figure 3. Here, the transformed result feature vector

[0078]

number

[0079] This is obtained, and this is the adapted anchor position

[0080]

number

[0081] It will be aligned to that.

[0082]

number

[0083] Transformed result feature vector

[0084]

number

[0085] , and the above feature vector calculated by encoding 50

[0086]

number

[0087] Y s This is supplied to further layers s of the decoder as a set of feature vectors, represented as follows.

[0088]

number

[0089] Each object query

[0090]

number

[0091] and each transformed result feature vector

[0092]

number

[0093] The sum of these functions as slots for possible objects (represented by individual boxes in Figure 2). Thus, a total of M slots are obtained. Decoder 6, in further layers s, adapts the anchor positions as described above.

[0094]

number

[0095] Object queries for current layers that depend on

[0096]

number

[0097] , the transformed result feature vector of the previous layer s0

[0098]

number

[0099] , as well as key vector k i and value vector v i From this, the resulting feature vector

[0100]

number

[0101] Next, calculate the resulting feature vector.

[0102]

number

[0103] This is also supplied to the fast-forward network 7, which receives box parameters related to objects O1 and O2.

[0104]

number

[0105] This is calculated. At this point, due to the refinement of the object query VQ, the position differences Δx, Δy, and Δz calculated here are small. Figure 2 shows further refinement QV of the object query for additional layers. Query 100 determines whether further refinement QV should be performed, thereby achieving further improvement in detection accuracy at additional layers. Here, S r This indicates the layer on which refined QV should be performed.

[0106]

number

[0107] In this case, the result feature vector of the current layer s

[0108]

number

[0109] This is used as an object query for subsequent layers (not shown here). s∈S r In this case, the corresponding refined QV of the object query is executed. As mentioned above, on the one hand, anchor position

[0110]

number

[0111] The fitting 140 was performed, and the box parameters calculated in the current layer s of decoder 6 were

[0112]

number

[0113] The positional differences Δx, Δy, and Δz are the anchor positions.

[0114]

number

[0115] By adding this, the suitable anchor position

[0116]

number

[0117] This is obtained. This adapted anchor position

[0118]

number

[0119] Starting from there, encoding 150 is performed (this corresponds to encodings 5 ​​and 50 mentioned above; please refer to those), which then generates the feature vector

[0120]

number

[0121] This is obtained. On the other hand, the anchor alignment module (AAM) is used to obtain the resulting feature vector.

[0122]

number

[0123] Transformation 190 (which corresponds to transformation 90 mentioned above; see also Figure 3 for details) is performed, and as a result, the transformed feature vector is obtained.

[0124]

number

[0125] You can obtain this. Generally, the set of feature vectors Y supplied to the layers s of decoder 6. s This is set as follows, depending on the number of layers and whether refined object queries (QVs) are performed on them:

[0126]

number

[0127] Here, j = max{l|l <s∧l∈S r ) and this is the current object query obtained by encoding 5, 50, 150 in the second case (second row).

[0128]

number

[0129] This is always supplied to decoder 6. The last line shows the case for the first layer s0. Figure 3 shows the flowchart of transformation 90. Result feature vector

[0130]

number

[0131] This is supplied to a fast-forward network consisting of two layers 91 and 92, including ReLU activation. The fast-forward network sets the positional differences Δx, Δy, and Δz from the previous anchor position to zero, and the two layers 91 and 92 are then processed to produce the resulting feature vector.

[0132]

number

[0133] It is trained to change the input. Layers 91 and 92 themselves are transformations of the input using the learned weights. After the first layer 91, an intermediate representation with dimension h is obtained. After the second layer 92, the transformed result feature vector is obtained.

[0134]

number

[0135] This is the resulting feature vector.

[0136]

number

[0137] It has the same dimension d as the original result feature vector.

[0138]

number

[0139] A bypass connection 94 is created to ensure that no information is lost. The above description can also be applied to transformations involving further layers, such as transformation 190.

Claims

1. Multiple objects from point cloud data using a transformer including an attention model (O １ , O ２ A method for detecting the plurality of objects (O １ , O ２ The state of ) is stored in the attention model within the feature space, and the following steps are taken: The step involves calculating feature vectors from point cloud data using a backbone (2), wherein the feature vectors are key vectors (k) related to the transformer. ｉ ) and value vector (v ｉ ) functions as a step, From the point cloud data, the first layer (s) of the transformer is obtained by sampling method (4). ０ The first anchor position related to ) [Math 1] Steps to calculate The first anchor position is determined using encoding (5) [Math 2] The step is to calculate a feature vector from the first layer (s) of the transformer, wherein the feature vector is obtained from the first layer (s) of the transformer. ０ Object queries related to ) [Math 3] A step that functions The first layer (s ０ ) of the decoder (6) of the transformer generates the object query [Math 4] and the key vector (k ｉ ) and value vector (v ｉ ) from the first layer (s) of the transformer ０ ) Result feature vector [Math 5] The steps to be calculated, The first layer (s) of the transformer ０ ) the aforementioned result feature vector [Math 6] Box parameters related to [Number 7] The step of calculating (7) The aforementioned box parameter [Number 8] The positional difference is the position of the first anchor. [Number 9] By being added to the anchor position with respect to at least one further layer(s) of the transformer, [Number 10] Steps to make it conform (40, 140), The adapted anchor position is determined by encoding (50, 150). [Math 11] The step is to calculate a feature vector from which the feature vector relates to the object query of the at least one further layer(s) of the transformer. [Math 12] A step that functions The aforementioned suitable anchor position [Number 13] The result feature vector of the first layer relating to the above [Number 14] The step is to convert (90), and the converted result feature vector [Number 15] However, the step functions as an object query relating to the at least one further layer(s) of the transformer, The decoder (60) of the transformer processes the first layer (s ０ The transformed result feature vector of ) [Number 16] , the calculated object query of the at least one further layer(s) [Number 17] , and the key vector (k ｉ ) and value vector (v ｉ ) from the result feature vector in the at least one further layer(s) of the transformer [Number 18] Calculation steps A method that includes this.

2. Box parameters relating to the aforementioned result feature vector [Number 19] Steps to calculate The aforementioned anchor position [Number 20] Steps to adapt (140), The aforementioned suitable anchor position [Math 21] From the above result feature vector [Number 22] The encoding (150) and conversion (190) of the adapted anchor position [Number 23] Feature vectors [Number 24] Calculation steps However, this is performed with respect to at least one further layer(s). The method according to feature 1.

3. To train the aforementioned transformer, follow these steps: A step of estimating multiple box parameter sets for the decoder output of each layer, The step of assigning the ground truth box parameter to the nearest estimated box parameter, The step of applying median regression to the deviation between the box parameter of the Grand Truth and the assigned estimated box parameter. Steps to train the transformer using the median regression described above. The method according to claim 1, characterized in that the following is performed.

4. The aforementioned suitable anchor position [Number 25] The aforementioned result feature vector [Number 26] The method according to claim 1, characterized in that the conversion (90, 190) is realized by a feedforward network consisting of two layers (91, 92) including ReLU activation.

5. The aforementioned suitable anchor position [Number 27] Regarding the aforementioned result feature vector [Number 28] To train the aforementioned conversion (90, 190), follow these steps: A step of calculating a result feature vector from the point cloud data, Steps to transform the aforementioned result feature vector, Steps include applying the estimation of the box parameters to the resulting feature vector and the transformed resulting feature vector, Steps to fit the transformed result feature vector until the position difference of the box parameters is zero. The method according to 4, characterized in that the following is performed.

6. The method according to claim 1, characterized in that the point cloud data is acquired by LiDAR.

7. The method according to claim 1, characterized in that the point cloud data is recorded from a vehicle (F).

8. A computer program designed to perform each step of the method according to any one of claims 1 to 7.

9. A machine-readable storage medium on which the computer program described in claim 8 is stored.

10. An electronically controlled device designed to perform detection of multiple objects from point cloud data using a transformer including an attention model, according to the method of any one of claims 1 to 7.

Images