A method for detecting objects from point cloud data using a transformer with an attention model
The method uses a backbone neural network and transformer decoder with refined anchor positions to enhance object detection in point cloud data, addressing inefficiencies in traditional transformer encoders and improving accuracy in sparse point clouds for applications like object tracking and prediction.
Patent Information
- Application Number
- JP2025517959
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-09-28
- Filing Date
- 2023-09-12
- Publication Date
- 2025-09-11
- Estimated Expiration
- 2043-09-12
AI Technical Summary
Existing methods for object detection in point cloud data, particularly in the context of autonomous driving, are inefficient and computationally costly due to the use of traditional transformer encoders, which are restrictive and unsuitable for sparse point clouds.
A method using a backbone neural network to calculate feature vectors, followed by a transformer decoder, with anchor positions refined through sampling and encoding, allowing for accurate object detection by reducing the distance between anchor positions and actual objects, and utilizing a feedforward network for position alignment.
This approach significantly enhances object detection accuracy by reducing computational costs and improving detection precision, especially in sparse point clouds, enabling effective use in downstream applications like object tracking and prediction.
Smart Images
Figure 2025530550000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a method for detecting multiple objects from point cloud data using a transformer that includes an attention model. [Background technology]
[0002] Today, object detection is performed in imaging sensors. There are typically multiple objects in the recorded environment, and therefore multiple object detection is performed. For example, object detection by sensors is used in vehicles to capture other vehicles, other road users, and infrastructure. This data can be used for (partially) automated or autonomous driving.
[0003] The concept of using transformers for object detection has been explored in recent years. Transformers were first described in the context of language processing in the paper "Attention is all you need" by Ashish Vaswani et al., arXiv preprint arXiv:1706.03762, 2017. Object detection involves computing a bounding box describing each object and its box parameters, e.g., its position, size, orientation, velocity, and / or class identifier. Transformers can also be used for downstream applications, e.g., object tracking, prediction, or (path) planning. When using transforms for object detection, the traditional post-processing approach of suppressing duplicate detections can be de-emphasized. Previously, such transforms have been applied to image data, for example. However, their use on large point clouds, such as those encountered in the context of autonomous and automated driving, is unknown. Summary of the Invention
[0004] The present invention relates to a method for detecting multiple objects from point cloud data using a transformer including an attention model. The point cloud data is captured by, for example, a LiDAR. However, the method is not limited to LiDAR, and other sensor types can also be used. Preferably, a sensor or sensor system is located on a vehicle so that point cloud data is recorded from the vehicle.
[0005] This method includes the following steps: First, feature vectors are calculated from point cloud data. This is done by a backbone, rather than by a transformer encoder, as is usually the case. The backbone is a neural network used to extract features from measured data or to convert inputs into specific feature representations that can be further processed later. Thus, the transformer encoder can be eliminated. Preferably, the output of the backbone is reformatted to obtain a sequence of feature vectors with a predetermined length. By using the backbone to calculate feature vectors, the input sequence in self-attention is less restrictive than when using a transformer encoder, and a grid-based backbone such as PointPillars allows for the selection of a sufficiently small cell size. The feature vectors thus calculated are then fed into the transformer and serve as key and value vectors for calculating cross-attention.
[0006] Furthermore, first anchor positions for the first layer of the transformer are calculated from the point cloud data using a sampling method, such as farthest point sampling (FPS). A feature vector is calculated from the first anchor positions using encoding, such as Fourier encoding. The encoding can be completed, inter alia, by a feedforward network. The calculated feature vector serves as an object query for the first layer of the transformer decoder. The object query for the anchor positions serves as a starting point for the object search. However, the search is not limited to these anchor positions; objects far from these anchor positions are also detected. The anchor positions do not correspond to anchor boxes used in other detection techniques. Therefore, the object query for the transformer is data-dependent, rather than trained as is typically the case. This is particularly advantageous for sparse point clouds, where finding positions that actually contain data typically requires significant computational resources. Such sparse point clouds are particularly common in LiDAR measurements. The object query calculated from the anchor positions serves as a slot for possible objects.
[0007] In the first layer, the decoder of the transformer computes a result feature vector (also called the decoder output vector) from the object query (i.e., the feature vector mentioned above) and the key vector and value vector (i.e., the feature vector mentioned at the beginning).
[0008] From the resulting feature vector, box parameters for a bounding box describing the object, i.e., for example, the object's position or position difference relative to the anchor position, size, direction, velocity, and / or class identifier, are calculated using a feedforward network. In this regard, preferably, a feedforward network different from the one described above is used to calculate the object query, this feedforward network having different weightings.
[0009] Then, the anchor position is adapted using the calculated box parameter for processing at least one further layer of the transformer. When adapting the anchor position, the position difference of the box parameter calculated from the result feature vector of the first layer of the transformer is added to the first anchor position. Generally, the box parameter can be obtained for the result feature vector of the first layer, and the box parameter is far from the first anchor position and therefore has a large position difference. By adapting the anchor position, an adapted anchor position is obtained, which is closer to the actual object. From the adapted anchor position, a feature vector is calculated using the encoding as described above, which serves as an object query for at least one further layer of the transformer.
[0010] In order to propagate the information of the high-dimensional result feature vector of the first layer in addition to the adapted anchor position, a transformation of the result feature vector of the first layer with respect to the adapted anchor position is performed. Here, the result feature vector is aligned to the adapted anchor position. Advantageously, this is realized by a feedforward network consisting of two layers including ReLU activation (Rectified Linear Unit). Since only a feedforward network with two layers is used here, this only brings about a small additional cost.
[0011] The above steps of adapting anchor locations, calculating feature vectors from the adapted anchor locations, and transforming the resulting feature vectors are referred to herein as refining the object query.
[0012] The transformed result feature vectors and the calculated object query, in particular their vector sum, are then fed as input for at least one further layer to the Transformer decoder, where they serve as slots for possible objects. The Transformer decoder calculates result feature vectors in the at least one further layer from the transformed result feature vectors calculated for the previous layer, from the calculated object query (calculated from the adapted anchor positions as described above), and from the key vectors and value vectors mentioned at the beginning.
[0013] As a result, the feature vectors of at least one further layer, and therefore the bounding boxes, and ultimately the objects calculated in at least one further layer, are calculated depending on the refined object query of the adapted anchor positions of the first layer. Here, the position of the refined object query is usually closer to the actual object than the position of the original object query. The distance between the position of the object query from which the capture is performed and the actual object affects the accuracy of the capture in each layer. By adapting the position of the refined object query to the previous box parameters, the distance between the position of the (refined) object query and the actual object is reduced, and therefore more accurate capture is achieved.
[0014] By transforming the resulting feature vector with respect to the adapted anchor positions, it can be subsequently used as an object query for evaluation in subsequent layers. Here, the shape of the resulting feature vector is not changed, and therefore a known type of encoding can be used. In particular, when the feedforward network described above with only two layers is used in this regard, the transformation can be performed with little additional cost. Furthermore, the same encoding of the anchor positions as for the first layer is used, and therefore no additional parameters need to be used.
[0015] Furthermore, the resulting feature vector is a position-based vector, which, when processed by the decoder and adapted to anchor positions, progressively captures more information about the object. Object information is encoded in the latent feature space, rather than being encoded only in low-dimensional box parameters as in the conventional method. In a further step, such vectors can then be propagated in time and used, for example, for object tracking and prediction. As a result, the transformer can also be used in downstream applications that presuppose object recognition and handle large point clouds.
[0016] In particular, a significant reduction in distance is achieved during the first refinement of the object query, and therefore the refinement of the object query already achieves a significant effect even between the first and second layers of the Transformer. Preferably, the steps of calculating box parameters for the result feature vector, adapting anchor positions, and calculating feature vectors for the adapted anchor positions by encoding and transforming the result feature vector from the adapted anchor positions are performed for at least one further layer of the Transformer other than the first layer, and in the above steps, the further layer is used instead of the first layer.
[0017] The term "first layer" is to be understood herein as the first layer of the transformer to which the method is applied. While it is advantageous to apply the method immediately to the first layer of the transformer, it is also conceivable to utilize the method from a subsequent layer onwards. In this case, this subsequent layer is interpreted as the "first" layer.
[0018] To train a transformer or a model of a transformer, the following steps are preferably performed: For the decoder output of each layer, multiple box parameter sets are calculated (preferably as many box parameter sets as object queries are provided at the decoder input); Furthermore, ground truth box parameters are provided which are assigned to the closest estimated box parameters; for this, preferably the Hungarian method is applied; box parameters that do not suit are assigned to the "non-object" class and discarded; median regression, also known as l1 loss, is applied to the deviations between the ground truth box parameters and the assigned estimated box parameters; Finally, the transformer is trained using median regression.
[0019] In particular, training of the transformation of the result feature vector with respect to the adapted anchor position using a feedforward network as described above can be trained independently of the transformer or model and then used with fixed weights. To calculate the input data for the transformation and the ground truth, the trained transformer with fixed weights is used, which calculates the result feature vector from the point cloud data as described above. These are then fed to the transformation, thereby obtaining the transformed result feature vector. To obtain the ground truth, box parameter estimation is applied to both the calculated result feature vector and the transformed result feature vector. Here, all box parameters other than the position difference remain unchanged. Finally, the transformed result feature vector is adapted until the position difference of the box parameters with respect to the new anchor position becomes zero after the transformation, and thus the box parameters overlap each other.
[0020] The computer program is specifically designed to execute the steps of the method when executed on a computing or control device, which allows the method to be implemented in conventional electronic control devices without the need for structural changes. For implementation, the computer program is stored on a machine-readable storage medium.
[0021] By loading a computer program into a conventional electronic control device, an electronic control unit designed to perform detection of multiple objects from point cloud data is obtained. Exemplary embodiments of the invention are illustrated in the drawings and are explained in more detail in the following description. [Brief explanation of the drawings]
[0022] [Figure 1a] FIG. 1 is a bird's-eye view visualizing bounding box calculation according to the prior art; [Figure 1b] 1 is a bird's-eye view visualizing the calculation of a bounding box according to an embodiment of the method according to the invention; [Figure 2] 1 is a flow diagram of an embodiment of a method according to the present invention; [Figure 3] 1 is a flow diagram of transforming a result feature vector with respect to adapted anchor locations according to a method in accordance with the present invention; DETAILED DESCRIPTION OF THE INVENTION
[0023] Figure 1a and Figure 1b show the ground truth bounding box B gt , and the estimated bounding box B calculated by the object detection method using a transformer. e , and the object query position P from which the calculations are performed. y,0 , P y,1 In Fig. 1a, the estimated bounding box B e is always the same position P y,0 The object query position P y,0 and the object position, i.e., the bounding box B gt The distance between the location of B and the estimated bounding box B e and the bounding box B according to the ground truth gtFigure 1b shows the results of our method. The estimated bounding box B in the first layer of the transformer e is calculated from the original position P y,0 This is done in the same way as in Figure 1a, starting from the bounding box B computed in the first layer, which is not shown in Figure 1b. As described below, object query refinement is then performed, which is done by using the bounding box B computed in the first layer. e The second layer of the transformer is fitted with a new anchor position that depends on the bounding box B shown here. e The calculation of the refined object query new position P y,1 This refined object query is then calculated by starting from this new position P y,1 is the actual object, i.e., the bounding box B according to the ground truth. gt is located near the estimated bounding box B e can be better determined and therefore objects can be detected more accurately.
[0024] Figure 2 shows a flow diagram of the method according to the invention for two layers of a transformer. Identical steps are designated with the same reference numerals and are described in detail only once. In the following, s denotes the number of the layer of the decoder of the transformer. i is used as a control variable for the feature vectors, and M feature vectors are provided.
[0025] First, the LiDAR sensor of the vehicle F captures the environment. A visual representation of this received point cloud data is shown at 1. From the point cloud data, the backbone 2 computes a feature vector, which is then amplified by position encoding using sine and cosine 3, finally resulting in a key vector k i and the value vector v i to the decoder 6 of the transformer.
[0026] At the same time, the first anchor position is obtained from the point cloud data by a sampling method 4 such as farthest point sampling.
[0027]
number
[0028] is calculated and then Fourier encoding 5 is performed.
[0029]
number
[0030] where B is a matrix with normally distributed entries, and FFN stands for feed-forward network, here consisting of two layers containing ReLU activation (Rectified Linear Unit).
[0031]
number
[0032] is the computed feature vector, which is fed to the decoder 6 of the transformer as the object query. First anchor position
[0033]
number
[0034] The first set of feature vectors, which are directly calculated from
[0035]
number
[0036] Each object query consists of
[0037]
number
[0038] s serve as slots (indicated by individual boxes in Figure 2) for possible objects. The decoder 6 of the transformer consists of six layers s, each with eight attention heads. In the first layer s0 (s=0), the decoder 6 calculates the object query
[0039]
number
[0040] , and the key vector k i and the value vector v i From the resulting feature vector
[0041]
number
[0042] Calculate the object query.
[0043]
number
[0044] , key vector k i , and the value vector v i The dimension of is, for example, 256. This results in the detection of two objects O1 and O2. The feedforward network 7 uses the result feature vector s0 of the first layer s0 as
[0045]
number
[0046] From the box parameters for objects O1 and O2,
[0047]
number
[0048] where Δx, Δy, and Δz are the anchor positions in three dimensions.
[0049]
number
[0050] represents the difference in position relative to the object O1, O2, w, l, and h represent the size of the object O1, O2 in three dimensions, γ is the orientation of the object O1, O2, and v x , v y represents the velocity of the objects O1, O2 in the horizontal plane, and cls represents the class identifier. The objects O1, O2 are detected and are shown here with a visual representation denoted 8.
[0051] According to the present invention, a refined VQ of the object query is performed. For this purpose, on the one hand, the anchor positions
[0052]
number
[0053] is adapted 40 to obtain the adapted anchor position s for the further layer s of the decoder 6.
[0054]
number
[0055] The box parameters calculated in the first layer s0 of the decoder 6 are obtained as follows:
[0056]
number
[0057] The position differences Δx, Δy, and Δz are the first anchor position
[0058]
number
[0059] , which results in an adapted anchor position
[0060]
number
[0061] is obtained.
[0062]
number
[0063] Result feature vector
[0064]
number
[0065] From box parameters
[0066]
number
[0067] can be obtained, which is the first anchor position
[0068]
number
[0069] The anchor positions 40 are located far away from the target position and therefore have high position differences Δx, Δy, Δz.
[0070]
number
[0071] is obtained, which is closer to the object. The locations of these adapted anchors
[0072]
number
[0073] Starting from , a further encoding 50 is performed (which corresponds to encoding 5 above, see there), thereby generating the object query s in a further layer s of the decoder 6 of the transformer.
[0074]
number
[0075] The feature vector is obtained as follows: Meanwhile, the anchor alignment module AAM (anchor alignment module) is used to align the result feature vector of the first layer s0.
[0076]
number
[0077] is performed, which will be described in more detail with reference to FIG. 3. Here, the resulting transformed feature vector
[0078]
number
[0079] which is the adapted anchor position
[0080]
number
[0081] is aligned to
[0082]
number
[0083] The transformed result feature vector
[0084]
number
[0085] , and the above feature vector calculated by encoding 50
[0086]
number
[0087] is Y s The feature vectors are fed to a further layer s of the decoder as a set of feature vectors denoted as
[0088]
number
[0089] Each object query
[0090]
number
[0091] and each transformed result feature vector
[0092]
number
[0093] The sum of s and s serves as slots for possible objects (represented by individual boxes in Fig. 2). Thus, a total of M slots are obtained. The decoder 6 then calculates the adapted anchor positions s as described above in a further layer s.
[0094]
number
[0095] The object query for the current layer s depends on
[0096]
number
[0097] , the transformed result feature vector of the previous layer s0
[0098]
number
[0099] , and the key vector k i and the value vector v i From the resulting feature vector
[0100]
number
[0101] Then, the resulting feature vector
[0102]
number
[0103] are also fed to the fast-forward network 7, which then calculates the box parameters for the objects O1 and O2.
[0104]
number
[0105] At this time, due to the refined VQ of the object query, the calculated position differences Δx, Δy, and Δz are small. Further refinement QV of the object query for further layers is shown in Figure 2. The query 100 determines whether further refinement QV should be performed, thereby achieving further improvement in the accuracy of detection at further layers. Here, S r indicates the layer on which the refined QV should be performed.
[0106]
number
[0107] If , the resulting feature vector of the current layer s
[0108]
number
[0109] is used as the object query for subsequent layers (not shown here). s∈S r If , the corresponding refinement QV of the object query is performed. As mentioned above, on the one hand, the anchor position
[0110]
number
[0111] The fitting 140 of the box parameter s is performed and the box parameter s is calculated in the current layer s of the decoder 6.
[0112]
number
[0113] The positional differences Δx, Δy, and Δz are the anchor positions.
[0114]
number
[0115] is added to the adapted anchor position
[0116]
number
[0117] This adapted anchor position is obtained.
[0118]
number
[0119] Starting from , a further encoding 150 is performed (which corresponds to encodings 5 and 50 above, see them), resulting in the feature vector
[0120]
number
[0121] On the other hand, the anchor alignment module AAM is used to obtain the result feature vector
[0122]
number
[0123] , a transformation 190 (which corresponds to the transformation 90 described above, see also FIG. 3 ) is performed, resulting in a transformed feature vector
[0124]
number
[0125] is obtained. In general, a set of feature vectors Y fed to layer s of the decoder 6 s is set as follows, depending on the number of layers and whether object query refinement QV is performed on them:
[0126]
number
[0127] where j=max{l|l <s∧l∈S r ), which in the second case (second row) is the current object query obtained by encoding 5, 50, 150.
[0128]
number
[0129] is always fed to the decoder 6. The last row shows the case for the first layer s0. Figure 3 shows a flow diagram of the transformation 90. Resulting feature vector
[0130]
number
[0131] is fed into a fast-forward network consisting of two layers 91 and 92 containing ReLU activations. The fast-forward network generates the result feature vectors 91 and 92 such that the positional differences Δx, Δy, and Δz with respect to the previous anchor position are set to zero.
[0132]
number
[0133] The layers 91 and 92 are themselves transformations of the input with learned weights. After the first layer 91, an intermediate representation with dimension h is obtained. After the second layer 92, the transformed resulting feature vector
[0134]
number
[0135] is obtained, which is the received result feature vector
[0136]
number
[0137] Furthermore, the original resulting feature vector
[0138]
number
[0139] A bypass connection 94 to is created to ensure that no information is lost. The above description can also be applied to transformations on further layers s, such as transformation 190.
Claims
1. Recognizing multiple objects (O) from point cloud data using a transformer with an attention model １ , O ２ ) detection, comprising: １ , O ２ ) are stored in the model in feature space and the following steps are performed: A step of calculating a feature vector from the point cloud data by a backbone (2), wherein the feature vector is a key vector (k ｉ ) and the value vector (v ｉ ) function as a step, From the point cloud data, a first layer (s ０ ) the first anchor position [Equation 1] Calculating The first anchor position is determined using encoding (5). [Equation 2] and the feature vector is calculated from the first layer (s ０ ) object query [Equation 3] Functioning as a step, The first layer (s) of the transformer decoder (6) ０ ) by the object query [Equation 4] and the key vector (k ｉ ) and the value vector (v ｉ ) to the first layer (s ０ ) resulting feature vector [Equation 5] Calculating The first layer (s ０ ) the result feature vector [Equation 6] Box parameters for [Equation 7] Calculating (7) The box parameter [Equation 8] The position difference is the first anchor position [Equation 9] by adding to the anchor position for at least one further layer(s) of the transformer [Equation 10] fitting (40, 140) The encoding (50, 150) determines the adapted anchor position. [0011] wherein said feature vector is a feature vector of an object query for said at least one further layer(s) of said transformer. [0012] Functioning as a step, the adapted anchor position [0013] The resulting feature vector of the first layer for [0014] Transforming (90) the resulting feature vector [Equation 15] serves as an object query for the at least one further layer(s) of the transformer; The decoder (60) of the transformer ０ ) the transformed result feature vector [0016] , the computed object query of the current layer(s) [Equation 17] , and the key vector (k ｉ ) and the value vector (v ｉ ) to generate a resultant feature vector in the at least one further layer(s) of the transformer. [Equation 18] Step of calculating A method comprising:
2. box parameters for the resulting feature vector [Equation 19] Calculating The anchor position [Equation 20] adapting (140) the adapted anchor position [0000] From the result feature vector [Equation 22] The adapted anchor position is obtained by encoding (150) and transforming (190) [Equation 23] With respect to the feature vector [0000] Step of calculating is performed with respect to at least one further layer(s) 2. The method of claim 1 .
3. To train the transformer, follow these steps: estimating multiple box parameter sets for the decoder output of each layer; assigning ground truth box parameters to the closest estimated box parameters; applying a median regression to the deviations between the box parameters of the ground truth and the assigned estimated box parameters. training the transformer with the median regression 3. The method according to claim 1, wherein the following is performed:
4. the adapted anchor position [Equation 25] The resulting feature vector for [Equation 26] 4. The method according to claim 1, wherein the transformation (90, 190) of is realized by a feedforward network consisting of two layers (91, 92) including ReLU activations.
5. the adapted anchor position [0000] With respect to the result feature vector [0000] To train the transformation (90, 190) of calculating a result feature vector from the point cloud data; transforming the resulting feature vector; applying the box parameter estimation for the resulting feature vector and the transformed resulting feature vector; Adapting the transformed resulting feature vectors until the positional difference of the box parameters is zero.
5. The method of claim 4, wherein:
6. 6. The method according to any one of claims 1 to 5, wherein the point cloud data is acquired by LiDAR.
7. 7. The method according to any one of claims 1 to 6, characterized in that the point cloud data is recorded from a vehicle (F).
8. A computer program designed to carry out the steps of the method according to any one of claims 1 to 7.
9. A machine-readable storage medium on which the computer program according to claim 8 is stored.
10. An electronic control device designed to perform detection of multiple objects from point cloud data using a transformer including an attention model according to the method of any one of claims 1 to 7.
Citation Information
Patent Citations
Object detection for a rotational sensor
US20220299649A1
Systems and methods for virtual and augmented reality
WO2021097318A1