Method for detecting and / or tracking and / or predicting position of plurality of objects

Abstract

The invention relates to a method for detecting and / or tracking and / or predicting the position of a plurality of objects on the basis of point cloud data by means of a Transformer with an attention model, the states of the tracked objects being stored in a feature space. The method comprises the steps that feature vectors are calculated from the point cloud data through a backbone network, and the feature vectors are used as key vectors and value vectors for the Transform. A result feature vector at a point in time is ascertained on the basis of an object query at the point in time and a key vector and a value vector, and the result feature vector is assigned to regression heads, each regression head containing regression parameters of a group (20-25) consisting of at least one category (10-19) of the object.

Description

TECHNICAL FIELD

[0001] The invention relates to a method for detecting and / or tracking and / or position prediction of a plurality of objects based on point cloud data by means of a Transformer with an attention model. Furthermore, the invention also relates to a computer program which implements all steps of the method and to a machine-readable data carrier on which the computer program is stored. Finally, the invention relates to an electronic control device which is set up for detecting and / or tracking and / or position prediction of a plurality of objects based on point cloud data by means of a Transformer with an attention model by means of the above-mentioned method. BACKGROUND

[0002] In publicly available data sets, different classes of objects are usually detected and their positions tracked, for example cars, bicycles and pedestrians. There are various approaches for achieving this task, in particular involving the extraction of bounding box parameters, i.e. the position, size and orientation of the object.

[0003] A group of 3D object detectors is based on anchor boxes. Here, the measurement region is divided into a grid and in each grid cell one anchor box is placed for each possible object class separately. The class of the object is thus already defined when the anchor box is placed. The bounding box parameters can thus be extracted by means of a regression head which is trained specifically for this class. It has proven advantageous here to merge similar classes, for example bicycles and motorcycles, into a group and to estimate this group in a common regression head.

[0004] In query-based approaches, the bounding box parameters are usually extracted from the feature vectors which have been output by the Transformer by means of a learned regression head. Here, it does not matter what the class of the object is. Instead, one common regression head is used for all classes. Similar classes cannot therefore be merged into a group as when using anchor boxes. SUMMARY

[0005] The invention relates to a method for detecting and / or tracking and / or position prediction of a plurality of objects based on point cloud data by means of a Transformer with an attention model. The point cloud data is detected for example by a laser radar. However, the method is not limited to laser radars, but other sensor types can also be used. Preferably, the sensor or sensor system is arranged on a vehicle in order to acquire the point cloud data from the vehicle.

[0006] The state of the object is stored in a feature space. The method comprises the following steps: First, a feature vector is calculated from the point cloud data, in particular by means of a backbone network. A backbone network is a neural network which is used to extract features from measured data or to bring an input into a specific feature representation which can then be processed further. In particular, a mesh-based backbone network is used. Alternatively, a point-based backbone network can also be used. The encoder of the Transformer can be omitted. However, if the size of the point cloud allows, a Transformer encoder can also be used. The feature vector thus calculated is then fed to the Transformer and serves as key vector and value vector for cross-attention.

[0007] Furthermore, an object query is provided. The object query can be, for example, a learned vector. Preferably, however, the object query is obtained by first providing anchor positions. In particular, the anchor positions are calculated from the point cloud data. To this end, the anchor positions can be calculated, in particular, by a sampling approach, such as Farthest-Point-Sampling (FPS), or by an estimation approach. Since the anchor positions are calculated from the data acquired, these anchor positions change at each time point (frame). Alternatively, the anchor positions can also be learned.

[0008] In particular, a feature vector can be calculated from the anchor positions by means of an encoding, for example a Fourier encoding. The feature vector thus calculated serves as an object query for the Transformer. The object query of the anchor positions serves as a starting point for searching for objects. However, the search is not limited to the anchor positions, but also detects objects at a distance from the anchor positions. The anchor positions do not correspond to anchor boxes as used in other detection approaches. Thus, the object query for the Transformer is data-dependent and not learned. This is particularly advantageous in the case of sparse point clouds, since otherwise many computing resources are wasted in searching for positions which actually have data. Such sparse point clouds arise, in particular, when measuring by means of a laser radar. The object query calculated from the anchor positions serves as a slot for possible objects.

[0009] At a certain time point (frame), a result feature vector, also referred to as decoder output vector, is calculated on the basis of the object query which has been calculated at the first time point, i.e. the feature vector described above, on the basis of the key vector and the value vector which have been calculated at the time point, i.e. the feature vector described at the beginning. The calculation takes place, in particular, by means of a decoder of the Transformer. From the result feature vector, bounding box parameters describing the object are calculated, for example the position, size, orientation, velocity and / or class identifier of the object, in particular with respect to the anchor positions.

[0010] The type of decoder used is not important, as long as the scheme for object recognition is query-based. For example, deformable attention, standard Transformer attention, or a simple sampling mechanism can be used. Similarly, the type of backbone network used for preprocessing and whether a Transformer encoder is used are also irrelevant.

[0011] The resulting feature vectors are assigned to regression heads, where each regression head contains regression parameters for groups comprised of at least one object category. It can be configured that each group contains multiple categories, or that some groups contain only one category while others contain multiple categories. Here, objects are understood both as three-dimensional objects, such as cars or pedestrians, and as two-dimensional objects, such as lane markings. The number of groups and the precise classification of categories can be changed as required. Furthermore, other object categories can be identified.

[0012] Preferably, the regression head is trained by assigning objects estimated from object queries to the ground truth. Here, the estimates are compared to the ground truth bounding boxes, the biases are determined and filled into the cost matrix, and the assignments are selected such that the biases are minimized overall, wherein each ground truth object is compared only with the estimate from the regression head that matches the object. This results in the cost matrix maintaining its size. Furthermore, existing matching algorithms can continue to be used, which similarly ensures that at most one ground truth object can be assigned to each object query.

[0013] In some query-based methods, object queries are assigned to locations in space (anchor locations). In this case, the locations of the object queries can be refined after each layer in the decoder. For this purpose, it is preferable to move the location of the object query to where the object is estimated at that time point after each layer in the decoder. In a preferred embodiment, the location is moved to the weighted median of multiple object candidates. In another preferred embodiment, the location is moved to the location of the object candidate with the highest weight score or group score. The group score is used to weight the object candidates. This group score is derived as the sum of the category scores of the categories contained in the corresponding group. The category scores are calculated, in particular, by a category head trained in conjunction with the regression head.

[0014] Therefore, at the first time point, multiple objects are classified based on the point cloud data. If object tracking is to be performed, these objects must be identified at subsequent, successive time points. Here, sensors, sensor systems, or systems with sensors deployed on them, such as vehicles, may move at their own speed. In the following text, a self-speed compensation module is used to compensate for the self-speed, and this compensation is applied to the resulting feature vector to obtain a self-speed compensated, transformed result feature vector.

[0015] At this point, the transformed feature vector is fed into the Transformer's decoder as input for the next time step, where it is used as a slot for potential objects in addition to object queries. After each time step, the Transformer's decoder at subsequent time steps calculates the resulting feature vector based on the transformed feature vector computed at the previous time step, the object query computed at that time step as described above, and the key and value vectors computed at that time step as introduced at the beginning. This computation can continue over any number of time steps. Finally, as described above, the computed feature vector is assigned to an object, thereby tracking multiple objects.

[0016] Therefore, object tracking of multiple objects based on point cloud data with its own velocity compensation is obtained.

[0017] In one preferred embodiment, for all objects already assigned to a regression head group before the current time step, the regression heads for all other groups are masked after the current time step. This ensures that objects do not change their groups during object tracking. In another preferred embodiment, all objects are allowed to change their group assignments after the current time step. This allows multiple hypotheses about which group an object is assigned to to to be maintained over time, with the outputs of other regression heads stored. This enables categories to change during object tracking and allows assignment to multiple category hypotheses.

[0018] The computer program is configured to perform each step of the method, particularly when executed on a computing or control device. This computer program enables the implementation of the method in conventional electronic control devices without requiring structural modifications. Only the Transformer needs to be implemented on the electronic control device. For this implementation, the computer program is stored on a machine-readable memory medium.

[0019] The electronic control device is obtained by loading the computer program and Transformer onto a conventional electronic control device, the electronic control device being configured to perform detection and / or tracking and / or position prediction of multiple objects based on point cloud data using the method and a Transformer with an attention model. Attached Figure Description

[0020] Embodiments of the present invention are shown in the accompanying drawings and explained in more detail in the following description.

[0021] Figure 1 This illustrates the summarization of object categories to object groups in one embodiment of the method according to the invention.

[0022] Figure 2 The chart shows the length deviation within the object category.

[0023] Figure 3 A flowchart illustrating an embodiment of the method according to the present invention is shown.

[0024] Figure 4a The illustration schematically shows the refinement of the location of the object query in one embodiment of the method according to the invention.

[0025] Figure 4b The illustration schematically shows a refinement of the location of the object query in another embodiment of the method according to the invention. Detailed Implementation

[0026] Figure 1 This illustrates how, in one embodiment of the method according to the invention, detectable objects are grouped into categories. The categories are cars 10, pedestrians 11, traffic cones 12, trucks 13, construction vehicles 14, buses 15, trailers 16, roadblocks 17, bicycles 18, and motorcycles 19. The first group 20 contains car category 10. The second group 21 contains pedestrians and traffic cones categories 11 and 12. The third group 22 contains trucks and construction vehicles categories 13 and 14. The fourth group 23 contains buses and trailers categories 15 and 16. The fifth group 24 contains roadblocks category 17. The sixth group 25 contains bicycles and motorcycles categories 18 and 19.

[0027] One regression head is trained for each of these groups 20-25. In this method, object queries can find objects of any category, therefore, these object queries are compatible with all regression heads.

[0028] During training, objects estimated from object queries are matched with the ground truth. To do this, each estimated... With each benchmark truth enclosure A comparison is made where m = 1…M and k = 1…K. The deviations are determined and filled into a cost matrix of size K×M, and the associations are selected such that the deviations are minimized overall. With N regression heads obtained, N potential objects are then generated from each of the M object queries. Where n=1…N, m=1…M. All potential objects should be considered in the assignment without increasing the cost matrix. Furthermore, each query should be assigned to at most one baseline truth object. To meet these requirements, each with a category… The benchmark truth object Each is compared only with the estimate from the regression head that matches the object. The comparison is performed, where m = 1…M. This results in the cost matrix still having a size of K×M. Furthermore, the existing matching algorithm can continue to be used, which similarly ensures that at most one baseline truth object can be assigned to each query.

[0029] Figure 2 The diagram shows the average length *l* of objects in each of these groups 20-25 and their 25% / 75% length deviations. It can be seen that the length deviation within a group is smaller than the length difference between groups. This allows for good distinguishability between objects in different groups.

[0030] Figure 3 A flowchart illustrating the method according to the invention at two successive time points t and t+1 is shown. Hereinafter, i and k will be referred to as indexed loop variables.

[0031] The left side relates to the first time point t. First, the LiDAR sensor of vehicle F detects the environment. The visualization of these acquired point cloud data is marked with 30. The backbone network 31 calculates feature vectors from these point cloud data, which are then enhanced by position encoding 32, for example, using sine and cosine position encoding, and finally used as the key vector at the first time point t. Sum value vector The deformable encoder 33 of the Transformer is fed to the decoder 34 of the Transformer.

[0032] At the same time, the anchor position at the first time point t is obtained from the point cloud data through sampling methods or estimation methods 35, such as the farthest point sampling method. These anchor positions are then encoded using position coding 36, for example, Fourier coding: Here, It is a matrix with a normal distribution term, and FFN is a feedforward network consisting of two layers with ReLU activation. These are the calculated feature vectors, which are then fed to the Transformer's decoder 34 as object queries.

[0033] For the set of feature vectors output at the first time point t, Mark and query by object Composition. Each object query Used as a slot for a possible object (in) Figure 3 (Shown as a single box in the image). The Transformer's decoder 34, for example, consists of six layers, each with eight attention heads. Decoder 34 queries the object at the first time point t. and key vector Sum value vector To obtain the feature vector of the result Object query Dimensions, key vectors Dimensions and value vectors The dimension is, for example, 256. However, object queries... The number can also be related to the key vector Sum value vector The quantities are different.

[0034] Object query after each layer in decoder 34 Location To refine this, object queries will be performed. Location Move to where the object was estimated at that point in time. For this, one can use methods such as those described in F. Ruppel, F. Faion, C. Gläser and K. Dietmayer, "Transformers for Object Detection in Large Point Clouds," 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China, 2022, pp. 832-838. The method described in .1109 / ITSC55140.2022.9921840. However, if N object candidates for each object query are obtained through the group regression head at this point, then "where the query location should be refined" is not uniquely clear. This is in Figure 4a The method embodiment shown in the text solves this problem by querying the object's location according to the following formula. Move to the object candidate respectively At the weighted median, where n=1…N, m=1…M: Here, This represents the group score used to weight the candidate objects. The group score is derived as the sum of the category scores of the categories contained within the corresponding group. The category scores are calculated using a category head trained together with this regression head. This method can be used both during training and inference.

[0035] In another embodiment, preferably during inference, the object query location can also be moved to the location of the object candidate with the highest group score. This is in Figure 4b It is displayed in the middle.

[0036] Therefore, two objects were detected at the first time point t. and In the first 37 of the regression, the resulting feature vector is... Assigning a regression header 37 to the object that matches it. , It was detected and is shown here in a visualization marked with 38.

[0037] exist Figure 3 The analysis and evaluation at the second time point t+1 is shown on the right. This second time point follows the first time point t one time step after a time step defined by the repetition rate of the data collected by the lidar sensor. Vehicle F moves at its own speed during this time step. This affects the analysis and evaluation of the measured data and the object... , The relative position and relative velocity. Perform self-velocity compensation 39. Self-velocity compensation 39 for the time step between the first time point t and the second time point t+1 can be performed using the method described in F. Ruppel, F. Faion, C. Gläser, and K. Dietmayer, Transformers for Object Detection in Large Point Clouds, in IEEEInt. Conf. Inteil. Transp. Syst., 2022. Here, the resulting feature vector from the first time point t... Anchor position The transformed feature vector is obtained by combining the pose change P. : Similar to the first time point t, here, the environment is first detected by the lidar sensor, and feature vectors are calculated from the point cloud data by the backbone network 31. These feature vectors are enhanced by position encoding 32, for example by using sine and cosine position encoding, and finally serve as the key vector for the second time point t+1. Sum value vector The data is fed to the Transformer's decoder 34 via the deformable encoder 33. Simultaneously, the anchor position at the second time point t+1 is obtained from the point cloud data using sampling method 35. These anchor positions are then Fourier encoded (36). This allows for the acquisition of object queries at the second time point t+1. .

[0038] For the feature vector set output at the second time point t+1, use Mark and query the object at the second time point t+1 and the transformed eigenvector The resulting eigenvectors after these transformations have been calculated at the first time point t, and this set of eigenvectors can be described as the union of the following: Query each object and the eigenvectors of each transformed result Used as a slot for a possible object (in) Figure 3 (Displayed using a single box).

[0039] Decoder 34 queries from objects The transformed feature vector and key vector Sum value vector To obtain the eigenvector of the result at the second time point t+1 Besides these two objects and In addition, a third object was detected. The path of the third object is continued to be tracked. In regression head 37, the resulting feature vector is... Assigned to matching object , , All regression headers 37. Here, object , , This is also shown in the visualization marked with 38. Therefore, multiple objects are probed at a subsequent time point t+1. , , .

[0040] In one embodiment of the method, for objects that have been identified at the first time point t and assigned to one of groups 20-25 , At the second time point t+1, the regression header 37 containing groups different from the already assigned groups is masked. Therefore, these objects... , Automatically retain their grouping attributes. Only objects first identified at the second time point will be included. Resulting feature vector Compare with all regression heads 37.

[0041] In another embodiment of the method, at the second time point t+1, for all objects... , , Re-determine their respective associations with one of the groups in groups 20-25.

[0042] Performing the above steps at subsequent points in time allows for the processing of multiple objects. , , Perform object tracking. Here, the above steps, inputs, and outputs are adjusted according to the points in time.

Claims

1. A method for applying attention to multiple objects based on point cloud data using a Transformer with an attention model. , , Methods for detection and / or tracking and / or location prediction, wherein, The tracked object ( , , The state of the feature space is stored in the feature space, and the method has the following steps: - A feature vector is computed from the point cloud data via a backbone network (31), wherein the feature vector is used as a key vector for the Transformer. , ) and value vector ( , ); - Object query based on time point (t) ) and key vector ( ) and value vector ( To obtain the eigenvector of the result at that time point (t) );as well as - The resulting feature vector ( ) is assigned to the regression head (37), where each regression head contains objects ( , , The regression parameters of a group (20-25) consisting of at least one category (10-19) of ).

2. The method according to claim 1, characterized in that: With the help of coding (36), based on anchor position ( , The feature vector is obtained by performing an object query. , ).

3. The method according to claim 2, characterized in that: The anchor position is calculated based on the point cloud data using a sampling method or estimation method (35). , ).

4. The method according to any one of claims 1 to 3, characterized in that: The regression head (37) is trained in the following manner: based on the object query ( , The estimated object ( , , The association between the estimate and the baseline truth is performed by comparing the estimate with the baseline truth bounding box, determining the deviation and filling the deviation into the cost matrix, and selecting the association such that the deviation is minimized overall, wherein each baseline truth object is compared only with the estimate from the regression head (37) matched with the object.

5. The method according to any one of claims 1 to 4, characterized in that: The object is queried after each layer in the decoder (34). The position of () ) Move to the estimated object at that point in time ( , , (There) 6. The method according to claim 5, characterized in that: The position ( Move to multiple object candidates respectively ( The weighted median of ).

7. The method according to claim 5, characterized in that: The position ( ) respectively move to the candidate objects with these objects ( The highest weighted score of the candidate object () (The location of) 8. The method according to any one of claims 1 to 7, characterized in that: Perform the operation on the resulting feature vector ( Self-velocity compensation (39) is performed to obtain the transformed result feature vector. ), and so that the resulting feature vector after one time step ( Based on the transformed feature vector at the previous time point (t), And based on the calculated object query at that time point (t+1) ) and key vectors ( ) and value vector ( ), which is obtained by means of the decoder (34) of the Transformer.

9. The method according to claim 8, characterized in that: For all objects in group (20-25) that have been assigned to the regression head (37) before the time step ( , , After the time step, the regression heads (37) of all other groups (20-25) are masked.

10. The method according to claim 9, characterized in that: Allow all objects ( , , After the time step, they change their association with group (20-25).

11. A computer program configured to perform each step of the method according to any one of claims 1 to 10.

12. A machine-readable storage medium having a computer program as claimed in claim 11 stored thereon.

13. An electronic control device configured to perform point cloud-based processing on a plurality of objects using a Transformer with an attention model, via the method according to any one of claims 1 to 10. , , ) detection and / or tracking and / or location prediction.

Images