Method for detecting and / or tracking and / or predicting the location of multiple objects

Abstract

The present invention relates to a method for detecting and / or tracking and / or predicting the location of multiple objects from point cloud data using a Transformer with an attention model, wherein the state of the objects being tracked is stored in a feature space. The present invention includes the step of calculating feature vectors from point cloud data using a backbone, which are used as key vectors and value vectors for the Transformer. This is followed by the steps of determining a result feature vector at a certain point in time from the object query and key vectors and value vectors at that point in time, and assigning this result feature vector to a regression head, each regression head containing regression parameters for a group (20-25) of at least one class (10-19) of objects.

Description

[Technical Field]

[0001] The present invention relates to a method for detecting and / or tracking and / or predicting the location of multiple objects from point cloud data using a Transformer equipped with an attention model. Furthermore, the present invention relates to a computer program that performs all steps of the method and a machine-readable data storage medium in which the computer program is stored. Finally, the present invention relates to an electronically controlled device adapted to perform the detection and / or tracking and / or prediction of multiple objects from point cloud data using a Transformer equipped with an attention model by the present method. [Background technology]

[0002] In publicly available datasets, it is common to detect and track the locations of objects of different classes, such as cars, bicycles, and pedestrians. Several approaches exist to achieve this, and this in particular concerns the extraction of box parameters, i.e., the position, size, and orientation of objects.

[0003] One group of 3D object detectors is anchor box-based. In this case, the measurement range is divided into a grid, and within each cell, an anchor box is placed for each possible object class. In other words, the object class is already defined when the anchor boxes are placed. Therefore, box parameters can be extracted using a regression head trained specifically for this class. In this regard, it has been found to be advantageous to group similar classes, such as bicycles and motorcycles, and estimate this group within a single common regression head.

[0004] In query-based methods, it's common to extract box parameters from the feature vector output by the Transformer using a trained regression head. In this case, the class of the object is irrelevant; rather, one common regression head is used for all classes. Therefore, similar classes cannot be grouped together as they would be when using anchor boxes. [Overview of the Initiative]

[0005] The present invention relates to a method for detecting and / or tracking and / or predicting the location of multiple objects from point cloud data using a Transformer with an attention model. The point cloud data is captured, for example, by LiDAR. However, the method is not limited to LiDAR, and other sensor types may also be used. The sensor or sensor system is preferably located in a vehicle, and therefore the point cloud data is recorded from the vehicle.

[0006] The state of an object is stored in the feature space. This method involves the following steps: firstly, feature vectors are calculated from the point cloud data, particularly by the backbone. The backbone is a neural network used to extract features from the measured data, or to transform the input into some feature representation that can be further processed later.

[0007] A grid-based backbone is particularly used. Alternatively, a point-based backbone may be used. The Transformer encoder can be omitted; however, the Transformer encoder may also be used if the size of the point cloud allows it. The resulting feature vectors are then sent to the Transformer and used as key and value vectors to determine the cross-attention.

[0008] Furthermore, an object query is provided. The object query can be, for example, a learned vector. However, it is preferable that the object query is obtained by first providing an anchor position. In particular, the anchor position is determined from point cloud data. For this purpose, the anchor position can be calculated by, in particular, a sampling method, such as Farthest-Point-Sampling (FPS), or an estimation method. Since the anchor position is determined from the recorded data, it changes for each time point (frame). Instead, it is also possible to learn the anchor position.

[0009] From the anchor position, a feature vector can be determined, in particular, by encoding, such as Fourier encoding. The feature vector thus calculated is used as an object query for the Transformer. This object query of the anchor position is used as a starting point for searching for an object. However, the search is not limited to these anchor positions, and objects spaced apart from these anchor positions are also detected. The anchor position does not correspond to an anchor box as used in other detection approaches. Therefore, the object query for the Transformer is data-dependent and not learned. This provides an advantage, especially in the case of a low-density point cloud, because otherwise a lot of computational resources would be wasted to actually find the positions with data. Such low-density point clouds occur especially during LiDAR measurements. The object query determined from the anchor position is used as a slot for possible objects.

[0010] At a certain point (frame), from an object query, that is, the aforementioned feature vector calculated at this first point, and key vectors and value vectors, that is, the feature vectors first described calculated at this point, a result feature vector (also referred to as a decoder output vector) is determined. This determination is performed, among other things, by the decoder of the Transformer. From this result feature vector, box parameters for describing the object, that is, for example, the position, size, orientation, speed, and / or class identifier of the object are calculated, among other things, with respect to the anchor position.

[0011] As long as the approach for object recognition is query-based, it is not important which type of decoder is used. For example, deformable attention, standard Transformer attention, or a simpler Sampling mechanism can be used. It is also not important which backbone is used for preprocessing and whether a Transformer encoder is used.

[0012] The result feature vector is assigned to regression heads, and each of these regression heads contains regression parameters for at least one group of classes of the object. Each group may contain multiple classes, or some groups may contain only one class, while it is also contemplated that some other groups may contain multiple classes. The object can be a three-dimensional object, such as a passenger car or a pedestrian, or a two-dimensional object, such as a road marking. The number of groups and the exact classification of the classes can be changed according to requirements. Moreover, further class recognition of the object is possible.

[0013] Preferably, the regression head can be trained by assigning estimated objects from object queries to ground truth. In this case, each estimate is compared to a ground truth box, the error for each is determined and incorporated into the cost matrix, and this assignment is selected so that the overall error is minimized, with each ground truth object being compared only to estimates from the regression head that match the object. This preserves the size of the cost matrix. Furthermore, existing matching algorithms can continue to be used, which also ensure that at most one ground truth object can be assigned to each object query.

[0014] In some query-based methods, object queries are assigned to a single location in space (anchor location). In this case, the location of the object query can be refined according to each layer in the decoder. For this purpose, it is preferable that the location of the object query be displaced according to each layer in the decoder to the location where the object is estimated at this point. In one preferred embodiment, each of these locations is displaced to a weighted average of multiple object candidates. In another preferred embodiment, each of these locations is displaced to the location of the object candidate having the highest weighted score or group score among the object candidates. The group score is used to weight the object candidates. The group score arises as the sum of the class scores of the classes contained within each group. The class score is calculated, among other things, by a class head that is trained together with the regression head.

[0015] As a result, multiple objects are classified from the point cloud data at the first time point. If object tracking is to be performed, these objects must be identified at subsequent consecutive time points. In this regard, sensors, sensor systems, or systems on which sensors are placed, such as a vehicle, may move at their own velocity. Next, the self-velocity is corrected by a self-velocity correction module, and this correction is applied to the resulting feature vector to obtain a transformed resulting feature vector with the self-velocity corrected.

[0016] This transformed feature vector is now sent to the Transformer's decoder as input for the next time step, where it is used as a slot for possible objects along with the object query. After a certain time step, the Transformer's decoder determines the final feature vector at each subsequent time step from the transformed feature vector computed for the previous time step, the object query computed at that time as described above, and the key and value vectors computed at that time as described earlier. This computation can continue for any number of time steps. The computed final feature vector is finally assigned to an object as described above, thereby tracking multiple objects.

[0017] As a result, object tracking of multiple objects from point cloud data is obtained through self-velocity correction. With respect to all objects assigned to a group of regression heads before this time step, in one preferred embodiment, it is intended that all other group regression heads are masked after this time step. This ensures that objects do not change their group during the progress of object tracking. In another preferred embodiment, it is intended that all objects can change their assignment to a group after this time step. This allows multiple hypotheses about which group an object may be assigned to to be maintained throughout this time, in which case the outputs of the other regression heads are stored. This enables class swapping and assignment to multiple class hypotheses during the progress of object tracking.

[0018] The computer program is adapted to perform each step of this method, particularly when implemented on a computing or control device. The computer program enables the implementation of this method within conventional electronic control devices without requiring any structural changes. The electronic control device only needs to have a Transformer implemented, or at least it should have one. For implementation, the computer program is stored on a machine-readable memory medium.

[0019] By installing this computer program and Transformer into conventional electronically controlled equipment, this method provides electronically controlled equipment adapted to perform detection and / or tracking and / or position prediction of multiple objects from point cloud data using a Transformer with an attention model.

[0020] Exemplary embodiments of the present invention are shown in the drawings and described in detail below. [Brief explanation of the drawing]

[0021] [Figure 1] This figure shows the integration of an object class into an object group in one exemplary embodiment of the method according to the present invention. [Figure 2] This graph shows the length deviation within an object class. [Figure 3] This is a flowchart of one exemplary embodiment of the method according to the present invention. [Figure 4a] This is a schematic diagram illustrating the refinement of object query positioning in one exemplary embodiment of the method according to the present invention. [Figure 4b] This is a schematic diagram showing a refinement of the object query location in another exemplary embodiment of the method according to the present invention. [Modes for carrying out the invention]

[0022] Figure 1 shows how classes of detectable objects are distributed into groups in one exemplary embodiment of the method according to the present invention. The classes are, for example, passenger cars 10, pedestrians 11, traffic cones 12, trucks 13, construction vehicles 14, buses 15, trailers 16, barricades 17, bicycles 18, and motorcycles 19. The first group 20 includes class 10 of passenger cars. The second group 21 includes classes 11 and 12 of pedestrians and traffic cones. The third group 22 includes classes 13 and 14 of trucks and construction vehicles. The fourth group 23 includes classes 15 and 16 of buses and trailers. The fifth group 24 includes class 17 of barricades. The sixth class 25 includes classes 18 and 19 of bicycles and motorcycles.

[0023] One regression head is trained for each of groups 20-25. Object queries have the potential to find objects of any class in this method and are therefore compatible with all regression heads.

[0024] During training, an assignment is made between the objects estimated from the object queries and the Ground Truth. For this purpose, each estimate d m Each ground truth box b kThis is compared with m=1...M and k=1...K, respectively. The error is determined for each and incorporated into a cost matrix of size K×M, and this assignment is chosen such that the overall error is minimized. Thus, if N regression heads are obtained, then N potential objects can be obtained from each of the M object queries.

[0025]

number

[0026] This occurs (n=1...N, m=1...M). All should be considered in the allocation without increasing the cost matrix. Furthermore, each query should be allocated to at most one ground truth object. To satisfy this requirement, each ground truth object b k is class i k Therefore, it is compared only with estimates from regression heads that are appropriate for each object.

[0027]

number

[0028] (m=1...M). This means the cost matrix still has a size of K×M. Moreover, the existing matching algorithm can still be used, which also ensures that each query can be assigned at most one ground truth object.

[0029] Figure 2 shows the average length I of objects in each of groups 20-25, along with their 25% and 75% length deviations. It can be seen that the length deviation within a single group is small compared to the length difference between groups. This allows for good differentiation of objects from different groups.

[0030] Figure 3 shows a flow diagram of the method according to the invention at two consecutive times t and t+1. Hereinafter, i and k as subscripts mean control variables. The left side relates to the first time t. First, the LiDAR sensor of vehicle F captures the surroundings. The visual display of this recorded point cloud data is labeled 30. The backbone 31 calculates a feature vector from the point cloud data, and this feature vector is then extended by the position encoding 32, for example using sine and cosine, and finally, the key vector k passed through the deformable encoder 33 of the Transformer for the first time t. t,i and the value vector v t,i are sent to the decoder 34 of the Transformer.

[0031] At the same time, from the point cloud data, the anchor position ρ at the first time t is determined by a sampling method or an estimation method 35, for example, Farthest-Point-Sampling. t,i This anchor position ρ t,i then passes through a position encoding 36, for example, Fourier encoding.

[0032] y i =FFN[sin(Bρ i ),cos(Bρ i )] where B is a matrix having components of a normal distribution, and FFN is a feed-forward neural network consisting of two layers having ReLU activation (Rectified Linear Unit) here. y i is the calculated feature vector and is sent to the decoder 34 of the Transformer as an object query.

[0033] The set of feature vectors output for the first time t is represented by Y t and consists of the object query y t,i Each object query y t,iHowever, these are used as slots for possible objects (shown by individual boxes in Figure 3). The Transformer's decoder 34 consists of six layers, each having, for example, eight attention heads. At the first time point t, the decoder 34 processes the object query y t,i and key vector k t,i and value vector v t,i From the resulting feature vector y' t,i Confirm the object query y t,i , key vector k t,i , and value vector v t,i The dimension of is, for example, 256. However, the object query y t,i The number of keys is the key vector k. t,i and value vector v t,i The number may be different from the number of .

[0034] Object Query t,i Position p m This is refined according to each layer in the decoder 34. For this purpose, the object query y t,i Position p mHowever, at this point, the object is displaced to its estimated location. For this, a method such as that described in "Transformers for Object Detection in Large Point Clouds" by F. Ruppel, F. Faion, C. Glaeser, and K. Dietmayer, 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China, 2022, pp. 832-838, doi:10.1109 / ITSC55140.2022.9921840. can be used. However, if the group regression head yields N object candidates for each object query, it is not clear where the query location should be refined. In the exemplary embodiment shown in Figure 4a of this method, the object query location p m These are, respectively, object candidates.

[0035]

number

[0036] The solution is to displace the weighted average of (n=1...N, m=1...M) based on the following equation.

[0037]

number

[0038] In the formula, s n This refers to the group score used to weight the object candidates. The group score is the sum of the class scores of the classes contained within each group. The class score is calculated by the class heads, which are trained together with the regression heads. This can be done in the manner described above, both during training and inference.

[0039] In another exemplary embodiment, preferably in the inference, the object query position may instead be displaced to the location of the object candidate with the best group score. This is shown in Figure 4b.

[0040] As a result, two objects O1 and O2 are detected at the first time point t. Within the regression head 37, the resulting feature vector y' t,i However, it is assigned to regression head 37 that matches the object. Objects O1 and O2 are detected and drawn into the visual display labeled 38.

[0041] On the right side of Figure 3, the evaluation for the second time point t+1 is shown, which follows the first time point t according to a time step defined by the repetition rate of the LiDAR sensor recordings. During this time step, vehicle F continues to move at its own velocity. This affects the evaluation of the measured data as well as the relative positions and velocities of objects O1 and O2. Self-velocity correction 39 is performed. Self-velocity correction 39 with respect to the time step between the first time point t and the second time point t+1 can be performed using the method described in "Transformers for Object Detection in Large Point Clouds" by F. Ruppel, F. Faion, C. Glaeser, and K. Dietmayer in IEEE Int.Conf.Intell.Transp.Syst. (IEEE International Conference on Intelligent Transportation Systems), 2022. In this regard, the resulting feature vector y' for the first time point t t,i , anchor position ρ t,i From the pose change P, the transformed result feature vector y” t,k This can be obtained.

[0042] y” t,k =EMC(y' t,i ρ i ,P) Following the first time point t, the LiDAR sensor first captures the surroundings, and the backbone 31 calculates a feature vector from the point cloud data. This feature vector is then expanded by the position encoding 32, for example, using sine and cosine, and finally, via the deformable encoder 33, a key vector k for the second time point t+1 is obtained. t+1,i and value vector v t+1,i This is then sent to the Transformer's decoder 34. Simultaneously, the anchor position ρ at the second time point t+1 is determined from the point cloud data using the sampling method 35. t+1,i This anchor position ρ is determined. t+1,i It then undergoes Fourier encoding 36. Object query y for the second time point t+1 t+1,i This can be obtained.

[0043] The set of feature vectors output for the second time point t+1 is Y t+1 This is represented by the object query y for the second time point t+1. t+1,i and the transformed result feature vector y” calculated for the first time point t. t,k It consists of, and is a union

[0044]

number

[0045] It can be shown as: Each object query y t+1,i and each transformed result feature vector y'' t,k However, these are used as slots for possible objects (shown by individual boxes in Figure 3).

[0046] Decoder 34 processes object query y t,i , the transformed result feature vector y” t,k and key vector k t+1,i and value vector v t+1,i From this, the resulting feature vector y' at the second time point t+1. t+1,iThis confirms the third object O3 is detected along with both objects O1 and O2, and its path is further traced. In the regression head 37, the resulting feature vector y' of all objects O1, O2, and O3 is determined. t,i However, it is assigned to the matching regression head 37. Here again, objects O1, O2, and O3 are drawn into the visual display labeled 38. As a result, multiple objects O1, O2, and O3 are detected in a further time step t+1.

[0047] In one exemplary embodiment of this method, at a second time point t+1, for objects O1 and O2 that were already recognized at the first time point t and assigned to one of groups 20-25, a regression head 37 containing a different group from the one they were already assigned to is masked. In other words, these objects O1 and O2 automatically retain their group assignments. The resulting feature vector y' of object O3, which was recognized for the first time at the second time point. t+1,i Only this is compared to all regression heads 37.

[0048] In another exemplary embodiment of this method, at a second time point t+1, the assignment of each object O1, O2, and O3 to one of groups 20-25 is determined again.

[0049] The above steps are performed for further points in time, thereby tracking multiple objects O1, O2, and O3. The above steps, inputs, and outputs are adapted accordingly to these points in time.

Claims

1. Multiple objects from point cloud data using Transformer with attention model (O １ , O ２ , O ３ A method for detecting and / or tracking and / or predicting the location of the object being tracked (O １ , O ２ , O ３ The state of ) is stored in the feature space, and the following steps are taken: - A step of calculating a feature vector from the point cloud data by a backbone (31), wherein the feature vector is a key vector (k ｔ，ｉ , k ｔ＋１，ｉ ) and a value vector (v ｔ，ｉ , v ｔ＋１，ｉ ) to be used as, step, - Result feature vector (y') at a certain point in time (t) ｔ，ｉ ) and the object query (y) at the aforementioned time (t). ｔ，ｉ ) and key vector (k ｔ，ｉ ) and value vector (v ｔ，ｉ ) The steps to be determined from, - The aforementioned result feature vector (y' ｔ，ｉ This is the step of assigning the object (O) to the regression head (37), and each regression head assigns the object (O) １ , O ２ , O ３ Step A method having

2. Anchor position (ρ) due to encoding (36) ｔ，ｉ ρ ｔ＋１，ｉ The feature vector is determined from ), and the feature vector is used in the object query (y ｔ，ｉ , y ｔ＋１，ｉ The method according to claim 1, characterized in that it is used as ).

3. The anchor position (ρ ｔ，ｉ ρ ｔ＋１，ｉ The method according to 2, characterized in that the above is calculated from the point cloud data by a sampling method or an estimation method (35).

4. The aforementioned object query (y ｔ，ｉ , y ｔ＋１，ｉ Objects (O) estimated from １ , O ２ , O ３ The method according to any one of claims 1 to 3, wherein the regression head (37) is trained by making an assignment between the ground truth and the ground truth, wherein the estimation is compared with the ground truth box, the error is determined for each, and incorporated into a cost matrix, and the assignment is selected such that the error is minimized overall, and in this respect each ground truth object is compared only with the estimation from the regression head (37) that is suited to the object.

5. The aforementioned object query (y ｔ，ｉ ) position (p ｍ ) at this point, according to each layer in the decoder (34), the object (O １ , O ２ , O ３ The method according to any one of claims 1 to 4, characterized in that the ) is displaced to a location where it is estimated.

6. The position (p ｍ ) each represents multiple object candidates [Math 1] The method according to 5, characterized in that the weighted average is displaced.

7. The position (p ｍ ) are each object candidates [Math 2] Candidates for the object with the highest weighted score 【Number 3】 The method according to claim 5, characterized in that it is displaced to the location.

8. The aforementioned result feature vector (y' ｔ，ｉ The self-velocity correction (39) of the transformed result feature vector (y'' ｔ，ｋ This is performed to obtain the resulting feature vector (y') after a certain time step. ｔ＋１，ｉ ) is the transformed result feature vector (y'') at the previous time point (t). ｔ，ｋ ) and the calculated object query y at this point (t+1) ｔ＋１，ｉ and key vector k ｔ＋１，ｉ and value vector v ｔ＋１，ｉ The method according to any one of claims 1 to 7, characterized in that the result is determined by the decoder (34) of the Transformer.

9. Before the aforementioned time step, all objects (O １ , O ２ , O ３ The method according to 8, characterized in that, with respect to the above time step, the regression heads (37) of all other groups (20-25) are masked after that time step.

10. All objects (O １ , O ２ , O ３ The method according to 9, characterized in that the agent can change its assignment to the group (20-25) after the aforementioned time step.

11. A computer program adapted to perform each step of the method according to any one of claims 1 to 10.

12. A machine-readable recording medium storing the computer program described in claim 11.

13. Multiple objects (O) from point cloud data using a Transformer equipped with an attention model by the method according to any one of claims 1 to 10 １ , O ２ , O ３ Electronically controlled equipment adapted to perform detection and / or tracking and / or location prediction of ).

Images