Computer-implemented method for classifying at least one object in environment of vehicle

Abstract

To provide an improved concept for improving detection of at least an object in an environment of a vehicle.SOLUTION: A computer-implemented method includes: a step of collecting first data from a first sensor; a step of collecting second data from a second sensor; a step of determining a first object representation using the first data; a step of determining a second object representation using the second data; a step of updating the first object representation and / or the second object representation depending on an arrival of third data from at least the second sensor, which is collected in a third data collection frame; a step of fusing the first object representation and at least the second object representation based on the received data; and a step of applying the updated representation as input data to a data-driven model in order to train the data-driven model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method for classifying at least objects within a vehicle's environment. [Background technology]

[0002] 2. Background of the Invention Current approaches to sensor fusion assume a predefined grid (structured grid) around the ego-vehicle. This grid is often referred to as the Bird's Eye View (BEV) grid, which has a fixed size grid around the vehicle. 3D native sensors, e.g., lidar, radar, provide their own information about the ego-vehicle in 3D. Hence, the projection onto this BEV grid is straight forward. To fuse the camera readings, an additional process is needed to project the information from the 2D camera pixels onto the 3D BEV grid.

[0003] However, the use of such a lattice to detect, classify or localize objects in the vehicle's environment has some limitations. It tries to represent sparse information about objects in the vicinity of the ego-vehicle in a dense lattice. This representation is redundant since most of the lattice remains unoccupied and requires a large memory footprint. Therefore, there is no scaling with range if the user wants to increase the detection range for a given sensor setup. Furthermore, the lattice assumes several synchronized data tuples, where data from all sensors are assumed to arrive at the same time, and the BEV model receives the already fused information as one measurement.

[0004] A further aspect in this context of detecting objects by using a sensor fusion approach is how to train models with the aim of automating object detection using deep learning (DL) and supervised learning (SL).

[0005] Current methods for deep learning (DL) in the context of supervised learning (SL) include some training data and some corresponding labels for training. The training data is fed to the model to generate some model predictions, and these predictions are compared to the labels via some loss scores. Current SL methods load a batch of data. In the current approach, the data batch is represented by a data sample with a fixed length, because the model knows that the data should arrive at some known size (e.g., number of images in the case of video applications) at some known time.

[0006] For this purpose, the prior art discloses image recognition tasks for detecting objects, where data is loaded synchronously into the model, i.e. the model usually waits for the GPU to load a predefined number of images into memory. Once this is done, it is passed to the model (either directly or to the GPU and then to the model). Now, the drawback of this approach is that the model waits for the data, and only after the data is loaded does the model become active. As a result, the process of object detection becomes inefficient.

[0007] There is a need to address these issues. Summary of the Invention [Problem to be solved by the invention]

[0008] Summary of the Invention The present invention therefore provides an improved concept for improving the detection of at least objects in the vehicle's environment. [Means for solving the problem]

[0009] The problem of the invention is solved by the subject matter of the independent claims, further embodiments are incorporated in the dependent claims.

[0010] In a first aspect of the present invention, there is provided a computer implemented method for classifying at least objects in an environment of a vehicle using a sensor fusion based approach and a data driven model, the method comprising: collecting first data from a first sensor within a first data collection frame; collecting second data from at least a second sensor within a second data collection frame; determining a first object representation using the first data; determining a second object representation using the second data; updating the first object representation and / or the second object representation in response to arrival of third data from at least a second sensor, the third data being collected in a third data collection frame after the first data collection frame; - fusing the first object representation and at least the second object representation to determine an updated representation of the object based on the received data; applying the updated representation as input data to a data-driven model for training the data-driven model to obtain output data containing information about a classification of the detected objects; Includes.

[0011] In other words, the core idea behind the invention is to use measurement data arriving or arriving from a sensor device, for example a radar sensor, to update a current object representation or an object list representation, which may for example be a track. In this connection, it is noted that the latent representation of an object list contains data related to the location and classification of this object list.

[0012] Thus, a system of a vehicle with multiple sensor devices will have multiple object representations, each object representation corresponding to a specific sensor device. Each time new sensor data is generated or arrives, this data is only used to update the corresponding object representation and not to update other object representations corresponding to other sensor devices. In this way, the fusion or merging of the various object representations to obtain a final object classification and / or object localization of the object becomes more efficient, since the relevant features or information of each sensor device are not ignored.

[0013] This approach of the present invention leads to a variety of advantages.

[0014] First, the present invention allows the detection range to be scaled depending on the sensor type application, so that a common BEV structural grid of a specific size, as used in the prior art, is not required when various sensor devices are merged.

[0015] Second, the present invention does not require synchronous reception of data from multiple sensor devices to obtain an object representation.

[0016] Third, the present invention allows information about tracks or object representations to be propagated over time, from time frame to time frame.

[0017] Fourth, the present invention enables robust and reliable object perception by using individual sensor configurations, e.g., cameras, radar, lidar, etc.

[0018] Fifth, the present invention leads to an increased detection range around the vehicle, improving the performance of the detection system.

[0019] According to one example, the first data acquisition frame and / or the second data acquisition frame are represented by a data acquisition window of fixed length and within a defined time interval during which data acquisition of the first sensor and / or at least the second sensor is performed, thereby allowing efficient asynchronous data loading.

[0020] According to an example, the first sensor and / or the second sensor are at least one of a camera sensor, a lidar sensor, and a radar sensor. In this way, object detection and object classification can be flexibly performed depending on the availability of sensor devices in the vehicle.

[0021] According to one example, the step of updating the first object representation and / or the second object representation comprises updating state information of the current first object representation and / or the second object representation at time t. In this way, object classification is efficiently performed.

[0022] According to one example, during the step of updating the first object representation and / or the second object representation at time t, a step of collecting state information of at least the potential second object at time t is included. In this way, object classification and object localization are performed more efficiently.

[0023] In a second aspect of the invention, there is provided a vehicle comprising a system implementing the computer-implemented method according to the invention.

[0024] In a third aspect of the invention there is provided a computer comprising a processor configured to carry out the method of the first aspect.

[0025] In a fourth aspect of the present invention there is provided a computer program product comprising instructions for causing a computer to perform the method of either the first or second aspect when the program is executed by a processor of a computer.

[0026] In a fifth aspect of the present invention there is provided a machine-readable data medium and / or download product embodying the computer program of the fourth aspect.

[0027] In the following, exemplary embodiments are described with reference to the following drawings: [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 shows a schematic flow chart of a computer-implemented method 100 according to the present invention. [Diagram 2] FIG. 2 illustrates a general concept of object detection according to an embodiment of the present invention. [Diagram 3] FIG. 2 illustrates a general concept of data collection using at least one sensor for object detection of an object according to an embodiment of the present invention. [Figure 4] FIG. 2 illustrates a schematic detection signal flow for object detection of an object according to an embodiment of the present invention; [Diagram 5] FIG. 2 illustrates a schematic detection signal flow for object detection of an object according to an embodiment of the present invention; [Figure 6] FIG. 2 illustrates a schematic detection signal flow for object detection of an object according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Detailed Description of the Drawings 1 shows a schematic flow chart of a computer-implemented method 100 according to the present invention for detecting at least one object 50 in an environment 62 of a vehicle 60 using a sensor fusion based approach and a data-driven model 70. It is noted that detecting the at least one object 50 may also include classification and / or localization of the object 50 as described above.

[0030] In a first step 102 , first data 12 is collected from a first sensor 10 in a first data collection frame 14 .

[0031] In a second step 104 , second data 22 is collected from at least a second sensor 20 in a second data collection frame 24 .

[0032] As mentioned above, the present invention can be used to analyze data obtained from sensors, which may be the first sensor 10 and / or the second sensor 20. The sensors 10, 20 can determine measurements of the environment in the form of sensor signals, which may be provided by digital images, such as video, radar, lidar, ultrasound, motion, thermal images, IMU data, GNSS data, etc. The time model fits well to realistic use cases, without requirements regarding ordering, timing or availability of data. Other types of sensors can be used.

[0033] In a third step 106 , a first object representation 30 is determined using the first data 12 .

[0034] In a fourth step 108 , a second object representation 32 is determined using the second data 22 .

[0035] In a fifth step 110, the first object representation 30 and / or the second object representation 32 are updated in response to the arrival of third data 36 from at least the second sensor 20 collected in a third data collection frame 38 after the first data collection frame 14 (i.e., later than the first data collection frame 14).

[0036] Optionally, updating 110 the first object representation 30 and / or the second object representation 32 at time t comprises collecting state information 35 of at least a potential second object 52 at time t.

[0037] Optionally, updating 110 the first object representation 30 and / or the second object representation 32 includes updating state information 33, 34 of the current first object representation 30 and / or the second object representation 32 at time t.

[0038] In a sixth step 112, the first object representation 30 and at least the second object representation 32 are fused to determine an updated representation 40 of the object 50 based on the received data 12,22.

[0039] In a seventh step 114, the updated representation 40 is applied as input data 72 to the data-driven model 70 to train the data-driven model 70 to obtain output data 74 containing information regarding the classification of the detected object 50.

[0040] FIG. 2 shows a general concept or approach for object detection of an object according to one embodiment of the present invention.

[0041] This approach is applicable to a system 64 implemented in a vehicle 60 .

[0042] A predicted track or object 77 arrives forming an object representation of the object 50 from a previous time frame. Additionally, measurements arrive from a first sensor 10, e.g. a radar sensor. A feature extractor 75 extracts features 75-2 of this radar data. The object representation 30 may include the feature extractor 75 and a feature association 76, which may also be an association for updating the object list representation.

[0043] Then, feature association 76 from the previous frame is performed and previous tracks or objects 77 are updated with the current data to obtain updated tracks 78. Ego-motion correction 80 is applied to these updated tracks to compensate for the motion of the vehicle 60, and these tracks are then considered ready for the next measurement. More specifically, ego-motion compensation or ego-motion correction 80 is applied from the previous timestamp to another timestamp. Furthermore, a prediction of the object list position is performed.

[0044] In a subsequent step of FIG. 2, the same is done by a second sensor 20, e.g., a camera, lidar, etc., to obtain a correspondingly updated track 86 using the object representation 32 that includes a feature extractor 84 that extracts features 84-2 from the sensor data 22 provided by the second sensor 20, as described above.

[0045] In some bounding box classification forms, whenever the data is queried by a downstream task, a detection head 79, 83 can be applied on top of the predicted or updated tracks 78, 83. Thus, the detection head 79 is applied to the potential representations of the object list to obtain a localization and classification representation list.

[0046] A track can be defined as a sensor-specific object representation of the objects 50 belonging to each sensor, as detected or classified by said sensors.

[0047] FIG. 3 shows a schematic concept of a data collection or data loading scheme using at least one sensor for object detection of an object according to an embodiment of the present invention.

[0048] As an introduction, in this context, current methods for deep learning (DL) in the context of supervised learning (SL) include some training data for training and some corresponding labels. The training data is fed to the model to generate some model predictions, and these predictions are compared to the labels via some loss scores. Current SL methods load a batch of data. In the current approach, the data batch is represented by a data sample with a fixed length, because the model knows that the data should arrive at some known size (e.g., number of images in the case of video applications) at some known time.

[0049] For this reason, the prior art discloses image recognition tasks for detecting objects, where data is loaded synchronously into the model. That is, the model usually waits for the GPU to load a predefined number of images into memory. Once this is done, it is passed to the model (either directly or to the GPU and then to the model). Now, the drawback of this approach is that the model waits for the data, and only after the data is loaded does the model become active. As a result, the process of object detection becomes inefficient.

[0050] However, when training a multimodal model, data is loaded from multiple sensors simultaneously. In this case, a data batch is loaded that contains data from all required sensors synchronized to one timestamp. The data type to be loaded is typically defined by the model requirements. Finally, the multimodal data is fed synchronously to the model, which assumes that all data refer to a single timestamp.

[0051] In the prior art, all sensors are loaded in one batch containing all data from all sensors, where the data is considered for one timestamp. The data considered to correspond to one timestamp is then fed into a model, where the data is also considered to be related to a single timestamp. This known approach results in inefficiencies in the image recognition process for detecting objects, limiting the ability of each sensor used in the system to detect objects incorrectly.

[0052] Therefore, in the present invention, a data loading scheme is introduced that shows how data from various sensors (see FIG. 2) is efficiently processed, thereby fully taking advantage of each sensor used in detecting objects in the vehicle's environment.

[0053] The present invention's solution to this problem is shown in FIG. 3 (in combination with FIG. 2).

[0054] The data loading scheme loads data from multiple sensor sources within a fixed time interval, so there is no need for a fixed number per source of measurements, nor an identical amount across all sensor sources.

[0055] With reference to FIG. 3, this data loading scheme is such that the first data acquisition frame 14 and / or the second data acquisition frame 24 are loaded over a fixed length of time t seq , and is implemented as represented by a data collection window 42 having a time period within which data collection of the first sensor 10 and / or at least the second sensor 20 is performed. The arrows 44 give the chronological order of collecting data from the various sensor types.

[0056] In this way, in comparison to known prior art approaches, the present invention uses an asynchronous data loading scheme to process data from different sensor types to detect an object by building multiple object representations of the object to be detected, which are then merged or fused to obtain a final object representation of the object.

[0057] In the following, an example with reference to FIG. 3 is provided for the data loading mechanism for the sensor fusion of the present invention.

[0058] For example, t seq Within a window of 1 = 300 ms, data from two different sensors was available to represent the samples, i.e., two samples from sensor 10 (e.g., LIDAR 1) and two samples from type 2 sensor 20 (e.g., CAMERA 1). The advantage of this data loading scheme is that it allows loading a non-fixed and inconsistent number of samples for each sensor, whereas existing data loaders use this assumption.

[0059] According to this approach, the data arrives at the network in batches containing a variable length of multimodal time samples within a specified time window (e.g., 300 ms). A batch in the sense of the present invention is a batch containing a variable length of multimodal time samples (e.g., sample 1, sample 2, sample 3 for a batch size of 3).

[0060] In this regard, further aspects of this embodiment of the invention are set out below.

[0061] As mentioned above, a relevance of the present invention is the ability to train neural networks using asynchronous data, where the batch size is constant, but the length of the samples within the batch is not constant.

[0062] In contrast, in known prior art approaches, sensor fusion is usually based on a soft (unstructured) grid approach, where the detection head is an attention-based detection head, where the data is assumed to be synchronized.

[0063] In the context of the present invention, asynchronous data is used rather than complete data tuples, and therefore the present invention relates to asynchronous data loading for sensor fusion.

[0064] A further aspect of the invention is the loading of several individual sensor data to train a single or several neural networks (NNs) for several tasks. In the present invention, these NNs are used for object detection (OD), but the invention is not limited to this case. Here, the data is loaded into the model in an asynchronous manner. Instead of loading data that is synchronous (matched in timestamp) or aligned to match a specific timestamp, a data loading scheme is proposed that loads sequential data that refer to measurements originating from different time points. Furthermore, our scheme does not synchronize the data by any means of post-processing, nor does it expect a fixed number of measurements to be loaded.

[0065] This general approach can be expressed in a more formal way.

[0066] Consider N sensor data sources and select one source as a reference. Then, select a time window t that defines the sequence length (time) to be loaded. seq For any measurement of the reference source, the time interval t ∈ [t ref -t seq,tref ], where t refdefines the timestamp of the reference measurement. The collected data is defined herein as a sample. It should be noted that the data of the requested data sources does not have to be within this time interval, nor is there an expectation of a fixed number of measurements between these sensor sources.

[0067] Furthermore, it is noted that the loaded data within this time interval holds data from all data sources as samples. To train the neural network, multiple samples are loaded and arranged in sets called batching. The present invention is intended to provide a data loading solution for asynchronous multimodal perception systems that use neural networks at its core. Thus, the present invention allows training a perception network that is compatible with real use cases where data arrives at the system on an asynchronous and inconsistent basis. This may include missing data, different sensor update frequencies, or multimodal sensor captures that do not follow a specific order.

[0068] Furthermore, note that our approach for loading data is also applicable to the case of unsupervised learning (USL), where data is loaded but no training labels exist. Furthermore, our approach is applicable to the case of self-supervised learning (SSL), where the model generates the labels for training by itself.

[0069] FIG. 4 shows a schematic detection signal flow for object detection of an object according to an embodiment of the present invention.

[0070] FIG. 4 details, in a further example, an attention-based detector for implementing the inventive detection approach.

[0071] The data from the lidar is data at time t. tThe PCs are then processed by the lidar backbone 91 to generate a lidar feature map t 92, which is a latent spatial representation of this lidar scan. Further position encoding 98 may be performed.

[0072] Further, potential object queries 99-1 are obtained from similar lidar scans using some algorithm to infer where the potential objects 99 are. The algorithms may vary from random initialization (a very simple initialization) to more complex initialization such as farthest point sampling. Optionally, these queries 99-1 are generated based on the object. t potential These queries 99-1 are derived from the state t To express it as feature t i.e., A t =Attention(key=feature t ,query=obj t potential ), where each key has a corresponding value.

[0073] Then some position encoding obj t potential The 99 potential objects at time t, including the current data feature t 92 is fed into an attention-based update 93. The output 94 of this attention mechanism is the number of objects detected in this lidar scan, i.e., obj at time t. t potential or state t / state information. The detector head 95 is in this state tt, a bounding box (bbx) representation 96 of this lidar scan at time t can be extracted. Otherwise, this map can be used for future use, e.g., as a state map before the next time trace or for other uses. t )

[0074] Figure 5 shows a schematic detection signal flow for object detection of an object according to an embodiment of the present invention. In Figure 5, a similar mechanism to that described in Figure 4 is shown, this time in a different context.

[0075] This time, some of the detection candidates are obj for tracking. t detections and is passed as a query at the next timestamp (t+1). In FIG. 5, objects 87, 88, and 89 are detected as objects to be tracked at time (t+1).

[0076] FIG. 6 shows a schematic detection signal flow for object detection of an object according to an embodiment of the present invention.

[0077] FIG. 6 illustrates a complete cycle of detection and tracking.

[0078] (i) Here, the updated track state at time t detected and tracked at time t+1 is t updated Objects 87, 88, and 89 (see box 94) are inputs from the previous time. These states are then t updated The track is ego-motion corrected and updated by some predictive motion model to become part of the predicted state at time (t+1), i.e., state t+1 predThe residual potential tracks for time (t+1) are derived by the same algorithm as above (random sampling, farthest point sampling, or other). Overall, the potential tracks are calculated by the propagated track state t+1 pred and potential new track obj t potential It is formed by overlapping the

[0079] (ii) Furthermore, data t+1 is acquired by the sensor and processed by the lidar backbone to obtain the data feature at time (t+1), i.e., feature t+1 is generated.

[0080] (iii) As mentioned above, an attention-based mechanism is applied. A t+1 =Attention(key=feature t+1 , query=state t+1 pred , and the value is from the key as well.)

[0081] (iv) The output of this attention mechanism in the context of tracking is the updated object state t+1 updated This latent spatial representation of the state can then be passed as the track for the next time cycle and input to the detection head for bbx extraction.

[0082] Similar principles to those described in Figures 4-6 can be applied to the radar PC and camera features projected onto the BEV 3D in several ways.

Claims

1. 1. A computer-implemented method (100) for classifying at least one object (50) in an environment (62) of a vehicle (60) using a sensor fusion based approach and a data-driven model (70), comprising: - collecting (102) first data (12) from a first sensor (10) within a first data collection frame (14); - collecting (104) second data (22) from at least a second sensor (20) within a second data collection frame (24); - determining (106) a first object representation (30) using said first data (12); - determining (108) a second object representation (32) using the second data (22); - updating (110) the first object representation (30) and / or the second object representation (32) in response to arrival of third data (36) from the at least second sensor (20), the third data (36) being collected in a third data collection frame (38) after the first data collection frame (14); - fusing (112) the first representation (30) and the at least second representation (32) to determine an updated representation (40) of the object (60) based on the received data (12, 14); applying (114) the updated representation (40) as input data (72) to the data-driven model (70) for training the data-driven model (70) to obtain output data (74) containing information about the classification of the detected object (50); A computer-implemented method (100), comprising:

2. 2. The computer-implemented method (100) of claim 1, wherein the first data collection frame (14) and / or the second data collection frame (24) are represented by a data collection window (42, 43) of fixed length and within a defined time interval during which data collection of the first sensor (10) and / or the at least second sensor (20) is performed.

3. 3. The computer-implemented method (100) of claim 1 or 2, wherein the updating (110) of the first object representation (30) and / or the second object representation (32) comprises updating state information (33, 34) of the current first object representation (30) and / or the second object representation (32) at a time t.

4. 4. The computer-implemented method (100) of claim 1, further comprising, during the step (110) of updating the first object representation (30) and / or the second object representation (32) at time t, a step (111) of collecting state information (35) of at least a potential second object (52) at time t.

5. A vehicle (200) comprising a system (202) implementing a computer-implemented method according to any one of claims 1 to 4.

6. A computer comprising a processor configured to perform a computer-implemented method according to any one of claims 1 to 4.

7. 7. A computer program product comprising instructions for causing a computer to carry out a method according to any one of claims 1 to 4 when said computer program is executed by a processor of a computer according to claim 6.

8. A machine-readable data medium and / or download product carrying a computer program according to claim 7.

Images