Computer-implemented method for classifying at least one object in vehicle environment

Abstract

The invention relates to a computer-implemented method for classifying at least one object in a vehicle environment using a sensor fusion-based method and a data-driven model, comprising the steps of:-collecting first data from a first sensor within a first data collection frame; -collecting second data from at least a second sensor within a second data collection frame; determining a first object representation using the first data; determining a second object representation using the second data; -updating the first object representation and / or the second object representation depending on arrival of third data from the at least second sensor, the third data having been collected in a third data collection frame following the first data collection frame; fusing the first representation and at least the second representation to determine an updated representation of the object based on the received data; -applying the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing information about the classification of the detected object.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for classifying at least one object in a vehicle environment. Background Art

[0002] Current approaches for sensor fusion assume a specific grid (regular grid) around the ego vehicle. This grid is often referred to as a bird's eye view (BEV) grid, which has a constant size grid around the vehicle. 3D local sensors (e.g., LiDAR, Radar) provide their information about the ego vehicle in 3D. As such, the projection to this BEV grid is straight forward. To fuse the camera readings, an additional process is needed to project the information from the 2D camera pixels onto the 3D BEV grid.

[0003] However, using such a grid to detect, classify or localize objects in the vehicle environment has some limitations: It attempts to represent sparse information about objects in the vicinity of the ego vehicle in a dense grid. The representation is redundant, as most of the grid remains unoccupied, and requires a large memory footprint. As such, it does not scale with range in the case where a user wishes to increase the detection range of a given sensor setup. Furthermore, the grid assumes some synchronized data tuples, where this data from all sensors is assumed to arrive simultaneously, and the BEV model receives the already fused information as one measurement.

[0004] In this context, when detecting an object by using a sensor fusion approach, an additional aspect is how to train a model in order to automate object detection by using deep learning (DL) and supervised learning (SL).

[0005] Current approaches for deep learning (DL) in the context of supervised learning (SL) consist of some training data and some corresponding labels for training. The training data is fed to the model to produce some model predictions, and these predictions are compared to the labels via some loss score. Current SL approaches load a batch of data. In current approaches, a data batch is represented by data samples of a fixed length, since the model knows that the data should arrive at some known time and in some known size, such as the number of images in the case of video applications, etc.

[0006] To this end, in the prior art, image recognition tasks for detecting objects are disclosed, in which data is loaded synchronously to the model. This means that the model usually waits for the GPU to load a predefined number of images into memory. Once this is completed, it is passed to the model (either directly to the model, or to the GPU and then to the model). Here, the disadvantage of this approach is that the model waits for the data, and once the data is loaded, the model is activated. This leads to an inefficient process of object detection.

[0007] There is a need to address these issues. Summary of the invention

[0008] The present invention therefore provides an improved concept for improving the detection of at least one object in the surroundings of a vehicle.

[0009] The objects of the invention are solved by the subject-matter of the independent claims, wherein further embodiments are incorporated in the dependent claims.

[0010] In a first aspect of the present invention, there is provided a computer-implemented method for classifying at least one object in a vehicle environment using a sensor fusion-based approach and a data-driven model, comprising the following steps:

[0011] - collecting first data from a first sensor within a first data collection frame;

[0012] - collecting second data from at least a second sensor within a second data collection frame;

[0013] - determining a first object representation using the first data;

[0014] - determining a second object representation using the second data;

[0015] - updating the first object representation and / or the second object representation in dependence on arrival of third data from at least a second sensor, the third data having been collected in a third data collection frame following the first data collection frame;

[0016] - fusing the first representation and at least the second representation to determine an updated representation of the object based on the received data;

[0017] - Applying the updated representation used for training the data-driven model as input data for the data-driven model to obtain output data containing information about the classification of the detected object.

[0018] In other words, the core idea behind the invention is that measurement data arriving at or coming from a sensor device (e.g. a radar sensor) is used to update a current object representation or object list representation, which may be, for example, a trajectory. It should be noted that in this context, the implicit representation of the object list includes data about the location and classification of the object list.

[0019] Therefore, a vehicle system with multiple sensor devices has multiple object representations, where each object representation corresponds to a certain sensor device. Whenever new sensor data is generated or arrives, the data is only used to update the corresponding object representation, and not to update other object representations corresponding to other sensor devices. In this way, fusing or merging different object representations to obtain a final object classification and / or object localization of an object becomes more efficient because the relevant features or information of each sensor device are not ignored.

[0020] This method of the invention leads to various advantages.

[0021] Firstly, when different sensor devices are incorporated, a normal regular BEV grid of a certain size (as it is used in the prior art) is not necessary, since the invention allows scaling the detection range depending on the sensor type used.

[0022] Second, the present invention does not require receiving data from sensor devices in a synchronous manner to obtain object representations.

[0023] Third, the present invention allows information about trajectories or object representations to be propagated from time frame to time frame over time.

[0024] Fourth, the present invention allows robust and reliable object perception by using a separate sensor configuration (e.g., camera, radar, lidar, etc.).

[0025] Fifth, the present invention results in an increase in the detection range around the vehicle and improves the performance of the detection system.

[0026] According to one example, the first data collection frame and / or the second data collection frame are represented by a data collection window of fixed length and within a defined time interval, during which data collection of the first sensor and / or at least the second sensor is performed. In this way, asynchronous data loading is possible in an efficient manner.

[0027] According to an example, the first sensor and / or the second sensor is at least one of: a camera sensor, a lidar sensor, a radar sensor. In this way, object detection and object classification may be performed in a flexible manner, depending on the availability of sensor devices in the vehicle.

[0028] According to an example, the step of updating the first object representation and / or the second object representation comprises updating state information of the current first object representation and / or the second object representation at time t. In this way, object classification is performed in an efficient manner.

[0029] According to an example, during the step of updating the first object representation and / or the second object representation at time t, a step of collecting state information of at least a potential second object at time t is included. In this way, object classification and localization is performed in a more efficient manner.

[0030] In a second aspect of the invention, a vehicle is provided, comprising a system implementing the computer-implemented method according to the invention.

[0031] In a third aspect of the present invention, there is provided a computer comprising a processor configured to execute the method of the first aspect.

[0032] In a fourth aspect of the present invention, there is provided a computer program product comprising instructions which, when executed by a processor of a computer, cause the computer to perform the method of any one of the first and second aspects.

[0033] In a fifth aspect of the invention there is provided a machine-readable data medium and / or a download product comprising the computer program of the fourth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Exemplary embodiments will be described below with reference to the following drawings:

[0035] Figure 1 A schematic flow chart of a computer-implemented method 100 of the present invention is illustrated;

[0036] Figure 2 The schematic concept of object detection according to an embodiment of the present invention is illustrated;

[0037] Figure 3 The schematic concept of data collection for object detection using at least one sensor according to an embodiment of the present invention is illustrated;

[0038] Figure 4 illustrates a schematic detection signal flow of object detection of an object according to an embodiment of the present invention;

[0039] Figure 5 illustrates a schematic detection signal flow for object detection of an object according to an embodiment of the present invention; and

[0040] Figure 6 FIG. 1 illustrates a schematic detection signal flow of object detection of an object according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Figure 1A schematic flow chart of a computer-implemented method 100 of the present invention is illustrated for detecting at least one object 50 in an environment 62 of a vehicle 60 using a sensor fusion-based approach and a data-driven model 70. It should be noted that the detection of at least one object 50 may also include classification and / or localization of said object 50.

[0042] In a first step 102 , first data 12 is collected from a first sensor 10 within a first data collection frame 14 .

[0043] In a second step 104 , second data 22 is collected from at least a second sensor 20 within a second data collection frame 24 .

[0044] As previously indicated, the present invention can be used to analyze data obtained from a sensor, which can be a first sensor 10 and / or a second sensor 20. The sensors 10, 20 can determine measurements of the environment in the form of sensor signals, which can be given by, for example, digital images, such as video, radar, lidar, ultrasound, motion, thermal images, IMU data, GNSS data, etc. The temporal model is well aligned with real-world use cases without any requirements on the ordering, timing or availability of the data. Other types of sensors can be used.

[0045] In a third step 106 , a first object representation 30 is determined using the first data 12 .

[0046] In a fourth step 108 , the second object representation 32 is determined using the second data 22 .

[0047] In a fifth step 110, the first object representation 30 and / or the second object representation 32 are updated depending on the arrival of third data 36 from at least the second sensor 20, which third data 36 has been collected in a third data collection frame 38 after (i.e. later than) the first data collection frame 14.

[0048] Optionally, the step 110 of updating the first object representation 30 and / or the second object representation 32 at time t comprises the step of collecting state information 35 of at least potentially the second object 52 at time t.

[0049] Optionally, updating 110 of the first object representation 30 and / or the second object representation 32 comprises updating the current state information 33 , 34 of the first object representation 30 and / or the second object representation 32 at time t.

[0050] In a sixth step 112 , the first representation 30 and the at least second representation 32 are fused to determine an updated representation 40 of the object 60 based on the received data 12 , 14 .

[0051] In a seventh step 114 , the updated representation 40 is applied to the training data-driven model 70 as input data 72 of the data-driven model 70 to obtain output data 74 containing information about the classification of the detected object 50 .

[0052] Figure 2 The diagram illustrates an exemplary concept or method for object detection of an object according to an embodiment of the present invention.

[0053] The method may be applied to a system 64 implemented in a vehicle 60 .

[0054] A predicted trajectory or object 77 arrives, which forms an object representation of the object 50 from the previous time frame. In addition, measurements from a first sensor 10 (e.g., a radar sensor) arrive. A feature extractor 75 extracts features 75-2 of the radar data. The object representation 30 may involve the feature extractor 75 and feature associations 76, which may also be updated associations of the object list representation.

[0055] Then, the association of features from the previous frame is done 76, and the previous tracks or objects are updated 77 with the current data to obtain updated tracks 78. For these updated tracks, ego motion correction 80 is applied to compensate for the movement of the vehicle 60, and then these tracks are considered ready for the next measurement. In more detail, the ego motion compensation or correction 80 is applied from a previous timestamp to another timestamp. In addition, the position prediction of the object list is performed.

[0056] exist Figure 2 In the next step, the same operation is performed on the second sensor 20 (e.g., a camera, lidar, etc. as described above) to obtain an updated trajectory 86 using the object representation 32 accordingly, and the object representation 32 includes a feature extractor 84 that extracts feature 84-2 from the sensor data 22 provided by the second sensor 20.

[0057] Whenever a downstream task queries data in the form of classification of some bounding boxes, the detection head 79, 83 can be applied on top of the predicted or updated trajectories 78, 83. Thus, the detection head 79 is applied on the latent representation of the list of objects to obtain a list of localization and classification representations.

[0058] A track may be defined as a sensor-specific object representation of an object 50 belonging to each sensor detected or classified by the sensor.

[0059] Figure 3 The diagram illustrates a schematic concept of a data collection or data loading solution for detecting an object using at least one sensor according to an embodiment of the present invention.

[0060] As an introduction and in this context, current approaches for deep learning (DL) in the context of supervised learning (SL) consist of some training data and some corresponding labels for training. The training data is fed to the model to produce some model predictions, and these predictions are compared to the labels via some loss score. Current SL approaches load a batch of data. In current approaches, a data batch is represented by data samples of a fixed length, since the model knows that the data should arrive at some known time and in some known size, such as the number of images in the case of video applications, etc.

[0061] To this end, in the prior art, image recognition tasks for detecting objects are disclosed, in which data is loaded synchronously to the model. This means that the model usually waits for the CPU to load a predefined number of images into memory. Once this is completed, it is passed to the model (either directly to the model, or to the GPU and then to the model). Here, the disadvantage of this approach is that the model waits for the data, and once the data is loaded, the model is activated. This leads to an inefficient process of object detection.

[0062] However, in the case of training a multimodal model, data is loaded from many sensors simultaneously. In this case, a data batch is loaded that contains data from all the desired sensors synchronized to one timestamp. The type of data loaded is usually defined by the model requirements. Ultimately, the multimodal data is synchronously fed to the model, which assumes that all data refers to a single timestamp.

[0063] Here, all sensors are loaded into one batch, which includes all data from all sensors, but here the data is considered for one timestamp. The data considered to correspond to one timestamp is then fed to the model. Here, too, the data is considered to be associated with a single timestamp. This known method makes the image recognition process for detecting objects inefficient and limits the capabilities of each sensor in the system for detecting objects in an inappropriate manner.

[0064] Therefore, in the present invention, a data loading scheme is introduced: how to process data from different sensors (see Figure 2 ) data so that the advantages of each sensor used are fully incorporated when detecting objects in the vehicle environment.

[0065] exist Figure 3 In (combined Figure 2 ) describes the present invention's solution to this problem.

[0066] The data loading scheme loads data from multiple sensor sources located at fixed time intervals. Therefore, a fixed number of measurements is required for each source, which does not need to be the same across all sensor sources.

[0067] about Figure 3 The data loading scheme is implemented in which the first data collection frame 14 and / or the second data collection frame 24 is composed of a data collection window 42 having a fixed length (e.g., a time t of 300 ms). seq ) represents and within a defined time interval, during which data collection of the first sensor 10 and / or at least the second sensor 20 is performed. Arrow 44 gives the temporal order of collecting data from the various sensor types.

[0068] In this way, compared to known prior art methods, the present invention uses an asynchronous data loading scheme to process data from different sensor types for detecting objects by constructing multiple object representations of the object to be detected, where the multiple object representations of the object are then merged or fused to obtain a final object representation of the object.

[0069] In the following, about Figure 3 An example is provided for the sensor fusion data loading mechanism of the present invention:

[0070] For example, at t seq = 300ms, data is available from two different sensors to formulate samples: 2 samples from sensor 10 (e.g., LiDAR 1), and 2 samples from type 2 sensor 20 (e.g., Camera 1). The advantage of this data loading scheme is that it can load a non-constant and non-uniform number of samples for each sensor, while existing data loaders do use this assumption.

[0071] According to this method, data arrives at the network as batches containing multimodal time samples of variable length within a specified time window (e.g., 300ms). In the sense of the present invention, a batch is a batch containing multimodal time samples of variable length (e.g., sample 1, sample 2, sample 3 for batch size 3).

[0072] In this regard, further aspects of this embodiment of the invention are presented hereinafter.

[0073] As mentioned before, the context of the present invention is to obtain the ability to train neural networks using unsynchronized data, and where the batch size is constant, but the length of the samples within the batch is not constant.

[0074] In contrast, in known prior art methods, sensor fusion is usually based on a soft (irregular) grid approach, where the detection head is an attention-based detection head. Here, the data is assumed to be synchronized.

[0075] In the context of the present invention, asynchronous data is used, rather than perfect data tuples. Thus, the present invention is about asynchronous data loading for sensor fusion.

[0076] Another aspect of the invention is to load some individual sensory data for training a single or multiple neural networks (NNs) for a certain task. In the present invention, these NNs are used for object detection (OD), but the invention is not limited to only this case. Here, the data is loaded to the model in an asynchronous manner. Instead of loading or synchronizing (timestamp matching) or aligning the data to match a certain timestamp, a data loading scheme is proposed that loads sequence data that references measurements originating from different time points. In addition, our scheme does not synchronize the data by any post-processing means and does not expect to load a fixed number of measurements.

[0077] This general approach can be expressed in a more formal way:

[0078] Given N sensor data sources, we select one source as the reference. Next, we provide a time window t seq , which defines the length of the sequence that should be loaded (in time units). For any measurement of the reference source, the loading is located in the time interval t∈[t ref -t seq,tref ] data from all sensor data sources, where t ref Defines the timestamp of the reference measurement. The collected data is defined as a sample in this paper. It must be noted that there is no requirement for the data of the requested data source to be present within the time interval, and there is no expectation of a fixed number of measurements between sensor sources.

[0079] Also note that data loaded in a time interval holds data from all data sources as samples. To train a neural network, multiple samples are loaded and arranged into sets, which is called batching. The present invention aims to provide a data loading solution for asynchronous multimodal perception systems that use neural networks at their core. Therefore, the present invention enables training perception networks that are compatible for real use cases where data arrives at the system asynchronously and on an inconsistent basis. This may include data loss, different sensor update frequencies, or multimodal sensor capture that does not follow a specific order.

[0080] It should also be noted that the present method for loading data is also applicable to the case of unsupervised learning (USL), where the data is loaded but the training labels do not exist. In addition, our method is applicable to the case of self-supervised learning (SSL), where the model generates its own labels for training.

[0081] Figure 4 FIG. 1 illustrates a schematic detection signal flow of object detection of an object according to an embodiment of the present invention.

[0082] Figure 4An attention-based detector implementing the detection method of the present invention in a further example is shown in detail.

[0083] The data 90 from the LiDAR is acquired in the form of a point cloud (PC) at time t as data t The PC is processed by the LiDAR hub 91 to form a LiDAR feature map feature t 92. This is the latent space representation of the LiDAR scan. Additionally, position encoding 98 can be performed.

[0084] Additionally, potential object queries 99-1 are obtained from the same lidar scan using some algorithm to guess where potential object 99 may be. The algorithm can vary from random initialization (a very simple initialization) to more complex initialization (such as farthest point sampling). Optionally, query 99-1 is derived from obj t potential , and where query 99-1 can include t The key value of , to formulate the attention vector as state t , that is, A t =Attention(Key=feature t ,Query=obj t potential ), where each key has a corresponding value.

[0085] At time t, the potential object 99 (including a position code obj t potential ) and then with the current data feature t 92 are fed together into an attention-based update 93. The output 94 of the attention mechanism is the object detected in the lidar scan, i.e., obj at time t t potential or state t / State information. If the detection head 95 will be applied to the state t If t is t, then the bounding box (bbx) representation 96 at time t of the lidar scan can be extracted. If not, the map will be saved as state t ) for future use, e.g. as a prior for the next tracking or any other use.

[0086] Figure 5 FIG. 1 illustrates a schematic detection signal flow of an object detection according to an embodiment of the present invention. Figure 4 The same mechanism described in , but now in a different context.

[0087] This time, some candidates for detection are considered for tracking because obj t detections t is passed as a query to the next timestamp (t+1). Here, objects 87, 88, 89 are detected as objects to be tracked at time (t+1).

[0088] Figure 6 FIG. 1 illustrates a schematic detection signal flow of object detection of an object according to an embodiment of the present invention.

[0089] Figure 6 The complete cycle of detection and tracking is presented:

[0090] (i) Here, the updated trajectory state at time t detected and tracked at time t+1 t updated Objects 87, 88, 89 (see box 94) are inputs from the previous time. Then, these states t updated The trajectory is ego-motion corrected and updated by a predicted mobility model to become part of the predicted state at time (t+1), i.e., state t+1 pred The remaining latent trajectories at time (t+1) are derived by the same algorithm described above (random sampling, farthest point sampling, or any other). In general, the latent trajectories are propagated by stacking the trajectory state t+1 pred and potential new trajectories obj t potential to form.

[0091] (ii) In addition, data t+1 The data is acquired by the sensor and processed by the lidar hub to generate data features at time (t+1), namely, feature t+1 .

[0092] (iii) Apply attention-based mechanisms as before:

[0093] A t+1 =Attention(Key=feature t+1 ,Query=state t+1 pred ), the value also comes from the key value.

[0094] (iv) The output of the attention mechanism is the updated object state in the context of tracking t+1 updated The Caintic space representation of the state can then be passed as the trajectory for the next time period and can be input to the detection head for BBX extraction.

[0095] As in Figures 4 to 6 The same principles described in apply to the radar PC and camera signatures which are projected to the BEV 3D by some method.

Claims

1. A computer-implemented method (100) for classifying at least one object (50) in an environment (62) of a vehicle (60) using a sensor fusion-based approach and a data-driven model (70), comprising the steps of: - collecting (102) first data (12) from a first sensor (10) within a first data collection frame (14); - collecting (104) second data (22) from at least a second sensor (20) within a second data collection frame (24); - determining (106) a first object representation (30) using the first data (12); - determining (108) a second object representation (32) using the second data (22); - updating (110) the first object representation (30) and / or the second object representation (32) in dependence on the arrival of third data (36) from at least the second sensor (20), the third data (36) having been collected in a third data collection frame (38) subsequent to the first data collection frame (14); - fusing (112) the first representation (30) and at least the second representation (32) to determine an updated representation (40) of the object (60) based on the received data (12, 14); - applying (114) the updated representation (40) for training a data-driven model (70) as input data (72) for the data-driven model (70) to obtain output data (74) containing information about the classification of the detected object (50).

2. A computer-implemented method (100) according to claim 1, wherein the first data collection frame (14) and / or the second data collection frame (24) are represented by a data collection window (42, 43) of fixed length and within a defined time interval, during which data collection of the first sensor (10) and / or at least the second sensor (20) is performed.

3. A computer-implemented method (100) according to any of the preceding claims, wherein updating (110) the first object representation (30) and / or the second object representation (32) comprises updating state information (33, 34) of the current first object representation (30) and / or the second object representation (32) at time t.

4. A computer-implemented method (100) according to any of the preceding claims, wherein during the step (110) of updating the first object representation (30) and / or the second object representation (32) at time t, a step (111) of collecting state information (35) of at least a potential second object (52) at time t is included.

5. A vehicle (200) comprising a system (202) implementing the computer-implemented method according to claims 1 to 4.

6. A computer comprising a processor configured to perform the computer-implemented method according to any one of the preceding claims 1 to 4.

7. A computer program product comprising instructions which, when a processor of a computer according to claim 6 executes the computer program, cause the computer to perform the method according to any one of claims 1 to 4.

8. A machine-readable data medium and / or a download product comprising a computer program according to claim 7.

Images