Computer-implemented method for classifying at least one object in the environment of a vehicle

Abstract

The present invention relates to a computer-implemented method (100) for classifying at least one object (50) in an environment (62) of a vehicle (60) using a sensor fusion-based approach and a data-driven model (70), comprising the following steps: - Acquisition (102) of first data (12) from a first sensor (10) within a first data acquisition frame (14); - Acquisition (104) of second data (22) from at least one second sensor (20) within a second data acquisition frame (24); - Determine (106) a first object representation (30) using the first data (12); - Determine (108) a second object representation (32) using the second data (22); - Updating (110) the first object representation (30) and / or the second object representation (32) depending on the arrival of third data (36) from the at least second sensor (20), which were acquired in a third data acquisition frame (38) after the first data acquisition frame (14); - Merging (112) the first representation (30) and the at least second representation (32) to determine an updated representation (40) of the object (60) based on the received data (12, 14); - Applying (114) the updated representation (40) to train the data-driven model (70) as input data (72) for the data-driven model (70) to obtain output data (74) containing information about a classification of the detected object (50).

Description

Field of invention

[0001] The present invention relates to a computer-implemented method for classifying at least one object in the environment of a vehicle. Background of the state of the art

[0002] Current approaches to sensor fusion start with a specific grid (a regular grid) around an ego vehicle. This grid is often referred to as a BEV (Bird's Eye View) grid and has a constant size around the vehicle. The native 3D sensors, such as lidar and radar, provide their information about the ego vehicle in 3D. Therefore, predictions for this BEV grid are straightforward. To fuse the camera data, an additional process is required to project the information from the 2D camera pixels onto the 3D BEV grid.

[0003] However, using such a grid for detecting, classifying, or localizing objects in the vehicle's environment has some limitations: It attempts to represent sparse information about the object in the Ego vehicle's environment in a dense grid. This representation is redundant, as most of the grid remains unused and requires significant memory. Therefore, there is no scaling with range if the user wants to increase the detection range for a specific sensor configuration. Furthermore, the grid relies on a few synchronized data tuples, assuming that data from all sensors arrives simultaneously and that the BEV model receives the already fused information as a single measurement.

[0004] Another aspect in this context when detecting an object using a sensor fusion approach is the way in which models are trained to automate object detection using deep learning (DL) and supervised learning (SL).

[0005] Current deep learning (DL) methods in the context of supervised learning (SL) have some training data and corresponding labels for the training. The training data is fed into the model to generate some model predictions, and these predictions are compared to the labels using a loss score. Current SL methods load a data stack. In current approaches, the data stack is represented by fixed-length data samples, since the model knows that the data should arrive at a known time and in a known size, e.g., a number of frames in the case of a video application, etc.

[0006] In the prior art, an image recognition task for object detection is known in which data is loaded synchronously with the model. This means that the model typically waits for the GPU to load a predefined number of images into memory. Once this is done, the images are passed to the model (either directly, or to the GPU and then to the model). The disadvantage of this approach is that the model waits for the data, and only activates the model once the data is loaded. This leads to an inefficient object detection process.

[0007] These problems need to be addressed. Summary of the invention

[0008] Therefore, the invention provides an improved concept for improving the detection of at least one object in the environment of a vehicle.

[0009] The object of the present invention is solved by the subject matter of the independent claims, with further embodiments being included in the dependent claims.

[0010] In a first aspect of the present invention, a computer-implemented method for classifying at least one object in a vehicle environment is provided using a sensor fusion-based approach and a data-driven model, comprising the following steps: - Capturing initial data from an initial sensor within an initial data acquisition frame; - Acquisition of second data from at least one second sensor within a second data acquisition frame; - Determining an initial object representation using the initial data; - Determining a second object representation using the second data; - Updating the first object representation and / or the second object representation depending on the arrival of third data from the at least second sensor, which was captured in a third data acquisition frame after the first data acquisition frame; - Merging the first representation and at least the second representation to determine an updated representation of the object based on the received data; - Applying the updated representation to train the data-driven model as input data for the data-driven model to obtain output data containing information about a classification of the detected object.

[0011] In other words, a key concept of the present invention is that measurement data arriving from a sensor device, e.g., a radar sensor, is used to update a current object representation or object list representation, which could, for example, be a track. In this context, it should be noted that the latent representation of the object list contains the data about the localization and classification of this object list.

[0012] A vehicle system with multiple sensor devices therefore has several object representations, each corresponding to a specific sensor device. Each time new sensor data is generated or received, this data is used only to update the corresponding object representation, and not other object representations corresponding to different sensor devices. In this way, the fusion or merging of different object representations to obtain a final object classification and / or localization is made more efficient, as relevant features or information from each sensor device are not neglected.

[0013] This approach of the present invention leads to various advantages.

[0014] Firstly, a common regular BEV grid of a specific size, as used in the prior art when fusing different sensor devices, is not required, since the present invention makes it possible to scale detection areas depending on the sensor type used.

[0015] Secondly, the present invention does not require the synchronous reception of data from sensor devices in order to obtain an object representation.

[0016] Thirdly, the present invention enables information about traces or object representations to propagate from time frame to time frame.

[0017] Fourthly, the present invention enables robust and reliable object perception through the use of individual sensor configurations, e.g. camera, radar, lidar, etc.

[0018] Fifthly, the present invention leads to an increase in the detection areas around the vehicle and improves the performance of the detection system.

[0019] For example, the first and / or second data acquisition frame is represented by a fixed-length data acquisition window within a defined time interval, during which data acquisition from the first and / or at least the second sensor is performed. This enables efficient asynchronous data loading.

[0020] For example, the first sensor and / or the second sensor is at least one of the following: camera sensor, lidar sensor, radar sensor. This allows object detection and classification to be performed flexibly, depending on the available sensors in the vehicle.

[0021] For example, the step of updating the first object representation and / or the second object representation involves updating the state information of the current first object representation and / or the second object representation at a time t. In this way, object classification is performed efficiently.

[0022] For example, the step of updating the first object representation and / or the second object representation at time t includes the step of capturing state information from at least one potential second object at time t. This makes object classification and localization more efficient.

[0023] In a second aspect of the present invention, a vehicle is provided which includes a system that implements the computer-implemented method according to the present invention.

[0024] In a third aspect of the present invention, a computer is provided which includes a processor configured to perform the method of the first aspect.

[0025] In a fourth aspect of the present invention, a computer program product is provided which includes instructions which, when the program is executed by a processor of a computer, cause the computer to carry out the method according to one of the first and second aspects.

[0026] In a fifth aspect of the present invention, a machine-readable data carrier and / or download product is provided which contains the computer program of the fourth aspect. Brief description of the drawings

[0027] Examples of implementation are described below using the following drawings: Fig. Figure 1 illustrates a schematic flowchart of a computer-implemented method 100 of the present invention; Fig. Figure 2 illustrates a schematic concept of object recognition according to an embodiment of the present invention; Fig. Figure 3 illustrates a schematic concept of data acquisition using at least one sensor for object detection of an object according to an embodiment of the present invention; Fig. Figure 4 illustrates a schematic recognition signal flow of an object recognition process according to an embodiment of the present invention; Fig. Figure 5 illustrates a schematic recognition signal flow of an object recognition system according to an embodiment of the present invention; and Fig. Figure 6 illustrates a schematic recognition signal flow of an object recognition system according to an embodiment of the present invention. Detailed description of the drawings

[0028] Fig. Figure 1 illustrates a schematic flowchart of a computer-implemented method 100 of the present invention for detecting at least one object 50 in an environment 62 of a vehicle 60 using a sensor fusion-based approach and a data-driven model 70. Note that the detection of the at least one object 50 may also include classification and / or localization of the object 50.

[0029] In a first step 102, initial data 12 are acquired from a first sensor 10 within a first data acquisition frame 14.

[0030] In a second step 104, second data 22 are acquired from at least one second sensor 20 within a second data acquisition frame 24.

[0031] As already mentioned, this invention can be used to analyze data obtained from a sensor, which may be the first sensor 10 and / or the second sensor 20. The sensor 10, 20 can determine environmental measurements in the form of sensor signals, which may be, for example, digital images, video, radar, LiDAR, ultrasound, motion, thermal images, IMU data, GNSS data, etc., as well as temporal models that are well suited to realistic applications without imposing requirements on the sequence, timing, or availability of the data. Other types of sensors can also be used.

[0032] In a third step 106, a first object representation 30 is determined using the first data 12.

[0033] In a fourth step 108, a second object representation 32 is determined using the second data 22.

[0034] In a fifth step 110, the first object representation 30 and / or the second object representation 32 is updated depending on the arrival of third data 36 from the at least second sensor 20, which was acquired in a third data acquisition frame 38 after (i.e. later than) the first data acquisition frame 14.

[0035] Optionally, step 110 of updating the first object representation 30 and / or the second object representation 32 at time t includes the step of acquiring state information 35 from at least one potential second object 52 at time t.

[0036] Optionally, updating 110 of the first object representation 30 and / or the second object representation 32 involves updating a state information 33, 34 of the current first object representation 30 and / or the second object representation 32 at a time t.

[0037] In a sixth step 112, the first representation 30 and the at least second representation 32 are merged to determine an updated representation 40 of the object 60 based on the received data 12, 14.

[0038] In a seventh step 114, the updated representation 40 is applied as input data 72 for the data-driven model 70 to train it, in order to obtain output data 74 containing information about a classification of the detected object 50.

[0039] Fig. Figure 2 illustrates a schematic concept or approach for object recognition of an object according to an embodiment of the present invention.

[0040] The approach can be applied to a system 64 that is implemented in the vehicle 60.

[0041] The predicted tracks or objects 77, which form an object representation of object 50 from a previous timeframe, arrive. In addition, measurements from the first sensor 10, e.g., a radar sensor, may arrive. The feature extractor 75 extracts the features 75-2 from this radar data. The object representation 30 may include the feature extractor 75 and the feature mapping 76, which may also be a mapping of an update to the object list representation.

[0042] Then, an assignment 76 of features from previous frames is performed, and the previous tracks or objects 77 are updated with the current data to obtain updated tracks 78. Ego motion correction 80 is applied to these updated tracks to compensate for the movement of the vehicle 60, and then these tracks are considered ready for the next measurement. More precisely, ego motion compensation or correction 80 is applied from one previous timestamp to another timestamp. Additionally, a prediction of the object list location is performed.

[0043] In the next step in Fig. 2 The same procedure is performed with the second sensor 20, e.g. a camera, lidar, etc., as previously described, in order to obtain an updated track 86 accordingly by using the object representation 32, which involves the feature extractor 84 extracting the features 84-2 from the sensor data 22 provided by the second sensor 20.

[0044] Whenever the data is queried by downstream tasks in the form of classifying certain bounding frames, a detection header 79, 83 could be applied to the predicted or updated tracks 78, 83. Therefore, a detection header 79 is applied to the latent representation of the object list to obtain a localization and classification representation list.

[0045] The traces can be defined as sensor-specific object representations of the object 50 belonging to a respective sensor, which is detected or classified by it.

[0046] Fig. Figure 3 illustrates a schematic concept of data acquisition or a data loading scheme using at least one sensor for object detection of an object according to an embodiment of the present invention.

[0047] As an introduction, and in this context, current deep learning (DL) methods in the context of supervised learning (SL) provide some training data and corresponding labels for training. The training data is fed into the model to generate some model predictions, and these predictions are compared to the labels using a loss score. Current SL methods load a data stack. In current approaches, the data stack is represented by fixed-length data samples, since the model knows that the data should arrive at a known time and in a known size, e.g., the number of frames in a video application, etc.

[0048] In the prior art, an image recognition task for object detection is known in which data is loaded synchronously with the model. This means that the model typically waits for the CPU to load a predefined number of images into memory. Once this is done, the images are passed to the model (either directly, or to the GPU and then to the model). The disadvantage of this approach is that the model waits for the data, and only activates the model once the data is loaded. This leads to an inefficient object detection process.

[0049] However, when training a multimodal model, data from many sensors is loaded simultaneously. In this case, a data stack is loaded containing data for all required sensors, synchronized to a single timestamp. The loaded data types are typically defined by the model requirements. Finally, the multimodal data is synchronously fed into the model, which assumes that all data refers to a single timestamp. There, all sensors are loaded into a stack containing all data from all sensors, but only the data corresponding to a single timestamp is considered. Subsequently, the data corresponding to a single timestamp is fed into the model. Again, it is assumed that the data refers to a single timestamp.This well-known approach makes the image recognition process for detecting an object inefficient and unduly limits the capabilities of each sensor used in the system to detect the object.

[0050] Therefore, the present invention provides a data loading scheme for how data from various sensors (see Fig. 2) are processed efficiently, so that the advantages of each sensor used in detecting an object in the vicinity of a vehicle are fully taken into account.

[0051] The solution of the present invention to this problem is in Fig. 3 (in combination with Fig. 2) shown.

[0052] The data loading scheme loads data from multiple sensor sources that occur within a fixed time interval. Therefore, neither a fixed number of measurements per source nor the same number of all sensor sources is required.

[0053] In relation to Fig. 3 This data loading scheme is implemented by separating the first data acquisition frame 14 and / or the second data acquisition frame 24 through a data acquisition window 42 with a fixed length, e.g., a time t. seq of 300 ms, and within a defined time interval during which data acquisition by the first sensor 10 and / or the at least second sensor 20 is carried out, is shown. Arrow 44 indicates a chronological sequence of data acquisition by the different sensor types.

[0054] In this way, the present invention, in comparison to the known prior art approach, uses an asynchronous data loading scheme to process data from different sensor types for the detection of an object by creating multiple object representations of the object to be detected, wherein the multiple object representations of the object are then combined or fused to obtain a final object representation of the object.

[0055] The following refers to Fig. 3. An example of the data loading mechanism for the sensor fusion of the present invention is provided: Within a window of t seqFor example, in 300 ms, data from two different sensors were available to formulate a sample: 2 samples from sensor 10 (e.g., Lidar1), 2 samples from type 2 sensor 20 (e.g., Camera1). The advantage of this data loading scheme is that it can load a non-constant and inconsistent number of samples for each sensor, whereas existing data loaders use this assumption.

[0056] According to this approach, the data arrives in the network as a batch containing multimodal temporal samples of variable length within a specific time window (e.g., 300 ms). A batch, as defined in the present invention, is a batch containing multimodal temporal samples of different lengths (e.g., sample 1, sample 2, sample 3 for batch size 3).

[0057] In this context, further aspects of this embodiment of the present invention will be presented below.

[0058] As already mentioned, the context of the present invention is to obtain the ability to train a neural network using data that is not synchronized and where the stack size is constant, but the length of the samples within a stack is not constant.

[0059] In contrast, known state-of-the-art sensor fusion approaches are typically based on a soft (non-regular) grid approach, where the detection head relies on attention. This approach assumes that the data is synchronized.

[0060] In the context of this invention, non-synchronized data is used instead of perfect data tuples. Therefore, the present invention relates to the asynchronous loading of data for sensor fusion.

[0061] Another aspect of the present invention is the loading of individual sensor data to train one or more neural networks (NNs) for a specific task. In the present invention, these NNs are used for object detection (OD), but the invention is not limited to this case. Data is loaded into the model asynchronously. Instead of loading data that is either synchronous (timestamps coincide) or aligned to a specific timestamp, a data loading scheme is proposed that loads sequential data relating to measurements taken at different times. Furthermore, our scheme does not synchronize the data through post-processing, and a fixed number of measurements are not expected to be loaded.

[0062] This general approach can be expressed more formally: Given N sensor data sources, one source is selected as a reference. Then, a time window t is defined. seq A parameter is provided that defines the sequence length (as a time duration) to be loaded. For each measurement of the reference source, the data from all sensor data sources within the time interval t ∈ [t] are loaded. ref - t seq , t ref ] lie, where t ref The timestamp of the reference measurement is defined. The acquired data is defined here as a sample. Note that the presence of data for a requested data source within the time interval is not required, nor is a fixed number of measurements expected from the sensor sources.

[0063] Note further that the loaded data within the time interval contains samples from all data sources. To train neural networks, multiple samples are loaded and arranged in batches, a process known as batch processing. The present invention aims to provide a data loading solution for asynchronous multimodal perception systems that use a neural network at their core. Thus, this invention enables the training of perception networks compatible with real-world applications where data arrives asynchronously and inconsistently in the system. This can include data loss, varying sensor update frequencies, or multimodal sensor acquisition that does not follow a specific sequence.

[0064] Note also that the presented approach to loading data is applicable to unsupervised learning (USL), where the data is loaded but the training labels are not present. Furthermore, our approach is applicable to self-supervised learning (SSL), where the model generates the training labels itself.

[0065] Fig. Figure 4 illustrates a schematic recognition signal flow of an object recognition system according to an embodiment of the present invention.

[0066] Fig. Figure 4 shows in detail an attention-based detector for implementing the recognition approach of the present invention in a further example.

[0067] Data 90 from a lidar are acquired as a point cloud (PC) at a specific time t. The PC is processed by a lidar backbone 91 to create a lidar feature map feature 92. This is a latent spatial representation of this lidar scan. Additionally, position coding 98 can be performed.

[0068] Furthermore, potential object queries 99-1 are obtained from the same LiDAR scan, using an algorithm to guess where potential objects 99 might be located. This algorithm can vary from a random initialization (a very simple one) to a more complex initialization, such as scanning at the furthest point. Optionally, queries 99-1 are derived from the obj t potenzielland where the queries can have 99-1 keys derived from the feature to formulate this attention vector as a state, i.e., At = attention(key = feature, query = obj t potenziell ), where each key has a corresponding value.

[0069] The potential objects 99 at time t, including some obj t potenziell The position coding data is then fed, along with the current data feature 92, to the attention-based update 93. The output 94 of this attention mechanism is objects detected in this lidar scan, i.e., obj t potenziell or state / state information at time t. If a detection head 95 is applied to this state, a bounding box representation (bbx) 96 can be extracted at time t of this lidar scan. If not, this map is recorded as state (state). t) for future use, e.g. before the next time recording or other use.

[0070] Fig. Figure 5 illustrates a schematic recognition signal flow of an object recognition system according to an embodiment of the present invention. It depicts the same mechanism as described in Fig. 4 described, but now in a different context.

[0071] This time, some of the candidates for detections, which are used for tracking as obj t erkennungen The queries are passed to the next timestamp (t+1). Here, objects 87, 88, and 89 were identified as objects to be tracked at time (t+1).

[0072] Fig. Figure 6 illustrates a schematic recognition signal flow of an object recognition system according to an embodiment of the present invention.

[0073] In Fig. Figure 6 shows the complete detection and tracking cycle: (i) Here, objects 87, 88, 89 of the tracks updated at time t are shown, state t aktualisiert , which were detected and tracked at time t+1 (see Box 94), from the previous time. These traces state t aktualisiert . Traces are then motion-corrected and updated by a predictive motion model to reflect part of the predicted state at time (t+1), i.e., state t+1 pred The remaining potential traces for time (t+1) are derived using the same algorithm described above (random sampling, sampling of the furthest point, or another). In total, the potential traces are obtained by stacking propagated traces. t+1 pred and potential new traces of object t potenziell educated. (ii) Furthermore, the datat+1 detected by the sensor and processed by the lidar backbone to determine the data characteristics at time (t+1), i.e., characteristic t+1 , to generate. (iii) The attention-based mechanism is applied as described above: A t+1 = Attention (Key = Feature) t+1 , query = state t+1 pred ), values ​​in turn come from the keys. (iv) The output of this attention mechanism in the context of tracking is updated object state t+1 aktualisiert . This latent spatial representation of the state can then be passed on as traces for the next time cycle and entered into the detection head for BBX extraction.

[0074] The same principle as in Fig. The description in sections 4 to 6 applies to radar PC and camera features that were 3D-projected onto the BEV using a specific method.

Claims

[1] Computer-implemented method (100) for classifying at least one object (50) in an environment (62) of a vehicle (60) using a sensor fusion-based approach and a data-driven model (70), comprising the following steps: - capturing (102) first data (12) from a first sensor (10) within a first data acquisition frame (14); - capturing (104) second data (22) from at least one second sensor (20) within a second data acquisition frame (24); - determining (106) a first object representation (30) using the first data (12); - determining (108) a second object representation (32) using the second data (22); - updating (110) the first object representation (30) and / or the second object representation (32) in dependence on the arrival of third data (36) from the at least second sensor (20) which were acquired in a third data acquisition frame (38) after the first data acquisition frame (14); - fusing (112) the first representation (30) and the at least second representation (32) to determine an updated representation (40) of the object (60) based on the received data (12, 14); - applying (114) the updated representation (40) for training the data-driven model (70) as input data (72) for the data-driven model (70) to obtain output data (74) containing information about a classification of the detected object (50). [2] Computer-implemented method (100) according to claim 1, wherein the first data acquisition frame (14) and / or the second data acquisition frame (24) is represented by a data acquisition window (42, 43) of fixed length and within a defined time interval during which the data acquisition of the first sensor (10) and / or the at least second sensor (20) is carried out. [3] Computer-implemented method (100) according to one of the preceding claims, wherein the updating (110) of the first object representation (30) and / or the second object representation (32) comprises updating a state information item (33, 34) of the current first object representation (30) and / or the second object representation (32) at a time t. [4] Computer-implemented method (100) according to one of the preceding claims, wherein during the step (110) of updating the first object representation (30) and / or the second object representation (32) at time t, the step (111) comprises detecting state information (35) of at least one potential second object (52) at time t. [5] Vehicle (200) comprising a system (202) implementing the computer-implemented method according to claims 1 to 4. [6] A computer comprising a processor adapted to perform the computer-implemented method according to any one of the preceding claims 1 to 4. [7] A computer program product comprising instructions which, when the computer program is executed by a processor of a computer according to claim 6, cause the computer to perform the method according to any one of claims 1 to 4. [8] Machine-readable data carrier and / or download product containing the computer program according to claim 7.

Images