Computer-implemented method and a driver assistance system for more efficiently processing sensor data, vehicle and computer program product

Abstract

The invention relates to a computer-implemented method for more efficiently processing sensor data for an at least semi-autonomous driver assistance system of a vehicle, comprising the steps of: acquiring (110) sensor data by means of at least two different vehicle sensors, the acquired sensor data representing the surroundings of the vehicle; preparing (120) each piece of the sensor data to form a feature vector by means of a neural network; selecting (150) the relevant sensor data on the basis of at least some of the feature vectors representing the sensor data by means of a sparse gating network; determining (160) perception data on the basis of the relevant sensor data by means of at least one trained machine recognition method; and controlling (170) the vehicle on the basis of the determined perception data.

Description

[0001] R.414411

[0002] - 1 -

[0003] Description

[0004] Computer-implemented method and driver assistance system for more efficient processing of sensor data, vehicle and computer program product

[0005] The present invention relates to a computer-implemented method and a driver assistance system for the more efficient processing of sensor data for an at least partially autonomous driver assistance system of a vehicle, in particular for driving assistance, maneuvering assistance, and / or parking assistance. The invention also includes a computer program for carrying out the method and a vehicle with the driver assistance system.

[0006] State of the art

[0007] Modern vehicles are equipped with a multitude of vehicle sensors of various types, such as cameras, radar sensors, LiDAR sensors, and / or ultrasonic sensors, each of which captures the vehicle's surroundings. The processing of this sensor data represents environmental perception, which is inherently essential for driver assistance systems if at least semi-autonomous driving functions are to be performed. However, the increasing number of sensors and the associated growing volume of data pose a challenge for the vehicle's bus systems, memory, and processing units. Although all sensors are active in principle, in a specific driving situation often only the sensor data from one or a few sensors, which may be of different types, are relevant.

[0008] An encoder-decoder architecture is a neural network architecture particularly useful in areas such as machine translation, text summarization, and image description. The basic idea of ​​an encoder-decoder architecture is to encode input data (e.g., a sentence in one language) into a compressed representation (also called a "context vector") and then decode this into the desired output data (e.g., the same sentence in another language). An attention mechanism allows the decoder to focus on specific parts of the input sequence at each step, improving performance with long sequences. Transformer models utilize this attention mechanism to model the relationships between words in the sequence. This enables R.414411

[0009] - 2 - faster training and higher performance, especially with large datasets.

[0010] Document US 12 067 476 B2 reveals a Mixture-of-Experts (MoE) approach.

[0011] The Mixture-of-Experts (MoE) approach is a machine learning model that aims to improve the output of a trained machine recognition algorithm, particularly a neural network, by combining multiple specialized models or experts. These experts are specialized models, each trained on different aspects of a problem. The specialized models can each incorporate different architectures, such as neural networks, decision trees, and / or other machine learning models. A gating mechanism determines which expert, or combination of experts, is best suited for a given input. The gating mechanism is a model that learns or decides how to distribute the inputs among the various experts. The gating mechanism can include a neural network, a decision tree, and / or other machine learning models.It calculates probabilities or weights for each input, determining how much each expert contributes to the final output. The experts' outputs are then combined based on the weights of the gating mechanism to generate the final prediction.

[0012] Ilya Sutskever et al. (2019) "Generating long sequences with sparse transformers" reveals a sparse gating network.

[0013] A sparse gating network is a special type of gating mechanism used in the Mixture-of-Experts (MoE) approach. The main difference between a traditional gating mechanism and a sparse gating network lies in how the experts are selected and combined. The sparse gating network calculates weights or probabilities for each expert, selecting only those with the highest weights or probabilities.

[0014] The paper “Mixture-of-Depths: Dynamically allocating compute in transformer-based language models” by D. Raposo et al. (2024) (arXiv:2404.02258) reveals a Mixture-of-Depths (MoD) approach. R.414411

[0015] - 3 -

[0016] The object of the present invention is to improve the efficiency of sensor data processing for a vehicle.

[0017] Disclosure of the invention

[0018] The above problem is solved according to the invention in accordance with independent claims 1, 6, 11 and 12.

[0019] The invention relates to a computer-implemented method for the more efficient processing of sensor data for an at least partially autonomous vehicle assistance system. The method comprises the acquisition of sensor data by means of at least two different vehicle sensors, in particular of different sensor types, specifically cameras, radar sensors, LiDAR sensors, and / or ultrasonic sensors of the vehicle, wherein the acquired sensor data of each sensor represents at least a part of the vehicle's environment. The vehicle sensors for sensing the environment can advantageously be of the same sensor type but oriented in different directions within the vehicle's environment. The particularly advantageous use of different sensor types allows the advantages of the individual sensor technologies to be combined and a more comprehensive and partially redundant picture of the vehicle's environment to be generated, which also makes the method safer.Subsequently, at least one feature vector, or processed sensor data, is determined based on the sensor data acquired by each vehicle sensor for a given feature vector using a neural network. This neural network specifically has an encoder-decoder structure. Advantageously, neural networks, particularly those with an encoder-decoder structure, are well-suited for recognizing complex data patterns and transforming the sensor data into a more suitable form for further processing. In the following steps, relevant sensor data is determined or selected based on the feature vectors representing the sensor data using a sparse gating network. The sparse gating network is advantageously configured to identify relevant sensor data for the respective driving situation and to ignore irrelevant and / or superfluous sensor data based on the determined feature vectors.This reduces the computational load. In a further step, perception data or output data are determined based on the selected relevant sensor data or the selected data that R.414411.

[0020] - 4 - feature vectors representing relevant sensor data are determined by at least one trained machine learning method, wherein the trained machine learning method includes, in particular, a transformer and advantageously downstream trained machine learning methods or neural networks for detecting or determining objects in the vehicle's environment and / or the occupancy of the vehicle's environment and / or a map of the environment. In other words, the feature vectors representing the relevant sensor data are subsequently used by a machine learning method, such as a transformer, to generate perceptual data. This perceptual data represents an interpreted representation of the relevant vehicle environment, for example, detected dynamic and / or static objects, a map, and / or a segmentation of the environment.Furthermore, the vehicle is controlled based on the acquired perception data. Specifically, the steering and / or drive of the vehicle are controlled according to driving trajectories determined from this perception data. Consequently, the perception data, derived solely from relevant sensor data, is used to guide the vehicle through traffic with maximum efficiency, thereby reducing the costs associated with the driver assistance system and the vehicle itself.

[0021] In a particularly preferred implementation, the feature vectors or the processed sensor data are merged. In other words, the feature vectors or the processed sensor data are fused in a common vector space. In this implementation, the relevant sensor data are determined or selected based on the merged or fused sensor data or on the vector space using a sparse gating network. This prior fusion allows, in particular, the identification of sensor data or their feature vectors representing currently irrelevant environmental areas as irrelevant, and the identification of sensor data from different sensors for relevant environmental areas as relevant.

[0022] As an optional step, the sensor data identified as unnecessary can be masked in the merged sensor data depending on the determined relevant R.414411.

[0023] - 5 -

[0024] Sensor data is processed. Masking irrelevant data can further increase the efficiency of subsequent processing steps.

[0025] Preferably, the sparse gating network is based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach. Using a sparse gating network based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach offers advantages in terms of system scalability and flexibility.

[0026] In one embodiment of the invention, the selection or determination of the relevant sensor data is additionally or in parallel performed by means of a filter algorithm depending on the acquired or combined sensor data or depending on the feature vectors representing the acquired or combined sensor data. For example, the additionally or in parallel filter algorithm may include a case distinction which, if the brightness in a camera image exceeds a predetermined first threshold value, representing driving in daylight, filters out or masks the radar data or selects or determines the sensor data derived from the sensor data acquired by the vehicle camera as the relevant sensor data.Alternatively or additionally, the filter algorithm includes the following: if the brightness in a camera image falls below a predefined second threshold, representing night driving or a defective vehicle camera, the sensor data derived from the vehicle camera is either hidden or masked, or the radar data acquired by the at least one radar sensor is determined as the relevant sensor data. The additional use of a filter algorithm can thus further improve the robustness of the method and the accuracy of selecting relevant sensor data.

[0027] In a further embodiment, an additional step involves determining the current driving situation. This determination is based, in particular, on the sensor data acquired from a vehicle sensor, especially the vehicle camera, or the corresponding feature vector, and / or on the fused sensor data, each using a trained machine recognition method. Alternatively or additionally, the driving situation is determined based on a vehicle position, which is determined, for example, by a satellite-based navigation system, and / or based on the vehicle's speed and / or other operating parameters. For example, R.414411

[0028] - 6 -

[0029] A highway drive or a parking situation is determined based on the vehicle's position. Subsequently, the relevant sensor data is selected or determined using the sparse-gating network, depending on the current driving situation. For example, during a highway drive, the radar and camera sensor data are identified as relevant, while the ultrasonic sensor data is deemed irrelevant. Conversely, for a parking situation, the ultrasonic sensor data might be identified as relevant, while at least some of the radar sensor data might be deemed irrelevant. Considering the current driving situation allows for a more targeted selection of relevant sensor data, as the relevance of individual sensor data can change depending on the driving situation.

[0030] The invention further relates to a driver assistance system for the more efficient processing of sensor data for an at least partially autonomous driving assistance system of a vehicle. The driving assistance system comprises at least two sensor units, preferably of different types, for acquiring sensor data, wherein the sensor data represents the vehicle's environment. The sensor units include, in particular, cameras, radar sensors, LiDAR sensors, and / or ultrasonic sensors. The driving assistance system further comprises a processing unit, which is configured to process the sensor data, in particular from each sensor type, into a feature vector using a neural network, wherein the neural network in particular has an encoder-decoder structure. The driving assistance system further comprises a selection unit, which is configured to determine relevant sensor data based on the feature vectors using a sparse gating network.Preferably, the sparse gating network of the selection unit is based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach. Furthermore, the driver assistance system comprises a recognition unit for determining perceptual data based on the relevant sensor data using at least one trained machine recognition algorithm, in particular a transformer. The driver assistance system also includes a control unit for controlling the vehicle based on the determined perceptual data.

[0031] In a particularly preferred embodiment, the driver assistance system includes a fusion unit configured to combine or fuse the feature vectors into a common vector space. In this embodiment, the selection unit is configured to use the sparse- R.414411

[0032] - 7 -

[0033] To determine relevant sensor data for a gating network depending on the feature vectors representing the sensor data to be merged, or depending on the common vector space.

[0034] Optionally, the selection unit is configured to mask or filter the merged sensor data depending on the relevant sensor data identified, whereby all irrelevant sensor data or sensor data recognized as unnecessary are masked or filtered, meaning they are not further processed by the transformer to control the vehicle.

[0035] In a further development, the selection unit is set up to determine the relevant sensor data depending on the combined sensor data, additionally or in parallel to the sparse gating network, using the filter algorithm.

[0036] In another embodiment, the driver assistance system includes a situation determination unit for determining a current driving situation, wherein the selection unit is designed to additionally determine the relevant sensor data depending on the determined current driving situation by means of the sparse gating network.

[0037] The invention also relates to a computer program product with program code stored on a computer-readable data carrier, wherein the program code, when executed by a processor, performs the steps of the method according to the invention.

[0038] The invention also relates to a vehicle which has the driver assistance system according to the invention.

[0039] The invention offers the following technical and economic advantages: By focusing on relevant sensor data, the computational load is reduced and / or more resources are available for other tasks. Furthermore, the energy consumption of a computing device configured to perform the method can be reduced. The combination of various sensors and the use of advanced algorithms for data evaluation increase the accuracy and robustness of the driver assistance system. The use of neural networks and machine learning methods enables flexible R.414411

[0040] - 8 -

[0041] Adaptation to different vehicle sensors, vehicle models, and driving situations. The potential use of more cost-effective sensors and efficient processing of sensor data can reduce the costs of driver assistance systems.

[0042] The invention is particularly relevant for use in semi-autonomous vehicles, as it provides the basis for reliable and safe environmental perception and interpretation. However, the invention can also be used in other fields where large amounts of sensor data need to be processed, such as robotics.

[0043] Further advantages will become apparent from the following description of exemplary embodiments with reference to the figures.

[0044] Figure 1: Flowchart of the process as a block diagram

[0045] Examples of implementation

[0046] Figure 1 schematically depicts a flowchart of the process as a block diagram. In step 110, sensor data is acquired. This is done using at least two sensors, preferably of different types. The sensors include, in particular, at least one camera, one radar sensor, one LiDAR sensor, and one ultrasonic sensor. The sensors continuously acquire data from the vehicle's surroundings, so that the sensor data represents the vehicle's environment. This sensor data, or raw data, can represent various aspects of the environment, such as visual information, distances, and movements. In a subsequent step 120, the acquired sensor data, or raw data, is transformed into feature vectors by neural networks, which are specifically trained for each sensor type. These feature vectors contain abstracted and relevant information suitable for further processing.The raw data from each sensor unit is advantageously processed by a neural network, e.g., with an encoder-decoder structure, and transformed into a compact feature vector. Subsequently, in optional step 130, data fusion is performed. The feature vectors of all sensor units are merged into a common vector space. The feature vectors of the different sensors are shown in R.414411.

[0047] - 9 - especially different sensor types, are combined in a common vector space in step 130. This enables a consolidated and comprehensive representation of the vehicle environment. In an optional step 140, a driving situation is determined. The optional determination 140 of the driving situation can be carried out depending on the sensor data of a vehicle sensor, depending on the vector space, depending on operating parameters of the vehicle, such as the vehicle speed and / or the vehicle position and their respective trajectories, and / or depending on map data, and / or by a trained machine recognition method, in particular a neural network. An optional module can therefore determine the current driving situation (e.g.,The current driving situation (e.g., highway, city traffic) is determined based on the acquired or fused sensor data or the feature vectors representing it, the vehicle position, and / or the vehicle speed. The determined current driving situation is provided to the sparse gating network. In a subsequent step 150, the relevance of the proportions of the acquired or fused sensor data is determined. A sparse gating network based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach identifies the relevant sensor data in step 150. The relevant sensor data can also be determined in step 150 depending on the specific driving situation. Identifying the relevant sensor data reduces the subsequent computational load by ignoring irrelevant data.In other words, a selection of 150 relevant sensor data points is made based on at least a portion of the acquired or merged sensor data, or the feature vectors representing it, using a sparse gating network. This sparse gating network analyzes the acquired or fused sensor data, the feature vectors representing it, or the vector space, and selects the most relevant information as relevant sensor data. Alternatively or additionally, the selection of relevant sensor data can be made by the sparse gating network based on the current driving situation. Optionally, the selection can be further supported by a filter algorithm.

[0048] Subsequently, in step 160, perceptual data are determined based on the relevant sensor data using at least one trained machine learning algorithm, in particular a neural network, preferably with a transformer structure. In other words, perceptual data are generated from the selected, relevant sensor data using a machine learning algorithm (e.g., a transformer). This data represents an R.414411

[0049] - 10 -

[0050] Interpretation of the environment, e.g., detected objects, a map, and / or segmentation of the environment, for example, into drivable and non-drivable areas. In other words, in step 160, the relevant sensor data is analyzed by a machine learning process, such as a transformer. This process generates perceptual data that represents an interpreted representation of the vehicle's environment, including detected objects, maps, and / or segmentations. Subsequently, in step 170, the vehicle is controlled based on the acquired perceptual data. In other words, control commands for the vehicle are generated based on the acquired perceptual data. These commands control the steering, drive, and other vehicle systems to guide the vehicle safely and efficiently through traffic.In other words, the control unit controls the vehicle, preferably the steering and / or the drive, based on the perception data, for example by calculating driving trajectories and passing them on to the appropriate vehicle systems.

Claims

R.414411 - 11 - Claims 1. A computer-implemented method for the more efficient processing of sensor data for an at least partially autonomous vehicle assistance system, comprising the steps of: a. Acquisition (110) of sensor data using at least two different vehicle sensors, wherein the acquired sensor data represent the vehicle's environment; b. Processing (120) the respective sensor data into a feature vector by a neural network; c. Selection (150) of the relevant sensor data depending on at least a part of the feature vectors representing the sensor data using a sparse gating network; d. Determination (160) of perception data depending on the relevant sensor data by at least one trained machine recognition method; and e. Control (170) of the vehicle based on the determined perception data.

2. Method according to claim 1, wherein after processing (120) the sensor data into a feature vector, a fusion (130) of the sensor data into a common vector space is carried out and the selection (150) of the relevant sensor data is performed based on the determined vector space.

3. Method according to any of the preceding claims, wherein the sparse gating network is based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach.

4. Method according to one of the preceding claims, wherein the selection (140) of the relevant sensor data is additionally or in parallel carried out by means of a filter algorithm based on the acquired or fused sensor data.

5. Method according to one of the preceding claims, wherein the following steps are performed: a. Determination (140) of a current driving situation; b. Selection (150) of the relevant sensor data additionally depending on the determined current driving situation by means of the sparse gating network. R.414411 - 12 - 6. A driver assistance system for the more efficient processing of sensor data for an at least partially autonomous driver assistance system of a vehicle, comprising: a. at least two sensor units for acquiring (110) sensor data representing the vehicle's environment; b. a processing unit configured to process (120) the sensor data into a feature vector using a neural network; c. a selection unit configured to determine relevant sensor data based on the feature vectors representing the sensor data using a sparse gating network; d. a recognition unit for determining (160) perception data based on the relevant sensor data using at least one trained machine recognition method; and e. a control unit for controlling (170) the vehicle based on the determined perception data.

7. System according to claim 6, a fusion unit for merging (130) the feature vectors into a common vector space, wherein the selection unit is configured to determine the relevant sensor data based on the fused sensor data or based on the determined vector space; 8. System according to claim 6 or 7, wherein the sparse gating network of the selection unit is based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach.

9. System according to any one of claims 6 to 8, wherein the selection unit is configured to determine the relevant sensor data additionally or in parallel to the sparse gating network by means of a filter algorithm based on the acquired or fused sensor data.

10. System according to one of claims 6 to 9, further comprising a situation determination unit for determining (140) a current driving situation, wherein the selection unit is configured to additionally determine the relevant sensor data depending on the determined current driving situation by means of the sparse gating network. R.414411 - 13 - 11. Computer program product comprising program code stored on a computer-readable data carrier, wherein the program code, when executed by a processor, performs the steps of the method according to any one of claims 1 to 5.

12. Vehicle, comprising at least a driver assistance system according to one of claims 6 to 9.

Images