Computer-implemented method and driver assistance system for more efficient processing of sensor data, vehicle and computer program product

Abstract

A computer-implemented method for the more efficient processing of sensor data for an at least partially autonomous vehicle assistance system, comprising the steps of: acquiring (110) sensor data using at least two different vehicle sensors, wherein the acquired sensor data represent the vehicle's environment; processing (120) the respective sensor data into a feature vector by a neural network; selecting (150) the relevant sensor data depending on at least a part of the feature vectors representing the sensor data using a sparse gating network; determining (160) perception data depending on the relevant sensor data by at least one trained machine recognition method; and controlling (170) the vehicle based on the determined perception data.

Description

[0001] The present invention relates to a computer-implemented method and a driver assistance system for the more efficient processing of sensor data for an at least partially autonomous driver assistance system of a vehicle, in particular for driving assistance, maneuvering assistance, and / or parking assistance. The invention also includes a computer program for carrying out the method and a vehicle with the driver assistance system. State of the art

[0002] Modern vehicles are equipped with a multitude of vehicle sensors of various types, such as cameras, radar sensors, LiDAR sensors, and / or ultrasonic sensors, each of which captures the vehicle's surroundings. The processing of this sensor data represents environmental perception, which is inherently essential for driver assistance systems if at least semi-autonomous driving functions are to be performed. However, the increasing number of sensors and the associated growing volume of data pose a challenge for the vehicle's bus systems, memory, and processing units. Although all sensors are active in principle, in a specific driving situation often only the sensor data from one or a few sensors, which may be of different types, are relevant.

[0003] An encoder-decoder architecture is a neural network architecture particularly useful in areas such as machine translation, text summarization, and image description. The basic idea behind an encoder-decoder architecture is to encode input data (e.g., a sentence in one language) into a compressed representation (also called a "context vector") and then decode this into the desired output data (e.g., the same sentence in another language). An attention mechanism allows the decoder to focus on specific parts of the input sequence at each step, improving performance with long sequences. Transformer models utilize this attention mechanism to model the relationships between words within the sequence. This enables faster training and higher performance, especially with large datasets.

[0004] Document US 12 067 476 B2 reveals a Mixture-of-Experts (MoE) approach.

[0005] The Mixture-of-Experts (MoE) approach is a machine learning model that aims to improve the output of a trained machine recognition algorithm, particularly a neural network, by combining multiple specialized models or experts. These experts are specialized models, each trained on different aspects of a problem. The specialized models can each incorporate different architectures, such as neural networks, decision trees, and / or other machine learning models. A gating mechanism determines which expert, or combination of experts, is best suited for a given input. The gating mechanism is a model that learns or decides how to distribute the inputs among the various experts. The gating mechanism can include a neural network, a decision tree, and / or other machine learning models.It calculates probabilities or weights for each input, determining how strongly each expert contributes to the final output. The experts' outputs are then combined based on the weights of the gating mechanism to generate the final prediction.

[0006] Ilya Sutskever et al. (2019) “Generating long sequences with sparse transformers” reveals a sparse gating network,

[0007] A sparse gating network is a special type of gating mechanism used in the Mixture-of-Experts (MoE) approach. The main difference between a traditional gating mechanism and a sparse gating network lies in how the experts are selected and combined. The sparse gating network calculates weights or probabilities for each expert, selecting only those with the highest weights or probabilities.

[0008] The paper “Mixture-of-Depths: Dynamically allocating compute in transformer-based language models” by D. Raposo et al. (2024) (arXiv:2404.02258) reveals a Mixture-of-Depths (MoD) approach.

[0009] The object of the present invention is to improve the efficiency of sensor data processing for a vehicle. Disclosure of the invention

[0010] The above problem is solved according to the invention in accordance with independent claims 1, 6, 11 and 12.

[0011] The invention relates to a computer-implemented method for the more efficient processing of sensor data for an at least partially autonomous driving assistance system of a vehicle. The method comprises the acquisition of sensor data by means of at least two different vehicle sensors, in particular of different sensor types, especially by means of cameras, radar sensors, LiDAR sensors and / or ultrasonic sensors of the vehicle, wherein the acquired sensor data of each sensor represents at least a part of the vehicle's environment. The vehicle sensors for acquiring the environment can advantageously be of the same sensor type but oriented in different directions within the vehicle's environment.

[0012] The particularly advantageous use of different sensor types allows the benefits of individual sensor technologies to be combined, generating a more comprehensive and partially redundant picture of the vehicle's environment, which also enhances the safety of the process. Subsequently, at least one feature vector, or processed sensor data, is determined based on the sensor data acquired by each vehicle sensor and assigned to a feature vector using a neural network. Specifically, the neural network has an encoder-decoder structure. Neural networks, especially those with an encoder-decoder structure, are advantageously well-suited for recognizing complex data patterns and transforming the sensor data into a more suitable format for further processing. In the following steps, relevant sensor data is determined or selected based on the feature vectors representing the sensor data using a sparse gating network.The sparse gating network is advantageously configured to identify relevant sensor data for the respective driving situation and to ignore irrelevant and / or superfluous sensor data based on the determined feature vectors. This reduces the computational load. In a further step, perception data or output data are determined based on the selected relevant sensor data or the selected feature vectors representing the relevant sensor data by at least one trained machine recognition method. This trained machine recognition method includes, in particular, a transformer and advantageously subsequent trained machine recognition methods or neural networks for detecting or determining objects in the vehicle's environment and / or the occupancy of the vehicle's environment and / or a map of the environment.In other words, the feature vectors representing the relevant sensor data are subsequently used by a machine learning algorithm, such as a transformer, to generate perceptual data. This perceptual data represents an interpreted representation of the relevant vehicle environment, for example, detected dynamic and / or static objects, a map, and / or a segmentation of the environment. Furthermore, vehicle control based on the acquired perceptual data is provided. Specifically, vehicle control involves regulating the steering and / or drive system to guide the vehicle along trajectories determined based on the perceptual data.The perception data, determined solely from the relevant sensor data, is therefore used to guide the vehicle through traffic with maximum efficiency, thereby also reducing the costs for the driver assistance system or the vehicle itself.

[0013] In a particularly preferred implementation, the feature vectors or the processed sensor data are merged. In other words, the feature vectors or the processed sensor data are fused in a common vector space. In this implementation, the relevant sensor data are determined or selected based on the merged or fused sensor data or on the vector space using the sparse gating network. This prior fusion allows, in particular, the identification of sensor data or their feature vectors representing currently irrelevant environmental areas as irrelevant, and the identification of sensor data from different sensors for relevant environmental areas as relevant.

[0014] As an optional step, sensor data identified as unnecessary can be masked within the merged sensor data, depending on the relevant sensor data determined. Masking irrelevant data can further increase the efficiency of subsequent processing steps.

[0015] Preferably, the sparse gating network is based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach. Using a sparse gating network based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach offers advantages in terms of system scalability and flexibility.

[0016] In one embodiment of the invention, the selection or determination of the relevant sensor data is additionally or in parallel performed by means of a filter algorithm depending on the acquired or combined sensor data or depending on the feature vectors representing the acquired or combined sensor data. For example, the additionally or in parallel filter algorithm may include a case distinction which, if the brightness in a camera image exceeds a predetermined first threshold value, representing driving in daylight, filters out or masks the radar data or selects or determines the sensor data derived from the sensor data acquired by the vehicle camera as the relevant sensor data.Alternatively or additionally, the filter algorithm includes the following: if the brightness in a camera image falls below a predefined second threshold, representing night driving or a defective vehicle camera, the sensor data derived from the vehicle camera is either hidden or masked, or the radar data acquired by the at least one radar sensor is determined as the relevant sensor data. The additional use of a filter algorithm can thus further improve the robustness of the method and the accuracy of selecting relevant sensor data.

[0017] In a further embodiment, an additional step involves determining the current driving situation. This determination is based primarily on the sensor data acquired from a vehicle sensor, particularly the vehicle camera, or the corresponding feature vector, and / or on the fused sensor data, each using a trained machine recognition algorithm. Alternatively or additionally, the driving situation is determined based on the vehicle's position, which is ascertained, for example, by a satellite-based navigation system, and / or based on the vehicle's speed and / or other operating parameters. For example, highway driving or a parking maneuver is defined as the driving situation based on the vehicle's positional trajectory.Subsequently, the relevant sensor data is further selected or determined based on the specific current driving situation using the sparse gating network. For example, for highway driving, the sensor data from the radar and camera sensors is determined to be relevant, while the ultrasonic sensor data is deemed irrelevant. Conversely, for a parking situation, for instance, the ultrasonic sensor data might be determined to be relevant, and at least some of the radar sensor data to be irrelevant. Considering the current driving situation allows for an even more targeted selection of relevant sensor data, as the relevance of individual sensor data can change depending on the driving situation.

[0018] The invention further relates to a driver assistance system for the more efficient processing of sensor data for an at least partially autonomous driving assistance system of a vehicle. The driving assistance system comprises at least two sensor units, preferably of different types, for acquiring sensor data, wherein the sensor data represents the vehicle's environment. The sensor units include, in particular, cameras, radar sensors, LiDAR sensors, and / or ultrasonic sensors. The driving assistance system further comprises a processing unit, which is configured to process the sensor data, in particular from each sensor type, into a feature vector using a neural network, wherein the neural network in particular has an encoder-decoder structure. The driving assistance system further comprises a selection unit, which is configured to determine relevant sensor data based on the feature vectors using a sparse gating network.Preferably, the sparse gating network of the selection unit is based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach. Furthermore, the driver assistance system comprises a recognition unit for determining perceptual data based on the relevant sensor data using at least one trained machine recognition algorithm, in particular a transformer. The driver assistance system also includes a control unit for controlling the vehicle based on the determined perceptual data.

[0019] In a particularly preferred embodiment, the driver assistance system includes a fusion unit configured to merge or fuse the feature vectors into a common vector space. In this embodiment, the selection unit is configured to determine relevant sensor data using the sparse gating network, depending on the feature vectors representing the sensor data to be merged, or depending on the common vector space.

[0020] Optionally, the selection unit is configured to mask or filter the merged sensor data depending on the relevant sensor data identified, whereby all irrelevant sensor data or sensor data recognized as unnecessary are masked or filtered, meaning they are not further processed by the transformer to control the vehicle.

[0021] In a further development, the selection unit is set up to determine the relevant sensor data depending on the combined sensor data, additionally or in parallel to the sparse gating network, using the filter algorithm.

[0022] In another embodiment, the driver assistance system includes a situation determination unit for determining a current driving situation, wherein the selection unit is designed to additionally determine the relevant sensor data depending on the determined current driving situation by means of the sparse gating network.

[0023] The invention also relates to a computer program product with program code stored on a computer-readable data carrier, wherein the program code, when executed by a processor, performs the steps of the method according to the invention.

[0024] The invention also relates to a vehicle which has the driver assistance system according to the invention.

[0025] The invention offers the following technical and economic advantages: By focusing on relevant sensor data, the computational load is reduced and / or more resources are available for other tasks. Furthermore, the energy consumption of a computing device configured to perform the process can be reduced. The combination of various sensors and the use of advanced algorithms for data evaluation increase the accuracy and robustness of the driver assistance system. The use of neural networks and machine learning methods enables flexible adaptation to different vehicle sensors, vehicle models, and driving situations. The potential use of more cost-effective sensors and the efficient processing of sensor data can reduce the costs of driver assistance systems.

[0026] The invention is particularly relevant for use in semi-autonomous vehicles, as it provides the basis for reliable and safe environmental perception and interpretation. However, the invention can also be used in other fields where large amounts of sensor data need to be processed, such as robotics.

[0027] Further advantages will become apparent from the following description of exemplary embodiments with reference to the figures. Fig. 1: Flowchart of the process as a block diagram Fig. 2: Description of the situation from a top view Examples of implementation

[0028] In Fig.Figure 1 is a schematic flowchart of the process, presented as a block diagram. In step 110, sensor data is acquired. This is done using at least two sensors, preferably of different types. The sensors include, in particular, at least one camera, one radar sensor, one LiDAR sensor, and one ultrasonic sensor. The sensors continuously acquire data from the vehicle's surroundings, so that the sensor data represents the vehicle's environment. This sensor data, or raw data, can represent various aspects of the environment, such as visual information, distances, and movements. In a subsequent step 120, the acquired sensor data, or raw data, is transformed into feature vectors by neural networks, which are specifically trained for each sensor type. These feature vectors contain abstracted and relevant information suitable for further processing.The raw data from each sensor unit is advantageously processed by a neural network, e.g., with an encoder-decoder structure, and transformed into a compact feature vector. Subsequently, in optional step 130, data fusion is performed. The feature vectors of all sensor units are merged into a common vector space. The feature vectors of the different sensors, especially those of different sensor types, are combined in this common vector space in step 130. This enables a consolidated and comprehensive representation of the vehicle's surroundings. In optional step 140, a driving situation is determined.The optional determination of the driving situation (step 140) can be performed based on sensor data from a vehicle sensor, the vector space, vehicle operating parameters such as vehicle speed and / or vehicle position and their respective profiles, map data, and / or a trained machine recognition method, particularly a neural network. Thus, in step 140, an optional module can determine the current driving situation (e.g., highway, city traffic) based on the acquired or fused sensor data or the feature vectors representing it, the vehicle position, and / or the vehicle speed. The determined current driving situation is then provided to the sparse gating network. In a subsequent step (step 150), the relevance of the contributions of the acquired or fused sensor data is determined.In step 150, a sparse gating network based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach identifies the relevant sensor data. This relevant sensor data can also be determined in step 150 based on the specific driving situation. Identifying the relevant sensor data reduces the subsequent computational load by ignoring irrelevant data. In other words, a selection of relevant sensor data is made based on at least a portion of the acquired or merged sensor data, or the feature vectors representing it, using a sparse gating network. This sparse gating network analyzes the acquired or fused sensor data, the feature vectors representing it, or the vector space, and selects the most relevant information as the relevant sensor data.Alternatively or additionally, the selection of relevant sensor data is carried out by the sparse gating network based on the current driving situation. Optionally, the selection can be further supported by a filter algorithm.

[0029] Subsequently, in step 160, perceptual data is determined based on the relevant sensor data using at least one trained machine learning algorithm, in particular a neural network, preferably with a transformer structure. In other words, perceptual data is generated from the selected, relevant sensor data using a machine learning algorithm (e.g., a transformer). This data represents an interpretation of the environment, such as detected objects, a map, and / or a segmentation of the environment, for example, into drivable and non-drivable areas. In other words, in step 160, the relevant sensor data is analyzed by a machine learning algorithm, such as a transformer. This algorithm generates perceptual data that represents an interpreted representation of the vehicle's environment, including detected objects, maps, and / or segmentations.The vehicle is then controlled in step 170 based on the acquired perception data. In other words, control commands for the vehicle are generated based on the acquired perception data. These commands control the steering, drive, and other vehicle systems to guide the vehicle safely and efficiently through traffic. In other words, the control unit controls the vehicle, preferably the steering and / or drive, based on the perception data, for example by calculating driving trajectories and transmitting them to the appropriate vehicle systems.

[0030] The ego-vehicle is driving on the highway and wants to overtake a truck. The vehicle's sensors detect its surroundings, including the truck and a car approaching from behind. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 12 067 476 B2

[0004]

Claims

[1] Computer-implemented method for more efficient processing of sensor data for at least a partially autonomous vehicle assistance system, comprising the steps: a. Acquisition (110) of sensor data using at least two different vehicle sensors, wherein the acquired sensor data represent the environment of the vehicle; b. Processing (120) the respective sensor data into a feature vector by a neural network; c. Selection (150) of the relevant sensor data depending on at least a part of the feature vectors representing the sensor data using a sparse gating network; d. Determination (160) of perception data depending on the relevant sensor data by at least one trained machine recognition method; and e. Control (170) of the vehicle based on the determined perception data. [2] Method according to claim 1, wherein after processing (120) the sensor data into a feature vector, a fusion (130) of the sensor data into a common vector space is carried out and the selection (150) of the relevant sensor data is carried out based on the determined vector space. [3] Method according to any of the preceding claims, wherein the sparse gating network is based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach. [4] Method according to one of the preceding claims, wherein the selection (140) of the relevant sensor data is additionally or in parallel carried out by means of a filter algorithm based on the acquired or fused sensor data. [5] A method according to any of the preceding claims, wherein the following steps are carried out a. Determination (140) of a current driving situation; b. Selection (150) of the relevant sensor data additionally depending on the specific current driving situation using the sparse gating network. [6] Driver assistance system for more efficient processing of sensor data for at least a partially autonomous driver assistance system of a vehicle, comprising: a. at least two sensor units for recording (110) sensor data representing the vehicle's environment; b. a processing unit which is designed to process (120) the sensor data into a feature vector by means of a neural network; c. a selection unit which is set up to determine relevant sensor data depending on the feature vectors representing the sensor data using a sparse gating network; d. a recognition unit for determining (160) perception data depending on the relevant sensor data by means of at least one trained machine recognition method; and e. a control unit for controlling (170) the vehicle based on the determined perception data. [7] System according to claim 6, a fusion unit for merging (130) the feature vectors into a common vector space, wherein the selection unit is configured to determine the relevant sensor data based on the fused sensor data or based on the determined vector space; [8] System according to claim 6 or 7, wherein the sparse gating network of the selection unit is based on the Mixture-of-Experts (MoE) and / or Mixture-of-Depths (MoD) approach. [9] System according to any one of claims 6 to 8, wherein the selection unit is configured to determine the relevant sensor data additionally or in parallel to the sparse gating network by means of a filter algorithm based on the acquired or fused sensor data. [10] System according to one of claims 6 to 9, further comprising a situation determination unit for determining (140) a current driving situation, wherein the selection unit is configured to additionally determine the relevant sensor data depending on the determined current driving situation by means of the sparse gating network. [11] Computer program product comprising program code stored on a computer-readable data carrier, wherein the program code, when executed by a processor, performs the steps of the method according to any one of claims 1 to 5. [12] Vehicle, comprising at least a driver assistance system according to any one of claims 6 to 9.

Images