Methods for data acquisition using a data acquisition system and data acquisition system

Abstract

One aspect of the invention relates to a method for data acquisition using a data acquisition system (10) in which sensor data from a plurality of sensor devices (12, 14, 16) are acquired, wherein the sensor data are first processed by corresponding encoders (12a, 14a, 16a) for each sensor device (12, 14, 16) and fused in a neural network. The invention further relates to such a data acquisition system (10).

Description

[0001] The invention relates to a method for data acquisition using a data acquisition system according to the preamble of claim 1. Furthermore, the invention relates to a data acquisition system.

[0002] According to the current state of technology, at least semi-autonomous and automated vehicles require a detailed understanding of their surroundings. Therefore, they are equipped with multiple sensors, such as cameras. Today, almost all perception systems are based on neural networks. The vehicles have specific sensor configurations, meaning that cameras with specific characteristics (such as fisheye, telephoto, and wide-angle lenses), radar, and lidar sensors are installed in particular positions. Autonomous / automated driving requires sophisticated software and hardware to ensure the necessary integrity of the output, such as the detection of objects.

[0003] These outputs are then passed on to subsequent processing steps such as trajectory planning. Modern methods / techniques often combine sensory information using neural networks, typically resulting in the loss of information about which sensors detected an object. However, this information is necessary to evaluate and improve the integrity of these output signals. To this end, we propose a simple approach to achieve integrity by validating a single modality within a single neural network.

[0004] German patent DE 11 2021 000 135 T5 discloses a processor comprising one or more circuits for performing various tasks. These tasks include receiving data representing the outputs of multiple deep neural networks (DNNs), each output associated with a specific sensor and its own field of view. The processor uses a fusion DNN to compute fused outputs based on this data, combining the information from the different sensors. The processor then uses this fused data to perform one or more operations with an autonomous machine.

[0005] The state of the art also reveals, for example, a large number of perception models, which, however, do not focus on secure and redundant multi-ECU architectures.

[0006] The object of the invention is to increase the integrity of sensor data from respective sensor devices and thus to increase the reliability of systems with such sensor devices.

[0007] This problem is solved by means of a method with the features of claim 1 and by means of a data acquisition system according to the invention. Advantageous embodiments of the data acquisition system according to the invention are to be regarded as advantageous embodiments of the method according to the invention, wherein the means of the data acquisition system are used to carry out the method steps. Furthermore, advantageous developments of the invention are described by the dependent claims, the following description, and the figures.

[0008] One aspect of the invention relates to a method for data acquisition using a data acquisition system in which sensor data is acquired from a plurality of sensor devices. According to the invention, the sensor data is first processed by corresponding encoders for each sensor device and then fused in a neural network. The encoders are necessary to provide the sensor data in a format suitable for the neural network.

[0009] The neural network is trained by feeding it datasets that encompass the different perspectives and characteristics of the sensor data. The goal of the training is to teach the network to recognize patterns and relationships in the combined sensor data. An electronic computing unit is, for example, positioned and coupled with the sensor devices. It implements the neural network used for fusing and processing the sensor data. This electronic computing unit is designed to receive the sensor data acquired by the sensor devices, route it through the encoders, and process the data within the neural network. This enables precise fusion of the sensor data to provide a comprehensive representation of, for example, an environment or objects along a trajectory for subsequent processing or other systems.

[0010] In addition to fusion, the sensor data collected from each sensor device can also be analyzed separately. This makes it possible to leverage the advantages of each individual sensor device without introducing unnecessary complexity into the process.

[0011] The fusion of sensor data is also based, for example, on the different detection perspectives of the sensor devices. This leads to better data acquisition and more accurate categorization of objects, since each sensor device captures different characteristics of the environment.

[0012] The integrity of the fusion output can also be assessed to ensure the data is more reliable and accurate. Outputs validated by preferred sensors for specific acquisitions are rated higher because they offer greater accuracy. This ensures optimal use of sensor data and improves the quality of decisions based on the fused data.

[0013] Finally, in one possible embodiment of the invention, only predefined sensor data are fused in the neural network, depending on the task to be solved by the sensor data. The sensor data are selectively filtered according to the respective task, so that only relevant data are used for the fusion. This reduces the computational effort and ensures that the neural network focuses on the information important for the respective task in order to deliver precise results.

[0014] Given a DNN-based sensor data fusion approach, the proposed method adds missing information to the fused outputs: which sensors detected an object. This information is crucial for assessing the integrity of an output and planning a safe response. Adding single-modality tasks provides this integrity information with minimal additional overhead. Since most computational costs are typically incurred by the encoders, the proposed solution is also computationally efficient. Furthermore, because the same training data can be used to train the entire network, including the single-modality tasks, the approach is highly efficient in terms of training data requirements.

[0015] Further advantages, features, and details of the invention will become apparent from the following description of a preferred embodiment and from the drawing. The features and combinations of features mentioned above in the description, as well as those mentioned below in the figure description and / or shown in the figure alone, can be used not only in the combinations specified, but also in other combinations or individually, without departing from the scope of the invention.

[0016] This shows: Fig. 1 A diagram illustrating a data fusion of sensor data using a method according to the invention.

[0017] The Fig. Figure 1 shows a diagram illustrating a data fusion of sensor data using a data acquisition method according to the invention using a data acquisition system 10.

[0018] A first sensor device 12 is depicted as a camera, a second sensor device 14 as a lidar, and a third sensor device 16 as a radar, each of these sensor devices 12, 14, 16 having an encoder. The first sensor device 12 is associated with a first encoder 12a, also called the camera encoder. The second sensor device 14 is associated with a second encoder 14a, also called the lidar encoder. The third sensor device 16 is associated with a third encoder 16a, also called the radar encoder.

[0019] An encoder in the respective sensor device 12, 14, 16 is a component that converts the sensor data or raw data from the respective sensor in the sensor device 12, 14, 16 into a format suitable for subsequent processing steps, such as data fusion in a neural network. The encoder extracts relevant features from the sensor data to make it usable for specific tasks, such as object detection or classification.

[0020] Now each sensor device 12, 14, 16 shows respective tasks 18, 20, 22, whereby the tasks in this simplified representation provide for the detection of objects, for example in the driving trajectory of a motor vehicle.

[0021] The first sensor device 12 (camera) captures visual data and identifies objects, e.g., based on image features within a corresponding field of view.

[0022] The second sensor device 14 (Lidar) measures distances to objects by emitting laser pulses and creates, for example, a three-dimensional point cloud for object recognition.

[0023] The third sensor device 16 (radar) uses electromagnetic waves to determine the position and speed of objects, especially in poor visibility conditions or at great distances.

[0024] The plan is now to combine all sensor data in a Deep Neural Network (DNN) fusion, with this fusion step 24 combining the data integration from the various sensor devices into a unified output.

[0025] In merging step 24, the data from the various sensor devices 12, 14, 16 (camera, lidar, radar) are fused in a deep neural network (DNN). Here, the sensor data extracted by the encoders 12a, 14a, 16a, or rather the features of each sensor device 12, 14, 16, are combined to obtain a better result, for example, to generate a more comprehensive picture of the environment or a detected object on the trajectory. The DNN analyzes the sensor data and compares the information to eliminate redundant or inaccurate data while simultaneously leveraging the strengths of each sensor device 12, 14, 16. This enables more robust object detection, as the different perspectives and measurements of the sensor devices 12, 14, 16 are combined. The fused data is then made available for subsequent process steps, such as trajectory planning or autonomous driving decisions.

[0026] The various sensor systems 12, 14, and 16, as previously mentioned, each offer specific strengths that can complement each other in data fusion. For example, the camera provides high-resolution visual information that is particularly well-suited for object recognition based on color, shape, and texture. Lidar creates 3D point clouds of the objects to capture them even more accurately, enabling precise distance measurements and three-dimensional mapping of the environment. Radar, on the other hand, can reliably confirm the data already acquired, even in poor visibility conditions such as rain, fog, or darkness. By combining these specific strengths in data fusion, more comprehensive and robust information about a vehicle's surroundings can be provided, thus increasing the reliability and safety of autonomous driving.It is also possible to integrate a variety of other sensor devices, not mentioned here, into the merging step 24.

[0027] Accordingly, the respective tasks are intended to be validated or adopted in a confirmation step 22 and applied to further process steps or systems that are coupled with the data acquisition system 10. However, it is also possible to combine the data in a merging step 24. After merging, this data can be processed mathematically and subsequently used for further tasks 26, 28 or forwarded to other systems.

[0028] In other words, in a typical fusion approach based on a deep neural network (DNN), the sensor data from sensor devices 12 (camera), 14 (lidar), and 16 (radar) can first be processed by specific encoders 12a, 14a, and 16a, respectively. The camera encoder 12a extracts visual features, the lidar encoder 14a creates a 3D point cloud, and the radar encoder 16a records distances and velocities. Subsequently, a DNN model combines this information in a fusion step 24, where the sensor data fusion can, for example, be performed in a bird's-eye view. A decoder then extracts relevant information from the fused network properties for tasks such as object detection. Typically, several tasks are performed on the fused data.

[0029] It is therefore proposed to additionally perform tasks on individual modality encoders. This means that each sensor device 12, 14, 16 retains its specific outputs, which are limited to its respective fields of view. In this way, the tasks of the camera, lidar, and radar can be trained separately, using the same datasets. A simple mapping of the individual modality outputs to the fused outputs makes it possible to assess their integrity by verifying which sensor modalities contributed to the results. For example, an object confirmed only by the radar task will be rated with lower integrity than an object confirmed by all sensor modalities. This integrity assessment is crucial for safe trajectory planning in subsequent processing steps or for coupled systems.

[0030] Accordingly, the plan is to apply a data acquisition method, specifically a multi-sensor fusion method based on machine learning with neural networks, which combines data from different sensor types, such as cameras, lidar, and radar. This sensor data is first processed by special encoders to extract relevant features. The information is then fused in a neural network, eliminating redundant or inaccurate data and combining the strengths of the individual sensor types. Finally, the neural network extracts the relevant information for a specific task, such as object detection, and delivers it as a consolidated output.

[0031] In summary, the invention proposes a single modality monitoring concept for multi-sensor fusion in machine learning. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 11 2021 000 135 T5

[0004]

Images