A method to fuse at least first data from a first sensor of a motor vehicle with second data from a second sensor of the motor vehicle

Abstract

A method of fusing data from vehicle sensors, comprising: receiving first data from a first sensor 34; receiving second data from a second sensor 34; determining missing data in the first data and / or the second data; providing a mathematical model (10, Fig.1) that predicts the missing data 42; and fusing the first data, second data and predicted missing data. The mathematical model may be a neural network comprising at last one vision transformer that predicts the missing data depending on the first data and / or second data and the context of the data 36. A second aspect of the invention (Fig. 1) relates to a method for training the model 10, comprising: providing first data of a first sensor 14; masking data as the missing data 26; using a neural network for predicting the missing data 30; and adapting the neural network depending on the prediction 32.

Description

[0001] The present invention relates to the field of automobiles. More specifically, the present invention relates to a method to fuse at least first data from a first sensor of a motor vehicle with second data with second sensor of the motor vehicle by an electronic computing device. Furthermore, the present invention relates to a method for training a mathematical model for predicting missing data by an electronic computing device, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as to a corresponding electronic computing device. BACKGROUND INFORMATION

[0002] To have better sensor fusion, it is critical for the algorithm to learn to combine information from different sensors. It is known that data from the sensors may have missing parts, wherein the fusion of data with missing parts is problematical. SUMMARY OF THE INVENTION

[0003] It is an object of the present invention to provide a method to fuse at least first data from a first sensor of a motor vehicle with second data from a second sensor of the motor vehicle, to a corresponding method for training a mathematical model for predicting missing data, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding electronic computing device, by which fusing of sensor data may be performed in an efficient way.

[0004] This object is solved by methods, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding electronic computing device according to the independent claims. Advantageous embodiments are presented in the dependent claims.

[0005] A first aspect of the invention relates to a method to fuse at least first data from a first sensor of a motor vehicle with second data from a second sensor of the motor vehicle by an electronic computing device. The first data are received from the first sensor and the second data are received from the second sensor. Missing data in the first data and / or the second data are determined. A mathematical model for predicting missing data is provided. The missing data is predicted depending on the determined missing data by the mathematical model and fusion of the first data and the second data is performed by taking the predicted missing data into consideration.

[0006] Therefore, a method is performed, wherein the relation between different sensors is calculated so that the mathematical model can predict part of missing values of one sensor based on the remain values of the first sensor or the second sensor.

[0007] According to an embodiment, missing data from the first data is predicted depending on the received first data and / or the received second data and / or missing data from the second data is predicted depending on the received first data and / or the received second data.

[0008] In another embodiment, the mathematical model comprises a neural network.

[0009] According to another embodiment, the neural network comprises at least one vision transformer.

[0010] In another embodiment, the missing data is predicted depending on the context of the data.

[0011] A second aspect of the invention relates to a method for training a mathematical model for predicting missing data by an electronic computing device. At least the first data of the first sensor are provided. At least missing data are masked out from the first data. A neural network is used for predicting the missing data and adapting the neural network depending on the prediction is performed.

[0012] In particular, the methods are computer-implemented methods. Therefore, another aspect of the invention relates to a computer program product comprising program code means for performing a method according to the first aspect of the invention and / or a second aspect of the invention.

[0013] Another aspect of the invention relates to a non-transitory computer-readable storage medium comprising at least a computer program product according to the preceding aspect.

[0014] Furthermore, the present invention relates to an electronic computing device to fuse at least first data from a first sensor of a motor vehicle with second data from a second sensor of the motor vehicle and / or for training a mathematical model for predicting missing data, wherein the electronic computing device is configured for performing a method according to the first aspect of the invention and / or the second aspect of the invention. In particular, the methods are performed by the electronic computing device.

[0015] A computing unit / electronic computing device may in particular be understood as a data processing device, which comprises processing circuitry. The computing unit can therefore in particular process data to perform computing operations. This may also include operations to perform indexed accesses to a data structure, for example a look-up table, LUT.

[0016] In particular, the computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units.

[0017] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0018] A memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0019] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figures and / or shown in the figures alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0021] The drawings show in:

[0022] Fig. 1 a schematic block diagram according to an aspect of the invention; and

[0023] Fig. 2 another schematic block diagram according to an aspect of the invention.

[0024] In the figures the same elements or elements having the same function are indicated by the same reference signs. DETAILED DESCRIPTION

[0025] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0026] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0027] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0028] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0029] Fig. 1 shows a schematic block diagram according to an embodiment of a method for training a mathematical model 10 for predicting missing data by an electronic computing device 12. The electronic computing device 12 comprises a visible input 14, a context encoder 16, a context embedding 18, a coordination of the mask / queries 20, a predictor 22, a predicted target embedding 24, a masked input 26, a target encoder 28, a target embedding 30 and a loss function 32.

[0030] In particular, fig. 1 shows in order to have sensor fusion, it is critical for the algorithm to learn to combine information from different sensors. One of the indications that the algorithm understands the relation between different sensors is that it can predict part of the missing value of sensors based on the remaining value of sensor and other sensors. Therefore, a masking out of some sensor input during a training phase of the mathematical model 10 is provided on purpose. The sensor is then asked to predict what is missing based on the rest of the information. This can be used as a pre-task or a pretraining before training for the actual sensor fusion task, in particular such as object detection and segmentation.

[0031] In particular, Fig. 1 shows a training method to pre-train the mathematical model 10, which may be provided as a deep neural network for a sensor fusion. The components listed are in particular software components. The context encoder may convert visible / unmasked (unmasked sensor measurement) into feature embedding, which describe the context. The predictor 22 predicts based on the context feature embeddings and the queries, which describe where the information is masked, the target feature embeddings on the visible (masked) sensor measurement. The target encoder 28 converts masked sensor measurement into feature embeddings, which describes the target. The loss function 32 compares the actual target feature embeddings, in particular from the target encoder 28, and the predicted target feature embeddings, in particular from the predictor 22, and calculates a loss. This loss is back propagated into the neural network to update the weights.

[0032] Fig. 2 shows another embodiment of an electronic computing device 12 in a block diagram. In particular, Fig. 2 shows a multi sensor input 44, a context encoder 26, an input embedding 38, the predictor 22, a query entire input 40 and the predicted input embedding 42. Furthermore, a downstream task 44, for example object detection or segmentation is provided. Furthermore, an output 46 is shown.

[0033] In particular, the components can be implemented in different ways as long as the input, for example the multi sensor input 34, and the output 46 is defined. In practice, all the components may be implemented as neural networks, except for the loss function 32, which is a differentiable function that calculates distance between two features embeddings. Specifically, the context encoder 16, the target encoder 28 and the predictor 22 may be three different vision transformers. Different in the sense of number of layers, and / or parameters per layer. reference signs Mathematical model Electronic computing device Visible input Context encoder Context embedding Cordinate of the mask Predictor Predicted target embedding Masked input Target encoder Target embedding Loss function Multi sensor input Context encoder Input embedding Query ...input Predicted input embedding Downstream task Output

Claims

1. A method to fuse at least first data from a first sensor of a motor vehicle with second data from a second sensor of the motor vehicle by an electronic computing device (12), comprising the steps of:- receiving the first data from the first sensor;- receiving the second data from the second sensor;- determining missing data in the first data and / or the second data;- providing a mathematical model (10) for predicting missing data;- predicting the missing data depending on the determined missing data by the mathematical model (10); and- fusing of the first data and the second data by taken the predicted missing data into consideration.

2. The method according to claim 1, characterized in thatmissing data from the first data is predicted depending on the received first data and / or the received second data and / or missing data from the second data is predicted depending on the received first data and / or the received second data.

3. The method according to claim 1 or 2, characterized in thatthe mathematical model (10) comprises a neural network.

4. The method according to claim 3, characterized in thatthe neural network comprises at last one vision transformer.

5. The method according to any one of claims 1 to 4, characterized in thatthe missing data is predicted depending on the context of the data.

6. A method for training a mathematical model (10) for predicting missing data by an electronic computing device (12), comprising the steps of:- providing at least first data of a first sensor;- masking out data as the missing data;- using a neural network for predicting the missing data; and - adapting the neural network depending on the prediction.

7. A computer program product comprising program code means for performing a method according to any one of claims 1 to 5 and / or according to claim 6.

8. A non-transitory computer-readable storage medium comprising at least a computer program product according to claim 7.

9. An electronic computing device (12) to fuse at least first data from a first sensor of a motor vehicle with second data from a second sensor of the motor vehicle and / or for training a mathematical model (10) for predicting missing data, wherein the electronic computing device (12) is configured for performing a method according to any one of claims 1 to 5 and / or according to claim 6.10

Images