A method for fusing at least two output signals of at least two sensor devices of a motor vehicle, a corresponding computer program product, a corresponding non-transitory computer-storage medium, as well as corresponding electronic computing device

Abstract

A method for fusing at least two output signals (22, 24) of at least two sensor devices (14, 16) of a motor vehicle (10) by an electronic computing device (12) of the motor vehicle, comprising the ste

Description

FIELD OF THE INVENTION

[0001] The present invention relates to the field of auto mobiles. More specifically, the present invention relates to a method for fusing at least two output signals of at least two sensor devices of a motor vehicle by an electronic computing device of the motor vehicle, according to the pending claim 1. Furthermore, the present invention relates to a corresponding computer program product, a corresponding non-transitory computerstorage medium, as well as two corresponding electronic computing device. BACKGROUND INFORMATION

[0002] From the state of the art it is known, that the sensor fusion algorithm usually require input sensor data to be synchronized to perform the best part. However, synchronization, in particular in time, is not always easy. This may be caused that the motor vehicles may not have well-designed- synchronization mechanism inside the sensor devices. Furthermore, the time tag / stamp may be inaccurate or missing.

[0003] SUMMARY OF THE INVENTION

[0004] It is an object of the present invention to provide a method, a corresponding computer program product, a corresponding non-transitory computer-storage medium, as well as a corresponding electronic computing device, by which the sensor fusion of at least two sensor devices can be provided in an approved manner.

[0005] This object is solved by a method, a corresponding computer program product, a corresponding non-transitory computer-storage medium, as well as a corresponding electronic computing device, according to the independent claims. Advantageous embodiments a presented in the dependent claims.

[0006] One aspect of the invention relates to a method for fusing at least two output signals of at least two sensor devices of a motor vehicle by an electronic computing device of the motor vehicle. At least a first output signal of at least a first sensor device of the at least two sensor devices is received by the electronic computing device. At least a second output signal of at least a second sensor device of the at least two sensor devices is received by the electronic computing device. A deep neural network for determining a time offset between the at least two received output signals is provided by the electronic computing device. The time offset is determined by the deep neural network and the at least two output signals are fused depending on the determined time offset by the electronic computing device.

[0007] Therefore, sensor fusion of in particular incorrectly synchronized or unsynchronized, i.e. multi-sensor-input signals, is provided. Therefore, the fusing of unsynchronized output signals of sensor devices is provided in an improved manner.

[0008] Therefore, the deep neural network is aware of the synchronization in accuracy and overcomes the synchronization issue.

[0009] Therefore, the deep neural network is trained with training data, in particular with training data from different sensor devices, which are unsynchronized. Therefore, the deep neural network can be provided and trained in order to fusion the unsynchronized data.

[0010] It is obvious for a person skilled in the art, that the motor vehicle may comprise more than two sensor devices, which can be fused by the deep neural network.

[0011] A sensor device can be for example a radar sensor device, a lidar sensor device, an ultrasonic sensor device or a camera. In particular, the different sensor devices may capture the surroundings of a motor vehicle with different time resolution. According to the invention this issue is overcome by fusing the output signals of the different sensor devices with a deep neutral network.

[0012] In particular, it is obvious, that a plurality of same sensor devices as well as a plurality of different sensor devices can be fused by the electronic computing device.

[0013] Therefore, the fused data can be used to provide a more efficient capturing of the surroundings, and therefore, for example can be used in an at least partially automated motor vehicle or fully automated motor vehicle. Furthermore, the improved captured surroundings can be used for presenting the surroundings on a display device for a user of the motor vehicle.

[0014] According to an embodiment the fusion is performed by the deep neural network. In particular, the deep neural network is aware of the synchronization and accuracy. The deep neural network is explicitly trained to predict the time-offset. The deep neural network is configured to fuse on the unsynchronized sensor input data directly. This is possible, because the deep neural network now already knows the time-offset information.

[0015] According to another embodiment the fusing is performed depending on at least one dynamical parameter of the motor vehicle. In particular, therefore, the fusion can be done ..manually", by compensating the time-offset using the dynamic parameter of the motor vehicle and do the fusion on a computationally synchronized sensor input.

[0016] In another embodiment the fusion is performed depending on a velocity as the dynamical parameter of the motor vehicle. Therefore, the time-offset is compensated by using the velocity information and the fusion is provided on a computational synchronized sensor input.

[0017] In another embodiment the at least two sensor devices are different in their way of detecting surroundings of the motor vehicle. For example, the first sensor device can be a camera. The second sensor device can be a radar sensor. Alternatively the sensor device can be configured as a radar sensor device or an ultrasonic sensor device. Therefore, different sensor devices can be fused in an approved manner.

[0018] In another embodiment the at least two output signals are desynchronized. For example, the desynchronized output signals can be generated by different sensor devices. The electronic computing device is now configured for fusing the desynchronized output signals in order to provide a better capturing of the surroundings.

[0019] In another embodiment at least one of the two output signals comprises an inaccurate time stamp. For example, an image of a camera may comprise a time stamp, when the image is captured. This time stamp may be inaccurate or, for example, a lidar sensor may comprise a higher time resolution and therefore the time stamp of the camera may be too inaccurate in order to be synchronized with the information from the lidar sensor. Therefore, in order to overcome this issue, the fusing of the output signals of the sensor devices is now provided by the deep neural network.

[0020] In particular, the present invention is computer-implemented method. Therefore, another aspect of the invention relates to a computer program product comprising program code means for performing a method according to the proceeding aspect.

[0021] A still further aspect of the invention relates to a non-transitory computer-readable storage medium comprising at least the computer program product according to the proceeding aspect.

[0022] Furthermore, the present invention relates to an electronic computing device of a motor vehicle for fusing at least two output signals of at least two sensor devices of the motor vehicle, wherein the electronic computing device is configured for performing a method according to the proceeding aspect. In particular, the method is performed by the electronic computing device.

[0023] A still further aspect of the invention relates to a motor vehicle comprising at least the electronic computing device. The motor vehicle can be for example configured as an at least in part automatically operated motor vehicle or a fully automatically operated motor vehicle.

[0024] Advantageous and embodiments of the method are to be regarded as advantages and embodiments of the computer program product, the non-transitory computer-storage medium, the electronic computing device, as well as the motor vehicle. Therefore, the electronic computing device as well as the motor vehicle comprises means for performing the method.

[0025] An artificial neural network / deep neural network can be understood as a software code or a compilation of several software code components, wherein the software code may comprise several software modules for different functions, for example one or more encoder modules and one or more decoder modules.

[0026] An artificial neural network can be understood as a non-linear model or algorithm that maps an input to an output, wherein the input is given by an input feature vector or an input sequence and the output may be an output category for a classification task or a predicted sequence.

[0027] For example, the artificial neural network may be provided in a computer-readable way, for example, stored on a storage medium of the vehicle, in particular of the at least one computing unit.

[0028] The neural network comprises several modules including the encoder module and the at least one decoder module. These modules may be understood as software modules or respective parts of the neural network. A software module may be understood as software code functionally connected and combined to a unit. A software module may comprise or implement several processing steps and / or data structures.

[0029] The modules may, in particular, represent neural networks or sub-networks themselves. If not stated otherwise, a module of the neural network may be understood as a trainable and, in particular, trained module of the neural network. For example, the neural network and thus all of its trainable modules may be trained in an end-to-end fashion before the method is carried out. However, in other implementations, different modules may be trained or pre-trained individually. In other words, the method according to the invention corresponds to a deployment phase of the neural network.

[0030] A computing unit / electronic computing device may in particular be understood as a data processing device, which comprises processing circuitry. The computing unit can therefore in particular process data to perform computing operations. This may also include operations to perform indexed accesses to a data structure, for example a look-up table, LUT.

[0031] In particular, the computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units.

[0032] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0033] A memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0034] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figures and / or shown in the figures alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0036] The drawings show in:

[0037] Fig. 1 a schematic side view according to an embodiment of a motor vehicle comprising an embodiment of an electronic computing device;

[0038] Fig. 2 a schematic block diagram according to an embodiment of the electronic computing device.

[0039] In the figures the same elements or elements having the same function are indicated by the same reference signs.

[0040] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0041] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0042] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0043] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0044] Fig. 1 shows a schematic side view according to an embodiment of a motor vehicle 10 comprising an embodiment of an electronic computing device 12. Furthermore, the motor vehicle 10 comprises a fist sensor device 14, a second sensor device 16, as well as a third sensor device 18. It is obvious for a person skilled in the art, that the motor vehicle 10 may comprise more than three sensor devices 14, 16, 18.

[0045] According to a shown embodiment the first sensor device 14 may be, for example, a camera, and a second sensor device 16 may be, for example, a lidar device. Furthermore, the third sensor device 18 may be provided as a back camera. Additionally or alternatively the sensor devices 14, 16, 18 may be configured as radar sensor devices or ultrasonic sensor devices.

[0046] Furthermore, the motor vehicle 10 may comprise an assistant system 20, for example, for an at least in part automated operation mode for the motor vehicle 10.

[0047] The fist sensor device 14 may generate a first output signal 22 and the second sensor device 16 may generate a second output signal 24. Furthermore, the third sensor device 18 may generate a third output signal 26. The output signals 22, 24, 26 are transmitted to the electronic computing device 12. The electronic computing device 12 may generate fused data 28 and may transmit the fused data 28 to the assistant system 20. The electronic computing device 12 shown in Fig. 1 is now more detailed described in Fig. 2.

[0048] Fig. 2 shows a schematic block diagram according to an embodiment of the electronic computing device 12. According to the shown embodiment the electronic computing device 12 is configured for fusing the at least two output signal 22, 24 of the at least two sensor devices 14, 16. Therefore, the electronic computing device 12 receives at least the first output signal 22 and at least the second output signal 24. A deep neural network 30 is provided for determining a time-offset between the at least two received output signals 22, 24 by the electronic computing device 12. The time-offset is determined by the deep neural network 30 and fusing of the at least two output signals is provided depending on the determined time-offset by the electronic computing device 12. In particular, the deep neural network 30 is trained to predict the time-offset by offering such data-label pair during the training time. The data from the at leat two sensor devices 14, 16 is the different sensor signal, and the label (output of the DNN) is the known timeoffset, either known from human labeling or known because it’s directly generated from simulation. The deep neural network 30 may be deployed ideally in the motor vehicle 10, for example as an edge device, to reduce latency. However, the training of such a deep neural network 30 can be done anywhere, for example in a cloud.

[0049] Fig. 2 further shows a so called motion compensation 32. Therefore, information 34, 36 for different time stamps t1, tn are provided. The motion compensation 32 is one example of how to use the predicted time-offset. The predicted time-offset is used to perform motion compensation 32 to the original signal so that their time-offset is compensated, and then they can be fused as there were time-offset.

[0050] Therefore, Fig. 2 shows, that the fusing is performed by either the deep neural network 30 or alternatively, the fusoning is performed depending on at least one dynamical parameter of the motor vehicle 10, in particular depending on a velocity of the motor vehicle 10 as the dynamical parameter of the motor vehicle 10.

[0051] Therefore, the deep neural network 30 is designed to be aware of synchronization inaccuracy of the sensor devices 14, 16. The deep neural network 30 is trained to explicitly predict the time-offset of the sensor devices 14, 16. Then, either the deep neural network 30 fusions on the unsynchronized sensor input directly. This is now possible because the deep neural network 30 is now already aware of the time-offset information. Alternatively, in particular shown with the motion compensation 32, a manually compensation of the time-offset is provided using the velocity information and the fusion is provided on a computational synchronized sensor input.

[0052] Furthermore, the at least two sensor devices 14, 16, as already mentioned, may be different and their way of detecting surroundings 38 of the motor vehicle 10.

[0053] Furthermore, the at least two output signals 22, 24 are desynchronized. Furthermore, at least one of the two output signals 22, 24 may comprise an inaccurate time stamp. signs motor vehicle electronic computing device first sensor device second sensor device third sensor device assistant system first output signal second output signal third output signal fusioned data deep neural network motion compensation first time data second time data surroundings time stamps

Claims

1. A method for fusing at least two output signals (22, 24) of at least two sensor devices (14, 16) of a motor vehicle (10) by an electronic computing device (12) of the motor vehicle (10), comprising the steps of:- receiving at least a first output signal (22) of at least a first sensor device (14) of the at least two sensor devices (14, 16) by the electronic computing device (12);- receiving at least a second output signal (24) of at least a second sensor device (16) of the at least two sensor devices (14, 16) by the electronic computing device (12);- providing a deep neural network (30) for determining a time offset between the at least two received output signals (22, 24) by the electronic computing device (12);- determining the time offset by the deep neural network (30);- fusing the at least two output signals (22, 24) depending on the determined time offset by the electronic computing device (12).

2. The method according to claim 1, characterized in thatthe fusing is performed by the deep neural network (30).

3. The method according to claim 1, characterized in thatthe fusing is performed depending on at least one dynamical parameter of the motor vehicle (10).

4. The method according to claim 3, characterized in thatthe fusing is performed depending on a velocity as the dynamical parameter of the motor vehicle (10).

5. The method according to any one of claims 1 to 4, characterized in thatthe at least two sensor devices (14, 16) are different in their way of detecting surroundings (38) of the motor vehicle (10).

6. The method according to any one of claims 1 to 5, characterized in thatthe at least two output signals (22, 24) are desynchronized.

7. The method according to any one of claims 1 to 6, characterized in thatat least one of the two output signals (22, 24) comprises an inaccurate time stamp.

8. A computer program product comprising program code means for performing a method according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium comprising at least the computer program product according to claim 8.

10. An electronic computing device (12) of a motor vehicle (10) for fusing at least two output signals (22, 24) of at least two sensor devices (14, 16) of the motor vehicle (10), wherein the electronic computing device (12) is configured for performing a method according to any one of claims 1 to 7.

Images