Method and device for fusing sensor signals by means of a neural network

Abstract

A computer-implemented method for fusing a plurality of sensor signals (x) by means of a neural network (60), wherein the sensor signals (x) comprise at least a first value and a second value, the first value characterizing an expected value of a physical quantity and the second value characterizing a dispersion of the physical quantity, further wherein the neural network (60) determines an output (y) based on the plurality of sensor signals (x), the output characterizing a fusion of the plurality of sensor signals (x) and the output (y) depending on a first intermediate output (z1, z2, z n ) of the neural network (60), wherein the first intermediate output (z1, z2, z n ) is determined by at least one first neuron and comprises a determined first value and a determined second value, the determined first value characterizing an expected value of the fusion of the plurality of sensor values and the determined second value characterizing a dispersion of the fusion, wherein the determined second value of the first intermediate output (z1, z2, z n ) is set to zero if a predefinable condition is fulfilled.

Description

Technical Field

[0001] The present invention relates to a method for fusing sensor signals, a hardware implementation of the method, a method for training, a training device, a computer program, and a storage medium. Background Technology

[0002] Unpublished DE 10 2020 201 133.8 discloses a neural network that includes random neurons for fusing sensor signals.

[0003] Advantages of the present invention

[0004] Signals recorded by sensors are typically subject to uncertainties, which may arise from environmental or operating conditions of the sensor or manufacturing tolerances during its production. To determine reliable sensor signals, multiple sensors of the same type are often used, and the correspondingly determined sensor signals are fused.

[0005] In particular, neural networks with random neurons have proven to be highly suitable for fusing sensor signals. These neural networks enable the fusion of sensor signals affected by uncertainties.

[0006] Calculating the output of a neural network with random neurons can place high energy demands on the device used to determine that output. Therefore, especially in mobile devices or in robotics, such as at least partially automated vehicles, it is desirable to keep the energy requirements of neural networks with random neurons as low as possible. Correspondingly, high-performance sensor signal fusion is required. In the context of this invention, the performance of sensor signal fusion can be understood as the ability to achieve a desired result based on multiple sensor signals. Here, this performance can be understood as a continuous measure indicating the degree of deviation of the output from the desired result. Summary of the Invention

[0007] The advantage of the method according to the present invention is that the number of mathematical operations required for a neural network with random neurons can be greatly reduced. This reduces the energy and storage space requirements of the device used to compute the output of the neural network. This means that the performance of the neural network can be improved while maintaining the same energy or storage space requirements.

[0008] In a first aspect, the present invention relates to a computer-implemented method for fusing multiple sensor signals by means of a neural network, wherein the sensor signals include at least a first value and a second value, the first value representing an expected value of a physical quantity, the second value representing a discreteness of the physical quantity, and wherein the neural network determines an output based on the multiple sensor signals, the output representing a fusion of the multiple sensor signals, and the output depends on a first intermediate output of the neural network, wherein the first intermediate output is determined by at least one first neuron and includes a determined first value and a determined second value, the determined first value representing an expected value of the fusion of the multiple sensor values, the determined second value representing a discreteness of the fusion, wherein the determined second value of the first intermediate output is set to zero if a pre-given condition is satisfied.

[0009] In the present invention, the method of combining signals from multiple sensors into a single sensor signal can be understood as the fusion of sensor signals, wherein these sensors are configured to measure the same physical quantity, and the combined sensor signal characterizes an improved measurement of that physical quantity.

[0010] For the purposes of this invention, it is conceivable that the first value is the expected value of the physical quantity. The second value may, for example, be the variance of the measured physical quantity. For numerical stability and for the sake of faster computation of neural networks, the second value may also advantageously and preferably be the reciprocal of the variance. In this case, the reciprocal of the variance is also known as the accuracy value. Thus, for the purposes of this invention, the sensor signal can be understood as a measurement of the physical quantity affected by uncertainty.

[0011] To measure this physical quantity, it is conceivable that a sensor measures a preliminary value and then determines a sensor signal, including a first value and a second value, based on this preliminary value. For example, an ultrasonic sensor can measure the transit time of an ultrasonic signal and other characteristics as preliminary values. Then, based on these preliminary values, the ultrasonic sensor can determine the first and second values ​​of the desired physical quantity, such as the layer thickness of a workpiece or the humidity level of a road surface. Another example is a camera sensor, which first measures an image as a preliminary value. Then, based on the image and an image classifier, the camera sensor can, for example, determine the first and second values ​​of the location of an object in the image, where this location represents the physical quantity.

[0012] Advantageously, the first neuron can be a random neuron. These neurons have proven particularly well-suited for fusing sensor signals with uncertainty.

[0013] The random neuron is configured to receive at least a first value and a second value of the input or intermediate result, and to determine the first and second values ​​based on these values. Preferably, the first values ​​are expected values, and the second values ​​are precision values. The random neuron can first be determined according to the formula...

[0014]

[0015] To determine the weighting of the received precision values, where The received precision value is located at position. The value, and This is the weight of that value. Furthermore, the weighting of the received expected values ​​can be calculated using the formula...

[0016]

[0017] To carry out, among which The location of the received expected value The value, and It is the weight of that value.

[0018] Based on the weighted sum of the received precision values ​​and the weighted sum of the received expected values, the random neuron can be calculated according to the formula...

[0019]

[0020] To determine the precision value and according to the formula

[0021]

[0022] The expected value is then determined. The determined expected value and the determined precision value can be passed on as at least part of the intermediate result to other random neurons of the neural network or used as at least part of the output. Therefore, the intermediate result or output can consist of at least one expected value and at least one precision value.

[0023] Therefore, the method executed by random neurons can be understood as the fusion of multiple sensor signals, where the weights of the random neurons determine how these sensor signals are fused. Multiple random neurons can be arranged in one layer of the neural network. In this case, the intermediate output of the neural network can be understood as several different possible results of the fusion of these sensor signals. This intermediate output can then be passed to other layers of the neural network so that different fusion results can be combined with each other. In this way, different fusion strategies can be mapped. The layers of the neural network may also include nonlinear activation functions, through which the multiple sensor signals are nonlinearly weighted to determine the output. In this case, the nonlinear weighting is determined by the weights of the corresponding layers. To train these weights, machine learning methods, especially stochastic gradient descent, can be used. This allows the method to learn the fusion strategy best suited to the data. This improves the performance of the fusion method.

[0024] It is conceivable that if the determined second value of the first intermediate output is lower than a predefined threshold, then the determined second value is set to zero. Alternatively, it is also conceivable that if the first precision value is less than or equal to the first threshold, then the first precision value is set to zero.

[0025] The advantage of setting a small second value to zero in a neural network is that the multiple mathematical operations required to determine the output can involve multiplication with zero and are thus computed significantly faster. Typically, neural network operations involve matrix multiplication and / or matrix addition. Therefore, this method results in matrix multiplication and / or matrix addition with sparse matrices. This significantly reduces the mathematical operations required by the neural network, especially by utilizing hardware specifically designed for sparse matrix operations. Consequently, the energy required to compute the output is reduced. Furthermore, the memory consumption required to compute the output is reduced. In turn, with the same energy or memory requirements, the performance of the neural network can be improved.

[0026] Based on these two advantages, a third advantage arises. Because of the reduced energy and computing power required, this method is particularly applicable to battery-powered devices, such as mobile phones or robots, to reduce energy consumption while maintaining the same performance. This results in the neural network only being usable in certain devices; otherwise, the energy consumption or required storage space would be too high.

[0027] In another embodiment of the method, it is conceivable that the first intermediate output is determined by a plurality of neurons and includes a plurality of determined first values ​​and a plurality of determined second values, wherein the determined second values ​​are set to zero if they belong to the minimum of a predefined number of these determined second values.

[0028] For this purpose, these second values ​​can first be sorted according to size. Then, the minimum of these second values ​​can be set to zero, or more precisely, to the same number of minimum values ​​as specified by the predefined number.

[0029] The advantage of this implementation is that the number of second values ​​set to zero can be determined within one layer of the neural network. This means that the reduction in mathematical operations can be precisely defined. This is particularly advantageous when using computational units to compute a first intermediate output for which a predefined number of operands are set to zero is advantageous, or where the first intermediate output is contingent upon a predefined number of elements being set to zero.

[0030] In another embodiment of the invention, it is conceivable that the step of determining the first intermediate output is performed by means of a computing unit for Sparse Matrix Operations, wherein the computing unit is configured to perform these operations by means of hardware acceleration.

[0031] The advantage of this implementation is that the efficiency of determining the first intermediate output is further improved.

[0032] The present invention also relates to a computer-implemented method for training a neural network, wherein the neural network is trained based on a loss function.

[0033] To train a neural network, machine learning methods can be used, especially those that determine the weights of the network through gradient descent, such as stochastic gradient descent (SGD) or Adam. The weights contained in the layers of a neural network can be understood as the weights of the network.

[0034] Preferably, for training, training data of sensor signals are used, wherein each training data includes multiple sensor signals that should be fused. Next, for training, the output of the neural network can be determined for at least one training data. The determined output, along with the desired output for that training data, can then be fed to a loss function that determines the difference between the determined output and the desired output. The weights can then be adapted based on this difference to improve the performance of the neural network.

[0035] In another implementation, it is conceivable that the loss function includes the norm of at least a portion of the weights of the random neurons.

[0036] For example, it can be envisioned that the loss function includes the L1 norm of at least a portion of the weights of the neural network and / or includes the L2 norm of at least a portion of the weights of the neural network.

[0037] The advantage of using at least a portion of the norms of these weights is that this training method provides an incentive such that, after training, many weights of the neural network are close to or equal to zero. Therefore, these weights are adjusted during training so that as many mathematical operations as possible during the operation of the neural network involve multiplications and / or additions with zero. This further reduces the energy and memory consumption of the required mathematical operations and thus leads to performance improvements. Attached Figure Description

[0038] Embodiments of the present invention will then be described in more detail with reference to the accompanying drawings. In the drawings:

[0039] Figure 1 The schematic diagram illustrates the construction of a neural network;

[0040] Figure 2 The schematic diagram illustrates the construction of a control system for manipulating actuators;

[0041] Figure 3 An embodiment for controlling at least a partially autonomous robot is illustrated schematically;

[0042] Figure 4 The diagram illustrates a training system used to train a neural network. Detailed Implementation

[0043] Figure 1 A neural network (60) for fusing multiple (x) input signals is shown. For this purpose, the neural network (60) includes layers (L1, L2, L... n These layers each contain random neurons. The corresponding random neurons determine the expected value and the precision value, respectively. Except for the last layer of the neural network (L... n In addition to the expected value and precision value determined by the random neurons of layer (L1, L2), the values ​​are combined in the respective layer outputs (a1, a2) of the corresponding layer (L1, L2).

[0044] The neural network (60) receives multiple (x) input signals in the first layer (L1) and determines the first layer output (a1) by means of random neurons in the first layer (L1). The first layer output is fed to a first comparison unit (V1). The first comparison unit (V1) determines multiple precision values ​​of the first layer output (a1) that are less than a first threshold (T1). Then, the determined precision values ​​are set to zero and the first layer output (a1) thus modified is provided to the second layer (L2) as a first intermediate output (z1). In an alternative embodiment (not shown), it is conceivable that the first comparison unit (V1) also determines for each precision value whether the precision value is lower than a threshold specifically defined for that precision value.

[0045] The second layer receives the first intermediate output (z1) and determines the second layer output (a2) using random neurons in the second layer (L2). The second layer output is fed to the second comparison unit (V2). The second comparison unit (V2) determines multiple precision values ​​of the second layer output (a2) that are less than a second threshold (T2). The determined precision values ​​are then set to zero, and the second layer output (a2) thus modified is provided as the second intermediate output (z2) to the third layer (not shown). In an alternative embodiment (not shown), it is conceivable that the second comparison unit (V2) also determines for each precision value whether the precision value is lower than a threshold specifically defined for that precision value.

[0046] Except for the last floor (L) n Apart from the intermediate outputs of the other layers, the other intermediate outputs are determined in a similar manner to those in the second layer. That is, the corresponding layer receives the corresponding previously determined intermediate output and generates a layer output for that intermediate output, and then the comparison unit compares the layer output with a threshold. The precision value of the layer output that is less than the threshold is set to zero, and the layer output thus modified is provided as an intermediate output to the next layer.

[0047] The last layer (L) n ) Receives the last intermediate output (z) of the layer above the last layer. n Next, based on the final intermediate output (z... n ), the last layer (L) n Determine the expected value (y) m ) and precision value (y e The expected value and the accuracy value together characterize the fusion of the multiple (x) sensor signals. To determine the expected value (y... m ) and precision value (y e ), the last layer (L) n Use random neurons.

[0048] In other embodiments (not shown), it is conceivable that these sensor signals represent vector physical quantities, such as expected and precise values ​​of position in three-dimensional space. In these embodiments, the last layer (L n This includes as many random neurons as the physical quantity has dimensions. Each random neuron can then determine the dimensions of the expected value and the precision value.

[0049] Figure 2An actuator (10) is shown that interacts with a control system (40) in its surrounding environment (20). Preferably, the surrounding environment (20) is detected by a plurality of first sensors (30) at regular intervals. Sensor signals (S) from the plurality of first sensors (30) are transmitted to the control system (40). Thus, the control system (40) receives a sequence of sensor signals (S). The control system (40) determines control signals (A) based on these signals, which are transmitted to the actuator (10). For this purpose, the sensor signals (S) include a desired value and a precision value.

[0050] The control system (40) receives a sequence of sensor signals (S) from the first sensor (30) in an optional receiving unit (50), which converts the sequence of sensor signals (S) into a sequence of input signals (x) (alternatively, the sensor signals (S) of the first sensor (30) can be used directly). The input signal (x) may be, for example, a fragment of these sensor signals (S) or further processing of these sensor signals. In other words, the input signal (x) is determined based on the sensor signals (S). The sequence of input signals (x) is fed to the neural network (60).

[0051] The neural network (60) preferably uses parameters ( These parameters are parameterized, and these parameters are stored in and provided by the parameter memory (P). This includes, in particular, the weights of neural networks.

[0052] The neural network (60) determines the fused output signal (y) based on the input signal (x), which includes the expected value (y). m ) and precision value (y e The output signal (y) is sent to the control unit (80), which determines control signals (A) accordingly. These control signals are then sent to the actuator (10) to control the actuator (10) accordingly. In other embodiments, the control unit (80) may receive other signals from other components of the control system to control the actuator (10). In particular, the control unit (80) may receive classification signals (c) from an image classifier (70), wherein the classification signals (c) preferably characterize at least one camera signal (S) obtained by the image classifier (70) based on at least one second sensor (30a), such as a camera sensor or video sensor, a LiDAR sensor, or a radar sensor. a The classification signal (c) is used to classify the surrounding environment (20). For example, the classification signal (c) can characterize the classification of objects in the surrounding environment (20) of the control system (40).

[0053] The actuator (10) receives the control signal (A), is correspondingly controlled, and performs a corresponding action. In this case, the actuator (10) may include (not necessarily structurally integrated) control logic that determines a second control signal to be used to control the actuator (10) based on the control signal (A).

[0054] In other embodiments, the control system (40) includes a sensor (30). In still other embodiments, alternatively or additionally, the control system (40) also includes an actuator (10).

[0055] In other preferred embodiments, the control system (40) includes at least one processor (45) and at least one machine-readable storage medium (46) on which commands are stored, which, when executed on the processor (45), cause the control system (40) to perform the method according to the invention.

[0056] In an alternative implementation, a display unit (10a) is provided at or in addition to the actuator (10), which is controlled by a control signal (A). Here, alternatively or additionally, the control signal (A) can be used to control the display unit (10a) and, for example, display the result of the fusion of sensor signals (30).

[0057] Figure 3 The control system (40) is shown as being able to be used to control at least a partially autonomous robot, which is here at least a partially autonomous motor vehicle (100).

[0058] The first sensor (30) may be, for example, an ultrasonic sensor preferably arranged in the motor vehicle (100) to measure the humidity value of the road on which the motor vehicle (100) moves. In this case, the ultrasonic sensors (30) respectively determine the desired value of the humidity value and the accuracy value of the humidity value.

[0059] The neural network (60) is configured to fuse sensor signals (S) from different ultrasonic sensors (30) and determine the desired value (y) for the humidity value. m ) and the accuracy value of that humidity value (y e Expected value (y) m ) and precision value (y e The output signal (y) is generated by the neural network (60). For this purpose, in this embodiment, the neural network (60) is in the last layer (L n The component includes random neurons that determine the expected value (y). m ) and precision value (y e ).

[0060] The image classifier (70) is configured to classify the surrounding environment (20) based on video recordings (S) taken by means of a camera sensor (30a). a The system detects objects that a motor vehicle (100) is not allowed to collide with, especially other road users, such as other motor vehicles, pedestrians, or cyclists. Objects classified by the image classifier (70) are transmitted to the control unit (80) via a classification signal (c).

[0061] The actuator (10) preferably arranged in the motor vehicle (100) may be, for example, the braking device, drive device, or steering device of the motor vehicle (100). Then, the control signal (A) can be determined to cause the actuator or these actuators (10) to be manipulated such that, especially when the object identified by the image classifier (70) is a certain type of object, such as a pedestrian, the motor vehicle (100) avoids collision with such objects. However, the control signal (10) of the actuator (10) is also determined by the expected value (y) of the humidity value determined by the neural network (60). m ) and the accuracy value of the humidity value (y e ) is determined by, for example, the precision value (y e If the value exceeds or equals a predefined third threshold, then it can be assumed that the expected value (y) m ) accurately characterizes the actual humidity of the road. In this case, if the expected value (y) m If the value is below the predefined fourth threshold, the motor vehicle (100) can continue its driving without restriction. If the expected value (y) is below the predefined fourth threshold, the motor vehicle (100) can continue its driving without restriction. m If the accuracy value (y) is greater than or equal to the fourth threshold, then, for example, the maximum speed at which the motor vehicle (100) is allowed to move can be reduced. e If the threshold is lower than the third threshold, then this restriction can also be selected.

[0062] For example, in the case of a motor vehicle (100) with a non-automatic steering system, it is also possible to use a control signal (A) to control the display unit (10a) so that if the accuracy value (y) e If the value is below the third threshold or if the expected value (y) is lower than the third threshold, or if the expected value (y) m If the value exceeds or equals the fourth threshold, the display unit will output an optical or audible alarm signal.

[0063] Alternatively, the first sensor (30) can also be a positioning sensor, such as a GPS sensor, a GLONASS sensor, a Galileo sensor, or a BeiDou sensor. In this case, the neural network (60) can determine four expected values ​​and four precision values ​​for the location and output these expected values ​​and precision values ​​in the output signal (y). In this embodiment, the neural network (60) is used in the last layer (L n The neural network (60) contains four random neurons, which determine the expected value and the precision value, respectively. In this embodiment, the number of expected values ​​and precision values ​​is merely illustrative. The desired number of expected values ​​and precision values ​​can be determined by adding a layer (L) to the last layer of the neural network (60). n The number of random neurons in the ) is specified.

[0064] Next, the actuator (10) can be manipulated, for example, to deactivate certain automated driving functions depending on the location of the vehicle (100). For example, it is conceivable that the vehicle (100) is only allowed to move automatically in a certain country, and that the function is deactivated once it crosses the border with another country.

[0065] Alternatively, the at least partially autonomous robot could be other mobile robots (not shown), such as those that move by flying, floating, diving, or walking. The mobile robot could also be, for example, a at least partially autonomous lawnmower or a at least partially autonomous cleaning robot. In these cases, the control signal (A) could also be determined to cause the drive and / or steering mechanisms of the mobile robot to be manipulated to prevent the at least partially autonomous robot from colliding with objects identified by the image classifier (70), for example.

[0066] Figure 4 An embodiment of a training system (140) is shown, which is configured to train a neural network (60). For training, the training data unit (150) accesses a computer-implemented database (St2), wherein the database (St2) includes at least one training dataset (T), wherein the training dataset (T) includes sensor records (x... i ) and the desired output signal (y i ) tuples, where these sensors record (x i ) is a record of multiple sensor signals, which should be fused by a neural network (60), and the desired output signal (y) i The value should be determined by a neural network.

[0067] The training data unit (150) determines the sensor records (x) of the training dataset (T). i) and the desired output signal (y i At least one tuple of ) and these sensors record (x i The data is transmitted to the neural network (60). The neural network (60) is based on sensor recordings (x...). i ) to determine the output signal ( ).

[0068] Desired output signal (y) i ) and the determined output signal ( ) is transmitted to the change unit (180).

[0069] Next, based on the determined output signal ( ) and the desired output signal (y i The new model parameters for the neural network are determined by changing the unit (180). ), especially the new weights. To this end, the unit (180) is modified using the loss function to determine the output signal ( ) and the desired output signal (y i The comparison is performed. The loss function is determined for the given output signal ( ) and the desired output signal (y i The degree of deviation is measured. Preferably, L1 loss (L1-Loss) or L2 loss (L2-Loss) can be chosen as the loss function. Preferably, the result of another loss function is added to the L1 loss or L2 loss, which is based on the model parameters ( The other loss function can be, for example, the Frobenius norm of the weights of the neural network (60).

[0070] The modified unit (180) determines new model parameters based on the determined metrics. In this embodiment, this is achieved using gradient descent, preferably stochastic gradient descent, or Adam.

[0071] The determined new model parameters ( The parameters are stored in the model parameter memory (St1).

[0072] In other embodiments, the described training is iteratively repeated a predefined number of iterations, or iteratively repeated until the metric falls below a predefined threshold. In at least one of these iterations, new model parameters (determined in previous iterations) are... ) are used as model parameters of neural networks ( ).

[0073] Furthermore, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146) containing instructions that, when executed by the processor (145), cause the training system (140) to perform the training method according to any of these aspects of the invention.

[0074] The term "computer" includes any device used to run computational rules that can be given in advance. These computational rules can exist in the form of software, hardware, or a combination of both.

Claims

1. A computer-implemented method for fusing multiple sensor signals (x) using a neural network (60), wherein, The sensor signal is an ultrasonic signal from an ultrasonic sensor, an image signal from a camera sensor, or a position signal from a positioning sensor. The sensor signal (x) includes at least a first value and a second value. The first value represents the expected value of a physical quantity, and the second value represents the discreteness of the physical quantity. Furthermore, the neural network (60) determines an output (y) based on the plurality of sensor signals (x). The output represents the fusion of the plurality of sensor signals (x), and the output (y) depends on the first intermediate output (z1, z2, z3) of the neural network (60). n ), wherein the first intermediate output (z1, z2, z n The first value is determined by at least one first neuron and includes a determined first value and a determined second value. The determined first value represents the expected value of the fusion of multiple sensor values, and the determined second value represents the discreteness of the fusion, wherein the discreteness is represented using a precision value, and wherein the determined second value is used as a weight for the weighting of the determined first value. Where the first intermediate output (z1, z2, z) is... n If the determined second value is lower than the predefined threshold (T1, T2), then the determined second value is set to zero, or Where the first intermediate output (z1, z2, z) n The first value is determined by multiple neurons and includes multiple determined first values ​​and multiple determined second values, wherein the determined second value is set to zero if it belongs to the minimum of a predefined number of determined second values.

2. The method of claim 1, wherein the step of determining the first intermediate output is performed by means of a computing unit for sparse matrix operations, wherein the computing unit is configured to perform the operations by means of hardware acceleration.

3. A computer for fusing multiple sensor signals (x) by means of a neural network (60), wherein the computer is configured to implement the method according to claim 1, wherein the computer has a computing unit for sparse matrix operations, wherein the step of determining the first intermediate output is performed by means of the computing unit, wherein the computing unit is configured to perform the operations by means of hardware acceleration.

4. A computer program product comprising a computer program configured to perform the method according to claim 1 or 2.

5. A machine-readable storage medium (46, 146) having a computer program stored thereon, the computer program being configured to perform the method according to claim 1 or 2.

Images