Detecting scenes in multi-modal domain

Abstract

Scenes are detected in a multi-modal domain. A method of training a neural network, the neural network predicting values of color information and other physical measurement variables associated with a location based on coordinates of the location in a scene and a description about a perspective of view at which the location is viewed, the method comprises the following steps: detecting camera images of a scene from a plurality of perspectives and values of other physical measurement variables as training examples; feeding coordinates of a location at which values of the color information and / or of at least one further physical measurement variable can be detected from each angle of view, together with information characterizing the angle of view, to the neural network; evaluating to what extent the values of the color information and / or of the at least one further physical measurement variable, which are subsequently provided by the neural network for the location, coincide with the values of the camera images actually detected from the respective perspective and / or of the at least one further measurement variable by means of a predefined cost function; the parameters characterizing the behavior of the neural network are optimized with the aim of improving the evaluation of the cost function when further processing the location coordinates and viewing angles.

Description

Technical Field

[0001] The invention relates to creating a representation of a scene as viewed, for example, from a vehicle or a robot. Background Art

[0002] At least partially automated driving of vehicles and / or robots on operating areas or in public transport requires continuous monitoring of the environment of the vehicles and / or robots. In addition to cameras, other sensors with other measuring modalities, such as radar or lidar, are also used for this purpose. The different measuring modalities complement each other in this case. In particular, this ensures that the environment monitoring is seamless even if one of the measuring modalities is inoperative.

[0003] If, for example, the low sun is incident on the lens of the camera, at least part of the image may be burned out because certain image pixels are saturated in intensity. The radar sensor may not be affected by this and thus still provide important information, for example: there is another vehicle or other object at a certain location in the vehicle environment. However, the image information in the burned-out area is still lost. Summary of the invention

[0004] The invention provides a method for training a neural network. The neural network predicts color information and the value of at least one other physical measurement variable based on the coordinates of a location in a scene and information about the viewing angle from which the location is viewed. The color information can, in particular, for example, indicate which light color the substance at the corresponding location emits when it is illuminated with white light. However, the color information can also, for example, indicate which light color a self-luminous object (e.g. a lamp) at the corresponding location emits.

[0005] In the context of the method, both camera images and values ​​of other physical measurement variables are detected from a plurality of viewing angles as training examples. For each of these viewing angles, the coordinates of a location at which color information and / or the value of at least one other physical measurement variable can be detected from the respective viewing angle are then fed to the neural network to be trained together with information characterizing the viewing angle.

[0006] These are not just locations that are in direct sight from the current viewing angle. Rather, for example, the radar beam can also penetrate objects or obstacles and provide information about locations that are not visible from the current vision.

[0007] The viewing angle can be characterized, for example, in particular by the angle from which the location is viewed and / or by the starting point from which the location is viewed. Thus, for example, in a complex scene, the view of a specific location from a starting point that is not far away may be free in a certain direction. From a starting point that is further away, however, the view of the location may be blocked (obstructed) by another object, such as a tree, another vehicle or a person.

[0008] Based on the input, the neural network provides color information and / or the value of at least one other physical measured variable. The extent to which the color information or the value agrees with the camera image actually detected from the corresponding viewing angle and / or the value of at least one other measured variable is evaluated using a predetermined cost function. The parameters characterizing the behavior of the neural network are optimized with the aim of improving the evaluation by the cost function during further processing of the coordinates of the location and the viewing angle.

[0009] It has been recognized that in this way a neural network can be made to generate a multimodal representation of a scene, which representation also provides color information and / or the value of at least one other measurement variable for an unseen combination of a queried location and a viewing angle, the at least one other measurement variable representing the corresponding location of the viewing angle. In particular, in this case different measurement modalities can be supplemented, so that gaps in the detection of a scene can be filled by one measurement modality based on information acquired with the other measurement modality.

[0010] The training is self-supervised. It does not rely on the measurement data being labeled in any way with the desired output of the neural network. Instead, the training uses only measurement data that can be detected in an automatic manner.

[0011] In the mentioned example where a low sun burns out a portion of a camera image, it can be estimated from radar measurements, for example, how to reconstruct the burned out portion of the camera image from its surroundings.

[0012] Even if all imaging modalities function without problems, they can also complement each other well. For example, it is possible to predict based on radar data how an object that is only partially visible in the camera image will continue in the area that is no longer visible.

[0013] Conversely, radar measurements with a relatively poor spatial resolution can also benefit from the high spatial resolution of the camera image. Thus, for example, the value of a measurement variable provided by the radar can be provided by the neural network with a higher spatial resolution than the radar measurement itself can provide. At the same time, that information that the radar measurement can provide particularly well, such as the material of the detected object or the speed relative to the object, is maintained.

[0014] The trained neural network thus functions in a manner similar to a stereo model, in which the scene is constructed from three-dimensional figures, buildings and other objects: points in the three-dimensional space are assigned color information and the value of at least one other physical measurement variable. This allows any view from any viewing angle to be generated. The neural network can be used in a similar manner to a "neural radiance field" which predicts only color information and optionally also light density for a location and viewing angle. However, as explained previously, this use is not limited to the boundaries between the measurement modalities.

[0015] Retrain for each scene. So here we are not targeting unseen scenes, but using the neural network's ability to generalize to unseen objects for unseen perspectives. In this regard, it is not a problem that a neural network trained specifically for a scene will strongly overfit to that scene.

[0016] An important practical application of this possibility is the behavior planning of vehicles and / or robots, which relies on continuous environmental monitoring. Systems for such behavior planning must be tested with views from a large number of perspectives, not all of which can be measured naturally. Therefore, in a particularly advantageous embodiment, at least one combination of location and perspective that has not been seen during training is fed to the trained neural network. The color information provided by the neural network and the value of at least one other physical measurement variable and / or the processing results consisting of these variables are then fed to at least one system for behavior planning of vehicles and / or robots.

[0017] The output of the system for behavior planning is compared with an expected output. Such an expected output can be derived, for example, from a known scene. The scene itself does not really change, but only a new view of the same scene is generated.

[0018] The result of the comparison is used to assess whether the system for behavior planning functions as intended. This functional test can be performed at many more viewing angles than if the scene had to be viewed naturally from every viewing angle.

[0019] Another important practical application is to simulate how changes in the configuration of cameras and / or other sensors affect the functionality of downstream systems.

[0020] Therefore, in a further particularly advantageous embodiment, at least one combination of location and viewing angle not seen during training is fed to the trained neural network, which combination represents a configuration of cameras and / or other sensors that is different from the configuration used for detecting the training examples. The color information subsequently provided by the neural network and the value of at least one other physical measurement variable and / or a processing result consisting of these variables are fed to at least one downstream system.

[0021] The behavior of the downstream system derived from this is compared with the expected behavior. Based on the result of this comparison, it is assessed whether the new configuration of the camera and / or other sensors in combination with the downstream system functions as intended.

[0022] Thus, for example, cameras and other sensors can be installed at different locations on each new vehicle type, since each new vehicle type has slightly different dimensions. If a vehicle type is now used to drive through a scene, the view of the same scene generated by the camera and other sensors on another vehicle can be simulated. Without having to record new data, it can then be checked whether, for example, the environmental monitoring and trajectory planning for the previous vehicle type can continue to be used on the new vehicle type as introduced. If this is not the case, the configuration of the cameras and / or other sensors and / or the downstream environmental monitoring and trajectory planning must be modified.

[0023] Therefore, in another particularly advantageous embodiment, the configuration of the camera and / or other sensors is optimized with respect to a predetermined optimization goal. The optimization goal can include, for example, improving the consistency of the behavior of downstream systems with the expected behavior. For example, the position of the camera and / or other sensors can be optimized so that depth information can also be detected by stereo vision and the type of objects in the vehicle's environment can thus be classified more accurately.

[0024] However, the optimization goal may also, for example, alternatively or in combination, include finding a configuration of the camera and / or other sensors that minimizes hardware expenditure and for which the behavior of the downstream system still corresponds to the expected behavior.

[0025] For example, in the case of a system for monitoring an area, the cost expenditure is thus decisively dependent on the number of cameras and / or other sensors required in total. Thus, the optimization result can be, for example, that a predetermined area can be covered with fewer cameras and / or other sensors if only fewer cameras and / or other sensors are arranged in a special manner in a suitable manner.

[0026] Vehicles, driver assistance systems, robots, systems for quality control, systems for monitoring areas, and / or systems for medical imaging are therefore particularly advantageously selected as downstream systems.

[0027] In another particularly advantageous embodiment, at least one measurement variable detected by a radar sensor, a lidar sensor and / or an ultrasonic sensor and / or a processing result of such a measurement variable is selected as the other physical measurement variable. These additional measurement modalities have a particularly good potential to complement the camera image and, in particular, can fill in the gaps that the imaging modality has when observing a specific location from a specific perspective.

[0028] The measured variables detected by means of radar sensors, lidar sensors and / or ultrasonic sensors may include, for example,

[0029] the measured location of where the radar reflection, lidar reflection or ultrasonic reflection came from, and / or

[0030] The covariance of radar, lidar or ultrasonic measurements, and / or

[0031] The speed of the radar sensor, lidar sensor or ultrasonic sensor relative to the location from which the radar reflection, lidar reflection or ultrasonic reflection comes. These are variables that are evaluated in particular when monitoring the environment of vehicles and / or robots.

[0032] Thus, at least one camera and / or at least one radar sensor, lidar sensor and / or ultrasonic sensor is advantageously carried by the vehicle or robot. However, not all sensors have to be mobile. Instead, fixed sensors, such as surveillance cameras installed at the location of the scene, can also make a complementary contribution.

[0033] The information characterizing the viewing angle may, for example, in particular comprise a direction from a reference point of the vehicle and / or robot to the observed location. The direction may, for example, be given in the form of one or more angles. Alternatively or in combination therewith, the information characterizing the viewing angle may also comprise the position of the vehicle and / or robot. As explained previously, the same location may be visible from the same viewing angle, for example from a closer position, but may be obscured from a more distant position.

[0034] In particular, for example, camera images and / or at least one other physical measurement variable can be used to determine the position of the vehicle and / or robot. For example, the position can be determined using a simultaneous localization and mapping method (SLAM). In this way, the position can be determined more accurately than would be possible using a satellite-assisted navigation system. This is particularly applicable, for example, in interior spaces with poor satellite reception, such as production halls.

[0035] In a particularly advantageous embodiment, the cost function contains the norm of the distance between a first vector of variables output by the neural network for at least one location and a second vector of measured variables corresponding thereto. In this way, evaluations of physically very different measuring modalities can be aggregated, so that, for example, a poor performance with respect to one measuring modality can be compensated by a better performance with respect to the other measuring modality.

[0036] In this case, in particular, different measurement modalities can be weighted relative to one another, for example in a cost function. For example, it can be balanced that different measurement modalities are represented in different numbers of measurement variables. For example, color information can be present in the form of intensity values ​​for the three primary colors red, green, blue, while a radar measurement can provide three spatial coordinates for the location of the radar reflection, a scalar covariance and additionally a scalar velocity, i.e. a total of five variables. For example, the various variables provided by a radar measurement can also have different values ​​in the context of the respective application.

[0037] In particular, the method can be completely or partially computer-implemented. Therefore, the invention also relates to a computer program with machine-readable instructions, which, when executed on one or more computers and / or computing instances, cause the one or more computers and / or computing instances to perform the described method. In this sense, control devices for vehicles and embedded systems for technical devices that are likewise capable of executing machine-readable instructions can also be regarded as computers. The computing instance can be, for example, a virtual machine, a container or a serverless execution environment, which can be provided in particular in the cloud.

[0038] The invention also relates to a machine-readable data carrier with a computer program and / or a download product. A download product is a digital product that can be transmitted via a data network, i.e. can be downloaded by a user of the data network, and can be sold for immediate downloading, for example in an online store.

[0039] Furthermore, one or more computers and / or computing instances can be equipped with the computer program, the machine-readable data carrier or the download product. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Further measures for improving the invention are shown in more detail below with reference to the figures together with the description of preferred exemplary embodiments of the invention.

[0041] Figure 1 An embodiment of a method 100 for training a neural network 1 is shown;

[0042] Figure 2 A diagram showing how the method 100 works at a three-dimensional scene is shown. DETAILED DESCRIPTION

[0043] Figure 1 is a schematic flow chart of an exemplary embodiment of a method 100 for training a neural network 1. The neural network 1 is designed to predict color information 4 and the value of at least one other physical measurement variable 5 related to a position 2 based on the coordinates of the position in a scene and information about the viewing angle 3 from which the position is viewed.

[0044] In step 110 , both camera images 4* of the scene and values ​​of other physical measured variables 5* are detected from a large number of viewing angles 3* as training examples.

[0045] According to block 111 , in particular, for example, at least one measured variable detected by means of a radar sensor, a lidar sensor and / or an ultrasonic sensor and / or a processing result of such a measured variable can be selected as further physical measured variable 5 .

[0046] According to block 111a, the measured variables 5 detected by means of a radar sensor, a lidar sensor and / or an ultrasonic sensor may include, for example,

[0047] the measured location of where the radar reflection, lidar reflection or ultrasonic reflection came from, and / or

[0048] The covariance of radar, lidar or ultrasonic measurements, and / or

[0049] The speed of the radar sensor, lidar sensor, or ultrasonic sensor relative to the location from which the radar reflection, lidar reflection, or ultrasonic reflection came.

[0050] According to box 112 , at least one camera and / or at least one radar sensor, lidar sensor, and / or ultrasonic sensor may be carried by the vehicle 50 or the robot 60 .

[0051] In step 120 , the coordinates of a location 2* at which color information 4 and / or a value of at least one other physical measured variable 5 can be detected from each viewing angle 3* are fed to the neural network 1 together with information characterizing the viewing angle 3*.

[0052] According to block 121, if according to block 112 at least one sensor or at least one camera is carried by the vehicle 50 and / or the robot 60, the information characterizing the viewing angle 3* may in particular include, for example

[0053] The direction from a reference point of the vehicle 50 and / or robot 60 to the location being observed, and / or

[0054] The location of the vehicle 50 and / or robot 60.

[0055] In step 130, a predetermined cost function 6 is used to evaluate the extent to which the color information 4 and / or the values ​​of at least one other physical measurement variable 5 subsequently provided by the neural network 1 for these locations are consistent with the camera image 4* and / or the values ​​of at least one other measurement variable 5* actually detected from the corresponding viewing angle 3*.

[0056] According to block 131 , the cost function 6 may contain, for example, in particular the norm of the distance between a first vector of variables output by the neural network for at least one location and a second vector of measured variables corresponding thereto.

[0057] According to block 131 a , the six different measurement modalities can be weighted relative to one another in the cost function.

[0058] In step 140, parameters 1a characterizing the behavior of the neural network 1 are optimized with the goal of improving the evaluation 6a by the cost function 6 during further processing of the coordinates of the location 2* and the viewing angle 3*. The optimized state of the parameters is denoted by reference numeral 1a* and represents the trained state 1* of the neural network 1.

[0059] The trained neural network 1* can be used in various ways to produce multimodal views of the scene used for training from any viewing angle 3, i.e., color information 4 combined with values ​​of other measured variables 5, similar to producing any views from a physical three-dimensional stereo model. Figure 1 Two exemplary practical applications are shown in FIG.

[0060] In step 150 , at least one combination of location 2 and viewing angle 3 which has not been seen during training can be fed to the trained neural network 1 *.

[0061] In step 160 , the color information 4 subsequently provided by the neural network 1 * and the value of at least one other physical measurement variable 5 and / or a processing result consisting of these variables can be fed to at least one system for behavior planning of the vehicle 50 and / or the robot 60 .

[0062] In step 170, the output 7 of the system for the behavior planning may be compared with the expected output 7*.

[0063] In step 180 , it can be assessed from the result 170 a of this comparison whether the system for behavior planning functions as intended (OK) or not as intended (not OK=NOK).

[0064] In step 190, at least one combination of location 2 and viewing angle 3 not seen during training can be fed to the trained neural network 1*, which combination represents a new configuration 8 of the camera and / or other sensors, i.e. a configuration different from the configuration used for detecting the training examples.

[0065] In step 200 , the color information 4 and the value of at least one further physical measured variable 5 subsequently provided by the neural network 1 * and / or a processing result consisting of these variables can be fed to at least one downstream system.

[0066] In this case, in particular, for example, a vehicle 50 , a driver assistance system 51 , a robot 60 , a system 70 for quality control, a system 80 for monitoring an area and / or a system 90 for medical imaging can be selected as downstream systems.

[0067] In step 210 , the behavior 9 of the downstream system resulting from the control using the values ​​of the color information 4 and other physical measured variables 5 or using the processing results formed therefrom can be compared with the expected behavior 9 *.

[0068] In step 220 , it can be assessed from the result 210 a of this comparison whether the combination of the new configuration 8 of the camera and / or other sensors with the downstream system functions as intended (OK) or not as intended (not OK=NOK).

[0069] In step 230, the new configuration 8 of the camera and / or other sensors can be optimized with regard to a predefined optimization goal. The optimization goal can include, for example, improving the consistency of the behavior 9 of the downstream system with the expected behavior 9*.

[0070] Figure 2 An example is used to illustrate how the neural network 1 enables always generating new multimodal views of the same scene.

[0071] To train the neural network 1, camera images 4* of the scene are recorded from a plurality of viewing angles 3* and values ​​of other physical measured variables 5* are used as training examples. As explained above, self-supervised training can thus be aimed in particular at reproducing information known from metrological measurements in the neural network 1.

[0072] After the neural network 1 has been trained, for any combination of position 2 and viewing angle 3, the neural network 1 can be asked which values ​​of the color information 4 (2, 3) and other measured variables 5 (2, 3) indicate the location 2 when viewed from the viewing angle 3. Figure 2 Two exemplary combinations 2 , 3 and 2 ′, 3 ′ are plotted for locations 2 , 2 ′ and viewing angles 3 , 3 ′.

Claims

1. A method (100) for training a neural network (1) for predicting color information (4) and the value of at least one other physical measurement variable (5) associated with a location based on the coordinates of the location (2) in a scene and information about the viewing angle (3) from which the location is viewed, the method comprising the steps of: Detecting (110) not only camera images (4*) of the scene from multiple viewpoints (3*) but also values ​​of other physical measurement variables (5*) as training examples; feeding (120) the coordinates of a location (2*) at which the color information (4) and / or the value of at least one other physical measurement variable (5) can be detected from each viewing angle (3*) to the neural network (1) together with information characterizing the viewing angle (3*); evaluating (130) using a predetermined cost function (6) the extent to which the color information (4) and / or the value of at least one other physical measurement variable (5) subsequently provided by the neural network (1) for the location agree with the value of the camera image (4*) and / or the value of at least one other measurement variable (5*) actually detected from the corresponding viewing angle (3*); Parameters (1a) characterizing the behavior of the neural network (1) are optimized (140) with a view to improving the evaluation (6a) by the cost function (6) during further processing of the coordinates of the location (2*) and the viewing angle (3*).

2. A method (100) according to claim 1, wherein at least one measurement variable detected by means of a radar sensor, a lidar sensor and / or an ultrasonic sensor and / or a processing result of such a measurement variable is selected (111) as another physical measurement variable (5).

3. The method (100) according to claim 2, wherein the measurement variable (5) detected by means of a radar sensor, a lidar sensor and / or an ultrasonic sensor comprises (111a) the measured location of where the radar reflection, lidar reflection or ultrasonic reflection came from, and / or The covariance of radar, lidar or ultrasonic measurements, and / or The speed of the radar sensor, lidar sensor, or ultrasonic sensor relative to the location from which the radar reflection, lidar reflection, or ultrasonic reflection came.

4. The method (100) according to any one of claims 1 to 3, wherein at least one camera and / or at least one radar sensor, lidar sensor and / or ultrasonic sensor is carried (112) by the vehicle (50) or the robot (60).

5. The method (100) according to claim 4, wherein the information characterizing the viewing angle (3*) comprises (121) The direction from a reference point of the vehicle (50) and / or robot (60) to the observed location, and / or • The location of the vehicle (50) and / or robot (60).

6. The method (100) according to claim 5, wherein the camera image (4*) and / or the at least one other physical measurement variable (5*) are used to determine the position of the vehicle (50) and / or the robot (60).

7. A method (100) according to any one of claims 1 to 6, wherein the cost function (6) comprises (131) the norm of the distance between a first vector of parameters output by the neural network for at least one location and a second vector of measured parameters corresponding thereto.

8. The method (100) according to claim 7, wherein different measurement modalities are weighted (131a) relative to each other in the cost function (6).

9. The method (100) according to any one of claims 1 to 8, wherein feeding (150) at least one combination of location (2) and viewpoint (3) not seen during training to the trained neural network (1*), feeding (160) the color information (4) subsequently provided by the neural network (1*) and the value of at least one other physical measurement variable (5) and / or a processing result consisting of these variables to at least one system for behavior planning of a vehicle (50) and / or a robot (60), comparing (170) the output (7) of the system for behavior planning with the expected output (7*), and From the result ( 170 a ) of this comparison, it is evaluated ( 180 ) whether the system for behavior planning is functioning as intended.

10. The method (100) according to any one of claims 1 to 9, wherein feeding (190) to the trained neural network (1*) at least one combination of location (2) and viewpoint (3) not seen during training, said combination representing a configuration (8) of cameras and / or other sensors different from the configuration used to detect the training examples; feeding (200) the color information (4) subsequently provided by the neural network (1*) and the value of at least one other physical measurement variable (5) and / or a processing result consisting of these variables to at least one downstream system; comparing (210) the resulting behavior (9) of the downstream system with the expected behavior (9*); and From the result of this comparison ( 210 a ), it is evaluated ( 220 ) whether the new configuration ( 8 ) of cameras and / or other sensors in combination with the downstream system functions as intended.

11. The method (100) according to claim 10, wherein a vehicle (50), a driver assistance system (51), a robot (60), a system for quality control (70), a system for monitoring an area (80) and / or a system for medical imaging (90) are selected (201) as downstream systems.

12. The method (100) according to any one of claims 10 to 11, wherein a new configuration (8) of cameras and / or other sensors is optimized (230) with regard to a predefined optimization goal.

13. A computer program comprising machine-readable instructions which, when executed on one or more computers and / or computing instances, cause the one or more computers and / or computing instances to perform the method (100) according to any one of claims 1 to 12.

14. A machine-readable data carrier having a computer program according to claim 13.

15. One or more computers and / or computing instances equipped with a computer program according to claim 13 and / or equipped with a machine-readable data carrier according to claim 14.

Images