Recognizing Scene Features with Higher Safety Integrity by Matching Sensor Data with Predictions

Abstract

1. A method (100) for evaluating location-resolved actual sensor data (2) recorded by at least one sensor (1), comprising: a step (110) in which a position (1a) and an orientation (1b) of the sensor (1) at the time of recording the sensor data (2) are determined; a step (120) in which a location-resolved prediction (4) is retrieved (120) from a location-resolved map (3) based on the position (1a) and orientation (1b) of the sensor (1); a step (130) in which how well the actual sensor data (2) matches the prediction (4); and a step (140) in which, at least for those positions where the actual sensor data (2) matches the prediction (4), the scene observed by the sensor (1) has a characteristic (5) stored in the map (3) associated with the prediction (4).

Description

[Technical field]

[0001] The present invention relates to the recognition of characteristics of a scene observed by at least one sensor, for example for the at least partially autonomous control of a vehicle or a robot. [Background technology]

[0002] Driving assistance systems and at least partially automated navigation systems for vehicles or robots capture the surroundings of the vehicle or robot by means of one or more sensors and, on the basis of this, determine a plan for the subsequent behavior of the vehicle or robot. Neural networks are often used to determine such plans or a description of the surroundings as a pre-product for the plans. DE 10 2018 008 685 A1 discloses a method for training a neural network for determining a path prediction for a vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] DE 102018008685 A1 Summary of the Invention [Means for solving the problem]

[0004] The invention provides a method for evaluating location-resolved real sensor data recorded by at least one sensor. The sensor may be, for example, a mobile sensor, in particular carried by a person or mounted on a robot or any land, water or air vehicle. However, the sensor may also be, for example, a stationary sensor observing, for example, a busy intersection. Location-resolved real sensor data is any sensor data that is assigned to a position determined in the context of its capture. For example, radar data and lidar data may be present as a point cloud of measurements in three-dimensional space, each of which is assigned to a point from which the scanning radiation used is reflected. An image assigns intensity values ​​to the pixels of the sensor used for its capture, which are usually arranged in a regular grid. Each pixel, on the other hand, also corresponds to a position in space from which the respective incident light comes, depending on the viewpoint at which the image was recorded and the optical system used.

[0005] The position and orientation of the sensor at the time of recording the sensor data is determined. This process is also called "registration" or "localization" of the sensor in the physical world. Any system can be used for this purpose, alone or in combination. For example, a navigation system based on radio signals emitted from satellites or even from ground stations can be used. Alternatively or in combination, for example, an inertial navigation system can be used. If the sensor is a stationary sensor, the registration or localization of the sensor is particularly simple, since the position and orientation of the sensor are known in advance.

[0006] From the geographically resolved map, a geographically resolved prediction for the actual sensor data is retrieved based on the sensor's location and orientation. The prediction is stored in the geographically resolved map in association with at least one characteristic of the observed scene. If the scene observed by the at least one sensor actually has the characteristic stored in the map, then the actual sensor data is predicted to be consistent with the prediction retrieved from the geographically resolved map.

[0007] This means that in reverse reasoning, if the actual sensor data matches the expectation, the property stored in the map also exists - and testing for the existence of this property is exactly the goal of the method.

[0008] Thus, the extent to which the actual sensor data matches the expectation is examined, and for at least those locations where the actual sensor data matches the expectation, it is determined that the scene observed by at least one sensor has the characteristics stored in the map associated with the expectation.

[0009] The predictions stored in the map are a sort of "fingerprint" of the scene observed by the sensor. This "fingerprint" may include any property that can be evaluated from the sensor data, such as geometry, texturing, multispectral response, etc. Basic physical properties, such as magnetic resonance, are also considered.

[0010] The sensor data may be, inter alia, sensor data recorded by sensors of a vehicle or a robot, for example. A vehicle may be any land, water or air vehicle. An important application of the method, in the context of vehicle and robot control, is the examination of the question which locations can be freely traversed by a vehicle or a robot. For this purpose, the properties stored in the map may include assertions of how freely a location associated with the prediction can be traversed by a vehicle or a robot. The method is also applicable in the same way with mobile sensors carried by people, for example to signal the traversable area to the visually impaired.

[0011] Let us clarify this with a simple example where camera images are used as sensor data. In this case, the map contains camera images, in which areas are respectively labeled as being freely traversable by the vehicle or not. Thus, if the areas determined during the vehicle's journey are re-recognized in their correct positions based on the camera images recorded by the vehicle at each time, it is guaranteed that the assertions stored in the map about these areas (how freely traversable these areas are by the vehicle or not) correspond to the current state of these areas. For example, if an obstacle is present at a certain location in the camera images recorded during the journey and this obstacle differs in at least one characteristic, for example the visual appearance or the geometry of the obstacle, from the same characteristic of the image stored in the map, this object can be recognized reliably.

[0012] In this way, the open classification task, in the sense of whether an area is freely passable (or there is any other characteristic therein that is important for the trip planning), can thus be reduced to a comparison with known information stored in the map. It has been found that this allows the recognition of which areas are freely passable to be performed with significantly better safety integrity. If a spatial area generates, at a certain position, the same or at least similar sensor data as stored in the map as a prediction, it necessarily follows that this area has the same occupancy of objects compared to the state at the time of the creation of the map (i.e., for example, no objects and therefore passable) and that no further objects are present between the sensor and this area. Any unexpected object in the area or in the visible path between the sensor and this area will destroy the correspondence between the actual sensor data and the prediction, so that this area is no longer identified as freely passable.

[0013] This has the advantage over open classification tasks solved, for example, by neural networks: non-freely traversable ranges are recognized as not being freely traversable even if the reason for non-traversability is highly unusual and therefore not present in the training data used to train the neural network. Events that are too unlikely to be present in the training data are also called the "long tail".

[0014] For example, sometimes on the highway, due to improper fastening of luggage, very unusual objects, such as furniture, large electrical appliances, skis or bicycles, go missing. The appearance of such objects on the roadway on which the vehicle is traveling is a highly dangerous situation, which is why it happens fortunately very rarely. However, this also means that when collecting real data for training neural networks in the context of test drives, such cases do not occur with a high probability. Consciously waiting for such situations in public transport is practically impossible.

[0015] Something similar applies to the dreaded "blow-ups" that occur suddenly during heat waves, when concrete roadways on highways turn into round bulges or cracks, but such conditions also cannot be trained for because by the time the "blow-up" appears on the camera images, an accident is almost impossible to prevent.

[0016] Moreover, the recognition of free-passage areas through re-identification in the map is not vulnerable to deliberate manipulation. Many neural network-based image classifiers can be misclassified by the introduction of malicious interference patterns. That is, for example, a stop traffic sign can be tampered with to be classified as a "70 km speed limit" sign by attaching a seemingly invisible sticker. It has already been successfully attempted to completely cut off the ability of a connected image classifier for pedestrian recognition by attaching a film with an inconspicuous semi-transparent dot pattern over a camera lens. Pedestrians were classified as free-passage areas.

[0017] Within the framework of the method presented here, such tampering attempts are either completely ignored or, in the worst case, areas that would otherwise be free to pass through are not recognized as such, i.e. any unexpected problems are avoided instead of being passed through ("fail-safe").

[0018] Furthermore, the prediction can already determine an upper limit for areas that can usually be recognized as freely passable. The recognition of an area in the prediction can only lead to a determination that this area is freely passable if this area was marked as freely passable in relation to the prediction. Areas that were not marked as freely passable here, such as concrete guardrails or trees at the side of the road, can never be recognized as freely passable.

[0019] A significantly better safety integrity in determining the characteristics of the scene observed by at least one sensor makes the control of the vehicle or robot based on the thus determined characteristics more likely to correspond to the respective situation. It is therefore advantageous to determine a control signal for the vehicle or robot taking into account the determination of at which location the scene observed by the sensor has the characteristics stored in the map associated with the expectation. The vehicle or robot is controlled by this control signal, whereby the driving dynamics of the vehicle or robot are influenced accordingly.

[0020] The comparison of the actual sensor data with the prediction is not limited to a 1:1 re-recognition, but instead any form of tolerance or even abstraction to specific features can be defined for this re-recognition, i.e. even two directly successively recorded camera images of one and the same scene are generally not completely identical.

[0021] In a further particularly advantageous embodiment, the actual sensor data and the predictions are transferred to a common spatial reference frame and / or a common working space. The actual sensor data are compared with the predictions in this reference frame or working space. In this way, sensor data captured with different modalities and the predictions can also be compared with each other. For example, the predictions can be based on the image of the scene observed by at least one sensor. Position-resolved 3D geometry and / or Texturing and / or Reflectance amplitude and / or Multispectral response and / or Magnetic Resonance Such a geometry may include, for example, a geometry that can be determined on the basis of image capture. Such a geometry may be amenable to labeling in terms of free-traverse areas or other characteristics of the landscape. In this case, for example, radar or lidar data may be examined to see how well it matches this geometry.

[0022] For the acquisition of sensor data, radar and lidar sensors as well as, inter alia, stereoscopically configured cameras or multi-camera systems are considered. Such camera assemblies also provide depth information that can be examined against an expected geometry. Moving monocular cameras are also conceivable for generating depth information. Furthermore, for example, multispectral cameras, Time of Flight (ToF) sensors, cameras with event-based reactive pixels, ultrasonic sensors or magnetic sensors can also be used.

[0023] For example, testing whether the sensor data recorded by the stereoscopic camera assembly is consistent with an expected three-dimensional geometry may involve testing a so-called Stereo-Hypothese, which is based on the fact that the geometry of the scene ties together the images coming from both cameras of the stereoscopic camera assembly, i.e., if one of the images has an expected geometry, this at least approximately defines the other image.

[0024] Thus, an image from a first camera of the camera assembly may be transformed into a prediction for an image from a second camera of the camera assembly based on the geometry of the prediction. This prediction may then be examined to see how well it matches the image that actually results from the second camera of the camera assembly. The transformation may include, among other things, warping the image from the first camera to match the viewpoint of the second camera based on the geometry.

[0025] In order to compare the image coming from the second camera with the expectation, features can be extracted, for example, from the image coming from the second camera on the one hand and from the expectation for this image on the other hand. These features can then be compared with each other. This abstraction to features can smooth out non-essential differences, for example differences regarding color or brightness, for the purpose of comparison.

[0026] In particular, for example, a binary determination can be made for every feature to be compared as to whether a feature from the image matches a corresponding feature from the prediction for this image.The degree of match between the image and the prediction can then be determined from the number of features that match each other, i.e., the Hamming distance of the examined feature combinations can be determined, which is greater the more inconsistent the image features on the one hand and the prediction on the other hand are with each other.

[0027] In a further advantageous embodiment, in addition to this, a predetermined test image that does not show the scene observed by the sensor is examined for how well it matches with the expectation for the image coming from the second camera of the camera assembly. This degree of match is then considered as the noise level for the match established between the image coming from the second camera of the camera assembly and the expectation for this image. In this way, a signal-to-noise ratio for the match between the image coming from the second camera and the expectation for this image can be established. This signal-to-noise ratio is more convincing than a match alone. For example, the convincingness of an image may be reduced by a large part of it being saturated by overexposure or underexposure.

[0028] The method can also be generalized to the effect that actual sensor data is recorded by a number of sensor modalities and then these results are compiled. Thus, in a further advantageous embodiment, the actual sensor data recorded by a number of different sensors is examined separately for which locations the actual sensor data corresponds to the predictions respectively retrieved from the map. Then, the actual sensor data is determined to collectively correspond to the predictions only for those locations where the actual sensor data of all sensors collectively correspond to the predictions respectively. That is, for example, an area is determined to be freely passable by a vehicle or robot only if the area is recognized as freely passable based on sensor data provided by a number of sensors of different sensor modalities (e.g. lidar and stereoscopic camera) independently of each other.

[0029] First, it can be examined how well, for example, the lidar data of the moment corresponds to the geometry of the scene stored in the map. In parallel to this, it can be examined how well, for example, the images coming from the stereoscopic camera assembly correspond to this geometry. For this purpose, for example, the geometry can be transformed into the reference coordinate system of the first camera of the camera assembly. Then, as described above, the images coming from the first camera of the camera assembly can be transformed based on the geometry into a prediction for the images coming from the second camera of the camera assembly, and this prediction can be compared with the images actually coming from the second camera, thereby examining the stereo hypothesis. Then, only for positions where both the lidar data and the images coming from the stereoscopic camera assembly correspond to the geometry stored in the map as a prediction, it can be determined that these positions have the searched properties (for example the free traversability of these positions) according to the map.

[0030] In a further advantageous embodiment, the actual sensor data, which is examined for a match with the expectation, is checked against the actual sensor data recorded by the further sensor. The match with the expectation is then determined or maintained only for those positions for which this check is positive. That is, instead of two comparisons between one sensor modality and the expectation, only one comparison is made. In addition, both sensor modalities are compared with each other. That is, for example, a certain area can be found free to pass on the one hand based on a match between the lidar data and the geometry stored as the expectation, and on the other hand, this area can be declared free to pass only if the lidar data is consistent with the images coming from the stereoscopic camera assembly.

[0031] In a further advantageous embodiment, the determined position and / or the determined orientation is optimized with the goal of maximizing the agreement between the actual sensor data and the prediction. As explained above, for the comparison of the actual sensor data with the prediction, it is important that the prediction for the correct position and the correct orientation of the sensor is called up from the map. Only in this case can the prediction be correctly recognized based on the actual sensor data at the time. However, all methods for determining the position and orientation have limited accuracy. Thus, for example, if the agreement between the sensor data and the prediction can be significantly improved by additionally displacing the determined position of the sensor and / or additionally tilting or rotating the determined orientation of this sensor, this indicates that the previously determined position or the previously determined orientation of the sensor was not necessarily correct. As an alternative or in combination, any other technique for transferring the actual sensor data and the map to a common reference coordinate system can be used.

[0032] The method may, inter alia, be fully or partly computer-implemented. The invention therefore also relates to a computer program having machine-readable instructions which, when executed on one or more computers, cause said one or more computers to carry out the aforementioned method. In this sense, embedded systems for control equipment and technical equipment for vehicles, which are also capable of executing machine-readable instructions, may also be considered as computers.

[0033] The invention also relates to a machine-readable data storage medium containing a computer program and / or a download product, which is a digital product that can be transmitted over a data network, i.e. downloaded by a user of the data network, and which can for example be offered for immediate download in an online shop.

[0034] Additionally, the computer may be loaded with a computer program, a machine-readable data storage medium, or a downloadable product. Further measures for improving the invention are detailed below together with the description of preferred exemplary embodiments of the invention based on the figures. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 illustrates an exemplary embodiment of a method 100 for evaluating location-resolved actual sensor data 2. [Diagram 2] FIG. 1 illustrates an exemplary application of the method 100 in a road scene. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] FIG. 1 is a schematic flow diagram of one exemplary embodiment of a method 100 for evaluating location-resolved actual sensor data 2 with respect to where a scene observed by a sensor 1 has a characteristic 5 of interest.

[0037] In step 105, sensor data 2 recorded by a sensor 1 of a vehicle 50 or a robot 60 may be selected. In step 106, sensor data 2 recorded by a radar sensor, a lidar sensor, and / or a stereoscopic camera assembly may be selected.

[0038] In step 110, the position 1a and orientation 1b of the sensor 1 at the time the sensor data 2 was recorded is determined. In step 120, a geo-resolved prediction 4 is retrieved from the geo-resolved map 3 based on the position 1a and orientation 1b of the sensor 1. The map 3 also stores a geo-resolved property 5 of interest. The property 5 is associated with the prediction 4, specifically to the effect that the actual sensor data 2 and the prediction 4 match, given that the scene observed by the sensor 1 from a certain position has the property 5.

[0039] In step 130, how well the actual sensor data 2 matches the expectations 4 is examined. In step 140, it is determined that the scene observed by the sensor 1 has characteristics 5 stored in the map 3 associated with the prediction 4, at least for locations where the actual sensor data 2 is consistent with the prediction 4. This is illustrated with respect to one example in FIG.

[0040] In step 150, a control signal 150a for the vehicle 50 or robot 60 is determined taking into account a determination of where the scene observed by the sensor 1 has characteristics 5 stored in the map 3 associated with the prediction 4.

[0041] In step 160, the vehicle 50 or robot 60 is controlled by this control signal 150a, which affects the driving dynamics of the vehicle 50 or robot 60 according to the control signal 150a.

[0042] Based on block 111, the determined position 1a and / or the determined orientation 1b may be optimized with the goal of maximizing the agreement between the actual sensor data 2 and the predictions 4. According to block 121 , the properties 5 stored in the map 3 may include, among other things, assertions about how freely the location associated with the prediction 4 may be traversed by a vehicle 50 or a robot 60 , for example.

[0043] Based on block 122 , prediction 4 may include, among other things, a position-resolved three-dimensional geometry of the scene observed by sensor 1 , for example. The actual sensor data 2 and the prediction 4 may be transferred to a common spatial reference frame and / or common working space, according to block 131. The actual sensor data 2 may then be compared to the prediction 4 within this reference frame or working space, according to block 132.

[0044] To determine whether the sensor data 2 recorded by the stereoscopic camera assembly is consistent with the three-dimensional geometry of expectation 4, images coming from a first camera of the camera assembly may be converted into a prediction for an image coming from a second camera of the camera assembly based on the geometry of expectation 4, according to block 133. Then, according to block 134, it may be determined how well this prediction matches the image actually coming from the second camera of the camera assembly.

[0045] This examination may in turn involve extracting features from the image coming from the second camera on the one hand and from the predictions for this image on the other hand, according to block 134a, and comparing these features with each other, according to block 134b.

[0046] This comparison may in turn include making a binary determination for each feature whether the feature from the image matches a corresponding feature from the prediction for that image, according to block 134c, and determining the degree of match between the image and the prediction from the number of features that match each other, according to block 134d.

[0047] In addition, it may be examined how well a given test image, not showing the scene observed by sensor 1, matches with an expectation for an image coming from the second camera of the camera assembly, according to block 135. This degree of match may be considered as a noise level for the match established between the image coming from the second camera of the camera assembly and the expectation for this image, according to block 136.

[0048] According to block 137, the actual sensor data 2 recorded by the multiple different sensors 1 can be examined separately for where this actual sensor data 1 matches the prediction 4 retrieved from the map 3. Then, according to block 138, it can be determined that the actual sensor data 2 collectively matches the prediction 4 only for those locations where the actual sensor data 2 of all sensors 1 collectively match the prediction.

[0049] According to block 141, the actual sensor data 2 that is being examined for a match with the prediction 4 may be matched against the actual sensor data 2 recorded by the further sensor 1. Then, according to block 142, a match with the prediction 4 may be determined or maintained only for those locations where this match is positive.

[0050] FIG. 2 illustrates an exemplary application of the method 100 to a road scene. The sensor 1 is mounted on a vehicle, which is not depicted in this example. From a viewpoint defined by the position 1a and the orientation 1b of the sensor 1, the sensor 1 captures actual sensor data 2 within its capture range 1c. In the example shown in Figure 2, the scene includes a road 10 with a vehicle 12 driving in front and trees 11 at the roadside.

[0051] The geo-resolved map 3 also captures the road 10 and trees 11, but there is no vehicle 12 ahead. A characteristic of interest 5 has been stored for a region of the road 10, and therefore this region is freely traversable.

[0052] The landscape and / or scenery geometry contained in the map 3 is compared with the actual sensor data 2 as a prediction 4. In this case, it is determined that the area of ​​the road 10 for which there is a large agreement between the actual sensor data 2 and the prediction 4 and for which characteristics 5 are simultaneously stored in the map 3 in association with the prediction 4, is a freely passable area. Accordingly, this area is considered to be freely passable.

[0053] The only exception here is the area with the leading vehicle 12. The actual sensor data 2 here differs from the expectation 4, since the leading vehicle 12 is not on the map 3. Accordingly, the area with the leading vehicle 12 is not considered as freely passable.

Claims

1. A method (100) for evaluating location resolved actual sensor data (2) recorded by at least one sensor (1), comprising: a step (110) in which the position (1a) and orientation (1b) of said sensor (1) at the time of recording said sensor data (2) are determined; A step (120) of retrieving a location resolved prediction (4) based on the position (1a) and the orientation (1b) of the sensor (1) from a location resolved map (3); a step (130) in which the degree to which the actual sensor data (2) matches the expectations (4) is examined; a step (140) of determining that the scene observed by said sensor (1) has characteristics (5) stored in said map (3) associated with said prediction (4), at least for locations where said actual sensor data (2) is consistent with said prediction (4); The method (100) comprising:

2. The method (100) of claim 1, wherein sensor data (2) recorded by a sensor (1) of a vehicle (50) or a robot (60) is selected (105).

3. The method (100) of claim 2, wherein the characteristics (5) stored in the map (3) include assertions (121) about how freely traversable a location associated with the prediction (4) may be by the vehicle (50) or robot (60).

4. a control signal (150a) for the vehicle (50) or robot (60) is determined (150) taking into account the determination of where the scene observed by the sensor (1) has the characteristic (5) stored in the map (3) associated with the prediction (4); the vehicle (50) or robot (60) is controlled (160) by the control signal (150a), whereby the driving dynamics of the vehicle (50) or robot (60) are influenced according to the control signal (150a); The method (100) of claim 2 or 3.

5. The method (100) according to any one of the preceding claims, wherein the actual sensor data (2) and the predictions (4) are transferred (131) into a common spatial reference system and / or a common working space, and the actual sensor data (2) is compared (132) with the predictions (4) in the reference system or working space.

6. The prediction (4) is based on the scene observed by the sensor (1). Positionally resolved 3D geometry and / or Texturing and / or Reflectance amplitude and / or Multispectral response and / or Magnetic resonance The method (100) of any one of claims 1 to 5, comprising (122)

7. The method (100) according to any one of the preceding claims, wherein sensor data (2) recorded by at least one radar sensor and / or at least one lidar sensor and / or at least one camera are selected (106).

8. Examining whether sensor data (2) recorded by a stereoscopic camera assembly is consistent with the three-dimensional geometry of the prediction (4), - transforming (133) an image coming from a first camera of said camera assembly into a prediction for an image coming from a second camera of said camera assembly, based on said geometry of said prediction (4); - examining (134) how well said prediction matches an image actually resulting from said second camera of said camera assembly; The method (100) of claims 6 and 7.

9. Features are extracted (134a) from an image coming from the second camera, on the one hand, and from the predictions for said image, on the other hand; and the features are compared to each other (134b); 9. The method (100) of claim 8.

10. a binary determination is made for each feature whether the feature from the image matches a corresponding feature from the prediction for that image (134c); and a degree of match between the image and the prediction is determined from the number of features that match each other (134d); 10. The method (100) of claim 9.

11. Additionally, it is examined (135) how well predetermined test images not showing the scene observed by the sensor (1) match the expectations for images coming from the second camera of the camera assembly; and the degree of match is considered as a noise level for the match established between an image coming from the second camera of the camera assembly and the prediction for said image (136); The method according to any one of claims 8 to 10.

12. actual sensor data (2) recorded by a plurality of different sensors (1) are examined (137) for each location where the actual sensor data (1) matches the predictions (4) retrieved from the map (3); and determining (138) that the actual sensor data (2) collectively agrees with the prediction (4) only for locations where the actual sensor data (2) of all sensors (1) collectively agrees with the prediction, The method (100) according to any one of the preceding claims.

13. the actual sensor data (2) to be examined for conformity with said prediction (4) is matched (141) against actual sensor data (2) recorded by a further sensor (1); and Only for those locations where the likelihood check is positive, a match with the expectation (4) is determined or maintained (142); The method (100) according to any one of the preceding claims.

14. The method (100) according to any one of the preceding claims, wherein the determined position (1a) and / or the determined orientation (1b) is optimized (111) with the goal of maximizing the agreement between the actual sensor data (2) and the predictions (4).

15. A computer program comprising machine-readable instructions which, when executed on one or more computers, cause said one or more computers to carry out the method (100) of any one of claims 1 to 14.

16. A machine-readable data carrier and / or download product comprising a computer program according to claim 15.

17. One or more computers comprising a computer program according to claim 15 and / or comprising a machine-readable data storage medium and / or a download product according to claim 16.

Images