Recognizing Scene Features with Higher Safety Integrity by Matching Sensor Data with Predictions

Abstract

1. A method (100) for evaluating location-resolved actual sensor data (2) recorded by at least one sensor (1), comprising: a step (110) in which a position (1a) and an orientation (1b) of the sensor (1) at the time of recording the sensor data (2) are determined; a step (120) in which a location-resolved prediction (4) is retrieved (120) from a location-resolved map (3) based on the position (1a) and orientation (1b) of the sensor (1); a step (130) in which how well the actual sensor data (2) matches the prediction (4); and a step (140) in which, at least for those positions where the actual sensor data (2) matches the prediction (4), the scene observed by the sensor (1) has a characteristic (5) stored in the map (3) associated with the prediction (4).

Description

[Technical Field]

[0001] The present invention relates to the recognition of characteristics of a scene observed by at least one sensor, for example for the at least partially autonomous control of a vehicle or robot. [Background technology]

[0002] Driving assistance systems and at least partially automated steering systems for vehicles or robots capture the vehicle's or robot's surroundings with one or more sensors and, based on this, determine a plan for the vehicle's or robot's subsequent behavior. Neural networks are often used to determine such plans or a representation of the surroundings as a pre-product for planning. DE 10 2018 008 685 A1 discloses a method for training a neural network to determine a path prediction for a vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] German Patent Application Publication No. 102018008685 Summary of the Invention [Means for solving the problem]

[0004] The present invention provides a method for evaluating location-resolved actual sensor data recorded by at least one sensor. The sensor may be, for example, a mobile sensor carried by a person, mounted on a robot, or any land, water, or air vehicle. However, the sensor may also be, for example, a stationary sensor observing, for example, a busy intersection. Location-resolved actual sensor data refers to any sensor data that is assigned to a location determined within the scope of its capture. For example, radar data and lidar data may exist as a point cloud of measurements, each assigned to a point in three-dimensional space from which the scanning radiation used is reflected. An image assigns intensity values ​​to the pixels of the sensor used for its capture, usually arranged in a regular grid. Each pixel, in turn, corresponds to a position in space from which the incident light originates, depending on the viewpoint at which the image was recorded and the optical system used.

[0005] The position and orientation of the sensor at the time of recording the sensor data are determined. This process is also called "registration" or "localization" of the sensor in the physical world. Any system can be used for this purpose, alone or in combination. For example, a navigation system based on radio signals emitted from satellites or even from ground stations can be used. Alternatively or in combination, for example, an inertial navigation system can be utilized. If the sensor is a stationary sensor, the position and orientation of the sensor are known in advance, so registering or localizing the sensor is particularly simple.

[0006] From the location-resolved map, a location-resolved prediction for the actual sensor data is retrieved based on the sensor's location and orientation. The prediction is stored in the location-resolved map in association with at least one characteristic of the observed scene. If the scene observed by the at least one sensor actually has the characteristic stored in the map, then the actual sensor data is expected to match the prediction retrieved from the location-resolved map.

[0007] This means that in reverse reasoning, if the actual sensor data matches the expectation, then the feature stored in the map also exists, and testing for the existence of this feature is precisely the goal of the method.

[0008] Thus, the degree to which the actual sensor data matches the expectation is examined, and for at least those locations where the actual sensor data matches the expectation, it is determined that the scene observed by at least one sensor has the characteristics stored in the map associated with the expectation.

[0009] The predictions stored in the map are a kind of "fingerprint" of the scene observed by the sensor. This "fingerprint" can include any characteristics that can be evaluated from the sensor data, such as geometry, texturing, multispectral response, etc. Fundamental physical characteristics, such as magnetic resonance, are also considered.

[0010] The sensor data may be, inter alia, sensor data recorded by sensors of a vehicle or a robot, for example. The vehicle may be any land, water, or air vehicle. An important application of the method, in the context of vehicle and robot control, is the examination of the question of which locations can be freely traversed by a vehicle or robot. For this purpose, the properties stored in the map may include assertions of how freely traversable a location associated with the prediction can be by a vehicle or robot. The method is also applicable in the same way to mobile sensors carried by people, for example to signal traversable areas to visually impaired people.

[0011] This can be clarified with a simple example in which camera images are used as sensor data. In this case, camera images are captured on a map, and in these images, areas are labeled as being freely traversable by the vehicle or not. Therefore, if the areas determined during the vehicle's travel are re-identified in their correct positions based on camera images recorded by the vehicle at each time, it is guaranteed that the assertions stored in the map about these areas (how freely traversable these areas are by the vehicle or not) correspond to the current state of these areas. For example, if an obstacle is found at a certain location in a camera image recorded during travel and this obstacle differs in at least one characteristic, such as visual appearance or obstacle geometry, from the same characteristic in the image stored in the map, this object can be reliably recognized.

[0012] In this way, the open classification task of whether an area is freely passable (or has any other characteristics important for trip planning) can be reduced to a comparison with known information stored in the map. It has been found that this allows the recognition of which areas are freely passable to be performed with significantly better safety integrity. If a spatial area generates sensor data at a specific location that is the same as or at least similar to the prediction stored in the map, it necessarily follows that this area has the same object occupancy (i.e., no objects, and therefore passable) compared to the state at the time the map was created, and that no further objects are present between the sensor and this area. Any unexpected object within the area or within the line of sight between the sensor and this area will disrupt the agreement between the actual sensor data and the prediction, and as a result, this area will no longer be identified as freely passable.

[0013] This has the advantage over open classification tasks solved by, for example, neural networks: a non-freely traversable range will be recognized as not being freely traversable even if the reason for the non-traversability is highly unusual and therefore not present in the training data used to train the neural network. Events that are too unlikely to be present in the training data are also called the "long tail."

[0014] For example, on highways, sometimes highly unusual objects, such as furniture, large electrical appliances, skis, or bicycles, go missing due to inadequate secure fastening of luggage. The appearance of such objects on the roadway where one's vehicle is traveling is a highly dangerous situation, which is why it is fortunately rare. However, this also means that such cases do not occur with a high probability when collecting real-world data for training neural networks in the context of test drives. It is practically impossible to deliberately ambush such situations in public transport.

[0015] A similar scenario applies to the dreaded "blow-ups" that occur suddenly during heat waves, when concrete roadways on highways become rounded or cracked. By the time the "blow-up" appears on camera images, accidents are all but impossible to prevent, so training for such situations is also impossible.

[0016] Furthermore, the recognition of free-traffic areas through re-identification within the map is not vulnerable to deliberate manipulation. Many neural network-based image classification systems can be misclassified by the introduction of malicious interference patterns. For example, a stop traffic sign can be tampered with so that it is classified as a "70 km / h speed limit" sign by attaching an inconspicuous sticker. Attempts have already been successful in completely blocking the ability of a connected image classifier for pedestrian recognition by attaching a film with an inconspicuous semi-transparent dot pattern over a camera lens. The pedestrian was classified as free-traffic space.

[0017] Within the framework of the method presented here, such attempts at manipulation are either completely ignored, or, in the worst case, areas that should be free to pass through are no longer recognized as free to pass, meaning that any unexpected problems are avoided instead of being passed through ("fail-safe").

[0018] Furthermore, the prediction can already set an upper limit for areas that can usually be recognized as free passage. The re-identification of an area in the prediction can only lead to the determination that this area is free passage if this area was marked as free passage in relation to the prediction. Areas that are not marked as free passage, such as concrete guardrails or trees at the side of the road, can never be recognized as free passage.

[0019] Significantly better safety integrity in determining the characteristics of the scene observed by at least one sensor makes the control of the vehicle or robot based on the thus determined characteristics more likely to be appropriate for the respective situation. Therefore, it is advantageous to determine a control signal for the vehicle or robot taking into account a determination of at which location the scene observed by the sensor has the characteristics stored in the map associated with the prediction. The vehicle or robot is controlled by this control signal, whereby the driving dynamics of the vehicle or robot are influenced accordingly.

[0020] The comparison of actual sensor data with predictions is not limited to a 1:1 re-recognition. Instead, any form of tolerance or even abstraction to specific features can be defined for this re-recognition. That is, even two camera images recorded directly after each other of the same scene are generally not completely identical.

[0021] In a further particularly advantageous embodiment, the actual sensor data and the predictions are transferred to a common spatial reference frame and / or a common working space. The actual sensor data are compared with the predictions in this reference frame or working space. In this way, sensor data captured with different modalities and the predictions can also be compared with each other. For example, the predictions can be based on the image of the scene observed by at least one sensor. Position-resolved 3D geometry and / or Texturing and / or Reflectance amplitude and / or Multispectral response and / or Magnetic Resonance Such a geometry may include, for example, a location where the location is not known. Such a geometry may be determined, for example, on the basis of image capture. Such a geometry may be amenable to labeling in terms of the extent of free movement of the landscape or other characteristics. In this case, for example, radar or lidar data may be examined to see how well it matches this geometry.

[0022] For capturing sensor data, radar and lidar sensors, as well as stereoscopically configured cameras or multi-camera systems, among others, are considered. Such camera assemblies also provide depth information that can be examined against the expected geometry. Moving monocular cameras are also considered for generating depth information. Furthermore, for example, multispectral cameras, Time of Flight (ToF) sensors, cameras with event-based response pixels, ultrasonic sensors, or magnetic sensors can also be used.

[0023] For example, testing whether sensor data recorded by a stereoscopic camera assembly is consistent with an expected three-dimensional geometry may involve testing a so-called stereo-hypothese, which is based on the geometry of the scene linking the images provided by both cameras of the stereoscopic camera assembly, i.e., if one of the images and the expected geometry are present, this at least roughly defines the other image.

[0024] Thus, an image provided by a first camera of the camera assembly can be transformed into a prediction for an image provided by a second camera of the camera assembly based on the geometry of the prediction. This prediction can then be examined to determine how well it matches the image actually provided by the second camera of the camera assembly. The transformation can include, among other things, for example, warping the image provided by the first camera to match the viewpoint of the second camera based on the geometry.

[0025] To compare the image from the second camera with the prediction, features can be extracted from the image from the second camera and from the prediction for this image. These features can then be compared with each other. This abstraction into features can smooth out non-essential differences, such as differences in color or brightness, for the purpose of comparison.

[0026] In particular, for every feature to be compared, a binary determination can be made as to whether a feature from the image matches a corresponding feature from the prediction for this image.The degree of match between the image and the prediction can then be determined from the number of features that match each other.That is, for example, the Hamming distance of the examined feature combination can be determined, and this Hamming distance increases the more inconsistent the image features on the one hand and the prediction on the other hand are.

[0027] In a further advantageous embodiment, a predetermined test image that does not show the scene observed by the sensor is also examined to determine how well it matches an expectation for an image provided by a second camera of the camera assembly. This degree of match is then considered as a noise level for the determined match between the image provided by the second camera of the camera assembly and the expectation for that image. In this way, a signal-to-noise ratio for the match between the image provided by the second camera and the expectation for that image can be determined. This signal-to-noise ratio is more convincing than a match alone. For example, the convincingness of an image may be reduced if a large portion of it is saturated due to overexposure or underexposure.

[0028] The method can also be generalized to the effect that actual sensor data is recorded by multiple sensor modalities and then these results are compiled. Thus, in a further advantageous embodiment, the actual sensor data recorded by multiple different sensors is individually examined to determine at which locations this actual sensor data matches the respective predictions retrieved from the map. Collectively, the actual sensor data is then determined to match the predictions only for locations where the actual sensor data of all sensors matches the respective predictions. That is, for example, an area is only considered to be freely traversable by a vehicle or robot if the area is recognized as freely traversable based on sensor data from multiple sensors of different sensor modalities (e.g., lidar and stereoscopic cameras) independently of one another.

[0029] First, for example, the degree to which current lidar data corresponds to the geometry of the scene stored in the map can be examined. In parallel, the degree to which images, for example, from a stereoscopic camera assembly correspond to this geometry can be examined. For this purpose, for example, the geometry can be transformed into the reference coordinate system of the first camera of the camera assembly. Then, as described above, the image from the first camera of the camera assembly is transformed based on the geometry into a prediction for an image from the second camera of the camera assembly, and this prediction is compared with the image actually obtained from the second camera, thereby examining the stereo hypothesis. Then, only for locations where both the lidar data and the images from the stereoscopic camera assembly correspond to the predicted geometry stored in the map, can these locations be determined to have the searched property (e.g., the free traversability of these locations) according to the map.

[0030] In a further advantageous embodiment, the actual sensor data examined for correspondence with the prediction is checked against actual sensor data recorded by a further sensor. The correspondence with the prediction is then determined or maintained only for locations where this check is positive. That is, instead of two comparisons between one sensor modality and the prediction, only one comparison is performed. In addition, both sensor modalities are compared with each other. For example, a certain area may be found to be free to travel based on a match between the lidar data and the geometry stored as the prediction, and this area may be declared free to travel only if the lidar data is consistent with the imagery provided by the stereoscopic camera assembly.

[0031] In a further advantageous embodiment, the determined position and / or the determined orientation are optimized with the goal of maximizing the agreement between the actual sensor data and the prediction. As explained above, for the comparison of the actual sensor data and the prediction, it is important that the prediction for the correct position and correct orientation of the sensor is retrieved from the map. Only in this case can the prediction be correctly re-established based on the actual sensor data at the time. However, all methods for determining the position and orientation have limited accuracy. Thus, for example, if the agreement between the sensor data and the prediction can be significantly improved by additionally displacing the determined position of the sensor and / or by additionally tilting or rotating the determined orientation of the sensor, this indicates that the previously determined position or the previously determined orientation of the sensor was not necessarily correct. As an alternative or in combination, any other technique for transferring the actual sensor data and the map to a common reference coordinate system can be used.

[0032] The method may, inter alia, be implemented entirely or partly on a computer. The invention therefore also relates to a computer program having machine-readable instructions which, when executed on one or more computers, cause the one or more computers to carry out the aforementioned method. In this sense, embedded systems for vehicle control devices and technical equipment, which are likewise capable of executing machine-readable instructions, may also be considered computers.

[0033] The invention also relates to a machine-readable data storage medium and / or a download product comprising a computer program, which is a digital product that can be transmitted over a data network, i.e. downloaded by a user of the data network, and which can be offered for immediate download, for example, in an online shop.

[0034] Additionally, the computer may be loaded with a computer program, a machine-readable data storage medium, or a downloadable product. Further measures for improving the invention are detailed below in conjunction with the description of preferred exemplary embodiments of the invention based on the figures. [Brief explanation of the drawings]

[0035] [Figure 1] FIG. 1 illustrates an exemplary embodiment of a method 100 for evaluating location-resolved actual sensor data 2. [Figure 2] FIG. 1 illustrates an exemplary application of the method 100 in a road scene. DETAILED DESCRIPTION OF THE INVENTION

[0036] FIG. 1 is a schematic flow diagram of one exemplary embodiment of a method 100 for evaluating location-resolved actual sensor data 2 with respect to where the scene observed by the sensor 1 has a characteristic of interest 5.

[0037] In step 105, sensor data 2 recorded by sensors 1 of the vehicle 50 or robot 60 may be selected. In step 106, sensor data 2 recorded by a radar sensor, a lidar sensor, and / or a stereoscopic camera assembly may be selected.

[0038] In step 110, the position 1a and orientation 1b of the sensor 1 at the time the sensor data 2 was recorded is determined. In step 120, a geo-resolved prediction 4 is retrieved from the geo-resolved map 3 based on the position 1a and orientation 1b of the sensor 1. The map 3 also stores a geo-resolved property 5 of interest. The property 5 is associated with the prediction 4, specifically, the association is such that the actual sensor data 2 and the prediction 4 match, assuming that the scene observed by the sensor 1 from a certain position has the property 5.

[0039] In step 130, the actual sensor data 2 is examined to see how well it matches the predictions 4. In step 140, it is determined that the scene observed by sensor 1 has characteristics 5 stored in map 3 associated with prediction 4, at least for locations where actual sensor data 2 matches prediction 4. This is illustrated for one example in FIG.

[0040] In step 150, a control signal 150a for the vehicle 50 or robot 60 is determined taking into account a determination of where the scene observed by the sensor 1 has characteristics 5 stored in the map 3 associated with the prediction 4.

[0041] In step 160, the vehicle 50 or robot 60 is controlled by this control signal 150a, which affects the driving dynamics of the vehicle 50 or robot 60 accordingly.

[0042] Based on block 111, the determined position 1a and / or the determined orientation 1b may be optimized with the goal of maximizing the agreement between the actual sensor data 2 and the predictions 4. According to block 121, the properties 5 stored in the map 3 may include, among other things, assertions about how freely traversable the location associated with the prediction 4 may be by vehicle 50 or robot 60, for example.

[0043] Based on block 122, prediction 4 may include, among other things, for example, a position-resolved three-dimensional geometry of the scene observed by sensor 1. The actual sensor data 2 and the prediction 4 may be translated into a common spatial reference frame and / or common workspace, per block 131. The actual sensor data 2 may then be compared to the prediction 4 within this reference frame or workspace, per block 132.

[0044] To determine whether the sensor data 2 recorded by the stereoscopic camera assembly is consistent with the three-dimensional geometry of prediction 4, images from a first camera of the camera assembly may be converted into a prediction for images from a second camera of the camera assembly based on the geometry of prediction 4, per block 133. Thereafter, how well this prediction matches images actually coming from the second camera of the camera assembly may be determined, per block 134.

[0045] This examination may in turn involve extracting features from the image coming from the second camera on the one hand and from the prediction for this image on the other hand, according to block 134a, and comparing these features with each other, according to block 134b.

[0046] This comparison may in turn include, in accordance with block 134c, making a binary determination for each feature as to whether the feature from the image matches the corresponding feature from the prediction for that image, and determining the degree of match between the image and the prediction from the number of features that match each other, in accordance with block 134d.

[0047] Additionally, a predetermined test image that does not show the scene observed by sensor 1 may be examined for its degree of agreement with an expectation for an image resulting from a second camera of the camera assembly, according to block 135. This agreement may be considered as a noise level for the agreement established between the image resulting from the second camera of the camera assembly and the expectation for this image, according to block 136.

[0048] According to block 137, actual sensor data 2 recorded by a plurality of different sensors 1 can be examined separately to determine at which locations this actual sensor data 1 matches the prediction 4 retrieved from the map 3. Then, according to block 138, it can be determined that the actual sensor data 2 collectively matches the prediction 4 only for locations where the actual sensor data 2 of all sensors 1 respectively match the prediction.

[0049] According to block 141, actual sensor data 2 that is being examined for conformance with prediction 4 may be matched against actual sensor data 2 recorded by additional sensors 1. Then, according to block 142, conformance with prediction 4 may be determined or maintained only for locations where this conformance is positive.

[0050] FIG. 2 shows an exemplary application of the method 100 to a road scene. The sensor 1 is mounted on a vehicle, which is not depicted in this example. From a viewpoint defined by the position 1a and orientation 1b of the sensor 1, the sensor 1 captures actual sensor data 2 within its capture range 1c. In the example shown in Figure 2, the scene includes a road 10 with a vehicle 12 in front and trees 11 along the roadside.

[0051] The resolved map 3 also captures the road 10 and the tree 11, but does not have a vehicle 12 ahead of it. The property of interest 5 is stored for the area of ​​the road 10, and therefore this area is freely traversable.

[0052] The landscape and / or scenery geometry contained in the map 3 is compared with the actual sensor data 2 as a prediction 4. In this case, for the most part, the actual sensor data 2 and the prediction 4 match for the area of ​​the road 10, for which characteristics 5 are simultaneously stored in the map 3 in association with the prediction 4, and which is determined to be a freely passable area. Accordingly, this area is considered to be freely passable.

[0053] The only exception here is the area with the leading vehicle 12. Since the leading vehicle 12 is not on the map 3, the actual sensor data 2 here differs from the expectation 4. Accordingly, the area with the leading vehicle 12 is not considered to be freely passable.

Claims

1. A method (100) for evaluating actual sensor data (2) recorded by at least one sensor (1), comprising: a step (110) in which the position (1a) and orientation (1b) of said sensor (1) at the time of recording said sensor data (2) are determined; a step (120) in which a prediction (4) is called from a map (3) based on the position (1a) and the orientation (1b) of the sensor (1); A step (130) in which the actual sensor data (2) is examined to see how well it matches the expectations (4); a step (140) of determining that the scene observed by said sensor (1) has the characteristics (5) stored in said map (3) associated with said prediction (4), at least for the locations where said actual sensor data (2) is consistent with said prediction (4); In the method (100), the actual sensor data (2) tested for conformance with the prediction (4) is validated (141) against actual sensor data (2) recorded by a further sensor (1); and Only for those locations where the likelihood check is affirmative, a match with the expectation (4) is determined or maintained (142); Method (100).

2. The method (100) of claim 1, wherein sensor data (2) recorded by sensors (1) of a vehicle (50) or a robot (60) are selected (105).

3. 3. The method (100) of claim 2, wherein the characteristics (5) stored in the map (3) include assertions (121) about how freely traversable the location associated with the prediction (4) is by the vehicle (50) or robot (60).

4. a control signal (150a) for the vehicle (50) or robot (60) is determined (150) taking into account the determination of where the scene observed by the sensor (1) has the characteristic (5) stored in the map (3) associated with the prediction (4); the vehicle (50) or robot (60) is controlled (160) by the control signal (150a), whereby the driving dynamics of the vehicle (50) or robot (60) are influenced accordingly; The method (100) of claim 2.

5. 10. The method (100) of claim 1, wherein the actual sensor data (2) and the predictions (4) are transferred (131) to a common spatial reference frame and / or a common workspace, and the actual sensor data (2) is compared (132) with the predictions (4) in the reference frame or workspace.

6. The prediction (4) is based on the scene observed by the sensor (1). spatially resolved 3D geometry and / or Texturing and / or reflectance amplitude and / or Multispectral response and / or Magnetic resonance The method (100) of claim 1, comprising (122)

7. The method (100) of claim 1, wherein sensor data (2) recorded by at least one radar sensor and / or at least one lidar sensor and / or at least one camera are selected (106).

8. Examining whether sensor data (2) recorded by a stereoscopic camera assembly is consistent with the three-dimensional geometry of the prediction (4), - transforming (133) an image coming from a first camera of said camera assembly into a prediction for an image coming from a second camera of said camera assembly based on said geometry of said prediction (4); examining (134) how closely the prediction matches an image actually provided by the second camera of the camera assembly; The method (100) of claim 6.

9. Features are extracted (134a) from the image coming from the second camera on the one hand and from the prediction for said image on the other hand, and the features are compared with each other (134b); The method (100) of claim 8.

10. a binary determination is made (134c) for each feature whether the feature from the image matches a corresponding feature from the prediction for that image; and a degree of match between the image and the prediction is determined from the number of features that match each other (134d); 10. The method (100) of claim 9.

11. Additionally, a predetermined test image not showing the scene observed by the sensor (1) is examined (135) for its conformity with the expectation for an image resulting from the second camera of the camera assembly; and the degree of match is considered as a noise level for the match established between an image coming from the second camera of the camera assembly and the prediction for said image (136); The method of claim 8.

12. Actual sensor data (2) recorded by a plurality of different sensors (1) are examined (137) separately to determine where the actual sensor data (1) matches the predictions (4) retrieved from the map (3), respectively; and determining (138) that the actual sensor data (2) collectively agrees with the prediction (4) only for locations where the actual sensor data (2) of all sensors (1) collectively agrees with the prediction; The method (100) of claim 1.

13. 2. The method (100) of claim 1, wherein the determined position (1 a) and / or the determined orientation (1 b) are optimized (111) with the goal of maximizing the agreement between the actual sensor data (2) and the prediction (4).

14. A computer program product containing machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100) of claim 1.

15. 15. A machine-readable data storage medium having a computer program according to claim 14.

16. One or more computers having the computer program of claim 14.

Images