Sensor fusion using interactive correspondence analysis of sensor data

Abstract

Sensor data (121-124) is received from a plurality of sensors (111-114), wherein the plurality of sensors (111-114) represent a common scene (80) using a plurality of measurement modes. At least one corresponding feature is determined in each of the sensor data (121-124), and corresponding performance specification data (172) is also obtained for each of the plurality of sensors (111-114). Furthermore, a correspondence analysis of interactions between features of the sensor data (121-124) is performed taking into account the corresponding performance specification data (172). Sensors are fused as a function of the correspondence analysis.

Description

Technical Field

[0001] Various examples of the invention generally relate to sensor fusion of sensor data from multiple sensors. Background Art

[0002] In many application areas, reliable and robust measurements of a scene, such as an interior or a road, are required. One example involves controlling the movement of a robot in a scene. If the scene is measured accurately and reliably, collisions with objects in the scene can be avoided and an optimal path can be planned.

[0003] In various examples, sensor fusion of sensor data from multiple sensors, each of which uses multiple measurement modes to image a common scene, is used to improve the accuracy of measuring the scene. For example, YE, Mao et al., "3D reconstruction in the presence of glasses by acoustic andstereo fusion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, pp. 4885-4893, describe an exemplary technique. Summary of the invention

[0004] Improved sensor fusion techniques are needed. In particular, there is a need for sensor fusion techniques that can accurately and reliably measure the scene.

[0005] In one example, a computer-implemented sensor fusion method includes receiving sensor data from a plurality of sensors. In this case, the plurality of sensors use a plurality of measurement modes to image a common scene. The method also includes determining at least one corresponding feature in the respective sensor data for a corresponding sensor of the plurality of sensors. The method also includes obtaining corresponding performance specification data for each of the plurality of sensors. The method also includes performing a mutual correspondence analysis between the features of the sensor data, wherein, in this case, the corresponding performance specification data is taken into account. The method also includes performing sensor fusion based on the correspondence analysis.

[0006] A computer program or computer program product, comprising a program code executable by at least one processor. Execution of the program code causes the at least one processor to perform a sensor fusion method. The method comprises receiving sensor data from a plurality of sensors. In this case, the plurality of sensors use a plurality of measurement modes to image a common scene. The method further comprises determining at least one corresponding feature in the respective sensor data for a respective sensor of the plurality of sensors. The method further comprises obtaining corresponding performance specification data for each of the plurality of sensors. The method further comprises performing a mutual correspondence analysis between features of the sensor data, wherein, in this case, the corresponding performance specification data is taken into account. The method further comprises performing sensor fusion on the basis of the correspondence analysis.

[0007] An apparatus includes at least one processor and a memory. The at least one processor is configured to load a program code from the memory and, based on the execution of the program code, perform the following steps: receiving sensor data from a plurality of sensors, wherein the plurality of sensors use a plurality of measurement modes to image a common scene; determining at least one corresponding feature in the respective sensor data for respective sensors of the plurality of sensors; obtaining corresponding performance specification data for each of the plurality of sensors; performing a mutual correspondence analysis between features of the sensor data taking into account the corresponding performance indicator data; and performing sensor fusion based on the correspondence analysis.

[0008] The features mentioned above and those described below can be used not only in the respective combinations explicitly stated but also in further combinations or alone without departing from the scope of protection of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 schematically illustrates a scene with multiple objects, where a robot moves according to various examples;

[0010] Figure 2 schematically illustrates exemplary processing of sensor data from a sensor capturing a scene;

[0011] Figure 3 is a flow chart of an exemplary method. DETAILED DESCRIPTION

[0012] The characteristics, features and advantages of the present invention described above and the manner of achieving them will become more clear and explicit with reference to the following description of exemplary embodiments, which will be explained in more detail with reference to the drawings.

[0013] Below, on the basis of preferred embodiments, and with reference to the accompanying drawings, the present invention is explained in more detail. In the drawings, the same reference numerals represent the same or similar elements. The accompanying drawings are schematic representations of various embodiments of the present invention. The elements shown in the drawings are not necessarily shown in true proportion. Instead, the various elements shown in the drawings are depicted in a manner that a person skilled in the art can understand their functions and versatility. As shown in the figure, the connection and coupling between the functional units and the elements can also be implemented as indirect connections or couplings. The connection or coupling can be implemented in a wired or wireless manner. The functional units can be implemented as hardware, software, or a combination of hardware and software.

[0014] The following is an introduction to techniques related to sensor fusion. Sensor fusion involves the combination of sensor data from multiple sensors. In this context, sensor fusion may involve multiple sensors imaging a common scene using multiple measurement modes. In other words, different sensors may therefore image the scene using different measurement modes. This may mean that the measurement fields of different sensors have overlapping areas.

[0015] In general, the techniques described herein can be used in a variety of application areas. For example, the movement of a robot in a scene can be controlled based on sensor fusion. For example, a mobile robot can be used, and the trajectory of the mobile robot can be planned based on sensor fusion. In another example, a robot that is mounted in a fixed manner and has one or more articulated robot arms can be used. The robot arm can then be moved based on sensor fusion. For example, the proximity of the robot arm to an operator or another restricted area can be monitored.

[0016] In general, the number and type of sensors associated with sensor fusion may vary in different embodiments. For example, a range of 2 to 10 sensors may typically be considered in sensor fusion. Exemplary sensors include active sensors, such as ToF (time of flight) sensors; radar sensors; lidar sensors, such as 1-D / 2-D scanning / rotating or flash lidars with fan-shaped illumination; ultrasonic sensors, such as single sensors or ultrasonic sensor arrays. Alternatively or in addition, passive sensors, such as cameras or stereo cameras, may also be used. Cameras may be used in the infrared spectrum, the visible spectrum, or the ultraviolet spectrum. Monocular cameras or stereo cameras may be used. Active cameras may also be used, which project patterns, such as line patterns, etc., and from which the surface topography may be determined. It is clear from these examples that the measurement mode varies depending on the type of sensor. For example, radar sensors and lidar sensors use electromagnetic waves in different wavelength ranges, radar sensors, for example, use waves in the gigahertz range, while lidar sensors typically use waves in the infrared range, such as 950nm or C-band waves. Ultrasonic sensors use density fluctuations of the air rather than electromagnetic waves as the measurement mode.

[0017] Generally speaking, both fixed sensors and movable sensors can be used relative to the scene. For example, one or more sensors can be mounted on a robot whose movement is controlled based on sensor fusion; alternatively or additionally, one or more sensors can be mounted in a fixed manner relative to the robot.

[0018] Sensor data can also vary depending on the sensor. Sensor data images a scene. Depending on the measurement mode, different physical observables are used to image the scene. For example, the information content of the sensor data can vary. Some sensor data may not have any spatial resolution: for example, an ultrasonic sensor may only provide a single distance value but no lateral resolution. Other sensor data may have 1-D resolution: for example, this is the case with a lidar sensor that scans in one dimension. However, other sensor data may have 2-D resolution, such as a 2-D scanning lidar sensor or a camera image or radar sensor based on a 2D coherent transmission array. In this case, the resolution of the sensor data may vary from sensor to sensor: for example, a radar sensor typically has a smaller number of lateral image points than a lidar sensor.

[0019] Various examples are based on the insight that various sensors may provide incorrect or erroneous information depending on the scene. One example relates to cameras as sensors, for example: cameras generally cannot detect objects that are transparent in the corresponding wavelength range or can only detect them to a limited extent. For example, a piece of glass cannot be detected by a camera or can only be detected to a limited extent. On the other hand, for example, ultrasonic sensors can detect glass particularly well. Another example relates to mirrors: for example, a camera can detect objects reflected in a mirror, but not the mirror itself and can therefore provide erroneous information about the scene. On the other hand, for example, a mirror may not be particularly reflective for radar waves and can therefore be reliably detected by a radar sensor. Another example relates to the limited range of passive cameras, for example, in the absence of scene illumination. Another example relates to the reduced range of light-based sensors in the presence of smoke or fog.

[0020] Therefore, there may often be a need to validate sensor data from various sensors. In this context, the various examples described herein are based on the recognition that it is not possible or only to a limited extent possible to validate sensor data from sensors based solely on the sensor data themselves. Therefore, the following describes techniques related to validation of sensor data in connection with sensor fusion by taking into account the availability of multiple sensors imaging a common scene using multiple measurement modalities.

[0021] In reference embodiments, sensor data from different sensors are usually evaluated separately. Obstacle detection is then performed, for example, based on a weighted combination of the sensor data as sensor fusion. For example, the weighting can be performed according to the probability of the presence of an obstacle. In such reference embodiments, such as described in US 2017 / 0 197 311 A1, the scene is usually perceived only to a limited extent, which can lead to inaccuracies or errors, for example, when controlling the movement of a robot.

[0022] In order to eliminate such disadvantages, at least one corresponding feature in the respective sensor data can be determined for each of the plurality of sensors. A mutual correspondence analysis can then be performed between the features of the sensor data. Thus, for example, it can be compared whether a feature determined in the first sensor data is also present in the second sensor data.

[0023] This mutual correspondence analysis makes it possible to verify different sensor data. For example, if a feature determined in the first sensor data is also determined in the second sensor data (and, possibly, vice versa), i.e. if the first sensor data corresponds to the second sensor data, then a positive verification of the first sensor and the second sensor can be assumed with high probability.

[0024] However, sometimes it may happen that there is no or only a limited correspondence between the sensor data from two different sensors. This may be due to different reasons: A first reason is a fault in the operation of the first sensor or the second sensor. The sensor may be damaged. A second reason is that there is no or a limited possibility to capture a certain feature by the measuring mode of the first sensor or the second sensor. For example, objects that are transparent to visible light often cannot be captured by conventional cameras or can only be captured to a limited extent.

[0025] The following describes a technique that is able to distinguish between these two causes of differences between sensor data, detected during the mutual correspondence analysis of sensor data features.

[0026] To this end, according to various examples, corresponding performance specification data may also be obtained for each of the multiple sensors. Then, a mutual correspondence analysis may be performed between the features of the various sensor data in consideration of the corresponding performance specification data. Then, sensor fusion may be performed based on the correspondence analysis.

[0027] Generally, the performance specification data is for a specific function of a corresponding sensor of an imaging feature. Thus, the performance specification data may indicate at least one limitation of a corresponding measurement modality of an imaging feature having a specific property.

[0028] Therefore, the performance specification data can usually be determined according to the corresponding measurement method. The performance specification data can be predefined.

[0029] The performance specifications of a camera may describe, for example, the fact that objects with high transparency in the corresponding spectral range cannot be imaged by the camera or can only be imaged to a limited range. The performance specifications of an ultrasonic sensor may describe, for example, the fact that objects more than 10 meters away from the ultrasonic sensor cannot be imaged by the ultrasonic sensor or can only be imaged by the ultrasonic sensor to a limited range - that is, the performance specifications may generally describe the range. The performance specifications of an ultrasonic sensor may describe, for example, the fact that objects with a lateral range (perpendicular to the depth measurement direction) less than 5 centimeters cannot be imaged by the ultrasonic sensor or can only be imaged by the ultrasonic sensor to a limited range - that is, the performance specifications may generally describe the resolution. The performance specifications may also describe, for example, the fact that surfaces of objects oriented in a specific way - for example, oriented relative to the position and / or orientation (pose) of the corresponding sensor - cannot be imaged or can only be imaged to a limited range. For example, one example involves a surface that is tilted relative to the direct connection line between the surface and the ultrasonic sensor and deflects the sound waves away from the ultrasonic sensor in different directions. Thus, even a lidar sensor cannot detect tilted reflecting surfaces or can only detect them to a limited range. This corresponds to the so-called stealth technology in relation to radar sensors. This property and other properties can be considered in conjunction with the reflectivity of an object, in particular including the material properties of the object's surface. For example, a LIDAR sensor cannot detect a specific object with particularly low reflectivity at a long distance.

[0030] Thus, in a more general case, at least one limitation indicated by the performance specification data may include a lower threshold for the lateral extent of an object in the scene. Alternatively or additionally, at least one limitation indicated by the performance indicator data may also include a geometric orientation of an object in the scene. Sometimes, it is possible to parameterize this geometric orientation based on the pose of the corresponding sensor relative to the object. Alternatively or additionally, at least one limitation may also include a reflectivity of a surface of an object in the scene.

[0031] By taking the performance specification data into account during the mutual correspondence analysis, the differences between the features determined in the different sensor data can be assigned to the limits of the performance specifications of the individual sensors. This can then be taken into account during the sensor fusion process. This allows for particularly accurate and reliable sensor fusion.

[0032] Figure 1 An exemplary system 70 is shown. A mobile robot 90 moves within a scene 80. The scene 80 includes a plurality of objects 81-83, such as solid obstacles 81 and 83 and a glass plate 82. Figure 1 It can be clearly seen that the obstacle 83 is arranged behind the glass pane 82 relative to the robot 90 .

[0033] There are two sensors 111, 112. Figure 1 In the example in FIG. 1 , the sensors 111 , 112 are mounted on the robot 90. However, fixed sensors may generally be used as an alternative or in addition to such mobile sensors.

[0034] The sensors 111, 112 each have a corresponding measurement region 211, 212. The measurement region 211, 212 has a specific range in the depth measurement direction (Z direction). The measurement region 211, 212 also has a specific range in the lateral direction (X direction and Y direction). Specific parameters related to the measurement region 211, 212 can, for example, be included in the range of the corresponding performance specification data of the sensors 111, 112.

[0035] Sensor fusion may be performed on the sensor data from sensors 111, 112. The movement of robot 90 in scene 80 may then be controlled based on the sensor fusion. For example, trajectory 91 may be planned to avoid collisions with objects 81-83.

[0036] Details related to sensor fusion will be combined Figure 2 Give a description.

[0037] Figure 2 Various aspects of processing sensor data from sensors 111, 112 are shown. In particular, Figure 2 Aspects of sensor fusion with respect to sensors 111 , 112 are shown.

[0038] The processing of the sensor data generally makes it possible to search for corresponding geometric properties or generally corresponding features between the various sensor data. On this basis, the plausibility of the various sensor data can be verified. For example, the sensor 111 can be an ultrasonic sensor and can provide distance information. For example, the sensor 112 can be, for example, a camera, which provides an image with a plurality of image points. In this case, contrast values ​​of the individual image points may also be associated with the distance information. Such depth image information items from the camera 112 can then be used to interpret the sensor data from the ultrasonic sensor 111. For example, differences between features detected by the ultrasonic sensor 111 and features detected by the camera 112 can be detected. If such a difference is detected, further processing or interpretation of the sensor data can be initiated, that is, for example, a scene reconstruction algorithm can be used. This will now be combined with the present invention to determine the depth of the sensor data. Figure 2 This technique and further techniques are described in detail.

[0039] In detail, Figure 2The fact is shown that a variable number of sensors 111-114 transmit sensor data 121-124 to a calibration module 131 (usually implemented in software). The performance of the calibration, and therefore the performance of the calibration module 131, is generally optional. The calibration makes it possible to calibrate the sensors 111-114 in an automatic, external way. The calibration can be initiated according to different events. For example, the calibration can be initiated repeatedly according to a predefined schedule. For example, if a number of differences exceed a threshold value, the calibration can also be initiated according to a mutual correspondence analysis. Another example of an event that can initiate a calibration involves a collision of the robot 90 with an obstacle 81-83.

[0040] The calibration data may be determined by calibration. The calibration data may describe the imaging characteristics of the respective sensors 111-114, optionally taking into account the respective measurement mode. In one example, the calibration data may compensate, for example, an offset of the measurement areas 211, 212 and / or a rotation of the measurement areas 211, 212 relative to each other. The sensor data 121-124 may then be further processed based on the calibration data, in particular features in the respective sensor data may be determined. This means that calibration may generally relate sensor data from different sensors to each other.

[0041] Features in the sensor data 121-124 are then determined in module 132, optionally taking into account calibration data. Module 132 can again be implemented in software. These features can describe specific characteristics of the sensor data. These features can correspond to objects 81-83 in scene 80.

[0042] As described above, the information content of the sensor data may vary depending on the type of sensor 111-114 used. Therefore, the complexity of the feature determination may also vary depending on the type of sensor. For example, for an ultrasonic sensor, the feature determination may include the determination of a single distance value. On the other hand, for a camera, the feature determination may include object detection, for example using machine learning and / or image segmentation.

[0043] Generally, at least one feature may be determined based on a single measurement of the sensor data, respectively. This means that sensor data from multiple time series measurements of the various sensors 111-124 are not combined. If separate measurements are used, a higher refresh rate may be used for downstream sensor fusion.

[0044] Then, a correspondence analysis between the features of the various sensor data 121 - 124 is performed in module 133 . The correspondence analysis is performed taking into account the performance specification data 172 .

[0045] The correspondence analysis is performed in particular between the features of the various sensor data 121-124. This means, for example, that the features determined in the sensor data 121 can be checked for correspondence with the features determined in the sensor data 122, and the features determined in the sensor data 123 can be checked for correspondence with the features determined in the sensor data 124. Correspondingly, the features determined in the sensor data 122 can be checked for correspondence with the features determined in the sensor data 123, and so on.

[0046] exist Figure 2 In some examples, the performance specification data is received by module 171. In some examples, the performance specification data may be permanently stored in the memory, that is, pre-defined. Figure 2 The examples in also illustrate the fact that sometimes the performance specification data may be adapted during the operation of the sensors 111-114. For example, a specific operating state of the sensors 111-114 may be taken into account, etc. For example, it may be beneficial if the measurement mode of the sensors 111-114 can be set.

[0047] Then, sensor fusion can be performed based on the result of the correspondence analysis. For example, if the correspondence analysis shows that there is a good correspondence between the features determined in the various sensor data 121-124, sensor fusion can be easily performed by combining the various sensor data 121-124, that is, by superimposing the various determined features, for example, the features corresponding to the output 141.

[0048] In case a discrepancy is detected during the correspondence analysis - that is, for example, in case a discrepancy exists in the interpretation of the various sensor data 121-124, a scene reconstruction algorithm may be used. In particular, a scene reconstruction algorithm may be used when the result of the correspondence analysis indicates a limited correspondence, which is, however, reasonable within the range of the corresponding performance specification data. If the correspondence is not reasonable within the range of the corresponding performance specification data, the sensor may be considered to be faulty.

[0049] During scene reconstruction, certain typical assumptions can be made about the scene, for example, within the scope of the reconstruction rules. For example, a glass sheet can be inserted into a frame detected by the camera 112. That is, it is often possible to add certain features, especially when they are missing in the corresponding sensor data 121-124, which are plausible within the scope of the performance specification data. For example, in Figure 1 In the example of , glass sheet 82 may be added in conjunction with a set of features determined from sensor data 122 of camera 112 .

[0050] For each scene hypothesis made, it can then be checked whether all sensor data can therefore be declared trustworthy, e.g. module 136. In this case, the performance specification data 172 can again be taken into account. If, for example, a flat glass sheet is reconstructed and added to the frame detected by camera 112, it can be verified that object 83 behind glass sheet 82 is still determined as a feature in the sensor data 122 of camera 112. This generally means that the scene reconstruction algorithm can correct limited correspondences based on predefined reconstruction rules for expected features of the scene. As described above, these reconstruction rules can describe, for example, geometric relationships between specific typical features. In Figure 2 In the example of , reconstruction rules are received from module 135 .

[0051] In some examples, it is possible to train the reconstruction rules on the basis of machine learning. For example, if a particularly reasonable reconstruction of a scene is obtained by using a specific variation of the corresponding rule, this information will likely be used to train a corresponding algorithm that provides the corresponding rule.

[0052] If the analysis in module 136 indicates that the scene reconstruction is successful, sensor fusion can be performed again by superimposing features (including features added by scene reconstruction), corresponding to output 142. In this case, the supplementary scene or specific attributes of the supplementary scene can also be used as prior knowledge to determine the feature 132 by feeding back the corresponding prior data 143.

[0053] The results of the scene reconstruction 134 may also be used to adapt the performance specification data 172. Accordingly, corresponding correction data 144 may be output to the module 171.

[0054] If a reasonable reconstruction of the scene is not possible, warning 149 may be output.

[0055] By means of such techniques, the scene - i.e. the environment - can thus be interpreted robustly. In particular, all available sensor data can be combined and possibly combined with assumptions about typical features during scene reconstruction. This sensor fusion provides a detailed image of the scene. The high level of detail leads to fewer spatial constraints. This can be helpful, for example, especially in connection with the planning of trajectories 91 (compare Figure 1 ). For example, in scenes with objects such as door frames, closely located obstacles such as pillars, etc., the environment can be correctly interpreted, allowing passage.

[0056] Figure 3 is a flow chart of an exemplary method. Figure 3 The method can be implemented on a processor by the latter reading a program code from a memory. Figure 3The method in the example is used to robustly find features in a scene based on multimodal sensor data.

[0057] First, in block 1001, sensor data is received. In this case, sensor data may typically be received from at least two different sensors, i.e., sensors assigned to different sensor types. The various sensors may use different modes to image a common scene. For example, sensor data may be received from a camera, and further sensor data may be received from an ultrasonic sensor.

[0058] Then, one or more features are determined in each case from the sensor data in block 1002. The evaluation in block 1002 can initially be performed separately for the different available sensor data. In particular, calibration data can be taken into account in block 1002. These calibration data can relate the different sensor data to each other. To this end, the calibration data can describe the imaging properties of the individual sensors, such as pose, magnification factor, delay, image refresh rate, etc.

[0059] Then, performance specification data for each sensor is obtained in block 1003. The performance specification data indicates at least one limitation of the corresponding measurement modality related to the ability of the corresponding sensor to image features having specific properties.

[0060] A mutual correspondence analysis of the features is then performed in block 1004, wherein, in this case, corresponding performance specification data are taken into account. This may mean finding corresponding features in the sensor data from different sensors. Thus, the features determined in the different sensor data can be associated or assigned to one another.

[0061] In this case, the performance specification data of the different sensors support the correspondence analysis and allow a more accurate assessment of the differences identified during the correspondence analysis. For example, the performance specification data can determine the boundaries of the overlapping measurement fields of the different sensors. For example, the performance specification data can determine the requirements imposed on the geometry of the object, i.e., for example, the surface orientation or the distance to the sensor, as a result of which the object can be detected as a feature.

[0062] The results of the correspondence analysis are then checked in block 1005. If a correspondence exists, sensor fusion can be performed directly in block 1007. To this end, the various features determined can be superimposed. For example, it can be determined whether a correspondence exists when all features of the sensor data from a sensor correspond to features of other sensor data (within a certain accuracy limit).

[0063] However, if there is no correspondence, then block 1006 may be performed before performing sensor fusion in block 1007. A scene reconstruction algorithm is used in block 1006. The algorithm may be used to reconstruct specific features for which there is no correspondence. In this case, block 1007 may be performed, in particular, when, despite the absence of a correspondence, the lack of correspondence is reasonable within the limits indicated by the performance specification data (otherwise, a warning may be output that the sensor fusion is not performed in the given state). Figure 3 ).

[0064] With regard to the scene reconstruction algorithm, a priori assumptions related to the scene can be used, for example within the scope of the reconstruction rules. For example, the a priori assumptions can include information items related to the field of use, for example, whether it involves an interior scene or an exterior scene, etc. Thus, it is possible to reconstruct more efficiently typical objects that often occur but are only difficult to detect with a specific sensor type. For example, a glass plate in a frame or a characteristic substrate / lining that is highly absorbent or shiny.

[0065] Machine learning methods can also be used as part of scene reconstruction algorithms. For example, corresponding rules describing feature reconstruction can be trained on the basis of machine learning. This enables more robust and efficient reconstruction of the environment.

[0066] The specific results of the scene reconstruction algorithm may be further utilized. For example, the performance specification data may be adjusted based on the results of the scene reconstruction algorithm. Specifically, for example, if it is detected that a specific feature is not detected in specific sensor data, and the feature is reconstructed, the attribute of the feature may be stored in the performance specification data as being undetectable by the corresponding sensor type. For example, machine learning, in particular reinforcement learning, may be used to implement such feedback.

[0067] The results of the scene reconstruction algorithm can also be used to detect systematic faults, for example, based on an increase in the difference frequency.

[0068] It goes without saying that the features of the above-mentioned embodiments and aspects of the present invention can be combined with each other. In particular, these features can be used not only in the described combinations, but also in other combinations or alone without departing from the scope of the invention.

Claims

1. A computer-implemented sensor fusion method, wherein: The method comprises: receiving sensor data (121-124) from a plurality of sensors (111-114), wherein the plurality of sensors (111-114) image a common scene (80) using a plurality of measurement modes, For each sensor of the plurality of sensors (111-114): determining at least one corresponding feature in the respective sensor data (121-124), For each sensor in the plurality of sensors (111-114): obtaining corresponding performance specification data (172), performing a mutual correspondence analysis between features of the sensor data (121-124) taking into account the corresponding performance specification data (172), performing sensor fusion based on the correspondence analysis, and If the correspondence analysis indicates as a result that there is a limited correspondence in the range of the corresponding performance specification data (172), a scene reconstruction algorithm is used before performing the sensor fusion.

2. The method according to claim 1, in, The performance specification data (172) indicates at least one limitation of a corresponding measurement mode of an imaging feature having a specific property.

3. The method according to claim 2, in, The at least one constraint includes a lower threshold for the lateral extent of objects (81-83) in the scene (80).

4. The method according to claim 2, in, The at least one constraint includes a geometric orientation of an object (81-83) in the scene (80), the geometric orientation being based on a pose of a corresponding sensor relative to the object (81-83).

5. The method according to claim 2, in, The at least one constraint includes a reflectivity of a surface of an object (81-83) in the scene (80).

6. The method according to any one of the preceding claims, in, The at least one corresponding feature is respectively determined based on an individual measurement of the sensor data (121-124).

7. The method according to claim 1, in, The scene reconstruction algorithm modifies the limited correspondence according to predefined reconstruction rules of expected features of the scene (80).

8. The method according to claim 7, in, The predefined reconstruction rules are trained based on machine learning.

9. The method according to claim 7 or 8, in, The reconstruction rules are continuously changed in multiple iterations of the scene reconstruction algorithm until the termination criteria are met. Therein, the termination criterion includes the credibility of the result of the scene reconstruction algorithm based on the sensor data (121-124).

10. The method according to claim 1, further comprising: The performance specification data (172) is adjusted according to the result of the scene reconstruction algorithm.

11. The method according to claim 1, in, The scene reconstruction algorithm uses a priori assumptions about the scene (80).

12. The method according to claim 1, in, The at least one corresponding feature is determined for each sensor of the plurality of sensors (111-114) based on respective calibration data, wherein the calibration data describes imaging characteristics of the various sensors.

13. The method according to claim 1, further comprising, The movement of the robot (90) in the scene (80) is controlled based on sensor fusion.

14. A device having at least one processor and a memory, wherein: The at least one processor is configured to load a program code from the memory and execute the program code, and based on the execution of the program code, perform the following steps: receiving sensor data (121-124) from a plurality of sensors (111-114), wherein the plurality of sensors (111-114) image a common scene (80) using a plurality of measurement modes, For each sensor of the plurality of sensors (111-114): determining at least one corresponding feature in the respective sensor data (121-124), For each sensor in the plurality of sensors (111-114): obtaining corresponding performance specification data (172), performing a mutual correspondence analysis between the features of the sensor data (121-124) taking into account the corresponding performance specification data (172), performing sensor fusion based on the correspondence analysis, and If the correspondence analysis indicates as a result that there is a limited correspondence in the range of the corresponding performance specification data (172), a scene reconstruction algorithm is used before performing the sensor fusion.

Images