Method for controlling robotic device

Abstract

According to various embodiments, a method for controlling a robotic device is described, the method having: receiving, from each sensor of a plurality of sensors, a respective sensor data set for the sensor; for each object of a set of objects comprising at least one object, determining, for each of a plurality of different combinations of sensor datasets, a location prediction of the object by sensor data fusion of the sensor datasets in accordance with the combinations of the sensor datasets; for each object of the object set, for each pair of combinations in the plurality of pairs, determining a distance between position predictions determined for the object according to the pair of combinations; delivering the determined distances to a neural network trained to: determine confidence information for the location predictions from the distances between the location predictions for the pair combinations; and controlling the robotic device using one or more of the location predictions, taking into account the confidence information.

Description

Technical Field

[0001] The present disclosure relates to methods for controlling a robotic device. Background Art

[0002] In various applications, good results can be obtained if sensor (data) fusion of different sensors is performed and a robotic device is controlled while using the result of the sensor data fusion. For example, for autonomous driving, sensor data from optical cameras, radar sensors, lidar sensors, and other sensors can be combined to increase the reliability of object detection.

[0003] There are sensor fusion methods based on machine learning and methods based on the Dempster-Shafer theory (DS Evidence Theory). Depending on the architecture used for sensor fusion, each sensor can contribute differently to the final result, more precisely based on the proportion of data that sensor contributes to the total data, the type of sensor, the time of day, weather, object characteristics, etc.

[0004] However, when performing sensor fusion, degradation in the measurement quality of a single sensor itself may significantly affect the final result.

[0005] In order to solve this problem, DE 102018222526 A1, hereinafter referred to as reference 1, describes a hybrid (or multipath) architecture. Here, the data of different sensors are fused in different ways. In this way, redundant identification of the same object is obtained. In this way, if a single sensor fails, one or more fault-free modes (i.e., the combination of sensor data of multiple sensors, in particular including the selection of which sensors' data are combined) are always still available. Different modes use sensor data from different sensors with different proportions (e.g., in the case of three sensors, 70%-30%-0% or 70%-15%-15%, etc.), and the dependence of the results on the sensor data may be different.

[0006] For multipath architectures, poor measurement quality of sensors may affect multiple modalities. Therefore, methods are needed that can determine which modalities can be trusted compared to other modalities, i.e., methods are needed to estimate the credibility of modalities so that the results from these modalities can then be used for reliable control. Summary of the invention

[0007] According to various embodiments, there is provided a method for controlling a robotic device, the method comprising:

[0008] receiving, from each sensor of the plurality of sensors, a corresponding sensor data set for the sensor;

[0009] For each object in the set of objects including at least one object, for each of a plurality of different combinations of sensor data sets, determining a position prediction of the object by performing sensor data fusion on the sensor data sets according to the combination of the sensor data sets;

[0010] For each object of the set of objects, for each pair of combinations in the plurality of pairs, determining a distance between position predictions determined for the object according to the pair of combinations;

[0011] The determined distances are fed to a neural network trained to: determine confidence information for the position predictions for the pairs of combinations based on the distances between the position predictions; and

[0012] One or more of these position predictions are used to control the robotic device taking into account the confidence information.

[0013] If the measurement quality of the sensor is poor and this has an impact on multiple modalities, the above method makes it possible to avoid the impact of the poor measurement quality on the overall result of the sensor (data) fusion or at least keep this impact low. The neural network learns the dependencies between the modalities and is therefore able to recognize failures or degradations in the measurement quality of the sensor and express them in the confidence information.

[0014] The confidence information may be provided, for example, to an object tracking device. The object tracking device may weight the results of the different modalities in combining the results, for example, according to their confidence levels.

[0015] Various embodiments are described below.

[0016] Embodiment 1 is a method for controlling a robot apparatus as described above.

[0017] Embodiment 2 is a method according to embodiment 1, wherein the object set includes multiple objects.

[0018] Thereby, confidence information is determined jointly for a plurality of objects, whereby it can be performed independently of the tracking of the individual objects.

[0019] Embodiment 3 is a method according to embodiment 2, wherein the set of objects has objects in a specified sub-area of ​​the environment of the robotic device detected by the sensors.

[0020] Thus, the confidence information is determined jointly for objects located in a sub-area of ​​the environment of the robot device covered by the sensors. Thereby, for the determination of the confidence information, on the one hand, it is not necessary to track the objects in these areas individually, since the confidence information is determined jointly for these objects, and on the other hand, a high accuracy of the determination (i.e., estimation) of the confidence information can be achieved, since in the sub-area of ​​the environment the fluctuation intensity of the sensor conditions is smaller than in the entire environment. Therefore, if the sub-area is small enough, it is reasonable to assume that each sensor has a similar accuracy for all objects in the sub-area.

[0021] Embodiment 4 is a method according to any one of embodiments 1 to 3, wherein, as input for each pair of the multiple pairs of combinations, the neural network obtains the distance between the position predictions determined for the object according to the pair of combinations and one or more results of object detection with the aid of these sensor data sets, and is trained to determine the confidence information based on the input.

[0022] One or more results of the object detection are, for example, the position, size, category membership (eg, as soft values) and orientation (and at least a portion thereof) of the object. These inputs increase the estimation accuracy of the confidence information.

[0023] Embodiment 5 is a method according to any one of embodiments 1 to 4, wherein the object set includes a plurality of objects, and the neural network is set up so that the neural network is invariant with respect to the arrangement of these objects.

[0024] This ensures that the confidence information is determined in such a way that it is independent of how the objects are numbered.

[0025] Embodiment 6 is a method according to embodiment 5, wherein the neural network has pooling of processing results on different objects.

[0026] In this way, invariance with respect to the arrangement of objects can be easily achieved.

[0027] Embodiment 7 is a method according to any one of embodiments 1 to 6, wherein the neural network is set up to process input data for a variable number of objects, the input data containing, for each of these objects, distances between position predictions for these pairs of combinations.

[0028] This allows the neural network to be trained for a different number of objects and to be used for such objects, i.e., it can be used regardless of how many objects are detected by the sensor. This can be achieved, for example, by the input data for each object being stored in input data elements (e.g., vectors), and the neural network being set up to perform convolutions with respect to these input data elements, since convolutions can be applied to a variable number of such input data elements. After the convolutions, for example, pooling of the convolution results with respect to the objects can be performed, so that a processing result is generated, the dimensionality of which is independent of the number of objects.

[0029] Embodiment 8 is a method according to any one of embodiments 1 to 7, the method comprising: training the neural network by means of supervised learning using training data elements, wherein each training data element has a training input element and a training target output element, and for each pair of combinations, the training input element has a distance between position predictions for one or more objects whose positions are known, wherein, for each of these combinations, the training target output element has a training target output for confidence information, and for each of the one or more objects, the training target output is given by the distance between the position prediction for the object according to the combination and the known position of the object.

[0030] For example, the distances may be averaged with respect to the objects to determine the combined training target output; or confidence information (ie, confidence values) may be determined for each object based on the corresponding distance and the confidence values ​​may be averaged with respect to the objects.

[0031] Embodiment 9 is a robot control device configured to execute the method according to any one of Embodiments 1 to 8.

[0032] Embodiment 10 is a computer program having instructions, which, when executed by a processor, cause the processor to execute the method according to any one of embodiments 1 to 8.

[0033] Embodiment 11 is a computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 8. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In the accompanying drawings, like reference numerals generally refer to the same parts in all the different views. These drawings are not necessarily to scale, with emphasis instead generally being placed on presenting the principles of the invention. In the description below, various aspects are described with reference to the following drawings.

[0035] Figure 1 A vehicle is shown.

[0036] Figure 2 An example of a multipath architecture for multimodal sensor fusion is shown.

[0037] Figure 3 The determination of confidence information of a modality according to specific embodiments is explained.

[0038] Figure 4 The distance between the position estimates of the two modalities and the distance to the true object position of the ground truth are clarified.

[0039] Figure 5 A flow chart is shown which presents a method for controlling a robotic device according to an embodiment. DETAILED DESCRIPTION

[0040] The detailed description below refers to the accompanying drawings, which show, for the purpose of illustration, the specific details and aspects of the present disclosure in which the present invention may be implemented. Other aspects may be used and structural changes, logical changes, and electrical changes may be performed without departing from the scope of protection of the present invention. Different aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0041] Various examples are described in more detail below.

[0042] Figure 1 A vehicle 101 is shown.

[0043] A vehicle 101 , for example a passenger car (PKW) or a truck (LKW), is equipped with a vehicle control device 102 .

[0044] The vehicle control device 102 has data processing components, such as a processor (eg, a CPU (central processing unit)) 103 and a memory 104 for storing control software according to which the vehicle control device 102 operates and data processed by the processor 103 .

[0045] For example, the stored control software has (computer program) commands that, when executed by the processor, cause the processor 103 to implement one or more neural networks 107 .

[0046] The data stored in memory 104 may include, for example, image data captured by one or more cameras 105. The one or more cameras 105 may, for example, take one or more grayscale or color pictures of the surroundings of vehicle 101.

[0047] Vehicle control device 102 can determine based on the image data whether and which objects, for example fixed objects such as traffic signs or road markings or mobile objects such as pedestrians, animals and other vehicles, are present in the environment of vehicle 101, i.e., can perform object detection.

[0048] Then, the vehicle 101 can be controlled according to the result of the object detection by the vehicle control device 102. Thus, the vehicle control device 102 can control the actuator 106 (eg brake) for example, in order to control the speed of the vehicle, for example, in order to brake the vehicle.

[0049] However, in addition to an optical camera 105 (or cameras 105), one or more other sensors 108 may also be provided. Examples thereof are radar sensors, LiDAR sensors, ultrasonic sensors or also thermal imaging cameras.

[0050] These sensors, including cameras, have different advantages, and these advantages should generally be exploited in an appropriate way in order to achieve high reliability, for example of object detection (which may include object recognition). For example, lidar sensors provide 3D point clouds with high resolution and are generally superior to radar sensors in terms of resolution. In this regard, however, radar sensors can penetrate water molecules without attenuation. Therefore, the radar sensor can be used in any type of weather conditions. In addition, for radar sensors, due to the Doppler effect, an estimate of the radial velocity of the detected object can be obtained, which is an additional advantage of radar sensors. On the other hand, optical cameras provide (even) better resolution, which helps in segmentation and classification of objects.

[0051] These sensors (i.e. the sensor data provided by them) can be fused at different levels. For early sensor fusion, the raw data of different sensors are usually fused in order to create an object hypothesis of the environment. For late fusion, each sensor performs a separate prediction and then the results of the individual sensors are fused. In various applications, such as autonomous driving (AD), different levels of fusion are usually required for safety reasons in order to generate redundant predictions of the environment.

[0052] In order to prevent the poor measurement quality of the sensors from affecting the quality of the sensor fusion result, a multipath architecture as described in reference 1 may be used.

[0053] Figure 2 An example of a multipath architecture for multimodal sensor fusion is shown.

[0054] In this example, three sensors (eg, a video sensor V, a radar sensor R, and a lidar sensor L) provide sensor data.

[0055] For each of these sensors, there is a corresponding path 201, 202, 203, and a corresponding object hypothesis 204 is determined from the sensor data of this sensor. Subsequently, feature selection 205 and feature combination are respectively performed, which are based on the object hypothesis of the corresponding path and include sensor data of other sensors depending on the modality. This results in a branch 206 for each modality. The fourth branch at the top, denoted by V+RL, corresponds, for example, to a modality in which the object hypothesis is determined using video data (V) and the feature selection and combination also include sensor data of a radar sensor (R) and a lidar sensor (L).

[0056] Then, for each of these branches 206, for example, classification and regression 207 (for example, for determining the bounding box of the object and its classification) are performed. Then, the results of the classification and regression 207 from the branches 206 can be combined in 208, for example, fused in an object-based manner. It is also possible to simply select the result from one of these branches 206.

[0057] Whether simply selecting the results from these branches 206 or combining these results in a more complex way, it is desired to exclude as much as possible from the determination of the final result (i.e., the result of 208) the result of one of these branches 206 with low accuracy due to poor measurement quality of one of these sensors (for example, due to rain), or at least to give this result a low weight.

[0058] Therefore, according to various embodiments, a method for separately determining confidence levels of multiple modalities (ie, confidence information of the corresponding modalities, ie, results from the corresponding branches 206 ) is provided.

[0059] This method can not only be applied to Figure 2 The multi-path (fusion) architecture shown in FIG. 1 and can be used for any object detection or object tracking system with redundant paths (or branches or routes) where different sensors contribute to these paths.

[0060] According to various embodiments, in particular, an estimation of confidence based on permutation-invariant machine learning (ML) using a distance metric between the modalities (i.e., the results of the modalities), i.e., a method that takes into account the dependencies between the modalities, is provided. For the distance between two modalities, for example, between bounding boxes of an object determined for the modalities, for example, the Mahalbonis distance is used.

[0061] According to various embodiments, the confidence of a modality is a value (eg, between 0 and 1) that indicates how far a location estimate (eg, a bounding box) made according to the modality is from the correct location of the object.

[0062] According to the following embodiments, a scene (eg, the environment of a vehicle) is divided into regions, and it is assumed that the confidence of the modality is the same for all objects in the region. Correspondingly, in the following embodiments, object detection results for all objects in the region are included.

[0063] Figure 3 The determination of confidence information of a modality according to specific embodiments is explained.

[0064] Assumption: for multiple modalities 301, results of object detection are provided respectively, such as (in this example) the position, size, category membership (e.g. as soft values), orientation and in particular the corresponding bounding box 302 of each object of multiple objects (from the area of ​​the scene).

[0065] An input matrix 303 for a neural network 304 is formed from the bounding box 302 .

[0066] Each row of the input matrix 303 is assigned to one of the objects. For this object, this row contains, for each pair of modalities in these modalities 301: the distance (e.g. Mahalanobis distance) between the bounding boxes determined for this object for both modalities from this pair. If the position cannot be estimated for a modality, then for example the maximum value is used as the estimated distance (to the other modality).

[0067] In case there are m modalities, this results in a vector of n2=m*(m-1) / 2 entries per subject.

[0068] Thus, if n1 is the number of objects in the region (which may be variable), the input matrix 303 contains at least one matrix of size n1 x n2. However, for each object, the input matrix 303 may also contain other entries, such as the estimated position of the corresponding object's bounding box for the modalities, the estimated size of the bounding boxes for the modalities, the predicted class of the objects for the modalities, the mean of the estimated position of the object for the modalities, and the variance of the estimated position of the object for the modalities, etc.

[0069] The neural network 304 is trained such that: the neural network provides an output 305 from the input matrix 303, which contains the confidence of each modality (or the result determined for the modality, such as the position prediction), that is, the vector

[0070] (C mod_1 ,C mod_2,…,C mod_n ).

[0071] To train the neural network, a ground truth vector (i.e., target output vector) of training data elements (with known object positions) is generated for this vector

[0072]

[0073] The entries of the ground truth vector for the i-th modality The distances between the position estimate for the modality and the actual object positions are derived, averaged over the objects.

[0074] Figure 4 The distance between the position estimates of the two modalities and the distance to the true object position of the ground truth are clarified.

[0075] The first bounding box 401 is the result of the object detection of the first modality. The second bounding box 402 is the result of the object detection of the second modality. The third bounding box 403 corresponds to the correct object position.

[0076] As entries in the input matrix 303 for the object (ie the corresponding rows) and the pair of modalities (ie the corresponding columns), the distance d between the two bounding boxes 402, 403 of these modalities is used. m1_m2 .

[0077] As the ground truth contribution for the object (these contributions are averaged over all objects), for the i-th two modalities use

[0078]

[0079] (Wherein, before or after applying the exponential function, the average may be calculated with respect to these objects, that is, the average may be calculated with respect to all objects. average or average the distances).

[0080] is equivalent to:

[0081]

[0082] Among them, Figure 4 In the example, d mi_GT is the distance d between the estimated bounding box 401 and the correct bounding box 403 for the first modality m1_GT Or the distance d between the estimated bounding box 402 and the correct bounding box 403 for the second modality m2_GT .

[0083] The above use of an exponential function provides high confidence for small distances and low confidence for large distances. Large distances correspond to low confidence because, with large distances (i.e., large errors in the estimate for a mode), that mode should be less trusted. Confidence value is in the range [0,1].

[0084] As described above, in the above examples, a confidence is determined for each region of the scene, and it is assumed that the confidence is the same for all objects in the region. However, alternatively, the confidence may also be determined for the entire scene, or it may also be determined only for individual objects. Since the accuracy of each sensor for each object depends on the distance of the object from the sensor, and some objects of the scene may be occluded from the sensor while other objects are not, determining a common confidence for all objects of the scene may not provide a good estimate. The highest accuracy can be achieved using a confidence for each individual object, but a common confidence (for example for objects in a region) enables independent object tracking.

[0085] Confidence (C mod_1 ,C mod_2 ,…,C mod_n ) should be independent of the permutation of the rows of the input matrix 303, since this corresponds only to a renumbering of the objects. However, the estimate should not be invariant with respect to the permutation of the columns, since these columns correspond to different attribute or modality pairs.

[0086] In order to achieve the permutation invariance of the neural network 304 with respect to these rows (and thereby achieve equal treatment of these objects), functions such as minimum, maximum, multiplication, addition, etc. on these rows can be inserted into the neural network along the column direction of each column. Alternatively, in the neural network, the same weights can be used for the processing of different rows.

[0087] According to one embodiment, a max-pooling layer is used. For example, the neural network 304 includes a 1D convolution layer that performs a 1D convolution on the row entries of each column and then performs a max-pooling on the obtained (column) vector.

[0088] In summary, according to various embodiments, there is provided a method for Figure 5 The method shown in .

[0089] Figure 5 A flow chart 500 is shown which presents a method for controlling a robotic device according to an embodiment.

[0090] At 501 , from each sensor of a plurality of sensors, a corresponding sensor data set of the sensor is received.

[0091] At 502 , for each object of a set of objects including at least one object, for each of a plurality of different combinations of sensor data sets, a position prediction of the object is determined by performing sensor data fusion on the sensor data sets according to the combination of the sensor data sets.

[0092] At 503 , for each object of the set of objects, for each pair of combinations in the plurality of pairs, a distance between position predictions determined for the object according to the pair is determined.

[0093] At 504, the determined distances are fed to a neural network that is trained to determine confidence information for the position predictions for the pairs of combinations based on the distances between the position predictions.

[0094] At 505, one or more of the position predictions are used to control the robotic device taking into account the confidence information.

[0095] Each combination corresponds to a modality of sensor fusion, that is, Figure 2 The specific combination of sensors contributing to the corresponding branches of the sensor fusion system of the multi-path architecture shown in . The sensor fusion system provides estimates (results) for each modality, such as the location of the object, the size of the object, the orientation of the object, the class membership of the object (e.g., in the form of soft values), etc.

[0096] Figure 5 The method can be performed by one or more computers with one or more data processing units. The term "data processing unit" can be understood as any type of entity capable of processing data or signals. For example, these data or signals can be processed according to at least one (that is, one or more than one) specific function, which is performed by a data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an integrated circuit of a programmable gate array (FPGA), or any combination thereof or be constructed by these. Any other means for implementing the corresponding functions described in more detail herein may also be understood as a data processing unit or a logic circuit device. One or more of the method steps described in detail here can be implemented (e.g., realized) by a data processing unit through one or more specific functions, which are performed by the data processing unit.

[0097] In the above detailed embodiments, an application of autonomous driving is described, but the above method can also be used for other applications in which sensor fusion is performed, and the method is not limited to the above sensors (ie, sensor types).

[0098] Figure 5 The method is generally used for generating control signals for a robotic device. The term "robotic device" may be understood to refer to any technical system having mechanical parts whose movement is controlled, such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system.

[0099] Various embodiments may receive and use sensor signals of various sensors, such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc., for example, to obtain a demonstration of the state of the system (robot and one or more objects) and sensor data of the configuration and scene. The sensor data may be processed. This may include classification of the sensor data or performance of semantic segmentation in terms of the sensor data, for example, to detect the presence of an object (in the environment in which the sensor data was obtained). Embodiments may be used to train machine learning systems and control robots, for example, autonomously controlled by a robot manipulator, to achieve different manipulation tasks in different scenarios. In particular, embodiments can be applied to control and monitor the implementation of manipulation tasks, for example in an assembly line.

[0100] Although specific embodiments are presented and described herein, it is recognized by those skilled in the art that the specific embodiments shown and described may be replaced with a variety of alternative and / or equivalent implementations without departing from the scope of protection of the present invention. The present application should encompass any adjustments or changes to the specific embodiments discussed herein. It is therefore intended that the present invention be limited only by the claims and their equivalents.

Claims

1. A method for controlling a robotic device, the method comprising: receiving, from each sensor of the plurality of sensors, a corresponding sensor data set for the sensor; For each object of the set of objects including at least one object, for each combination of a plurality of different combinations of sensor data sets, determining a position prediction of the object by performing sensor data fusion on the sensor data sets according to the combination of the sensor data sets; For each object of the set of objects, determining, for each pair of combinations in the plurality of pairs, a distance between position predictions determined for the object according to the pair of combinations; delivering the determined distances to a neural network trained to: determine confidence information for the position predictions based on the distances between the position predictions for the pairs of combinations; and One or more of the position predictions are used to control the robotic device taking into account the confidence information.

2. The method according to claim 1, wherein: The object set includes multiple objects.

3. The method according to claim 2, wherein: The set of objects has objects in a specified sub-area of ​​an environment of the robotic device detected by the sensor.

4. The method according to any one of claims 1 to 3, wherein: The neural network obtains as input for each of the plurality of pairs the distance between position predictions determined for the object according to the pair and one or more results of object detection using the sensor data set, and is trained to determine the confidence information based on the input.

5. The method according to any one of claims 1 to 4, wherein: The object set includes a plurality of objects, and the neural network is configured such that the neural network is invariant with respect to an arrangement of the objects.

6. The method according to claim 5, wherein: The neural network has pooling of processing results on different objects.

7. The method according to any one of claims 1 to 6, wherein: The neural network is set up to process input data for a variable number of objects, the input data comprising, for each of the objects, the distances between position predictions for these pair combinations.

8. The method according to claim 1 , comprising: training the neural network by means of supervised learning using training data elements, wherein: Each training data element has a training input element and a training target output element, wherein for each pair of combinations, the training input element has a distance between position predictions for one or more objects whose positions are known, wherein, for each of the combinations, the training target output element has a training target output for the confidence information, and for each of the one or more objects, the training target output is given by the distance between the position prediction for the object according to the combination and the known position of the object. 9 . A robot controller configured to carry out the method according to claim 1 . 10 . A computer program comprising instructions which, when executed by a processor, cause the processor to perform the method according to claim 1 .

11. A computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

Images