Method for probabilistic calculation of poses, computer program and robot with a control device

Abstract

Method for probabilistically calculating poses based on a model of a robot with a linking kinematic chain and an end effector, wherein poses and correlating uncertainties are calculated along the kinematic chain taking into account time-variable and configuration-dependent errors, and calculation results are continuously updated; computer program comprising instructions which, when the program is executed by a computer and / or a control device of a robot, cause the latter to execute such a method; and robot with a control device designed to execute such a method.

Description

[0001] The invention relates to a method for probabilistically calculating poses based on a model of a robot kinematics with a kinematic chain comprising links and an end effector. The invention also relates to a computer program. Furthermore, the invention relates to a robot with a control device.

[0002] The publication “Meyer, Lukas and Strobl, Klaus H. and Triebel, Rudolph (2022) The Probabilistic Robot Kinematics Model and Its Application to Sensor Fusion. In: 2022 IEEE / RSJ International Conference on Intelligent Robots and Systems, IROS 2022. 2022 IEEE / RSJ International Conference on Intelligent Robots and Systems, 23 - 27 Oct 2022, Kyoto, Japan. doi: 10.1109 / IROS47612.2022.9981399 <https: / / doi.org / 10.1109 / IROS47612.2022.<9981399>. ISBN 978 - 166547927 - 1. ISSN 2153 - 0858.”, hereinafter also referred to as [1], concerns a probabilistic robot kinematics model and its application to sensor fusion. In this publication, a method for the kinematic modeling of robots or a robot model is presented that is based on a probabilistic product of exponentials. The method orThe model not only considers a deterministic robot geometry, but also models time-varying and configuration-dependent errors in a probabilistic manner. This method and model allows for the description of errors along the kinematic chain and the calculation of their influence on an end-effector pose. It is proposed to apply this method and model in the context of sensor fusion to correct a manipulator pose for robot systems.

[0003] The method and robot model presented in publication [1] are part of the teachings of the present invention. This publication is therefore incorporated into the present disclosure by reference. In particular, the features contained under points "III. MATHEMATICAL BACKGROUND," "IV. PROBABILISTIC PRODUCT-OF-EXPONENTIALS ROBOT MODEL," and "V. APPLICATION: PROBABILISTIC KINEMATICS IN SENSOR FUSION" are incorporated into the present disclosure.

[0004] The invention is based on the object of structurally and / or functionally improving a method mentioned above. Furthermore, the invention is based on the object of providing a computer program for executing the method. Furthermore, the invention is based on the object of providing a robot with a control device for executing the method.

[0005] The problem is solved by a method having the features of claim 1. Furthermore, the problem is solved by a computer program having the features of claim 13. Furthermore, the problem is solved by a robot having the features of claim 14. Advantageous embodiments and / or further developments are the subject of the subclaims.

[0006] The method is designed for the probabilistic calculation of poses. The method can be designed for calculating poses of a robot. The method can be designed for calculating poses of a dynamic robot. In this context, "probabilistic" means in particular that uncertainties are taken into account when calculating the poses. Uncertainties can arise in particular due to elastic deformation, decalibration, aging, and / or wear. In this context, "pose" refers in particular to a combination of position and orientation. The poses are calculated based on a model of a robot. The model of the robot has a kinematic chain with links and an end effector. The robot described using the model can be a conventional robot with discrete links and joints. The robot described using the model can be a continuum robot with a joint-free moving structure.To model the kinematic chain, the joint-free structure can be described with virtual links and / or virtual joints. The model of the robot and / or the kinematic chain can have a base. The model of the kinematic chain can have a start and an end. The base can form the start of the kinematic chain. The end effector can form the end of the kinematic chain. The kinematic chain can be an open kinematic chain. The model of the robot and / or the kinematic chain can have joints. The joints can be effective between the base and a link, between two links, and / or between a link and the end effector. The joints can be adjustable. The joints can be adjustable using joint drives. Adjustment parameters can be assigned to the joints and / or joint drives. The model can be a mathematical model.The model can be designed to describe the robot, in particular the kinematic chain. Poses and correlating uncertainties are calculated. Poses of the robot, the kinematic chain, the links, and / or the end effector can be calculated. "Correlating" in this context describes in particular that the calculated poses and the calculated uncertainties are interrelated. "Uncertainty" in this context refers in particular to an uncertainty regarding a correspondence between a calculated pose and a real pose. The uncertainties can be described using measures of dispersion, in particular variances. The poses and correlating uncertainties are calculated along the kinematic chain. The poses and correlating uncertainties can be calculated starting from the beginning of the kinematic chain towards the end of the kinematic chain.The poses and correlating uncertainties can be calculated for each link and / or the end effector. Each calculated pose and correlating uncertainty can take into account the calculated poses and correlating uncertainties of the preceding links in the kinematic chain. The poses and correlating uncertainties are calculated taking into account time-varying and configuration-dependent errors. "Configuration" in this context refers in particular to joint settings. Calculation results are continuously updated. "Continuous" in this context means in particular that the calculation results are updated repeatedly, continuously, continuously, and / or consecutively. The calculation results can be continuously updated for at least one specific period of time. The at least one period of time can be a time period before and / or during robot operation.The time period can be within a preparation time or encompass the entire preparation time. The time period can be within a robot operating time or extend over the entire robot operating time. The calculation results can be updated before and / or during robot operation.

[0007] The calculation results can be considered as virtual sensor measurements. In this context, this means, in particular, that the calculated poses and correlated uncertainties can be considered according to the measured values ​​of real sensors.

[0008] Measurement values ​​from at least one sensor can be taken into account. The at least one sensor can be a real sensor. The at least one sensor can be an image sensor and / or a force sensor. The measurement values ​​from the at least one sensor can be taken into account to correct the calculation results. The measurement values ​​from the at least one sensor can be taken into account to update the model. The measurement values ​​from the at least one sensor can be taken into account to obtain a corrected end effector pose.

[0009] The calculation results considered as virtual sensor measurements and the measurement values ​​of at least one sensor can be fused. The calculation results considered as virtual sensor measurements and the measurement values ​​of at least one sensor can be fused in a weighted manner. The calculation results considered as virtual sensor measurements and the measurement values ​​of at least one sensor can be fused in a weighted manner using covariances.

[0010] A state estimator can be used to calculate more precise poses. An extended Kalman filter and / or an SMC filter (SMC: Sequential Monte Carlo Methods) can be used for state estimation.

[0011] The calculated poses can be compared with target poses. The target poses can also be referred to as nominal poses. The comparison between the calculated pose and the target pose can be performed for each of the limbs and / or the end effector. Comparison results can be obtained from the comparisons. Taking the comparison results into account, an end effector pose can be corrected. The end effector pose can be corrected taking all comparison results into account. A calculated end effector pose and / or a real end effector pose can be corrected. The end effector pose can be corrected using inverse kinematics. To correct the end effector pose, the comparison results can be transferred to the joints using inverse kinematics. The joint drives can be controlled based on the comparison results.The calculation results can be converted to the joints at a differential geometry and / or velocity level using a pseudoinverse of a Jacobian matrix. The pseudoinverse can be weighted based on the weighted uncertainties.

[0012] At least one of the correlated uncertainties can be selected, explicitly modeled, and parameterized. Using the at least one explicitly modeled uncertainty, parameters can be calculated. The parameters can be stored as additional state variables for state estimation and actively estimated.

[0013] The calculation results can be used to calibrate a robot. The calculation results can be used in an offline calibration process. Measurement values ​​from at least one sensor can be recorded and processed together with the calculation results from multiple configurations.

[0014] The kinematic chain can be extended by another element on the end effector side. The previous end effector can become a link, and the additional element can become a new end effector. The new end effector can be included in the calculation of the poses and correlating uncertainties. The calculation results can then also include a pose and a correlating uncertainty of the additional element. The calculation results relating to the additional element can be corrected as described, and / or a pose of the additional element can be corrected. For example, the calculation results can include a grasped object and a grasping uncertainty, and a pose of the grasped object can be corrected.

[0015] Limits can be defined taking the calculation results into account. The limits can be defined using the covariance matrix. 6-dimensional limits can be defined using entries in the covariance matrix. For example, a distance from an expected value based on a standard deviation can be defined. The limits can be used for passive monitoring, introspection, and / or diagnosis. Reaching and / or exceeding the defined limits can be monitored. When a defined limit is reached and / or exceeded, a predetermined reaction can be triggered. For example, an application can be stopped and / or a process can be adjusted.

[0016] The method can be implemented in a computer program. The method can be a computer-implemented method. The computer program includes instructions that, when executed by a computer and / or a control device of a robot, cause the robot to execute the method.

[0017] The robot can be a robot described using the model. The robot can be a real robot. The robot can be a conventional robot with discrete limbs and joints. The robot can be an industrial robot, a high-precision robot, an agricultural robot, an agricultural or construction machine, a space robot, a medical robot, a care robot, an assistance robot, a household robot, an elastic robot, an autonomous robot, a mobile robot, a humanoid robot, a service robot, and / or a personal robot. The robot can have a base, limbs, joints, and an end effector. The base, limbs, joints, and / or end effector can form a kinematic chain. The joints can be adjustable using joint drives. The end effector can be a gripper and / or a tool, such as a shovel, scientific instrument, or welding head.The robot can be a continuum robot with a jointless movable structure.

[0018] The robot has a control device. The control device is designed to carry out the method. The control device can have a computer or be formed with the aid of a computer. The control device and / or the computer are / is can / can be designed and / or arranged to execute the computer program. The control device and / or the computer can have a processor, a main memory, a data memory and / or a signal interface. The robot can have at least one sensor. The at least one sensor can be an optical sensor. The at least one sensor can be a laser sensor. The at least one sensor can be an image sensor. The robot can have a camera with at least one image sensor. The camera can be arranged on the base. The camera can be designed and / or arranged to detect the manipulator, the end effector and / or another element.The at least one sensor can be a force transducer. The force transducer can be arranged on the end effector. The force transducer can be designed and / or arranged to detect a gripping force. The at least one sensor, on the one hand, and the control device and / or the computer, on the other hand, can be connected to one another in a signal-conducting manner. The control device and / or the computer, on the one hand, and the joint drives, on the other hand, can be connected to one another in a signal-conducting manner.

[0019] In summary, and in other words, the invention thus provides, among other things, a method for considering configuration-dependent robot inaccuracies in the application. The method may include the concept of sensor data fusion of probabilistically weighted measured values ​​from different sensors and the resulting improvement of the robot application. The method may include the use of the probabilistic information about the robot's uncertainties obtained using the method from [1].

[0020] The starting point for implementing the method can be a holistic modeling of uncertainties in the robot kinematics and their mutual, configuration-dependent impact in a probabilistic manner. As an "output," a pose with the associated uncertainty estimate in the form of a covariance matrix can be obtained for all structural elements or other defined reference systems, such as links and end effectors, along the robot kinematics.

[0021] The probabilistic information obtained with the method from [1] can be updated during a robot application run to obtain configuration-dependent knowledge about a robot state or the effect of kinematic errors. For example, an end-effector pose and associated uncertainty can be continuously calculated during a robot arm movement.

[0022] To reduce kinematic measurement uncertainties, measurements from other sensors can be continuously incorporated. These could, for example, be estimated position data from cameras. If current measurements are available from all considered sensors, they can be merged.

[0023] To obtain a continuous measurement of the end-effector's position corrected by additional sensor information, the robot kinematics can be considered as a probabilistic sensor measurement and fused with the additional sensors (e.g., cameras) based on the covariances, and the more precise position of the robot elements can be calculated using a state estimator. For state estimation, an Extended Kalman Filter or a Particle Filter, for example, can be used. The result is a continuously updated pose estimate and the associated uncertainty.

[0024] Not only the case of a static robot can be considered, but probabilistic kinematics can be used as an additional sensor source and not only as an initial measurement value.

[0025] To actively correct the position of the end effector, the calculated difference between the target pose and the estimated pose of the individual robot elements (e.g., the end effector) can be used. For this purpose, the pose difference can be transferred to the robot joints using inverse kinematics. The resulting differences in the joint angles can be commanded as motor speeds. If in the previous sensor fusion step not only a pose is estimated but also its uncertainty, the latter can also be used here. At the differential geometry / velocity level, poses can be converted to the robot joints using the pseudoinverse of the Jacobian matrix. The pseudoinverse can be weighted based on the calculated uncertainty and thus bring about a correction primarily in those dimensions in which the sensor fusion results exhibit a particularly low uncertainty.

[0026] To actively estimate parameters that model the sources of measurement uncertainty, some of the most important measurement uncertainties in robot kinematics can be explicitly modeled and parameterized. These parameters can be stored as additional state variables in the state estimation and actively estimated during robot application.

[0027] To improve existing offline calibration procedures, these parameters can be estimated in an offline calibration procedure, where sensor measurements (e.g., from cameras) are recorded and processed together with the probabilistic poses of many different robot configurations. Furthermore, the probabilistic poses of the robot elements can extend and improve classic robot calibration procedures.

[0028] To robustly describe the interaction of a robot with a grasped object as an additional, uncertain, kinematic connection in the robot structure, when the robot grasps an object with its end effector, this can be viewed as an extension of the robot's own kinematic chain, i.e., as an extension of the arm. The object itself can become the new end effector. The uncertainty of the object's pose is then composed of the uncertainty of the actual robot arm and the uncertainty in the object's grip. This can also be described using the mathematical methods from the publication [1]. The combined uncertainty of the object's pose can then be corrected as described above.

[0029] To perform an introspection of the system, which describes a pose deviation outside of permissible parameters, the modeling from publication [1] can also be used to diagnose the robot system. The probabilistic description allows for the definition of 6-dimensional limits based on the entries in the covariance matrix, for example, a distance from the expected value based on the standard deviation. If a measurement system detects that the robot system is operating outside these limits, the application can be stopped for safety reasons or the process can be adjusted.

[0030] The invention can increase the accuracy of robot manipulation, application safety, reliability, and application performance. In particular, it can increase speed and / or reduce the probability of failure.

[0031] The method is specifically designed to measure and correct the end effector pose, but can also consider all other reference systems in the robot, for example the pose of a segment or link in the middle of the kinematic structure.

[0032] The invention is characterized in particular by a holistic, probabilistic modeling of measurement uncertainties in robot kinematics, a consideration of the configuration dependence of measurement uncertainties and a continuous consideration of the kinematics and the associated measurement uncertainty in the application.

[0033] Because the directions of kinematic measurement uncertainties are more precisely described and differentiated, a variety of sensors can be used to refine pose estimation, even those that were previously unavailable. In particular, it is possible to use low-dimensional measurements to refine the 6D pose: for example, 2D camera information or 1D contact information.

[0034] The execution can be performed with all robot systems that i) interact with the environment, objects in the environment or other robots, and ii) have additional measurement sensors.

[0035] The following describes exemplary embodiments of the invention for a mobile robot comprising a base, a manipulator designed as a robot arm, and a gripper as the end effector. A camera system with an image sensor is arranged on the base to observe the movements of the robot arm and detect objects. As an exemplary task, the robot is to grasp a shovel with a gripping adapter and then take a stone from the ground in front of the robot as a sample.

[0036] The relative position of the objects to be grasped to the camera's coordinate system is determined using established image processing algorithms, such as "object detection" or "object tracking" algorithms.

[0037] The robot has a kinematic chain with several uncertainties: The geometric information regarding a camera mount is inaccurate; the base of the robot arm is mounted on a metal plate made of thin material. This causes deflection when the arm is extended. Additionally, inaccuracies and elasticities exist in the robot arm's joints.

[0038] Since the objects to be gripped are registered in the camera coordinate system, this results in a kinematic chain from the camera coordinate system, via the base of the robot arm, along the individual segments or links to the gripper.

[0039] Since individual uncertainties propagate along this chain, it cannot initially be guaranteed that the gripper is correctly positioned relative to the object in order to successfully grasp it. The invention addresses this point by reducing the kinematic measurement uncertainties and ensuring correct positioning.

[0040] The listed uncertainties are modelled using the method according to [1] as “probabilistic robot kinematics”, where the probabilistic end-effector pose is described by the nominal pose with an additional covariance matrix, which describes the accumulated measurement uncertainty as a result of the combined individual uncertainties.

[0041] The camera system measures the position of the gripper or end effector using existing image processing methods, for example, tag-based methods such as April tags. For this purpose, an April tag is attached to the gripper. The measurement result can be either the 6D end effector pose or just the 2D image coordinates of the April tag. The method described here, improved over other methods, can utilize a wide variety of sensor information, including low-dimensional ones.

[0042] A force sensor on the gripper makes it possible to register contact with the object to be gripped.

[0043] As the robot moves toward the object, a continuous calculation of the robot pose is performed using camera measurements. The 2D camera data is fused with the calculation results, the poses calculated taking into account time-varying and configuration-dependent errors, and correlating uncertainties. The covariance matrix of the robot kinematics describes the error correlation between the six pose dimensions of the end effector, allowing the 2D measurement to correct the overall pose in all six dimensions. This results in a continuous measurement of the end effector's position, corrected by additional sensor information. The 2D measurement described here can correct the position in all three dimensions, but there is a possibility that the rotation error will not be substantially corrected due to ambiguities.However, this depends on the configuration of the robot, for example in a singularity.

[0044] If contact between the gripper and the object is detected, the 1D contact information is also fused. This provides a continuous measurement of the end effector's position, corrected by additional sensor information.

[0045] The elastic deflection of the plate is explicitly modeled as a function of the arm's extension using a parameter. This parameter is included as a state variable in the state estimator. This actively estimates a parameter that models the sources of measurement uncertainty.

[0046] Using the pose estimation result or the calculation results, a difference between a nominal arm pose and an estimated arm pose is calculated. A correction signal is calculated from this difference, which corrects the arm pose to approximate the nominal arm pose. The correction signal is transmitted to the joint motors as a velocity command using differential kinematics, for which the pseudoinverse of the "Jacobian" robot kinematics is used. The covariance information of the state estimation is used to weight the pseudoinverse. This allows the position of the end effector to be actively corrected.

[0047] If the grasping of an object (in this case, the shovel) is successful, the kinematic chain is updated: The shovel is added to the kinematic chain as a new end effector, with the gripper as the immediate parent element. The uncertainty of the grip is modeled as a further, relative covariance. The uncertainty of the new end effector (the shovel) can thus be efficiently described using the method from [1] as a combination of the previous gripper uncertainty and the new relative uncertainty of the grip. This allows a robust description of the robot's interaction with the grasped object.

[0048] The pose error of the new end effector can be measured and corrected in the same way, using a continuous measurement of the end effector position corrected by additional sensor information and an active correction.

[0049] The listed steps will be applied analogously to the interaction of the robot with the stone to be collected.

[0050] In addition to the active correction of the kinematic error, passive monitoring of the system takes place. The covariance matrix of the end effector describes the main propagation directions of the measurement uncertainty in a probabilistic manner. Limit values ​​can thus be defined based on the covariance values. In this application example, the robot should stop moving as soon as the error in one dimension exceeds the limit of two standard deviations. This is monitored by the camera system for all observable dimensions. The limit value can be exceeded if, for example, the load of the grasped object is higher than expected and thus the deflection of the system is significantly higher. Alternatively, this can indicate a decalibration event. This can be used to introspect the system to determine whether a pose deviation lies outside of permissible parameters.

[0051] An application of the method according to the invention is particularly advantageous when system-intrinsic accuracy is insufficient for an application and modeling of the uncertainties is necessary. Since the method is not dependent on the accuracy requirements of the application, but only on the accuracy of the measuring systems used, the methodology of the method according to the invention can be applied scaling with the measurement accuracy. This can be just as relevant in the high-precision range, for example in applications in industrial robotics, as in applications with lower accuracy requirements, for example in care robotics. In general, the method according to the invention can cover all areas in which kinematic structures of linked elements have a position inaccuracy that is corrected by external sensors.Among others, the following areas of application are possible: industrial robotics, high-precision robotics, agricultural robotics, agricultural and construction machinery (e.g. excavators), space robotics, medical robotics, care robotics, assistance robotics, household robotics, elastic robots.

[0052] "May" refers in particular to optional features of the invention. Accordingly, there are also further developments and / or embodiments of the invention that additionally or alternatively comprise the respective feature(s).

[0053] If necessary, features can also be selected from the combinations of features disclosed in the description and used alone or in combination with other features to define the subject matter of the claim, dissolving any structural and / or functional connection that may exist between the features. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents listed by the applicant has been automatically generated and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited Non-Patent Literature

[0000] Meyer, Lukas and Strobl, Klaus H. and Triebel, Rudolph (2022) The Probabilistic Robot Kinematics Model and Its Application to Sensor Fusion. In: 2022 IEEE / RSJ International Conference on Intelligent Robots and Systems, IROS 2022. 2022 IEEE / RSJ International Conference on Intelligent Robots and Systems, 23-27 Oct 2022, Kyoto, Japan. doi: 10.1109 / IROS47612.2022.9981399 <https: / / doi.org / 10.1109 / IROS47612.2022.<9981399>. ISBN 978-166547927-1. ISSN 2153-0858

[0002]

Claims

[1] Method for the probabilistic calculation of poses based on a model of a robot with a kinematic chain comprising links and an end effector, whereby poses and correlating uncertainties are calculated along the kinematic chain, taking into account time-varying and configuration-dependent errors, and calculation results are continuously updated. [2] Method according to claim 1, characterized by that the calculation results are considered as virtual sensor measurements. [3] Method according to at least one of the preceding claims, characterized by that measured values ​​from at least one sensor are taken into account. [4] Method according to claims 2 and 3, characterized by that the calculation results considered as virtual sensor measured values ​​and the measured values ​​of at least one sensor are fused. [5] Method according to claim 4, characterized bythat the calculation results considered as virtual sensor measured values ​​and the measured values ​​of at least one sensor are fused in a weighted manner. [6] Method according to at least one of the preceding claims, characterized by that precise poses are calculated using a state estimator. [7] Method according to at least one of the preceding claims, characterized by that the calculated poses are compared with target poses and an end effector pose is corrected taking into account the comparison results. [8] Method according to claim 7, characterized by that the end effector pose is corrected using inverse kinematics. [9] Method according to at least one of the preceding claims, characterized by that at least one of the correlating uncertainties is selected, explicitly modeled and parameterized. [10] Method according to at least one of the preceding claims, characterized bythat the calculation results are used to calibrate a robot. [11] Method according to at least one of the preceding claims, characterized by that the kinematic chain is extended by another element on the end effector side, whereby the previous end effector becomes a link and the further element becomes a new end effector, and the new end effector is included in the calculation of the poses and correlating uncertainties. [12] Method according to at least one of the preceding claims, characterized by that, taking into account the calculation results, defined limit values ​​are used for passive monitoring. [13] Computer program, characterized by that the computer program comprises instructions which, when the program is executed by a computer and / or a control device of a robot, cause the robot to carry out a method according to at least one of claims 1 to 12. [14] Robot with a control device, characterized by that the control device is designed to carry out a method according to at least one of claims 1 to 12.