Method for calibrating a multi-axis robotic kinematics of a medical visualization system and medical visualization system

Abstract

The invention relates to a method for calibrating a multi-axis robotic kinematics (2) of a medical visualization system (1), wherein the robotic kinematics (2) is described by means of a kinematic model (10), comprising: acquiring and / or determining a scale, alternatively a) acquiring at least two images (11) of an object (20) arranged in the acquiring area (9) at different magnification levels and determining a magnification center (19), b) acquiring a set of further images (12), wherein the further images (12) are acquired at different positions of a selected axis (15-6) of the robotic kinematics (2), c) at least once: varying a position of at least one other axis (15-x) of the robotic kinematics (2) and repeating step b);Determining a set of distance measures (13), each based on the acquired further images (12), wherein the respective magnification center (19) forms at least one reference point for the distance measure (13), adjusting the kinematic model (10) by changing parameters so that a deviation between certain at least one distance measure (13) and a distance measure (13) calculated using the kinematic model (10) is minimized or minimized.

Description

[0001] The invention relates to a method for calibrating a multi-axis robotic kinematics of a medical visualization system and to a medical visualization system.

[0002] Modern medical visualization systems, such as surgical microscopes, can have robotic kinematics, at the end of which at least one camera is arranged, which can be moved using the robotic kinematics. The robotic kinematics particularly supports a movement of the camera manually initiated by a user or carries out this movement automatically. In order to be able to control a target pose of the robotic kinematics and thus of the camera with high precision in space, the robotic kinematics is described in the form of a kinematic model. The accuracy of the kinematic model is critical for the accuracy of an (absolute) pose of the robotic kinematics and thus for approaching a target pose without a control loop. Individual calibration of the robotic kinematics by adjusting parameters of the kinematic model is therefore desirable.

[0003] Robotic kinematics are often described using the Denavit-Hartenberg model (DH model). Here, each joint is modeled using four parameters. Forward kinematics can be used to calculate the location of an end effector or a tool center point (TCP) in relation to a robot base's coordinate system. Inverse kinematics, on the other hand, is used to calculate which positions or angles must be set at the joints or axes to achieve a specified pose in space with the end effector or TCP. How accurately a calculated pose corresponds to reality depends on how precisely the aforementioned parameters of the kinematic model are known. Manufacturing tolerances of manufactured parts and assembly accuracies (e.g., reproducibility, mechanical stops, etc.)) determine, for example, the exact position and orientation of joints or their encoder sensors for reading a joint angle. Nominal models are therefore not sufficient for high levels of accuracy. The models or model parameters are determined individually using kinematic calibration procedures.

[0004] Known calibration methods use external tracking systems that detect and track the movement of a marker attached to the robot's kinematics. However, these methods require expensive tracking systems and thus a complex infrastructure to track the robot's kinematics within the entire workspace. For calibrations in the field (e.g., on-site in hospitals), this also involves time and additional costs.

[0005] Alternatively, it is known to use the internal cameras of a surgical microscope to calibrate a tripod. Such a method is described, for example, in Abdullah Aamir Hayat et al., "A geometric approach for kinematic identification of an industrial robot using a monocular camera," Robotics and Computer-Integrated Manufacturing, Vol. 57, June 2019, pp. 329-346.

[0006] The invention is based on the object of improving a method for calibrating a multi-axis robotic kinematics of a medical visualization system and a medical visualization system.

[0007] The object is achieved according to the invention by a method having the features of patent claim 1 and a medical visualization system having the features of patent claim 11. Furthermore, the object is also achieved according to the invention by a method having the features of patent claim 12 and a medical visualization system having the features of patent claim 13. Advantageous embodiments of the invention emerge from the subclaims.

[0008] One of the basic ideas of the invention is to detect or determine and / or establish a reference point as a reference point and, based on this, to determine a distance measure in detected images. In one alternative (referred to as alternative i), the reference point is formed by a magnification center of a magnification lens of at least one camera. In another alternative (referred to as alternative ii), a distinguished point with reference to an image sensor of the at least one camera is determined and / or established as the reference point. Alternative ii) can be used in particular when no mechanical magnification lens is present, but magnification occurs exclusively digitally. The distance measure is determined with the aid of a movement of a selected axis by detecting images (hereinafter referred to as "further images") at several positions on the selected axis.As a result, a set of captured additional images is available for a pose of the robotic kinematics, from which the distance measure can be determined. According to a first aspect of the invention, this is repeated at least once, wherein beforehand a position of at least one other axis of the robotic kinematics is varied and with the varied position a set of additional images is again captured, while a position of the selected axis is changed. In particular, for each variant of the position of the at least one other axis according to the first aspect, a set of additional images is captured and a respective distance measure is determined, such that a distance measure is available for each variant. In this way, a set of distance measures for different positions of the at least one other axis can be provided. According to a second aspect of the invention, in particular only one set of additional images is captured overall.However, after varying the position of at least one other axis, at least one additional image is acquired (and in particular, not a set of further images). In particular, according to the second aspect, no further movement of the selected axis occurs, i.e., at least one additional image is acquired at a position of the selected axis.

[0009] In particular, it is intended that the position of other axes is varied so that the largest possible number of distance measurements are available for different poses of the robot kinematics.

[0010] According to the first aspect, the kinematic model is adapted based on the set of distance measures by changing the parameters (which can also be referred to as "fitting"), so that a deviation between the at least one distance measure determined for each set and a corresponding distance measure calculated for each variation using the kinematic model is or will be minimized. In other words, for each of the poses of the robotic kinematics for which a distance measure is also available, precisely this distance measure is calculated based on the kinematic model. For this purpose, the values ​​for the positions (e.g. angles) of the axes, recorded for the respective pose by means of suitable sensors on the axes (e.g. joints) of the robotic kinematics, are fed to the kinematic model.For a fixed position of at least one other axis, the respective distance measurement is calculated using the kinematic model by changing the position of the selected axis and compared with the determined distance measurement. By varying the parameters of the kinematic model, a deviation between the respectively calculated value and the determined value for the at least one distance measurement can be minimized. The result is an adapted kinematic model that better describes the actual state of the robot kinematics. Based on this, it can be provided, for example, to determine and provide offsets for the sensor values. Furthermore, other parameters can also be adjusted, such as the length specifications of the axes.

[0011] According to the second aspect of the invention, the kinematic model is adapted by changing the parameters (which can also be referred to as “fitting”) on the basis of the at least one distance measure determined for the set of acquired further images and the at least one distance measure determined on the basis of the at least one additional image, such that a deviation between the at least one distance measure determined for the set and the at least one additional image and a corresponding distance measure calculated for each variation using the kinematic model is or will be minimized. In other words, for each of the poses of the robot kinematics for which a distance measure is also available, precisely this distance measure is calculated on the basis of the kinematic model. For this purpose, the kinematic model is assigned the values ​​determined for the respective pose by means of suitable sensors on the axes (e.g.The values ​​for the positions (e.g., angles) of the axes recorded by the robot's kinematics (e.g., joints) are fed into the model. By varying the parameters of the kinematic model, a deviation between the calculated value and the determined value for at least one distance measurement can be minimized. The result is an adapted kinematic model that better describes the actual state of the robot's kinematics. Based on this, it can be provided, for example, to determine and provide offsets for the sensor values. Furthermore, other parameters can also be adjusted, such as the length specifications of the axes.

[0012] In particular, according to the first aspect, a method for calibrating a multi-axis robotic kinematics of a medical visualization system is provided, wherein the medical visualization system has at least one camera at the end of the robotic kinematics, and wherein the robotic kinematics is described by means of a kinematic model, comprising: detecting and / or determining a scale with respect to a detection range of the at least one camera, a) i) capturing at least two images of an object arranged in the detection area at different magnification levels of a magnification optics of the at least one camera and determining a magnification center of the magnification optics based on the captured images, or ii) determining and / or setting a distinguished point with respect to an image sensor of the at least one camera, b) capturing a set of further images, wherein the further images are captured at different positions of a selected axis of the robot kinematics, c) at least once, in particular several times: varying a position of at least one other axis of the robot kinematics and repeating step b);

[0013] Determining a set of distance measures, each comprising at least one distance measure for each of the sets of the acquired further images, each starting from the acquired further images, wherein the respective magnification center or the distinguished point forms at least one reference point for the distance measure, adapting the kinematic model by changing parameters so that a deviation between the at least one distance measure determined for the sets and a distance measure corresponding thereto calculated for each variation by means of the kinematic model is or will be minimized.

[0014] Furthermore, according to the first aspect, a medical visualization system is provided, comprising a multi-axis robotic kinematics, at least one camera arranged at the end of the robotic kinematics, and a control device, wherein the control device is configured to describe the robotic kinematics by means of a kinematic model, and to carry out a calibration of the robotic kinematics: to detect and / or determine a scale with respect to a detection range, a) i) to capture at least two images of an object arranged in the detection area at different magnification levels of a magnification optics of the at least one camera and to determine a magnification center of the magnification optics based on the captured images, or ii) to determine and / or establish a distinguished point with respect to an image sensor of the at least one camera, b) to acquire a set of further images, wherein the further images are acquired at different positions of a selected axis of the robot kinematics, c) at least once, in particular several times: to vary a position of at least one other axis of the robot kinematics and to repeat step b); to determine a set of distance measurements, comprising at least one distance measurement for each of the sets of the acquired further images, in each case starting from the acquired further images, wherein the respective magnification center or the designated point forms at least one reference point for the distance measurement, to adapt the kinematic model by changing parameters so that a deviation between the at least one distance measurement determined for the sets in each case and a distance measurement corresponding thereto calculated for each variation by means of the kinematic model is or will be minimized.

[0015] In step c), it can optionally be provided that step a) is also repeated. In general, however, it is particularly possible to carry out the method by executing step a) only once. In alternative i), it is particularly assumed that the magnification center with respect to an image sensor of the at least one camera remains the same even after varying the position of at least one other axis. In alternative ii), it is particularly assumed that the designated point only needs to be determined once, since a reference to the image sensor remains the same even after varying the position of at least one other axis.

[0016] According to a second aspect, in particular, a method for calibrating a multi-axis robotic kinematics of a medical visualization system is provided, wherein the medical visualization system has at least one camera at the end of the robotic kinematics, and wherein the robotic kinematics is described by means of a kinematic model, comprising: detecting and / or determining a scale with respect to a detection range of the at least one camera, a) i) capturing at least two images of an object arranged in the detection area at different magnification levels of a magnification optics of the at least one camera and determining a magnification center of the magnification optics based on the captured images, or ii) determining and / or setting a distinguished point with respect to an image sensor of the at least one camera, b) capturing a set of further images, wherein the further images are captured at different positions of a selected axis of the robot kinematics, c) at least once, in particular several times: varying a position of at least one other axis of the robot kinematics and capturing at least one additional image;

[0017] Determining distance measures, comprising at least one distance measure for the set of acquired further images, starting from the acquired further images, and for the acquired at least one additional image, wherein the magnification center or the distinguished point forms at least one reference point for the distance measure,

[0018] Adapting the kinematic model by changing parameters so that a deviation between the at least one distance measure determined for the set and the at least one additional mapping and a corresponding distance measure calculated for each variation using the kinematic model is or will be minimized.

[0019] Furthermore, according to the second aspect, a medical visualization system is provided, comprising a multi-axis robotic kinematics, at least one camera arranged at the end of the robotic kinematics, and a control device, wherein the control device is configured to describe the robotic kinematics by means of a kinematic model, and to carry out a calibration of the robotic kinematics: to detect and / or determine a scale with respect to a detection range, a) i) to capture at least two images of an object arranged in the detection area at different magnification levels of a magnification optics of the at least one camera and to determine a magnification center of the magnification optics based on the captured images, or ii) to determine and / or establish a distinguished point with respect to an image sensor of the at least one camera, b) to acquire a set of further images, wherein the further images are acquired at different positions of a selected axis of the robot kinematics, c) at least once, in particular several times: to vary a position of at least one other axis of the robot kinematics and to capture at least one additional image;

[0020] Determining distance measures, comprising at least one distance measure for the set of acquired further images based on the acquired further images and for the acquired at least one additional image, wherein the magnification center or the distinguished point forms at least one reference point for the distance measure, to adapt the kinematic model by changing the parameters so that a deviation between the at least one distance measure determined for the set and the at least one additional mapping and a corresponding distance measure calculated for each variation using the kinematic model is or will be minimized.

[0021] The method and the medical visualization system allow the robotic kinematics to be calibrated without the need for an additional tracking system. Calibration of the robotic kinematics in the field (e.g., in a hospital) can thus be performed more easily using at least one camera of a surgical microscope. This saves effort and costs.

[0022] The selected axis can be a linear axis or a rotational axis. For a linear axis, the at least one distance measurement is determined, in particular, based on a track defined by the moving magnification center or the designated point.

[0023] Robotic kinematics can be described, for example, using the Denavit-Hartenberg model (DH model) mentioned above. Here, each joint is modeled using four parameters. Forward kinematics can be used to calculate the location of an end effector or a tool center point (TCP) in relation to a coordinate system of a robotic kinematics basis. Inverse kinematics is used to calculate which positions or angles must be set at the joints or axes in order to achieve a specified pose in space with the end effector or the TCP. At least one camera is arranged at the end of the robotic kinematics, in particular at an end effector. In forward kinematics, the positions of the axes (linear position or angle), detected in particular by a sensor, are fed to the kinematic model.However, these values ​​represent nominal values ​​provided by the sensors on the axes and may therefore deviate from reality. Using the kinematic model, the pose of the end effector and, if the relationship between the at least one camera and the end effector is known, the pose of the at least one camera and / or its optical axis can then be calculated (or estimated) in the forward kinematics. After performing the calibration, the nominal values ​​of the sensors, in particular, will better match the actual values ​​for the linear positions or angles of the axes.

[0024] The scale serves, in particular, to describe a relationship between a physical unit of length (e.g., meters or millimeters) and a corresponding number of image elements in the acquired (further) images. Based on the scale, the corresponding physical quantity can be determined from a number of image elements in the acquired (further or additional) images. The scale makes it possible, in particular, to compare a quantity determined from the acquired (further or additional) images with a quantity calculated using the kinematic model.

[0025] The magnification center is, in particular, the point or area that does not move across at least two magnification levels. It is assumed, in particular, that this point or area coincides with an optical axis of the camera, in particular the optical axis of an imaging and magnification optics of the camera. The point or image area is identified, in particular, in the captured images using conventional methods, for example, by determining and evaluating an optical flow. The point or area can then serve as a reference point for determining the at least one distance measurement.

[0026] The designated point is a point that is determined and / or specified with reference to an image sensor of the at least one camera. The designated point is determined and / or specified in particular with reference to an image element of the image sensor of the at least one camera. For example, it can be provided that the point or image element that corresponds to a center of symmetry of a distortion of an imaging optics of the at least one camera is determined and / or specified as the designated point. It is particularly assumed here that such a center of symmetry corresponds to the optical axis of the imaging optics. In order to determine the center of symmetry, a suitable calibration object, for example comprising a checkerboard pattern, can be arranged in the detection range of the at least one camera.An image of this is captured by the at least one camera, and based on the captured image, a distortion map of the imaging optics of the at least one camera is determined using methods known per se. The center of symmetry is determined in this distortion map, and based on this, the point or image element that coincides with the determined center of symmetry is determined and / or defined as the distinguished point. Another possibility is to set the distinguished point to an image center of the image sensor of the at least one camera. In other words, the image element that lies in the center of an active area of ​​the image sensor of the at least one camera is defined as the distinguished point.

[0027] Parts of the medical visualization system, in particular the control device, can be implemented individually or collectively as a combination of hardware and software, for example, as program code executed on a microcontroller or microprocessor. However, it can also be provided that parts are implemented individually or collectively as an application-specific integrated circuit (ASIC) and / or a field-programmable gate array (FPGA). The control device is particularly configured to initiate the execution of the method steps, in particular by controlling the at least one camera and the robotic kinematics.

[0028] The solution according to the second aspect of the invention can be used in particular when the at least one other axis is subordinate to the selected axis in a kinematic chain of the robot kinematics. In other words, in this case, the at least one other axis is moved when the selected axis is moved, but not vice versa (in the kinematic chain, which is shown by way of example in the Fig. 2, the other axes would be, for example, axes 15-4 and 15-5 if the selected axis is axis 15-6. In particular, the solution according to the second aspect can thus be used if, for example, only a partial kinematics of a robotic tripod with the camera arranged therein is to be calibrated.

[0029] The following embodiments apply in particular to the first aspect and to the second aspect of the invention.

[0030] In one embodiment, steps b) and c) are performed for at least two working distances. This allows the amount of data available for adapting the kinematic model to be increased. In particular, this prevents the kinematic model from only aiming for a local minimum, which would result in the calibration becoming invalid at a different working distance or when the medical visualization system is restarted. Preferably, steps b) and c) are performed for at least two working distances. This allows a typical operating range to be well covered without the calibration effort becoming too great. For example, it can be provided to use at least the following working distances: 300 mm, 600 mm. Optionally, step a) can also be performed for the at least two working distances.

[0031] In one embodiment, it is provided that the selected axis is a rotation axis, wherein the rotation axis is or will be arranged perpendicular to the detection area, and wherein the at least one distance measure is a distance between, on the one hand, the respectively determined magnification center or the distinguished point and, on the other hand, the respective rotation center of the rotation axis. This makes it possible to improve the determination of the distance measure. In particular, it is possible to define at least one circular trajectory for the magnification center or the distinguished point in the superimposed further images on the basis of the plurality of images acquired at different angular positions of the rotation axis and to determine the at least one distance measure based on a radius of this circular trajectory. In other words, the optical axis (oranother designated point with respect to the image sensor) of the at least one camera by rotating it in a circle around the center of rotation. For the positions in the further images that lie on the circle defined in this way, a circular path can be adapted (fitted), and in this way a radius to the center of rotation in the middle of this circle can be determined and used as a distance measure. The radius is in particular the distance from the determined magnification center or the designated point to the center of rotation of the rotation axis, that is, in particular to the point or area whose position does not move or moves the least in the acquired further images. The center of rotation of the selected axis can then be described on a (calibration) object (in a coordinate system defined on the object).Subsequently, the position of at least one other axis between the calibration object and the selected axis can be varied. Since the center of rotation of the rotation axis on the calibration object does not change its position, another distance measurement can be quickly generated by varying the other axis.

[0032] In one embodiment, the scale of a calibration object arranged in the detection area is detected and taken into account when determining the distance measurements. This allows the scale to be detected and defined directly via a scale arranged on the calibration object. In particular, a number of image elements in the detected (further or additional) images that coincides with a size of the scale on the calibration object can be directly determined. Based on this relationship, other quantities can be converted back into a physical length based on a number of image elements in the detected (further or additional) images.

[0033] In one embodiment, it is provided that, to determine the scale, an image is captured at at least two different positions of the at least one other axis, wherein the scale is determined based on values ​​from a position sensor on the at least one other axis and a current value for a working distance. In this way, a (virtual) scale can be generated with the aid of the position sensor. The at least one other axis is, in particular, a rotational axis. The position sensor can, for example, be an encoder that encodes an angle into a sensor signal. Such encoders generally operate very precisely. This can be used to determine the (virtual) scale given a known working distance in the detection range.For this purpose, either a nominal working distance provided by an actuator and / or sensor system of the robot kinematics is used, for example, and / or an autofocus system, such as an autofocus laser, can be used to determine the working distance. Based on trigonometric considerations, the scale is then determined, particularly with reference to a plane detected within the detection range.

[0034] In one embodiment, the robotic kinematics are controlled such that the magnification center or the designated point is located at the rotation center of the rotation axis. This allows a plausibility check to be performed. After calibration, the positions of the axes should be known or determinable for which the magnification center or the designated point coincides with the rotation center upon rotation of the rotation axis.

[0035] In one embodiment, it is provided that a loading state of the robot kinematics is taken into account in the kinematic model. If different configurations are possible on the end effector (e.g., an additional environment camera, a lateral co-observer, a stereo bridge, etc.), the properties of the robot kinematics change, in particular with regard to a torque acting on the individual axes or joints. A calibration for one loading state (e.g., without an environment camera) may then be too inaccurate or even invalid for a different loading state (e.g., with an environment camera). However, this embodiment makes it possible to adapt the kinematic model using correction values, so that reliable pose estimation is possible even with a changed loading state.

[0036] In a further embodiment, it is provided that, based on at least one original first loading state and a second loading state that has been changed compared to this, a torque change is determined on at least one axle, and based on the determined torque change, at least one correction parameter is determined and taken into account in the kinematic model. In particular, a correction value for an angle can be determined directly using the determined torque change and a stiffness of a considered axle. If, for example, a changed loading state causes a torque change from M1→M2, which leads to an angle change on the considered axis from Phi1→Phi2, then ΔM=M2−M1 ΔPhi=Phi2−Phi1 and a constant for the stiffness: k=ΔM / ΔPhi with the unit Nm / (rad) a corrected angle: Phi0′=Phi0+ΔM / k.

[0037] The corrected angle Phi0' can then be used in the kinematic model.

[0038] In one embodiment, it is provided that at least some of the method steps of the method are carried out during normal operation of the medical visualization system. This allows calibration to take place continuously. For example, it can be provided that the method is always carried out at those times during normal operation when the medical visualization system has not been used for a predetermined period. Furthermore, it can be provided that the images are acquired when moving through magnification levels and then a movement of at least one axis is used to acquire the further images. In particular, it is provided that the acquisition of the further images takes place when a position of the selected axis is changed during normal operation. This makes it possible to utilize movements of the robot kinematics that are caused by a user during normal operation.

[0039] Alternatively or additionally, the procedure may also be performed after startup and / or before shutdown of the medical visualization system. Furthermore, calibration may also be performed by personnel at regular intervals outside of normal operation.

[0040] Further features for the design of the medical visualization system are described in the embodiments of the method. The advantages of the medical visualization system are the same as those of the embodiments of the method.

[0041] The invention will be explained in more detail below using preferred embodiments with reference to the figures. Fig. 1 is a schematic diagram illustrating an embodiment of the medical visualization system; Fig. 2 a schematic representation to illustrate a robotic kinematics of the medical visualization system; Fig. 3a, Fig. 3b schematic representations to illustrate an embodiment; Fig. 4 a schematic representation to illustrate an embodiment; Fig. 5 is a schematic diagram illustrating an embodiment; Fig. 6 is a schematic flow diagram illustrating an embodiment of the method according to the first aspect; Fig. 7a, Fig. 7b schematic representations to illustrate the second aspect of the invention.

[0042] The Fig. Figure 1 shows a schematic representation illustrating an embodiment of the medical visualization system 1. The medical visualization system 1 comprises a multi-axis robotic kinematic system 2, at least one camera 3 arranged at the end of the robotic kinematic system 2, and a control device 5. The medical visualization system is, for example, a surgical microscope. The method described in this disclosure is described in more detail below using the medical visualization system 1.

[0043] The robotic kinematics 2 comprises several links 6, which are connected to one another via rotary joints 7. The rotary joints 7 each enable rotation about a rotation axis corresponding to the rotary joint. This illustration is merely exemplary; in principle, the robotic kinematics 2 can also be simpler or more complex and, in particular, comprise fewer or more rotary joints 7 and rotation axes, as well as linear axes.

[0044] The at least one camera 3 is arranged at the end of the robotic kinematics 2 on a microscope head 8 and can thus be moved in particular in all degrees of freedom of the robotic kinematics 2. The at least one camera 3 captures a capture area 9. The at least one camera 3 can be, for example, a monocular or a stereo camera.

[0045] The control device 5 comprises a computing device 5-1 and a memory 5-2. The computing device 5-1 is configured to perform the computing operations necessary for carrying out method steps of the method described in this disclosure. For this purpose, program code and data can be stored in the memory 5-2, which the computing device 5-1 can access. The control device 5 is configured, in particular, to initiate the execution of the method steps, in particular by controlling the at least one camera 3 and the robotic kinematics 2.

[0046] The control device 5 is configured to describe the robotic kinematics 2 using a kinematic model 10. The kinematic model 10 can, for example, be based on the Denavit-Hartenberg model with four parameters for each of the axes or joints 7. However, other suitable models can also be used. The control device 5 is further configured to perform the following steps to calibrate the robotic kinematics 2: A scale 30 is detected and / or determined with reference to the detection area 9. This can be done in different ways, as described below.

[0047] Furthermore, a) in an alternative i) at least two Fig. an object 20 arranged in the detection area 9, for example a calibration object or any object, in different magnification levels of a magnification optics 4 of the at least one camera 3 and a magnification center of the magnification optics 4 starting from the detected Fig. certainly.

[0048] In particular, if no magnification optics 4 are present, an alternative to a) or an alternative to ii) can be used to determine and / or establish a distinguished point with reference to an image sensor of the at least one camera 3. For example, a distortion map can be acquired and / or determined using a suitable calibration object, and a center of symmetry can be determined in this distortion map, which center is established as the distinguished point. Alternatively, the distinguished point can also be defined, for example, as the image element of an image sensor of the at least one camera 3 that lies in the center of the image sensor.

[0049] Further b) a set of further Fig. recorded, with the further Fig. at different positions of a selected axis of robot kinematics 2.

[0050] c) A position of at least one other axis of the robot kinematics 2 is varied at least once, and according to the first aspect, step b) is repeated with the varied position. Optionally, step a) can also be repeated. In particular, step c) is performed more than once.

[0051] A typical process includes the following steps: For one axis configuration (axis configuration 1) of the robot kinematics 2, several further Fig. at different positions of the selected axis, so that for this axis configuration a set of further Fig. is available. The position of another axis is then changed (slightly), resulting in a different axis configuration (axis configuration 2) of robot kinematics 2. For this changed axis configuration, several additional Fig. at different positions of the selected axis, so that a set of further Fig. This is repeated for further axis configurations (axis configuration 3, 4, 5, ..., n etc.). For each of the n axis configurations, there is then a set of further Fig. The various positions of the selected axis can be the same for each axis configuration. However, other positions can also be used.

[0052] The control device 5 determines a set of distance measures 13, comprising at least one distance measure 13 for each of the sets of the detected further Fig. , each based on the additional data recorded Fig. , wherein the respective magnification center or the designated point forms at least one reference point for the distance measurement 13. As a result, at least one distance measurement 13 is then available for each of the n axis configurations mentioned as examples.

[0053] The control device 5 is configured to adapt the kinematic model 10 by changing the parameter(s) so that a deviation between the at least one distance measurement 13 determined for each set and a corresponding distance measurement calculated for each variation using the kinematic model 10 is minimized or will be minimized. In other words, parameters of the kinematic model 10, in particular the Denavit-Hartenberg parameters, are varied until the distance measurements calculated by the kinematic model 10 exhibit the smallest possible (cumulative) deviation from the determined distance measurements 13. For this purpose, the kinematic model 10 is supplied with sensor data from a sensor system of the robotic kinematics 2, e.g., the set angles of the respective joints 7 and / or a respective position of linear axes (if present) for the respective axis configuration.For each of the different positions of the selected axis, at least one distance measurement is then calculated using the kinematic model.

[0054] In particular, it is intended that the parameters of the kinematic model 10 discovered through the adaptation are subsequently used in the kinematic model 10. The adapted model 10 describes the robotic kinematics 2 better than the original kinematic model 10.

[0055] If the selected axis is a linear axis, the distance measure 13 can be determined, in particular, as a length resulting from a movement of the magnification center or the designated point across the various positions of the selected axis. If the selected axis is a rotational axis, the distance measure 13 can be determined, in particular, as a radius of a movement of the magnification center or the designated point around a center point of the rotational axis identifiable in the additional images acquired (cf. Fig. 3a and Fig. 3b).

[0056] The medical visualization system 2 may also include a display and operating device (not shown) and / or other devices.

[0057] According to the second aspect, in c) after varying a position of the at least one other axis, at least one additional Fig. Distance measures 13 are then determined, comprising at least one distance measure 13 for the set of further Fig. , based on the additional data recorded Fig. , and for the recorded at least one additional Fig. determined, whereby the respective magnification center or the designated point forms at least one reference point for the distance measurement 13. Once the distance measurements 13 are determined, the kinematic model 10 is adapted by changing the parameters, so that a deviation between the value for the set and the at least one additional Fig. determined at least one distance measure 13 and a corresponding distance measure 13 calculated for each variation by means of the kinematic model 10 is minimized or is minimized.

[0058] The Fig. Figure 2 shows a schematic representation to illustrate a more realistic robotic kinematics 2. A total of several links 6 are provided, each of which is connected to one another via joints 7, wherein each joint 7 corresponds to an axis 15-x and a degree of freedom. A total of seven axes 15-x are provided, wherein all axes 15-x in the example shown are designed as rotation axes. Furthermore, a further degree of freedom with the focus or the working distance 14 of the at least one camera is provided at the end of the robotic kinematics 2. In total, the robotic kinematics 2 therefore has eight degrees of freedom. The kinematic chain of the robotic kinematics 2 is described, for example, using the Denavit-Hartenberg model. For each of the joints 7 or for each of the axes 15-x, the model then comprises four parameters (Θ n [rad], d n [m], a n [m], α n [rad]). Here, the nand a n each in the Fig. 2 shows the lengths of the links 6 and the distances between the joints 7. The Denavit-Hartenberg convention describes how the transformation from one coordinate system to the next is performed. The coordinate systems always lie along the active axes of the robot. This is done by first rotating around the active axis (angle "Θ", which is determined by the rotational position of the corresponding axis), then performing two translations ("d" and "a"), and finally performing another rotation (angle "α"; fixed value, often zero, π / 2, or π), so that the coordinate system then coincides with the next active axis.

[0059] It can be provided that steps b) and c) are performed for at least two working distances 14. For this purpose, one working distance 14 is changed, and the steps described above are performed for each of the working distances 14. This can, in particular, prevent the kinematic model 10 from being optimized to a local minimum and becoming invalid at other settings. For example, it can be provided to provide at least the following working distances 14: 300 mm, 600 mm. Optionally, step a) can also be performed for the at least two working distances.

[0060] In one embodiment, it is provided that the selected axis is a rotation axis, wherein the rotation axis is arranged or will be arranged perpendicular to the detection area, and wherein the at least one distance measure 13 is a distance between the respectively determined magnification center or distinguished point and the respective rotation center of the rotation axis. For example, it can be provided that the Fig. 2 is used as the selected axis, whereas axes 15-4 and 15-5 are the other axes whose position is varied during the process. The positions of the remaining axes 15-1, 15-2, 15-3, and 15-7 are then kept constant. Only the focus, i.e., the working distance 14, can still be varied, as already described above. The selected axis 15-6 is a rotation axis, which in the illustration shown is perpendicular to the detection area 9.

[0061] The Fig. 3a and Fig. 3b show schematic representations to illustrate the embodiment described above. Fig. Figure 3a shows a simplified representation of the robot kinematics 2, which illustrates an exemplary axis configuration of the embodiment. Fig. 3b is an image content of three captured additional images displayed one above the other. Fig. recorded in this axis configuration. The three other recorded Fig. were measured at different angular positions of the selected axis 15-6 ( Fig. 3a, see also Fig. 2). Since the selected axis 15-6 is a rotation axis (which is Fig. 2 runs along the direction of d5), features 16-x, 17-x (the value of x indicates the position number of three exemplary positions or angles of rotation of the selected axis 15-6) move on circular paths with respect to the rotation axis 18, just like the magnification center 19 or the designated point 19a. The rotation axis 18 intersects the image plane at the point around which the features 16-x, 17-x rotate. This point on the rotation axis 18 can therefore be determined using circles fitted to the features 16-x, 17-x. In other words, the respective circle center corresponds to the point at which the rotation axis 18 intersects the image plane. A distance determined from the point on the rotation axis 18 to the magnification center 19 or to the designated point 19a can then be used as the distance measure 13.This distance corresponds to the radius with which the magnification center 19 or the designated point 19a rotates around the rotation axis 18. The magnification center 19 or the designated point 19a forms a reference point for the distance measurement 13. In other words, the distance measurement 13 is limited and / or defined on one side in particular by the magnification center 19 or the designated point 19a. Subsequently, a position of at least one other axis 15-5 is changed, and with the changed position, several further . Fig. at various positions of the selected axis 15-6 and evaluated as described. This can generally be performed for all axes 15-x. A set of specific distance measurements 13 (i.e., specific radii) is then available for adapting the kinematic model, i.e., one distance measurement 13 (radius) for each of the axis configurations used.

[0062] It can be provided that the scale 30 of a calibration object 20 arranged in the detection area 9 is detected and taken into account when determining the distance dimensions 13. At least one scale 30 is then arranged on the calibration object 20, the dimensions of which are known and which can be detected by means of the at least one camera 3. Based on the detected known scale 30, a ratio between a number of image elements in the detected (further) Fig. and real physical dimensions and taken into account when determining the distance dimensions 13.

[0063] It can further be provided that for determining the scale 30 a Fig. at at least two different positions of the at least one other axis 15-5, wherein the scale 30 is determined based on values ​​of a position sensor on the at least one other axis 15-5 and a current value for a working distance 14. This embodiment is schematically illustrated by the Fig. 4. For small angular changes ΔΘ5 of the axis 15-5, it can be approximately assumed, with a known working distance 14, that a displacement v of the magnification center 19 or the designated point 19a caused by the angular change is proportional to the angle (in rad), i.e., the following applies: v=f*ΔΘ5 where v is the displacement of the magnification center 19 or the designated point 19a in the object plane, f is the working distance 14, and ΔΘ5 is the angular change of the axis 15-5. The working distance 14 and the angular change ΔΘ5 are each determined using the values ​​of a sensor system of the robot kinematics. In particular, this takes advantage of the fact that an angular change can be detected very precisely using an encoder sensor system. The displacement v can then be calculated using the values ​​recorded at the respective angles. Fig. be assigned to a number of image elements. The displacement v thus forms the scale 30.

[0064] It can be provided that the robotic kinematics 2 ( Fig. 1 and Fig. 2) is controlled in such a way that the magnification center 19 or the distinguished point 19a is arranged in the rotation center 18 of the rotation axis.

[0065] It can be provided that a loading state of the robotic kinematics 2 is taken into account in the kinematic model 10. A loading state can change, for example, if an additional observer camera, for example an environment camera, is attached to the robotic kinematics 2, in particular to the microscope head 8 ( Fig. 1). Such an observer camera or another device additionally arranged on the microscope head 8 represents an additional mass, which leads to a change in torque on the robotic kinematics 2 or on the microscope head 8. This additional torque can be taken into account in the form of correction values ​​for the parameters of the kinematic model 10.

[0066] In particular, it can be provided that, starting from at least one original first loading state and a second loading state changed compared to this, a torque change is determined on at least one axle and, starting from the determined torque change, at least one correction parameter is determined and taken into account in the kinematic model.

[0067] This is exemplified in the Fig. 5. There, a first loading state 21 and a second loading state 22 of the robotic kinematics or the microscope head 8 are shown, wherein in the second loading state 22, the mass m (e.g., another sensor, an illumination source, or an environmental camera, etc.) is additionally arranged on the robotic kinematics or the microscope head 8. The additional mass m results in a torque change ΔM on the axis 15-5 (and also on existing, but not shown, additional axes 15-x) from M1 to M2, where ΔM = M2 - M1. The torque change ΔM causes a movement of the axis 15-5 by an angle ΔΘ5. Based on these values, which can be determined, for example, in a separate measurement using a suitable sensor on the axis 15-5, a stiffness for the axis 15-5 can be determined: k = ΔM / ΔΘ5 with the unit [Nm / rad].Based on this, an angle of the axis 15-x in the kinematic model can be corrected using a torque change ΔM currently detected by a sensor on the axis 15-5:. Θ5new=Θ50+ΔM / k

[0068] In the same way, stiffness constants can be determined for the other axes 15-x, so that the angles of these axes 15-x can also be corrected. The corrected angles, in particular, are then fed into the kinematic model.

[0069] It can be provided that at least some of the method steps of the method are carried out during normal operation of the medical visualization system 1 ( Fig. 1). For example, the calibration can be carried out completely or partially if the medical visualization system 1 has not been moved for a predetermined minimum period. Furthermore, it is also possible to record further Fig. possible if one of the axes 15-x is moved. This moved axis is then the selected axis. In particular, this allows sets of recorded further Fig. for various axis configurations set by a user during regular use, which can be used to determine the respective at least one distance measurement 13. If a predetermined number of distance measurements 13 is available, the kinematic model 10 can be adapted. In this way, the kinematic model 10 of the medical visualization system 1 can always be kept up-to-date, even in the field. Alternatively or additionally, the method can also be performed after booting up and / or before shutting down the medical visualization system 1.

[0070] The Fig. 6 shows a schematic flow diagram to illustrate an embodiment of the method according to the first aspect.

[0071] In a method step 100, a scale is detected and / or determined with reference to a detection area. For example, a known scale can be detected on a calibration object arranged in the detection area.

[0072] In a method step 101, in an alternative i), at least two images of an object arranged in the detection area, for example, a calibration object, are captured at different magnification levels, and a magnification center of the magnification optics is determined based on the captured images. In an alternative ii), a distinguished point is determined and / or specified with reference to an image sensor of the at least one camera.

[0073] In a method step 102, a set of further images is acquired, with the further images being acquired at different positions of a selected axis of the robot kinematics. For this purpose, method steps 102a to 102c are executed. In method step 102a, a position of the selected axis is set, in method step 102b, another image is acquired for the set position, and in method step 102c, a check is made to determine whether all specified positions of the selected axis have already been passed through. If this is not the case, the system returns to method step 102a and sets the next position of the selected axis; otherwise, the system continues with method step 103.

[0074] In process step 103, a check is made to determine whether all the specified axis configurations have already been run through. If this is not the case, the program continues with process step 104; otherwise, it continues with process step 105.

[0075] In method step 104, a position of at least one other axis of the robotic kinematics is varied at least once, and the system returns to method step 102. Optionally, it is also possible to return to method step 101.

[0076] In a method step 105, a set of distance measures, each comprising at least one distance measure for each of the sets of acquired further images, is determined in each case based on the acquired further images, wherein the respective magnification center (alternative i) or the distinguished point (alternative ii) forms at least one reference point for the distance measure.

[0077] In a method step 106, the kinematic model is adapted (fitted) by changing the parameters so that a deviation between the at least one distance measure determined for the sets and a corresponding distance measure calculated for each variation by means of the kinematic model is or will be minimized.

[0078] In a method step 107, the adapted kinematic model is provided for an application in the medical visualization system, for example, loaded into a memory and activated.

[0079] The Fig. 7a and Fig. 7b show schematic representations to illustrate the second aspect. This assumes a set of additional images acquired for different positions of the selected axis, with the selected axis in the example being a rotation axis. The result is a distance measurement 13 between the rotation axis 18 and the magnification center 19 ( Fig. 7a) or the excellent point 19a, as is the case with reference to the Fig. 3a and Fig. 3b has already been explained. Subsequently, the position of at least one other axis of the robotic kinematics is changed. The at least one other axis is in particular an axis subordinate to the selected axis in the kinematic chain of the robotic kinematics (in the example of Fig. 2, for example, the axis 15-6 is the selected axis, while the other axes are the axes 15-4 and 15-5). The position of the magnification center 19 or the designated point 19a remains the same with respect to a corresponding image element in particular in all acquired images, since the camera is arranged at the end of the kinematic chain. However, by changing the position of at least one other axis, the rotation axis 18 of the selected axis moves in the acquired additional Fig. ( Fig. 7b). However, the rotation axis 18 remains fixed to the object and does not move in the world coordinate system. Based on the additional Fig. a distance measure 13 can be determined by detecting a feature that coincides with the rotation axis 18 in the detected additional Fig. is identified, for example, by means of pattern recognition and / or by means of an artificial intelligence method (machine learning model, etc.). Alternatively, a coordinate system can be defined on the object and the position of the selected axis can be described in the object coordinate system. Using the detected and / or determined scale, the distance measure 13 can then be determined from the additional Fig. can be determined. For example, a calibration object with a checkerboard pattern or a ChArUco pattern can be used as an object.

[0080] In particular, it is planned that several additional Fig. and starting from each of the additional images a distance measure 13 is determined, which together with the distance measure 13, which starting from the set of further Fig. was determined, forms the basis for the subsequent adaptation of the kinematic model. List of reference symbols 1 medical visualization system 2 robotic kinematics 3 Camera 4 Magnifying optics 5 Control device 5-1 Computing device 5-2 Memory 6 link (robotic kinematics) 7 joint 8 microscope head 9 Detection range 10 kinematic model 11 captured image 12, 12-x captured further image (at position x) 13 Distance measurement 14 Working distance 15-x axis 16-x feature for position x 17-x Feature for position x 18 Rotation axis 19 Magnification Center 19a excellent point 20 objects 21 first loading condition 22 second loading condition 23 captured additional images 30 scale 100-107 Procedural steps f Working distance m mass v Shift ΔΘ5 angle change

Claims

[1] Method for calibrating a multi-axis robotic kinematics (2) of a medical visualization system (1), wherein the medical visualization system (1) has at least one camera (3) at the end of the robotic kinematics (2), and wherein the robotic kinematics (2) is described by means of a kinematic model (10), comprising: Detecting and / or determining a scale with reference to a detection area (9) of the at least one camera (3), a) i) capturing at least two images (11) of an object (20) arranged in the detection area (9) at different magnification levels of a magnification optics (4) of the at least one camera (3) and determining a magnification center (19) of the magnification optics (4) based on the captured images (11), or ii) determining and / or setting a distinguished point (19a) with reference to an image sensor of the at least one camera (3), b) capturing a set of further images (12), wherein the further images (12) are captured at different positions of a selected axis (15-6) of the robot kinematics (2), c) at least once: varying a position of at least one other axis (15-x) of the robotic kinematics (2) and repeating step b); determining a set of distance measurements (13), each comprising at least one distance measurement (13) for each of the sets of acquired further images (12-x), each starting from the acquired further images (12), wherein the respective magnification center (19) or the distinguished point (19a) forms at least one reference point for the distance measurement (13), Adapting the kinematic model (10) by changing parameters so that a deviation between the at least one distance measure (13) determined for each set and a corresponding distance measure (13) calculated for each variation by means of the kinematic model (10) is or will be minimized. [2] Method according to claim 1, characterized by that steps b) and c) are carried out for at least two working distances (14,f). [3] Method according to claim 1 or 2, characterized by that the selected axis (15-6) is a rotation axis, wherein the rotation axis (15-6) is or will be arranged perpendicular to the detection area (9), and wherein the at least one distance measure (13) is a distance between, on the one hand, the respectively determined magnification center (19) or the distinguished point (19a) and, on the other hand, the respective rotation center (18) of the rotation axis (15-6). [4] Method according to one of the preceding claims, characterized by that the scale of a calibration object (20) arranged in the detection area (9) is detected and taken into account when determining the distance measurements (13). [5] Method according to one of the preceding claims, characterized by in that, in order to determine the scale, an image (11) is acquired at at least two different positions of the at least one other axis (15-x), the scale being determined on the basis of values from a position sensor on the at least one other axis (15-x) and a current value for a working distance (14). [6] Method according to one of claims 3 to 5, characterized by that the robotic kinematics (2) is controlled in such a way that the magnification center (19) or the distinguished point (19a) is arranged in the rotation center (18) of the rotation axis. [7] Method according to one of the preceding claims, characterized bythat a loading state of the robotic kinematics (2) is taken into account in the kinematic model (10). [8] Method according to claim 7, characterized by that, starting from at least one original first loading state and a second loading state changed compared to this, a torque change is determined on at least one axle (15-x) and, starting from the determined torque change, at least one correction parameter is determined and taken into account in the kinematic model (2). [9] Method according to one of the preceding claims, characterized by that at least some of the method steps of the method are carried out during normal operation of the medical visualization system (1). [10] Method according to one of the preceding claims, characterized bythat in step a) alternative ii) for determining the distinguished point (19a) by means of a suitable calibration object, a distortion map is recorded and / or determined by means of the at least one camera (3) and in this distortion map a center of symmetry is determined, which is defined as the distinguished point (19a). [11] Medical visualization system (1), comprising: a multi-axis robotic kinematics (2), at least one camera (3) arranged at the end of the robotic kinematics (2), and a control device (5), wherein the control device (5) is configured to describe the robotic kinematics (2) by means of a kinematic model (10), and to carry out a calibration of the robotic kinematics (2): to detect and / or determine a scale with reference to a detection area (9), a) i) to capture at least two images (11) of an object (20) arranged in the capture area (9) at different magnification levels of a magnification optics (4) of the at least one camera (3) and to determine a magnification center (19) of the magnification optics (4) based on the captured images (11), or ii) to determine and / or establish a distinguished point (19a) with reference to an image sensor of the at least one camera (3), b) to acquire a set of further images (12-x), wherein the further images (12-x) are acquired at different positions of a selected axis (15-6) of the robot kinematics (2), c) at least once: to vary a position of at least one other axis (15-x) of the robot kinematics (2) and to repeat step b); to determine a set of distance measures (13), each comprising at least one distance measure (13) for each of the sets of acquired further images (12-x), each starting from the acquired further images (12-x), wherein the respective magnification center (19) or the distinguished point (19a) forms at least one reference point for the distance measure (13), to adapt the kinematic model (10) by changing parameters so that a deviation between the at least one distance measure (13) determined for the sets in each case and a distance measure (13) corresponding thereto and calculated for each variation by means of the kinematic model (10) is or will be minimized. [12] Method for calibrating a multi-axis robotic kinematics (2) of a medical visualization system (1), wherein the medical visualization system (1) has at least one camera (3) at the end of the robotic kinematics (2), and wherein the robotic kinematics (2) is described by means of a kinematic model (10), comprising: Detecting and / or determining a scale with reference to a detection area (9) of the at least one camera (3), a) i) capturing at least two images (11) of an object (20) arranged in the detection area (9) at different magnification levels of a magnification optics (4) of the at least one camera (3) and determining a magnification center (19) of the magnification optics (4) based on the captured images (11), or ii) determining and / or setting a distinguished point (19a) with reference to an image sensor of the at least one camera (3), b) capturing a set of further images (12), wherein the further images (12) are captured at different positions of a selected axis (15-6) of the robot kinematics (2), c) at least once: varying a position of at least one other axis (15-x) of the robot kinematics (2) and capturing at least one additional image; Determining distance measures (13), comprising at least one distance measure (13) for the set of acquired further images (12-x), starting from the acquired further images (12), and for the acquired at least one additional image, wherein the magnification center (19) or the distinguished point (19a) forms at least one reference point for the distance measure (13), Adapting the kinematic model (10) by changing parameters so that a deviation between the at least one distance measure (13) determined for the set and the at least one additional mapping and a distance measure (13) corresponding thereto and calculated for each variation by means of the kinematic model (10) is or will be minimized. [13] Medical visualization system (1), comprising: a multi-axis robotic kinematics (2), at least one camera (3) arranged at the end of the robotic kinematics (2), and a control device (5), wherein the control device (5) is configured to describe the robotic kinematics (2) by means of a kinematic model (10), and to carry out a calibration of the robotic kinematics (2): to detect and / or determine a scale with reference to a detection range (9), a) i) to capture at least two images (11) of an object (20) arranged in the capture area (9) at different magnification levels of a magnification optics (4) of the at least one camera (3) and to determine a magnification center (19) of the magnification optics (4) based on the captured images (11), or ii) to determine and / or establish a distinguished point (19a) with reference to an image sensor of the at least one camera (3), b) to acquire a set of further images (12-x), wherein the further images (12-x) are acquired at different positions of a selected axis (15-6) of the robot kinematics (2), c) at least once: to vary a position of at least one other axis (15-x) of the robot kinematics (2) and to capture at least one additional image; Distance measures (13), comprising at least one distance measure (13) for the set of acquired further images (12-x) based on the acquired further images (12-x) and for the acquired at least one additional image, wherein the magnification center (19) or the distinguished point (19a) forms at least one reference point for the distance measure (13), to adapt the kinematic model (10) by changing parameters so that a deviation between the at least one distance measure (13) determined for the set and the at least one additional image and a distance measure (13) corresponding thereto and calculated for each variation by means of the kinematic model (10) is or will be minimized.

Images