Method for patient registration on a medical visualization system and medical visualization system

Abstract

The invention relates to a method for patient registration on a medical visualization system (1), comprising: performing an initial patient registration, wherein parameters of a transformation rule (10) between a reference coordinate system (20) and a patient coordinate system (21) are determined, further for at least one camera pose (30) of a main camera (2) of the medical visualization system (1): Determining the camera pose (30) in the reference coordinate system (20), Capturing a patient image (12) using the main camera (2), and at least once: - generating a two-dimensional projection image (13) of preoperatively acquired three-dimensional patient data (60) starting from the determined camera pose (30) of the main camera (2) taking into account the transformation rule (10), and - minimizing a deviation (14) between the acquired patient image (12) and the corresponding generated two-dimensional projection image (13) by adapting parameters of the transformation rule (10); and Providing the adapted transformation rule (10).

Description

[0001] The invention relates to a method for patient registration on a medical visualization system and a medical visualization system.

[0002] To use preoperatively acquired data, such as computed tomography (CT) data or magnetic resonance imaging (MRI) data, in surgery, a relationship between a reference coordinate system and a patient, or between a patient coordinate system during the operation, must be known. Determining this relationship, or a transformation rule describing the relationship between the coordinate systems, is usually performed before the operation and is referred to as patient registration. Patient registration specifically involves determining the relative pose (translation and rotation) of a patient in another coordinate system that does not move relative to the patient and therefore serves as the reference coordinate system.

[0003] To register the patient, the patient's surface and the pose of a reference object located in the reference coordinate system are determined, e.g., using a unique tracker or marker firmly attached to the patient (e.g., a Mayfield clamp). The patient's surface can be determined in the reference coordinate system, e.g., using stereoscopic topography determination (e.g., derived from stereoscopic image data from a surgical microscope) or using contact-based or contactless probing devices, such as a Brainlab Softtouch or a Brainlab Z-Touch (both from Brainlab AG). The pose of the reference object (the marker or tracker on the patient) can be recorded and determined using a navigation system (e.g., from NDI, Canada) or, alternatively, using an internal tracking system of a medical visualization system.

[0004] The accuracy of patient registration depends significantly on the determination of the surface (topography, soft touch, or Z-touch) on the patient. Topography determination using stereoscopic image data requires very precise camera calibrations. Furthermore, the stereo base of surgical microscopes, with a working distance typically ranging from 200 mm to 650 mm, is small (especially between 2 and 3 cm), which makes estimating the patient's depth in the camera system of the medical visualization system inaccurate. Therefore, improving the accuracy of patient registration is desirable.

[0005] From EP 1 142 536 A1 a method for referencing a patient or a body part of a patient in a camera-supported, medical navigation system is known, comprising the following steps: the body part of the patient to be referenced is brought into the detection range of a navigation system supported by at least two cameras, which computer-aidedly detects the three-dimensional spatial positions of light markers, by means of a light beam light markers are generated on the surface of the body part to be referenced, the three-dimensional position of which is determined by the camera-supported navigation system, by means of the position data for the light markers the spatial position of the surface of the body part to be referenced is determined.

[0006] J. Kang, W. Liu, W. Tu and L. Yang, YOLO-6D+: Single Shot 6D Pose Estimation Using Privileged Silhouette Information, 2020 International Conference on Image Processing and Robotics (ICIP), Negombo, Sri Lanka, 2020, pp. 1-6, doi: 10.1109 / ICIP48927.2020.9367354, describes a method for pose estimation.

[0007] The invention is based on the object of improving a method for patient registration on a medical visualization system and a medical visualization system, in particular with regard to accuracy in patient registration.

[0008] The object is achieved according to the invention by a method having the features of patent claim 1 and a medical visualization system having the features of patent claim 14. Advantageous embodiments of the invention emerge from the subclaims.

[0009] One of the basic ideas of the invention is to improve the accuracy of a transformation rule between a reference coordinate system and a patient coordinate system, starting from an initial patient registration, by performing an optimization based on a deviation between two-dimensional images. The idea behind this is that a main camera of the medical visualization system uses those degrees of freedom to optimize the transformation rule whose respective value can be estimated with greater accuracy. These degrees of freedom are those degrees of freedom that lie in the image plane of an image from the main camera, namely the x-direction, the y-direction, and the rotation (in particular the roll angle of the main camera around an optical axis) with respect to the image or the image plane (plane of an image sensor of the main camera).Values ​​of the other degrees of freedom, such as the z-direction along the optical axis of the main camera and the other rotation directions (in particular the yaw and pitch angles of the main camera), can be estimated less well from a captured image and are therefore not used. The accuracy of the transformation rule is improved by capturing a patient image for at least one camera pose, preferably at least two or more camera poses, of the main camera. For the same camera pose, a two-dimensional projection image is additionally generated based on preoperatively acquired three-dimensional patient data (e.g. CT or MRI data, etc.). The two-dimensional projection image is generated in such a way that it is arranged at the same position, with the same orientation and scale with respect to the captured patient image in the reference coordinate system.The two-dimensional projection image is generated, in particular, by projecting the preoperatively acquired three-dimensional patient data onto a plane of an image sensor of the main camera according to the laws of geometric optics. The pose of the image sensor can be derived, in particular, from the camera pose (taking into account intrinsic and extrinsic calibration). Ideally, i.e., if the transformation rule were error-free, the acquired (two-dimensional) patient image and the two-dimensional projection image corresponding to it via the camera pose would have to be congruent, i.e., in particular, they would have the same image content or encompass the same image section. In reality, however, a deviation will occur if the transformation rule is faulty.To minimize this deviation, parameters of the transformation rule are adjusted based on the acquired patient image and the generated two-dimensional projection image. The deviation is determined and minimized based on the acquired two-dimensional patient image and the generated two-dimensional projection image. At the end of the process, a transformation rule with improved accuracy can then be provided.

[0010] In particular, a method for patient registration on a medical visualization system is provided, comprising: performing an initial patient registration, wherein parameters of a transformation rule between a reference coordinate system and a patient coordinate system are determined, further for at least one camera pose of a main camera of the medical visualization system: Determining the camera pose in the reference coordinate system, Capturing a patient image using the main camera, and at least once: - generating a two-dimensional projection image of preoperatively acquired three-dimensional patient data based on the determined camera pose of the main camera, taking into account the transformation rule, and - Minimizing a deviation between the acquired patient image and the corresponding generated two-dimensional projection image by adjusting parameters of the transformation rule; and providing the adjusted transformation rule.

[0011] Furthermore, in particular, a medical visualization system is created, comprising a main camera and a control device, wherein the control device is configured to obtain and / or determine parameters of a transformation rule between a reference coordinate system and a patient coordinate system within the scope of an initial patient registration, and for at least one camera pose of the main camera: to obtain a camera pose determined in the reference coordinate system or to determine the camera pose, to initiate the capture of a patient image by means of the main camera in the at least one camera pose and / or to obtain a patient image captured in the at least one camera pose, and at least once: - to generate a two-dimensional projection image of preoperatively acquired three-dimensional patient data based on the determined camera pose of the main camera, taking into account the transformation rule, and - to minimize a deviation between the acquired patient image and the corresponding generated two-dimensional projection image by adjusting parameters of the transformation rule; and to provide the adjusted transformation rule.

[0012] An advantage of the method and the medical visualization system is that the adaptation and / or optimization of the transformation rule is carried out based on those degrees of freedom in the images of the main camera whose values ​​can be determined most accurately with respect to a deviation. These are, in particular, the degrees of freedom that correspond to the two-dimensional pixel plane of an image sensor of the main camera. These are, in particular, the degrees of freedom that correspond to the coordinate axes of the pixel sensor and a rotation of the pixel sensor about a surface normal running perpendicularly through the plane of the pixel sensor (i.e., the x-direction, y-direction, and rotation degrees of freedom).

[0013] The medical visualization system is, in particular, a surgical microscope. However, the medical visualization system can also be a microscope used for medical examinations and / or diagnostic purposes. In principle, the method can be used particularly with microscopes that operate at high magnification and where, based on this, there are three of six degrees of freedom whose values ​​can be determined more precisely in an image captured by the main camera than those of the other three degrees of freedom.

[0014] The two-dimensional projection image is determined based on the respective camera pose of the main camera. For this purpose, a known extrinsic and intrinsic calibration of the main camera are taken into account, which can be determined in a conventional manner. The intrinsic calibration describes in particular properties or spatial relationships of an imaging optics of the main camera. Properties include, for example, a description of distortions or a focal length. The intrinsic calibration parameters make it possible to image an object whose position in the coordinate system of the main camera is known onto the corresponding sensor of the main camera. The extrinsic calibration describes in particular a relative spatial relationship of the main camera to the reference coordinate system or to a tracking system operating in the reference coordinate system (e.g., a surround camera).Using the known camera pose, in which the patient image was also captured, a spatial relationship to the patient coordinate system in which the preoperatively acquired three-dimensional patient data is available can be established using the transformation rule (and the known intrinsic and extrinsic calibration). This preoperatively acquired three-dimensional patient data is projected into the image plane or image sensor plane of the main camera using known relationships and methods to obtain the two-dimensional projection image. For example, a pinhole camera model can be used for the main camera. A point from the 3D image space (X w , Y w , Z w) can be transformed into the image space (u, v) using the pinhole camera model. This is described as an example for the OpenCV software package at https: / / docs.opencv.org / 4.x / d9 / d0c / group__calib3d.html. The generated two-dimensional projection image has, in particular, the same number of image elements and the same width and height—i.e., the same image format—as the acquired patient image. If the number of image elements and / or the format differ, the images are scaled accordingly to enable direct comparison.

[0015] Within the scope of the method, a deviation between the acquired patient image and the projection image corresponding to it via the camera pose is determined and minimized. By adjusting the parameters, a subsequently generated two-dimensional projection image is changed, so that the deviation from the acquired patient image can be influenced and thus minimized, particularly gradually.

[0016] Using the transformation rule, coordinates of the reference coordinate system can be converted into coordinates of the patient coordinate system and vice versa. Using the transformation rule, the preoperatively acquired three-dimensional patient data can be located in the reference coordinate system. Using the camera pose, which is also known in the reference coordinate system, the two-dimensional projection image can be generated based on the preoperatively acquired three-dimensional patient data arranged in this way. Ideally, this projection image—that is, with an error-free transformation rule—is congruent with the patient image acquired in this camera pose. The deviation between the images can be used to quantify the extent of an error in the real case of an erroneous transformation rule and to correct the parameters of the transformation rule.The relationship between the patient coordinate system and the reference coordinate system, i.e., the transformation rule, can be described using a 6D pose. This can be done, for example, using a 3x1 rotation vector and a 3x1 translation vector, or alternatively, using quaternions. Assuming a point P is described with a normal to the surface in a coordinate system K1, its position P1 and orientation R1 can be represented as follows: P1=[R1 P1;0 0 0 1] where R1 has the dimension 3x3 and P1 has the dimension 3x1. Using a transformation matrix with the rotation R 12 (Dimension 3x3) and the translation T 12 (Dimension 3x1) the description of the point P can be transferred from the coordinate system K1 to a coordinate system K2. P2=[R12 T12;0 0 0 1]*[R1 P1;0 0 0 1]

[0017] Alternatively, a point P can be described in one coordinate system with its three coordinates (x, y, z). Using a transformation notation, this point P can be transformed into another coordinate system K2. [Px,2;Py,2;Pz,2;1]=[R12T12; 0 0 0 1]*[Px,1;Py,1;Pz,1;1] [P x,2 ;P y,2 ;P z,2 ;1] describes the point P in the second coordinate system K2, [P x,1 ;P y,1 ;P z,1 ;1] the point P in the coordinate system K1 and the 4x4 matrix [R 12 T 12 ;0 0 0 1] is the transformation matrix from the coordinate system K2 to the coordinate system K1.

[0018] Using this method, the entire transformation chain between the reference coordinate system and the patient coordinate system can be considered. In particular, errors in a tracking system and intrinsic and extrinsic calibration can be compensated for by reactivating these initially set calibration parameters (allowing them to be optimized).

[0019] During the initial patient registration, the transformation rule between a reference coordinate system and a patient coordinate system is estimated. This estimate forms a starting point for the optimization performed using the method to increase the accuracy of the transformation rule. The initial patient registration can be performed in various ways. In particular, the initial patient registration can be performed using methods known per se. Some further exemplary possibilities for the initial patient registration are described later in this disclosure.

[0020] At least one additional main camera can be provided, for example, as part of a stereo camera of the medical visualization system. The method can then also be performed for the at least one additional main camera, whereby the procedure is fundamentally analogous to that for the main camera.

[0021] After performing the minimization, the adapted transformation rule is provided. Providing can, in particular, comprise loading the adapted parameters of the transformation rule into a memory of the control device so that the adapted transformation rule is available for subsequent use. In particular, the preoperatively acquired three-dimensional patient data can be overlaid on acquired images captured by the main camera for augmentation.

[0022] Parts of the medical visualization system, in particular the control device, can be implemented individually or collectively as a combination of hardware and software, for example, as program code executed on a microcontroller or microprocessor. However, it can also be provided that parts are implemented individually or collectively as an application-specific integrated circuit (ASIC) and / or a field-programmable gate array (FPGA). The control device comprises, in particular, at least one computing device and at least one memory. Furthermore, the control device can comprise a communication interface for communication.

[0023] The procedure and the medical visualization system can be used in particular in the following areas: neurosurgery, spinal surgery, dental surgery, ophthalmic surgery, etc.

[0024] In particular, it is provided that the generation of the two-dimensional projection image and the minimization are carried out for at least one further camera pose. In particular, the camera poses differ from one another (e.g., a different perspective, different detection angle, etc.). This allows, in particular, all six degrees of freedom of the medical visualization system to be taken into account when minimizing the deviation. The transformation rule is then optimized or improved, in particular with regard to all six degrees of freedom. In other words, in particular, a patient image is acquired in at least two camera poses. In particular, a two-dimensional projection image is generated for each of the patient images. The minimization of the deviation then takes place taking into account the patient images and the associated two-dimensional projection images of the at least two camera poses.

[0025] In one embodiment, features in the acquired patient image and the corresponding two-dimensional projection image are detected, with the deviation being minimized based on the detected features. This allows a deviation to be determined directly based on corresponding features. For example, features such as edges, contours, patterns, silhouettes, shadows, and the like can be detected in the images. Known pattern recognition methods from the fields of computer vision and / or machine learning can be used to detect the features. A deviation between the images can be quantified based on the detected features. For example, it can be provided to determine a distance between corresponding features in the images.A translation (in particular an x-direction and / or a y-direction of the image plane or an image sensor) and a rotation (rotation angle) can be considered. The distance can be expressed, for example, as a number of image elements. For multiple features, a cumulative distance can be determined as a measure of a cumulative deviation. The deviation or the cumulative deviation are minimized within the process by adjusting the parameters of the transformation rule (stepwise or iteratively).

[0026] In one embodiment, the camera pose is determined using a surround camera of the medical visualization system used as an internal tracking system and at least one marker (e.g., a Mayfield clamp) arranged on the patient. This eliminates the need for an external tracking system. It is assumed that a spatial relationship between the surround camera and the main camera is known (i.e., in particular, an extrinsic calibration is known).

[0027] In one embodiment, the camera pose is determined using an external tracking system and at least one marker arranged on the patient (e.g., a Mayfield clamp) and at least one marker arranged on the main camera. This allows the poses of the markers on the patient and the main camera to be determined directly. The external tracking system then defines the reference coordinate system.

[0028] In one embodiment, the reference coordinate system is a coordinate system of a robotic tripod, wherein the camera pose is determined in the reference coordinate system based on position data of an actuator of the robotic tripod. This has the advantage that no optical tracking is necessary. A relative initial pose of the robotic tripod to the patient can be estimated using image-based pose estimation. For this purpose, for example, a focal point of the main camera of the medical visualization system can be used for the initial position of the patient. A camera pose can be determined in the coordinate system of the robotic tripod using a kinematic model, with which the camera pose can be calculated based on the settings of actuators (rotational axes and / or linear axes) of the robotic tripod.Using a further transformation that takes the optical imaging into account, the position of the focal point of the main camera in the coordinate system of the robotic tripod can then be determined (this essentially involves a translation of the camera pose). In particular, it can be assumed that the focal point, if it is located at a designated position on the patient, defines the patient coordinate system or coincides with an origin of the patient coordinate system. The designated point is chosen in particular such that it initially defines the origin of the patient coordinate system. The coordinate system of the robotic tripod itself can, for example, be defined with reference to a designated point on a base of the robotic tripod as a reference coordinate system. The main camera is arranged in particular at a distal end of the robotic tripod.The camera pose can be changed and / or adjusted using the robotic tripod. The position data is acquired using sensors arranged at the joints of the robotic tripod, corresponding to the respective actuators. Such sensors can be, for example, encoders that encode the respective rotation angles of the joints of the robotic tripod. Based on this, a camera pose can be calculated using a model for the forward kinematics.

[0029] It can be provided that the camera pose is determined using an internal tracking system and / or an external tracking system and / or a robotic tripod. By combining different methods, the camera pose can be determined more precisely. Values ​​for the camera pose determined based on the individual methods are then provided, in particular, in an averaged manner. In particular, it can also be provided that the camera pose is determined using an internal tracking system and an external tracking system and a robotic tripod.

[0030] In one embodiment, the initial patient registration is carried out by capturing at least one selected region of a patient using a capturing device in a reference coordinate system and adapting the preoperatively acquired three-dimensional patient data, which are available with reference to a patient coordinate system, to the captured region and determining parameters of the transformation rule between the reference coordinate system and the patient coordinate system. The initial patient registration is then carried out in particular using methods known per se from the prior art. The capturing device can, for example, be a contact-based or contactless capturing device that is configured to capture the selected region of the patient with regard to its spatial properties (e.g., surface profile, contour, etc.).The detection device may also be part of the medical visualization system or comprise at least a part of the medical visualization system.

[0031] In one embodiment, the initial patient registration comprises capturing and / or deriving a surface contour of at least the selected region, wherein the adaptation to the preoperatively acquired three-dimensional patient data is carried out based on the captured and / or derived surface contour. This makes it possible to achieve a high level of accuracy even for the initial patient registration. To capture the surface contour, the patient (or the selected region) can be captured monoscopically or stereoscopically, for example using a stereo camera. In particular, a marker permanently arranged on the patient (tracker, e.g. a Mayfield clamp) can also be captured, so that a pose of the surface contour in the reference coordinate system defined by the marker can be captured.Alternatively or additionally, the surface contour can be determined using a contact-based or contactless probing device (detection device), such as a Brainlab Softtouch or a Brainlab Z-Touch. Such probing devices typically have markers that can be detected and located in the reference coordinate system using an internal tracking system (e.g., the medical visualization system's field camera) or an external tracking system (navigation system), allowing the pose of the probing device to be detected in the reference coordinate system. Additionally, a marker permanently attached to the patient can be detected.

[0032] In one embodiment, the initial patient registration includes a pose estimation of the main camera relative to the patient. The pose estimation can be performed, in particular, with reference to a coordinate system of a robotic tripod used as a reference coordinate system. Based on the estimated pose of the main camera relative to the patient, the patient's pose can be determined in the reference coordinate system. The reference coordinate system can, in particular, be the coordinate system of the robotic tripod. The camera pose can be determined using a kinematic model or a transformation derived therefrom in the coordinate system of the robotic tripod and thus in the reference coordinate system. Based on this, the transformation rule between the reference coordinate system and the patient coordinate system can be determined using the estimated pose of the main camera relative to the patient.Pose estimation can be performed, for example, using a method as described in J. Kang, W. Liu, W. Tu and L. Yang, YOLO-6D+: Single Shot 6D Pose Estimation Using Privileged Silhouette Information, 2020 International Conference on Image Processing and Robotics (ICIP), Negombo, Sri Lanka, 2020, pp. 1-6, doi: 10.1109 / ICIP48927.2020.9367354.

[0033] In one embodiment, for initial patient registration, a captured patient image is displayed superimposed with a projection image generated from the preoperatively captured patient data, and the main camera is positioned by a user in a desired camera pose based on the superimposed images. This allows initial patient registration to be carried out with the assistance of a user. For example, it can be provided that, depending on the type of planned surgery, a suitable projection image is generated based on patient data captured preoperatively in preparation for the surgery. Typically, a surgical intervention is performed from a predetermined direction. For example, the projection image is generated for this direction from the preoperatively captured patient data and displayed, for example, on a display device of the medical visualization system.At the same time, a patient image is captured of the patient using the main camera and also displayed on the display device. The projection image is displayed, in particular, superimposed with the patient image. In particular, it is provided that patient images are captured continuously or repeatedly using the main camera, in particular in the form of a real-time video in which a change in the arrangement of the main camera appears immediately. A user can then adjust the pose of the main camera based on the superimposed display of the images such that the projection image and the patient image contain the same content (e.g., part of the head, etc.). If the images are aligned for a camera pose, the corresponding camera pose is determined.If the reference coordinate system is a coordinate system of a robotic tripod, the coordinates of the camera pose can be determined, for example, based on sensor values ​​from an actuator. Based on the determined camera pose, the transformation rule for the initial patient registration is determined.

[0034] In one embodiment, a set of camera poses is provided, with the minimization being performed based on all acquired patient images and the corresponding two-dimensional projection images of the set. This allows the accuracy of the transformation rule to be further improved. In particular, the method determines and minimizes a total deviation for the set.

[0035] In one embodiment, it is provided that markers are placed on the patient and / or selected areas are identified, wherein the minimization of the deviation is carried out taking into account the markers and / or the selected areas. This allows specific areas of the patient to be marked and / or identified, which can be used to determine the deviation. In particular, it is provided that the markers are used and / or recorded (or registered) as part of the initial patient registration.

[0036] In one embodiment, at least one region in which the patient is located is structurally illuminated during the acquisition of the patient image, wherein the minimization of the deviation is carried out taking the structural illumination into account. This can assist in determining the deviation. For this purpose, in particular, a pose of a lighting unit that generates the structural illumination relative to the main camera must be known. This can be known in advance or determined via an extrinsic calibration. Furthermore, a direction of light rays of the structural illumination emanating from the lighting unit must be known. In this embodiment, the light rays of the lighting unit must be taken into account in the generated two-dimensional projection image accordingly via the known direction.In other words, the structural illumination is also generated in the projection image based on the known direction of the light rays. For example, if a line projection is used, the patient's topography causes the lines of the projection to become curves in both the acquired patient image and the generated two-dimensional projection image, which can be recognized and associated with each other. The optimization of the transformation rule is then achieved, in particular, by minimizing any deviation between the positions of the curves in the superimposed images. Alternatively, other projection patterns or lasers can be used, whereby the procedure is analogous.

[0037] In one embodiment, at least one camera pose of the main camera is set automatically using a robotic tripod. This allows for the automated setting of various camera poses. In particular, this allows for fully automated acquisition of the patient image in multiple camera poses. This allows the method to be performed automatically at the location of the medical visualization system without significant personnel expenditure and without special training.

[0038] Further features for the design of the medical visualization system are described in the various embodiments of the method. The advantages of the medical visualization system are the same as those of the various embodiments of the method.

[0039] The invention will be explained in more detail below using preferred embodiments with reference to the figures. Fig. 1 a schematic representation to illustrate embodiments of the medical visualization system; Fig. 2 a schematic representation to illustrate the determination of the deviation; Fig. 3 a schematic flow diagram to illustrate embodiments of the method.

[0040] The Fig. Figure 1 shows a schematic diagram illustrating embodiments of the medical visualization system 1 and the method. The medical visualization system 1 comprises a main camera 2 and a control device 3. The control device 3 comprises a computing device 3-1 and a memory 3-2.

[0041] At least one additional main camera (not shown) can be provided, for example, as part of a stereo camera. The method can also be performed using the at least one additional main camera, whereby the procedure is analogous to using the main camera. The respective camera pose of the at least one additional main camera is then taken into account accordingly.

[0042] The control device 2 is configured to obtain and / or determine parameters 11 of a transformation rule 10 between a reference coordinate system 20 and a patient coordinate system 21 as part of an initial patient registration. In the embodiment shown, the medical visualization system 1 comprises, in particular, a robotic stand 40.

[0043] It can be provided that the initial patient registration is carried out by capturing at least one selected region of a patient 22 using a capturing device in a reference coordinate system 20 and adapting preoperatively acquired three-dimensional patient data 60, which are available with reference to a patient coordinate system 21, to the captured region and determining parameters of the transformation rule 10 between the reference coordinate system 20 and the patient coordinate system 21. The capturing device can be, for example, the main camera 2.

[0044] The control device 2 is further configured to obtain a camera pose 30 determined in the reference coordinate system 20 for at least one camera pose 30 of the main camera 2 or to determine the camera pose 30, to initiate the acquisition of a patient image 12 by means of the main camera 2 in the at least one camera pose 30 and / or to obtain a patient image 12 acquired in the at least one camera pose 30.

[0045] Furthermore, by means of the control device 2, a two-dimensional projection image 13 of the preoperatively acquired three-dimensional patient data 60 is generated at least once, starting from the determined camera pose 30 of the main camera 2, taking into account the transformation rule 10, and a deviation 14 between the acquired patient image 12 and the corresponding generated two-dimensional projection image 13 is minimized by adapting the parameters 11 of the transformation rule 10. In particular, for this purpose, a two-dimensional projection image 13 is repeatedly and / or iteratively generated, taking into account the adapted transformation rule 10, and a deviation 14 is determined again in order to adapt the parameters 11 again. This is repeated, in particular, until the deviation 14 falls below a predetermined threshold value.

[0046] The adapted transformation rule 10 is subsequently provided. In particular, providing may include loading the adapted transformation rule 10 into a memory 3-2 or memory area of ​​the control device 3 in order to subsequently make the transformation rule 10 available for application. Such an application may, for example, include overlaying the preoperatively acquired patient data 60 with a patient image 12 acquired by the main camera 2, in particular for augmenting the acquired patient image 12.

[0047] The Fig. Figure 2 shows a schematic representation to illustrate the determination of the deviation 14 between the acquired patient image 12 and the generated two-dimensional projection image 13. In the example shown, it is assumed that the transformation rule is faulty, which means that the patient image 12 and the projection image 13 have a different image content. In other words, the Fig. different sections of the patient 22, since the faulty transformation rule leads to the coordinates of the camera pose in which the patient image 12 was acquired being incorrectly transformed into the patient coordinate system in which the preoperatively acquired three-dimensional patient data 60 are present, and vice versa. Due to this faulty transformation, the projection image 13 generated from the preoperatively acquired patient data in projection onto the camera pose shows a different image section than the acquired patient image 12. The same features 15-1, 15-2 in the Fig. , such as edges, contours or patterns, therefore have different positions in the Fig. (described e.g. with reference to image element coordinates of an image sensor in x and y directions). This is shown by means of a superimposed Fig. which is also in the Fig. 2. There are the Fig. shown superimposed on one another. It can be seen that the same features 15-1, 15-2 have a respective distance 17-1, 17-2 from one another. This distance 17-1, 17-2 can, for example, be quantified in the coordinate system of the image elements. By adjusting the parameters of the transformation rule, the respective distance 17-1, 17-2 can be reduced, so that the features 15-1, 15-2 ideally lie directly above one another, i.e., in particular, are each located at the same image element coordinates. For example, it can be provided that a measure for the deviation 14 is determined from the distances 17-1, 17-2, for example by forming a sum of the distances 17-1, 17-2 or another suitable measure (e.g., mean square deviation, etc.).By gradually adjusting the transformation parameters using known optimization methods and regenerating the two-dimensional projection image 13 with the thus adjusted transformation rule, the deviation 14 can be determined again. In this way, the deviation 14 can be minimized step by step or iteratively.

[0048] In particular, it can be provided that the features 15-1, 15-2 are recognized in the acquired patient image 12 and the corresponding two-dimensional projection image 13, wherein the minimization of the deviation 14 is carried out based on the recognized features 15-1, 15-2. To recognize the features 15-1, 15-2, known computer vision and / or artificial intelligence methods, such as machine learning and / or pattern recognition methods, can be used.

[0049] It can be provided that the determination of the camera pose 30 ( Fig. 1) is carried out by means of an environment camera 5 of the medical visualization system 1 used as an internal tracking system 4 and at least one marker 23 (tracker, e.g., Mayfield clamp) arranged on the patient 22. The environment camera 5 has a detection range 6 that detects the patient 22 and the marker 23, as well as, in particular, the surroundings of the patient 22. The detection range 6 of the environment camera 5 is generally significantly larger than a detection range 7 of the main camera 2.

[0050] It can be provided that the determination of the camera pose 30 is carried out by means of an external tracking system 50 and at least one marker 23 arranged on the patient 22 and at least one marker 23 arranged on the main camera 2. The external tracking system 50 can, for example, be a navigation system (e.g., from the company NDI, Canada).

[0051] It can be provided that the reference coordinate system 20 is a coordinate system 41 of a robotic tripod 40, wherein the determination of the camera pose 30 in the patient coordinate system 21 is carried out based on position data of an actuator of the robotic tripod 40.

[0052] It can be provided that the initial patient registration comprises detecting and / or deriving a surface contour of at least the selected region, wherein the adaptation to the preoperatively acquired three-dimensional patient data 60 is carried out based on the detected and / or derived surface contour.

[0053] It can be provided that the initial patient registration includes a pose estimation of the main camera 2 relative to the patient 22. This can be done, for example, using the method described in J. Kang, W. Liu, W. Tu and L. Yang, YOLO-6D+: Single Shot 6D Pose Estimation Using Privileged Silhouette Information, 2020 International Conference on Image Processing and Robotics (ICIP), Negombo, Sri Lanka, 2020, pp. 1-6, doi: 10.1109 / ICIP48927.2020.9367354. Based on this, the transformation rule 10 is available and can then be optimized according to the method.

[0054] It can also be provided that for the initial patient registration, a recorded patient image 12 is displayed superimposed with a projection image 13 generated from the preoperatively recorded patient data 60 and the main camera 2 is controlled by a user based on the superimposed Fig. is arranged in a desired camera pose. The display is performed, for example, on a display device (not shown) of the medical visualization system.

[0055] It can be provided that a set of camera poses 30 is provided, wherein the minimization is carried out starting from all acquired patient images 12 and the corresponding generated two-dimensional projection images 13 of the set. In particular, the set can comprise at least two camera poses 30. Preferably, the set comprises further camera poses 30. For all image pairs, then - as exemplified by the Fig. 2 - the respective deviation 14 is determined and minimized by adjusting the parameters of the transformation rule 10. In particular, a cumulative deviation 14 is determined and minimized for the entire set.

[0056] It can be provided that markers 24-1, 24-2 are arranged on the patient 22 and / or selected areas are identified, wherein the minimization of the deviation 14 is carried out taking into account the markers 24-1, 24-2 and / or the selected areas.

[0057] It may be provided that at least one area in which the patient 22 is arranged is structurally illuminated during the acquisition of the patient image 12, wherein the minimization of the deviation 14 is carried out taking into account the structural illumination.

[0058] It can be provided that the at least one camera pose 30 of the main camera 2 is set automatically by means of the robotic tripod 40. For this purpose, the robotic tripod 40 is controlled in particular by means of the control device 3. For example, two or three different camera poses 30 (e.g., via different angular positions of an axis of the robotic tripod 40) of the main camera 2 can be provided.

[0059] The Fig. Figure 3 shows a schematic flow diagram to illustrate embodiments of the method for patient registration on a medical visualization system. The medical visualization system can, for example, be configured as described with reference to Fig. 1 described medical visualization system.

[0060] In a method step 100, an initial patient registration is performed, for example, by capturing at least one selected region of a patient using a capture device in a reference coordinate system and adapting preoperatively acquired three-dimensional patient data (e.g., CT or MRI), which are available with reference to a patient coordinate system, to the captured region and determining parameters of a transformation rule between the reference coordinate system and the patient coordinate system. Alternatively or additionally, the initial patient registration can also be performed in another way.

[0061] In a method step 101, a camera pose of a main camera is set. This can be done, for example, automatically using a robotic tripod or manually.

[0062] In a method step 102, the camera pose in the reference coordinate system is determined. This can be done using an internal tracking system (e.g., a field camera) or an external tracking system (e.g., a navigation system).

[0063] In a method step 103, a patient image is captured by the main camera in the set camera pose.

[0064] In a method step 104, it is checked whether further camera poses are planned. If this is the case, the process returns to step 101. If this is not the case, i.e., if all planned camera poses have already been used, the process continues with step 105.

[0065] In method step 105, a two-dimensional projection image of the preoperatively acquired three-dimensional patient data is determined based on the respective camera pose of the main camera, taking the transformation rule into account. For each acquired patient image, a corresponding two-dimensional projection image is then available via the respective camera pose. Overall, a set of pairs, each comprising a patient image paired with a projection image, is thus available.

[0066] In a method step 106, a deviation is determined for each pair of acquired patient images and corresponding projection images. For this purpose, feature recognition can be performed, for example, during which features and / or contours, etc., are recognized and used to determine the deviation. The deviation of the entire set is, in particular, a cumulative value of the deviations between the individual pairs within the set.

[0067] In process step 107, a check is made to determine whether the (cumulative) deviation is below a specified threshold. If this is the case, the process continues with step 109. If this is not the case, the process continues with step 108.

[0068] In method step 108, parameters of the transformation rule are adjusted to minimize the deviation. Known optimization methods can be used to determine the type and extent of adjustment. The process then returns to method step 106, which is performed taking the adjusted parameters of the transformation rule into account.

[0069] In method step 109, the adapted transformation rule is provided. For this purpose, the adapted transformation rule can be loaded, for example, into a memory or memory area of ​​the medical visualization system. Subsequently, the preoperatively acquired patient data can be overlaid with acquired patient images to augment and support a surgeon.

[0070] Further embodiments of the method have already been described with reference to the medical visualization system. List of reference symbols 1 medical visualization system 2 main cameras 3 Control device 3-1 Calculation device 3-2 Memory 4 internal tracking system 5 Surround camera 6 Detection range (surround camera) 7 Detection range (main camera) 10 Transformation rule 11 parameters 12 Patient image 13 Projection image 14 Deviation 15-x feature 16 superimposed image 17-x distance 20 Reference coordinate system 21 Patient coordinate system 22 patients 23 markers (trackers) 24-x marking 30 Camera Pose 40 robotic tripod 41 Coordinate system (robotic tripod) 50 external tracking system 60 preoperatively recorded three-dimensional patient data 100-109 procedural steps QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] EP 1 142 536 A1

[0005] Cited non-patent literature

[0000] J. Kang, W. Liu, W. Tu and L. Yang, YOLO-6D+: Single Shot 6D Pose Estimation Using Privileged Silhouette Information, 2020 International Conference on Image Processing and Robotics (ICIP), Negombo, Sri Lanka, 2020, pp. 1-6, doi

[0006] 10.1109 / ICIP48927.2020.9367354

[0006] https: / / docs.opencv.org / 4.x / d9 / d0c / group__calib3d.html

[0014] J. Kang, W. Liu, W. Tu and L. Yang, YOLO-6D+: Single Shot 6D Pose Estimation Using Privileged Silhouette Information, 2020 International Conference on Image Processing and Robotics (ICIP), Negombo, Sri Lanka, 2020, pp. 1-6, doi: 10.1109 / ICIP48927.2020.9367354 [0032, 0053]

Claims

[1] Method for patient registration on a medical visualization system (1), comprising: Carrying out an initial patient registration, wherein parameters of a transformation rule (10) between a reference coordinate system (20) and a patient coordinate system (21) are determined, further for at least one camera pose (30) of a main camera (2) of the medical visualization system (1): Determining the camera pose (30) in the reference coordinate system (20), capturing a patient image (12) using the main camera (2), and at least once: - generating a two-dimensional projection image (13) of preoperatively acquired three-dimensional patient data (60) starting from the determined camera pose (30) of the main camera (2) taking into account the transformation rule (10), and - minimizing a deviation (14) between the acquired patient image (12) and the corresponding generated two-dimensional projection image (13) by adapting parameters of the transformation rule (10); and Providing the adapted transformation rule (10). [2] Method according to claim 1, characterized by that features (15-x) are recognized in the acquired patient image (12) and the two-dimensional projection image (13) corresponding thereto, wherein the minimization of the deviation (14) is carried out on the basis of the recognized features (15-x). [3] Method according to claim 1 or 2, characterized by that the determination of the camera pose (30) is carried out by means of an environmental camera (5) of the medical visualization system (1) used as an internal tracking system (4) and at least one marker (23) arranged on the patient (22). [4] Method according to one of claims 1 or 2, characterized bythat the determination of the camera pose (30) is carried out by means of an external tracking system (50) and at least one marker (23) arranged on the patient (22) and at least one marker (23) arranged on the main camera (2). [5] Method according to claim 1 or 2, characterized by in that the reference coordinate system (20) is a coordinate system (41) of a robotic tripod (40), wherein the camera pose (30) in the reference coordinate system (20) is determined on the basis of position data of an actuator of the robotic tripod (40). [6] Method according to one of the preceding claims, characterized bythat the initial patient registration is carried out by detecting at least one selected area of ​​a patient (22) by means of a detection device in a reference coordinate system (20) and adapting the preoperatively detected three-dimensional patient data (60), which are present with reference to a patient coordinate system (21), to the detected area and determining parameters of the transformation rule (10) between the reference coordinate system (20) and the patient coordinate system (21). [7] Method according to claim 6, characterized by that the initial patient registration comprises detecting and / or deriving a surface contour of at least the selected area, wherein the adaptation to the preoperatively acquired three-dimensional patient data (60) is carried out on the basis of the detected and / or derived surface contour. [8] Method according to one of the preceding claims, characterized bythat the initial patient registration includes a pose estimation of the main camera (2) to the patient (22). [9] Method according to one of the preceding claims, characterized by that for the initial patient registration, a recorded patient image (12) is displayed superimposed with a projection image (13) generated on the basis of the preoperatively recorded patient data (60), and the main camera (2) is arranged in a desired pose by a user based on the superimposed images (12, 13). [10] Method according to one of the preceding claims, characterized by that a set of camera poses (30) is provided, wherein the minimization is carried out on the basis of all acquired patient images (12) and the two-dimensional projection images (13) of the set that correspond thereto. [11] Method according to one of the preceding claims, characterized bythat markings (24-x) are arranged on the patient (22) and / or selected areas are identified, wherein the minimization of the deviation (14) is carried out taking into account the markings (24-x) and / or the selected areas. [12] Method according to one of the preceding claims, characterized by that at least one area in which the patient (22) is arranged is structurally illuminated during the acquisition of the patient image (12), wherein the minimization of the deviation (14) is carried out taking into account the structural illumination. [13] Method according to one of the preceding claims, characterized by that the at least one camera pose (30) of the main camera (2) is set automatically by means of a robotic tripod (40). [14] Medical visualization system (1), comprising: a main camera (2), a control device (3), wherein the control device (3) is designed to to obtain and / or determine parameters of a transformation rule (10) between a reference coordinate system (20) and a patient coordinate system (21) as part of an initial patient registration, and for at least one camera pose (30) of the main camera (2): to obtain a camera pose (30) determined in the reference coordinate system (20) or to determine the camera pose (30), to cause the capture of a patient image (12) by means of the main camera (2) in the at least one camera pose (30) and / or to obtain a patient image (12) captured in the at least one camera pose (30), and at least once: - to generate a two-dimensional projection image (13) of preoperatively acquired three-dimensional patient data (60) starting from the determined camera pose (30) of the main camera (2) taking into account the transformation rule (10), and - to minimize a deviation (14) between the acquired patient image (12) and the corresponding generated two-dimensional projection image (13) by adapting parameters of the transformation rule (10); and to provide the adapted transformation rule (10).

Images