Method for patient registration on a medical visualization system and medical visualization system

Abstract

The invention relates to a method for patient registration on a medical visualization system (1), wherein an image (10) of a body part (21) of a patient (20) is captured by a camera (2) of the medical visualization system (1), wherein, based on the captured image (10), a three-dimensional surface profile (11) of the body part (21) is estimated in a coordinate system (30) of the camera (2) by means of a trained machine learning method (6) and / or a computer vision method (7), wherein a scaling factor of the three-dimensional surface profile (11) is determined and / or estimated, and wherein preoperatively captured three-dimensional patient data (22) which are present in a patient coordinate system (31) are adapted to the estimated three-dimensional surface profile (11),wherein, based on a resulting adaptation result, a transformation rule (T) is determined between the coordinate system (30) of the camera (2) and the patient coordinate system (31), wherein the adaptation and / or the determination of the transformation rule (T) are carried out taking into account the determined and / or estimated scaling factor, and wherein the determined transformation rule (T) is provided. Furthermore, the invention relates to a medical visualization system (1).

Description

[0001] The invention relates to a method for patient registration on a medical visualization system and a medical visualization system.

[0002] To use preoperatively acquired data, such as computed tomography (CT) data or magnetic resonance imaging (MRI) data, in surgery, a relationship between a reference coordinate system and a patient, or between a patient coordinate system during the operation, must be known. Determining this relationship, or a transformation rule describing the relationship between the coordinate systems, is usually performed before the operation and is referred to as patient registration. Patient registration specifically involves determining the relative pose (translation and rotation) of a patient in another coordinate system that does not move relative to the patient and therefore serves as the reference coordinate system.

[0003] To register the patient, for example, a surface of the patient and a pose of a reference object located in the reference coordinate system can be determined, e.g. a unique tracker or a unique marker that is firmly attached to the patient (e.g. a Mayfield clamp). The patient's surface can be determined in the reference coordinate system, for example, using stereoscopic topography determination (e.g. derived from stereoscopic image data from a surgical microscope) or using contact-based or contactless probing devices, such as a Brainlab Softtouch or a Brainlab Z-Touch (both from Brainlab AG). The pose of the reference object (the marker or tracker on the patient) can be recorded and determined using a navigation system (e.g. from NDI, Canada) or, alternatively, using an internal tracking system of a medical visualization system.

[0004] From EP 1 142 536 A1 a method for referencing a patient or a body part of a patient in a camera-supported, medical navigation system is known, comprising the following steps: the body part of the patient to be referenced is brought into the detection range of a navigation system supported by at least two cameras, which computer-aidedly detects the three-dimensional spatial positions of light markers, by means of a light beam light markers are generated on the surface of the body part to be referenced, the three-dimensional position of which is determined by the camera-supported navigation system, by means of the position data for the light markers the spatial position of the surface of the body part to be referenced is determined.

[0005] Further methods for patient registration are known, for example, from DE 10 2022 202 555 A1, DE 10 2022 100 626 A1 and US 2023 / 0 074 362 A1.

[0006] T. Mane et al., Single-camera 3D head fitting for mixed reality clinical applications, Computer Vision and Image Understanding, Vol. 218, April 2022, 103384, https: / / doi.org / 10.1016 / j.cviu.2022.103384, provides an overview of methods for estimating a three-dimensional surface gradient.

[0007] Computer vision methods for estimating a three-dimensional surface contour are known from D. Jiang et al., Efficient 3D reconstruction for face recognition Pattern Recognition, Vol. 38, No. 6, June 2005, pp. 787-798, https: / / doi.org / 10.1016 / j.patcog.2004.11.004.

[0008] Surface matching methods are known from Christian Wöhler, 3D Computer Vision - Efficient Methods and Applications, Springer Berlin, Heidelberg, 2009, https: / / doi.org / 10.1007 / 978-3-642-01732-2.

[0009] The invention is based on the object of improving a method for patient registration on a medical visualization system and a medical visualization system.

[0010] The object is achieved according to the invention by a method having the features of patent claim 1 and a medical visualization system having the features of patent claim 15. Advantageous embodiments of the invention emerge from the subclaims.

[0011] One of the basic concepts of the invention is to estimate a three-dimensional surface contour of a patient's body part based on an image acquired by a camera of the medical visualization system using a trained machine method and / or a computer vision method in a coordinate system of the camera. The camera is, for example, a main observer camera of the medical visualization system. Based on this, the estimated three-dimensional surface contour of the body part is adapted to preoperatively acquired three-dimensional patient data of the body part (e.g., computed tomography (CT) data or magnetic resonance imaging (MRI) data, etc.), for example by means of surface matching.Since the preoperatively acquired three-dimensional patient data is available in a patient coordinate system, an adaptation result allows the determination of a transformation rule between the camera's coordinate system and the patient's coordinate system. The preoperatively acquired three-dimensional patient data can then be located in the camera's coordinate system using the determined transformation rule. This allows, for example, the preoperatively acquired patient data to be correctly overlaid with images acquired with the camera. When adapting and / or determining the transformation rule, a specific and / or estimated scaling factor of the three-dimensional surface contour is taken into account.

[0012] In particular, a method for patient registration on a medical visualization system is provided, wherein an image of a body part of a patient is captured by a camera of the medical visualization system, wherein, based on the captured image, a three-dimensional surface profile of the body part is estimated in a coordinate system of the camera by means of a trained machine learning method and / or a computer vision method, wherein a scaling factor of the three-dimensional surface profile is determined and / or estimated, and wherein preoperatively acquired three-dimensional patient data, which are present in a patient coordinate system, are adapted to the estimated three-dimensional surface profile, wherein, based on a resulting adaptation result, a transformation rule between the coordinate system of the camera and the patient coordinate system is determined,wherein the adaptation and / or the determination of the transformation rule are carried out taking into account the determined and / or estimated scaling factor, and wherein the determined transformation rule is provided.,

[0013] Furthermore, in particular, a medical visualization system is created, comprising a camera configured to capture an image of a body part of a patient, and a control device, wherein the control device is configured to estimate a three-dimensional surface profile of the body part in a coordinate system of the camera based on the captured image by means of a trained machine learning method and / or a computer vision method, to determine and / or estimate a scaling factor of the three-dimensional surface profile, to adapt preoperatively acquired three-dimensional patient data, which are present in a patient coordinate system, to the estimated three-dimensional surface profile, to determine a transformation rule between the coordinate system of the camera and the patient coordinate system based on a resulting adaptation result,wherein the adaptation and / or the determination of the transformation rule are carried out taking into account the determined and / or estimated scaling factor, and the determined transformation rule is provided.

[0014] The method and the medical visualization system, in particular, enable patient registration using only a single (monoscopic) image. This can save effort, time, and costs, since stereoscopic acquisition and / or the successive point-by-point acquisition of the three-dimensional surface contour of the body part can be omitted. Patient registration can thus be carried out more efficiently and automatically. The method and the medical visualization system, in particular, enable automated, marker-free patient registration. The method and the medical visualization system, in particular, allow the provision of a robotic, geometrically calibrated visualization system.

[0015] The camera is, in particular, a main observer camera of the medical visualization system. However, in principle, another camera of the medical visualization system can also be used. The camera is, in particular, a monoscopic camera that captures monoscopic images. The camera is, in particular, a (true) color camera (RGB). However, the camera can also, in principle, operate at other wavelengths (e.g., NIR).

[0016] The medical visualization system is, in particular, a surgical microscope. However, the medical visualization system can also be a microscope used for medical examinations and / or diagnostic purposes. Furthermore, the medical visualization system can also be a handheld or hand-guided micro-inspection tool.

[0017] The trained machine learning method can, for example, comprise a trained neural network or be designed as such. The trained neural network is, in particular, a deep neural network, especially a convolutional neural network (CNN).

[0018] The machine learning method is trained in particular to estimate a three-dimensional surface profile of the patient's body part depicted in the acquired image based on the acquired (particularly monoscopic) image. The three-dimensional surface profile can be estimated and / or described, for example, in the form of a three-dimensional mesh, consisting in particular of grid points. The machine learning method is or was trained, in particular during a training phase, with the aid of training data by means of supervised or unsupervised learning. The training data comprises pairs in which an image of a body part is paired with a three-dimensional surface profile (as ground truth).In particular, it is provided that a scaling factor of the three-dimensional surface structure is taken into account, so that the three-dimensional surface profile describes the real dimensions of the body part in (particularly metric) coordinates. Alternatively, the three-dimensional surface profile can also be estimated in relative (arbitrarily chosen) coordinates. Such a surface profile can then be scaled to the real dimensions using a suitable scaling factor. The machine learning method is then trained, in particular, by means of supervised learning with the aid of the training data set. The training data can be created based on acquired images and corresponding surface profiles. Alternatively or additionally, simulations can also be used to create pairs of images and associated surface profiles.For example, the machine learning method can be based on one of the methods described in T. Mane et al., Single-camera 3D head fitting for mixed reality clinical applications, Computer Vision and Image Understanding, Vol. 218, April 2022, 103384, https: / / doi.org / 10.1016 / j.cviu.2022.103384.

[0019] A computer vision method can, for example, be based on one of the methods described in D. Jiang et al., Efficient 3D reconstruction for face recognition Pattern Recognition, Vol. 38, No. 6, June 2005, pp. 787-798, https: / / doi.org / 10.1016 / j.patcog.2004.11.004. In particular, it is envisaged that a database containing three-dimensional body models is used to find the corresponding three-dimensional body model based on two-dimensional features in the captured image.

[0020] The scaling factor, in particular, allows relative distances of the three-dimensional surface contour to be assigned to real distances or a real unit of length (e.g., in the metric system of units with the unit meter). The scaling factor, in particular, contains or describes such an assignment rule.

[0021] It can be provided that a camera pose of the camera is determined in a reference coordinate system. The coordinate system of the camera then corresponds to the reference coordinate system. For this purpose, an internal tracking system can be used, which, for example, comprises a surround camera of the medical visualization system and a marker arranged on the patient (also referred to as a tracker, e.g. a Mayfield clamp). A relative pose between the surround camera and the camera is known or can be determined using extrinsic calibration. The surround camera can determine a pose of the marker arranged on the patient. Based on this, the camera pose can also be determined using extrinsic calibration. Furthermore, an external tracking system can also be used, which detects both a marker arranged on the patient and a marker arranged on the camera and determines their respective pose.Starting from a known relative pose of the marker to the camera, the camera pose can be determined.

[0022] Preoperatively acquired patient data includes, for example, data acquired using a computed tomography (CT) or magnetic resonance imaging (MRI) scanner. However, other preoperatively acquired patient data can also be used. In particular, the term preoperatively acquired patient data should also include non-acquired, but rather simulated or otherwise generated three-dimensional patient data. The preoperatively acquired three-dimensional patient data is fed to and / or received from the control device.

[0023] Surface matching can be performed, for example, using a method as described in Christian Wöhler, 3D Computer Vision - Efficient Methods and Applications, Springer Berlin, Heidelberg, 2009, https: / / doi.org / 10.1007 / 978-3-642-01732-2.

[0024] Using the specified transformation rule, coordinates of the camera's coordinate system or a reference coordinate system can be converted into coordinates of the patient's coordinate system and vice versa. Using the transformation rule, the preoperatively acquired three-dimensional patient data can be located in the camera's coordinate system or the reference coordinate system. This makes it possible, in particular, to correctly overlay the preoperatively acquired patient data with a captured image from the camera for augmentation. The transformation rule can be expressed mathematically as follows: PmV=RmV / Patient*PPatient+TmV / Patient, where P mV has the dimension 3x1 and denotes a point in the coordinate system of the camera (or in the reference coordinate system), P Patient also has the dimension 3x1 and denotes a point in the patient coordinate system, R mV / Patient is a matrix of dimension 3x3 and describes as a linear mapping a rotation and T mV / Patient denotes a translation vector of dimension 3x1.

[0025] Parts of the medical visualization system, in particular the control device, can be implemented individually or collectively as a combination of hardware and software, for example, as program code executed on a microcontroller or microprocessor. However, it can also be provided that parts are implemented individually or collectively as an application-specific integrated circuit (ASIC) and / or a field-programmable gate array (FPGA). The control device comprises, in particular, at least one computing device and at least one memory.

[0026] The procedure and the medical visualization system can be used in particular in the following areas: neurosurgery, spinal surgery, dental surgery, ophthalmic surgery, etc.

[0027] In one embodiment, the scaling factor of the estimated three-dimensional surface profile is determined based on the settings of the camera and / or optical elements of the medical visualization system at the time the image is acquired. The settings of the camera and / or the optical elements relate in particular to operating parameters during image acquisition, in particular a working distance and / or a focus and / or a magnification. Using the operating parameters, a distance between points in the acquired image can be determined, and the scaling factor can be determined from this.

[0028] In a further embodiment, it is provided that the scaling factor is determined as follows: querying and / or determining at least one focus value and one magnification value as settings of the medical visualization system; determining a ratio between a relative distance of selected points of the three-dimensional surface profile and an absolute distance of the selected points determined from a known pixel size of an image sensor of the camera, taking into account at least the focus value and the magnification value; providing the determined ratio as a scaling factor. A physical size corresponding to a pixel in the captured image (e.g., in meters or millimeters) is determined by a focus value (or working distance), a magnification value, and a physical size of a pixel of an image sensor of the camera.If two three-dimensional points from the estimated surface contour are considered, which correspond to image elements in the acquired image, and it is also assumed, as a first approximation, that the two points lie in a focal plane corresponding to the focus value, an absolute distance between the three-dimensional points can be determined. Based on this distance, the scaling factor for the three-dimensional surface contour can be determined as the ratio between the relative and absolute distance between the points. The three-dimensional surface contour can then be provided, in particular, with an absolute length unit (e.g., with the unit meter).

[0029] In one embodiment, the scaling factor of the estimated three-dimensional surface profile is determined based on a marker arranged on the patient, captured by the camera or a surrounding camera of the medical visualization system, and at least one marker captured by the camera or the surrounding camera. This allows the scaling factor to be determined based on known properties (in particular, known dimensions) of the at least one marker, which in particular has a known 3D geometry.

[0030] In a further embodiment, the following steps are carried out: detecting the at least one marker in the image captured by the camera; determining properties of the detected at least one marker based on the captured image; determining a relationship between the determined properties and known properties of the detected at least one marker; providing the determined relationship as a scaling factor. Based on the known properties (in particular known dimensions), the scaling factor for the three-dimensional surface profile can be determined.Since the image elements of the image correspond directly to the estimated three-dimensional surface contour, the known properties (in particular the known dimensions) of the marker can be used to infer the properties (in particular the dimensions) of the three-dimensional surface contour, so that the scaling factor can be determined via the ratio. For example, a relative distance of the marker in the image can be compared to a (known) absolute distance of the marker. In a simple case, the marker can include a scale attached to the patient, from which a length measurement can be directly determined.

[0031] In another further embodiment, it is provided that the following steps are carried out for this purpose: Detecting the at least one marker in an environment image captured by the environment camera; Projecting the detected at least one marker into the captured image taking into account a known relative pose between the environment camera and the camera; Determining properties of the projected at least one marker based on the captured image; Determining a relationship between the determined properties and known properties of the projected at least one marker; Providing the determined relationship as a scaling factor. Since the relative pose between the camera (orIf the coordinate system of the camera (the camera coordinate system) and the coordinate system of the surrounding camera are known (or can be determined by means of extrinsic calibration), a marker detected by the surrounding camera can be projected to scale into the camera image. Based on this and known properties of the marker (in particular its dimensions), the scaling factor of the estimated three-dimensional surface contour can be determined, since the image elements of the image correspond to the estimated three-dimensional surface contour. In other words, the required metric information originates in particular from the marker (i.e., in particular, a known 3D geometry of the marker and an estimated 6D pose with respect to the camera coordinate system).The 3D positions of the marker are projected onto the camera image using the camera's intrinsic calibration parameters, meaning that the 2D position of the marker is determined specifically in the image. This creates a mapping between the 3D and 2D marker positions, which can be used to calculate the scaling factor.

[0032] In one embodiment, a camera pose is determined in a reference coordinate system, and the transformation rule between the reference coordinate system and the patient coordinate system is determined. This allows direct reference to the reference coordinate system.

[0033] In one embodiment, after a camera pose is changed, an image of the patient's body part is acquired again, and the adaptation and determination of the transformation rule are performed again. This allows the transformation rule to be determined anew for each new camera pose. In this way, the accuracy of the transformation rule can be improved after a change in the camera pose.

[0034] In one embodiment, it is provided that the trained machine learning method and / or the computer vision method comprises at least a first method and at least one second method, wherein the at least one first method determines distinguished points on the body part in the captured image, and wherein the at least one second method estimates the three-dimensional surface profile based on the determined distinguished points. In other words, the step in which the three-dimensional surface profile of the body part is estimated in a coordinate system of the camera can be carried out in particular by determining distinguished points on the body part in the captured image and estimating the three-dimensional surface profile based on the determined distinguished points. This allows a step-by-step evaluation of the captured image to be realized.In particular, this can increase accuracy, as the individual procedures can be specialized for specific subtasks. Selected points are, in particular, rigid and / or bony regions and / or predefined or predetermined regions. In other words, selected points do not coincide with regions on the body part that are subject to significant change due to the mobility / flexibility of the external tissue (e.g., skin). For craniotomies, for example, prominent points in the area of ​​the nasal bone, frontal bone, and / or zygomatic bone can be used as selected points.The estimation of the three-dimensional surface contour using the at least one second method can be carried out, for example, by assigning the determined points to one or more three-dimensional models of the body part (which can also be referred to as "mapping") and deforming them in three dimensions. It can be provided that a three-dimensional model of the respective body part is provided to the at least one second method and / or that such a model is used by the at least one second method to estimate the three-dimensional surface contour.

[0035] In one embodiment, the camera is a surround camera of the medical visualization system. This allows a larger detection area to be captured than, for example, with a main observer camera. In particular, the larger detection area allows for the simultaneous detection of a marker placed on the patient (e.g., a Mayfield clamp).

[0036] In one embodiment, at least one further image of the patient's body part is captured in at least one other camera pose of the camera or by means of another camera of the medical visualization system arranged in the at least one other camera pose, wherein the adaptation takes place taking into account the at least one other captured image. This can improve the accuracy of determining the transformation rule. The camera pose and the at least one other camera pose of the camera can be adjusted, for example, using a robotic tripod of the medical visualization system.

[0037] In one embodiment, it is provided that, based on the at least one further acquired image, a three-dimensional surface profile of the body part is also estimated using the trained machine learning method and / or the computer vision method. The three-dimensional surface profile of the body part estimated from the at least one further image is fused with the three-dimensional surface profile of the body part estimated from the acquired image, with the adaptation taking place based on the fused three-dimensional surface profile. This allows a three-dimensional surface profile to be estimated separately for each of the acquired images; only then are the results, i.e., the individually estimated three-dimensional surface profiles, fused with one another.By fusing the estimated three-dimensional surface profiles, a more accurate estimation of the three-dimensional geometry (or surface profile) can be achieved.

[0038] In one embodiment, it is provided that, based on the acquired at least one further image, a three-dimensional surface profile of the body part is also estimated using the trained machine learning method and / or the computer vision method. The adaptation to the preoperatively acquired three-dimensional patient data is also performed for the three-dimensional surface profile estimated based on the acquired at least one further image. The transformation rule is determined taking the adaptation results into account. This allows the transformation rule to be determined based on each individual adaptation.

[0039] In one embodiment, the camera and / or the at least one additional camera for capturing the images is / are arranged in at least two camera poses using a robotic tripod of the medical visualization system. This allows images to be adjusted and captured in the multiple camera poses, particularly automatically. Automated capture allows the images to be captured systematically and reproducibly. Errors caused by manually setting a camera pose can be avoided.

[0040] The invention will be explained in more detail below using preferred embodiments with reference to the figures. Fig. 1 a schematic representation to illustrate embodiments of the medical visualization system and the method; Fig. 2 a schematic representation to illustrate the estimation of the three-dimensional surface profile; Fig. 3 is a schematic diagram illustrating an embodiment of the medical visualization system and method; Fig. 4 is a schematic diagram illustrating an embodiment of the medical visualization system and method; Fig. 5 a schematic flow diagram to illustrate embodiments of the method.

[0041] The Fig. 1 shows a schematic representation to illustrate embodiments of the medical visualization system 1. The medical visualization system 1 comprises a (in particular a single, in particular monoscopic) camera 2 and a control device 3. Furthermore, the medical visualization system 1 can comprise a robotic stand 40, at the distal end of which the camera 2 is arranged. The robotic stand 40 is controlled, for example, by the control device 3. The method described in this disclosure is explained in more detail below using the medical visualization system 1.

[0042] Camera 2 is set up to Fig. of a body part 21 of a patient 20. For this purpose, the body part 21, for example the head of the patient 20, is arranged and recorded in a recording area 4 of the camera 2. For this purpose, the camera 2 is arranged in a camera pose 50, for example by means of the robotic tripod 40. The camera 2 records a Fig. from the body part 21 and feeds it to the control device 3.

[0043] The control device 3 comprises at least one computing device 3-1 and at least one memory 3-2. The control device 3 is designed to, based on the detected Fig. by means of a trained machine learning method 6 and / or a computer vision method 7 a three-dimensional surface profile 11 (see also Fig. 2) of the body part 21 in a coordinate system 30 of the camera 2. For this purpose, the trained machine learning method 6 is provided by the control device 3. In particular, a structural description and parameters of the trained machine learning method 6 are stored in the memory 5. Alternatively or additionally, the computer vision method 7 is provided by the control device 3, with a description of the method being stored in the memory 3-2.

[0044] The control device 3 is further configured to determine and / or estimate a scaling factor of the three-dimensional surface profile 11.

[0045] The control device 3 is further configured to adapt preoperatively acquired three-dimensional patient data 22, which are present in a patient coordinate system 31, to the estimated three-dimensional surface profile 11. This is done, for example, by means of surface matching.

[0046] Furthermore, the control device 3 is configured to determine a transformation rule T between the coordinate system 30 of the camera 2 and the patient coordinate system 31 based on a resulting adaptation result.

[0047] It is intended that the adaptation and / or determination of the transformation rule T is carried out taking into account the determined and / or estimated scaling factor.

[0048] The control device 3 is configured to provide the specific transformation rule T. In particular, the provision may comprise loading the specific transformation rule T into the memory 3-2, so that the transformation rule T can subsequently be used, in particular to augment the preoperatively acquired patient data 22 with an acquired Fig. of body part 21.

[0049] The Fig. Figure 2 shows schematic representations to illustrate the estimation of the three-dimensional surface profile 11. The three-dimensional surface profile 11 is estimated in the camera's coordinate system 30. The three-dimensional surface profile 11 is subsequently adapted to the preoperatively acquired three-dimensional patient data, and the transformation rule is determined based on the resulting adaptation result. The adaptation and / or determination of the transformation rule T are carried out taking into account the determined and / or estimated scaling factor.

[0050] It can be provided that the scaling factor of the estimated three-dimensional surface profile 11 is determined based on settings of the camera 2 and / or optical elements of the medical visualization system 1 at the time of acquiring the image. In particular, the settings can include operating parameters, such as a working distance and / or a focus and / or a magnification. In particular, the settings include a focus and a magnification. Based on these, metric coordinates of the three-dimensional surface profile 11 can be determined.

[0051] For this purpose, it can be provided that the scaling factor is determined as follows: querying and / or determining at least one focus value and one magnification value as settings of the medical visualization system 1; determining a ratio between a relative distance of selected points of the three-dimensional surface profile 11 and an absolute distance of the selected points determined from a known pixel size of an image sensor of the camera 2, taking into account at least the focus value and the magnification value; providing the determined ratio as a scaling factor.

[0052] It can be provided that the scaling factor of the estimated three-dimensional surface profile 11 is calculated starting from a value determined by means of the camera 2 or an environment camera 8 ( Fig. 1) of the medical visualization system 1 and at least one marker 23 arranged on the patient 20, captured by the camera 2 or by the environment camera 8, and is taken into account when determining the transformation rule T. In particular, a pose of the camera 2 relative to the marker 23 can be determined using the environment camera 8. This requires, in particular, an extrinsic calibration, i.e., the determination of a relative pose between the environment camera 8 and the camera 2. The coordinate system 30 of the camera 2 is then defined, in particular, via the pose of the marker 23.

[0053] It can be provided that the following steps are carried out for this purpose: Detection of the at least one marker 23 in the image captured by the camera 2 Fig. ; Determining properties of the detected at least one marker 23 based on the detected Fig. ; Determining a relationship between the determined properties and known properties of the detected at least one marker 23; Providing the determined relationship as a scaling factor. The properties relate in particular to at least dimensions in at least one dimension of the at least one marker 23. The relationship can be determined, for example, between a relative distance of the marker 23 in the image and a (known) absolute distance of the marker 23.

[0054] It can be provided that the following steps are carried out for this purpose: Detecting the at least one marker 23 in an environment image captured by the environment camera 8; Projecting the detected at least one marker 23 into the captured Fig. taking into account a known relative pose between the environment camera 8 and the camera 2; determining properties of the projected at least one marker 23 based on the detected Fig. ; Determining a relationship between the determined properties and known properties of the projected at least one marker 23; Providing the determined relationship as a scaling factor. The properties relate in particular to at least dimensions in at least one dimension of the at least one marker 23.

[0055] It can be provided that a camera pose 50 of the camera 2 is determined in a reference coordinate system 32, wherein the transformation rule T between the reference coordinate system 32 and the patient coordinate system 31 is determined. This can be done by means of an internal tracking system, for example by means of an environment camera 8 and a marker 23 arranged on the patient 20, as already described above. Alternatively or additionally, the camera pose 50 can also be determined by means of an external tracking system 60 (e.g. from NDI, Canada) and a marker 23 arranged on the camera 2. The three-dimensional surface profile 11 is then estimated in the reference coordinate system 32.In particular, it is provided that the determined camera pose 50 is fed to the trained machine learning method 6 and / or the computer vision method 7 as input data so that they can determine the three-dimensional surface profile 11 with reference to the reference coordinate system 32.

[0056] It can be provided that after changing the camera pose 50 of the camera 2, another Fig. of the body part 21 of the patient 20 is detected and the adaptation and determination of the transformation rule T are carried out again. In particular, this can be done with each change of the camera pose 50.

[0057] It can be provided that the trained machine learning method 6 and / or the computer vision method 7 comprises at least a first method and at least one second method, wherein the at least one first method awards points 24 ( Fig. 2) on the body part 21 in the recorded Fig. and wherein the at least one second method estimates the three-dimensional surface profile 11 based on the determined distinguished points 24. Some distinguished points 24, such as a nose tip, a nose root or the eyebrows, are exemplary and schematically shown in the Fig. 2. The identified points 24 are transferred to the at least one second method, and the at least one second method estimates the three-dimensional surface profile 11 based thereon, in particular taking scaling into account.

[0058] It can be provided that the camera 2 is a peripheral camera 8 of the medical visualization system 1. In other words, the method can also be carried out using the peripheral camera 8. The procedure is fundamentally analogous. In particular, the peripheral camera 8 is also a monoscopic camera that captures monoscopic images.

[0059] It may be provided that at least one further Fig. of the body part 21 of the patient 20 in at least one other camera pose 50 of the camera 2 or by means of a further camera (not shown) of the medical visualization system 1 arranged in the at least one other camera pose 50, wherein the adaptation is carried out taking into account the recorded at least one further Fig. occurs.

[0060] It may be provided that, based on the recorded data, at least one further Fig. by means of the trained machine learning method 6 and / or the computer vision method 7, a three-dimensional surface profile 11-2 of the body part 21 is also estimated, wherein the surface profile 11-2 of the body part 21 is determined from the at least one further Fig. estimated three-dimensional surface profile 11-2 of the body part 21 with the one from the recorded Fig. estimated three-dimensional surface profile 11-1 of the body part 21, wherein the adaptation is carried out starting from the fused three-dimensional surface profile 11-f. This embodiment is shown schematically in the Fig. 3 illustrates this.

[0061] It may be provided that, based on the recorded data, at least one further Fig. by means of the trained machine learning method 6 and / or the computer vision method 7, a three-dimensional surface profile 11-2 of the body part 21 is also estimated, wherein the adaptation to the preoperatively acquired three-dimensional patient data 22 is additionally also carried out for the at least one further Fig. estimated three-dimensional surface profile 11-2, wherein the determination of the transformation rule T is carried out taking into account the adaptation results. In particular, all existing adaptation results are taken into account when determining the transformation rule T. This embodiment is shown schematically in the Fig. 4 illustrates this.

[0062] It can further be provided that the camera 2 and / or the at least one further camera for capturing the Fig. by means of the robotic tripod 40 ( Fig. 1) of the medical visualization system 1 is or are arranged in at least two camera poses 50.

[0063] The Fig. Figure 5 shows a schematic flow diagram to illustrate embodiments of the method for patient registration on a medical visualization system. The method can be carried out, for example, using a medical visualization system, as described with reference to Fig. 1 was described.

[0064] In a method step 100, an image of a patient's body part is captured using a (particularly monoscopic) camera of the medical visualization system. A camera pose is known in a camera coordinate system during the capture of the image.

[0065] In a method step 101, based on the captured image, a three-dimensional surface contour of the body part is estimated in a camera coordinate system using a trained machine learning method and / or a computer vision method. The three-dimensional surface contour is estimated in particular with reference to the camera coordinate system, i.e., following the estimation, coordinates of points of the three-dimensional surface contour are available in the camera coordinate system. As already described above, the camera coordinate system can also be a reference coordinate system.

[0066] In a method step 102, a scaling factor for the three-dimensional surface profile is determined and / or estimated. Method step 102 can, in principle, also be part of method step 101.

[0067] In a method step 103, preoperatively acquired three-dimensional patient data, which are available in a patient coordinate system, are adapted to the estimated three-dimensional surface contour. This is done, in particular, by means of surface matching.

[0068] In a method step 104, based on the resulting adaptation result, a transformation rule between the camera coordinate system and the patient coordinate system is determined. The transformation rule includes, in particular, a coordinate transformation between the camera coordinate system (or the reference coordinate system) and the patient coordinate system. During the determination process, parameters of this coordinate transformation are determined.

[0069] The adaptation in method step 103 and / or the determination of the transformation rule (T) in method step 104 are carried out taking into account the scaling factor determined and / or estimated in method step 102.

[0070] In a method step 105, the determined transformation rule is provided. Providing may, in particular, include loading the transformation rule into a memory of the medical visualization system so that the transformation rule can subsequently be used.

[0071] In an optional method step 106, it can be provided that the determined transformation rule is used to overlay the preoperatively acquired patient data with an acquired image, thereby providing an augmented image. This can assist a user of the medical visualization system, in particular a surgeon, during a procedure and / or examination.

[0072] Further embodiments of the method have already been described with reference to the medical visualization system. List of reference symbols 1 medical visualization system 2 cameras 3 Control device 3-1 Calculation device 3-2 Memory 4 Detection range 6 trained machine learning algorithm 7 Computer Vision Methods 8 Surround camera 10 captured image 11 three-dimensional surface profile 11-1 three-dimensional surface profile 11-2 three-dimensional surface profile 11-f fused three-dimensional surface profile 12 further illustration 20 patients 21 body part 22 preoperatively recorded three-dimensional patient data 23 markers 24 excellent point 30 Coordinate system (camera) 31 Patient coordinate system 32 Reference coordinate system 40 robotic tripod 50 Camera Pose 60 external tracking system 100-106 procedural steps T Transformation rule QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] EP 1 142 536 A1

[0004] DE 10 2022 202 555 A1

[0005] DE 10 2022 100 626 A1

[0005] US 2023 / 0 074 362 A1

[0005] Cited non-patent literature

[0000] T. Mane et al., Single-camera 3D head fitting for mixed reality clinical applications, Computer Vision and Image Understanding, Vol. 218, April 2022, 103384, https: / / doi.org / 10.1016 / j.cviu.2022.103384 [0006, 0018] D. Jiang et al., Efficient 3D reconstruction for face recognition Pattern Recognition, Vol. 38, Nr. 6, Juni 2005, S. 787-798, https: / / doi.org / 10.1016 / j.patcog.2004.11.004 [0007, 0019] Christian Wöhler, 3D Computer Vision - Efficient Methods and Applications, Springer Berlin, Heidelberg, 2009, https: / / doi.org / 10.1007 / 978-3-642-01732-2 [0008, 0023]

Claims

[1] Method for patient registration on a medical visualization system (1), wherein an image (10) of a body part (21) of a patient (20) is captured by means of a camera (2) of the medical visualization system (1), wherein, starting from the captured image (10), a three-dimensional surface profile (11) of the body part (21) is estimated in a coordinate system (30) of the camera (2) by means of a trained machine learning method (6) and / or a computer vision method (7), wherein a scaling factor of the three-dimensional surface profile (11) is determined and / or estimated, and wherein preoperatively recorded three-dimensional patient data (22), which are present in a patient coordinate system (31), are adapted to the estimated three-dimensional surface profile (11), wherein, based on a resulting adaptation result, a transformation rule (T) is determined between the coordinate system (30) of the camera (2) and the patient coordinate system (31), wherein the adaptation and / or determination of the transformation rule (T) is carried out taking into account the determined and / or estimated scaling factor, and where the particular transformation rule (T) is provided. [2] Method according to claim 1, characterized by that the scaling factor of the estimated three-dimensional surface profile (11) is determined based on settings of the camera (2) and / or optical elements of the medical visualization system (1) at the time of capturing the image (10). [3] Method according to claim 2, characterized by that the scaling factor is determined as follows: - querying and / or determining at least one focus value and one magnification value as settings of the medical visualization system (1); - determining a ratio between a relative distance of selected points of the three-dimensional surface profile (11) and an absolute distance of the selected points determined from a known pixel size of an image sensor of the camera (2) taking into account at least the focus value and the magnification value; - Providing the specific ratio as a scaling factor. [4] Method according to one of the preceding claims, characterized bythat the scaling factor of the estimated three-dimensional surface profile (11) is determined on the basis of a marker (23) arranged on the patient (20) and detected by the camera (2) or the surroundings camera (8) of the medical visualization system (1) and at least one marker (23). [5] Method according to claim 4, characterized by that the following steps are carried out: - detecting the at least one marker (23) in the image (10) captured by the camera (2); - determining properties of the detected at least one marker (23) based on the captured image (10); - determining a relationship between the determined properties and known properties of the detected at least one marker (23); - Providing the specific ratio as a scaling factor. [6] Method according to claim 4, characterized bythat the following steps are carried out: - detecting the at least one marker (23) in an image of the surroundings captured by the surroundings camera (8); - Projecting the detected at least one marker (23) into the captured image (10) taking into account a known relative pose between the environment camera (8) and the camera (2); - determining properties of the projected at least one marker (23) based on the captured image (10); - determining a relationship between the determined properties and known properties of the projected at least one marker (23); - Providing the specific ratio as a scaling factor. [7] Method according to one of the preceding claims, characterized bythat a camera pose (50) of the camera (2) is determined in a reference coordinate system (32), wherein the transformation rule (T) between the reference coordinate system (32) and the patient coordinate system (31) is determined. [8] Method according to one of the preceding claims, characterized by that after changing a camera pose (50) of the camera (2), an image (10) of the body part (21) of the patient (20) is again acquired and the adaptation and determination of the transformation rule (T) are carried out again. [9] Method according to one of the preceding claims, characterized byin that the trained machine learning method (6) and / or the computer vision method (7) comprises at least a first method and at least one second method, wherein the at least one first method determines distinguished points (24) on the body part (21) in the captured image (10), and wherein the at least one second method estimates the three-dimensional surface profile (11) based on the determined distinguished points (24). [10] Method according to one of the preceding claims, characterized by that the camera (2) is an environment camera (8) of the medical visualization system (1). [11] Method according to one of the preceding claims, characterized bythat at least one further image (12) of the body part (21) of the patient (20) is captured in at least one other camera pose (50) of the camera (2) or by means of a further camera of the medical visualization system (1) arranged in the at least one other camera pose (50), wherein the adaptation takes place taking into account the captured at least one further image (12). [12] Method according to claim 11, characterized bythat, starting from the at least one further image (12) acquired, a three-dimensional surface profile (11-2) of the body part (21) is also estimated by means of the trained machine learning method (6) and / or the computer vision method (7), wherein the three-dimensional surface profile (11-2) of the body part (21) estimated from the at least one further image (12) is fused with the three-dimensional surface profile (11-1) of the body part (21) estimated from the acquired image (10), wherein the adaptation is carried out starting from the fused three-dimensional surface profile (11-f). [13] Method according to claim 11, characterized bythat, starting from the at least one further image (12) acquired, a three-dimensional surface profile (11-2) of the body part (21) is also estimated by means of the trained machine learning method (6) and / or the computer vision method (7), wherein the adaptation to the preoperatively acquired three-dimensional patient data (22) is additionally also carried out for the three-dimensional surface profile (11-2) estimated starting from the at least one further image (12) acquired, wherein the determination of the transformation rule (T) takes place taking into account the adaptation results. [14] Method according to one of claims 11 to 13, characterized by that the camera (2) and / or the at least one further camera for capturing the images (10, 12) is arranged in at least two camera poses (50) by means of a robotic tripod (40) of the medical visualization system (1). [15] Medical visualization system (1), comprising: a camera (2) configured to capture an image (10) of a body part (21) of a patient (20), and a control device (3), wherein the control device (3) is arranged to based on the captured image (10) by means of a trained machine learning method (6) and / or a computer vision method (7), to estimate a three-dimensional surface profile (11) of the body part (21) in a coordinate system (30) of the camera (2), to determine and / or estimate a scaling factor of the three-dimensional surface profile (11), adapt preoperatively recorded three-dimensional patient data (22), which are present in a patient coordinate system (31), to the estimated three-dimensional surface profile (11), to determine a transformation rule (T) between the coordinate system (30) of the camera (2) and the patient coordinate system (31) based on a resulting adaptation result, wherein the adaptation and / or determination of the transformation rule (T) is carried out taking into account the determined and / or estimated scaling factor, and to provide the specific transformation rule (T).

Images