Medical system and method for checking registration accuracy

Abstract

The invention relates to a medical system (1) for a surgical procedure on a patient (P), comprising image data of the patient (P), a recording unit (2) which is designed to generate and provide an up-to-date image of the patient (P), a control unit (12) which is designed to register the patient (P) on the basis of the image data of the patient (P) and the up-to-date image of the patient (P), a visual display device (10) which is designed to display visual information for the surgical procedure, wherein the control unit (12) is designed to spatially calculate a registration accuracy of the patient and to display an image of the patient (P) projected on the display device (10). In addition, the present invention relates to a method for checking registration accuracy of a medical system (1).

Description

[0001] Medical system and procedures for verifying registration accuracy

[0002] Description

[0003] Technical field

[0004] The present disclosure relates to a medical system for a surgical, in particular neurosurgical, procedure on a patient. Furthermore, the present disclosure relates to a method for verifying the registration accuracy of a medical system.

[0005] Background of the Revelation

[0006] It is already known from the state of the art to register a patient using (preoperative and / or intraoperative) image data and a real-time recording of the patient, i.e., to capture the position and orientation of a patient's anatomy, in the case of neurosurgery especially the patient's head or face, in a global coordinate system in order to support the surgical procedure by navigating instruments, devices, etc. relative to the patient.

[0007] To verify patient registration, current technology typically involves manually operating a tracked instrument by a user / operator, which can disrupt the operation, prolong the operating time, and thus negatively impact patient safety overall.

[0008] Furthermore, registration, especially with complex geometries such as patient anatomy, can vary in accuracy depending on the spatial location, meaning that the registration must be checked at different points and that deficient registration results at individual / certain positions may still not be detected, which can have significant disadvantages for patient safety.

[0009] Summary of Revelation

[0010] Therefore, the present disclosure aims to avoid or at least reduce the disadvantages of the prior art. In particular, it seeks to provide a medical system and an associated procedure in which registration accuracy can be verified particularly easily, reliably, and intuitively, and patient safety can be improved.

[0011] The problem addressed in this disclosure is solved by a medical system having the features of claim 1 or by a method having the features of the dependent claim. Advantageous further developments are the subject of the dependent claims.

[0012] Accordingly, the task described in this disclosure is solved by a medical system, in particular a medical augmented reality system, for a surgical, especially neurosurgical, procedure on a patient (or patient's head). The medical system comprises image data of the patient, a visualization unit or a recording unit, a control unit, and a visual display device.

[0013] The image data can be radiological image data and / or ultrasound image data and / or image data / captures in the visible spectrum. The image data can preferably be preoperative and / or intraoperative image data. The image data are particularly important in three-dimensional form. The image data can, for example, be stored on a storage device.

[0014] The acquisition unit is designed to generate and provide a real-time (three-dimensional) image of the patient, specifically a live image feed. The control unit is designed to register / identify the patient using the patient's image data and the real-time image. This means that during registration, the image data is linked to the patient or the real-time image.

[0015] The visual display device is designed to display visual information for the surgical procedure.

[0016] According to the present disclosure, the control unit is configured to spatially calculate the patient's registration accuracy, particularly at multiple local points and / or areas, and to display it projected onto a patient image on the display device. The patient image onto which the registration accuracy is projected can preferably be the image data and / or the current patient image. That is to say, the control unit is configured to spatially display the patient's registration accuracy preferably onto the preoperative image data (or onto one or more two-dimensional cross-sectional images of the preoperative image data, i.e., 2D CT slices) and / or the current patient image (or onto one or more two-dimensional views of the current image or one or more snapshots of the current image).

[0017] The core of the present disclosure therefore consists in the fact that the medical system is configured to calculate the registration accuracy and to display it visually, preferably by means of augmented reality (directly) projected onto a patient image (i.e., onto the live image feed and / or the 2D CT slices). In particular, the system is configured to calculate the registration accuracy spatially, namely as a function of a relevant local (patient) area / region / surface segment, and to display the registration accuracy spatially, namely for each relevant local (patient) area / region / surface segment individually. In other words, the registration accuracy is displayed by means of augmented reality on the display device within a patient image, in particular, each area / region / segment individually.

[0018] This has the advantage that augmented reality, whose use in surgical procedures is already established for simple applications such as displaying enhanced information in the image feed of a connected external microscope, is employed in this medical system for more advanced, sophisticated functions. By verifying registration accuracy via augmented reality, a contactless way of perceiving and interacting with patient data becomes possible. In particular, it is no longer necessary to manually operate a navigated instrument to check registration accuracy, as the quality of a patient registration can be assessed at a glance in a simple and intuitive way. Specifically, it can simultaneously determine in which areas the registration was accurate or inaccurate.This knowledge can be used to accelerate conventional accuracy checks with the navigated instrument or to eliminate manual accuracy checks altogether. This allows for a smoother surgical procedure and reduces operating time, thereby increasing overall patient safety. Furthermore, this approach offers the advantage of verifying registration accuracy in any area of ​​the patient.

[0019] According to a preferred embodiment, the registration accuracy can be determined by the distance between an (actual) surface area of ​​the patient and a corresponding virtual surface area created based on the registration. The actual surface area can preferably be captured using the recording unit (or a wide-angle camera). In particular, the actual surface area can be captured by the real-time image or a point cloud of the patient. Furthermore, the distance can be determined by comparing a 3D depth perception of the actual surface area and the virtual surface area. In this way, the registration accuracy can be spatially determined.According to a preferred embodiment, the control unit can be configured to display the spatially calculated registration accuracy as a map, in particular a heatmap, in which the spatially calculated registration accuracy (i.e., locally broken down) is encoded according to size or accuracy, in particular color-coded. The map-like representation, especially as a heatmap, allows for particularly simple spatial mapping. The (color) coding enables a particularly quick visual identification of areas with lower registration accuracy and areas with higher registration accuracy. This means that areas with lower registration accuracy have a different encoding / identification, in particular color, than areas with higher registration accuracy.For example, areas with lower registration accuracy can be marked in red and areas with higher registration accuracy in green. This makes it particularly easy for the operator to identify areas where the registration accuracy is insufficient.

[0020] According to an alternative preferred embodiment, the control unit can be configured to display the spatially calculated registration accuracy as a map, where the spatially calculated registration accuracy (i.e., locally broken down) is shown with a numerical value. This allows the operator to analyze the registration accuracy with particular precision.

[0021] According to a preferred embodiment, the control unit can be configured to statically calculate the registration accuracy and display it statically on the display device. This means that the display of the registration accuracy does not change when the recording unit is moved or modified. In other words, the registration accuracy is calculated statically and only once. This has the advantage of requiring very little computing power. In other words, a point cloud (generated by the current recording) is used for registration, and the registration accuracy is checked against the point cloud (used for the registration itself). In particular, a distance to a surface that would be expected based on the registration is calculated, colored, and projected. During the static display, the spatial location of the point cloud (or its position) does not change.the (color) coding) not work if the recording unit is moved.

[0022] According to further training, the control unit can be trained to register the patient by calculating individual transformations between corresponding points in the image data and the current image capture, and then calculating an overall transformation from these individual transformations. Furthermore, the control unit can be trained to statically calculate the registration accuracy by comparing the overall transformation with the individual transformations. This means that the difference between the individual transformations already used to determine the overall transformation and the (average) overall transformation indicates the registration accuracy. This allows for a particularly simple calculation of the registration accuracy.

[0023] According to a preferred embodiment, the recording unit can be a (preferably navigated) camera. Alternatively, the recording unit can be configured as a pointer or a laser.

[0024] According to a preferred embodiment, the control unit can be configured to dynamically calculate the registration accuracy and display it dynamically, preferably live, on the display device. This means that the display of the registration accuracy changes as the recording unit is moved or repositioned. According to this embodiment, the recording unit can be configured to generate and provide a real-time depth perception of the patient (or point cloud of the patient or patient head). In other words, the registration accuracy is calculated dynamically and repeatedly. This has the advantage that the registration accuracy can be checked at any time (or at least at a specific point in time) in any area of ​​the patient or patient head, and not only based on the point cloud used for the registration itself.In other words, a point cloud (generated by the current recording) is used for registration, and the registration accuracy is checked against constantly recalculated point clouds. Specifically, a distance to a surface that would be expected based on the registration is calculated, colored, and projected. With dynamic display, the spatial location of the point cloud (or its color coding) changes when the recording unit is moved, because the movement of the recording unit alters the point cloud being used.

[0025] According to a further training, the control unit can be configured to register the patient by calculating initial individual transformations between the first corresponding points of the image data and the current image acquisition, and then calculating an overall transformation from these initial individual transformations. The control unit is also configured to dynamically calculate the registration accuracy by calculating second individual transformations between the second corresponding points of the image data and the current image acquisition, and comparing the overall transformation with these second individual transformations. This means that the difference between those individual transformations that are not used to determine the overall transformation and the (average) overall transformation indicates the registration accuracy. In this way, the registration accuracy can be calculated with particular precision.

[0026] According to a training course, the first and second corresponding points can be located in different areas of the patient. This means that the verification of registration accuracy is performed using these (second) points, which (unlike the first points) were not used for registration. This increases the validity of the registration accuracy.

[0027] In other words, static registration accuracy means that the image or point cloud created by the recording unit at the time of registration is used to calculate the registration accuracy, while dynamic registration accuracy means that the image or point cloud created live by the recording unit is used to calculate the registration accuracy. The image or point cloud created live can differ from the image or point cloud created at the time of registration. According to a preferred embodiment, the medical system can include a navigation system with a navigation camera that is preferably separate from the recording unit. The navigation camera can, in particular, be configured to track the recording unit relative to the patient. This ensures that the pose of the recording unit is known at all times.

[0028] According to a preferred embodiment, the control unit can be configured to display the registration accuracy as points or lines / contours on the display device. This has the advantage that, even with a superimposed display, a desired area of ​​interest can be viewed depending on the focus being placed on the points or lines / contours or on the patient image.

[0029] According to a preferred embodiment, the control unit can be configured to define the spatial calculation, in particular boundaries between local areas, using a surface or grid or individual anatomical landmarks. This allows for easy assignment to the patient, which supports a faster and simpler interpretation of the registration accuracy.

[0030] According to a preferred embodiment, the display device can be a 2D monitor and / or a 3D monitor. This allows the registration accuracy to be displayed reliably and easily.

[0031] According to a preferred embodiment, the display device can be an AR headset (augmented reality headset). This allows the display to be integrated particularly advantageously into the operating environment.

[0032] According to a preferred embodiment, the display device can be a VR headset (virtual reality headset). This allows the registration accuracy to be checked even in a simulated environment or a digital twin. According to a preferred embodiment, the display device can be configured to enable 3D perception. This has the advantage that the display device can be interpreted easily and intuitively by the operator.

[0033] According to a preferred embodiment, the recording unit can be a digital surgical microscope and / or a surgical endoscope or an optical camera. In particular, the recording unit can be a depth-sensing camera or a stereo camera.

[0034] According to a preferred embodiment, the control unit can be configured to display the registration accuracy directly after registration or at a later time, particularly upon manual request, especially from a surgeon. This means that the registration accuracy can be visualized during the operation, for example, if the surgeon wants to check a previously performed registration after draping the patient. This can increase patient safety.

[0035] According to a preferred embodiment, the medical system can be trained to perform a verification of the registration accuracy in a simulated environment or in a digital twin.

[0036] The objective of the present disclosure is also achieved by a (preferably computer-implemented) method for verifying the recording accuracy of a medical system, in particular the medical system described. The method comprises the following steps:

[0037] - Creating and providing a current image of a patient,

[0038] - Registering the patient using (preoperative and / or intraoperative) image data of the patient and the current patient recording,

[0039] - Spatial calculation of patient registration accuracy, especially at multiple local points and / or areas, and

[0040] - Displaying the registration accuracy. Preferably, the registration accuracy is displayed as a projection onto a patient image, such as the patient's preoperative image data (or two-dimensional sectional views of the preoperative image data) and / or the current patient image.

[0041] Preferably, in a preliminary step, preoperative, in particular radiological, image data of the patient are generated and provided. The preoperative image data are three-dimensional image data. The preoperative image data are generated, for example, by a CT scanner or an MRI scanner.

[0042] In other words, the present disclosure relates to a system and a method for quickly verifying the result of a patient registration by enriching a relevant registration region, such as the patient's face, within a camera image with spatial registration accuracy information as a three-dimensional heatmap. The spatial registration accuracy information is based on the distance between registration information collected in the real world, such as a point cloud of the patient's face, and a virtual fitting target, such as a surface within the scan volume. In this way, the user can assess the quality of a performed patient registration at a glance and simultaneously determine in which areas the registration was accurate or inaccurate.This knowledge can be used to speed up conventional accuracy testing with a navigated instrument or to allow the user to skip manual accuracy testing altogether.

[0043] Spatial registration accuracy information can be displayed in two ways. In a static display, the spatial registration accuracy is calculated once and then displayed. For each data point—that is, for individual landmarks or points in a point cloud—used to calculate the registration, the shortest distance to the target surface (based on a scan) is calculated, and this information is then statically displayed as a three-dimensional heatmap. This heatmap can be inspected by moving the camera using AR visualization. In a dynamic display, live depth perception from a calibrated and navigated camera is used. The depth of this camera is compared to the depth perception of a virtually cloned camera aligned with the registered scan surface. The offset between the two depth maps yields the spatial registration accuracy.The movement of the camera affects the visualized information, as the heatmap is generated live.

[0044] Furthermore, the spatial registration accuracy information can also be represented in 2D by projecting it as points or lines / contours onto 2D layers of a 3D image volume, such as a CT scan.

[0045] The visualization or registration accuracy can be inspected directly after registration or later during the procedure, for example when the surgeon begins the procedure and wants to check the accuracy of the previously performed registration.

[0046] Brief description of the characters

[0047] Fig. 1 shows an exemplary view of a medical system according to the present disclosure;

[0048] Fig. 2 shows a representation of a registration accuracy by the medical system according to the present disclosure;

[0049] Figures 3a to 3c show various 2-dimensional views to represent the registration accuracy by points;

[0050] Figures 4a to 4c show various two-dimensional views illustrating registration accuracy by lines; and

[0051] Fig. 5 shows a schematic representation of a method for verifying the registration accuracy according to the present disclosure. A preferred embodiment of the present disclosure is described below based on the accompanying figures.

[0052] Fig. 1 is an exemplary view of a medical system 1 according to the present disclosure. The medical system 1 is used for a surgical, in particular neurosurgical, procedure on a patient P. The procedure can be controlled, for example, by a user / operator S.

[0053] Medical system 1 contains patient image data. This image data can be radiological images, ultrasound images, and / or images / captures in the visible spectrum. The image data can preferably be preoperative and / or intraoperative images. In particular, the image data can be three-dimensional images.

[0054] The medical system 1 comprises a (movable) visualization unit 2 or a recording unit 2. The recording unit 2 is designed to generate and provide a real-time image of the patient P, in particular of a surgical area, such as the patient's head. The recording unit 2 is specifically designed as a digital surgical microscope, a surgical endoscope, or an optical camera. In particular, the recording unit 2 can be a camera with depth perception or a stereo camera.

[0055] The medical system 1 comprises a medical robot 4 with a movable robot arm 6. The acquisition unit 2 is attached, in particular, to the movable robot arm 6, preferably its end effector. The position (x, y, z) and orientation of the acquisition unit 2 can be set, in particular, by controlling the robot arm 6 in space relative to the patient P.

[0056] The medical system 1 may include a navigation system 8. In particular, the navigation system 8 may include an integrated navigation camera, such as a stereo camera, which is designed to navigate (during the surgical procedure). The navigation camera is specifically configured to track the recording unit 2 relative to the patient P.

[0057] The medical system 1 includes a visual display device 10. The display device 10 is configured to display visual information for the surgical procedure. In particular, the display device 10 is a 2D monitor and / or a 3D monitor. Preferably, the display device 10 is configured to enable 3D perception. In particular, the display device 10 is an AR headset (augmented reality headset) and / or a VR headset (virtual reality headset).

[0058] The medical system 1 comprises a control unit 12. The control unit 12 is configured to register / capture the patient P based on the (three-dimensional) image data of the patient P and the current image of the patient P. The control unit 12 is configured to calculate the registration accuracy of the patient P spatially, in particular at several local points and / or areas. The control unit 12 is configured to display the registration accuracy on the display device 10, preferably projected onto an image of the patient P. The image of the patient P can be formed, in particular, from the preoperative image data or two-dimensional sectional views of the preoperative image data and / or the current image of the patient P. The control unit 12 can, in particular, be a component of the medical robot 4.

[0059] As can be seen in Fig. 2, the registration accuracy is projected onto the patient image and displayed. This means that the registration accuracy in different spatial areas of the image is displayed. In the embodiment shown in Fig. 2, the registration accuracy is visualized by a coding system, in particular a color coding system, in the form of dots. First areas 14, which have very high registration accuracy, second areas 16, which have high registration accuracy, third areas 18, which have medium registration accuracy, fourth areas 20, which have somewhat inaccurate registration accuracy, and fifth areas 22, which have inaccurate registration accuracy, are each assigned a different coding system.For example, the first areas can be colored 14 dark green, the second areas 16 light green, the third areas 18 yellow, the fourth areas 20 light red and the fifth areas 22 dark red, or marked with such a colored dot.

[0060] Figures 3a to 3c show various two-dimensional sectional views of the preoperative image data, on which the registration accuracy is projected. The sectional view in Figure 3a is a sagittal section of the patient's head. The sectional view in Figure 3b is an axial section of the patient's head. The sectional view in Figure 3c is a coronal section of the patient's head. Different colored dots are depicted along a surface of the patient's head, particularly in the area of ​​the patient's face. The color of these dots encodes / represents / visualizes the registration accuracy in the corresponding spatial area. In the illustrated embodiment, a higher registration accuracy is present in an area near the forehead or nose than in an area near the ear, mouth, or chin.

[0061] Figures 4a to 4c show various two-dimensional sectional views of the preoperative image data, on which the registration accuracy is projected. The sectional view in Figure 4a is a sagittal section of the patient's head. The sectional view in Figure 4b is an axial section of the patient's head. The sectional view in Figure 4c is a coronal section of the patient's head. Different colored lines / contours are depicted along a surface of the patient's head, particularly in the area of ​​the patient's face. The color of these lines / contours encodes / represents / visualizes the registration accuracy in the corresponding spatial area. In the illustrated embodiment, a higher registration accuracy is present in an area near the forehead or nose than in an area near the ear, mouth, or chin.

[0062] Fig. 5 shows a (preferably computer-implemented) method for verifying the recording accuracy of a medical system, in particular the medical system 1 described above. The method comprises the following steps:

[0063] In a preliminary step, image data of patient P is generated and provided. This image data is preferably preoperative and / or intraoperative. Specifically, it is radiological image data, ultrasound image data, and / or image data in the visible spectrum. The image data is particularly important in three dimensions. The image data is generated, for example, by a CT scanner, an MRI scanner, ultrasound, or a camera.

[0064] In the first step S1, a current image of patient P is generated and provided. In the second step S2, patient P is registered using the image data of patient P and the current image. In the third step S3, the registration accuracy of patient P is spatially calculated, particularly at several local points and / or areas. In the fourth step S4, the registration accuracy is displayed, particularly on an image of patient P, such as the image data of patient P (or two-dimensional cross-sectional views of the preoperative image data) and / or the current image.

[0065] Reference symbol list

[0066] 1 medical system

[0067] 2 recording units

[0068] 4 medical robots

[0069] 6 movable robot arm

[0070] 8 Navigation system

[0071] 10 Display device

[0072] 12 Control unit

[0073] 14 First area with very high registration accuracy

[0074] 16 Second area with high registration accuracy

[0075] 18 third area with medium registration accuracy

[0076] 20 fourth area with somewhat less precise registration accuracy

[0077] 22 fifth area with inaccurate registration accuracy

[0078] Patient

[0079] 5 Surgeon

[0080] 51 Creating and providing a current recording of a patient

[0081] 52 Registering the patient using image data and the current recording

[0082] 53 Spatial calculation of patient registration accuracy

[0083] 54 Representing the registration accuracy

Claims

Claims 1. Medical system (1 ) for a surgical, in particular neurosurgical, procedure on a patient (P), with Image data of the patient (P), a recording unit (2) configured to generate and provide a real-time recording of the patient (P), a control unit (12) configured to register the patient (P) using the image data of the patient (P) and the real-time recording of the patient (P), and a visual display device (10) configured to display visual information for the surgical procedure, wherein the control unit (12) is configured to calculate a spatial registration accuracy of the patient, in particular at several local points or areas, and to project this onto a recording of the patient (P), preferably onto the image data and / or the real-time recording of the patient (P), on the display device (10).

2. Medical system (1) according to claim 1, characterized in that the registration accuracy is determined by a distance between a surface area of ​​the patient (P) and a corresponding virtual surface area of ​​the patient (P) created on the basis of the registration.

3. Medical system (1 ) according to claim 1 or 2, characterized in that the control unit (12) is configured to display the spatially calculated registration accuracy as a map, in particular a heatmap, in which the spatially calculated registration accuracy is encoded according to size, in particular color-coded.

4. Medical system (1 ) according to one of claims 1 to 3, characterized in that the control unit (12) is configured to statically calculate the registration accuracy and statically display it on the display device (10).

5. Medical system (1) according to claim 4, characterized in that the control unit (12) is configured to register the patient by calculating individual transformations between corresponding points of the image data and the current recording, and calculating an overall transformation from the individual transformations, and the control unit (12) is configured to statically calculate the registration accuracy by comparing the overall transformation with the individual transformations.

6. Medical system (1 ) according to one of claims 1 to 5, characterized in that the recording unit (2) is a navigated camera and the control unit (12) is configured to dynamically calculate the registration accuracy and dynamically display it on the display device (10).

7. Medical system (1) according to claim 6, characterized in that the control unit (12) which is configured to register the patient by calculating first individual transformations between first corresponding points of the image data and the current recording and from the first individual transformations an overall transformation, and the control unit (12) which is configured to dynamically calculate the registration accuracy by calculating second individual transformations between second corresponding points of the image data and the current recording and comparing the overall transformation with the second individual transformations.

8. Medical system (1 ) according to claim 7, characterized in that the first corresponding points and the second corresponding points are arranged in geographically different areas of the patient.

9. Medical system (1 ) according to one of claims 1 to 8, characterized in that the medical system (1 ) has a navigation system (8) with a navigation camera preferably designed separately from the recording unit (2), which is configured to track the recording unit (2) relative to the patient (P).

10. Medical system (1 ) according to one of claims 1 to 9, characterized in that the control unit (12) is configured to display the registration accuracy as points or lines on the display device (10).

11. Medical system (1 ) according to one of claims 1 to 10, characterized in that the control unit (12) is configured to define the spatial calculation, in particular boundaries between the local areas, on the basis of a surface or a grid or individual anatomical reference points.

12. Medical system (1) according to one of claims 1 to 11, characterized in that the display device (10) is a 2D monitor and / or a 3D monitor and / or an AR headset and / or a VR headset.

13. Medical system (1) according to one of claims 1 to 12, characterized in that the recording unit (2) is a digital surgical microscope and / or a surgical endoscope or an optical camera.

14. Method for verifying the recording accuracy of a medical system (1), in particular according to one of claims 1 to 13, characterized by the steps: - Generating and providing (S1 ) a current recording of a patient (P), - Registering (S2) the patient (P) using image data of the patient (P) and the current recording of the patient (P), - Spatial calculation (S3) of a patient registration accuracy (P), especially at multiple local areas, and - Displaying (S4) the registration accuracy.

Images