Medical AR system for a surgical procedure and method for verifying navigation accuracy

Abstract

The disclosure relates to a medical AR system (1) for a surgical procedure on a patient (10), comprising a visualization unit (2) for generating a preoperative image and a current image with a reference point (18), a navigation system (8) for detecting the visualization unit (2), and a control unit (16) for generating an AR overlay representation from a digitized reference point (20) from the preoperative image and the current image and for displaying a comparison of the AR overlay representation. The disclosure also relates to a method for checking the navigation accuracy of the medical AR system (1), as well as a computer-readable storage medium and / or a computer program.

Description

Technical area

[0001] The present disclosure relates to a medical augmented reality (AR) system for a surgical, in particular neurosurgical, procedure involving a patient. The AR system comprises a visualization unit that generates a preoperative image, in particular a 3D image, of the patient with at least one (current) reference point marked thereon, and a time-current image, in particular a 3D image, of the patient with the at least one (current) reference point. Furthermore, the AR system comprises a control unit configured to generate a digitized reference point registered to the patient from the at least one (current) reference point of the preoperative image. Furthermore, the present disclosure relates to a method for checking the navigation accuracy of the medical AR system, as well as to a computer-readable storage medium and a computer program. Background of the Revelation

[0002] Traditional approaches to surgical navigation and planning rely on preoperative imaging to obtain detailed representations of surgical target areas. These methods typically involve the use of MRI (magnetic resonance imaging), CT (computed tomography), and ultrasound images to generate comprehensive views of the surgical areas. Despite the high quality of these images, a challenge remains to effectively link these preoperative images to the actual situation during the surgical procedure. Previous systems rely on external tracking systems to connect the preoperatively generated images with the intraoperative reality. However, these approaches often require cumbersome setup processes and can limit the surgeon's freedom of movement.

[0003] In the field of medical imaging and surgery, the integration of augmented reality (AR) technologies has enabled significant advances, particularly in the precision and efficiency of surgical procedures. Augmented reality offers a touchless way to interact with and perceive patient data. With the introduction of AR into the operating room, new possibilities have opened up, allowing for more direct and intuitive interaction with preoperative data. By overlaying digital images directly onto the surgeon's field of view, AR systems can provide seamless integration of imaging information into the surgical workflow. Despite these considerable advances, limitations remain regarding the accuracy of the overlay and the ability to incorporate changes in real time.The accuracy of positioning and the stability of image overlay are critical factors that influence the usefulness of AR in surgery. Particularly in high-precision applications such as neurosurgery, minor inaccuracies in the surgical procedure can have significant impacts, sometimes with catastrophic consequences for patient safety. Additionally, the effective use of AR systems in surgery requires seamless integration into the surgical workflow, which presents a challenge, as the systems often impose additional cognitive burden on the surgeon, and interaction with the systems during the procedure must be intuitive and seamless.

[0004] Conventional surgical (AR) assistance systems are described, for example, in US 2013 / 0 293 578 A1, US 2022 / 0 215 532 A1 or CN 1 12 043 382 A.

[0005] Despite considerable advances in medical imaging and AR technology, there remains a need for improved methods and systems that enable reliable and user-friendly integration of AR into surgical procedures without adversely affecting patient safety. In particular, there is a need for systems that offer greater navigation accuracy and better adaptability to intraoperative changes without interrupting the surgical workflow or increasing the surgeon's cognitive load. Summary of Revelation

[0006] Therefore, the object of the present disclosure is to avoid or at least mitigate the disadvantages described above and, in particular, to provide a medical AR system and a method which enables precise and reliable integration of AR into surgical procedures.

[0007] This object is achieved by a medical AR system according to the features of claim 1, by a method for checking navigation accuracy, or by a computer-readable storage medium and / or a computer program according to the features of the independent claims. Advantageous embodiments are claimed in the subclaims and / or are explained below.

[0008] The disclosure therefore initially relates to a medical AR system for a surgical, in particular neurosurgical, procedure on a patient. The medical AR system has a visualization unit. The visualization unit is designed to generate and provide a preoperative (3D) image of the patient with at least one (current / momentary) reference point marked on the patient. The visualization unit is designed to generate and provide a time-updated (3D) image of the patient, for example in the form of a video feed, with the at least one (current) reference point / marker. The medical AR system has a navigation system. The navigation system is designed to detect a position and orientation of the visualization unit in a global coordinate system relative to the patient.The medical AR system has a visual display device for displaying visual information for the surgical procedure. The medical AR system has a control unit. The control unit is configured and prepared to generate and store at least one digitized reference point registered to the patient from the at least one (current) reference point of the preoperative (3D) image of the patient. The control unit is additionally configured and prepared to generate an AR overlay display with the at least one digitized reference point and the current image of the patient (which has the at least one current reference point), to display this on the visual display device, and to compare the at least one digitized reference point with the at least one (current) reference point of the current image.This is particularly useful to check the navigation accuracy of the medical AR system during the procedure.

[0009] In other words, the system comprises a visualization unit that generates, in real time, up-to-date images of the patient with at least one, preferably at least two or three, marked current reference points, and a preoperative current 3D image of the patient and the at least one, preferably at least two or three, current reference points. The visualization unit of the medical AR system is designed to generate the preoperative 3D image of the patient, including the at least one current reference point in this preoperative 3D image, which is then transmitted to a navigation system and serves as the basis for the navigation processes in the subsequent surgical procedure. A navigation system records the position and orientation of the visualization unit relative to the patient in a global coordinate system.The visual display device displays visual information for the surgical procedure, while a control unit generates and stores digital reference points from the current reference points of the preoperative image.

[0010] According to a central aspect of the disclosure, the control unit generates an AR overlay representation that simultaneously (overlaid) the digitized reference points with the current reference points of the patient's real-time image. This enables the user, or preferably the system, to continuously check and (automatically) adjust the navigation accuracy during the procedure. A visual display device is used to display this visual information and presents the augmented images and data in a form that is understandable and usable by the user. The ability to seamlessly integrate real-time data and images into the surgeon's field of view improves decision-making and precision during the procedure. The control unit is configured to generate and store digitized reference points from the preoperative 3D images of the patient.These digitized points serve as anchors for overlaying the AR representations and are crucial for maintaining spatial consistency between the virtual information and the physical patient. The ability to precisely generate and store these reference points is fundamental to the accuracy and reliability of the entire system.

[0011] Real-time, especially contactless, verification of navigation accuracy improves surgical precision, which in turn increases the safety and effectiveness of the procedure and addresses the challenge of maintaining accuracy in dynamic surgical environments. These high-resolution 3D images are fundamental to the planning and execution of the procedure, providing a three-dimensional basis for navigation and orientation. The reference points from preoperative images are compared with the actual images acquired during the procedure to ensure continuous accuracy and timeliness of visual information.Thus, planning and navigation information can be conveyed during all phases of a surgical procedure, allowing the user to use augmented reality and, with minimal manual interaction, check the accuracy of the registration and navigation of medical systems, especially during the procedure.

[0012] In a further preferred embodiment of the disclosure, the control unit can be configured and prepared to determine a distance between the at least one digitized reference point and the at least one current reference point of the current time recording as a navigation error of the medical AR system and preferably to correct it automatically.

[0013] In other words, the control unit can, especially before the actual surgical procedure starts, determine and preferably quantify a deviation between the digitized reference points and the current reference points on the patient of the current (video feed) recording and output / display this on the display device.

[0014] This enables more precise and dynamic adjustment and verification of navigation accuracy during the surgical procedure. The control unit, which is already configured to generate an AR overlay display with the digitized reference point and the time-stamped image of the patient containing the current reference point, is thus able to quantify the spatial discrepancy between the digitized and current reference points. This quantification of the distance enables the system to provide an objective measurement of the deviation in real time, allowing for correction / compensation by the user, especially before the surgical procedure begins.The identification and quantification of navigation error as a specific distance measurement between reference points provides a direct feedback loop that allows the surgical team to monitor and adjust the precision of the procedure. The implementation of these specific communication mechanisms between components, particularly between the control unit and the visual display device, enables seamless integration of real-time monitoring of navigation accuracy into the surgical workflow. This innovation leads to increased safety and efficiency during surgical procedures by minimizing the risks of navigation errors and enabling more precise alignment with surgical targets.The ability to identify and correct navigation errors, preferably in real time, represents a significant advance in medical imaging and navigation that has the potential to significantly improve patient outcomes and reduce the challenges faced by surgeons during complex procedures.

[0015] In a further advantageous aspect of the disclosure, the navigation system can be configured to (additionally) detect a position and orientation of a medical instrument tip in a global coordinate system relative to the patient, and the control unit is configured and configured to generate the at least one digitized reference point using the position and orientation of the medical instrument tip.

[0016] This extension enables direct and precise real-time tracking of the instrument tip, for example, using a dedicated marker / tracker. The control unit can be configured to generate a digitized reference point based on the position and orientation of the medical instrument tip. This allows the position of the instrument tip to be updated and refined in relation to the preoperative planning data and the current intraoperative conditions, and allows the selection of reference points on the patient to be digitized using the instrument tip.

[0017] Alternatively or additionally, the control unit can be configured and prepared to select and digitize the reference points using a laser pointer and / or a focal point of the visualization unit. The use of a laser pointer or a focal point enables significantly more precise localization and selection of the reference points on the patient. The focal point, which is part of the visualization unit, allows the positioning of the reference points with high accuracy by being directed directly at the relevant part of the patient's body. Accordingly, the control unit can generate a digitized reference point, which is then fed into the augmented reality (AR) overlay display. This display, displayed on the visual display device, combines the digital information with the real world.

[0018] In a further or alternative advantageous embodiment of the disclosure, the control unit can be configured and designed to automatically generate the at least one digitized reference point using the at least one current reference point of the preoperative 3D image of the patient generated by the visualization unit. This means that the control unit can be configured to automatically generate digitized reference points based on the preoperative 3D image of the patient generated by the visualization unit and the current reference points / markings arranged on the patient in this image, i.e., without additional manual selection of the current reference points on the patient.

[0019] Advantageously, an automatic generation of the at least one digitized reference point by the control unit is thus possible, based on the current reference point identified in the preoperative 3D image.

[0020] In a further preferred embodiment of the disclosure, the medical AR system may comprise an AR headset and / or a 2D monitor and / or a 3D monitor and / or a VR headset as the display device.

[0021] In other words, the visual AR overlay representation of the current recording and the digitized reference points can be displayed alternatively or additionally via a VR or AR headset that the user wears on their head.

[0022] Integrating an AR or VR headset as a display device allows the user to experience an immersive, augmented reality experience, with digital information superimposed directly into the user's field of vision. This promotes intuitive interaction with the visual data and supports more precise navigation and orientation during the surgical procedure by reducing the need to shift the patient's gaze away from the surgical site. Using a 3D monitor as a display device offers an alternative visualization method, allowing the surgical team to view the three-dimensional anatomical structures and the positioning of landmarks in real time, facilitating collective assessment and decision-making during the procedure.

[0023] In a further preferred embodiment of the disclosure, the navigation system may comprise an infrared-based tracking system or an electromagnetic tracking system or an optical machine vision tracking system.

[0024] In other words, the visualization unit is detected primarily via the navigation system, for example, using an infrared-based, electromagnetic, or optical tracking system. In all cases, the tracking system detects the relative position of the visualization system to the patient. A tracking system can also be used that calculates the relative position of the visualization system to the patient from kinematic data / information from the robot or robot arm to which the visualization unit is connected.

[0025] In other words, the navigation system can be either an infrared-based tracking system, an electromagnetic tracking system, or an optical machine vision tracking system. With an infrared-based tracking system, communication occurs through infrared signals reflected from markers attached to the patient, enabling highly precise positioning and orientation of the visualization unit relative to the patient. An electromagnetic tracking system uses electromagnetic fields to detect the spatial position and orientation of the visualization unit. An optical machine vision tracking system, on the other hand, uses image processing algorithms to detect and track the position of markers on the patient.

[0026] According to a further advantageous aspect of the disclosure, the visualization unit can be designed as a surgical microscope, a surgical exoscope, a surgical endoscope or as an optical camera.

[0027] Preferably, the AR system according to the invention enables 3D perception of the displayed AR information. Further preferably, the entire AR system can run in a simulated environment or in a digital twin.

[0028] The present disclosure further relates to a method for checking the navigation accuracy of a medical AR system, comprising the steps of setting at least one, preferably at least two, current reference point / marker on a patient, generating and providing a preoperative 3D image of the patient with the at least one current reference point by a visualization unit, generating and storing at least one digitized reference point from the at least one current reference point of the preoperative 3D image of the patient by a control unit, generating and providing a current image of the patient with the at least one current reference point by the visualization unit, generating an AR overlay representation with the at least one digitized reference point and the current image of the patient, which has the at least one current reference point,Outputting the AR overlay representation by a visual display device, and comparing the at least one digitized reference point with the at least one current reference point of the current recording in order to check a navigation accuracy of the medical AR system.

[0029] According to a further advantageous embodiment of the disclosure, the method may comprise an additional step of calculating a navigation error of the medical AR system from a distance / deviation between the at least one digitized reference point and the at least one current reference point of the current recording.

[0030] By calculating the distance between the respective reference points, the control unit can detect any navigation error. The ability to immediately detect and correct deviations minimizes the risk of errors during the procedure, improves surgical outcomes, and increases patient safety. Furthermore, this feature enables an objective evaluation of the medical AR system's performance by providing a quantifiable measure of navigation accuracy.

[0031] According to a further embodiment, the navigation system can detect a position and orientation of a medical instrument tip in a global coordinate system relative to the patient, and the step of generating and storing the at least one digitized reference point can be carried out using the position and orientation of the medical instrument tip.

[0032] The present disclosure further relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method steps according to the disclosure.

[0033] Furthermore, the present disclosure further relates to a computer program with instructions which, when executed by a computer, cause the computer to carry out the method steps according to the disclosure. Short description of the characters

[0034] The disclosure is explained in more detail below using preferred embodiments with the aid of the accompanying figures. They show: Fig. 1 is an exemplary perspective side view of a medical AR system according to a preferred embodiment of the present disclosure; Fig. 2 shows an exemplary time-based image of a patient with a plurality of reference points marked thereon and an exemplary medical instrument according to the preferred embodiment of the present disclosure; Fig. 3 shows an exemplary acquisition process of a preoperative 3D image of the patient and the plurality of marked reference points by a visualization unit, which are automatically transmitted to the navigation system as digitized reference points, according to the preferred embodiment of the present disclosure; Fig. 4 an exemplary perspective AR overlay display with a plurality of digitized reference points and the current image of a draped patient, and Fig. 5 is a flowchart of a method for checking navigation accuracy of the medical AR system according to a preferred embodiment of the present disclosure.

[0035] The figures are schematic in nature and serve only to facilitate understanding of the disclosure. Like elements are provided with the same reference numerals. The features of the various embodiments can be interchanged and may occur in any combination. Description of the embodiments

[0036] In the following, the present disclosure will be described by means of an advantageous embodiment with reference to the Fig. 1 to 5 described.

[0037] Fig. 1 is an exemplary perspective side view of a medical AR system 1 according to a preferred embodiment of the present disclosure. The AR system 1 comprises a movable visualization unit 2, in particular in the form of a surgical microscope, which is connected to a movable robot arm 4 of a medical robot 6 in order to adjust both a position (x, y, z) and an orientation of the visualization unit 2 in space relative to the patient by controlling the robot arm 4. The user / operator 12 controls the position of the visualization unit 2, which generates a time-current image of a patient 10 as well as of preoperatively placed reference points / markings on the patient.Furthermore, the AR system 1 has a navigation system 8, which uses an integrated navigation camera, in this case a stereo camera, to detect the position and orientation of the visualization unit 2 relative to a (registered) patient 10. The AR system 1 also has a display device 14, for example in the form of a 2D monitor, and a control unit 16. The control unit 16 is adapted to generate an AR overlay display with at least one digitized reference point 20 and the current image of the patient 10, which has the at least one current reference point 18, and to display it on the display device 14.

[0038] Fig. 2 is an exemplary up-to-date image of the patient 10 with a plurality of (current) reference points 18 marked thereon and an exemplary medical instrument 17, which has an instrument tip 21 trackable / traceable by the navigation system 8, according to the preferred embodiment of the present disclosure. The up-to-date image of the patient 10, which is captured by the visualization unit 2 and can be displayed as a up-to-date video transmission on the display device 14, thus represents a up-to-date image of the patient 10 including the up-to-date / true-to-date markings / reference points 18 and preferably also a up-to-date image of a medical instrument 17.

[0039] Fig. 3 shows an exemplary acquisition process of a preoperative 3D image of the patient 10 and the plurality of marked reference points 18 by a visualization unit 2, which are automatically transmitted to a navigation system 8 as digitized reference points 20, according to the preferred embodiment of the present disclosure. The digitized reference points 20 are generated by the control unit 16 of the medical AR system 1 based on the acquired, up-to-date reference points 18.

[0040] Fig. 4 is an exemplary perspective AR overlay representation with a plurality of digitized reference points 20 and the current image of a patient 10 with a plurality of current reference points 18. In this representation, the patient 10 is draped by a medical sheet / cover 22 for a subsequent surgical procedure, and only a part (in this representation, the forehead) of the patient 10, which has the reference points 18, is not covered by the medical sheet 22. The characteristic anatomical landmarks of the patient 10 are thus no longer easily accessible to the visualization unit 2. Nevertheless, it is possible to use augmented reality to display the previously digitized reference points 20 to the user 12, who is then able to visually compare them with the reference points 18 marked on the patient 10 and currently displayed in the display device 14.The shift between the digitized reference points 20 and the respective current reference points 18 is then the navigation error 24 of the AR system 1, which is preferably indicated or highlighted separately on the display device 14.

[0041] Fig. 5 is a flowchart of a (computer-implemented) method for checking navigation accuracy of the medical AR system 1 according to a preferred embodiment of the present disclosure.

[0042] In a first step S1, at least one current reference point 18 is set on a patient 10 preoperatively. For example, this is done using a medical marker. In a second step S2, a preoperative 3D image of the patient 10 with the at least one current reference point 18 is generated and provided by a visualization unit 2. Subsequently, in the third step, at least one digitized reference point 20 is generated from the at least one current reference point 18 of the preoperative 3D image of the patient 10 by the control unit 16 and stored. Simultaneously, before or alternatively after step S3, in the fourth step S4, a time-current image of the patient 10 with the at least one current reference point 18 is generated and provided by the visualization unit 2.Subsequently, in the fifth step, an AR overlay representation with the at least one digitized reference point 20 and the current image of the patient 10, which has the at least one current reference point 18, is generated and output on the visual display device 14 in the sixth step S6. In the final step S7, the control unit 16 compares the at least one digitized reference point 20 with the at least one current reference point 18 of the current image.

[0043] In a further step S8, a navigation error 24 of the medical AR system 1 can be calculated from a distance between the at least one digitized reference point 20 and the at least one current reference point 18 of the current recording by the control unit 16.

[0044] In a further preferred embodiment of the disclosure, the navigation system 8 can detect a position and orientation of a medical instrument tip 21 in a global coordinate system relative to the patient 10, and the step S3 of generating and storing the at least one digitized reference point 20 can be carried out using the position and orientation of the medical instrument tip 21. List of reference symbols 1 medical AR system 2 Visualization unit 4 movable robot arm 6 medical robots 8 Navigation system 10 patients 12 users 14 Display device 16 Control unit 17 medical instrument 18 current reference point 20 digitized reference points 21 Instrument tip 22 medical tarpaulins 24 navigation errors S1 Step Setting at least one current reference point S2 Step Generating and providing a preoperative 3D image S3 Step Generate and save at least one digitized reference point S4 Step Creating and providing a current patient image S5 Step Creating an AR overlay representation S6 Step Output of the AR overlay representation by the display device S7 Step Comparing the at least one digitized reference point with the at least one current reference point S8 Step Calculating a navigation error of the medical AR system

Claims

[1] Medical AR system (1) for a surgical, in particular neurosurgical, intervention of a patient (10), comprising: a visualization unit (2) which generates and provides a preoperative image of the patient (10) with at least one reference point (18) marked thereon and a current image of the patient (10) with the at least one reference point (18), a navigation system (8) which is designed to detect a position and orientation of the visualization unit (2) in a global coordinate system relative to the patient (10), a visual display device (14) for displaying visual information for the surgical procedure, a control unit (16) which is set up and prepared to generate and store at least one digitized reference point (20) registered on the patient (10) from the at least one reference point (18) of the preoperative image of the patient (10), characterized by , that the control unit (16) is additionally configured and prepared to generate an AR overlay representation with the at least one digitized reference point (20) and the current image of the patient (10), to display the AR overlay representation on the visual display device (14), and to compare the at least one digitized reference point (20) with the at least one reference point (18) of the current image. [2] Medical AR system (1) according to claim 1, characterized byin that the control unit (16) is set up and prepared to determine a distance between the at least one digitized reference point (20) and the at least one reference point (18) of the current recording as a navigation error (24) of the medical AR system (1) and preferably to correct it automatically. [3] Medical AR system (1) according to claim 1 or 2, characterized by in that the navigation system (8) is designed to detect a position and orientation of a medical instrument tip (21) in a global coordinate system relative to the patient (10), and the control unit (16) is set up and designed to generate the at least one digitized reference point (20) using the position and orientation of the medical instrument tip (21). [4] Medical AR system (1) according to one of claims 1 to 3 characterized bythat the control unit (16) is set up and designed to generate the at least one digitized reference point (20) with the aid of a laser pointer and / or a focal point of the visualization unit (2). [5] Medical AR system (1) according to one of claims 1 to 4, characterized by that the control unit (16) is set up and designed to automatically generate the at least one digitized reference point (20) with the aid of the at least one reference point (18) of the preoperative image of the patient (10) produced by the visualization unit (2). [6] Medical AR system (1) according to one of the preceding claims, characterized by that the medical AR system (1) has an AR headset and / or a 3D monitor and / or a VR headset as the display device (14). [7] Medical AR system (1) according to one of the preceding claims, characterized bythat the navigation system (8) has an infrared-based tracking system or an electromagnetic tracking system or an optical machine vision tracking system. [8] Method for checking a navigation accuracy of a medical AR system (1), in particular according to one of claims 1 to 7, characterized by the steps: - setting (S1) at least one reference point (18) on a patient (10); - generating and providing (S2) a preoperative image of the patient (10) with the at least one reference point (18) by a visualization unit (2); - generating and storing (S3) at least one digitized reference point (20) from the at least one reference point (18) of the preoperative image of the patient (10) by a control unit (16); - generating and providing (S4) a time-current image of the patient (10) with the at least one reference point (18) by the visualization unit (2); - generating (S5) an AR overlay display with the at least one digitized reference point (20) and the current image of the patient (10); - outputting (S6) the AR overlay representation by a visual display device (14); and - comparing (S7) the at least one digitized reference point (20) with the at least one reference point (18) of the current recording by the control unit (16). [9] Method according to claim 8, characterized by an additional step of calculating (S8) a navigation error (24) of the medical AR system (1) from a distance between the at least one digitized reference point (20) and the at least one reference point (18) of the current recording by the control unit (16). [10] Method according to claim 8 or 9, characterized by in that a navigation system (8) detects a position and orientation of a medical instrument tip (21) in a global coordinate system relative to the patient (10) and the step of generating and storing (S3) the at least one digitized reference point (20) is carried out using the position and orientation of the medical instrument tip (21). [11] A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method steps according to any one of claims 8 to 10.

Images