Medical AR system and method for intraoperative planning during surgical procedures
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- B BRAUN NEW VENTURES GMBH
- Filing Date
- 2024-07-17
- Publication Date
- 2026-08-05
Smart Images
Figure 2026526080000001_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a medical AR system for intraoperative planning during surgical procedures on patients, particularly neurosurgical procedures. The AR system has a spatially movable visualization unit, particularly the microscope head of a surgical microscope, an endoscope, or an optical camera, that creates and digitally provides real-time images of the patient, particularly internal body images, whose spatial position and orientation can be adjusted. Furthermore, the AR system has a navigation system with a visual display device, particularly a surgical monitor or AR glasses, for displaying visual information for the surgical procedure, and a data providing unit that digitally provides a stored preoperative 3D image of the patient. The navigation system is adapted to detect the position and orientation of the visualization unit in a global coordinate system for the aligned patient. In addition, the present disclosure relates to an intraoperative planning method, a computer-readable storage medium, and a computer program for a surgical procedure on a patient as described in the preamble of the adjacent claims.
Background Art
[0002] For example, in neurosurgery, surgical navigation systems are commonly used. Most procedures require intensive preoperative planning, but surgical procedures can be performed guided by these systems. For example, in a conventional biopsy, the surgical procedure is pre-planned using pre-generated image data of the entire (surgical) trajectory or at least the target points of the trajectory without navigation.
[0003] There are approaches to facilitating intraoperative point or trajectory planning, i.e., intraoperative planning performed during a procedure, using standard navigation equipment. When using standard navigation equipment such as pointers / navigation pointers, points in image data are typically defined based on generated navigation information, usually using the tip or a virtual extension of the tip. However, these approaches are very inconvenient and do not provide the user with all the information necessary to make sound judgments. In particular, when planning or completing a trajectory from an existing starting point, users want to obtain or access navigation information along the entire trajectory to ensure that there are no dangerous structures too close to the planned procedure path.
[0004] Here, it is crucial that navigation information is presented to the user in a clear and reliable format. Furthermore, it would be advantageous if users could access a wide range of navigation information and filter or select the information specifically necessary for their actions. [Overview of the project]
[0005] Therefore, this disclosure is based on the objective of avoiding or at least mitigating the aforementioned shortcomings and, in particular, providing a medical AR system and intraoperative planning method in which sufficient information for surgical procedures is presented to the user during surgery and accessible to the user during the operation. Fundamental challenges are seen in confirming the planning of the surgical (intervention) trajectory, and in particular in simplifying it for the user without unnecessarily increasing the risks of the procedure. Another sub-challenge is to integrate preoperatively generated (model) data into the actual surgical (intraoperative) procedure in a reliable and meaningful manner. Furthermore, a sub-challenge is seen in making the process of manipulating and selecting the information required by the user easy and intuitive.
[0006] These problems are solved according to the features of claim 1 relating to a general medical AR system, the features of claim 14 relating to a planning procedure, and the features of claims 17 and 18 relating to a computer-readable storage medium and a computer program.
[0007] Accordingly, this disclosure relates to a medical augmented reality (AR) system for intraoperative planning in patient surgery, particularly neurosurgical procedures, comprising: a movable visualization unit, particularly a microscope head of a surgical microscope, an endoscope, or an optical camera, that can adjust its spatial position and orientation to create and digitally provide real-time images of the patient, particularly internal images; a visual display device, particularly a surgical monitor or a portable head-mounted display, such as AR glasses / AR headset or VR glasses / VR headset or MR glasses / MR headset, for displaying visual information for the surgical procedure; and a navigation system further comprising a data delivery unit that digitally provides stored preoperative 3D images of the patient. The navigation system is designed and adapted to detect the position and orientation of the visualization unit in a global coordinate system relative to the aligned patient, and preferably to detect the (current) depth of focus of the visualization unit. According to this disclosure, the medical AR system further comprises a control unit adapted to generate a cross-section or cross-sectional disk, particularly a virtual cross-section or cross-sectional disk, in relation to the position and orientation of the visualization unit, particularly at a predetermined distance along the optical axis to the visualization unit or at the currently set focal depth of the visualization unit, and preferably at a certain angle of the cross-section with respect to the optical axis of the visualization unit, particularly at an angle perpendicular to the optical axis. The control unit is further adapted to generate a cross-sectional view of a 3D image of the patient generated preoperatively based on the cross-section, generate an AR overlay display of the real-time image and the generated cross-sectional view of the 3D image data, output this AR overlay display through or via a visual display device, thereby visually displaying it to the user.
[0008] In this context, the term "depth of focus" refers to the focal distance from the optical system of the visualization unit to the (focal) plane through the optical system that forms a sharp image on the sensor.
[0009] In other words, a spatially movable visualization unit of the AR system, such as a surgical microscope with a movable microscope head, generates a real-time image of the patient, which is visually displayed via a display device, such as a surgical monitor. Simultaneously, the AR system projects or overlays a preoperative 3D image of the patient onto the display device. The 3D image is generated preoperatively, for example, using computed tomography (CT) or magnetic resonance imaging (MRI). The 3D image forms a virtual (3D) model of the patient, which is virtually overlaid onto the patient's real-time image so that the position and orientation of the virtual model match those of the actual patient. The virtual model is pseudo-overlaid onto the patient's real-time image so that its position and orientation match those of the patient. Furthermore, the AR system has a navigation system, for example, in the form of one or more stereo cameras, which specifically detect the position and orientation of the movable visualization unit in a global coordinate system and store information about its position and orientation. According to this disclosure, the control unit of the AR system generates a cross-sectional image of a virtual model, i.e., selects a cross-section from a preoperatively generated image of the patient based on the (relative) position and orientation detected by the navigation system, i.e., the location of the movable visualization unit relative to or with the patient as the reference point. The cross-sectional view or preoperatively generated image of the virtual 3D model selected by / from the control unit is then visually output by a display device in an AR overlay display combined with the patient's real-time image, so that the (virtual) cross-sectional view of the currently selected model is displayed in the corresponding / anatomically "correct" position in the patient's real-time image. In particular, points in space can then be defined in this cross-sectional view or overlay view, which can serve, for example, as further waypoints.
[0010] In other words, this disclosure particularly describes the planning of points in a preoperative (scanned) model based on AR-based navigation information, in which cross-sectional images from preoperative 3D images / 3D scans are projected / can be projected onto the video output of a visualization unit via a display device. The currently displayed (virtual) image cross-section changes according to the position and orientation / alignment of the movable visualization unit, e.g., a movable microscope head, in relation to the patient. Thus, based on preoperatively generated image data, individual (image) section views are projected using AR onto the anatomically correct position, i.e., the corresponding location, of the patient's real-time image, real-time image, in particular the video feed / video output of the visualization unit. These section views are, in particular, cross-sections of the image data perpendicular to the optical axis of the visualization unit at a certain distance from the visualization unit (this distance can be fixed or adjustable) or at the currently set depth of field (which can also be understood as a distance in some sense, but is a distance that particularly depends on the currently set focus of the visualization unit). In particular, by moving and / or rotating the visualization unit in the real world, the user can change the displayed / selected section view and examine the entire preoperatively generated (scanned) image information at any desired axis.
[0011] In particular, the section view can be adjusted by changing the focus or depth of focus of the visualization unit (and preferably zooming in or out as appropriate by enlarging or reducing the section view), or via an independent control, for example, via a slider control output by a slider / touch display for setting the distance along the optical axis between the section plane and the visualization unit. Thus, for example, if the depth of focus is increased (i.e., an increase in the distance between the plane that is sharply imaged and the visualization unit), the depth of the section or section plane can also be changed. This means that it is not necessary to move the visualization unit toward the patient to produce sections at greater depths.
[0012] This allows for the examination of the entire 3D model / 3D scan volume, and the advantage is that the user can select and define any point on the model for intraoperative planning while complete navigation information is available to the user. Thus, surgical (intervention) trajectories can be planned during surgery, i.e., during the surgical procedure. Furthermore, the operation of the AR system regarding the selection of a cross-sectional view of the virtual model to be displayed as needed is designed to be intuitive for the user, as control or movement of the visualization unit to display information, changing the focus and zoom of the visualization unit, and / or user-independent controls are simple and clear.
[0013] Although this disclosure is described in relation to neurosurgery, it is also applicable to other medical procedures such as spinal surgery, otolaryngology surgery, and general surgery.
[0014] Advantageous embodiments are described in the dependent claims and are described in particular below.
[0015] In an advantageous embodiment, the control unit is adapted to output an AR overlay display and an additional point shape or cross shape indicator through / via a visual display device, thereby allowing the user to select and set the target point or target axis in the currently displayed AR overlay display.
[0016] In further embodiments, the control unit is adapted to set a target point in a global coordinate system, and the display device displays additional point-shaped or cross-shaped indicators in the AR overlay display, thereby allowing the selection of a target point or target axis in the currently displayed section view via the position and / or orientation of the visualization system relative to the patient.
[0017] In other words, a point indicator, preferably in the shape of a crosshair, can be displayed to the user on the AR system's display device, thereby allowing the user to select a point in the currently displayed overlay. Selection is also made by changing the position and / or orientation of the visualization unit (relative to the patient). This allows the user to select or plan, preferably save, a number of points when reviewing / changing / switching virtual dissection views, preferably two or more points connected together to form a trajectory for the surgical procedure. This allows the trajectory to be planned and completed as needed during the procedure, i.e., intraoperatively. At the same time, the user has access to all information from the preoperative model.
[0018] In another preferred embodiment of the present disclosure, the target point setting is stored in a storage unit, and the control unit is adapted to use this target point setting as an entry / initial / starting point or waypoint or target / endpoint of a surgical plan or trajectory.
[0019] In another preferred embodiment of the present disclosure, the medical AR system has a foot pedal, and the control unit is adapted to set and save the (current) indicator upon input via the foot pedal, and / or has a button or key, and the control unit is adapted to set and save the (current) indicator upon input via the button or key, and / or has a microphone, and the control unit is adapted to set and save the (current) indicator upon detection of an acoustic signal such as voice input, in particular a predefined acoustic command by the user, and / or has a touch display, and the control unit is adapted to set and save the (current) indicator upon input. By utilizing these various input means, the user can use an indicator on the display device to set the currently selected point in the currently displayed section view, and the AR system saves this set point in order to eventually form a trajectory with a plurality of these saved points, or at least to save the start or end point of the trajectory. As the visualization unit is moved laterally, the position of the section image changes in relation to the fixed indicator. The indicator allows the user to select a specific point at any location on the current display. By moving the visualization unit near the target point or along an axis of interest, users can also view all the image information necessary for a well-founded intraoperative plan (of the trajectory), especially preoperative scans.
[0020] In another preferred embodiment of the present disclosure, the medical AR system includes an AR headset and / or a 3D monitor and / or a VR (virtual reality) headset as a display device.
[0021] In further embodiments of the present disclosure, the AR system includes a head-mounted display in the form of an AR headset, VR headset, or MR headset worn on the user's head as a visual display device. In this way, a particularly intuitive display method can be provided to a surgeon that enables spatial visualization in a flexible and compact design.
[0022] In other words, real-time images and visual AR overlays from virtual dissected views can be displayed alternately or additionally via a VR or AR headset worn by the user. Furthermore, VR or AR headsets can be equipped with cameras that generate additional real-time reality images. This allows the user to become more deeply immersed in the interaction process.
[0023] In another preferred embodiment of this disclosure, the navigation system incorporates an infrared-based tracking system and / or an electromagnetic tracking system and / or an optical machine vision tracking system and / or a robotic kinematics-based tracking system.
[0024] In other words, the visualization unit is detected, particularly via a navigation system, using, for example, an infrared-based, electromagnetic, or optical tracking system. In either case, the tracking system detects the relative position of the visualization unit to the patient. Similarly, a tracking system may be used that calculates the relative position of the visualization unit in relation to the patient from kinematic data / information from the robot or robotic arm to which the visualization unit is connected.
[0025] In another preferred embodiment of the present disclosure, the visualization unit is connected to the robotic arm of a robot, particularly as an end effector, and the visualization unit has, for example, a joystick on the housing that allows the user to adjust the position and orientation of the visualization unit in space, or the medical AR system has a separate remote control unit adapted to adjust the position and orientation of the visualization unit in space by remote control of the robotic arm. The visualization unit can thus be changed / adjusted with respect to position and orientation by the user's (manual) input, particularly using the joystick or remote control unit, to display a desired (virtual) cutting view.
[0026] In further embodiments of the present disclosure, a visualization unit is connected to a robotic arm of a robot and is defined or definable by user input via a movement axis within a room, in particular the (current) optical axis or a vertical axis in relation to the operating room, or a horizontal axis in the direction of the patient, and a control unit relies on this to perform translational movement of the visualization unit precisely along this movement axis, or the control unit is adapted to control the movement of the visualization unit in a circular mode so that the visualization unit orbits the current focal point at a constant distance.
[0027] In other words, the user can input a way to restrict the freedom of movement of the visualization unit so that it can only move along a single axis (translation). For example, this axis of movement could be perpendicular or horizontal to the operating room floor, or it could run along the currently set optical axis of the visualization unit. Thus, the current optical axis can be locked / selected by the user. If desired, this simplifies the operation of the AR system because there is only one degree of freedom for moving the visualization unit. This allows the user to concentrate solely on the precise selection of the cutting view. Another way to restrict the freedom of movement of the visualization unit is to set the rotation of the robotic arm to which the visualization unit is connected to a fixed rotation point. In this case, the visualization unit performs predictable rotations (around the patient), especially at a constant distance from the patient.
[0028] According to a further embodiment, the control unit may be adapted to generate an AR overlay display such that a generated cut view of the 3D image data is displayed within a circle around a center point through which the optical axis of the visualization unit passes, and preferably the radius of the circle is adjustable, particularly by user input (e.g., via a slider), and a real-time image is displayed outside the circle.
[0029] In particular, the control unit may be alternatively adapted to generate an AR overlay display such that the outer radius and / or the inner radius is adjustable (e.g., via two sliders displayed on a touch display or via default values stored in a storage unit), and within a ring whose center is defined by the optical axis of the visualization unit, the generated cut view of the 3D image data is displayed within the outer and inner radii of the ring, and a real-time image is displayed outside the outer radius and within the inner radius of the ring.
[0030] Alternatively, the control unit may be adapted to generate an AR overlay display such that the generated cut view of the 3D image data and the real-time image are alternately displayed at regular time intervals, particularly as a fade-in fade-out animation. For example, a fixed cross-section of the generated cut view of the 3D image data can be displayed at a fixed position and orientation of the visualization unit, while the depth of the cross-section can be zoomed in and out via an animation, for example, around the current focus of the visualization unit. This has the advantage of making it easier to see what is in front of and behind the current surgical field.
[0031] This disclosure further relates to an intraoperative planning method for surgical treatment of a patient using a medical AR system according to this disclosure, comprising the steps of: providing a preoperative 3D image of the patient stored by a data provision unit of a navigation system; capturing and providing a real-time image of the patient by a movable visualization unit; detecting the position and orientation of the visualization unit in a global coordinate system relative to the patient, and preferably the current depth of focus of the visualization unit, by the navigation system; generating a sectioned view of the patient's preoperative 3D image by a control unit based on the detected position and orientation of the visualization unit (2) and preferably a predetermined distance along the optical axis to the visualization unit (2) or the currently set depth of focus; generating an AR overlay display from the real-time image and sectioned view by the control unit; and outputting the AR overlay display by a visual display device. These steps enable intraoperative planning of surgical procedures, particularly neurosurgical procedures.
[0032] It is preferable that additional point-shaped or cross-shaped indicators (20) are output by / through a visual display device together with or within the AR overlay display.
[0033] In particular, the intraoperative planning method includes an additional step of selecting and setting the target point or target axis in the AR overlay display currently shown by the user.
[0034] This disclosure further relates to computer-readable storage media containing instructions that, when executed by a computer, cause the computer to perform the method steps of this disclosure.
[0035] Furthermore, this disclosure also relates to computer programs that, when executed by a computer, include instructions causing the computer to perform the method steps provided by this disclosure. [Brief explanation of the drawing]
[0036] This disclosure will be described in more detail below with reference to preferred embodiments and the accompanying drawings. [Figure 1] An exemplary perspective side view of a medical AR system according to a first preferred embodiment of the present disclosure is shown. [Figure 2A] This disclosure shows an exemplary oblique AR overlay depiction including cross-sectional views of a virtual 3D model at various cross-sections according to a preferred embodiment of this disclosure. [Figure 2B] This disclosure shows an exemplary oblique AR overlay depiction including cross-sectional views of a virtual 3D model at various cross-sections according to a preferred embodiment of this disclosure. [Figure 2C] This disclosure shows an exemplary oblique AR overlay depiction including cross-sectional views of a virtual 3D model at various cross-sections according to a preferred embodiment of this disclosure. [Figure 2D] This disclosure shows an exemplary oblique AR overlay depiction including cross-sectional views of a virtual 3D model at various cross-sections according to a preferred embodiment of this disclosure. [Figure 3] Figure 2A shows an oblique AR overlay depiction including an indicator according to a preferred embodiment of the present disclosure. [Figure 4] This is a flowchart of the intraoperative planning procedure according to a preferred embodiment of the present disclosure.
[0037] The figures are schematic in nature and are intended only to aid in understanding this disclosure. Identical elements are provided with the same reference numerals. Features of various embodiments are interchangeable and can appear in any combination. [Modes for carrying out the invention]
[0038] This disclosure will be described below with reference to Figures 1-4 and with reference to advantageous embodiments.
[0039] Figure 1 is an exemplary perspective top view of a medical AR system 1 according to a first preferred embodiment of the present disclosure. The AR system 1 has a movable visualization unit 2, which is connected to or attached to the end of a movable robotic arm 3 of a medical robot 4 to adjust both its spatial position (x,y,z) and orientation relative to the patient by controlling the robotic arm 3. In particular, the visualization unit has a joystick (not shown) protruding from its housing, thereby allowing a user 8 to control the movement of the visualization unit. The user / operator 8 thus controls the position of the visualization unit 2, which generates a real-time image 18 of the patient 7.
[0040] Furthermore, the AR system 1 has a navigation system 6 that uses an integrated navigation camera, in this case a stereo camera, to detect the position and orientation of the visualization unit 2 relative to the (aligned) patient 7.
[0041] The AR system 1 further includes a display device 10 and a control unit 12. The control unit 12 is adapted so that the selection of a virtual section view 16 of a digital 3D model of the patient, generated preoperatively, for example by CT or MRI images, is performed based on the current position and orientation of the visualization unit 2. The section view 16 of the selected section of the 3D model is displayed on the display device 10 (overlaid) in the form of an AR overlay display 14 together with a real-time image 18, and is visually displayed to the user 8. The selected virtual section view 16 is thus projected onto the real-time image 18 on the display device 10 using AR. The displayed section view 16 can then be modified by the user 8 by changing the movement / position of the visualization unit 2 in relation to the patient 7. Thus, a pointer is not required, and instead, the user 8 can virtually set points or axes for the navigation system using the section view 16 and the definition of points in the section view 16, in the simplest form being the center point 14 of the AR overlay display, or points selected by the user in the section view 16, in particular entry points (incision points), waypoints, or destination points. The specific creation of the sectioned view 16 and the AR overlay display 14 will be explained below with reference to Figures 2A to 2D and Figure 3.
[0042] Figures 2A–2D are exemplary oblique AR overlay depictions 14 of a reclining patient 7, including section views 16 of various cross-sections of a preoperatively generated virtual 3D model, according to a preferred embodiment of the present disclosure. The (virtual) section views 16 of various cross-sections are indicated by diagonal lines in Figures 2A–2D, particularly running vertically through the patient 7. In addition, in each of the figures 2A–2D, different section views 16 are projected onto the same real-time image 18 of the patient 7. The various cross-sections for each section view 16 can be selected by the user 8 by changing the position of the visualization unit 2, particularly by changing its translational position along the horizontal axis through the head and feet of the patient 7.
[0043] For example, Figure 2A shows a vertically running section view 16 that runs through the lower half of patient 7's head, while Figure 2D shows a vertically running section view 16 that runs through patient 7's skull. In particular, these different section views 16 run parallel to each other.
[0044] User 8 can, of course, define the cross-section 16 themselves, as shown in Figure 2C. For example, the cross-section can be defined perpendicular to the optical axis of the visualization unit at a predetermined distance from the visualization unit to, for example, the front lens of the optical system, or it can be set to the currently set depth of field of the visualization unit 2. Thus, the visualization unit 2 has a virtual cross-section 16 that is statically connected to the front of the visualization unit 2 and can move with it. As the visualization unit 2 is moved translationally along its optical axis, the cross-section 16 moves accordingly through the 3D model of the patient, creating a new section view in real time. When User 8 stops moving the visualization unit 2, i.e., the microscope head, in space, an oblique AR overlay display 14 is output to the user, and in this view, User 8 can define points for intervention during surgery without having to use a navigation pointer. Such point definitions are shown and explained below in Figure 3.
[0045] In particular, the surgeon can change the depth of focus by manually adjusting the focus, for example, via a touch display or a focus setting dial, and the depth of the cross section (perpendicular to the optical axis in this embodiment) can also be adjusted accordingly using the depth of focus. Therefore, the surgeon does not need to move the entire visualization unit 2 to change the depth of the cross section view 16; changing the focus is sufficient.
[0046] Figure 3 shows an oblique AR overlay display 14 from Figure 2A including (with) an indicator 20 according to a preferred embodiment of the present disclosure. The indicator 20 of the AR overlay display 14 currently displayed on the display device 10 is preferably designed in the shape of a crosshair, which the user 8 can use to select a point (or axis) in the currently selected / displayed section view 16 via the position and / or orientation of the visualization unit 2 relative to the patient 7. By manual input by the user 8, for example via a foot pedal, the user 8 can also save the point currently selected on the indicator 20 via the control unit 20, for example as an entry point, waypoint, or target point of the surgical plan. This allows points in the 3D model of the patient to be defined very precisely, in particular, so that they can be set as waypoints or as selections of characteristic landmarks. In this way, the plan can be further adjusted and navigation can be performed during surgery.
[0047] Alternatively, in Figure 3, user 8 can manually define or set the indicator 20 in section 16 by using the touch display, for example, by pressing their finger to a desired location on the touch display, similar to focusing a camera on the touch display. The control unit 12 then accurately calculates this position on section 16 accordingly and defines a point that can thus be used for navigation.
[0048] Figure 4 shows a flowchart of an intraoperative (computer-aided) planning method for AR System 1 according to a preferred embodiment of the present disclosure.
[0049] In the first step S1, the planning method uses the data provision unit of the navigation system 6 to provide (digital or computer-readable) preoperative (model) 3D images of the patient 7.
[0050] In the second step S2, the planning method uses the movable visualization unit 2 to capture and make available real-time images 18 of the patient 7.
[0051] In the third step S3, the planning method uses the navigation system 6 to detect the (current) position and (current) orientation of the visualization unit 2 in a global coordinate system relative to the patient 7.
[0052] In the fourth step S4, the planning method generates a sectioned view 16 of the patient 7's preoperative 3D image based on the position and orientation of the visualization unit 2 detected using the control unit 12.
[0053] In the fifth step S5, the planning method uses the control unit 12 to generate an AR overlay display 14 from the real-time image 18 and the sectioned view 16.
[0054] In the final step S6, the planning method outputs an AR overlay display 14 via the visual display device 10. It is preferable that additional point-shaped or cross-shaped indicators are also output by / through the visual display device.
[0055] In a further step (S7), the indicator 20 is then set in the AR overlay display 14, thereby selecting and defining an axis that spans a (3D) point (in space) or multiple points.
[0056] In any further optional step, this defined point can then be used for navigation, particularly as a waypoint or destination. [Explanation of symbols]
[0057] 1. Medical AR System 2 Visualization Units 3 Robotic Arms 4 Medical robots 6. Navigation System 7 patients 8 users 10 Display Devices 12 Control Unit 14 AR Overlay Display 16 Cutting View 18 Real-time images 20 Indicators Step S1: Providing preoperative 3D images. Steps to capture and provide S2 real-time images S3 Step to detect the position and orientation of the visualization unit. Steps to generate an S4 disconnected view Steps to generate an S5 AR overlay display Steps to output S6 AR overlay display S7 Step to select and set the target point or target axis
Claims
1. A medical AR system (1) for intraoperative planning during surgical, particularly neurosurgical, procedures on a patient (7), A movable visualization unit (2) that can adjust its position and orientation to create and digitally provide real-time images (18), particularly intracellular images, of the patient (7), A visual display device (10) for displaying visual information for the aforementioned surgical procedure, A navigation system (6) comprising a data provision unit that digitally provides stored preoperative 3D images of the patient (7), wherein the navigation system (6) is adapted to detect the position and orientation of the visualization unit (2) in a global coordinate system relative to the aligned patient (7), and preferably the depth of focus of the visualization unit (2). Includes, The aforementioned medical AR system (1) is In relation to the position and orientation of the visualization unit (2), a cross-section is generated at a predetermined distance along the optical axis to the visualization unit (2) or at the currently set depth of focus of the visualization unit (2). Based on the cross-section, a sectioned view (16) of the 3D image of the patient (7) generated before the surgery is created. An AR overlay display (14) is generated using the current image (18) and the generated section view (16) of the 3D image data, and this AR overlay display (14) is output through the visual display device (10). Further includes a control unit (12) adapted to such a configuration. A medical AR system (1) characterized by the following.
2. The medical AR system (1) according to claim 1, wherein the control unit (12) is adapted to output the AR overlay display (14) and an additional point shape or cross shape indicator (20) through / via the visual display device (10), thereby enabling the user (8) to select and set the target point or target axis in the currently displayed AR overlay display (14).
3. The medical AR system (1) according to claim 2, wherein the control unit (12) is adapted to set an objective point in a global coordinate system, and an objective point or target axis in the currently displayed section view (16) can be selected via the position and / or orientation of the visualization system (2) relative to the patient (7).
4. The medical AR system (1) according to claim 2 or 3, characterized in that the setting of the objective point is stored in a storage unit, and the control unit (12) is adapted to use this setting of objective point as an entry point, waypoint, or target point for a surgical plan.
5. The aforementioned medical AR system (1) is It has a foot pedal, and the control unit (12) is adapted to set and save the indicator (20) when input is made using the foot pedal, and / or Having a button or key, the control unit (12) is adapted to set and save the indicator (20) when input is received via the button or key, and / or The control unit (12) is equipped with a microphone and is adapted to set and save the indicator (20) when it detects an acoustic signal such as a voice input, in particular a predefined acoustic command, and / or The control unit (12) is configured to set and save the indicator (20) when an input is received, and has a touch display. A medical AR system (1) according to any one of claims 2 to 4, characterized in that...
6. The medical AR system (1) according to any one of claims 1 to 5, characterized in that the medical AR system (1) has an AR headset and / or a 3D monitor and / or a VR headset as the display device (10).
7. The medical AR system (1) according to claim 1, characterized in that the AR system (1) has a head-mounted display in the form of an AR headset, VR headset, or MR headset that can be worn on the head by a user (8) as the visual display device (10).
8. The medical AR system (1) according to any one of claims 1 to 7, characterized in that the navigation system (6) has an infrared-based tracking system, an electromagnetic tracking system, an optical machine vision tracking system, or a robot kinematics-based tracking system.
9. The visualization unit (2) is connected to the robot arm (3) of the robot (4), The visualization unit (2) has a joystick that allows the user (8) to adjust the position and orientation of the visualization unit (2) in space, or The medical AR system (1) has a separate remote control unit adapted to adjust the position and orientation of the visualization unit (2) in space via remote control of the robotic arm (3). A medical AR system (1) according to any one of claims 1 to 8, characterized in that
10. The visualization unit (2) is connected to the robot arm (3) of the robot (4), The axis of movement in space can be defined or defined by user input via the optical axis, or vertical axis, or horizontal axis in the direction of the patient, and the control unit (12) relies on this to precisely perform the translational movement of the visualization unit (2) along this axis of movement, or The control unit (12) is adapted to control the movement of the visualization unit (2) in a circular mode so that the visualization unit (2) circles the current focus at a constant distance. A medical AR system (1) according to any one of claims 1 to 9, characterized in that
11. The medical AR system (1) according to any one of claims 1 to 10, characterized in that the control unit (12) is adapted to generate the AR overlay display (14) such that the generated section view (16) of the 3D image data is displayed within a circle, which is preferably a circle whose radius is particularly adjustable by user input, and the real-time image (18) is displayed outside the circle.
12. The medical AR system (1) according to any one of claims 1 to 10, characterized in that the control unit (12) is adapted to generate the AR overlay display (14) such that the generated section view (16) of the 3D image data is displayed within the outer ring radius and the inner ring radius in a ring whose center is defined by the optical axis of the visualization unit (2), and the real-time image (18) is displayed outside the outer ring radius and inside the inner ring radius.
13. The medical AR system (1) according to any one of claims 1 to 10, characterized in that the control unit (12) is adapted to generate the AR overlay display (14) such that the generated section view (16) of the 3D image data and the real-time image (18) are displayed alternately at regular time intervals.
14. In particular, an intraoperative planning method for surgical treatment of a patient (7) using a medical AR system (1) as described in any one of claims 1 to 13, - Step (S1) of providing a preoperative 3D image of the patient (7) stored by the data provision unit of the navigation system (6), - Step (S2) of capturing and providing a real-time image (18) of the patient (7) using a movable visualization unit (2), - Step (S3) of detecting the position and orientation of the visualization unit (2) in a global coordinate system relative to the patient (7) by the navigation system (6), and preferably the depth of focus of the visualization unit (2), - A control unit (12) generates a sectioned view (16) of the preoperative 3D image of the patient (7) based on the detected position and orientation of the visualization unit (2) and preferably a predetermined distance along the optical axis to the visualization unit (2) or the currently set depth of field; - The control unit (12) generates an AR overlay display (14) from the real-time image (18) and the sectioned view (16) (S5), - Step (S6) of outputting the AR overlay display (14) using the visual display device (10) and An intraoperative planning method characterized by the following.
15. The intraoperative planning method according to claim 14, characterized in that in the output step (S6), an additional point-shaped or cross-shaped indicator (20) is output by / through the visual display device (10).
16. The intraoperative planning method according to claim 14 or 15, further comprising the additional step (S7) of selecting and setting an objective point or target axis in the AR overlay display (14) currently displayed by the user (8).
17. A computer-readable storage medium that includes instructions, when executed by a computer, causing the computer to perform the steps according to any one of claims 14 to 16.
18. A computer program that, when executed by a computer, includes instructions causing the computer to perform the steps according to any one of claims 14 to 16.