Medical ar system for intraoperative planning during a surgical procedure, and intraoperative planning method

Abstract

The disclosure relates to a medical AR system (1) for intraoperative planning, comprising a mobile visualisation unit (2) the position and orientation of which can be adjusted, and which creates and digitally supplies a real-time image (18) of the patient (7), in particular an intracorporeal image, and also comprising a visual display device (10) and a navigation system (6) with a data supply unit, which digitally supplies stored preoperative 3D images of the patient (7), wherein the navigation system (6) is adapted to detect a position and orientation of the visualisation unit (2) in a global coordinate system with respect to the recorded patient (7). The medical AR system (1) also has a control unit (12) that is designed to generate a sectional plane in relation to the position and orientation of the visualisation unit (2), to generate a sectional view (16) of the preoperatively generated 3D images of the patient (7) on the basis of the sectional plane, to generate an AR overlay representation (14) of the current image (18) and the generated sectional view (16) of the 3D image data, and to output this representation on the visual display device (10). The disclosure also relates to an intraoperative planning method and to a computer-readable storage medium and computer program according to the additional independent claims.

Description

[0001] Medical AR system for intraoperative planning during a surgical procedure and method for intraoperative planning

[0002] Description

[0003] Technical area

[0004] The present disclosure relates to a medical AR system for intraoperative planning during a surgical, in particular neurosurgical, procedure on a patient. The AR system comprises a spatially movable visualization unit, in particular a microscope head of a surgical microscope, an endoscope, or an optical camera, whose spatial position and orientation are adjustable, and which creates and digitally provides a current image of the patient, in particular an intracorporeal image. Furthermore, the AR system comprises a visual display device, in particular a surgical monitor or AR glasses, for displaying visual information for the surgical procedure, and a navigation system with a data provision unit, which digitally provides stored preoperative 3D images of the patient.The navigation system is adapted to detect a position and orientation of the visualization unit in a global coordinate system relative to the registered patient. Furthermore, the present disclosure relates to an intraoperative planning method for a surgical procedure on a patient, a computer-readable storage medium, and a computer program according to the preambles of the independent claims.

[0005] Background of the Revelation

[0006] In neurosurgery, for example, surgical navigation systems are commonly used. They enable the performance of navigated surgical procedures, although most procedures require intensive preoperative planning. For example, in a conventional biopsy, the surgical procedure is planned in advance along an entire (procedural) trajectory or at least a target point of the trajectory without navigation using preoperatively generated image data.

[0007] There are approaches to facilitate intraoperative point or trajectory planning using standard navigation instruments, i.e., intraoperative planning that takes place during the procedure. When using standard navigation instruments such as a pointer / navigation pointer, either their tip or a virtual extension of it is typically used to define a point in the image data based on generated navigation information. However, these approaches are quite cumbersome and do not provide the user with all the information needed to make an informed decision. In particular, when planning or completing a trajectory with an existing starting point, the user wants to obtain or access navigation information along the entire trajectory to ensure that no high-risk structures are located too close to the path of the planned procedure.

[0008] It is crucial that navigation information is presented clearly and reliably to the user. It would also be advantageous if the user had access to a wide range of navigation information and could filter or select the information specifically required for the procedure.

[0009] Summary of Revelation

[0010] Therefore, the present disclosure is based on the object of avoiding or at least mitigating the disadvantages described above and, in particular, of providing a medical AR system and an intraoperative planning method with which sufficient information for a surgical procedure is presented to a user intraoperatively and which the user can access during the operation. A fundamental object can be seen, in particular, in ensuring the planning of a surgical (procedure) trajectory and, in particular, simplifying it for the user without unnecessarily increasing the risk of the procedure. A further sub-objective is to reliably and meaningfully integrate preoperatively generated (model) data into the actual (intra)operative procedure.Furthermore, a sub-task can be seen in making the operation and selection process for the required information easier and more intuitive for the user.

[0011] The objects are achieved according to the invention with regard to a generic medical AR system by the features of claim 1, with regard to a planning method by the features of claim 14 and with regard to a computer-readable storage medium and a computer program by the features of claims 17 and 18.

[0012] The disclosure accordingly relates to a medical AR (Augmented Reality) system for intraoperative planning during a surgical, in particular neurosurgical, intervention on a patient, comprising a movable visualization unit, in particular a microscope head of a surgical microscope, an endoscope or an optical camera, the spatial position and orientation of which is adjustable, and which creates and digitally provides a current image of the patient, in particular an intracorporeal image, a visual display device, in particular an operating room monitor or a portable head-mounted display, e.g. AR glasses / AR headset or VR glasses / AR headset or MR glasses / MR headset, in order to display visual information for the surgical intervention, and furthermore a navigation system with a data provision unit which digitally provides stored preoperative 3D images of the patient.The navigation system is provided and adapted to detect a position and orientation of the visualization unit in a global coordinate system relative to the registered patient and preferably a (current) depth of focus of the visualization unit. According to the disclosure, the medical AR system further comprises a control unit adapted to generate a, in particular virtual, sectional plane or slice in relation to the position and orientation of the visualization unit, in particular at a predetermined distance along an optical axis from the visualization unit or at a currently set depth of focus of the visualization unit and preferably at an angle of the sectional plane to an optical axis of the visualization unit, in particular perpendicular to the optical axis.The control unit is further adapted to generate a sectional view of the preoperatively generated 3D images of the patient based on the sectional plane, to generate an AR overlay display with the current image and the generated sectional view of the 3D image data, and to output this AR overlay display through or via the visual display device and thus to visually display it for a user.

[0013] The term focus depth refers in this case to a focus distance from an optical system of the visualization unit to a (focus) plane, which is sharply imaged on a sensor via the optical system.

[0014] In other words, the (spatially) movable visualization unit of the AR system, e.g., a surgical microscope with a movable microscope head, generates a current image of the patient, which is visually displayed on a display device, e.g., an operating room monitor. At the same time, the AR system projects preoperative 3D images of the patient onto the display device or overlays them. The 3D images are generated preoperatively, for example, using computed tomography (CT) or magnetic resonance imaging (MRI). The 3D images form a virtual (3D) model of the patient and are virtually overlaid on the current image of the patient in such a way that the position and orientation of the virtual model match the position and orientation of the real patient.The virtual model is, so to speak, superimposed on the currently displayed patient in a position and orientation that matches that of the patient. Furthermore, the AR system has a navigation system, for example in the form of one or more stereo cameras, which records the position and orientation of the movable visualization unit, in particular in a global coordinate system, and stores information about its position and orientation. According to the disclosure, a control unit of the AR system generates sectional images of the virtual model, i.e. selects sections from the preoperatively generated images of the patient based on the (relative) position and orientation recorded by the navigation system, i.e. the position of the movable visualization unit, in particular relative to the patient or with the patient as a reference point. The sectional view of the virtual 3D model or the resulting sectional view of the virtual 3D model or the resulting sectional view of the movable visualization unitThe preoperatively generated images are then visually output by the display device in an AR overlay display combined with the current image of the patient, such that the currently selected (virtual) sectional view of the model is displayed at the corresponding / anatomically "correct" position in the current image of the patient. In particular, a point in space can then be defined in this sectional view or overlay view, which serves, for example, as an additional waypoint.

[0015] In other words, the disclosure describes, in particular, the planning of points in a preoperative (scan) model based on navigation information using AR, in which cross-sectional images from the preoperative 3D recordings / 3D scans are projected / can be projected into the video output of a visualization unit via the display device. The change in the currently displayed (virtual) image section occurs depending on the position and orientation / alignment of the movable visualization unit, for example, a movable microscope head, with respect to the patient. Thus, based on preoperatively generated recording data, individual (image) cross-sectional views are projected using AR at an anatomically correct position, i.e., at the corresponding location in the current recording of the patient, in the current recording, in particular a video feed / video output, of the visualization unit.These sectional views are, in particular, cross-sections of the acquisition data perpendicular to an optical axis of the visualization unit at a specific distance (the distance can be fixed or adjustable) or a currently set focus depth (also understood as a distance, which, however, depends specifically on the currently set focus of the visualization unit) from the visualization unit. By moving and / or rotating the visualization unit in the real world, the user can change the displayed / selected sectional view and inspect the entire preoperatively generated (scanned) acquisition information in any desired axis.

[0016] In particular, the cross-sectional view can be created by changing the focus or depth of focus (and preferably also the zoom, by enlarging or reducing the cross-sectional view accordingly) of the visualization unit or by an independent control, for example via a slider / slider displayed on a touch display for setting a distance along the optical axis between the cutting plane and the visualization unit. For example, if the depth of focus is increased (i.e., an increase in the distance between a sharply imaged plane and the visualization unit), the depth of the cut or cutting plane can also be changed. This means that it is not necessary to move the visualization unit towards the patient in order to create cuts at a greater depth.

[0017] Advantageously, this enables inspection of the entire 3D model / 3D scan volume and allows the user to select and define any points in the model for intraoperative planning, while the complete navigation information is available to the user. A surgical (procedure) trajectory can thus be planned intraoperatively, i.e., during the surgical procedure. Furthermore, operation of the AR system and the selection of the cross-sectional views of the virtual model to be displayed as needed are designed to be intuitive for the user, as the information display is simple and clear by controlling or moving the visualization unit, by changing the focus and zoom of the visualization unit, and / or by a separate control via the user.

[0018] The present disclosure is described particularly in connection with neurosurgery, but also applies to other medical procedures such as spinal, ENT and general surgery.

[0019] Advantageous embodiments are claimed in the subclaims and are explained in particular below.

[0020] In an advantageous embodiment, the control unit is adapted to output the AR overlay representation and an additional point-shaped or crosshair-shaped indicator through / via the visual display device, with which a target point or a target axis in the currently displayed AR overlay representation can be selected and set by a user. In a further embodiment, the control unit is adapted to perform the target point setting in the global coordinate system, and the display device displays the additional point-shaped or crosshair-shaped indicator in the AR overlay representation, whereby the target point or the target axis in the currently displayed sectional view can be selected via the position and / or orientation of the visualization system relative to the patient.

[0021] In other words, a point indicator can be displayed to the user on the display device of the AR system, preferably in the form of crosshairs, with which the user can select a point in the currently displayed overlay representation. The selection is also made by changing the position and / or orientation of the visualization unit (relative to the patient). This allows the user to select or plan, and preferably save, a large number of points when browsing through / changing / switching the virtual sectional views, with two or more connected points preferably forming a trajectory for a surgical procedure. This enables the planning and possible completion of the trajectory during the procedure, i.e. intraoperatively. At the same time, all information from the preoperative model is accessible to the user.

[0022] In a further preferred aspect of the disclosure, the target point setting is stored in a storage unit and the control unit is adapted to use this target point setting as an entry point / starting point / starting point or as a waypoint or as a destination point / end point for an operational plan or trajectory.

[0023] In a further preferred aspect of the disclosure, the medical AR system comprises a foot pedal and the control unit is adapted to set and store a (current) indicator upon input via the foot pedal, and / or comprises a button or push button and the control unit is adapted to set and store a (current) indicator upon input via the button or push button, and / or comprises a microphone and the control unit is adapted to set and store a (current) indicator upon detection of an acoustic signal such as a voice input, in particular a predefined acoustic command by the user, and / or comprises a touch display and the control unit is adapted to set and store a (current) indicator upon input.Using these different input devices, the user is able to set the currently selected point in the currently displayed cross-sectional view using the indicator on the display device. The AR system saves this set point, ultimately creating a trajectory from a multitude of these saved points, or at least saving a start or end point of a trajectory. If the visualization unit is moved laterally, the position of the cross-sectional image changes relative to the fixed indicator. The indicator allows the user to select a specific point anywhere in the current display.By moving the visualization unit near the target point or along an axis of interest, the user is also able to view all acquisition information, especially preoperatively scanned information, required for sound intraoperative planning (a trajectory).

[0024] In a further preferred aspect of the disclosure, the medical AR system comprises an AR headset and / or a 3D monitor and / or a VR (Virtual Reality) headset as the display device.

[0025] In a further embodiment of the disclosure, the AR system comprises a head-mounted display in the form of an AR headset, a VR headset, or an MR headset worn by the user as the visual display device. This provides a surgeon with a particularly intuitive display method that enables flexible, compact spatial visualization.

[0026] In other words, the visual AR overlay of the current recording and the virtual cross-sectional view can be displayed alternatively or additionally via a VR or AR headset worn by the user. It is also possible to equip the VR or AR headset with a camera that generates additional, real-time recordings. This allows the user to be even more immersed in the surgical procedure.

[0027] In a further preferred aspect of the disclosure, the navigation system comprises an infrared-based tracking system and / or an electromagnetic tracking system and / or an optical machine vision tracking system and / or a robot kinematics-based tracking system.

[0028] In other words, the visualization unit is detected primarily via the navigation system, for example, using an infrared-based, electromagnetic, or optical tracking system. In all cases, the tracking system detects the relative position of the visualization system to the patient. A tracking system can also be used that calculates the relative position of the visualization system to the patient from kinematic data / information from the robot or robot arm to which the visualization unit is connected.

[0029] In a further preferred aspect of the disclosure, the visualization unit is connected to a robot arm of a robot, in particular as a terminal end effector, and the visualization unit has a joystick, for example on a housing, with which the user can adjust the position and orientation of the visualization unit in space. Or the medical AR system has a separate remote control adapted to adjust the position and orientation of the visualization unit in space by means of a remote control of the robot arm. The position and orientation of the visualization unit can thus be changed / adjusted by a (manual) input from the user, in particular using a joystick or a remote control, in order to display the desired (virtual) sectional views.

[0030] In a further embodiment of the disclosure, the visualization unit is connected to a robot arm of a robot and is defined or definable by means of a user input via a movement axis in space, in particular the (current) optical axis or a vertical axis, in particular with respect to an operating room, or a horizontal axis in the patient direction, which the control unit uses to carry out the translational movement of the visualization unit precisely along this movement axis, or the control unit is adapted to control the movement of the visualization unit in an orbit mode such that it orbits the current focal point at a constant distance. In other words, the user is able to make an input and thereby restrict the freedom of movement of the visualization unit such that it can only be moved (translationally) along a single axis.For example, this axis of movement is perpendicular or horizontal to the floor of the operating room, or the axis of movement runs along the currently set optical axis of the visualization unit. A current optical axis can therefore be locked / selected by the user. This makes it easier to operate the AR system if desired, as only one degree of freedom is available for moving the visualization unit. This allows the user to concentrate solely on precisely selecting the cross-sectional views. Another option for restricting the freedom of movement of the visualization unit is to set a rotation of the robot arm, to which the visualization unit is connected, to a fixed rotation point. The visualization unit then performs a predictable rotation (around the patient), in particular at a constant distance from the patient.

[0031] According to a further embodiment, the control unit can be adapted to generate the AR overlay representation in such a way that within a circle around a center point through which the optical axis of the visualization unit runs, and whose circle radius is preferably adjustable, in particular by a user input (for example via a slider), the generated sectional view of the 3D recording data is displayed and outside the circle the current recording is displayed.

[0032] In particular, the control unit can alternatively be adapted to generate the AR overlay display in such a way that, in a ring whose outer ring radius and / or inner ring radius can be adjusted (for example via two sliders displayed on a touch display or by a specification stored in a memory unit) and whose center point is defined by the optical axis of the visualization unit, the generated sectional view of the 3D recording data is displayed within the ring's outer radius and inner radius, and the current recording is displayed outside the ring's outer radius and within the ring's inner radius. Preferably, the control unit can alternatively be adapted to generate the AR overlay display in such a way that the generated sectional view of the 3D recording data and the current recording are displayed alternately at regular time intervals, in particular as a fade-in-fade-out animation.For example, a fixed section of the generated cross-sectional view of the 3D acquisition data can be displayed with a fixed position and orientation of the visualization unit, while the depth of the section can be zoomed in and out via animation, for example, around the current focal point of the visualization unit. This advantageously makes it easy to see what lies in front of and behind the current surgical field.

[0033] The present disclosure further relates to an intraoperative planning method for the surgical treatment of a patient, in particular with a medical AR system according to the disclosure, comprising the steps of providing stored preoperative 3D images of a patient by a data provision unit of a navigation system, recording and providing a current image of the patient by a movable visualization unit, detecting a position and an orientation of the visualization unit in a global coordinate system relative to the patient and preferably a current focus depth of the visualization unit by the navigation system, generating, by a control unit,a sectional view of the preoperative 3D images of the patient based on the detected position and orientation of the visualization unit (2) and preferably a predetermined distance along an optical axis to the visualization unit (2) or a currently set depth of focus; generating an AR overlay representation from the current images and the sectional view by the control unit; and outputting the AR overlay representation by the visual display device. These steps enable intraoperative planning of a surgical, in particular neurosurgical, intervention.

[0034] Preferably, an additional point-shaped or crosshair-shaped indicator (20) is output together with or in the AR overlay representation by / via the visual display device. Particularly preferably, the intraoperative planning method comprises an additional step of selecting and setting a target point or a target axis in the currently displayed AR overlay representation by a user.

[0035] The present disclosure further relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method steps according to the disclosure.

[0036] Furthermore, the present disclosure further relates to a computer program with instructions which, when executed by a computer, cause the computer to carry out the method steps according to the disclosure.

[0037] Short description of the characters

[0038] The disclosure is explained in more detail below using preferred embodiments with the aid of the accompanying figures. They show:

[0039] Fig. 1 is an exemplary perspective side view of the medical AR system according to a first preferred embodiment of the present disclosure;

[0040] 2A to 2D show exemplary perspective AR overlay representations with sectional views in different section planes of a virtual 3D model according to a preferred embodiment of the present disclosure,

[0041] Fig. 3 is the perspective AR overlay representation of Fig. 2A with an indicator according to a preferred embodiment of the present disclosure, and

[0042] Fig. 4 is a flowchart of an intraoperative planning method according to a preferred embodiment of the present disclosure.

[0043] The figures are schematic in nature and serve only to facilitate understanding of the disclosure. Like elements are provided with the same reference numerals. The features of the various embodiments can be interchanged and may occur in any combination.

[0044] Description of the embodiments

[0045] The present disclosure will now be described using an advantageous embodiment with reference to Figures 1 to 4.

[0046] Figure 1 is an exemplary perspective top view of a medical AR system 1 according to a first preferred embodiment of the present disclosure. The AR system 1 has a movable visualization unit 2, which is connected to a movable robot arm 3 of a medical robot 4 or is connected to the robot arm 3 at its end in order to set both a position (x, y, z) and an orientation in space relative to the patient by controlling the robot arm 3. In particular, the visualization unit has a joystick (not shown) protruding from its housing, with which a user 8 controls the movement of the visualization unit. The user / operator 8 thus controls the position of the visualization unit 2, which generates a time-current image 18 of the patient 7.

[0047] Furthermore, the AR system 1 has a navigation system 6, which uses an integrated navigation camera, in this case a stereo camera, to detect the position and orientation of the visualization unit 2 relative to a (registered) patient 7.

[0048] The AR system 1 also has a display device 10 and a control unit 12. The control unit 12 is adapted such that a selection of virtual sectional views 16 of a digital 3D model of the patient, generated preoperatively, for example, using CT or MRI scans, is made based on the current position and orientation of the visualization unit 2. The selected sectional view 16 of a sectional plane of the 3D model is displayed (superimposed) on the display device 10 together with the current image 18 in the form of an AR overlay display 14 and is visually displayed to the user 8. The selected virtual sectional view 16 is thus, in a sense, projected onto the current image 18 using AR in the display device 10. This displayed sectional view 16 can then be changed by the user 8 via a movement / change in position of the visualization unit 2 in relation to the patient 7.A pointer is therefore no longer required. Instead, the user 8 can virtually set a point or an axis for the navigation system, in particular an entry point (incision point), a waypoint, or a destination point, using the sectional view 16 and a definition of a point in the sectional view 16—in the simplest form, the center of the AR overlay display 14 or a self-selected point in the sectional view 16. The specific creation of the sectional view 16 and the AR overlay display 14 is explained below with reference to Figs. 2A to 2D and Fig. 3.

[0049] Figures 2A to 2D are exemplary perspective AR overlay representations 14 of a lying patient 7 with sectional views 16 in various sectional planes of the virtual preoperatively generated 3D model according to a preferred embodiment of the present disclosure. The (virtual) sectional views 16 in various sectional planes are shown hatched in Figures 2A-2D and run, in particular, vertically through the patient 7. Furthermore, in each of Figures 2A-2D, a different sectional view 16 is projected onto the same current image 18 of the patient 7. Different sectional planes for a respective sectional view 16 can be selected by the user 8 via a change in the position of the visualization unit 2, here, in particular, a translational change in position along the horizontal axis through the head and feet of the patient 7.

[0050] For example, Figure 2A shows a vertically extending sectional view 16 that runs through the lower half of the patient's head 7, whereas Figure 2D shows a vertically extending sectional view 16 that runs through the skullcap of the patient 7. In particular, these different sectional views 16 run parallel to one another. The user 8 can of course also define the sectional plane 16 themselves, as indicated in Fig. 2C. For example, the sectional plane can be defined perpendicular to an optical axis of the visualization unit at a predetermined distance from the visualization unit, such as a front lens of an optical system, or can be set at a currently set depth of focus of the visualization unit 2. The visualization unit 2 therefore has a virtual sectional plane 16, which is, so to speak, statically attached to the front of the visualization unit 2 and can be moved with it.If the visualization unit 2 is moved translationally along its optical axis, the section plane 16 moves accordingly through the patient's 3D model and creates the new section view in real time. If the user 8 stops a movement of the visualization unit 2, for example, the microscope head in space, the perspective AR overlay representations 14 are displayed, and in this view, the user 8 can then define a point for an intervention intraoperatively without having to use a navigation pointer. Such a definition of the point is shown and described below in Fig. 3.

[0051] In particular, the surgeon can change the focus depth by manually changing the focus, for example, via an input using a touch display or a focus adjustment wheel. Using the focus depth, they can also adjust the depth of the cutting plane (which, in this embodiment, is perpendicular to the optical axis). Thus, to change the depth of the cross-sectional view 16, the surgeon does not need to move the entire visualization unit 2; a change in the focus is sufficient.

[0052] Figure 3 shows the perspective AR overlay representation 14 from Fig. 2A with a (set) indicator 20 according to a preferred embodiment of the present disclosure. In the AR overlay representation 14 currently displayed in the display device 10, the indicator 20 is preferably crosshair-shaped, with the aid of which the user 8 can select a point (or axis) in the currently selected / displayed sectional view 16 via the position and / or orientation of the visualization system 2 relative to the patient 7. Through manual input by the user 8, e.g., via a foot pedal, the user 8 is also able to save the point currently selected with the indicator 20, e.g., as an entry point, a waypoint, or a destination point for an operative plan, via the control unit 20.This allows a point in the patient's 3D model to be defined very precisely, which can be set as a waypoint or by selecting a characteristic landmark. This allows for intraoperative planning and navigation adjustments.

[0053] Alternatively, in Fig. 3, it is possible for the user 8 to manually define or set an indicator 20 in the section plane 16 using a touch display, for example by pressing a finger on the touch display at the desired location, similar to focusing a camera on a touch display. The control unit 12 then calculates this position accordingly on precisely this section plane 16 and thus establishes a point that can be used for navigation.

[0054] Figure 4 shows a flowchart of an intraoperative (computer-implemented) planning method for the AR system 1 according to a preferred embodiment of the present disclosure.

[0055] In a first step S1, the planning method provides stored preoperative (model-like) 3D images of the patient 7 using a data provision unit of the navigation system 6 (digital or computer-readable).

[0056] In a second step S2, the planning method takes a current image 18 of the patient 7 through the movable visualization unit 2 and makes it available.

[0057] In a third step S3, the planning method records the (current) position and the (current) orientation of the visualization unit 2 in a global coordinate system relative to the patient 7 using the navigation system 6;

[0058] In a fourth step S4, the planning method generates a sectional view 16 of the preoperative 3D images of the patient 7 based on the detected position and orientation of the visualization unit 2 using the control unit 12. In a fifth step S5, the planning method generates an AR overlay representation 14 from the current image 18 and the sectional view 16 using the control unit 12.

[0059] In a final step S6, the planning method outputs the AR overlay representation 14 via the visual display device 10. Preferably, an additional point-shaped or crosshair-shaped indicator is also output by / via the visual display device.

[0060] In a further step (S7), an indicator 20 can then be set in the AR overlay display 14 and thus a (3D) point or an axis across several points (in space) can be selected and defined.

[0061] In a further optional step, this defined point can then be used for navigation, in particular as a waypoint or as a destination point.

[0062] List of reference symbols

[0063] 1 Medical AR system

[0064] 2 Visualization unit

[0065] 3 Robot arm

[0066] 4 Medical robot

[0067] 6 Navigation system

[0068] 7 patients

[0069] 8 users

[0070] 10 Display device

[0071] 12 Control unit

[0072] 14 AR overlay display

[0073] 16 Sectional view

[0074] 18 Current recording

[0075] 20 Indicator

[0076] S1 Step Providing preoperative 3D images

[0077] S2 Step Recording and providing a timely recording

[0078] S3 Step Determining a position and orientation of the

[0079] Visualization unit

[0080] S4 Step Creating a section view

[0081] S5 Step Creating an AR overlay representation

[0082] S6 Step Outputting the AR overlay representation

[0083] S7 Step Selecting and setting a target point or a target

[0084] axis

Claims

Claims 1. A medical AR system (1) for intraoperative planning during a surgical, in particular neurosurgical, intervention on a patient (7), comprising: a movable visualization unit (2), the position and orientation of which can be adjusted, and which creates and digitally provides a current image (18) of the patient (7), in particular an intracorporeal image, a visual display device (10) for displaying visual information for the surgical intervention, and a navigation system (6) with a data provision unit which digitally provides stored preoperative 3D images of the patient (7), wherein the navigation system (6) is adapted to detect a position and orientation of the visualization unit (2) in a global coordinate system relative to the registered patient (7) and preferably a depth of focus of the visualization unit (2);characterized in that the medical AR system (1) further comprises a control unit (12) which is adapted to: generate a sectional plane in relation to the position and orientation of the visualization unit (2), in particular at a predetermined distance along an optical axis to the visualization unit (2) or at a currently set depth of focus of the visualization unit (2), generate a sectional view (16) of the preoperatively generated 3D images of the patient (7) on the basis of the sectional plane, generate an AR overlay representation (14) with the current image (18) and the generated sectional view (16) of the 3D image data, and output this AR overlay representation (14) through / via the visual display device (10); 2. Medical AR system (1) according to claim 1, characterized in that the control unit (12) is adapted to display the AR overlay representation (14) and an additional point-shaped or crosshair shaped indicator (20) through / via the visual display device (10), with which a target point or a target axis in the currently displayed AR overlay display (14) can be selected and set by a user (8).

3. Medical AR system (1) according to claim 2, characterized in that the control unit (12) is adapted to carry out the target point setting in the global coordinate system, wherein the target point or the target axis in the currently displayed sectional view (16) is selectable via the position and / or the orientation of the visualization system (2) relative to the patient (7).

4. Medical AR system (1) according to claim 2 or 3, characterized in that the target point setting is stored in a storage unit and the control unit (12) is adapted to use this target point setting as an entry point or as a waypoint or as a target point for an operative plan.

5. Medical AR system (1) according to one of claims 2 to 4, characterized in that the medical AR system (1) has a foot pedal and the control unit (12) is adapted to set and store the indicator (20) upon input via the foot pedal, and / or has a button or push button and the control unit (12) is adapted to set and store the indicator (20) upon input via the button or push button, and / or has a microphone and the control unit (12) is adapted to set and store the indicator (20) upon detection of an acoustic signal such as a voice input, in particular a predefined acoustic command, and / or has a touch display and the control unit (12) is adapted to set and store the indicator (20) upon input.

6. Medical AR system (1) according to one of the preceding claims, characterized in that the medical AR system (1) is an AR headset and / or a 3D monitor and / or a VR headset as the display device (10) 7. Medical AR system (1) according to claim 1, characterized in that the AR system (1) has a head-mounted display in the form of an AR headset worn on the head by a user (8) or a VR headset or an MR headset as the visual display device (10). 8 Medical AR system (1) according to one of the preceding claims, characterized in that the navigation system (6) has an infrared-based tracking system or an electromagnetic tracking system or an optical machine vision tracking system or a robot kinematics-based tracking system.

9. Medical AR system (1) according to one of the preceding claims, characterized in that the visualization unit (2) is connected to a robot arm (3) of a robot (4) and the visualization unit (2) has a joystick with which the user (8) can adjust the position and orientation of the visualization unit (2) in space, or the medical AR system (1) has a separate remote control which is adapted to adjust the position and orientation of the visualization unit (2) in space by means of a remote control of the robot arm (3).

10. Medical AR system (1) according to one of the preceding claims, characterized in that the visualization unit (2) is connected to a robot arm (3) of a robot (4) and is defined or definable by means of a user input via a movement axis in space, in particular the optical axis or a vertical axis or a horizontal axis in the patient direction, which the control unit (12) uses to carry out the translational movement of the visualization unit (2) exactly along this movement axis, or the control unit (12) is adapted to control the movement of the visualization unit (2) in an orbit mode so that it orbits the current focus point at a constant distance.

11. Medical AR system (1) according to one of the preceding claims, characterized in that the control unit (12) is adapted to generate the AR overlay representation (14) in such a way that within a circle around a center point through which the optical axis of the visualization unit (2) runs, and whose circle radius is preferably adjustable, in particular by a user input, the generated sectional view (16) of the 3D recording data is displayed and outside the circle the current recording (18) is displayed.

12. Medical AR system (1) according to one of claims 1 to 10, characterized in that the control unit (12) is adapted to generate the AR overlay display (14) in such a way that in a ring, the outer ring radius and / or the inner ring radius of which is adjustable and the center point of which is defined by the optical axis of the visualization unit (2), the generated sectional view (16) of the 3D recording data is displayed within the outer ring radius and the inner ring radius, and the current recording (18) is displayed outside the outer ring radius and within the inner ring radius.

13. Medical AR system (1) according to one of claims 1 to 10, characterized in that the control unit (12) is adapted to generate the AR overlay display (14) such that the generated sectional view (16) of the 3D recording data and the current recording (18) are displayed alternately at regular time intervals.

14. Intraoperative planning method for the surgical treatment of a patient (7), in particular with a medical AR system (1) according to one of claims 1 to 13, characterized by the steps: - Providing (S1) stored preoperative 3D images of a patient (7) by a data provision unit of a navigation system (6); - recording and providing (S2) a current image (18) of the patient (7) by a movable visualization unit (2); - detecting (S3) a position and an orientation of the visualization unit (2) in a global coordinate system relative to the patient (7) and preferably the focus depth of the visualization unit (2) by the navigation system (6); - generating (S4), by a control unit (12), a sectional view (16) of the preoperative 3D images of the patient (7) based on the detected position and the orientation of the visualization unit (2) and preferably a predetermined distance along an optical axis to the visualization unit (2) or a currently set focus depth; - generating (S5) an AR overlay representation (14) from the current recording (18) and the sectional view (16) by the control unit (12); - Outputting (S6) the AR overlay representation (14) by the visual representation device (10).

15. Intraoperative planning method according to claim 14, characterized in that in the output step (S6) an additional point-shaped or crosshair-shaped indicator (20) is output by / via the visual display device (10).

16. Intraoperative planning method according to claim 14 or 15, characterized by an additional step of selecting and setting (S7) a target point or a target axis in the currently displayed AR overlay representation (14) by a user (8).

17. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method steps according to any one of claims 14 to 16.

18. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method steps according to any one of claims 14 to 16.

Images