Navigation system and navigation method with annotation function

Abstract

The present disclosure relates to a surgical navigation system (1) having an annotation function for inputting a position-dependent medical annotation and for outputting a medical annotation superimposed on a visual image during a surgical operation at a patient (P), the surgical navigation system having: a presentation device (2) for visually outputting the medical annotation; at least one robot (4) as a positioning system, having a robot arm (8) movably connected to a robot base (6) and a robot head (10) arranged on the robot arm (8) at the end, and having a stereoscopic recording system (12) arranged on or in the robot head (10), which stereoscopic recording system has at least two recording units (14) spaced apart from each other, in order to perform 3D shooting (3DA) of the structure or surface with respect to the shooting system (12); a tracking system (16) adapted to detect and track the orientation of the photographing system (12) and computer-readable provide the orientation of the photographing system (12); an annotation system (18) having a storage unit in which a centralized digital annotation database having a global coordinate system for annotating objects is stored; and a control unit (20) adapted to process the provided 3D photography (3DA) and photography parameters and the provided orientation and to determine the orientation of the 3D photography (3DA) in a global coordinate system based on the provided photography parameters and the orientation of the photography system (12), and outputting at least one annotation object of the annotation database in the 3D recording (3DA) by means of the presentation device (2) in the superimposed presentation (U) at least in a position-correct manner. Furthermore, the present disclosure relates to a navigation method and a computer readable storage medium according to the parallel claims.

Description

Technical Field

[0001] The present disclosure relates to a surgical navigation system having an annotation function / marking function for inputting position-related, especially visual medical annotations / markings or photographing and / or for outputting medical annotations / markings when performing surgery on a patient, the surgical navigation system having: a (visual) presentation device for visually outputting medical annotations, especially a surgical monitor or display or AR glasses or VR glasses; and at least one, preferably a plurality of robots that can be actuated independently of each other as a positioning system, (respectively) having a robot arm that is movably connected to a robot base, especially a multi-link robot arm, and a robot head arranged at the end on the robot arm. A module to be positioned, especially an end effector, such as an instrument, can be connected to the robot head. In addition, the present disclosure relates to a navigation method according to the preamble of the parallel claims, as well as a computer-readable storage medium and a computer program. Background Art

[0002] In the field of surgical navigation, there are generally two different groups / types / models / classes of surgical navigation systems.

[0003] The first group of surgical navigation systems uses recordings / images that were produced before surgery (i.e. before the actual surgery on the patient), such as computed tomography recordings (CT recordings), magnetic resonance tomography recordings (MRT recordings), X-ray recordings, ultrasound recordings, so-called image-based navigation. The preoperative recordings must be linked / compared with the position or orientation of the patient during surgery in order to enable corresponding navigation of instruments or implants. In this case, the patient is registered with respect to the preoperative recordings in order to link the patient's virtual anatomy of the preoperative (3D) recording data with the actual anatomy during surgery (in real time) and to display the instruments with respect to the preoperative recording data.

[0004] The second group of surgical navigation systems works without recording or images (image-free navigation). These navigation systems create a model of the patient during surgery, i.e., during surgery, usually by scanning anatomical landmarks / orientation points with a navigation probe / pointer / navigation pointer or by kinematic analysis using a correspondingly simplified axis model.

[0005] The first group of navigation systems, namely image-based navigation systems, have the disadvantage of complex and time-consuming registrations which often do not provide the required accuracy, although the recordings are necessary for the operation.

[0006] In contrast, image-free navigation systems do not require pre-operative imaging / photographing and intra-operative registration, but have the disadvantage of using only a very basic model of the patient's anatomy as a reference, for example an axis model of the leg in the case of knee surgery.

[0007] Current efforts in plastic surgery navigation are to combine the advantages of both technologies, i.e. preoperative imaging and intraoperative registration are no longer necessary, but the patient or the patient's surgical area must be photographed as accurately as possible as a reference in order to orient the instruments or implants accordingly and ultimately achieve the best result for the patient.

[0008] Image-based navigation methods are currently the standard in modern computer-assisted surgery. The central components of image-guided / image-based navigation are the computed tomography / CT visualization of the target position, the target trajectory, and the display of the CT risk objects / structures with reference to the medical imaging data detected before the operation. This (guidance) information is currently derived only from the medical image data detected before the operation, i.e., before the operation in terms of time, and is used as the main guidance form during certain steps of the operation.

[0009] Typically, optical tracking systems (as measuring systems), in particular active or passive optical infrared systems or electromagnetic tracking systems, are used in image-based navigation in order to orient the preoperative recording or its information about applied markers, in particular reflecting reference points, to the surgical area / operating position with all necessary objects. Unfortunately, the visualization of this preoperative information in relation to the (time-real-time) live view of the microscope or endoscope of the surgical area / operating area of ​​the patient during the current operation is highly inaccurate. In particular, this inaccuracy can be based on inaccurate registration. Particularly high inaccuracies can occur if the patient's anatomical structures move between the preoperative (image) recording and the situation during the operation, as is the case, for example, with so-called brain shifts when the patient's skull is opened.

[0010] The inaccuracy or precision of the navigation is influenced in particular by the following aspects: tracking errors of each individual tracked rigid body (reference set); errors in the patient registration; errors in the recording structure; changes in the anatomical structure with respect to the pre-operative recording data, which are caused, for example, by tissue shifts, such as so-called brain shifts; errors in the hand-eye camera calibration; and / or errors in the optical (camera) calibration.

[0011] Furthermore, optical tracking technology also requires a (continuous) line of sight between the tracking camera on the one hand and the instrument to be tracked on the other hand, which in turn strongly limits the surgeon's practically possible working area. An unfavorable position of the surgeon (e.g. other instruments or a robot arm) can obstruct the view or interrupt the line of sight, which leads to undesired tracking / tracking interruptions during the operation and endangers the patient's safety. This also prolongs the duration of the operation. Summary of the invention

[0012] Therefore, the aim and purpose of the present disclosure is to avoid or at least reduce the disadvantages of the prior art and in particular to provide a surgical navigation system, a navigation method, a computer-readable storage medium and a computer program with an annotation function, which provide improved navigation with improved flexible annotations, in particular a higher accuracy of navigation. Another sub-purpose can be regarded as collecting information from different subsystems and storing it centrally and accessing it centrally for navigation accordingly, so that all data can be accessed in an orderly manner. In particular, the sub-purpose is to provide navigation with high accuracy during surgery even without preoperative information (photographing).

[0013] The objects of the present disclosure are achieved according to the invention by the features of claim 1 with respect to a surgical navigation system of this type, by the features of claim 14 with respect to a navigation method of this type, and by the features of claim 15 with respect to a computer-readable storage medium.

[0014] The core idea of ​​the present disclosure is therefore to detect feature points or objects in a recording, in particular in a 3D recording, in real time, intraoperatively (i.e. during surgery on the patient) and to register accordingly (or position identification, in particular orientation identification, of points, objects or regions of tissue structures) and to store them in a (single) centralized global coordinate system with at least the corresponding coordinates (X, Y and Z coordinates). The global coordinate system can be oriented relative to the patient, for example as the patient's coordinate system, or alternatively at the navigation system, in particular with the zero point of the coordinate system (KOS) of the tracking system, preferably the zero point in the tracking camera. It is only decisive that all information is converted into a single centralized coordinate system, so that subsequent conversions and error sources are omitted.

[0015] When determining the (position) coordinates of the annotation object, different position data are taken into account, which are introduced by a combination of navigation (global position detection), visualization by a recording system (local position detection) and a robot or robotics (sensor-based position detection by a robot arm). In other words, according to the present disclosure, an improvement in navigation can be achieved by a combination of visualization, robotics and navigation.

[0016] In other words, a surgical navigation system is provided, having an annotation function / marking function for inputting position-related, especially visual information or also neural signal information or histological information, medical annotations or photographing and / or for outputting medical annotations at a certain position of an anatomical structure during surgery on a patient, the surgical navigation system having: a presentation device, especially a surgical monitor, for visually outputting the medical annotations; a robot as a positioning system, having a robot arm, especially a multi-segment robot arm, which is movably connected to a robot base, and a robot head arranged at the end of the robot arm, wherein the navigation system also includes: a stereo photography system, especially a stereo camera, particularly preferably a stereo microscope, arranged at or in the robot head, having at least two photography units spaced apart from each other, so as to perform (temporally real-time) 3D photography of structures or surfaces (of the target area) relative to the photography system and to display the 3D images of the structures or surfaces (of the target area) in real time. a tracking system adapted to detect and track the position of the recording system, in particular indirectly via the position (i.e., location and orientation) of the robot head and to provide the position of the recording system in a computer-readable manner; an annotation system / marking system having a storage unit in which a digital annotation database / marking database having a global coordinate system is stored / stored; and a control unit adapted to process the provided 3D recording together with the recording parameters and the provided position, and to determine the position of the 3D recording in the global coordinate system based on the provided recording parameters and the position of the recording system, and to output at least one annotation object of the annotation database (which is also stored in the global coordinate system and is therefore associated with the position of the 3D recording) in a superimposed presentation via a presentation device in a positionally correct, in particular orientationally correct manner, and to improve navigation. Therefore, in particular during 3D photography (in real-time surgery), at every point in time and at any position of the photography system, the stored annotation objects or annotations can be presented superimposed on the 3D photography in the correct position, in particular in the correct orientation (as long as the position, in particular the orientation, of the annotation is within the visible area of ​​the photography system, i.e. in the 3D photography).

[0017] In particular, the control unit can be adapted to determine the orientation of the 3D recording based on the provided recording parameters and the orientation of the recording system in the case of an annotation command / marking command (entered by the user) and to store at least one partial cutout or target area of ​​the 3D recording as an annotation object / marking object together with the recording parameters in a global coordinate system in a determined position, in particular in an orientation, in an annotation database of the annotation system in order to output the annotation object in a correctly positioned, in particular correctly positioned, in a superimposed presentation via the presentation device at a later point in time and to improve navigation. With the navigation system, annotations on position data, in particular orientation data, can be centrally stored with particularly high accuracy.

[0018] In other words, a novel surgical navigation system is provided, which has one or more stereo / filming systems / optical camera systems mounted / fastened or arranged on a positioning system (especially a robot arm, an optical tracking system, an electromagnetic system). The (primary) filming system is used during surgery (i.e. during surgery) to display or visually output (in real time) (3D) recordings on a presentation device, such as a live 3D video view of the operating position / surgical position. Image-internal or spatial annotations / annotations can be manually or automatically set on the live video image of the operating area / surgical area by an additional tracking system (measurement system or tracking system). These image-internal or spatial annotations are detected and stored in the same (single) global coordinate system and can therefore be visualized and re-established by the user when necessary for surgical guidance. This can be achieved with high reproducibility and higher overall accuracy, even if the positioning system (robot arm, optical tracking system, electromagnetic system) moves during this period and camera parameters, such as focus or focal plane, change. The term “focus” in this case defines a point in space which is located in the focal plane of the optical system of the recording system or of the recording unit and in particular on the optical axis.

[0019] In particular, for example, the robot carries out a position determination of the stereo recording system in the robot's coordinate system based on the robot kinematics, and the recording system carries out a position determination of the 3D recording in the recording system's coordinate system. The two position determinations are linked to one another in that, for example, starting from the determined position of the recording system in the robot's coordinate system, the determined position of the 3D recording is transformed into the robot's coordinate system in order to finally determine the position of the 3D recording relative to the robot (or its robot coordinate system) similar to a serial transformation. This position of the 3D recording is then determined in the global coordinate system, for example, by a fixed or determinable transformation between the robot coordinate system and a global coordinate system (which in one embodiment can be a (virtual) coordinate system of the patient / patient coordinate system). In other words, data for annotation can be obtained from different modalities, converted or transformed into a single global coordinate system and stored there in the global coordinate system with corresponding positions (with coordinates X, Y, Z in space), in particular corresponding positions and orientations (i.e., positions). In this way, different annotations are no longer provided in different databases with different (local) coordinate systems, but the annotations are integrated into a common database with a common global coordinate system. This results in a certain degree of standardization or normalization and a uniform system is provided by centrally stored annotations (e.g. a database with respectively standardized annotations in the form of a data array with: annotation ID, annotation position, annotation orientation, annotation data (such as images, vector graphics, texts, etc.) and optional annotation transparency (0% to 100%, at 100% the annotation is switched off / invisible). These annotations can then be selected and presented, in particular on a computer basis, so that they appear in a superimposed presentation with the 3D recording of the recording system in a positionally correct (i.e. in the correct position relative to the 3D recording and thus relative to the patient), in particular in an orientationally correct (i.e. in the correct position in the correct orientation). In particular, when tracking and registering the patient, it is advantageous to use a correspondingly associated patient coordinate system as the global coordinate system. In this way, different annotations of a central database with a uniform global coordinate system can be displayed in a 3D recording, for example as an opaque or partially transparent superimposition.

[0020] Therefore, a surgical navigation system is described in particular, which has at least one stereo optical recording system, which is mounted or arranged on a positioning system (especially a robot arm), and which produces a 3D recording during surgery in order to display a live video view of the surgical location (live 3D recording). Further cameras can additionally be used to detect further spatial and visual information of the surgical area. The recording system mounted on the positioning system (especially a robot arm, an optical tracking system, electromagnetic tracking) can be moved by a user, such as a surgeon, to a specific spatial position and oriented, in particular with respect to an orientation, wherein the recording system provides the user with a live view of the surgical area.

[0021] The surgeon can then either use manual methods to identify and mark different anatomical or physiological regions, for example by setting points as annotations with a mouse pointer in the output 3D recording, or can use additional measuring systems and methods to automatically annotate, in particular to mark the corresponding regions at the moment. In this case, at least one 3D recording, in particular a live video image (annotation reference recording), as well as its spatial position (coordinates) are converted into a global coordinate system by a positioning system (in particular by the current robot arm configuration) together with the current camera settings as an annotation object and stored.

[0022] Unlike existing waypoint functions in robot-assisted microscopy or annotations in microscope systems, the present navigation system stores more and more precise data and features for accurate and convenient visualization of the annotated objects relative to the patient's anatomy. Furthermore, the navigation system provides the possibility to move the entire navigation system or change camera settings during surgery and still recreate and visualize the annotated objects. Changes in the anatomy can be reliably identified by the disclosed navigation system because the data is detected during surgery and not before surgery.

[0023] The term “tracking system” describes, for example, a technical system which enables spatial localization and which can detect a position and / or an orientation.

[0024] The term "position" refers to a geometric position in three-dimensional space, which is described in particular with the aid of coordinates of a Cartesian coordinate system. In particular, the position can be described by three coordinates X, Y and Z.

[0025] The term "orientation" again describes the orientation in space (approximately at this position). It can also be said that the orientation is described by the orientation in a direction or rotation in three-dimensional space. In particular, the orientation can be described with the aid of three angles.

[0026] The term “position” includes both the position and the orientation. In particular, the position can be specified using six coordinates, three position coordinates X, Y and Z and three angular coordinates for the orientation.

[0027] The term "positionally correctly" is defined as reflecting the annotation or 3D shot at its actual, correct position in the overlay presentation, similar to a CAD system.

[0028] The term "positionally correct" defines a combination of positional correctness and orientational correctness, wherein in the superimposed presentation, recordings are indicated in the correct position and in the correct orientation. In the case of parallel, coherent orientation of the recording directions, the superimposed recordings are shown with the same aspect ratio as in the actual recordings. If, for example, the recording directions are now turned, the extent or size of the recordings in the associated directions is compressed by this angle until, in the case of an angle of 90° between the two recording directions, only a one-dimensional line is produced in the superimposed presentation.

[0029] Here, the term "target region" describes a part of the surgical area that is of interest and should be targeted by the recording system. In particular, the target region can be a single target point in space (X, Y, Z coordinates) and can also be specified as such coordinates. However, the target region can also be a section of a two-dimensional plane, for example a small section of a tissue surface. The target region can also be described as a small cubic or spherical volume.

[0030] Unlike image-based navigation, the proposed navigation system and navigation method only have the following negative influences on the accuracy: the hand-eye calibration between the camera and the robot base coordinate system; and the calibrated camera system, which must be taken care of and can be maintained accordingly accurately.

[0031] Advantageous embodiments are claimed in the dependent claims and are explained in particular below.

[0032] According to one embodiment, the robot arm configuration of the robot and the camera settings of the recording system are stored as recording parameters in the annotation database so that the robot can be moved / displaced into exactly this position and orientation with the corresponding camera settings at a later point in time during the operation and the 3D recordings stored previously are output in real time as a coherent reference in an overlay presentation, for example as a side-by-side presentation or as a transparent overlay presentation, in order to visualize the changes over time. In particular, for so-called intra-image annotations, i.e. annotations within the image, the camera of the recording unit must be calibrated and the method requires, for repeated visualization, that the recording system be set to the same zoom and the same focus as during the annotation process (coherent view).

[0033] In particular, zoom (factor) and focus can be detected and stored as recording parameters as camera settings of a stereo camera, so that the position of at least a partial area of ​​the 3D recording can be determined precisely by the position of the recording system tracked by the tracking system and by the zoom and focus and / or can also be recreated for visualization.

[0034] In addition, preferably, the control unit can be adapted so that when a command is output to an annotation object stored in the annotation database, the shooting system moves / moves to a position stored relative to the annotation object and outputs the real-time 3D shooting and the stored annotation object, especially a partial cutout of the stored 3D shooting, in a superimposed presentation via a presentation device.

[0035] Annotations, in particular intra-image annotations, may also be assisted by computer vision algorithms, in particular by tracking corresponding areas during zoom level and focus / focal length changes.

[0036] According to a further preferred embodiment, preoperative 3D recording data, in particular CT recording data and / or MRT recording data and / or segmented X-ray images can also be stored in the storage unit, the surgical navigation system is adapted to register the patient with respect to the preoperative 3D recording data, and the control unit is adapted to display the preoperative 3D recording data in the superimposed presentation of the 3D recording for further assistance. In other words, the preoperative recording data / imaging data (3D-CT data / 3D-MRT data) of the medical operation can therefore be additionally accessed as auxiliary information, which, however, is not displayed as primary guidance information, but only as secondary guidance information. The primary guidance information forms the recording or the associated annotations during the operation.

[0037] According to a further embodiment, the control unit can be adapted to display a warning message in the superimposed presentation if the partial cutout of the 3D recording stored in the annotation database deviates from the 3D recording in real time, in particular by means of an image analysis method, by exceeding a predefined tolerance.

[0038] In particular, the control unit can be adapted to determine, when a deviation of the 3D shot stored in the annotation database relative to the 3D shot in real time exceeds a preset tolerance, in particular by means of an image analysis method, that there is a movement of the entire patient when the stored 3D shot or at least a partial cut-out of the 3D shot moves as a whole relative to the 3D shot in real time, and the control unit is further preferably adapted to realign the navigation system relative to the patient, in particular to perform a registration in order to precisely correct the determined movement.

[0039] Furthermore, the control unit can be adapted in particular to determine, when a deviation of the 3D shot stored in the annotation database relative to the 3D shot in real time exceeds a preset tolerance, in particular by means of an image analysis method, that there is only a movement of the patient's tissue when the stored 3D shot or at least a partial cut-out of the 3D shot is distorted, for example compressed, relative to the 3D shot in real time, and additionally preferably an alignment relative to the relevant tissue is performed.

[0040] In particular, the control unit can be adapted to detect the instrument in the 3D recording in real time and set / appear in the front plane in the superimposed presentation, while at least one annotation object is presented in the intermediate plane between the live recording and the instrument, that is, it is presented in the rear plane observed relative to the instrument, so that in the superimposed presentation, the annotation object does not interfere with the instrument or the instrument is not covered by the annotation object. In addition, the control unit can be adapted in particular to detect the instrument in the 3D recording in real time and set / appear in the rear plane in the superimposed presentation, while at least one annotation object is presented in the front plane, that is, in the front plane observed relative to the instrument. The annotation can therefore be in front of or behind the instrument. When the annotation is in front of the instrument, the annotation covers the instrument opaquely or translucently (for example 70% coverage), and when the annotation object is behind the instrument, the instrument covers the annotation opaquely or translucently (for example 70% coverage).

[0041] According to a further embodiment, the recording system can provide a real-time / live 3D recording during the operation, which is output via a presentation device on the one hand and is stored in an annotation database on the other hand in the event of an annotation command.

[0042] In particular, a transparency parameter between 0 and 1 (i.e. a continuous value between 0% and 100%) can be stored in the annotation database for each annotation object and thus stored between hiding and visibility, and the control unit can be adapted to display all annotation objects at least positionally correctly, in particular orientationally correctly, according to their transparency in the superimposed presentation together with the current 3D recording in real time.

[0043] According to one embodiment, the control unit may be adapted to change the threshold value for displaying the contour / outline, in particular based on a manual input, in order to mark the area for the user as required and to move the boundaries of the area according to the set boundary value.

[0044] In particular, the patient's head can be fixed relative to the robot base or the robot system of the robot. This is common in particular in image-based navigation systems. Alternatively, the patient can be tracked using the same positioning system, in particular a tracking system.

[0045] According to a further embodiment, the navigation system, in particular the recording system, can be adapted to perform a camera calibration in order to provide information about how the collected annotation objects move with respect to a change of zoom / (zoom level) or focus / focal length, in particular of the spatial pyramid, to the control unit, so that the control unit is adapted to change the camera settings during the movement. In other words, the navigation system requires a positioning system (in particular a robot arm, an optical tracking system, an electromagnetic tracking) with high repeatability of spatial positioning and a (mainly) camera system with high repeatability of zoom and focus / focal length, so it is advantageous when the camera system is also calibrated.

[0046] Preferably, as associated data of the annotation object, the incremental number / marker ID of the pointing device (navigation pointer / pointer) being tracked and / or the intensity information of the fluorescence shooting and / or the convex hull of the trajectory and / or the spherical distance field can be stored in the annotation database. In other words, the following data can be stored as associated data of the annotation object: for example, the incremental number / marker ID from the pointing device / computer vision tracking device; for example, the intensity information of the fluorescence shooting; for example, the convex hull of the trajectory; for example, the spherical distance field from neuron monitoring. In particular, the annotation can be:

[0047] Physiological information, especially neural signaling and / or histology;

[0048] geometric objects, in particular points, circles, lines, trajectories and / or contours;

[0049] And / or secondary image information, in particular fluorescence. By means of the disclosed navigation system, different display forms can be presented in a (temporally real-time) 3D recording where they are actually located, ie where they are correctly positioned, in particular where they are correctly oriented.

[0050] Preferably, the following methods can be used as manual marking methods: the focus of the recording system, in particular the focus of a microscope (1D object with XYZ coordinates in the global coordinate system, which point can be marked accordingly in space); a cursor / mouse pointer in the live image of the presentation device (with XY coordinates and an offset to the focus in the image plane as Z coordinate); a navigation point or instrument, such as a navigation pointer / pointer; computer vision methods for identifying edges or tissue patterns; detecting the origin of the blood and the direction of blood flow in time (identification of ML) in order to detect the blood source.

[0051] In particular, automatic labeling methods can also be used as annotation methods, in particular: neuron monitoring and / or fluorescence imaging (eg DUV400, DFS560, DIR800) and / or other optical measurement methods, such as OCT or the like.

[0052] In particular, the navigation system can store (temporally limited) 3D video recordings, in particular 3D video recordings with a length of substantially five seconds, as annotation objects, and thus store moving images / videos as annotation objects. In other words, in particular 4D annotation objects (i.e. 3D with a time-varying component) can therefore be stored and correspondingly re-imaged at another point in time. In particular, the video recording can come from the ICG.

[0053] Preferably, the surgical navigation system can also have an overview camera, which is adapted in such a way that the surroundings of the operating area are detected and provided in a computer-readable manner. In other words, the additional camera system can assist in the automatic generation of annotation objects.

[0054] Preferably, the navigation system can be adapted to perform in-image / in-waypoint annotation. In so-called in-image / in-waypoint annotation, the user moves the recording system to the desired position and sets the camera to the desired parameters regarding zoom and focus (zoom level and focus / focal length). The annotation objects within the 2D recording can now be identified manually or automatically. This category requires a robot system with high reproducibility and a primary recording system / camera system with high repeatability of zoom and focus. In particular, a prerequisite for the focus is that the robot head, in particular the robot end effector, is also moved in an appropriate orientation relative to the original orientation (waypoint with stored camera parameters).

[0055] In particular, a specific camera calibration can provide information about how the collected annotation objects move with respect to changes in zoom level or focus / focal length (spatial pyramid). This enables the user to change camera settings during movement. Traditional annotation systems do not allow such flexibility during this movement.

[0056] Preferably, the navigation system can be adapted to perform spatial annotation. During spatial annotation, the user moves the shooting system (stereoscopic shooting system / camera system) to the desired position and sets the camera to the desired parameters related to the zoom level and focus / focal length. Now the annotation object in the 2D shooting / 2D image can be identified manually or automatically. Preferably, through a specific camera calibration (stereoscopic), the spatial 3D position of the marked object can be derived in the coordinate system of the positioning system. This category of spatial annotation requires a robot system with high reproducibility and a shooting system with high repeatability of zoom and focus / focal length. The prerequisite for the focus is that the robot head, especially the robot end effector, also moves in a posture or orientation close to the original posture (i.e., similar position and orientation within tolerance; path points with stored camera parameters). In addition, special camera calibration and hand-eye calibration for 3D reconstruction are particularly required to transfer the 3D reconstructed data to the coordinate system of the positioning system. This enables the user to move the camera and freely change the camera settings. The annotation object can be visualized under the consideration of the adapted posture and camera settings. The method can also preferably be assisted by computer vision algorithms by tracking the corresponding areas during zoom and focus / focal length changes and during robot movements.

[0057] In particular, the following user interactions can be implemented by the navigation system or the navigation system can be adapted to perform the following functions: turning annotation objects on and off (such as by corresponding entries in an annotation database), supplemented by a live view; moving thresholds to display the contours / outline of a point cloud (e.g. a tumor boundary) of a regular sample with corresponding intensities (e.g. a fluorescence image) (user-guided segmentation of structures by simple marking (based on so-called max-cut min-flow)).

[0058] In particular, free movement of the recording system (camera) can be achieved by extending the spatial annotation object, which requires calibration in particular. The annotation object is extended relative to a fixed operating position.

[0059] Preferably, at least one direction for the stored annotations can also be displayed or visualized in the superimposed representation, in particular during an (automatic or manual) movement of the robot arm.

[0060] Preferably, the annotation object may be measured (in mm) to be recorded and stored in the annotation database as data associated with the annotation object.

[0061] Further preferably, the focus and zoom for the spatial annotation object can be stored and / or set accordingly, in particular depending on the contours of the fluorescence image.

[0062] In particular, as annotation objects, (1D or 2D) heat maps / temperature maps can be supplemented to better present the target or risk structure.

[0063] An important part of the navigation system is the visualization of the detected and processed annotations. The visualization of the annotation objects can preferably be adapted so that the superposition of the objects does not interfere with the instruments used (e.g. masking, similar to the depth blurring of the background). This improves the surgeon's visualization and reinforces the impression that the annotation objects are associated with the patient's surgical area and are not just simply overlapped. For example, in one embodiment, the control unit can be adapted to present the annotation objects / annotation objects vaguely so that the surgical instruments appear to be above (masked). The annotations can also preferably be revealed through different focus images (1-2 shooting planes upward or downward) with a certain degree of transparency.

[0064] The navigation system may in particular be adapted to mark the detected spatial annotation as invalid if the underlying anatomical structure has been altered by the surgery. This may be achieved by comparing the annotated reference image with the current live view of the camera.

[0065] Preferably, the navigation system, in particular the control unit, can also be adapted to suppress the visualization of the spatial annotation object if the visualization error increases after a movement of the microscope head.

[0066] Preferably, the navigation system can also be adapted to store the biopsy or biopsy information as annotations. In particular, if the annotation points also provide diagnostic information, in particular from the biopsy, then surgical documentation can be easily automatically generated.

[0067] In particular, spatial annotation objects can be used to register pre-operative CT / MRT data in order to obtain additional support during surgery.

[0068] Regarding a surgical navigation method, the object of the present disclosure is achieved by providing a navigation method, which has an annotation function for inputting position-related visual medical annotations and photographing and outputting the medical annotations during a surgical operation on a patient, and the navigation method is particularly used for a surgical navigation system according to the present disclosure, the method comprising the following steps: photographing (in real time) 3D photography by a stereoscopic photography system, particularly a stereo camera, particularly preferably a stereo microscope, arranged on or in a robot head, the stereoscopic photography system having at least two photography units spaced apart from each other, and providing the 3D photography together with photography parameters to a control unit; detecting and tracking the orientation of the photography system by a tracking system and providing the orientation to the control unit; determining the orientation of the 3D photography based on the orientation of the photography system and the photography parameters when an annotation command is detected; storing an annotation object having at least one local cutout of the 3D photography and the photography parameters in an annotation database of an annotation system having a global coordinate system; and outputting a superimposed presentation having content with a correct position, particularly a correct orientation, of the at least one annotation object by a presentation device.

[0069] With regard to the computer-readable storage medium and the computer program, the objects are met in that the computer-readable storage medium and the computer program respectively comprise instructions which, when executed by a computer, cause the computer to perform the method steps of the navigation method disclosed herein.

[0070] Any disclosure associated with the surgical navigation system of the present disclosure is also applicable to the navigation method of the present disclosure, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The present disclosure will be explained in more detail below with the aid of the accompanying drawings according to preferred embodiments.

[0072] FIG. 1 shows a schematic diagram of a navigation system according to the prior art, in which various error sources are shown.

[0073] Figure 2 is a stereoscopic view of a surgical navigation system according to a preferred embodiment of the present disclosure, which can show annotations in a coherent view

[0074] Figure 3 shows a further stereoscopic view of a surgical navigation system according to a further preferred embodiment, which detects annotated objects with depth information,

[0075] Figure 4 Show Figure 3 a stereoscopic view of a navigation system in a second position, the navigation system detecting or outputting an adapted representation of the annotation in a second position,

[0076] Figure 5 Show Figure 3 and Figure 4 Another stereoscopic view of the navigation system in which the annotation moves out of the detection area,

[0077] Figure 6 Schematic diagram showing an exemplary 3D recording with circular and dot-shaped annotations,

[0078] Figure 7 A schematic diagram showing an exemplary 3D recording with contour annotations,

[0079] Figure 8 A schematic diagram showing an exemplary 3D acquisition with trajectory annotations,

[0080] Fig. 9 Schematic diagram showing an exemplary 3D acquisition with fluorescence annotation,

[0081] Fig.10 A further schematic diagram shows an exemplary 3D recording with fluorescence annotation,

[0082] Fig.11 shows a schematic diagram of an exemplary 3D recording with a tracked instrument and the associated distance field as annotation,

[0083] Fig.12 and Fig.13 schematically shows an exemplary 3D recording with motion vectors of angiofluorescence (ICG) as stored annotations,

[0084] Fig.14 and Fig.15 Schematic diagram showing an exemplary 3D photo with a biopsy as annotation,

[0085] Fig.16 Schematic diagram showing an exemplary 3D recording with different settings of the threshold value for detecting the boundary as an annotation,

[0086] Fig.17 shows a schematic diagram of an exemplary 3D recording with dimensions stored in mm as annotations,

[0087] Fig.18 Schematic diagram showing an exemplary 3D recording with an added heat map as annotation,

[0088] Fig.19 Schematic diagram showing an exemplary 3D recording with reproducible 4D annotation (video),

[0089] Fig. 20 A schematic diagram of an exemplary 3D shot is shown with a tracked instrument and annotations placed in the background such that the instrument appears in the foreground.

[0090] The accompanying drawings are schematic in nature and are only used to understand the present invention. The same elements are provided with the same reference numerals. The features of the various embodiments may be substituted for each other. DETAILED DESCRIPTION

[0091] FIG. 1 shows a surgical navigation system according to the prior art in a schematic perspective view. Many error sources that negatively affect the accuracy occur here. The instrument is tracked by a tracking camera via an installed rigid body (infrared marker), wherein for each individual rigid body, tracking errors occur with a certain error tolerance. The patient is also tracked by the rigid body, which is accompanied by tracking errors. In addition, errors may also occur in the recording structure, and changes in the anatomical structure with respect to the preoperative recording data may occur, for example, due to tissue displacement, such as so-called brain displacement. Other sources of faults may be in the hand-eye camera calibration or in the optical calibration. Such errors and error chains are considered to be avoided or at least minimized in order to achieve better and more precise navigation.

[0092] Figure 2 A surgical navigation system 1 (hereinafter referred to as navigation system) with an annotation function is shown in a stereoscopic view. The annotation function is used to combine position-dependent visual medical annotations and recordings during surgery and for guiding and performing surgical operations on a patient P (shown here).

[0093] In order to be able to visually represent the inputted annotations, the navigation system 1 has a representation device 2 in the form of a surgical monitor in order to visually output the medical annotations for a medical professional, in particular a surgeon.

[0094] The navigation system 1 also has a robot 4 as a positioning system. Specifically, the robot has a fixed robot base 6 and a multi-link robot arm 8 movably connected to the robot base 6, which can be actively controlled and moved. The robot head 10 located at the end is movably connected to the end section of the robot arm 8, and the robot head can rotate at least relative to the longitudinal axis of the last segment / link of the robot arm 8.

[0095] A stereo recording system 12, in particular a stereo camera, in this case a stereo microscope, is housed in the robot head 10 and has at least two recording units 14, in this case recording cameras, spaced apart from one another, which record a 3D recording 3DA of a structure or surface via two optical systems, each with a sensor connected downstream. The 3D recording 3DA is provided together with recording parameters, in this case in particular camera parameters, in a computer-readable manner.

[0096] In order to track the robot head 10, the navigation system 1 has a tracking system 16, which is designed as a mechanical sensor-type tracking system 16 in the present embodiment and detects the position of the robot head 10 relative to the robot base 6 or the coordinate system of the robot 4 through the robot arm configuration, and indirectly detects the position of the camera system 12 through the position of the robot head 10 and tracks the position in space. The detected position of the camera system 12 is provided in a computer-readable manner.

[0097] For data management of annotations, the navigation system 1 comprises a centralized internal annotation system 18 with a storage unit (not shown) in which a digital annotation database with a single global coordinate system is stored, wherein the global coordinate system is the coordinate system of the robot 4. New annotation objects can be created in this annotation database as list entries (similar to a SQL database) and data associated with the annotation objects can be stored.

[0098] A specially adapted control unit 20 (of which the annotation system 18 is a part / subunit) is configured to process the provided 3D recording 3DA together with the recording parameters of the recording system and the provided orientation. If the annotation object is stored in an annotation database, the control unit 20 can first determine the orientation of the 3D recording 3DA in space, more precisely relative to a centralized global coordinate system, based on the provided recording parameters and the orientation of the recording system. When the stored annotation object falls into the visible area of ​​the 3D recording, the stored annotation object or the data of the annotation object are then output in the superimposed presentation with the correct position, in particular with the correct orientation. For example, images, texts or other medical data for surgery can be stored as annotation objects, and the images can be displayed with the correct orientation or the texts can be displayed with the correct position in the 3D recording.

[0099] Furthermore, the control unit 20 is particularly adapted to process the provided 3D recording 3DA together with the recording parameters of the recording system and the provided orientation and, in the case of an annotation command, for example via a touch display of a surgical monitor as a presentation device or automatically based on an internal annotation program, first determine the orientation of the 3D recording in space, more precisely with respect to a centralized global coordinate system, based on the provided recording parameters and the orientation of the recording system. In other words, starting from the robot base 6, via the robot arm 8, via the robot head 10, via the recording system 12 and the optical parameters, the orientation of the 3D recording is finally determined. This orientation is transferred to the centralized global coordinate system.

[0100] For example, at least one partial cutout of the 3D recording of the region of interest, such as a tumor, is then stored in the annotation database of the annotation system together with the recording parameters in the global coordinate system in a specific position (here even a specific orientation) as an annotation object. In particular, if the navigation system 1 has not only one robot 4, but a plurality of robots 4 with robot arms, the important data for the operation can be stored centrally, wherein all annotations are recorded centrally in a (single) global coordinate system. This global coordinate system can be conceived as a three-dimensional space with a Cartesian coordinate system, which stores different types of annotations recorded during the operation at different XYZ positions.

[0101] At a later point in time, the surgeon can output the previously recorded annotation objects via the presentation device 2 as a superimposed presentation U in the live presentation of the real-time recording of the recording system 12 at least in the correct position (e.g. a point in space determined by a navigation pointer) or in the correct orientation (e.g. a 3D recording of a surface). This significantly improves surgical navigation.

[0102] exist Figure 2 A coherent view is shown in , which means that an annotation command is given at a first point in time in a first orientation of the filming system, and the annotation is stored in a central annotation database in the global coordinate system at the correct position with additional information about the orientation based on the corresponding filming data, i.e. based on the main parameters of the camera settings, such as focus and zoom and the robot configuration (the configuration of the links of the robot arm 8 with each other and with the robot base 6), and the robot is moved after the first point in time and the operation is further performed accordingly.

[0103] If the surgeon now wants to return to the first annotation, the surgeon selects the corresponding annotation by means of an output command, by manual input or by operating the touch display of the surgical monitor, and the control unit controls the robot arm 8 to the configuration set for this purpose and thereby controls the filming system 12 to the same position as at the first time point based on the stored data about the annotation object, adjusts the camera to be sharp according to the stored camera settings and outputs an overlay presentation of a 3D recording 3DA with real time on the one hand and a partial cutout of the stored 3D recording 3DA on the other hand, for example a side-by-side presentation or an overlay with the aid of transparency, so that the time changes can be visually presented to the surgeon.

[0104] Figure 3 and Figure 4 A further preferred embodiment of the surgical navigation system 1 is shown in a stereoscopic view. This embodiment differs only in that the control unit is adapted to perform spatial annotation, wherein the control unit 20 is adapted to also perform spatial annotation from a second position different from the first position (see Figure 4The first orientation shown in dashed lines, the second orientation shown in solid lines) is used to display the spatial representation of the previously recorded annotation object, for example, by means of a corresponding matrix transformation. In this case, the centralized global coordinate system with the stored annotation objects in combination with the image analysis and recording processing method helps the annotation to appear in the second orientation of the recording system 12 in the correct orientation (correct orientation).

[0105] exist Figure 5 Schematically shown in FIG. 1 is the situation when the robot 4 moves and the object to be annotated moves out of the visible area (eg in a coherent view).

[0106] exist Figure 6 An example of annotations stored in the annotation database is schematically shown in . Points in space of a global coordinate system having coordinates X, Y and Z are stored as annotation objects with corresponding identifications, which can be displayed in an overlay presentation U as required.

[0107] Figure 7 A further alternative to the stored annotation is shown, in which an area of ​​the tissue surface is (colored) as a pentagon. During the operation, this area can also be displayed to the surgeon.

[0108] Figure 8 A further annotation is shown in the form of a trajectory which is also visualized into the tissue in order to provide guidance to the surgeon.

[0109] Fig. 9 and Fig.10 A 3D recording of the tissue structure with stored fluorescence as annotations is shown.

[0110] exist Fig.11 , an instrument 22 , here a pointer, is shown which has, on the one hand, a ring marking on the instrument 22 itself and also a distance field at the distal tip of the instrument 22 as a further embodiment of the annotation.

[0111] Fig.12 and Fig.13 The 3D flow vectors are shown in the stored 3D recordings as annotations or annotation objects.

[0112] Fig.14 shows the area where the biopsy was performed, and Fig.15 Also shown is a schematic diagram with biopsy results of different areas.

[0113] Fig.16 (From left to right) shows the change of the limit / border of the area where the user steadily lowers the threshold. In other words, the user can preset the boundary value and visualize the border around the tissue area according to the boundary value. This is another alternative for annotation.

[0114] Fig.17 The dimensions in mm are shown as a note. Here, a sterile measuring instrument in the form of a ruler is positioned inside the tissue and the 3D recording is performed. Thus, at a later point in time, if the recording system is again in a similar position, the dimensions can be visualized if necessary without having to physically reconnect the measuring instrument.

[0115] Fig.18 A further alternative of annotation is shown in the form of a heat map.

[0116] Fig.19 The video of the stream is shown as an annotation so that the moving image can also be reflected.

[0117] Fig. 20 The instrument 22 is shown in the real-time recording, while a marking point (here represented by a circle) appears as an annotation in the background of the instrument. The control unit 20 is adapted to recognize the instrument 22 and to bring it forward into the plane so that the annotation does not cover the instrument 22 .

[0118] Reference numerals list

[0119] 1 Surgical Navigation System

[0120] 2. Presentation Device

[0121] 4 Robots

[0122] 6 Robot base

[0123] 8 Robotic Arm

[0124] 10 Robot Head

[0125] 12 Shooting System

[0126] 14 Shooting Unit

[0127] 16 Tracking System

[0128] 18 Annotation System

[0129] 20 Control unit

[0130] 22 Equipment

[0131] Patient P

[0132] 3DA 3D Shooting

[0133] U Overlay Presentation

Claims

1. A surgical navigation system (1) having an annotation function for inputting position-dependent, in particular visual, medical annotations or recordings and / or for outputting medical annotations during a surgical operation on a patient (P), the surgical navigation system comprising: a presentation device (2), in particular a surgical monitor, for visually outputting the medical annotation; At least one robot (4) as a positioning system, having a robot arm (8), in particular a multi-linked robot arm (8), which is movably connected to a robot base (6) and a robot head (10) arranged at the end of the robot arm (8), Features: a stereo recording system (12), in particular a stereo camera, particularly preferably a stereo microscope, arranged at or in the robot head (10), having at least two recording units (14) spaced apart from one another, in order to carry out a 3D recording (3DA) of structures or surfaces relative to the recording system (12) and to make the 3D recording available together with recording parameters in a computer-readable manner; a tracking system (16) adapted to detect and track the position of the camera system (12), in particular indirectly via the position of the robot head (10), and to provide the position of the camera system (12) in a computer-readable manner; An annotation system (18) having a storage unit in which a centralized digital annotation database having a global coordinate system for annotated objects is stored; as well as A control unit (20) is adapted to process a provided 3D recording (3DA) as well as recording parameters and a provided orientation and to determine the orientation of the 3D recording (3DA) in a global coordinate system based on the provided recording parameters and the orientation of the recording system (12), and to output at least one annotation object of the annotation database in the 3D recording (3DA) in a superimposed presentation (U) via the presentation device (2) in a positionally correct, in particular orientationally correct manner and to improve navigation.

2. The surgical navigation system (1) according to claim 1, characterized in that: The control unit is adapted to determine the orientation of the 3D recording (3DA) in the annotation command based on the provided recording parameters and the orientation of the recording system (12), and to store at least one partial cutout of the 3D recording (3DA) in the determined position, in particular orientation, in the global coordinate system together with the recording parameters in an annotation database of the annotation system as an annotation object, so that at a later point in time the annotation object can be output in a superimposed presentation (U) via the presentation device (2) at least in the correct position, in particular in the correct orientation, and navigation can be improved.

3. The surgical navigation system (1) according to claim 1 or 2, characterized in that: The robot arm configuration of the robot (4) and the camera settings of the filming system (12) are stored as filming parameters in the annotation database so that the robot (4) can be moved to exactly that position and orientation using the corresponding camera settings at a later point in time during the operation and output in the overlay presentation (U) a real-time 3D film as well as previously stored 3D films as a coherent reference in order to visualize changes over time.

4. The surgical navigation system (1) according to any one of the preceding claims, characterized in that Zoom (factor) and focus are detected and stored as camera settings of a stereo camera of the filming system (12) as filming parameters, so that the position of at least a partial area of ​​the 3D filming can be accurately determined by the orientation of the filming system (12) tracked by the tracking system and by the zoom and the focus.

5. The surgical navigation system (1) according to any one of the preceding claims, characterized in that The control unit (20) is adapted to, when outputting a command to an annotation object stored in the annotation database, cause the filming system (12) to move to a position stored relative to the annotation object and output, via the presentation device, a temporally real-time 3D filming (3DA) and the stored annotation object, in particular a partial cutout of the stored 3D filming (3DA), in the superimposed presentation (U).

6. The surgical navigation system (1) according to any one of the preceding claims, characterized in that Preoperative 3D imaging data, in particular CT imaging data and / or MRT imaging data and / or segmented X-ray images, are also stored in the storage unit; The surgical navigation system (1) is adapted to register the patient (P) relative to pre-operative 3D imaging data; and The control unit (20) is adapted to visualize pre-operative 3D recording data for further assistance in the superimposed presentation (U) of the 3D recording (3DA).

7. The surgical navigation system (1) according to claim 3, characterized in that: The control unit (20) is adapted to display a warning message in the superimposed presentation (U), in particular by means of an image analysis method, when the local cutout of the 3D recording (3DA) stored in the annotation database deviates from the 3D recording (3DA) in real time by more than a preset tolerance.

8. The surgical navigation system (1) according to any one of the preceding claims, characterized in that The control unit (20) is adapted to detect the instrument (22) in a temporally real-time 3D recording (3DA) and to place the instrument in a front plane in the superimposed presentation (U) and to present the at least one annotation object in a rear plane, so that in the superimposed presentation (U) the annotation object does not interfere with the instrument (22).

9. The surgical navigation system (1) according to any one of the preceding claims, characterized in that The recording system (12) provides a 3D recording (3DA) in real time during the operation, which is output via the presentation device on the one hand and is stored in the annotation database on the other hand in the event of an annotation command.

10. The surgical navigation system (1) according to any one of the preceding claims, characterized in that In the annotation database, a parameter for transparency between 0 and 1 and therefore between concealment and visibility is stored for each annotation object, and the control unit (20) is adapted to display all annotation objects in the superimposed presentation together with the real-time 3D recording (3DA) at least positionally correctly, in particular orientationally correctly, corresponding to the transparency of the annotation objects.

11. The surgical navigation system (1) according to any one of the preceding claims, characterized in that The control unit (20) is adapted to vary the threshold value to display the outline / contour as annotation to mark the area for the user as desired.

12. The surgical navigation system (1) according to any one of the preceding claims, characterized in that The filming system (12) is adapted to perform a camera calibration to provide the control unit (20) with information about how the collected annotation objects move, in particular the spatial pyramid, with respect to a change of zoom or focus, so that the control unit (20) can change the camera settings during the movement.

13. The surgical navigation system (1) according to any one of the preceding claims, characterized in that In particular, an incremental ID of the tracked pointing device and / or intensity information of the fluorescence recordings and / or a convex hull and / or a spherical distance field of the trajectory are stored in the annotation database as associated data of the annotation object.

14. A surgical navigation method having an annotation function for inputting position-related, in particular visual, medical annotations and recording them during a surgical operation on a patient (P) and for outputting the medical annotations, the surgical navigation method being in particular for use in a surgical navigation system (1) according to any one of the preceding claims, characterized in that step: recording a 3D recording (3DA) by means of a stereo recording system (12), in particular a stereo camera, particularly preferably a stereo microscope, arranged at or in the robot head (10), the stereo recording system having at least two recording units (14) spaced apart from one another, and providing the 3D recording (3DA) together with recording parameters to a control unit (20); detecting and tracking the position of the camera system (12) by a tracking system (16) and providing the position to the control unit (20); When detecting the annotation command, determining the orientation of the 3D shooting (3DA) by the orientation of the shooting system (12) and the shooting parameters; storing an annotation object having at least one partial cutout of the 3D acquisition (3DA) and the acquisition parameters in an annotation database of an annotation system having a global coordinate system; as well as A superimposed presentation (U) with the correctly positioned, in particular correctly oriented, content of the at least one annotation object is outputted by a presentation device (2).

15. A computer-readable storage medium comprising instructions which, when executed by the computer, cause the computer to implement the method steps of the navigation method according to claim 14.

Images