Navigation system and navigation method having an annotation function

Abstract

The present disclosure relates to a surgical navigation system (1) having an annotation function for inputting position-related, particularly visual, medical annotations or records and / or for outputting medical annotations during a surgical procedure on a patient (P), the surgical navigation system (1) comprising a display device (2), particularly an operating monitor, for visual output of medical annotations, at least one robot (4) as a positioning system, particularly a multi-link robotic arm (8) movably connected to a robot base (6), and a robot head (10) arranged at an end of the robotic arm (8), a stereoscopic recording system (12), particularly a stereo camera, particularly preferably a stereomicroscope, arranged on or within the robot head (10), the stereoscopic recording system comprising at least two recording units (14) arranged spaced apart from each other for performing a 3D recording (3DA) of a structure or surface for the recording system (12) and providing it in a computer-readable format together with recording parameters, a tracking system (16) adapted to detect and track the orientation of the recording system (12), particularly indirectly via the orientation of the robot head (19), and provide the orientation of the recording system (12) in a computer-readable format, an annotation system (18) comprising a storage device in which a central digital annotation database having a global coordinate system for annotation objects is stored, and a control unit (20) adapted to process the provided 3D recording (3DA) together with the recording parameters and the provided orientation, determine the orientation of the 3D recording (3DA) in the global coordinate system based on the provided recording parameters and the orientation of the recording system (12), and output at least one annotation object of the annotation database in a superimposed display (U) via the display device (2) at least at a correct position, particularly in a correct orientation, within the 3D recording (3DA) to improve navigation. The present disclosure relates to a navigation method and a computer-readable storage medium according to the independent and dependent claims.

Description

Technical Field

[0001] The present invention relates to a surgical navigation system having an annotation function / marker function for inputting medical annotations / markers or records related to a position, particularly a visual one, and / or for outputting medical annotations / markers during a surgical procedure on a patient, the system comprising a (visual) display device for visual output of medical annotations, particularly an operating monitor or display, AR glasses, or VR glasses, and at least one robot as a positioning system having a robot arm movably connected to a robot base, particularly a multi-link robot arm, and a robot head arranged at an end of the robot arm, preferably a plurality of robots operable independently of each other. An end effector, particularly an instrument, to be positioned may be connected to the robot head. Further, the present disclosure relates to a navigation method according to the preamble of the independent claims and a computer-readable storage medium and a computer program.

Background Art

[0002] In the field of surgical navigation, there are currently generally two different groups / types / types / classes of surgical navigation systems.

[0003] The first group of surgical navigation systems is so-called image-based navigation, which uses recordings / images such as computed tomography recordings (CT recordings), magnetic resonance imaging recordings (MRI recordings), X-rays, ultrasound scans, etc., generated before surgery, i.e., before the actual surgical procedure on a patient. The preoperative recordings must be associated / compared with the position or posture of the patient during the surgical procedure in order to enable proper navigation of the instrument or implant. In so doing, the virtual anatomical structure of the patient in the preoperative (3D) recording data is associated with the actual anatomical structure during the operation (currently), and the patient is registered in relation to the preoperative recording data in order to display the instrument related to the preoperative recording data.

[0004] The second group of surgical navigation systems operates without records or images (image - less navigation). These navigation systems typically create a patient model during a surgical procedure, i.e., intra - operatively, by scanning anatomical landmarks / reference points with a navigation probe / pointer / navigation pointer or by performing kinematic analysis using a corresponding simplified axis model.

[0005] However, the first group of navigation systems, i.e., image - based navigation systems, has the drawback of complex and time - consuming registration and often does not provide the accuracy required for surgical procedures.

[0006] In contrast, image - less navigation systems do not require pre - operative images / records and intraoperative registration, but have the drawback that they can only use a very basic model of the patient's anatomical structure (such as the axis model of the leg in knee surgery) as a reference.

[0007] In orthopedic navigation, there are current efforts to combine the advantages of both techniques, i.e., to have as accurate a record as possible of the patient and / or the patient's surgical area without performing pre - operative imaging and intraoperative registration, to properly align instruments or implants, and ultimately to achieve the best results for the patient.

[0008] The image - based navigation method is the current standard for computer - based surgery. One central part of image - guided / image - based navigation is the computer tomography visualization / CT visualization of the target position, target trajectory, and CT risk objects / structures related to pre - detected medical record data. This (guide) information is currently derived only from pre - detected medical image data and thus, from a pre - treatment temporal perspective, is used as the main guide modality during specific steps of a surgical procedure.

[0009] Typically, in image-based navigation, preoperative records and / or information from preoperative records about markers placed on all necessary objects, in particular fiducial points that reflect, are used to align (as a measurement system) an optical tracking system, in particular an active or passive optical infrared system, or an electromagnetic tracking system, to the surgical area / surgical site. As a result, the visualization of preoperative information regarding the intraoperative microscopic or endoscopic (current) live view of the patient's surgical area / surgical field is unfortunately very inaccurate. In particular, such inaccuracies can be due to inaccurate registration. Particularly high inaccuracies can result from the fact that the patient's anatomical structure shifts between the preoperative (image) record and the intraoperative situation, such as the so-called brain shift when the patient's skull is opened.

[0010] In particular, the following aspects affect the inaccuracy or accuracy of navigation. Changes in the anatomical structure relative to the preoperative recorded data due to tracking errors of individual tracked rigid bodies / reference sets, patient registration errors, errors in the reconstruction of the recording, tissue shifts such as the so-called brain shift, errors in hand-eye camera calibration, and / or errors in optical (camera) calibration.

[0011] Furthermore, optical tracking technology requires a (continuous) line of sight between the tracking camera and the instrument being tracked, which greatly limits the actual working area of the surgeon. For example, if the position of the surgeon, other instruments, or the robotic arm is unfavorable, the line of sight may be blocked and / or inhibited, resulting in tracking being undesirably inhibited during the surgical procedure, threatening the safety of the patient. Also, the time of the surgical procedure is lengthened. SUMMARY OF THE INVENTION

[0012] Accordingly, the object and purpose of the present disclosure is to eliminate or at least reduce the drawbacks of the prior art, and in particular, to provide a surgical navigation system, a navigation method, a computer-readable storage medium, and a computer program having an annotation function that provides improved navigation, in particular, accurate navigation with improved flexible annotation. Another partial object is to collect information from different subsystems and store it centrally, and to appropriately centrally integrate the information for navigation in order to make all data organizationally available. In particular, it is a partial object to provide high-precision navigation during the operation even without preoperative information (records).

[0013] The object of the present disclosure is achieved, according to the present invention, with respect to a general surgical navigation system by the features of claim 1, and is achieved, according to the present invention, with respect to a general navigation method by the features of claim 14, and is achieved, according to the present invention, with respect to a computer-readable storage medium by the features of claim 15.

[0014] Therefore, the core idea of the present disclosure is the current intraoperative detection of characteristic points or objects in a recording, in particular a 3D recording (i.e., during a surgical procedure on a patient), the corresponding registration (or position recognition, in particular the pose recognition of a point, object, or region of tissue structure), and the storage in at least the corresponding coordinates (X, Y, and Z coordinates) in one (single) central global coordinate system. The global coordinate system can be aligned with the patient, for example, as the patient's coordinate system, or alternatively, can be aligned with the navigation system, in particular the zero point of the coordinate system (COS) of the tracking system, preferably the zero point of the tracking camera. It is important that all information is transferred to one single central coordinate system, thereby eliminating subsequent conversions and sources of error.

[0015] The determination of the (position) coordinates of the annotation object incorporates different position data included by a combination of navigation (global position detection), visualization by a recording system (local position detection), and robotics or robotics engineering (sensory position detection by a robotic arm). In other words, according to the present disclosure, the improvement of navigation can be achieved by a combination of visualization, robotics, and navigation.

[0016] Put another way, an annotation function / marking function for inputting nerve signals or histological information, medical annotations or records at specific points of a position-related, especially visual, otherwise anatomical structure, and / or for outputting medical annotations during a surgical procedure on a patient, a surgical navigation system having a display device for visual output of medical annotations, especially an operating monitor, a positioning system as a robot having a robot arm movably connected to a robot base, especially a multi-link type, and a robot head arranged at an end of the robot arm, the navigation system comprising a stereoscopic recording system arranged on or in the robot head, especially a stereo camera, particularly preferably a stereomicroscope, having at least two recording units arranged spaced apart from each other for performing a (current) 3D recording of a structure or surface (of the target area) for the recording system and providing it in a computer-readable format together with the recording parameters of the recording system, a tracking system adapted to detect and track the pose of the recording system, especially indirectly via the pose (i.e., position and orientation) of the robot head, and provide the pose of the recording system in a computer-readable format, an annotation system / marking system having a storage device in which a digital annotation database / marking in a global coordinate system is stored / kept, and a control unit adapted to process the provided 3D recording together with the recording parameters and the provided pose, determine the pose of the 3D recording in the global coordinate system based on the provided recording parameters and the pose of the recording system, and output at least one annotation object (stored in the global coordinate system in the annotation database and thus associated with the pose of the 3D recording) in the 3D recording at least at the correct position, especially in the correct pose, in an overlay display via the display device to improve navigation.As a result, in particular for the (current intraoperative) 3D recording, at any time and at any position of the recording system, the stored annotation objects and / or annotations can be overlaid and displayed on the 3D recording in the correct position, in particular in the correct orientation (if the position, in particular the orientation, of the annotation is within the field of view of the recording system, i.e., within the 3D recording).

[0017] In particular, the control unit outputs the annotation object for overlay display via the display device at a later point in time at least in the correct position, in particular in the correct orientation, and determines the orientation of the 3D recording for the annotation instruction / marking instruction (input by the user) based on the provided recording parameters and the orientation of the recording system in order to improve navigation, and stores at least a part of the 3D recording or the target area as an annotation object / marking object in the annotation database of the annotation system in the global coordinate system at the determined position, in particular in the orientation, together with the annotation parameters. The navigation system can store the annotations in a unified manner with particularly high accuracy with respect to the position data, in particular the orientation data.

[0018] Putting it yet another completely different way, there is provided a novel surgical navigation system comprising one or more volumetric recording systems / optical camera systems that are attachable to or arranged in a positioning system (in particular a robotic arm, an optical tracking system, an electromagnetic system). The (main) recording system is used during the operation (i.e., during the surgical procedure) to display and / or visually output a (current) (3D) recording, such as a live 3D video view of the surgical site, to a display device. Image internal or spatial annotations / memos can be set manually or automatically by an additional tracking system (measurement system or tracking system) on the live video image of the surgical field / surgical area. The image internal or spatial annotations are detected and stored in the same (single) global coordinate system and thus can be visualized and reconstructed by the user if necessary for a surgical guide. This is possible with high reproducibility and overall high accuracy even if the positioning system (robotic arm, optical tracking system, electromagnetic system) moves in between and camera parameters such as the focus position or focal plane change. The term "focus" defines a focal plane of the optical system of the recording system and / or recording unit, in particular a spatial point located on the optical axis.

[0019] In particular, the determination of the posture based on the robot kinematics of the volumetric recording system in the coordinate system of the robot is performed, for example, via the robot, and the determination of the posture of the 3D recording in the coordinate system of the recording system is performed via the recording system. Similar to the serial transformation, these two posture determinations are connected to each other in order to ultimately determine the posture of the 3D recording relative to the robot (and the robot coordinate system). For example, by converting the determined posture of the 3D recording into the robot coordinate system, based on the determined posture of the recording system in the robot coordinate system. Subsequently, the posture of the 3D recording is determined, for example, by a defined or determinable transformation in the global coordinate system from the robot coordinate system to the global coordinate system (in one embodiment, this may be the (virtual) coordinate system of the patient / patient coordinate system). In other words, the data for the annotations transferred or converted to a single global coordinate system can be obtained from different modalities and are stored together with the corresponding positions (in X-Y-Z coordinates in space), in particular the corresponding positions and orientations (i.e., postures) in the global coordinate system. In this way, different annotations are not provided to different databases with different (local) coordinate systems, and the annotations are integrated into a joint database with a joint global coordinate system. In this way, in fact, standardization is performed, and the unified system is generated by a database with standardized annotations in the form of an array of data having annotations (e.g., annotation ID, annotation position, annotation orientation, annotation data (image, vector graphics, text, etc.), and optionally annotation transparency (from 0% to 100%, 100% means the annotation is off / invisible)). The annotations can then be selected and rendered, in particular computer-based, so as to be superimposed and displayed in the correct position (i.e., the correct position relative to the 3D recording and thus relative to the patient), in particular in the correct posture (i.e., the correct orientation in the correct position) on the 3D recording of the recording system. In particular, when the patient is tracked and registered, it is advantageous to use a patient coordinate system that correlates appropriately as the global coordinate system.In this way, different annotations of the central database having a uniform global coordinate system can be displayed in the 3D recording, for example, as an opaque or translucent overlay.

[0020] In this case, in particular, a surgical navigation system having at least one stereoscopic optical recording system for generating an intraoperative 3D recording, which is attached to or arranged on a positioning system (in particular a robotic arm) for displaying a live video view (live 3D recording) of the surgical site, is described. A camera can also be used to detect further spatial and visual information from the surgical field. The recording system attached to the positioning system (in particular a robotic arm, an optical tracking system, an electromagnetic tracking) can be moved to a specific spatial position by a user such as a surgeon and can be aligned in particular with respect to orientation, and the recording system provides a live view on the surgical field to the user.

[0021] The surgeon can then use methods for identifying and labeling different anatomical or physiological regions. For example, in the output 3D recording, points can be set as annotations by means of a mouse pointer, or additional measurement systems and methods can be applied to automatically annotate and in particular label the appropriate regions in this case. Thus, at least one 3D recording, in particular a live video image (annotation reference recording), and the spatial position (coordinates) and the current camera settings via the positioning system (in particular via the current configuration of the robotic arm) are transferred and stored as annotation objects in the global coordinate system.

[0022] In contrast to the waypoint function in existing robot-based microscopes or the annotation function of microscope systems, this navigation system stores both more accurate data and features in order to accurately and simply visualize annotation objects with respect to the anatomical structure of the patient. Furthermore, the navigation system provides the possibility of moving the entire navigation system or changing the camera settings during a surgical procedure, and the annotation objects can be reconstructed and visualized. With the disclosed navigation system, since the data is detected during the surgery rather than preoperatively, changes in the anatomical structure can be safely recognized.

[0023] The term "tracking system" refers to a technical system that enables spatial positioning and can detect position and / or orientation.

[0024] The term "position" means a geometric position in a three-dimensional space, particularly indicated using the coordinates of a rectangular coordinate system. In particular, a position can be indicated by three coordinates X, Y, Z.

[0025] The term "orientation" indicates alignment in space (e.g., with respect to a position). It can also be said that the alignment with a direction and / or rotation in a three-dimensional space is indicated by the orientation. In particular, an orientation can be indicated by three angles.

[0026] The term "pose" includes both position and orientation. In particular, a pose can be indicated by six coordinates, three position coordinates X, Y, Z and three angular coordinates for the orientation.

[0027] The term "in the correct position" defines that the annotation and / or 3D recording is reproduced in the actual correct position in an overlay display, similar to a CAD system.

[0028] The term "in the correct pose" defines a combination of correct position and correct orientation where the recording is displayed in both correct position and correct orientation in the overlay display. When the recording directions are consistently adjusted in parallel, the overlaid recordings are displayed with a certain aspect ratio similar to the actual recording. When there is an angle in the recording direction, for example, the extension or dimension of the recording in the relevant direction is compressed corresponding to the angle until only a one-dimensional line is generated in the overlay display when the angle between the two recording directions is 90°.

[0029] The term "target area" refers to an area of interest and represents a part of the surgical area that should be assumed by the recording system. In particular, the target area may be a single target point (having X, Y, Z coordinates) in space and may be indicated as such coordinates. However, the target area may also be a section from a two-dimensional plane, such as a small section of the tissue surface. Additionally, the target area may be indicated as a small cubic or spherical volume.

[0030] In contrast to image-based navigation, the proposed navigation system and navigation method include, as factors that adversely affect the accuracy shown below, the hand-eye calibration between the camera and the robot-based coordinate system, and only the calibrated camera system, which need to be considered and can be appropriately maintained for accuracy.

[0031] Advantageous embodiments are claimed in the dependent claims and are particularly described below.

[0032] According to one embodiment, the robot arm configuration of the robot and the camera settings of the recording system are such that, at a point in time after a surgical procedure, the robot is moved / displaced in position and orientation with the corresponding camera settings, and also, in an overlay display, the current 3D recording and the previously saved 3D recording are output as a consistent reference, for example as a juxtaposed display or as a transparent overlay display for visualizing changes over time, and can be saved as recording parameters in an annotation database. In particular, in the case of so-called in-image annotations, i.e., annotations within an image, the camera of the recording unit needs to be calibrated, and when visualizing repeatedly, this technique requires that the annotation process (consistent field of view) be set to the same zoom and the same focus as the recording system.

[0033] In particular, as recording parameters, the zoom (magnification) and focus can be detected and saved as camera settings of the stereo camera, whereby at least the position of the segment of the 3D recording is accurately determined and / or can be obtained for visualization via the zoom and focus through the posture of the recording system tracked by the tracking system.

[0034] Furthermore, the control unit is preferably adapted to move / displace the recording system to the posture saved in relation to the annotation object in the case of an output instruction regarding the annotation object saved in the annotation database, and in the overlay display, output the current 3D recording and the saved annotation object, in particular the section of the saved 3D recording, via the display device.

[0035] In particular, annotations, especially in-image annotations, can also be assisted by computer vision algorithms by tracking the corresponding regions while the zoom step and the focus / focal length change.

[0036] According to another preferred embodiment, further preoperative 3D recording data, in particular CT recording data and / or MRI recording data and / or sectional X-rays, can be stored in the memory device, the surgical navigation system can be adapted to register the patient with respect to the preoperative 3D recording data, and the control unit can be adapted to additionally display the preoperative 3D recording data in a superimposed display in addition to the 3D recording for further assistance. In other words, the preoperative medical recording data / image data (3DCT data / 3DMRI data) can thus be additionally used as assistance information that is not displayed as the main guidance information but only as auxiliary guidance information. The main guidance information is composed of the intraoperative recording and the associated annotations respectively.

[0037] According to another embodiment, the control unit can be adapted to display a warning message in a superimposed display when a section of the 3D recording stored in the annotation database deviates from a preset tolerance range, in particular by means of an image analysis method, with respect to the current 3D recording.

[0038] In particular, the control unit can be adapted to determine that there is a shift of the entire patient when the 3D recording stored in the annotation database deviates from the current 3D recording by more than a preset tolerance range, in particular by means of an image analysis method, or when at least a section of the stored 3D recording is shifted as a whole with respect to the current 3D recording. Preferably, the control unit can further be adapted to newly register the navigation system for the patient, in particular to accurately correct the registration according to the established shift.

[0039] Furthermore, particularly when the deviation of the stored 3D recording from the current 3D recording in the annotation database exceeds a preset tolerance range, especially by means of an image analysis method, and when the stored 3D recording or at least a section of the 3D recording is distorted, such as being compressed with respect to the current 3D recording, it is determined that only the patient's tissue has shifted. More preferably, it can be adapted to perform registration on the relevant tissue.

[0040] In particular, the control unit detects the instrument in the current 3D recording and arranges / displays the instrument in the front plane in the superimposed display, while at least one annotation object is displayed in the intermediate plane between the live recording and the instrument, that is, in the rear plane when viewed relative to the instrument, and can be adapted so that the annotation object does not interfere with the instrument and / or the instrument is not covered by the annotation object in the superimposed display. Furthermore, in particular, the control unit detects the instrument in the current 3D recording and arranges / displays the instrument in the rear plane of the superimposed display, while at least one annotation object can be adapted to be displayed in the front plane, that is, the front plane when viewed relative to the instrument. In this way, the annotation may be arranged in front of or behind the instrument. When the annotation is arranged in front of the instrument, the annotation covers the instrument non-transparently or semi-transparently (e.g., with a 70% coverage rate). When the annotation object is behind the instrument, the instrument covers the annotation non-transparently or semi-transparently (e.g., with a 70% coverage rate).

[0041] According to another embodiment, the recording system can provide the current / live 3D recording during the operation, which is output via a display device on the one hand and stored in the annotation database in the case of an annotation instruction on the other hand.

[0042] In particular, the transparency parameter between 0 and 1 (i.e., continuous values between 0% and 100%), and thus between non-display and display, may be stored for each annotation object of the annotation database, and the control unit may be adapted to display, in a superimposed display, all annotation objects corresponding to that transparency, at least at the correct position and in particular in the correct orientation, together with the current 3D recording.

[0043] According to one embodiment, the control unit may be adapted to vary a threshold value for the display of the contour / outline, in particular based on a manual input, mark areas for the user if necessary, and shift the boundaries of the areas corresponding to the set threshold value.

[0044] In particular, the patient's head can be fixed relative to the robot and / or the robot base of the robot system. This is common in particular in image-based navigation systems. As an alternative, the patient can also be tracked by the same positioning system, in particular a tracking system.

[0045] According to another embodiment, the navigation system and in particular the recording system may be adapted to perform a camera calibration to provide the control unit with information on how the collected annotation objects move with respect to a zoom / (zoom step) or a change in focus / focus width, in particular information on a three-dimensional pyramid, and the control unit may be adapted to change the camera settings during the method. In other words, the navigation system requires a positioning system (in particular a robotic arm, an optical tracking system, an electromagnetic tracking) with high reproducibility of positioning in space and a (main) camera system with high reproducibility of zoom and focus / focus width, which is why it is advantageous when the camera system is calibrated.

[0046] Preferably, the incremental number / marker ID of the particularly tracked pointing device (navigation pointer / pointer), and / or the intensity information of the fluorescence recording, and / or the convex hull of the trajectory, and / or the spherical distance field can be stored in the annotation database as the relevant data of the annotation object. In other words, the incremental number / marker ID such as that of a pointing device / computer vision tracking device, the intensity information such as fluorescence recording, for example, the convex hull of the trajectory, for example, the spherical distance field from neuromonitoring can be stored, particularly as the relevant data of the annotation object. The annotation specifically is, physiological information, particularly nerve signals and / or histological information, geometric objects, particularly points, circles, lines, trajectories, and / or contours, and / or auxiliary image information, particularly fluorescence may be. Different display modalities can be displayed in the (current) 3D recording by the navigation system of the present disclosure where they are actually located, i.e., in the correct position, particularly in the correct orientation.

[0047] Preferably, as manual labeling techniques, the following techniques can be used: the focus of the recording system, particularly the focus of the microscope (in a one-dimensional object with X - Y - Z coordinates in a global coordinate system, points in space can be appropriately labeled); the cursor / mouse pointer in the live image of the display device (having X - Y coordinates and a Z coordinate as an offset with respect to the focus on the image plane); a navigated point or instrument such as a navigation pointer / pointer; computer vision techniques for recognizing edges or tissue patterns; the source of blood and the direction of blood flow over time for detecting the source of bleeding (using ML recognition).

[0048] In particular, automatic labeling techniques, particularly neuromonitoring, and / or fluorescence imaging (e.g., DUV400, DFS560, DIR800), and / or other optical measurement processes such as OCT can be used as annotation techniques.

[0049] In particular, the navigation system can store, as an annotation object, a 3D video recording, especially having a length of substantially 5 seconds (with a time limit), and a moving image / video is stored as an annotation object. In other words, in particular, a 4D annotation object (i.e., 3D with time-varying calibration elements) is stored and can be reproduced at a further point in time accordingly. In particular, this video recording may be transmitted from the ICG.

[0050] Preferably, the surgical navigation system can further include an aligned and adapted overview camera that is positioned to detect the environment of the surgical area and provide it in a computer-readable format. In other words, an additional camera system can assist in the automatic generation of annotation objects.

[0051] Preferably, the navigation system may be adapted to perform in-image / waypoint annotations. In so-called in-image / waypoint annotations, the user moves the recording system to the desired position and adjusts the camera to the desired parameters with respect to zoom and focus (zoom step and focus / focus width). This enables the annotation objects within the 2D recording to be identified manually or automatically. This category requires a robotic system with high reproducibility and a primary recording system / camera system with high reproducibility of zoom and focus adjustment. In particular, focus adjustment requires that the robotic head, especially the robotic end effector, moves in an appropriate pose with respect to the original pose (waypoint with stored camera parameters).

[0052] In particular, a specific camera calibration can provide information (3D pyramid) on how the collected annotation objects move with respect to changes in the zoom step or focus / focus width. This allows the user to change the camera settings during the procedure. Conventional annotation systems do not allow such flexibility during the procedure.

[0053] Preferably, the navigation system may be adapted to perform spatial annotation. With spatial annotation, the user can move the recording system (stereoscopic recording system / camera system) to a desired position and adjust the camera with respect to zoom steps and focus / focus width to desired parameters. This enables the annotation objects in the 2D recording / 2D image to be identified manually or automatically. Preferably, via a (stereoscopic) specific camera calibration, the spatial 3D position of the labeling object can be derived in the coordinate system of the positioning system. This category of spatial annotation requires a robotic system with high reproducibility and a recording system with high reproducibility of zoom and focus / focus width. Focus adjustment requires that the robot head and, in particular, the robot end effector move in a posture or pose close to the original posture (i.e., similar position and orientation within an acceptable range or a waypoint with stored camera parameters). Further, in particular, specific camera calibration for 3D reconstruction and hand-eye calibration are required to transfer the data of 3D reconstruction to the coordinate system of the positioning system. Thereby, the user can move the camera and freely change the camera settings. The visualization of the annotation object can be visualized taking into account the adapted posture and camera settings. This method can also preferably be assisted by computer vision algorithms by tracking the corresponding regions when changing the zoom and focus / focus width and when the robot is moving.

[0054] In particular, the following user interactions can be realized by the navigation system and / or the navigation system can be adapted to perform the following functions. Activation and deactivation of annotation objects (e.g., by corresponding entries in an annotation database) supplemented by a live view, shift of thresholds for the display of contours / contours (such as tumor boundaries) of a regular sample of point clouds (e.g., fluorescence images) with appropriate intensity, user-guided segmentation of structures by simple plots (based on so-called maximum cut minimum flow).

[0055] In particular, the free movement of the recording system (camera head) can be carried out by means of an extension of the spatial annotation object which requires calibration in particular. The annotation object is extended with respect to the fixed surgical site.

[0056] Preferably, in the overlay display, at least one direction regarding the stored annotation may also be displayed or indicated, in particular during the (automatic or manual) movement of the robotic arm.

[0057] Preferably, the measurement (in mm) of the annotation object may be carried out for documentation and may be stored as data associated with the annotation object in an annotation database.

[0058] More preferably, the focus and zoom may be stored and / or appropriately set for the spatial annotation object, in particular at the contour of the fluorescence image.

[0059] In particular, a (1D or 2D) heat map may be added as an annotation object in order to more clearly display the target structure or risk structure.

[0060] An important part of the navigation system is constituted by the visualization of the detected and processed annotations. The visualization of the annotation object may preferably be adapted so as not to interfere with the instrument into which the overlay of the object is inserted (for example in the same way as masking-background depth of field blur). As a result, the visualization for the surgeon is improved, and the impression that the annotation object is connected to the patient's surgical field and not just overlaid is enhanced. For example, in one embodiment, the control unit may be adapted to display the annotation object in a blurred manner such that the surgical instrument appears to be lying (masked) on top. The annotation may preferably be displayed with a certain transparency through different focus images (1-2 recording planes up or down).

[0061] In particular, the navigation system may be adapted to invalidate and mark detected spatial annotations when the underlying anatomical structure has changed due to a surgical procedure. This can be achieved by comparing the reference image of the annotation with the current live view of the camera.

[0062] Also, the navigation system, in particular the control unit, may preferably be adapted to suppress the visualization of spatial annotation objects if the visualization error increases after movement of the microscope head.

[0063] Preferably, the navigation system may be adapted to store biopsies or biopsy information as annotations. In particular, when the annotation points provide diagnostic information from a biopsy in particular, it is easily possible to automatically create a surgical document.

[0064] In particular, the spatial annotation object may be used to register preoperative CT / MRI data in order to obtain additional assistance during the operation.

[0065] Regarding a surgical navigation method, an object of the present disclosure is to have an annotation function for inputting visual medical annotations and records related to a position and outputting medical annotations during a surgical procedure on a patient. In particular, it is a navigation method for a surgical navigation system according to the present disclosure, which is a stereoscopic recording system arranged on or in a robot head, particularly a stereo camera, particularly preferably a stereomicroscope, and includes at least two recording units arranged at intervals from each other. The (current) 3D recording is recorded by the stereoscopic recording system, the 3D recording is provided to a control unit together with recording parameters, the posture of the recording system is detected and tracked by a tracking system, the posture is provided to the control unit, and when an annotation instruction is detected, the posture of the 3D recording is determined based on the posture of the recording system and the recording parameters, and an annotation object having at least one section of the 3D recording and the recording parameters is stored in an annotation database of an annotation system having a global coordinate system, and at least one annotation object is output via a display device in a correct position, particularly in a correct posture, including an integrated superimposed display. This is achieved by the fact that a navigation method is provided.

[0066] Regarding a computer-readable storage medium and a computer program, each object is achieved by a computer program including instructions that, when executed by a computer, cause the computer to execute the method steps of the navigation method of the present disclosure.

[0067] All disclosures related to the surgical navigation system of the present disclosure are also applicable to the navigation method of the present disclosure, and vice versa.

[0068] Hereinafter, the present disclosure will be described in detail based on preferred embodiments with reference to the drawings.

Brief Description of the Drawings

[0069]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

[0070] The figures are schematic and are only intended to aid in the understanding of the present invention. The same reference numerals are assigned to similar elements. The features of different embodiments can be interchanged.

DETAILED DESCRIPTION OF THE INVENTION

[0071] FIG. 1 shows a conventional surgical navigation system in a schematic perspective view. In this case, a plurality of error sources that adversely affect accuracy occur. The instrument is tracked by a tracking camera via an attached rigid body (infrared marker), and a tracking error with a specific error tolerance occurs for each individual rigid body. Also, the patient is tracked via a rigid body, but this is accompanied by a tracking error. Furthermore, errors can occur due to recording reconstruction and anatomical changes to preoperative recording data, such as tissue shifts like so-called brain shift. Another source of error can be hand-eye camera calibration or optical calibration. In order to enable better and more accurate navigation, these errors and chains of errors must be avoided or at least minimized.

[0072] Figure 2 shows a perspective view of a surgical navigation system 1 having an annotation function (hereinafter referred to as a navigation system). The annotation function integrates visual medical annotations and records related to positions during the operation and serves as a guide and for surgical procedures on the patient P (shown here).

[0073] To be able to visually display the input annotations, the navigation system 1 includes a display device 2 in the form of an operating monitor for visually outputting medical annotations for qualified medical personnel, especially surgeons.

[0074] The navigation system 1 further includes a robot 4 as a positioning system. Specifically, the robot has a stationary robot base 6 and a multi-link robot arm 8 that is movably connected to the robot base 6, actively controlled, and movable / displaceable. An end robot head 10 that is at least rotatable with respect to the longitudinal axis of the last segment / link of the robot arm 8 is movably connected to the end of the robot arm 8.

[0075] The robot head 10 is equipped with a stereoscopic recording system 12 having at least two recording units 14 (recording cameras in this case) arranged at intervals from each other, especially a stereo camera, in this case a stereomicroscope. The stereoscopic recording system 12 records a 3D recording 3DA of a structure or surface via two optical systems each having a subsequent sensor. The 3D recording 3DA is provided in a computer-readable format together with recording parameters, especially camera parameters in this case.

[0076] To track the robot head 10, in this embodiment, the navigation system 1 is configured as a tracking system 16 with mechanical sensors, and through the configuration of the robot arm, it detects the posture of the robot head 10 relative to the robot base 6, detects the posture of each relative to the coordinate system of the robot 4, and indirectly detects the posture of the recording system 12 through the posture of the robot head 10, and spatially tracks it, and is equipped with a tracking system 16. The detected posture of the recording system 12 is provided in a computer-readable format.

[0077] For the data management of annotations, the navigation system 1 includes a central internal annotation system 18 having a storage device (not shown here) in which a digital annotation database having a single global coordinate system is stored. In this case, the global coordinate system is the coordinate system of the robot 4. In the annotation database, new annotation objects can be input as list items (similar to an SQL database), and data related to the annotation objects can be stored.

[0078] A particularly adapted control unit 20 (the annotation system 18 is part / sub-unit of the control unit 20) is configured to process the provided 3D recording 3DA together with the recording parameters and the provided posture of the recording system. When an annotation object is stored in the annotation database, based on the provided recording parameters and the posture of the recording system, the control unit 20 can first determine the posture of the 3D recording 3DA in space, that is, the posture relative to the central global coordinate system. The stored annotation objects, and the data of the corresponding annotation objects respectively, are then output in a superimposed display at the correct position, especially in the correct posture, when they enter the field of view of the 3D recording. For example, images, texts, or other medical data for surgical procedures may be stored as annotation objects, the image may be displayed in the 3D recording in the correct posture, and the text may be displayed in the 3D recording in the correct position.

[0079] Furthermore, the control unit 20 is adapted to, in particular, process the provided 3D recording 3DA together with the recording parameters and the pose of the provided recording system, and in the case of an annotation instruction, automatically, for example, via the touch display of the surgical monitor as a display device or based on an internal annotation program, determine first the pose of the 3D recording in space, i.e., the pose with respect to the central global coordinate system, based on the provided recording parameters and the pose of the recording system. In other words, the pose of the 3D recording is finally determined via the recording system 12 and the optical parameters from the robot base 6 via the robot head 10. The pose is transferred to the central global coordinate system.

[0080] In the annotation database of the annotation system, at least sections of the 3D recording of regions of interest, such as tumors, are stored as annotation objects at a specific position, here furthermore at a specific pose, together with the recording parameters in the global coordinate system. In particular, when the navigation system 1 has not only one robot 4 but a plurality of robots 4 including robotic arms, important data for a surgical procedure can be stored centrally, and one (single) global coordinate system centrally records all annotations. The global coordinate system can be assumed to be a three-dimensional space with a Cartesian coordinate system that stores different types of annotations recorded during the operation at different X - Y - Z positions.

[0081] And at a later point in time, the surgeon can output previously recorded annotation objects in the overlay display U in the live display of the current recording of the recording system 12, at least at the correct position, such as a point in the space fixed by the navigation pointer, or at the correct pose, such as a 3D recording of the surface, via the display device 2. The surgical navigation is clearly improved in this way.

[0082] Figure 2 shows a consistent perspective at a first point in the first posture of the recording system, where an annotation instruction is given and the annotation is stored corresponding to the relevant recording data, i.e., the annotation data. Substantive parameters of camera settings such as focus and zoom, and the robot configuration (the configuration of the links of the robot arm 8 relative to each other and to the robot base 6) are stored in the central annotation database of the global coordinate system at the correct position together with additional information regarding the orientation. After the first point in time, the robot is moved and the surgical procedure is continued as appropriate.

[0083] If the surgeon intends to return to the first annotation, the surgeon selects the corresponding annotation by an output instruction, manually, or by actuating the touch display of the surgical monitor. The control unit controls the robot arm 8 to the configuration intended for this purpose, thus to the same posture at the first point in time of the recording system 12, based on the data stored regarding the annotation object, aligns the focus of the camera corresponding to the stored camera settings, and outputs on the one hand the current 3D recording 3DA and on the other hand a superimposed display including a part of the stored 3D recording 3DA, e.g., by side-by-side display or superimposition by transparency, presenting the change over time visually to the surgeon.

[0084] Figures 3 and 4 show perspective views of a surgical navigation system 1 in a further preferred embodiment. The said embodiment differs only in that the control unit is adapted to perform spatial annotations and the control unit 20 is adapted to display a spatial representation of a previously recorded annotation object, such as by a corresponding matrix transformation, even from a second posture different from the first posture (see Figure 4 where the first posture is shown by a dashed line and the second posture by a solid line). By doing so, the central global coordinate system with the stored annotation object serves to display the annotation in the correct posture in the second posture of the recording system 12 in relation to the image analysis and recording processing procedures.

[0085] FIG. 5 schematically shows a situation where the robot 4 moves and the object to be annotated moves out of the field of view (e.g., a consistent field of view).

[0086] FIG. 6 schematically shows an example of an annotation stored in an annotation database. A point in space in a global coordinate system having X, Y, Z coordinates is stored as an annotation object having a corresponding labeling that can also be displayed in an overlay display U as needed.

[0087] FIG. 7 shows a further alternative of the stored annotation where an area of the tissue surface is marked (in color) as a pentagon. In a surgical procedure, the area can also be displayed to the surgeon.

[0088] FIG. 8 shows a further annotation in the form of a trajectory displayed within the tissue to provide guidance to the surgeon.

[0089] FIGS. 9 and 10 show 3D recordings of tissue structures having fluorescence stored as annotations.

[0090] In FIG. 11, an instrument 22 (in this case a pointer) having a ring mark provided on the instrument 22 itself and a distance field at the distal tip of the instrument 22 is shown as a further configuration of the annotation.

[0091] FIGS. 12 and 13 show flux vectors as annotations or annotation objects in the stored 3D recording 3DA.

[0092] FIG. 14 shows the area where the biopsy was performed, and FIG. 15 schematically shows biopsy results having different areas.

[0093] FIG. 16 shows the change of the boundary / limit of the area where the threshold value is continuously decreased by the user (from left to right). In other words, the user can predetermine the threshold value, and based on the threshold value, a boundary is visualized around the tissue area. This is one alternative of the annotation.

[0094] FIG. 17 shows the dimensions in mm as an annotation. Thus, a sterile measuring instrument in the form of a ruler was placed inside the tissue and a 3D recording was made. In this way, at a later point in time, when the recording system is in the same situation again, there is no need to physically integrate the measuring instrument again, and the dimensions can be displayed at the required locations.

[0095] FIG. 18 shows another alternative of the annotation in the form of a heat map.

[0096] FIG. 19 shows a flowing video as an annotation so that the video can also be reproduced.

[0097] FIG. 20 shows the instrument 22 during the current recording. In this case, the marker points shown as circles are displayed as an annotation on the background of the instrument. The control device 20 is adapted to recognize the instrument 22 and arrange the instrument 22 forward in the plane so that the annotation does not cover the instrument 22.

Description of the reference numerals

[0098] 1 Surgical navigation system 2 Display device 4 Robot 6 Robot base 8 Robot arm 10 Robot head 12 Recording system 14 Recording unit 16 Tracking system 18 Annotation system 20 Control unit 22 Instrument P Patient 3DA 3D recording U Overlay display

Claims

1. A surgical navigation system (1) having an annotation function for inputting medical annotations or records related to position, particularly visual ones, and / or for outputting medical annotations during a surgical procedure on a patient (P), a display device (2), particularly an operating monitor, for visual output of the medical annotations, at least one robot (4) as a positioning system, comprising a robot arm (8), particularly a multi-link type, movably connected to a robot base (6), and a robot head (10) arranged at an end of the robot arm (8), a stereoscopic recording system (12), particularly a stereo camera, particularly preferably a stereomicroscope, arranged on or within the robot head (10), the stereoscopic recording system comprising at least two recording units (14) arranged at intervals from each other to perform a 3D recording (3DA) of a structure or surface for the recording system (12) and provide it in a computer-readable format together with recording parameters, a tracking system (16) adapted to detect and track the posture of the recording system (12), particularly indirectly via the posture of the robot head (19), and provide the posture of the recording system (12) in a computer-readable format, an annotation system (18) comprising a storage device storing a central digital annotation database having a global coordinate system of annotation objects, a control unit (20) adapted to process the provided 3D recording (3DA) together with the provided recording parameters and the provided posture, determine the posture of the 3D recording (3DA) in the global coordinate system based on the provided recording parameters and the posture of the recording system (12), and output at least one annotation object of the annotation database in a superimposed display (U) at least at a correct position, particularly in a correct posture, within the 3D recording (3DA) via the display device (2) to improve navigation, comprising a surgical navigation system.

2. Output the annotation object at a later time, at least in the correct position, in particular in the correct orientation, via the display device (2) in a superimposed display (U), and in order to improve navigation, the control unit, in the case of an annotation instruction, determines the orientation of the 3D recording (3DA) based on the provided recording parameters and the orientation of the recording system (12), and as the annotation object, at least a section of the 3D recording (3DA), together with the recording parameters, is adapted to be stored in the annotation database of the annotation system in the global coordinate system at a specific position, in particular in an orientation. The surgical navigation system according to claim 1.

3. At a later time after the surgical procedure, move the robot (4) to the exact position and orientation with the corresponding camera settings, and output the current 3D recording and the previously stored 3D recording as a consistent reference in a superimposed display (U) to visualize the changes over time. For this purpose, the robot arm configuration of the robot (4) and the camera settings of the recording system (12) are stored in the annotation database as the recording parameters. The surgical navigation system (1) according to claim 1 or 2.

4. The detected zoom (magnification) and focus as the camera settings of the stereo camera of the recording system (12) are stored as recording parameters so that the position of at least a segment of the 3D recording can be accurately determined via the orientation of the recording system (12) tracked by the tracking system and via the zoom and the focus. The surgical navigation system (1) according to any one of claims 1 to 3.

5. In the case of an output instruction regarding the annotation object stored in the annotation database, the control unit (20) moves the recording system (12) to the orientation stored for the annotation object, and is adapted to output the current 3D recording (3DA) and the stored annotation object, in particular the stored portion of the 3D recording (3DA), via the display device in a superimposed display (U). The surgical navigation system (1) according to any one of claims 1 to 4.

6. Preoperative 3D recording data, in particular CT recording data and / or MRI recording data and / or segmented X-ray data, is further stored in the memory device, the surgical navigation system (1) is adapted to register the patient (P) with respect to the preoperative 3D recording data, the control unit (20) is adapted to also display the preoperative 3D recording data for further assistance in the superimposition display (U) related to the 3D recording (3DA), according to any one of claims 1 to 5, the surgical navigation system (1).

7. the control unit (20) is adapted to display a warning message in the superimposition display (U) when, in particular by means of an image analysis method, a section of the 3D recording (3DA) stored in the annotation database deviates beyond a preset tolerance range in relation to the current 3D recording (3DA), according to claim 3, the surgical navigation system (1).

8. the control unit (20) detects an instrument (22) in the current 3D recording (3DA) and, in the superimposition display (U), at the superimposition position (U), arranges at least one annotation object in a rear plane and the instrument (22) in a front plane so that the annotation object does not interfere with the instrument (22), according to any one of claims 1 to 7, the surgical navigation system (1).

9. the recording system (12) provides the current 3D recording (3DA) during the operation, which is output on the one hand via the display device and stored in the annotation database on the other hand in the case of an annotation instruction, according to any one of claims 1 to 8, the surgical navigation system (1).

10. For each annotation object in the annotation database, a transparency parameter between 0 and 1, i.e., between non-display and display, is stored, and the control unit (20) is adapted to display all annotation objects corresponding to the transparency, together with the current 3D recording (3DA), at least in the correct position and in particular in the correct orientation, in the superimposition display, according to any one of claims 1 to 9, the surgical navigation system (1).

11. The surgical navigation system (1) according to any one of claims 1 to 10, wherein the control unit (20) is adapted to change a threshold value for indicating a contour / outline as an annotation in order to mark an area for the user if necessary.

12. The recording system (12) is adapted to perform camera calibration in order to provide the control unit (20) with information regarding how the collected annotation objects move with respect to a change in zoom or focus, in particular information about a three-dimensional pyramid, in order to enable the control unit (20) to change the camera settings during movement. The surgical navigation system (1) according to any one of claims 1 to 11.

13. An incremental ID, in particular an incremental ID of a tracked pointing device, and / or intensity information of a fluorescence recording, and / or a convex hull of a trajectory, and / or a spherical distance field are stored as associated data of the annotation objects in the annotation database. The surgical navigation system (1) according to any one of claims 1 to 12.

14. A surgical navigation method for a surgical navigation system (1) according to any one of claims 1 to 13, having an annotation function for inputting position-related, in particular visual, medical annotations and recordings and outputting the medical annotations during a surgical procedure on a patient (P), recording a 3D recording (3DA) by means of a stereoscopic recording system (12) arranged on or in a robot head (10), in particular a stereo camera, particularly preferably a stereomicroscope, the stereoscopic recording system (12) comprising at least two recording units (14) arranged at a distance from each other, and providing the 3D recording (3DA) together with recording parameters to a control unit (20), detecting and tracking the pose of the recording system (12) by means of a tracking system (16) and providing the pose to the control unit (20), determining the pose of the 3D recording (3DA) from the pose of the recording system (12) and the recording parameters when an annotation instruction is detected, storing an annotation object having at least one section of the 3D recording (3DA) and recording parameters in an annotation database of an annotation system having a global coordinate system. Outputting a superimposed display (U) while integrating the at least one annotation object at a correct position, particularly in a correct orientation, via the display device (2). A method comprising this.

15. A computer-readable storage medium including instructions for causing a computer to execute the method steps of the navigation method according to claim 14 when executed by the computer.

Images