Surgical assistance system and representation method

Abstract

The present disclosure relates to a surgical assistance system, comprising: a representation device; an endoscope having a distal imaging shooting head, the shooting head being suitable for creating an in-vivo shooting of a patient; a data providing unit, the data providing unit being suitable for providing digital 3D shooting data of the patient; and a control unit, the control unit being suitable for processing the in-vivo shooting and 3D shooting data of the endoscope, wherein the in-vivo shooting is initially registered with the 3D shooting data by corresponding anatomical landmarks and / or orientations. The tracking system continuously detects the position and / or movement and / or orientation of the endoscope, and the control unit generates an associated representation having not only a representation of the in-vivo shooting but also a representation of a view of the 3D shooting data by means of the detected position and / or movement and / or orientation, in which the in-vivo shooting and the view of the 3D shooting data are associated with each other in terms of the endoscope. In addition, the present disclosure relates to a shooting representation method, a storage medium, and a sterilization chamber.

Description

Technical Field

[0001] The present disclosure relates to a surgical assistance system for use in surgical interventions on a patient, comprising a display device and an endoscope. In addition, the present disclosure relates to a method for recording and displaying, a computer-readable storage medium, and a sterilization chamber. Background Art

[0002] In minimally invasive surgeries on soft tissues (e.g. abdominal surgeries), surgical assistance systems with (only) computer-assisted navigation according to the current prior art still do not work accurately enough. Due to the intraoperative movement of soft tissues and the lack of hard tissue references, the three-dimensional 3D recording data recorded before the operation (e.g. computed tomography (CT) recording data or magnetic resonance tomography (MRT) recording data) cannot be linked to the patient accurately enough during the surgery. Therefore, the surgeon also relies in particular on endoscopic recordings to find the target surgical area, identify the tissue and perform the treatment accordingly.

[0003] However, endoscopic recordings themselves are limited in terms of information content and do not allow, in particular, path planning or tissue recognition based on preoperative (three-dimensional) 3D recording data. This drawback can lead to unintentional damage to critical anatomical structures, such as nerves or blood vessels, as well as insufficient or excessive resection of target pathological tissue. For example, special mathematical models have been developed for the liver organ in order to correct for deformations during surgery based on surface extraction from endoscopic recordings. However, these techniques are not reliable enough and do not guarantee the required high precision.

[0004] Therefore, the surgeon generally benefits if he can provide not only endoscopic recordings during surgery / in vivo but also 3D recordings before surgery. However, it is currently impossible to connect these two modes of recording to each other without additional complexity while maintaining high accuracy requirements.

[0005] Surgical assistance systems for endoscopic soft tissue navigation are known from the prior art, in which two different representations of two shots are shown on a display next to each other or superimposed. In addition to the first endoscopic in-vivo shot in a first sub-area of ​​the display, a three-dimensional virtual CT shot is statically shown in a second sub-area of ​​the display. However, in surgical assistance systems according to the prior art, the surgeon must manually change the virtual CT image in order to readjust the representation of the CT shot and adapt it to the current view of the endoscope with the in-vivo shot. This process is labor-intensive, demanding, tiring and does not guarantee the necessary precision and therefore safety during the surgical operation.

[0006] Furthermore, a navigation-based endoscope is known from the prior art, in which the endoscope and the patient are tracked / tracked by means of a camera. For this purpose, marking elements / trackers are attached to the endoscope and to the patient, with which the registration is performed. In this way, the CT recording data can be linked to the in vivo recordings of the endoscope. However, such systems and methods require very high additional costs. Summary of the invention

[0007] Therefore, the object of the present disclosure is to avoid or at least reduce the disadvantages of the prior art and in particular to provide a surgical assistance system, a (photographing) representation method, a storage medium and a sterilization chamber, which provide the surgical participants, in particular the (senior) surgeon, with an intuitive and assisted fusion of the information of the in vivo recording of the endoscope and the digital 3D recording data of the patient at a glance, improve the hand-eye coordination and ensure reliable access to the surgical field, in particular to avoid tissue damage caused by negligent and unintentional movements and to minimize the duration of the surgical operation. In particular, the surgical participants should benefit from both modes (i.e., local navigation through the endoscope and global navigation through the recording data of the patient) in a simple manner and can perform the surgical operation intuitively and effortlessly.

[0008] The tasks and objectives of the present disclosure are achieved in a surgical assistance system of this type by the features according to the invention, in a (photographing) representation method of this type by the features according to the invention, in a storage medium of this type by the features according to the invention, and in a sterilization chamber of this type by the features according to the invention.

[0009] The surgical assistance system provides the two (filming) modes during the operation to the surgical participant, such as the surgeon, in an informative and coordinated manner by linking the two (filming) modes to one another and allowing the surgeon to use the in vivo films of the endoscope for local guidance and to use (views of) 3D filming data, in particular pre-operative films, for global guidance and for further information, for example about the tissue.

[0010] In the surgical assistance system, the in-vivo recording of the endoscope is therefore initially registered with the digital 3D recording data of the patient's body, and specifically by a "recording-based" registration, wherein in the in-vivo recording, at least one characteristic landmark / orientation point (in the recording direction) and / or the anatomical orientation is detected and determined. This landmark and / or orientation is also found in the (virtual / digital) 3D recording data. The in-vivo recording can be (geometrically) linked to the 3D recording data via this at least one reference point (landmark) and / or orientation. In particular, a plurality of characteristic landmarks, preferably two characteristic landmarks, particularly preferably three characteristic landmarks, are detected and determined in the in-vivo recording.

[0011] In other words, there are two aspects / spaces in the surgical assistance system: a real first (geometric) space with a real endoscope and the real body of the patient and a virtual second (geometric) space with a virtual endoscope and the virtual body (region) of the patient. The aim is to "connect", i.e. associate, the real space with the virtual space in a simple manner so that the geometric relationship of the recording head, in particular the endoscope, to the body in the real space corresponds to the geometric relationship of the recording head, in particular the endoscope, to the body in the virtual space. It is assumed here that the digital 3D recording data corresponds geometrically at least locally to the real body of the patient and approximately represents a 1:1 imaging in the virtual space.

[0012] If the endoscope is inserted into the patient's body during a surgical operation, either manually or guided by a robot, the endoscope detects the internal region of the body through its camera head (especially optically) and performs an in-vivo capture, such as a two-dimensional or three-dimensional image. As described above, the captured image or the captured anatomical structure can be found not only in the real body but also in the virtual digital body. Now, in order to perform an initial alignment between the real endoscope and the real patient (or the patient's body), it can be said that an alignment between the virtual endoscope or at least one virtual camera head and the patient's virtual body is performed with the aid of in-vivo capture and finally "transferred" to the real space. Then, the real space and the virtual space can be said to be connected to each other and the virtual space can be regarded as an "imaging" of the real space.

[0013] The registration can be performed in particular using the following different registration methods:

[0014] - Point-to-point registration, wherein at least three points are required for unambiguous position identification. In this case, at least three (anatomical) points spaced apart from one another are detected in the in vivo recording as landmarks (these points define a plane). This detection can be performed, for example, automatically and / or by means of manual input. In particular, the control unit can then be adapted to detect or determine three corresponding points / landmarks in the 3D recording data and to perform the registration accordingly;

[0015] - Point and orientation registration, whereby at least one point is required as a marker and orientation. In this case, a unique determination is achieved by reference to the marker and the orientation; in this case, the term "orientation" means, in particular with respect to a reference coordinate system, that three angles are specified for determining the orientation, so that the orientation is unique.

[0016] - Surface equalization, in which a three-dimensionally detected surface (3D surface) acquired in vivo (with a plurality of landmarks approximating the 3D surface) is associated with a corresponding 3D structure (e.g. a segmented (3D) structure) in the 3D acquisition data and thus initially registered. For example, a (partial) surface of the liver in real space can be associated with a segmented 3D structure of the liver in the MRT acquisition data.

[0017] By means of the initial registration using at least one landmark and / or orientation, a relationship between the in-vivo recordings and the 3D recording data (with respect to the at least one recording head) exists at least initially.

[0018] The movement (orientation over time) of the actual endoscope (especially the camera head) in the real space is thus detected by the tracking system. The movement or endoscope movement data may have translations (transformations) and / or rotations, which may be added up to six degrees of freedom. In particular, the transformation or transformation matrix from the operating section (especially the handpiece) of the endoscope to its camera head is known or determinable, so that based on the movement of the operating section, the movement of the camera head can be determined in particular by the control unit, and vice versa. Further preferably, the transformation or transformation matrix from the camera head to the in vivo (endoscope) shooting and / or the transformation from the operating section to the in vivo shooting may also be known or determinable, so that the movement of the in vivo shooting, the movement of the camera head or the operating section, and vice versa, can be determined in particular by the control unit. Therefore, in particular translational movements in three directions and rotational movements around three axes can be provided as endoscope movement data of the control unit. The control unit transfers the endoscope movement data of the real space to the 3D shooting data, in particular to the virtual camera head in the virtual space. It can also be said that an absolute association is performed using the registration and a subsequent relative association is performed using the detection of the movement.

[0019] It can also be said that the endoscope movement data change the transformation matrix between the local coordinate system of the recording head and the local coordinate system of the patient's body (3D recording data) in the virtual space and adapt the view of the 3D recording data accordingly. It should be noted that of the maximum six degrees of freedom detected in the endoscope movement data, in particular only a selected set can also be forwarded as endoscope movement data into the virtual space. A subset of the endoscope movement data is meaningful, for example, when the position of the endoscope changes only very slightly during the operation, but mainly only one orientation changes.

[0020] The surgical assistance system uses the generated association representation to provide the surgeon with local and global navigation modes and improve the surgeon's orientation. The surgeon can "see" the back of the anatomical structure in a natural image / in vivo shot with the help of an association representation output by a surgical monitor, for example. The surgeon can navigate better and more efficiently to reach the desired area and know where the surgeon needs to operate. In addition, safety during surgery is improved and the quality of surgery is improved. Better orientation leads to shorter operating times. In particular, the surgeon no longer has to superimpose cross-sectional images in the head or estimate where the surgeon is exactly. This leads to a reduction in complications and therefore a reduction in additional costs and risks for patients and hospitals.

[0021] In other words, the task is achieved in a surgical assistance system for use in a surgical operation on a patient, wherein the assistance system comprises: a display device, in particular a surgical monitor, for displaying visual content, in particular an image, an endoscope, in particular a video endoscope, with a distal imaging camera, the camera head being configured and suitable for creating in-vivo shots of the patient and providing the in-vivo shots digitally / computer-readable, a data provision unit, in particular a storage unit, the data provision unit being configured and suitable for providing, in particular collating and providing digital 3D shot data of the patient, in particular pre-operative 3D shot data, digitally / computer-readable, and a control unit / computing unit / computer unit, the control unit / computing unit / computer unit being suitable for processing the in-vivo shots of the endoscope and the 3D shot data. Furthermore, the surgical assistance system has a registration unit, which is configured and adapted to detect at least one, in particular a plurality, preferably two, particularly preferably three, in particular predetermined anatomical landmarks and / or anatomical orientations in the in-vivo recording of the endoscope and to determine at least one corresponding anatomical landmark and / or orientation in the provided 3D recording data and to at least initially register the in-vivo recording with the 3D recording data by means of the corresponding anatomical landmark and / or orientation and to register the endoscope, in particular the recording head, relative to the (body) of the patient; and a tracking system, which is configured and adapted to continuously / continuously detect the orientation of the endoscope and / or the movement of the endoscope, in particular the recording head, and to provide it as endoscope movement data to the control unit. The control unit is further adapted to generate an associated representation having (at least one) representation of the in-vivo recording and (at least one) representation of the view of the 3D recording data, in which the in-vivo recording and the view of the 3D recording data are associated with each other with respect to the endoscope, in particular the recording head. In this case, the detected endoscope movement data are transferred to at least one virtual position and / or orientation of a virtual recording head, in particular a virtual endoscope, in the view of the 3D recording data for the associated movement, and the control unit is adapted to visually output the associated representation generated thereby via the representation device. The control unit can, for example, have a processor (CPU) for corresponding processing and control. The data providing unit can also prepare data when changes of the model related to the environment occur, for example, by deformation or separation of the tissue.

[0022] Therefore, in principle, in a surgical assistance system, the present disclosure provides a first (actual) image, i.e., an in-vivo shooting through an endoscope and a second (virtual) shooting in the form of a (predetermined) view of 3D shooting data (such as CT shooting data or MRT shooting data). The two shots are visually displayed to surgical participants, such as surgeons, on a surgical monitor, in particular. Specifically, in particular in a (partial) area of ​​an associated representation and therefore in a (partial) area of ​​the surgical monitor, the real-time actual endoscopic video shooting is represented as an in-vivo shooting, while in another (partial) area of ​​the surgical monitor, a (selected) associated view of 3D shooting data is represented. In particular, the two representations are arranged side by side. Alternatively, two representations (augmented reality) can also be shown superimposed. Thus, the surgeon can perform global navigation to reach the target surgical area by viewing the 3D shooting data, while the in-vivo shooting of the endoscope provides a detailed image of the actual current position of the soft tissue. Because the two shots are associated with each other, the surgeon does not have to perform manual adaptation or virtual rotation or shooting displacement between each other. The association of the two shots is especially automatically performed by a surgical assistance device.

[0023] In other words, the present disclosure relates to an intraoperative surgical assistance system, which in particular shows an endoscopic image (in vivo recording) of a patient and a 3D image (3D recording data) in particular before the operation side by side. The two images (recordings) are associated and relate to the current position and / or orientation of the recording head, in particular the endoscope. Preferably, when the endoscope is introduced into the operating area, the surgeon selects, in particular predefined anatomical landmarks and / or orientations based on the endoscopic image (in vivo recording) and associates them with corresponding anatomical landmarks and / or orientations in the 3D data set (3D recording data), wherein this step is called registration. Different registration methods are possible:

[0024] -Point to point, at least 3 points

[0025] -Points and Orientation

[0026] Surface equalization, associating a 3D surface from an endoscopic image (in vivo recording) with a structure in the 3D image (3D recording data), in particular a corresponding segment.

[0027] Although registration with the aid of landmarks has the advantage that the distance to the object, for example the liver, is also recorded and the absolute dimensions of the in vivo recordings and the 3D recording data can thus be "matched", it is sufficient if only the object is correctly rotated (merely angular adaptation). This is especially useful in a segmented representation, since the 3D recording data may also include only one organ.

[0028] The term 3D defines that the patient's recording data are presented spatially, ie three-dimensionally. The patient's body or at least a partial area of ​​the body with a spatial extension can be digitally presented as recording data in a three-dimensional space with, for example, a Cartesian coordinate system (X, Y, Z).

[0029] The term "position" defines both location and orientation. The location can be expressed, for example, in a Cartesian coordinate system using (X, Y, Z) and the orientation can be expressed using three angles (α, β, γ).

[0030] The term "view" describes, in particular in 3D recording data, in particular at a (predeterminable) point, a (predeterminable) viewing direction / viewing angle with in particular a (predeterminable) rotation similar to a selectable view in a CAD model. The orientation of the viewing point is defined to a certain extent.

[0031] The term "real" defines the "aspect" of the physical (actual) endoscope and the patient on whom the actual operation is performed, whereas "virtual" defines the "aspect" of the digital virtual endoscope and the captured virtual patient data, which is present in a computer-readable manner and can be digitally manipulated in a simple manner in order to show individual information.

[0032] Advantageous embodiments are described below.

[0033] In a preferred embodiment, the tracking system can be designed in particular as an endoscope-internal tracking system, and the endoscope, in particular the operating section and / or the recording head, can have internal motion sensors, preferably acceleration sensors and / or rotational speed sensors, in particular inertial measurement units (IMUs), for detecting endoscope movement data. The surgical assistance system thus does not require additional external systems or measuring devices, but rather only one spatially small sensor integrated in the endoscope is sufficient. The internal motion sensor can detect the movement of the endoscope, in particular the recording head, so to speak, autonomously and provide it digitally as endoscope movement data to the control unit. In other words, in order to determine the movement of the endoscope, in particular the recording head, the endoscope can have at least one motion sensor, preferably an internal sensor, such as in particular an IMU.

[0034] According to further embodiments, as an alternative or in addition to the internal sensors of the endoscope for the tracking system, external sensors or reference marks detected by the camera system can also be provided. In particular, the tracking system can have a camera system that (continuously) spatially detects the endoscope, in particular the operating section, determines the position of the endoscope and provides it to the control unit as endoscope movement data. From this, the position of the recording head can be inferred from the predetermined geometric relationship. In other words, the tracking system can also have an external camera, in particular a 3D camera, which detects the operating area and, in the case of a rigid endoscope, continuously detects the position of the operating section and determines the position and thus the movement of the endoscope, in particular the recording head, in time based on the detected operating section.

[0035] According to a preferred embodiment, in order to form a tracking system, the control unit can be adapted to determine the movement of the endoscope by image analysis / data evaluation of the moving anatomical structures in the in-vivo shooting and provide it as endoscope movement data. The movement can be inferred by changing the image by a suitable image analysis method. This can be compared, for example, with the video shooting of the front camera of the car, wherein the movement is detected according to the lane markings or, for example, the moving horizontal line. Therefore, the tracking system in the form of an adapted control unit calculates the movement, in particular the movement of the shooting head, by means of the in-vivo shooting that changes over time. For this purpose, in particular, the optical parameters of the in-vivo shooting of the endoscope and / or the calibration shooting head, in particular the endoscopic camera of the shooting head, are known. In other words, the movement of the endoscope can also be identified by the endoscopic image / in-vivo shooting, by analyzing in which direction and in particular at which speed the (anatomical) structure in the image moves. The advantage of this image analysis is, for example, that the endoscope movement data can be obtained even without a motion sensor.

[0036] In particular, the tracking system can have a motion sensor arranged in the endoscope, in particular an acceleration sensor and / or a rotation speed sensor, particularly preferably an inertial measurement unit, and the control unit can be adapted to detect endoscope movement data by image analysis, wherein a combined value of the two results, in particular an adapted average value or a weighted value of the endoscope movement data, is calculated from at least two results of the endoscope movement data as the endoscope movement data (as a final result). Each individual element / component of the rotation and translation (six elements / degrees of freedom) can be weighted. For example, an inertial measurement unit (IMU) can determine the rotation more accurately than the translation. In particular, the rotation data are provided from at least two results with respect to the inertial measurement unit (IMU) and the translation data of the endoscope movement data are provided by image analysis. In particular, weighting can be performed so that in the result of the IMU, the rotation component is weighted with more than 80%, in particular 100%, and the translation component is weighted with less than 20%, in particular 0%, wherein in the result of the image analysis, the rotation component is weighted complementarily with less than 20%, in particular 0%, and the translation component is weighted with more than 80%, in particular 100%. The average value can be, for example, an arithmetic mean, a geometric mean or a square mean, which is formed in particular in each individual component. In this way, (at least) two different "sensor systems" are connected to each other. By using a combination for redundant detection, in particular mean value formation or weighting, the inaccuracy errors caused by the system can be further minimized and the accuracy can be increased. In particular, internal sensors and image analysis can be combined to determine the movement of the endoscope. As a result, the surgeon can track the actual position of the endoscope, in particular the actual position of the recording head, in the 3D image / 3D recording data, and thus identify critical structures or track the path from the preoperative planning.

[0037] According to a further preferred embodiment, the control unit can be adapted to positionally correctly identify at least one (stored) medical instrument in the in vivo recording, in particular based on a database with geometrically predefined structures of medical instruments stored in the storage unit, and to positionally correctly, in particular associatedly, display the identified medical instrument as a virtual medical instrument in the view of the 3D recording data, so that the medical instrument is displayed in both recordings. The term "positionally correct" defines a combination of positionally correct and orientationally correct, i.e. in a consistent position and orientation. In other words, in particular, at least one instrument can also be displayed in both recordings, so that the surgeon can visualize not only the position and / or orientation of the endoscope, for example, but also the position and / or orientation of the medical instrument in the 3D recording data. For this purpose, the medical instrument is detected in the in vivo recording and virtually superimposed on its (actual) position in the 3D recording data (3D image). A prerequisite for this is that the instrument is visible in the endoscopic image and its geometry is known.

[0038] In particular, data about the recording head, in particular the camera type, and / or data about the endoscope, in particular the type of integrated endoscope and / or "virtual endoscope" for the camera, can also be stored in the database stored in the storage unit. For example, the stored geometric data of the recording head can be correspondingly inserted into the associated representation.

[0039] The control unit can be adapted in particular to determine data stored in the storage unit about optical parameters of the endoscope, in particular about the recording head, as a function of the position and / or orientation of the endoscope, in particular the recording head, and preferably determine a viewing cone of the recording head as a function of the distance to the target anatomical structure, and to insert together a virtual viewing cone of the virtual recording head in the associated representation in the representation of the 3D recording data. In this way, the user sees which area he is viewing on the 3D recording data. This is advantageous, for example, when the scale is not the same and, for example, the 3D recording data is shown reduced. The term "viewing cone" describes a three-dimensional volume, which is formed by the field of view defined by the optics and the corresponding distance to the front side.

[0040] Preferably, the registration unit can perform a re-registration in addition to the initial registration at at least one further, in particular predetermined, point in time in order to further improve the accuracy of the association. Since the endoscope movement data are relative data (without external absolute reference) based on the registered (absolute) starting position of the recording head, a re-registration (as an absolute association) can further improve the accuracy. In particular, a subsequent registration can be performed after a predetermined time or after a predetermined sum of movements. In particular, a plurality of repeated registrations can also be performed as a "reset" after a respectively predetermined time, a predetermined path or a predetermined sum of movements. Thus, a re-registration between the in vivo recording of the endoscope and the 3D recording data (3D model / 3D image) can be performed again at any time in order to further improve the accuracy in the event of soft tissue movement.

[0041] In a preferred embodiment, the endoscope can be designed as a rigid video endoscope. In the case of a rigid endoscope, there is a constant geometric relationship between the operating section and the recording head. For example, a motion sensor can be provided in the handpiece that registers the movements in the handpiece, and the control unit can be adapted to infer the movement of the recording head from the detected movement data and the geometric relationship.

[0042] Preferably, the control unit can be adapted to display, in the representation of the view of the 3D recording data and / or the associated representation in the representation of the in vivo recording, in addition to the actual and / or virtual endoscope, also the pre-planned medical instruments and / or implants stored in the storage unit, in particular the pre-planned trocars, in a correctly positioned (virtual) manner. Thus, in particular, the pre-planned trocars can be displayed in the 3D recording data or in the view of the 3D recording data relative to the current position of the endoscope.

[0043] According to a further preferred embodiment, the control unit may be adapted to show the planned path / trajectory in a representation of a view of the 3D recording data and / or in an associated representation of a representation of the in vivo recording, in order to indicate the path towards the operating area to the surgeon. This further improves the surgeon's orientation. The trajectory can be illustrated in particular by an inserted arrow with an arrow tip in the direction of the planned path, which is adapted to the movement of the endoscope and always points in the direction of the path. Alternatively or additionally, the planned path can also be inserted into the 3D recording data, for example in the form of a three-dimensional line along the path. Alternatively or additionally, annotation points planned before surgery can also be inserted into the 3D recording data during surgery, in order to, for example, approach a target area or identify critical structures.

[0044] Preferably, the control unit can be adapted to establish the view of the 3D recording data as a 3D scene / 3D reconstruction and / or a two-dimensional cross section relative to the image coordinate system of the endoscope and / or along the recording axis of the endoscope, especially the recording head and / or create it as a virtual endoscopic image. The 3D recording data can therefore be, for example, a rendered 3D model with or without segmented anatomical structures, which can be freely moved in the virtual space and can be represented by different positions and views from different viewing angles with different transparency of different anatomical structures, similar to a CAD model. The control unit then creates the selected view of the 3D scene. For example, a large number of anatomical structures can also be masked in the virtual space in order to obtain a better view of the relevant structures. Alternatively or additionally, the 3D recording data can also exist as a (segmented) coordinate system relative to the endoscope, especially the image coordinate system of the endoscope and / or a two-dimensional cross section along the axis (for example in MRT or CT recording data). The image coordinate system corresponds to the (local) coordinate system recorded in the body. Alternatively, a two-dimensional cross section relative to the local coordinate system of the recording head can also be created. The image coordinate system captured in vivo can be back-calculated by transforming to the local coordinate system of the shooting head and further by transforming to the local coordinate system of the handpiece. Then, the control unit can create a view of the 3D shooting data along the axis of the shooting direction or the axis of the shooting head, especially the endoscope, corresponding to the position of the shooting head, especially the endoscope, which is similar to the ultrasound image. This can, for example, realize the recognition of the anatomical structure behind the image captured by the endoscope. As an alternative or additional scheme to the cross section along the shooting axis, a diagonal or individual cross-sectional view depending on, for example, the anatomical structure or a predefined cross-sectional view can also be created. Preferably, an X section and / or a Y section and / or a Z section observed relative to the reference coordinate system of the endoscope (with XYZ axes) can be created. In the case of a Z section (i.e., a section parallel to the field of view of the endoscope and perpendicular to the local Z axis of the reference coordinate system of the endoscope), the section can be moved accordingly, especially by moving the endoscope forward and backward. The range of movement toward the object can be limited, for example, by the working distance. In the case of a rigid endoscope, the Z axis is in particular coaxial with its longitudinal axis, and the Z section is accordingly arranged orthogonal to the Z axis and therefore parallel to the field of view. Alternatively or additionally, the control unit can create a view of the 3D recording data as a virtual endoscopic image / endoscopic perspective, which simulates the actual recording of the recording head. In other words, a view of the 3D recording data is created, wherein the virtual recording head of the virtual endoscope has the same orientation (i.e. position and orientation) as the actual recording head, and a virtual recording is created with optical parameters that are exactly the same as the optical parameters of the actual endoscope, so that the actual in-vivo recording is represented as a virtual in-vivo recording in virtual space (1:1 representation of actual recording - virtual recording). Here, the surgeon "sees" in exactly "the same direction and at the same position" not only in the actual representation but also in the virtual representation.In other words, 3D recording data, in particular views of 3D recording data (3D images), can be presented as 3D scenes / 3D renderings of anatomical structures with or without segmentation. Alternatively or additionally, the 3D images can be presented as 2D cross-sections along the axis of the endoscope, similar to ultrasound images. The 3D images can alternatively or additionally be presented as virtual endoscopic images, which simulate the line of sight of the endoscope similar to a fly-through model.

[0045] Preferably, the surgical assistance system has a view selection unit in which a predefined number of different views are stored. In particular, a bird's eye view, an endoscopic view (along the recording direction), a side view and a bottom view are stored. When a view is selected, in particular by the surgeon, the control unit adapts the view of the 3D recording data accordingly to the selected view. In other words, the surgical assistance system can have preconfigured settings that allow the surgeon to obtain different views of the 3D recording data (representation of the 3D image) for the area to be examined, such as a bird's eye view, an endoscopic view, a side view, a bottom view, thereby adopting different perspectives and taking into account the individually available and different contents of the surgeon's decision.

[0046] In particular, the control unit performs the functions of the tracking system.

[0047] According to an embodiment, the surgical assistance system can also have a robot, and the endoscope, in particular the operating section of the endoscope, can be robot-guided, in particular by means of a robot arm, and the tracking system determines the position and / or orientation and / or movement of the endoscope, in particular the camera head, by means of the associated position and / or orientation and / or movement of the robot, in particular the robot arm. In the case of robot-assisted surgery in which the endoscope is guided by the robot, the position and / or orientation can preferably be calculated directly by the axis system or the control command using the corresponding transformation and the movement of the endoscope, in particular the camera head, can be calculated in time. The detection of the position and / or orientation and / or movement is therefore not carried out in the endoscope itself, but in the robot. In other words, in a robot system, the movement and / or position and / or orientation of the endoscope is derived by the robot, in particular the robot arm.

[0048] With respect to a recording representation method for associated representation of two different recordings, the tasks and objectives of the present disclosure are achieved by the following steps: reading digital 3D recording data of a patient, in particular pre-operative 3D recording data; creating an in-vivo recording by means of an endoscope; detecting at least one landmark and / or orientation in the in-vivo recording; determining a corresponding landmark and / or orientation in the 3D recording data; registering the 3D recording data with the in-vivo recording via at least one detected landmark and / or orientation; generating at least one associated representation having a representation of the in-vivo recording and a representation of a view of the 3D recording data, and preferably outputting the associated representation by means of the representation device; continuously detecting the position and / or orientation and / or movement of an endoscope, in particular a recording head; transferring the detected movement of the endoscope, in particular the recording head, to at least one virtual position and / or orientation of a virtual recording head, in particular a virtual endoscope, in the view of the 3D recording data, and generating an updated associated representation having the in-vivo recording and an updated view of the 3D recording data, and outputting the associated representation by means of the representation device.

[0049] According to an embodiment independently claimed for protection if necessary, the shooting representation method may preferably have the following steps: detecting 3D shooting (data), in particular pre-operative 3D shooting (data); preferably segmenting 3D shooting (data); providing 3D shooting data, in particular segmented shooting, in a surgical assistance system; preferably operating an endoscope and / or creating an in-vivo shooting; aiming and detecting anatomical landmarks in the natural endoscopic in-vivo shooting; determining and detecting corresponding landmarks in the 3D shooting data; aligning two shots (in-vivo shooting and 3D shooting data) with respect to the detected landmarks; (detecting and) determining the movement of the shooting head, in particular the endoscope, in particular by using at least one endoscope internal sensor and / or image analysis / shooting analysis of the in-vivo shooting of the endoscope; transferring the acquired movement to the 3D shooting; and in particular outputting the two shots in parallel and / or superimposed association.

[0050] Preferably, the representation method has a further step of realignment.

[0051] According to a further preferred embodiment, the representation method can have the steps of: detecting the medical instrument and its position in the in vivo recording; positionally correctly virtually superimposing the detected instrument in the 3D recording data;

[0052] Preferably, the representation method can include the step of representing a virtual endoscope, in particular a virtual recording head, in a positionally correct manner in the 3D recording data.

[0053] According to a further preferred embodiment, the representation method can have the step of inserting a preplanned medical instrument, in particular a preplanned trocar, into the in-vivo recordings and / or the 3D recording data.

[0054] In particular, the representation method can have the steps of: detecting an input of a view selection, in particular a selection from the set of a bird's eye view, an endoscopic view, a side view and a bottom view; applying the view selection to a view of the 3D recording data in order to represent the selected view in an associated representation.

[0055] Preferably, the representation method may also include the steps of determining a deformation of the actual anatomical structure, in particular a deformation relative to the initial anatomical structure, based on the detected landmarks and the determined corresponding landmarks, and transferring the determined deformation to the virtual anatomical structure in the 3D recording data in order to adapt the 3D recording data accordingly. In other words, the deformation can be determined based on the corresponding landmarks and their relative changes, in particular to the initial landmarks, and then the deformation can be transferred to the 3D recording data accordingly in order to correct the 3D recording data. The method has the advantage, firstly, in registration by means of surface registration, in which not only a rigid association with the 3D recording data is now carried out, but also the segmented object, for example the liver, is deformed accordingly in the 3D recording data so that the 3D recording data is adapted to the actual 3D structure recorded by the endoscope. These steps may be implemented in particular by a control unit of the auxiliary system.

[0056] It should be noted here that the features of the representation method of the present disclosure can be transferred to the surgical assistance system of the present disclosure, and vice versa.

[0057] With respect to a computer-readable storage medium, the tasks and objects of the present disclosure are solved in the following manner: the computer-readable storage medium includes instructions, which, when executed by a computer, cause the computer to perform the method steps of the photographic representation method according to the present disclosure.

[0058] With regard to a sterile room of this type, the object of the disclosure is achieved in that the medical sterile room has a surgical assistance system according to the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present disclosure will be explained below according to preferred embodiments with the aid of the accompanying drawings. It is shown:

[0060] Figure 1 A schematic diagram showing a surgical assistance system according to a first preferred embodiment,

[0061] Figure 2 A schematic partial view of a rigid endoscope of a surgical assistance system of a further preferred embodiment with associated in vivo recordings, the movements of the rigid endoscope being transmitted to a virtual endoscope with a view of the 3D recording data;

[0062] Figure 3 A schematic diagram showing an imaging-based registration of a surgical assistance system;

[0063] Figure 4 A schematic stereoscopic view showing a video endoscope used in a surgical assistance system of a preferred embodiment;

[0064] Figure 5 is a schematic diagram of different perspectives for a view of 3D captured data; and

[0065] Figure 6 A flowchart showing a photographic display method according to a preferred embodiment is shown.

[0066] Among them: 1- surgical auxiliary system; 2- endoscope; 2'- virtual endoscope; 4- handle; 6- camera head; 6'- virtual camera head; 8- surgical monitor; 10- storage unit; 12- control unit; 14- registration unit; 15- mark; 15'- virtual mark; 16- tracking system; 18- IMU; 20- acceleration sensor; 22- speed sensor; 24- medical instrument; 24'- virtual medical instrument; 26- field of view of endoscope; 26'- virtual field of view of endoscope; 10 0-operating room; S1-step of reading recording data; S2-step of creating or detecting an in-vivo recording; S3-step of detecting at least one landmark in the in-vivo recording; S4-step of determining a corresponding landmark in the 3D recording data; S5-step of registering by means of the corresponding landmark; S6-step of generating an associated representation; S7-step of detecting a movement of the endoscope; S8-step of associating by transferring the detected movement; S9-step of generating and outputting an updated associated representation; B1-condition for re-registration?; B2-condition for ending the method?; P-patient; IA-in-vivo recording; 3DA-3D recording data; K-associated representation.

[0067] The accompanying drawings are merely schematic in nature and are only useful for understanding the present disclosure. Identical elements are provided with identical reference numerals. The features of the various embodiments may be substituted for one another. DETAILED DESCRIPTION

[0068] Figure 1A schematic diagram of a surgical assistance system 1 (hereinafter referred to as the assistance system) of a first preferred embodiment of use is shown. The assistance system 1 is used in a medical sterilization room in the form of an operating room 100 of a preferred embodiment in order to support surgical participants, in particular surgeons, by suitable visualization when performing surgical operations on a patient P (here only schematically shown). Minimally invasive surgery is performed on the patient in a sterile surgical area in the middle. A rigid video endoscope 2 (hereinafter referred to as the endoscope only) with a handpiece / handle 4 and a spaced front imaging camera head 6 is located inside the patient's body on the endoscope handle side for this purpose. For imaging, the camera head has an endoscope video camera (not shown), which creates a two-dimensional in-vivo shot IA of the patient P's body interior as a time-current (live video) shot and provides it digitally in the direction of the longitudinal axis of the endoscope 2 with the help of a CMOS sensor on the end side. Then, via a display device in the form of a surgical monitor 8, visual content, such as a display of the in-vivo shot IA, can be reproduced for the surgical participants. It should be noted that the endoscope 2 can also be optionally guided by a robot arm on its operating section.

[0069] The surgical assistance system 1 also has a data provision unit in the form of a storage unit 10 in which digital preoperative 3D recording data 3DA of the patient P to be treated are stored digitally / computer-readable. For example, (segmented) MRT recording data or CT recording data can be stored as 3D recording data 3DA, which virtually or digitally image the body of the patient P or at least a part of the body.

[0070] For detection, processing and calculation as well as control, the assistance system 1 has a control unit 12 which is suitable for processing the in-vivo recordings IA and 3D recording data of the endoscope 2 and for correspondingly actuating the surgical monitor 8 .

[0071] The surgical assistance system also has a registration unit 14 which is constructed as a specially adapted control unit 12 in this embodiment. Specifically, the control unit 12 is suitable for detecting anatomical landmarks 15 in the in-vivo shooting IA of the endoscope 2 and determining corresponding anatomical landmarks 15' in the provided 3D shooting data. In the in-vivo shooting IA of the endoscope 2 inserted into the body of the patient P and shooting the target anatomical structure as a two-dimensional image, in this embodiment, the surgeon selects three predefined characteristic (actual) landmarks 15. These three landmarks 15 in the real space also appear in the virtual space as three (virtual) landmarks 15'. Through a total of two times three corresponding anatomical landmarks 15, 15', the in-vivo shooting IA is initially associated with the 3D shooting data, that is, registered. Therefore, the endoscope 2, especially the shooting head 6, is also registered relative to the patient P. That is to say, a geometric relationship (3D shooting data) is established between the virtual shooting head 6' and the virtual body of the patient P, which corresponds to the actual geometric relationship between the actual shooting head 6 and the actual body of the patient P. In the virtual space, the control unit 12 calculates the corresponding transformation matrix and transfers it to the real space. The orientation of the recording head 6 of the real endoscope 2 relative to the body of the patient P is thus determined. In other words, the (real) in-vivo recording IA is first compared with the (virtual or digital) 3D recording data 3DA with respect to at least one characteristic mark 15, 15', which is initially linked, that is, registered, in both recordings in order to register or determine the orientation of the (real) endoscope 2, in particular the recording head 6, relative to the patient P.

[0072] In particular, the control unit 12 can be suitable for determining such a view of the 3D recording data 3DA when the optical parameters of the recording head 6 or the distance between the recording head 6 and the anatomical landmarks are known, wherein the three landmarks have essentially the same distance from each other in the 3D recording data 3DA as in the in vivo recording IA.

[0073] Furthermore, the surgical assistance system 1 has a tracking system 14 which continuously detects the movements of the endoscope 2 and the recording head 6 and provides them as endoscope movement data to the control unit 12. In this embodiment, a combination of endoscope-internal sensors and an image analysis performed by the control unit 12 is used.

[0074] Specifically, the handle 4 of the rigid endoscope 2 has an inertial measurement unit (IMU) 18, which has a plurality of inertial sensors, here an acceleration sensor 20 and a rotation speed sensor 21 (a three-axis acceleration sensor and a three-axis gyroscope), for constructing an inertial navigation system (see also Figure 4). The individual accelerations in three directions produce a total acceleration (vector) in one direction in real space. The detected rotational speeds around the three axes can be imaged as a rotation matrix, for example. Therefore, the movement of the endoscope in real space is continuously detected (due to technical limitations, approximately at discrete time intervals with as small a time interval as possible) and forwarded to the control unit 12 as endoscope movement data with translation and rotation. Through the fixed geometric relationship between the handle 4 and the shooting head 6, the IMU 18 can detect not only the movement of the handle 4, but also the movement of the shooting head 6.

[0075] Furthermore, in the surgical assistance system 1, the control unit 12 is adapted to perform image analysis with the aid of the in-vivo recording IA. The control unit 12 performs data processing in such a way that it detects changes in the anatomical structure over time and thereby determines the movement of the structure (in particular the change in direction) and determines the movement of the recording head 6 and thereby the movement of the endoscope 2 based on the change. Thus, in addition to the sensor-type endoscope movement data, there are also image-analyzed endoscope movement data, which can be linked to one another in order to minimize measurement errors and increase accuracy. In this case, an average value of the two endoscope movement data is formed and provided as a result to the control unit 12.

[0076] Because the control unit 12 is further suitable for generating an associated representation K, which has a parallel representation of the representation of the in-vivo recording IA and the representation of the view of the 3D recording data 3DA, wherein the representation of the in-vivo recording IA and the representation of the 3D recording data 3DA are associated with each other with respect to the endoscope 2, in particular the recording head 6. For this purpose, the detected endoscope movement data are transferred to at least one virtual position and / or orientation of the virtual recording head 6', in particular the virtual endoscope 2', in the view of the 3D recording data 3DA for the associated movement. In particular, the virtual recording head 4' moves together in the same way as the actual recording head 4. The view of the 3D recording data is adapted accordingly. Finally, the control unit 12 is suitable for visually outputting the associated representation K generated in this way via the surgical monitor 8. The associated representation K is a parallel representation of the endoscope in-vivo recording IA and the pre-operative 3D recording data, wherein the two recordings are associated with respect to the movement of the endoscope.

[0077] The view of 3D shooting data realizes global orientation, while the endoscope in vivo shooting realizes accurate spatial orientation. The surgical auxiliary system 1 disclosed in the present invention is suitable for showing two modes in parallel and in an associated manner. Therefore, although the endoscope in vivo shooting and 3D shooting data are not always completely consistent due to the deformation of the tissue during surgery, the surgeon can better perform the operation and directly use the 3D shooting data with preoperative information in particular. For precise surgery, the surgeon can also rely on the in vivo shooting of the endoscope. The associated 3D shooting data or 3D image enables the surgeon to avoid critical structural damage, better find its surgical target, and accurately and effectively implement the preoperative planning. Through the global guidance with 3D shooting data, the auxiliary system 1 can also shorten the learning curve when using the in vivo shooting of the endoscope. In particular, the orientation of the position and / or orientation of the endoscope, especially the shooting head, relative to the 3D shooting data or relative to the segmented 3D model leads to the surgeon's improved orientation in the area of ​​interest.

[0078] After the initial alignment, it can be said that in the surgical assistance system 1 of the present invention, the transformation matrix can be changed in the virtual space to the same extent as the transformation matrix in the real space, so that the endoscope 2', in particular the shooting head 6' in the virtual space can be guided together with the endoscope 2 or the shooting head 6 in the real space and the view of the 3D shooting data 3DA can be adapted (associated) accordingly, in particular moved together.

[0079] The initial registration (as an absolute association) by means of the registration unit 14 can be performed in particular as follows. A local camera head coordinate system can be assigned to the camera head 6 of the endoscope 2 in real space. The local coordinate system can also be assigned to the body of the patient P. For the initial registration, a transformation matrix between the two local coordinate systems must be calculated and provided so that the initial position (position and orientation) of the camera head and thus the initial position (position and orientation) of the endoscope relative to the body of the patient P can be determined by the transformation matrix. In virtual space, the local camera head coordinate system can also be assigned to the virtual camera head 6 'and the local coordinate system can be assigned to the "virtual body" of the patient. It can also be said that the virtual body or at least one partial area, such as an organ, is consistently assigned to the real body. A basic association between the real space and the virtual space is thus established. This is obvious because, in particular, preoperative 3D recording data, such as CT recording data or MRT recording data, which record the entire body or at least one partial area of ​​the body, of course, are 1:1 images of the real body in space. The actual in-vivo recording IA of the anatomical structure with at least one detected landmark is also found in the virtual 3D recording data and a registration can be achieved by comparison, in particular between the virtual recording head 6' and the virtual body. This results in an initial transformation matrix in the virtual space, which can be "transferred" to the real space in order to carry out an initial registration. After the registration in both spaces, at least the geometrical position of the recording head is thus associated with the patient's body and thus also the in-vivo recording is associated with the 3D recording data.

[0080] Thus, with the aid of the disclosed surgical assistance system 1, in vivo recordings IA can be linked to views of 3D recording data 3DA and output via the surgical monitor 8 with only very minor hardware changes to existing endoscopes, namely by supplementing the (standard) endoscope with an internal (movement) sensor 18 and / or by detecting the movement of the endoscope 2 via an external camera, such as a 3D stereo camera, and by correspondingly adapting the control unit, which adaptation can be achieved in particular by a correspondingly stored program, without the need for additional space-intensive and error-prone hardware devices. The assistance system 1 improves the surgeon's orientation and thus also the safety of the patient P.

[0081] The control unit 12 is also suitable for identifying such medical instruments 24 with the aid of a database with geometrically predefined structures of medical instruments 24 in the in vivo recording IA stored in the storage unit 10, and for detecting the associated position in real space. Such detected medical instruments 24 are then virtually inserted or virtually superimposed in their detected position in the 3D recording data 3DA with the aid of their stored geometric structures as virtual medical instruments 24' in the correct position, so that the medical instruments 24, 24' are shown in both representations of the recordings IA, 3DA. As a result, the orientation can be further improved.

[0082] In addition to identifying the medical instruments 24 in real space, the control unit 12 is also suitable for displaying pre-planned virtual medical instruments 24', in particular pre-planned trocars, in a view of 3D recording data (3DA) in virtual space (inserted or superimposed). These pre-planned virtual medical instruments 24', including their position relative to the (virtual) body of the patient, are also stored in the storage unit 10.

[0083] Furthermore, a planned path to the operating area is stored in the memory unit 10 , which can be correspondingly inserted into the 3D recording data, for example as an arrow or as a three-dimensional path line, in order to indicate the route to the surgeon.

[0084] Figure 2 A schematic partial view of an assistance system 1 of a further preferred embodiment is shown, with a comparison of real space and virtual space to illustrate the mode of operation. Figure 1 The assistance systems 1 differ primarily in the selected view of the 3D recording data 3DA. Figure 2 The real space is depicted in the left partial area and the virtual space is depicted in the right partial area. The real endoscope 2 detects the in-vivo shots IA at the end side through its shooting head 6, which are used for local navigation for the surgeon. The corresponding virtual endoscope 2' is shown with its virtual shooting head 6' in the CT-shot segment (as 3D shooting data 3DA) at its current position using an inserted virtual field of view 26'. That is, for the associated representation K, a CT-shot segment is used for the view of the 3D shooting data 3DA, which corresponds to the position of the shooting head 6 in the upper front direction. In the view of the 3D shooting data, the shooting head 6' and its orientation are then schematically inserted virtually in the correct orientation. The surgeon can therefore see his global position in the body of the patient P in a simple and intuitive way at a glance in the view of the 3D shooting data 3DA.

[0085] If now in Figure 2 In the registration (if the Figure 1If the endoscope 2 is subsequently moved (see the movement arrow in the real space), the detected movement is transferred to the virtual endoscope 2' with the virtual recording head 6' (see the movement arrow in the virtual space). As a result, on the one hand, the shown orientation of the virtual recording head 6' is changed and, in addition, when the recording head 6 moves in the upper front direction, the segment of the CT recording adapted to this position is displayed. Moreover, when the endoscope 2 is rotated about its longitudinal axis, this rotation is also transferred to the view of the 3D recording data 3DA. For example, in the simplest example, when the endoscope 2 is oriented exactly in the upper front direction and the view of the 3D recording data 3DA is also oriented in this direction, the rotation is transferred 1:1 to the view. If, for example, the endoscope 2 is rotated 90° in the clockwise direction, the view of the 3D recording data 3DA is rotated 90° in the clockwise direction. The in-vivo recording IA is thus linked by the assistance system 1 to a (selectable and variable) view of the 3D recording data 3DA and the movement of the endoscope 2 is transferred to the view of the 3D recording data 3DA and the latter is adapted accordingly.

[0086] For example, two representations of in vivo recordings can be included in an associated representation. Here, the first representation can be a (split screen) real-time representation in fluorescence mode, while the second representation shows a "normal" endoscopic image. Thus, multiple systems and modes and thus multiple real-time representations of in vivo recordings are possible. Likewise, multiple representations of different viewing angles of the same 3D recording data can be shown in an associated representation.

[0087] Figure 3 A preferred embodiment of an assistance system 1 with registration of an in-vivo recording IA with 3D recording data 3DA via three points is schematically shown. In the simplest form, the surgeon selects three characteristic landmarks 15 spaced apart from one another in the in-vivo recording IA. These three landmarks 15 are again visible in the 3D recording data 3DA as corresponding virtual landmarks 15'. Either the control unit 12 can be adapted to automatically determine the landmarks 15', or the surgeon can manually determine the three virtual landmarks 15' in the 3D recording data. In this way, the in-vivo recording IA is registered with the 3D recording data 3DA, and the endoscope 2 is registered with the patient P.

[0088] Figure 4 A detailed perspective partial view shows a rigid endoscope 2 (with a rigid handle) according to an embodiment, which can be used in Figures 1 to 3The endoscope 2 has a handle 4 and a distal shooting head 6, which optically detects the in vivo shooting IA (shown only schematically here) in the shooting direction and digitally provides it to the control unit 12. In the handle 4, a motion sensor inside the endoscope is provided in the form of an inertial measurement unit 18 for constructing an inertial navigation system, which detects the movement of the endoscope 2. By means of defined geometric parameters between the position of the internal sensor 18 and the tip of the endoscope 2 with the shooting head 6, the movement of the shooting head 6 can also be calculated by the control unit 12. This movement is then transferred accordingly to the 3D shooting data 3DA in the virtual space. Figure 4 . The first local coordinate system is the coordinate system of the handpiece, the second local coordinate system is the coordinate system of the recording head, and the third local coordinate system is the image coordinate system recorded in the body. All three coordinate systems can be connected to one another by suitable (serial) transformations, so that (especially using corresponding (geometric) data of the endoscope) one coordinate system can be derived from another coordinate system.

[0089] Figure 5 Shows exemplary different views of 3D recording data which can be viewed with the aid of Figure 1 The view selection unit of the auxiliary system 1 in the rigid endoscope 2 is selected by the surgeon. This representation of the view of the 3D recording data 3DA is then reflected in the associated representation K to provide the surgeon with flexible adaptation and improved orientation. Therefore, if the surgeon can select a side view, a front view and a top view with viewing directions perpendicular to each other, the surgeon can approximately analyze whether there is a misalignment between the longitudinal axis of the rigid endoscope 2 and the target surgical field.

[0090] Figure 6 The flowchart shows a method for displaying images according to a preferred embodiment of the disclosure for displaying two different images in a centrally linked manner, in particular in a surgical assistance system 1. It should be noted that the sequence of the steps can also be changed.

[0091] In a first step S1 , digital 3D recording data 3DA, in particular preoperative 3D recording data, are read out of a patient P. For example, the read out 3D recording data, such as a CT recording of the entire body of the patient P, can be stored in a storage unit with corresponding segmentation.

[0092] In the next step S2, an in-vivo recording IA is created or recorded by the endoscope 2, here by the recording head. It should be noted that the order of steps S1 and S2 is not important, so that the in-vivo recording IA can also be performed first and then read out, or both steps S1 and S2 can be performed simultaneously.

[0093] In a subsequent step S3, a marker 15, in particular three markers 15, is targeted and detected in the in-vivo recording IA. After the marker 15 has been detected, in a step S4, the corresponding marker 15' is 'determined' in the 3D recording data 3DA. In a step S5, the 3D recording data 3DA are registered with the in-vivo recording IA via the detected markers 15, 15'. Steps S3 to S5 thus together form a registration step.

[0094] In step S6, a linked representation K having a representation of the in-vivo recording IA and a linked representation of the view of the 3D recording data 3DA is generated and output, for example, by a representation device in the form of a surgical monitor 8. It can also be said that a transformation linking of the in-vivo recording of the endoscope (endoscopic image) and the pre-operative 3D recording data (3D image) is performed.

[0095] In step S7, the position and / or movement of the endoscope 2 is (continuously) detected, in particular indirectly also of the recording head 6 via the operating section 4. The detected movement can be provided as endoscope movement data.

[0096] In step S8 , the detected movement of the endoscope 2 with the recording head 6 is transferred to the virtual position and / or orientation of the virtual recording head 6 ′ of the virtual endoscope 2 ′ in the view of the 3D recording data 3DA.

[0097] In a subsequent step S9, an updated associated representation K is generated with a parallel representation of the in-vivo recording IA and the updated views of the 3D recording data 3DA, which is displayed, for example, via the surgical monitor 8. In this way, the surgeon is provided with a current associated representation of the in-vivo recording IA and the views of the 3D recording data 3DA.

[0098] In a first condition B1 , it is then checked whether a re-registration should be performed. If a re-registration is to be performed, the method jumps to step S3 of the detection of the landmark 15 .

[0099] If a re-registration is not to be performed, the method jumps to the second condition B2 and checks whether the method should be ended. If the method is not to be ended, the method jumps again to step S7 of movement detection and carries out a new iteration.

[0100] If condition B2 indicates that the method should be terminated, the loop is interrupted and the acquisition representation method is finally terminated.

[0101] In the case of a new registration, it is not necessary to redefine the landmarks. In particular, previously determined landmarks can also be used which are merely “remeasured” or linked.

[0102] In particular, the above-mentioned photographing and representing method is stored in the form of instructions on a computer-readable storage medium and when read and executed by a computer, causes the computer to implement the method steps.

Claims

1. A surgical assistance system (1) for use in a surgical operation on a patient (P), comprising: A presentation device for presenting visual content, an endoscope (2) having an imaging head (6) which is arranged and adapted to create an in-vivo image (IA) of the patient (P), a data providing unit adapted to provide digital 3D recording data (3DA) of the patient (P), a control unit (12) adapted to process the in-vivo imaging (IA) of the endoscope (2) and the 3D imaging data (3DA), Features a registration unit (14) adapted to detect at least one anatomical landmark (15) and / or anatomical orientation in the in-vivo recording (IA) and to determine a corresponding anatomical landmark (15') and / or orientation in the provided 3D recording data (3DA) and to at least initially register the in-vivo recording (IA) with the 3D recording data (3DA) by means of the corresponding anatomical landmark (15, 15') and to register the endoscope (2) relative to the patient (P), a tracking system (16) adapted to continuously detect the position and / or orientation and / or movement of the endoscope (2) and to provide it to the control unit (12) as endoscope movement data, The control unit (12) is further adapted to generate an associated representation (K) having at least one representation of the in-vivo recording (IA) and at least one representation of the view of the 3D recording data (3DA), in which the in-vivo recording (IA) and the view of the 3D recording data (3DA) are associated with each other with respect to the endoscope (2) by transferring the detected endoscopic movement data to at least one virtual position and / or orientation of a virtual recording head (6') in the view of the 3D recording data (3DA) for the associated movement, and the control unit (12) is adapted to visually output the associated representation (K) generated thereby via the representation device, wherein the tracking system (16) is constructed as an endoscope-internal tracking system (16), and the endoscope (2) has an endoscope-internal motion sensor for detecting endoscope movement data, and / or For the tracking system (16), the control unit (12) is adapted to determine the movement of the endoscope by image analysis of moving anatomical structures in the in vivo filming (IA).

2. The surgical assistance system (1) according to claim 1, characterized in that: The tracking system (16) has an acceleration sensor (20) and / or a rotation speed sensor (22) arranged in the endoscope, and in addition, the control unit (12) is suitable for detecting the endoscope movement data by means of the image analysis, wherein a combined value of the two results of the endoscope movement data is calculated as the endoscope movement data.

3. The surgical assistance system (1) according to claim 1 or 2, characterized in that: The control unit (12) is adapted to positionally correctly identify at least one medical instrument (24) in the in-vivo recording (IA), and to positionally correctly indicate the identified medical instrument (24) in the views of the 3D recording data (3DA), so that the medical instrument (24, 24') is shown in both recordings (IA, 3DA).

4. The surgical assistance system (1) according to claim 1, characterized in that: The registration unit (14) performs a re-registration in addition to the initial registration at at least one further point in time in order to further improve the accuracy of the association.

5. The surgical assistance system (1) according to claim 1, characterized in that: The endoscope (2) is robot-guided by means of a robot arm, and the tracking system (16) determines the position and / or orientation and / or movement of the endoscope (2) via the associated position and / or orientation and / or movement of the robot and thus via the robot.

6. The surgical assistance system (1) according to claim 1, characterized in that: The control unit (12) is suitable for displaying, in an associated representation (K) in the representation of the 3D recording data (3DA) and / or the in vivo recording (IA), in addition to the endoscope (2; 2') also pre-planned medical instruments (24') and / or implants stored in the storage unit (10) in a correctly oriented manner.

7. The surgical assistance system (1) according to claim 1, characterized in that: The control unit (12) is adapted to display a planned path and / or annotations in the associated representation (K) in the view of the 3D recording data (3DA) and / or in the representation of the in-vivo recording (IA) in order to indicate a path to an operating area for a surgeon.

8. The surgical assistance system (1) according to claim 1, characterized in that: The control unit (12) is adapted to create a view of the 3D recording data (3DA) As a 3D scene, or as a two-dimensional cross section relative to an image coordinate system of the endoscope (2) and / or along a recording axis of the endoscope (2), or As a virtual endoscopic image.

9. A computer-readable storage medium comprising instructions, which, when executed by a computer, cause the computer to implement the method steps of a method for collectively and associatively representing two different shots, characterized in that step: (S1) reading digital 3D imaging data (3DA) of a patient (P); (S2) creating or detecting an in-vivo image (IA) by means of an endoscope (2) in a surgical assistance system (1) according to any one of claims 1 to 8; (S3) detecting at least one landmark (15) and / or anatomical orientation in the in vivo recording (IA); (S4) determining corresponding landmarks (15') and / or orientations in the 3D recording data (3DA) by means of a registration unit (14); (S5) registering the 3D recording data (3DA) with the in-vivo recording (IA) by means of the registration unit (14) via at least one detected corresponding landmark (15, 15') and / or orientation; (S6) generating, by means of a control unit (12), an associated representation (K) having at least one representation of the in-vivo recording (IA) and at least one representation of a view of the 3D recording data (3DA); (S7) continuously detecting the position and / or orientation and / or movement of the endoscope (2) in the surgical assisting system (1) according to claim 1 by means of a tracking system (16) in the surgical assisting system (1) according to claim 1; (S8) transmitting the detected movement of the endoscope (2) via the control unit (12) to at least one virtual position and / or orientation of a virtual recording head (6') in the view of the 3D recording data (3DA), and (S9) The control unit (12) generates at least one updated associated representation (K) of the updated view of the in-vivo recording (IA) and the 3D recording data (3DA) and outputs the associated representation (K) via the representation device.

10. The computer-readable storage medium according to claim 9, wherein: The shooting and representing method further comprises the steps of: Determining a deformation of the actual anatomical structure based on the detected landmarks and the determined corresponding landmarks, and The determined deformation is transferred to the virtual anatomy in the 3D recording data in order to adapt and correct the 3D recording data accordingly.

11. A medical disinfection room, characterized in that: The medical sterilization chamber has a surgical assistance system (1) according to claim 1.

Images