Surgical assistance system and display method equipped with a surgical microscope and camera

Abstract

The present disclosure relates to a surgical assistant system (1) for use in a surgical intervention on a patient (P), the assistant system comprising a surgical microscope (2) comprising a housing (6), a movable microscope head (4) including an optical system (8) adapted to provide optical magnification of a region of interest (B) in the direction of an optical microscope axis (10) and to create a digital microscope image (14) by a downstream microscope imaging unit (12), and a movable microscope arm (16) connected to a base (18) on which the movable microscope head (4) is mounted and adapted to adjust the position and / or orientation of the microscope head (4), at least one display device (20) for displaying visual content, and a display unit (22) for processing the microscope image (14). and a control unit (22) adapted to appropriately control the display device (20) for display, the surgical microscope (2) further comprising a camera system (24) having at least one surrounding camera (26) arranged on the microscope head (4), the surrounding camera (26) adapted to capture both an area (B) targeted by the optical system (8) and the surroundings (U) around the target area (B) by the surrounding camera (26) such that the field of view (36) of the surrounding camera (26) includes the field of view (34) of the optical system (8) to provide a surrounding image (30), the control unit (22) further adapted to process the surrounding image (30) and generate a composite view (32) including the microscope image (14) and the surrounding image (30) and visually output it by the display device (20). Further, the present disclosure relates to an image display method, a storage medium, and a sterile space.

Description

Technical Field

[0001] The present disclosure relates to a surgical assistance system comprising a surgical microscope / operating microscope used in surgical interventions / treatments on patients. The surgical microscope comprises a housing and, in particular, is provided within the housing and provides an optical magnification of a target area (intervention area) in the direction of the optical microscope axis, particularly at a distance between 5 cm and 80 cm, particularly preferably between 20 cm and 60 cm, and is particularly provided with a sensor for creating (digital) microscope images and an optical system adapted to provide the microscope images in a digitally / computer-readable manner, and comprises a movable microscope head. Further, the surgical microscope comprises a joint-movable / movable microscope arm provided at a base, on which the movable microscope head is arranged and particularly supported, and the microscope arm is adapted to adjust the position and / or orientation of the microscope head in space. The surgical assistance system further comprises at least one display device for displaying visual content and a control unit adapted to process the microscope images and control the display device for corresponding display. Further, the present disclosure relates to an image display method, a storage medium, and a sterile space according to the preamble of the claims.

Background Art

[0002] Surgical microscopes are standard instruments used in surgical interventions, particularly in neurosurgery, spinal surgery, and microsurgery. These surgical microscopes typically feature binocular optics, allowing surgeons to directly view optically magnified microscopic images / photographs, or to view magnified images / photographs via a display device / external notification device such as an OR monitor for visually displaying (digitally) recorded microscopic images. However, in either case, surgeons view only the microscopic image of the intervention area at very high magnification through binoculars or an OR monitor. As a result, surgeons are unable, or at least find it difficult, to see the surrounding surgical area / field / intervention area (of the microscopic image) to confirm the position of their hands or the medical instruments used during the surgical intervention. Only the tips of surgical instruments, when properly positioned, are within the field of view of a surgical microscope.

[0003] However, looking around the microscopic image is crucial to avoiding damage outside the actual intervention area. To look around, surgeons must take their eyes off the binocular system or the microscope's display, quickly visually grasp the area around the surgical microscope's target, and then return to the microscopic image, which is a significant burden during intervention. Constantly switching between different field-of-view directions of two field-of-view modalities and maintaining a constant correlation in the surgeon's mind is not only tiring but often leads to unexpected mistakes.

[0004] Furthermore, surgical microscopes must be positioned in the operating room so that, on the one hand, the surgeon in particular can easily handle and look through the microscope, or view the OR monitor with a good field of view, and on the other hand, so that the surgical field / surgical area / intervention area can be viewed clearly. Therefore, the OR monitor must be positioned so that the surgeon has a good field of view of both modalities, but this limits the flexibility of the positioning in the operating room. Also, it is usually difficult for the surgeon to directly view the intervention area because the volume of the microscope head prevents them from directly viewing the patient. Moreover, this position and the corresponding good field of view are secured only for the surgeon, and it is difficult for other surgical staff in other positions in the operating room to see the microscope image or the intervention area displayed on the OR monitor. Therefore, medical staff are constantly forced to turn their heads and change their viewing direction if they are in an unfavorable position.

[0005] For example, US 201902901 A1 discloses an auxiliary system comprising a surgical microscope used in combination with a portable visualization system mounted on the surgeon's head. This visualization system allows the surgeon to view either a microscopic image or a type of magnified image / magnification display, depending on the field of view of their eye. While this allows the surgeon to switch between two modalities, such visualization systems have limitations and difficulties. Because the optics are integrated, the visualization system is heavy, expensive to manufacture, and prone to errors because the beam path to the surgeon's eye must always be precise. Furthermore, the surgeon loses their sense of their surroundings due to the visualization system.

[0006] For example, US9,936,863 B2 discloses a system comprising a surgical microscope having an additional camera mounted on a wound dilator on the patient, in order to provide the surgeon with central microscopic images from the surgical microscope and peripheral images from an additional camera mounted on the wound dilator. However, such a system has the drawback that the wound dilator equipped with the camera must be attached to the patient, and that the surrounding environment of the intervention site, which would provide the surgeon with important information, is not optically captured. [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Therefore, the objective of this disclosure is to avoid, or at least reduce, the shortcomings of prior art, and in particular to provide a surgical assistance system, (image) display method, storage medium, and sterile space that enable participants / medical staff in the operating room, especially the (leading) surgeon, to view a unified and intuitive information fusion of (magnified) microscopic images / micrographs of the target area of ​​the intervention site and a display of the surrounding environment of the target area of ​​the intervention site. Furthermore, hand-eye coordination should be improved and safe access to the intervention area / surgical area / surgical field should be ensured, in particular to avoid, or at least minimize, tissue damage due to careless or unintended movements or the duration of surgical intervention. In particular, the head movements and mental stress of the surgeon that inevitably accompany the intervention should be minimized by linking microscopic images with information from the environment. [Means for solving the problem]

[0008] These problems are solved with respect to a general-purpose surgical assistance system according to the features of claim 1, a general-purpose image display method according to the features of claim 12, a general-purpose computer-readable storage medium according to the features of claim 13, and a general-purpose sterile space according to the features of claim 14.

[0009] Basically, a surgical microscope, particularly a microscope head, includes a camera system that includes at least one additional camera. The camera is positioned on the microscope head, particularly on the housing of the microscope optics, and is especially fixed / securely mounted. Thus, the axis of the optical microscope oriented in a similar field of view direction to the axis of the optical camera. Therefore, both the microscope image and the camera image are particularly interrelated. The additional ambient / environmental camera produces images at lower magnification compared to the surgical microscope, or preferably wide-angle images / wide-angle photographs with a similar field of view direction to the optics of the surgical microscope. The ambient camera or another camera in the camera system may be positioned on the microscope head so as to be located within the microscope head, particularly within the housing, or integrated with it.

[0010] Both the microscopic image from the surgical microscope (or optical system) and the ambient image from the surrounding camera are processed by the control unit to generate a composite view containing both images. This (digitally generated) composite view, which provides visual information from both images, is then centrally output by a display device.

[0011] Thus, this assistive system allows users, particularly surgeons, to visually grasp both the interior of the magnified intervention area and the surrounding / exterior overview of the intervention area without changing head position or viewing direction. In particular, the unified display improves coordination, reduces fatigue, and shortens intervention time. Furthermore, it improves patient safety by providing surgeons with a better overview.

[0012] In other words, a surgical microscope, particularly a microscope head, comprises a camera system including at least one ambient camera, the ambient camera being positioned on or connected to the microscope head. In particular, the ambient camera is mounted on the housing. Preferably, the ambient camera is firmly fixed to the microscope head, particularly the housing, and its orientation relative to the microscope head and optics is always kept constant. In order for the ambient camera to detect both the area targeted by the optics and the surrounding area of ​​the area of ​​target, and to provide a digitally / computer-readable ambient image, the ambient camera is adapted so that its field of view includes / encompasses / companies / accompanys the field of view of the surgical microscope's optics, particularly through the orientation of its optical camera axis and / or through its camera optics, from a predetermined distance, preferably from a distance of 5 cm from the front of the camera or the side of the microscope head. The control device is further adapted to process the ambient image, generate a composite view of the microscope image and the ambient image, and output the composite view visually by a display device.

[0013] Here, a composite view means that information from both the microscope image and the surrounding image is displayed centrally in a single view. In other words, a composite view displays the microscope image and the surrounding image, which may be image-processed as needed.

[0014] In particular, the ambient camera has a field of view direction similar to or parallel to the optical system of a surgical microscope. Therefore, in addition to the target area, the surrounding environment of this target area is also optically recorded. The optical axes of the optical system and the ambient camera are arranged in the same way, or more specifically, parallel or substantially parallel to each other. It should be noted here that optical detection does not necessarily have to be limited to the visible light range, but may include other frequencies below and above the visible light range. For example, an infrared image may be recorded as an ambient image by the ambient camera, processed by the control unit, and appropriately output. Preferably, the ambient camera, in particular the ambient camera and the control unit, or another camera in the camera system, may be adapted to record or generate a three-dimensional image and / or a so-called depth map. In this way, not only a two-dimensional image but also the three-dimensional structure of the surroundings is obtained. The depth map contains information about the distance from the viewpoint to the surface, so the three-dimensional structure is detected.

[0015] The term "field of view" refers to the area within the field of view of an optical system or a peripheral camera that can capture an image. In other words, it is the image captured by an image unit or sensor, limited by its peripheral edge. This can be likened to a cone positioned coaxially with the optical axis, its tip located in front of the optical system. All "objects" within the internal volume of the cone can be recorded. The aperture angle of the cone defines the possible image area. Here, the cone of the microscope is schematicly located within the cone of the peripheral camera. Therefore, the peripheral camera can detect the target area, since the target area is also within the peripheral camera's cone, and can also detect the surrounding environment of this area. The peripheral camera has a wide aperture angle (of the cone) and consequently low magnification, while the optical system of a microscope has a very narrow aperture angle and consequently high magnification.

[0016] Advantageous embodiments are described in the dependent claims and are explained below.

[0017] In a preferred embodiment, the auxiliary system may further include a storage unit storing preoperative three-dimensional image data, particularly MRI image data and / or CT image data, and the auxiliary system, in particular the control unit, may be adapted to spatially detect the 3D (i.e., three-dimensional) structure of the patient, particularly the head, particularly preferably the patient's face, using a camera system, particularly using only a perimeter camera, and to associate the detected 3D structure with 3D image data in order to register the patient, particularly with respect to a surgical microscope, particularly preferably the microscope head. In other words, the control unit, in particular, may be adapted to associate the recorded 3D structure with 3D image data based on the patient's stored 3D image data, particularly stored magnetic resonance imaging (MRI) or computed tomography (CT) data, in order to register the patient. Thus, via a transformation matrix, the patient's local coordinate system is linked to the surgical microscope's local coordinate system, particularly the microscope head's local coordinate system. In this way, the 3D image data may be geometrically associated with the patient or linked to the microscope head. Therefore, patient registration can be easily performed using a perimeter camera. When the microscope head moves, the movement of the microscope head may be detected / tracked and associated accordingly with 3D image data. Thus, another function of an auxiliary system equipped with a perimeter camera is to register the patient when using a surgical navigation system. In other words, an additional perimeter camera may be used to register the patient by pattern recognition. The patient's 3D surface, e.g., face, is extracted and associated with the 3D surface of 3D image data from a preoperative 3D dataset, such as CT image data or MRI image data. The perimeter camera thus generates a 3D image of the patient's surface, e.g., a 3D image of the face, which may be associated with preoperative 3D image data such as MRI image data, and the control unit is adapted accordingly. External references for association may be rigid bodies and / or optical patterns and / or distinctive landmarks, in particular. The perimeter camera is positioned on the patient in the area used for registration, e.g., the face. The term 3D means that the patient's structure, image data, or surface is spatial, i.e., three-dimensional.A patient's body, or at least a part of their body, which has spatial dimensions, can be digitally utilized as image data in a three-dimensional space, for example, in a Cartesian coordinate system (X, Y, Z).

[0018] In a further embodiment, registration may be performed via at least one marker attached to the patient, in addition to or as an alternative to registration via a 3D structure. The control unit is adapted to register the patient to a surgical microscope, particularly a microscope head, via a predetermined detected position of at least one marker. This registration may be performed using or without a marker by employing image processing techniques. The term "pose" refers to both position and orientation.

[0019] For 3D structure detection, the camera system may have a 3D camera and / or a 2D camera, and it is particularly preferable that the camera system consists only of a perimeter camera as the sole 2D camera or 3D camera. In the case of a 2D camera, the 2D camera may be moved over the patient's region of interest to obtain various views of the region of interest. The control unit is then adapted to calculate the 3D structure from the various views captured using image analysis (machine vision). Thus, to generate a 3D surface / 3D structure, a 3D camera is used, or a 2D camera is used that is moved over the patient to generate images from various viewpoints. The 3D structure / 3D surface thus generated may be associated with preoperative 3D image data, in particular, via the control unit.

[0020] According to one aspect of the present disclosure, the auxiliary system may, in particular, include at least one sensor adapted to detect the position, in particular the pose, of the microscope head and provide it to the control unit as position data or pose data. The control unit is preferably adapted to store the position, in particular the pose, of the microscope head at the time of registration in a storage unit as the registration position and associated ambient images, in particular the detected 3D structure. When re-registering with the stored registration position or pose, the control unit may determine that a registration misalignment exists if the superposition of the stored 3D structure and the re-detected 3D structure is misaligned by more than a predetermined threshold, particularly in a sub-region of the superposition that does not change during the intervention (such as a cover or wound edge / incision boundary). In particular, the control unit may be adapted to calculate the amount of misalignment. For example, a set of characteristically predetermined points may be defined, and the distances between corresponding points of the stored 3D structure and the newly detected 3D structure may be summed. If the sum of the distances exceeds a threshold, a misalignment may be determined to exist. The control unit may preferably be adapted to automatically correct registration by offset, particularly by translational offset in space and / or rotational offset about an axis. In other words, the auxiliary system may preferably be adapted to detect the displacement between two ambient images (of the same pose), calculate the offset / displacement / shift, and correct a predetermined offset. In particular, the control unit may notify the user of the presence of the displacement, for example, visually via a display device and / or acoustically via an audio device. In particular, an alarm signal may be output via the auxiliary system. In particular, the auxiliary system may, by the ambient camera and the adapted control unit, automatically detect and correct navigation inaccuracies caused, for example, by unintended movements of the patient tracking device. The optics of the ambient camera and the surgical microscope are geometrically related. During or after initial registration, the ambient camera captures an image of the intervention area from a specific position (registration position) or pose. This position or pose is calculated as a relative transformation (transformation matrix) from the patient tracking device to the microscope tracking device.To verify accuracy, the microscope head may be moved back to this registration position at any time during surgery, and another photograph (surrounding image) may be taken. Both the initial registration image and the accuracy check image (new surrounding image) are superimposed. If there is no inaccuracy or misalignment, these images will match in all specific image parts / regions that are not changed during the intervention (such as the cover or wound / incision boundary). On the other hand, if there is misalignment or inaccuracy, the two images / photographs will be offset from each other. In particular, the user may be notified of this misalignment. Furthermore, the amount of misalignment may be informed by calculating the offset. Preferably, the inaccuracy may be automatically corrected by the control unit by adjusting the registration using the determined offset.

[0021] In a further aspect of this disclosure, a geometric position, in particular the pose of the microscope head, may also be predetermined in place of or in addition to a stored registration position, and the microscope head is moved to that position, in particular to take a periphery image from there via detection of an intervention area or detection of a patient's face. The above-described features relating to the configuration of an auxiliary system for calculating the displacement are also found in this aspect.

[0022] In particular, a surgical microscope may be equipped with at least one actuator for actively moving and controlling a movable microscope arm and a movable microscope head, and the control unit may be further adapted to control the actuator to move the microscope head to a predetermined position and / or orientation in space. Thus, the microscope head may be "moved" automatically rather than manually, in particular to a predetermined position (position and orientation). In this way, for example, the registered position described above can be reproduced with clear repeatability, or a stored view can be "recalled" by the surgical microscope.

[0023] According to one aspect of the present invention, which may be independently described in the claims, the ambient camera and / or image unit may preferably comprise a high-resolution sensor of at least 35 megapixels, and the control unit may be adapted to digitally magnify one area of ​​the sensor with respect to the image, particularly the central area, while using a lower magnification for other areas of the sensor.

[0024] According to further aspects of the present disclosure, the control unit is adapted to first generate an ambient image, extract at least one characteristic marker in the ambient image, and, based on the detected pose of the microscope head (where the geometric relationship between the microscope head and the ambient camera is defined), to determine the position of the extracted marker relative to the microscope head, in particular its pose, i.e., to determine the position of the marker in space relative to a reference coordinate system, and to store the position or pose determined in association with the characteristic marker in the ambient image as a reference value in a storage unit, the control unit is further adapted to continuously recognize this extracted marker in the current ambient image, continuously determine the current position of the marker in space, in particular its current pose, and to determine that there is a registration misalignment if there is a discrepancy between the stored position or pose and the current position or pose. In particular, the control unit outputs a corresponding message via a display device. In other words, in one embodiment, instead of detecting a registration misalignment, a method or a control unit adapted accordingly may be used to determine a registration misalignment of an undetermined position or pose. In this case, a portion of the region observed or detected by the ambient camera that does not change during the intervention is used. In this region, traceable markers are extracted using image processing techniques. These markers may be initially found or determined at the start of an intervention or procedure, or they may be continuously found or determined during the intervention. The markers must be detectable by the ambient camera even if the microscope head's position or orientation changes. The position of each marker within the ambient camera's field of view is continuously identified relative to the ambient camera, particularly by image processing techniques. Since the geometric relationship or pose of the ambient camera relative to the microscope head is known, and the microscope head's position is measurable, the observed marker positions are also known, or can be calculated or identified by the control unit. The marker positions thus identified may be compared to expected marker positions or poses (target positions). Any discrepancies suggest registration errors. The expected marker positions are stored in the system or memory unit, particularly at the time of the initial observation or detection of the marker, so that they can be used as a reference value for further detection of the markers.

[0025] In a further aspect of this disclosure, the control unit may be adapted to improve the registration of the camera system and / or minimize errors in the camera system by using an image processing method to identify the position, in particular the pose, of a landmark relative to the surrounding camera, and comparing this pose with a known pose of the landmark, in particular the pose of the landmark detected by an external camera of the navigation system and / or the pose detected by the optical system and image unit of the microscope. In other words, according to one embodiment, the identified position or pose of an observed or characteristic landmark relative to the surrounding camera can also be used to improve the accuracy of the navigation system or registration. For this purpose, it is assumed that the pose of the microscope head, the pose of the surrounding camera, and the pose of the landmark are known or identifiable. If the position, in particular the pose, of the landmark relative to the surrounding camera is identified (by the control unit) using an image processing method, there is another way to identify the position of the landmark via the position, in particular the pose, of the microscope or microscope head. Due to inaccuracies in the camera system or registration, the position of the landmark differs in the two methods. From a mathematical standpoint, this becomes an over-deterministic system. By using minimization techniques, this over-determination can be leveraged to continuously improve errors in the camera system and registration errors in the navigation system, thereby improving the overall accuracy of the system.

[0026] According to a further aspect of the present disclosure, in the case of an actively movable microscope head, the assistance system may comprise an input unit, particularly a touch screen, for selecting a focus point in the ambient image of the ambient camera. The control unit is adapted to actively control and move the microscope arm and the microscope head via at least one actuator based on the selected focus point such that the axis of the optical microscope is aligned with the selected focus point in the patient, particularly at a predetermined relevant distance and / or at a predetermined image angle. In this way, an intuitive and automatically adjustable surgical microscope is provided that allows the surgeon to focus the microscope appropriately by simply touching the selected point in the ambient image. Thus, the additional function of the assistance system with an ambient camera is to move the robotic-guided surgical microscope to the target field by determining the target / focus point in the ambient image of the ambient camera and moving the microscope such that the focus point is at the focus of the microscope image or at the focus of the optical system of the surgical microscope.

[0027] In a further aspect of this disclosure, the control unit may be adapted to detect objects within the area of ​​the microscope head and the distance of the movable microscope head to these objects by image analysis of the surrounding image and / or by sensors provided on the microscope head. If there is a possibility of collision with the object, particularly if the distance is less than a predetermined distance, the auxiliary system, in particular the control unit, is adapted to emit an alarm signal, in particular an alarm sound via an audio device and / or an alarm display via a display device, and / or to stop the overall movement of the microscope head, in particular the entire surgical microscope, in order to limit at least the degrees of freedom of movement of the microscope head and / or microscope arm. Thus, a further function of the auxiliary system with an additional surrounding camera is to prevent collisions between the microscope and objects by the auxiliary system determining the distance to the microscope head and issuing a visual or auditory warning if the distance is insufficient, or by stopping the movement in the case of a robot-guided surgical microscope. The surrounding camera / overview camera can also detect collisions during the autonomous movement of the microscope head, particularly when guided on a microscope arm in the form of a robotic arm, and its movement can be stopped in a timely manner, for example by a brake or locking system. Even when microscope arms or robotic arms are manually guided, collisions can be detected and further movement prevented.

[0028] The control unit may preferably be adapted to correctly detect, and in particular correctly track, the pose of medical instruments, especially surgical instruments, and / or the pose of the microscope head, via a camera system, and in particular, at least ambient images, and in particular markers and / or patient features. Accordingly, the ambient camera may continuously detect the pose of medical instruments. This detected pose of the medical instrument may be reproduced in 3D image data, in particular by schematically displaying a virtual position and / or orientation, or by superimposing a virtual instrument in the corresponding position. In particular, the relative pose of the patient and / or medical instrument to the microscope head may be accurately displayed in the 3D image data. Furthermore, in particular, the detected microscope head may also be displayed in the 3D image data as a virtual microscope head. In other words, an auxiliary system with an ambient camera may be used in particular to track the patient's position, in particular, pose, during the use of a surgical microscope. To identify or find the patient's position, in particular, pose, relative to the microscope head, it is preferable that external references such as rigid bodies, optical patterns, or unique anatomical landmarks may be used for this purpose. This allows for the display of microscope images associated with preoperative 3D image data, such as MRI image data. The surrounding camera may also preferably be used to track surgical instruments, particularly those equipped with markers, and to overlay their position, especially their pose, onto the microscopic image of the preoperative 3D image data. Thus, another function of an auxiliary system with an additional surrounding camera is to provide an auxiliary system for tracking / following the patient and / or surgical instruments and / or the surgical microscope itself during the use of the surgical navigation system. For example, markers within the field of view of the surrounding camera may be used for tracking, or tracking may be performed without markers by utilizing features unique to the patient or instrument. In this case, the surrounding camera can replace the external camera that is normally used in the surgical navigation system. Furthermore, the surrounding camera can also temporarily replace or complement the external camera of the surgical navigation system to detect features and instruments, especially to pinpoint their position, even if the external camera's field of view is temporarily limited (e.g., due to overlap).

[0029] According to a further aspect of the present disclosure, data of at least one medical instrument, particularly geometric relationships, and associated usage instructions may be stored / memorized in a memory unit. The control unit is further adapted to detect surgical instruments in the surrounding image based on the stored data, particularly based on the stored geometric relationships, and output the associated stored usage instructions to the user via a display device and / or an audio device. Thus, an additional function of the assistance system with a surrounding camera is the recognition of the medical instruments used, particularly surgical instruments, and the provision of associated usage instructions to the surgeon.

[0030] According to one embodiment, the at least one display device may be an OR monitor and / or a head-mounted display (HMD) and / or a binocular system having superimposable data. When the display device is an OR monitor, the composite view is displayed on the OR monitor as an image. When the assistance system has a head-mounted display / virtual reality goggles as the display device, the composite view may be output via the display. Particularly, when the surgical microscope and / or the surrounding camera are adapted to generate a 3D image, the composite view may be output three-dimensionally via the head-mounted display so as to provide a spatial view to the surgeon. In the case of a binocular system that the surgeon observes with both eyes, the surrounding data may be superimposed on the image plane, for example, via a prism and a display. This constitutes an augmented reality system. In other words, the at least one display device may be an (OR) monitor and / or a head-mounted display and / or a binocular system.

[0031] In particular, the control unit may be adapted to generate a side-by-side or superimposed display of the microscope image and the surrounding image as a composite view and output it via a display device. In other words, the microscope image and the surrounding image may be displayed side-by-side or superimposed. In the case of superimposed display, the microscope image (e.g., without transparency) may be generated by the control unit in the central region of the composite view, and the surrounding image may be displayed as a background around this microscope image.

[0032] In further embodiments, the magnification of the optical system may be at least 5x, particularly at least 10x, and especially preferably at least 40x. The optical system of the surgical microscope may preferably have a zoom function. The magnification of the ambient camera may more preferably be up to 5x, particularly up to 1x. While the surgical microscope with the optical system and image unit provides relatively high magnification, the ambient camera with the optical system is adapted to generate wide-angle images and provide them as ambient images for environmental display.

[0033] According to one aspect of the present disclosure, the camera system may further include, in addition to the perimeter camera, a second camera, particularly in addition a third camera, and especially preferably in addition a fourth camera, to expand the field of view around the intervention area. Thus, the camera system may use additional cameras in addition to the perimeter camera to expand the field of view outside the surgical field.

[0034] In particular, in order to provide a wide-angle ambient image, the focal length of the ambient camera's optical system may be a minimum of 10 mm and / or a maximum of 50 mm, or the field of view may be at least 45°, preferably at least 70°.

[0035] The optical microscope axis may intersect or pass through the optical camera axis at a distance of up to 10 cm, and particularly preferably up to 5 cm. In particular, the angle between the optical microscope axis and the optical camera axis may be a minimum of 2° and / or a maximum of 15°. Alternatively, it is preferable that the optical microscope axis and the optical camera axis are parallel to each other.

[0036] With respect to a method of displaying two different images, the problems and objectives of this disclosure are solved by the following steps: preferably, placing a perimeter camera on a movable microscope head of a surgical microscope; targeting a region through an optical system located on the movable microscope head; generating a microscope image by a microscope image unit located on or within the microscope head at a magnification provided by the optical system; generating a perimeter image by a perimeter camera located on the microscope head, wherein the field of view of the perimeter camera, particularly from a predetermined distance, includes the field of view of the optical system; generating a composite view including the microscope image and the perimeter image; and outputting the composite view by a display device. This display method provides both image modalities to the surgeon in a unified manner.

[0037] The display method may further preferably include the steps of: reading preoperative 3D image data, particularly MRI image data and / or CT image data; detecting the patient's 3D structure using a surrounding camera; associating the detected 3D structure with the 3D image data; and registering the patient based on the association.

[0038] According to one embodiment, the display method may include the steps of: initially registering a patient; detecting the pose of the microscope head; generating a surrounding image using a surrounding camera; storing the detected pose (registration position) and the surrounding image; moving the microscope head and then returning it to the stored pose; reacquiring the surrounding image; comparing the stored surrounding image with a new surrounding image; determining the discrepancy between the stored surrounding image and the new surrounding image; outputting an acoustic signal or a visual signal if the discrepancy exceeds a threshold; and / or calculating an offset and applying the offset to the initial registration.

[0039] In particular, as described above with respect to surgical assistance systems and applicable to this method as well, the position of the observed marker may be continuously compared with the reference position of the marker to determine the deviation. If a deviation is detected, it is preferable that a registration deviation is determined, and a message is issued. When using a method for continuously detecting registration deviations, it is possible to omit storing the registration position, imaging the environment at the registration position, and returning to the registration position with imaging the environment.

[0040] In a further embodiment, the display method may include the steps of detecting a medical device and its position and / or orientation in an ambient image, transferring the pose of the medical device to 3D image data, and displaying a virtual view or at least a schematic view of the medical device in the 3D image data.

[0041] The display method preferably includes the steps of tracking the patient and / or surgical instruments and / or microscope head using a surrounding camera, and providing corresponding motion data.

[0042] Furthermore, the display method may include, in particular, the steps of determining the distance to an object detected in the surrounding image, outputting an alarm signal if the distance is short, and / or limiting the degrees of freedom of movement of the microscope head, in particular stopping / preventing the movement of the microscope head. The surrounding image from the surrounding camera / overview camera can also be used to detect collisions, in particular during the autonomous or coordinated movement of a microscope arm in the form of a robotic arm. Thus, the braking system can stop the movement in a timely manner. For example, even when the microscope arm or robotic arm is manually guided by a surgeon, collisions can be detected and further movement can be prevented.

[0043] According to one aspect of the present disclosure, the display method may include the steps of reading a focus point in the surrounding image and controlling a microscope arm and a microscope head so that the optical microscope axis is aligned with the selected focus point, particularly at a predetermined distance and / or a predetermined image angle.

[0044] The display method may preferably include the steps of detecting a medical device in the surrounding image and outputting usage instructions related to the medical device via a display device.

[0045] It should be noted that the characteristics of the presentation method of this disclosure can also be applied to the surgical aid system of this disclosure, and vice versa.

[0046] With respect to computer-readable storage media, the problems and objectives of this disclosure are solved by providing the computer-readable storage media with instructions that cause the computer to perform method steps of the image display method according to this disclosure when executed by the computer.

[0047] Regarding general sterile spaces, the issues of this disclosure are resolved when medical sterile spaces are equipped with the surgical support system described herein.

[0048] This disclosure is described below with reference to preferred embodiments using the accompanying drawings. [Brief explanation of the drawing]

[0049] [Figure 1] This is a schematic perspective view of a surgical assistance system of a first preferred embodiment, the surgical assistance system being provided as a mobile surgical microscope. [Figure 2] Figure 1 is a schematic perspective view of the microscope head of a surgical microscope. [Figure 3] Figures 1 and 2 show a further schematic perspective of the support system during intervention. [Figure 4] This is a schematic diagram of a surgical assistance system equipped with a head-mounted display according to a more preferred embodiment. [Figure 5] This is a schematic diagram illustrating the detection of 3D structures using a surrounding camera. [Figure 6] A flowchart of an image display method according to a preferred embodiment is shown. [Modes for carrying out the invention]

[0050] The figures are schematic and are intended solely to facilitate understanding of this disclosure. The same elements are denoted by the same reference numerals. Features of various embodiments are interchangeable.

[0051] Figures 1 to 3 are schematic perspective views of a surgical support system 1 (hereinafter referred to as the support system) of a first preferred embodiment used in a surgical intervention on patient P. The support system 1 is used in a medical sterile space in the form of a preferred embodiment of an operating room 100 to support participants in the operating room, particularly the surgeon, with appropriate visualization during a surgical intervention on patient P (illustrated schematicly only). A minimally invasive intervention is performed on patient P in a central sterile surgical intervention area.

[0052] In this preferred embodiment, the surgical assistance system 1 comprises a surgical microscope 2, which is configured as a movable and autonomously operating surgical microscope 2. Specifically, the surgical microscope 2 (hereinafter simply referred to as the microscope) has a movable, front-mounted, movable / adjustable microscope head 4. Thus, the position and orientation of the microscope head 4 can be adjusted individually.

[0053] The movable microscope head 4 comprises a housing 6 and an optical system 8 located within the housing, which has multiple optical elements such as lenses and lens systems. In this embodiment, a monocular optical system is provided. However, a binocular optical system 8 may also be provided, for example, to create a three-dimensional microscope image. The optical system is adapted to provide optical magnification of the target area B in the direction of the optical microscope axis 10, as is common in surgical microscopes. In particular, the optical system is adapted to sharply focus on the target area B opposite a front lens (not shown) at a distance between 5 cm and 80 cm and provide magnification. The magnification thus provided is detected by a downstream microscope image unit 12 (hereinafter referred to as the image unit) equipped with a sensor such as a CMOS sensor or a CCD sensor, and the image unit generates a digital / computer-readable microscope image 14.

[0054] Furthermore, the microscope head 4 is attached to a movable microscope arm 16 supported on a trolley-like base 18. The microscope arm 16 has the form of a robotic arm, particularly a cooperative robotic arm, having two arm elements corresponding to the base 18 and the microscope head 4. This allows the microscope 2 to set the position and / or orientation of the microscope head 4 in space and position the microscope head 4 to the target area B that the surgeon wants to view magnified.

[0055] To visually display this magnified view, the auxiliary system 1 or the movable microscope 2 is equipped with an OR monitor 20 on the cantilever. The control unit 22 processes the microscope image 14 and outputs a display via the OR monitor 20.

[0056] In contrast to known auxiliary systems, the microscope head 4 of microscope 2 further includes a camera system 24 equipped with a perimeter camera 26 (hereinafter referred to as camera). This camera 26 is securely mounted to the housing 6 of microscope head 4 so that the optical system 8 of microscope 2 and the camera 26 are always in the same relative orientation to each other. Specifically, in order for the perimeter camera 26 to detect both the target area B of the optical system 8 and the surrounding area U around the target area B, and for the camera 26 to provide a corresponding perimeter image 30, the perimeter camera 26 is adapted, through the orientation of its optical camera axis 28 and the camera optical system, so that the (camera) field of view 36 of the perimeter camera 26 includes the (microscope) field of view 34 of the optical system 8 from a predetermined distance.

[0057] The control device 22 is configured to process the ambient image 30 and generate a composite view 32 in a side-by-side display format that includes the microscope image 14 and the ambient image 30, and to output it visually via the display device 20.

[0058] Thus, the movable microscope head 4 is equipped with an additional ambient camera 26. The viewing direction or orientation of the optical camera axis 28 of the camera 26 is the same as the viewing direction of the optical microscope axis 10 of the optical system 8 of the microscope 2. Since this additional camera 26 generates a wide-angle view, it is possible to photograph the surrounding area U, which is the surgical field around the target area B of the optical system 8, and in particular, it is possible to photograph the incision boundary, the surgeon's hand, the instruments used, etc. By displaying the image / video from the camera 26 together with the microscope image, the surgeon can see both the "inside" and "outside" of the intervention area without changing the position of their head or the direction they are looking. In this respect, the camera 26 is fixedly attached to the microscope head 4 and is adapted to image or detect the surrounding area U of the intervention area even when the distance between the microscope head 4 and the patient P is close, for example, 10 to 30 cm. In this auxiliary system 1, the surgeon views both the high-magnification microscope image 14 and the low-magnification additional ambient image 30 on the OR monitor 20. Therefore, surgeons can intuitively see both the high-resolution area inside the intervention zone and the clear area outside it at a glance.

[0059] The microscope head 4 has two different "optical systems": the microscope 2's optical system 8 and the camera 26's optical system, which are related to each other by their arrangement. The microscope 2's optical system 8 is responsible for generating high-magnification images, while the additional camera 26, with its related optical system, is responsible for generating wide-angle images. Both optical systems are related, have similar field-of-view directions, and may be directed towards the intervention area / surgical field. The camera system 24 with camera 26 has a low-magnification, wide field of view 36 to detect, for example, the incision boundary around the intervention site or the surgical environment with hands and / or instruments, while the optical system 8 has a high-magnification, narrow field of view 34. The composite view 32 displays the two images 14, 30 in a unified manner, allowing surgeons as well as other medical professionals to see both related image modalities at a glance.

[0060] In addition to camera 26, the camera system 24 may have another camera attached to the microscope head, for example, to create a stereo image or to capture multiple views and combine them appropriately in the composite view 32. Furthermore, camera 26 itself may be a 3D camera or a stereo camera in addition to a 2D camera.

[0061] Camera 26 has an autofocus function to focus on the target area B or its surroundings. Furthermore, camera 26 may have a zoom function to optimally adjust the surrounding image 30 for a specific intervention. Furthermore, the optical system 8 may also have a zoom function to adjust the magnification of the microscope 2. The zoom function may be continuous via lenses that move relative to each other, or discrete, such as a rotating nosepiece. In particular, the surgeon may be able to adjust the zoom / magnification using a knob on the microscope head. Also, camera 26 and / or the optical system 8 may have an adjustable iris for controlling focus and light input.

[0062] Since the control unit 22 digitally processes the two images, it can perform various digital analyses, image processing, and controls, in particular, to generate a suitable composite view 32. For example, the control unit 22 may be adapted to change various image parameters such as brightness and contrast of the images. Other digital data may be included in the composite view 32. The control unit 22 may include a central processing unit (CPU), volatile memory such as RAM, ROM, and non-volatile memory such as an SSD.

[0063] The microscope arm 16 is actively controllable by multiple actuators. Sensors on each element of the microscope arm 16 identify the pose of the microscope arm 16 and the microscope head 4, and transmit this information to the control unit 22. The surgeon can manually select a focus point using their finger on the ambient image 30 on the OR monitor 20, which is designed as a touchscreen. Once this set focus point is communicated to the control unit 22, the control unit 22 actively controls and moves the actuators of the microscope arm 16 and the bearings of the microscope head, aligning the optical microscope axis 10 to the focus point on the patient P at a predetermined distance and perpendicular image angle based on the set focus point. In this way, the target area B is set as the focus point, allowing the surgeon to easily intervene or view a magnified image at this point. The focus point may also be saved to automatically switch between different views.

[0064] Furthermore, the control unit 22 is further adapted to detect objects by image analysis of the surrounding image 30 and to calculate the distance to the objects. Alternatively or additionally, distance sensors may be provided on the microscope head 4 to determine the distance to objects, and in particular, multiple distance sensors may be provided in different directions. If there is a possibility of collision with an object thus detected within the operating range, the auxiliary system 1 emits an audible alarm signal (collision warning) via an audio device (not shown) and prevents the movement of the microscope arm 16. In particular, the movement of the microscope arm 16 and microscope head 4 is completely stopped (collision prevention). This prevents collision with an object.

[0065] Figure 4 shows a surgical assistance system 1 of a further second preferred embodiment. Similar to the first embodiment, the microscope head 4 is movably mounted to a fixed base 18 via a microscope arm 16, and the optical system 8 can target region B. The ambient camera 26 is oriented such that the optical camera axis 28 intersects the optical microscope axis 10 in the target region B. This ensures that the ambient image is concentric or symmetric with respect to the target region. Again, the control unit 22 processes the two images 14, 30 to generate a composite view 32.

[0066] However, in contrast to the first embodiment, the composite view 32 is output not via a monitor, but via a head-mounted display / HMD glasses 38 (hereinafter referred to as "glasses"). For this reason, the glasses 38 may further include a display and an optical system. Specifically, the control unit controls a transceiver unit 40 that wirelessly transmits data to the glasses 38. The received data is then displayed on the glasses 38.

[0067] The auxiliary system 1 also includes a storage unit 42 that uses a surgical navigation system and stores MRI and CT images of patient P as 3D image data. The control unit 22 is adapted to detect the 3D structure of the patient's face (see Figure 5) in three dimensions when the microscope head 4 is properly aligned with the camera 26. The 3D structure of the patient's face thus detected is compared with the virtual 3D structure of the patient's face in the 3D image data, and the two 3D structures are associated to perform initial registration and register patient P with the microscope head 4. When the microscope head 4 moves, sensors detect the corresponding movement.

[0068] Figure 5 schematically illustrates the registration function for the 3D structure of the patient's face performed by the auxiliary system in Figure 4. Camera 26 is a 2D camera that is moved over the region of interest of the patient's face (see dashed line) to acquire various views of the patient's face and calculates the 3D structure of the face by machine vision or image analysis performed by the control unit 22. The 3D structure of the face thus obtained may be superimposed, or rather associated, with the virtual 3D structure of 3D image data, or the control unit 22 itself may determine the position of patient P based on the 3D structure.

[0069] Furthermore, the control unit 22 of the auxiliary system 1 in Figure 4 is adapted to detect the medical instrument 44 via the ambient image 30 by markers 46 or features, particularly geometric features, attached to the medical instrument 44, and to identify its pose and / or function. In the case of features, a virtual geometric relationship of the instrument 44 may be stored in the memory unit 42, which the control unit 22 may use for matching. The medical instrument 44 thus detected may be superimposed on 3D image data at the detected position and may be used for surgical navigation. In particular, the control unit may further superimpose usage instruction data on the composite view to provide the surgeon with further guidance for intervention. Specifically, the detection of the instrument 44 by the ambient camera 30 may be used by the control unit 22 to optimally position and / or align the microscope head 4 via the robot guide microscope arm 16 to obtain a good view of the target area B, for example, without the instrument 44 obscuring the field of view or the instrument 44 being positioned and / or aligned with its tip in the field of view.

[0070] The control unit 22 is also adapted to automatically detect navigation deviations caused, for example, by undesirable movements of the patient tracking device / marker. For this purpose, the auxiliary system 1 is adapted to move the microscope head 4 to a predetermined position (registration position and orientation, hereinafter referred to as the registration position) after initial registration, and generate a surrounding image 30 by the camera 26, which is stored in the memory unit 42. The registration position is calculated as a relative transformation between the patient tracking device (not shown) and the microscope head tracking device (not shown). To ensure there is no deviation, the surgeon (or someone instructed by the surgeon) may occasionally move the microscope head 4 to the registration position during surgery to generate a new surrounding image 30. The new surrounding image is then superimposed on the surrounding image 30 stored in the memory unit 42 and compared. If there is no deviation or a small deviation that is acceptable, the unchanging subregions of the two surrounding images 30, such as the incision boundary or surgical drape, remain related to each other, and the control unit 22 uses the corresponding image analysis to determine that there is no deviation or the deviation is acceptable. However, if a portion of the two surrounding images 30 (including structures that do not actually change) is misaligned with each other by more than an acceptable limit, the control unit 22 detects this and outputs a message to the surgeon via the glasses 38 or monitor. Furthermore, the control unit 22 calculates the degree of misalignment using an appropriate calculation method and outputs the amount of misalignment to the surgeon. This allows the surgeon to determine whether the misalignment is excessive and take appropriate action. Alternatively or additionally, an image processing algorithm for continuous detection of registration misalignment or continuous improvement of registration accuracy may be stored in the memory unit 42 of the control unit 22.

[0071] In addition or alternatively, the control unit 22 may also determine the displacement / offset between the stored ambient image 30 and the new ambient image 30, and be adapted to correct this offset by fitting it to the registration or, as appropriate, to a transformation matrix between a local coordinate system such as the local coordinate system of the microscope head or surgical navigation system and the local coordinate system of the patient P.

[0072] Figure 6 shows a flowchart of an image display method (hereinafter referred to as the "display method") according to a preferred embodiment of the present disclosure for displaying a microscope image 14 and a surrounding image 30 in association with each other. This display method can be used as appropriate in the surgical assistance system 1 described above.

[0073] In the first step S1, the ambient camera 26 may be positioned on the movable microscope head 4 of the surgical microscope 2, and may be particularly securely attached.

[0074] In step S2, region B is targeted by the optical system 8 located on the movable microscope head 4. In the subsequent step S3, a microscope image unit 12 located on the microscope head 4 generates a microscope image 14 at the magnification provided by the optical system 8. Furthermore, in step S4, a surrounding image 30 is generated by a surrounding camera 26 located on the microscope head 4, and the field of view 36 of the surrounding camera 26 includes the field of view 34 of the optical system 8, particularly from a predetermined distance.

[0075] In step S5, the control unit 22 creates a composite view 32 that includes the microscope image 14 and the surrounding image 30, and in step S6, the display devices 20 and 38 finally output the composite view 32. (Item 1) A surgical support system (1) used in surgical intervention on a patient (P), wherein the support system (1) is A surgical microscope (2), A movable microscope head (4) includes a housing (6) and an optical system (8) adapted to provide optical magnification of a target area (B) in the direction of the optical microscope axis (10) and to generate a digital microscope image (14) via a downstream microscope image unit (12), A microscope (2) comprising: a movable microscope arm (16) connected to a base (18), on which a movable microscope head (4) is mounted, and which is particularly supported and adapted to adjust the position and / or orientation of the microscope head (4); At least one display device (20;38) for displaying visual content, The system includes a control unit (22) adapted to process the microscope image (14) and appropriately control the display device (20;38) for display, The surgical microscope (2), in particular the microscope head (4), further comprises a camera system (24) having at least one ambient camera (26) preferably firmly positioned on the microscope head (4), particularly on the housing (6), The surrounding camera (26) is configured such that, in particular, the orientation of its optical camera axis (28) and the camera optics allow the surrounding camera (26) to detect both the region (B) targeted by the optics (8) and the surrounding area (U) around the targeted region (B), and to provide a surrounding image (30). In this configuration, the field of view (36) of the surrounding camera (26) is adapted to include the field of view (34) of the optics (8), particularly from a predetermined distance. A surgical assistance system (1), characterized in that the control device (22) is further adapted to process the ambient image (30), generate a composite view (32) having the microscope image (14) and the ambient image (30), and output it visually by the display device (20;38). (Item 2) The surgical support system (1) further comprises a storage unit (42) having preoperative 3D image data, particularly MRI image data and / or CT image data. The surgical assistance system (1) according to item 1, characterized in that the assistance system (1), in particular the control unit (22), is adapted to spatially detect the 3D structure of the patient (P), in particular the 3D structure of the patient (P)'s face, using the camera system (24), in particular the ambient camera (26), and to associate the detected 3D structure with the 3D image data in order to register the patient (P). (Item 3) To detect 3D structures, the camera system (24) has a 3D camera and / or a 2D camera, and in particular the surrounding camera (26) is composed of the only 2D or 3D camera of the camera system (24). In the case of the 2D camera, the 2D camera is moved over the patient's (P) region of interest to obtain various views of the region of interest, and the control unit (22) is adapted to calculate a 3D structure from the various views using image analysis, characterized in that, as described in item 1 or item 2, the surgical assistance system (1). (Item 4) The auxiliary system (1) is configured to detect the position, particularly the pose, of the microscope head (4) and provide it to the control unit (22). The control unit (22) is adapted to store the position of the microscope head (4), particularly the pose, in the storage unit (42) as a registered position and associated surrounding image (30), particularly as a detected 3D structure. A surgical assistance system (1) according to item 2 or item 3, characterized in that when updating the surrounding image (30) with the stored registered position or pose, the control unit (22) determines that a misalignment exists if the superposition of the stored surrounding image (30) and the newly detected surrounding image (30), particularly the superposition of a partial region, is misaligned by more than a predetermined threshold. (Item 5) The surgical microscope (2) is equipped with at least one actuator to actively move the movable microscope arm (16) and the movable microscope head (4), The surgical assistance system (1) according to one of the preceding items, further characterized in that the control device (22) is further adapted to control the actuator to actively move the microscope head (4) to a predetermined position and / or orientation. (Item 6) The auxiliary system (1) includes an input unit, particularly a touchscreen, for selecting a focus point in the surrounding image (30). The surgical assistance system (1) according to item 5, characterized in that the control unit (22) is adapted to actively control the microscope arm (16) and the microscope head (4) via at least one actuator based on the selected focus point such that the optical microscope axis (10) is aligned with the selected focus point in the patient (P) in particular at a predetermined related distance and / or a predetermined image angle. (Item 7) The control unit (22) is configured to detect, via image analysis of the surrounding image (30) and / or sensors provided on the microscope head (4), an object within the area of ​​the microscope head (4) and the distance of the movable microscope head (4) to the object, and to issue an alarm signal and / or restrict at least the degrees of freedom of movement of the microscope head (4), in particular to stop the overall movement of the microscope head (4), as described in one of the preceding items (1). (Item 8) The surgical assistance system (1) according to one of the preceding items, characterized in that the control unit (22) is adapted to correctly detect, in particular to continuously track, the pose of the medical instrument (44) and / or the microscope head (4) using the camera system (24), in particular at least the ambient image (30), in particular the marker and / or the patient (P) features. (Item 9) Data for at least one medical device (44), particularly its geometric relationships and associated usage instructions, are stored in the memory unit (42). The surgical assistance system (1) according to one of the preceding items, characterized in that the control unit (22) is adapted to detect the surgical instrument (44) in the surrounding image (30) based on the stored data, particularly based on the geometric relationship, and to output the relevant usage instructions to the user via the display device (20;38). (Item 10) A surgical assistance system (1) according to one of the preceding items, characterized in that at least one of the display devices is an OR monitor (20) and / or a head-mounted display (38) and / or a binocular system having fade-in data. (Item 11) The magnification of the optical system (8) is at least 5x, and more preferably at least 10x, and the optical system (8) preferably has a zoom function, and / or A surgical assistance system (1) as described in one of the preceding items, characterized in that the magnification of the surrounding camera (26) is a maximum of 5x, and in particular a maximum of 1x. (Item 12) In order to display a microscope image (14) and a surrounding image (30) in association, an image display method for a surgical assistance system (1) described in one of items 1 to 11, wherein the image display method is Preferably, the steps include (S1) positioning a perimeter camera (26) on the movable microscope head (4) of a surgical microscope (2), A step (S2) is performed targeting region (B) via an optical system (8) located on the movable microscope head (4), Step (S3) of generating a microscope image (14) at the magnification provided by the optical system (8) using a microscope image unit (12) located on the microscope head (4), Step (S4) of generating an ambient image (30) by an ambient camera (26) positioned on the microscope head (4), wherein the field of view (36) of the ambient camera (26) includes, in particular, the field of view (34) of the optical system (8) from a predetermined distance, Step (S5) of generating a composite view (32) including the microscope image (14) and the surrounding image (30), An image display method comprising the step (S6) of outputting the composite view (32) using a display device (20;38). (Item 13) A computer-readable storage medium comprising instructions, when executed by a computer, causing the computer to perform the method steps of item 12. (Item 14) A medical sterile space, such as an operating room (100), characterized by being equipped with a surgical support system (1) described in one of items 1 to 11. [Explanation of symbols]

[0076] 1: Surgical support systems 2: Surgical microscope 4: Microscope head 6: Housing 8:Optical system 10: Optical microscope axis 12: Microscope Image Unit 14: Microscope image 16: Microscope Arm 18: Bass 20: OR Monitor 22: Control Unit 24: Camera System 26: Surrounding Camera 28: Optical camera axis 30: Surrounding image 32: Composite View 34: Field of view of the optical system 36: Camera field of view 38: Head-mounted display / glasses 40: Transceiver Unit 42: Memory Unit 44: Medical devices 46: Marker 100: Operating room P:Patient B: Target area U: Surroundings S1: Step to position the surrounding camera. S2: Steps targeting a region S3: Step to generate a microscope image S4: Step to generate a surrounding image. S5: Step to generate a composite view S6: Step to output a composite view

Claims

1. A surgical support system used in surgical interventions on patients, wherein the support system is A surgical microscope, A movable microscope head, including a housing and an optical system adapted to provide optical magnification of the area of ​​interest in the direction of the optical microscope axis and to generate a digital microscope image via a downstream microscope image unit, A microscope comprising: a movable microscope arm connected to a base, on which a movable microscope head is mounted, and which is adapted to adjust the position and / or orientation of the microscope head; At least one display device for displaying visual content, The system comprises a control unit adapted to process the microscope image and appropriately control the display device for display, The surgical microscope further comprises a camera system having at least one ambient camera positioned on the microscope head, The surrounding camera is configured such that its field of view includes the field of view of the optical system in order to detect both the region targeted by the optical system and the surrounding area around the targeted region, and to provide a surrounding image. The control unit is further adapted to process the ambient image, generate a composite view having the microscope image and the ambient image, and output it visually by the display device. The surgical support system further comprises a storage unit having preoperative 3D image data, A surgical assistance system characterized in that the control unit of the assistance system is adapted to spatially detect the 3D structure of the patient using the camera system and to associate the detected 3D structure with the 3D image data in order to register the patient.

2. To detect a 3D structure, the camera system includes a 3D camera and / or a 2D camera. The surgical assistance system according to claim 1, characterized in that, in the case of the 2D camera, the 2D camera is moved over the patient's region of interest to obtain various views of the region of interest, and the control unit is adapted to calculate a 3D structure from the various views using image analysis.

3. The auxiliary system is configured to detect the position of the microscope head and provide it to the control unit. The control unit is configured to store the position of the microscope head in the storage unit as a registered position and associated ambient image. The surgical assistance system according to claim 1, characterized in that when updating the surrounding image at the stored registered position, the control unit determines that a misalignment exists if the superposition of the stored surrounding image and the newly detected surrounding image is misaligned by more than a predetermined threshold.

4. The surgical microscope is equipped with at least one actuator to actively move the movable microscope arm and the movable microscope head. The surgical assistance system according to claim 1, further characterized in that the control unit is further adapted to control the actuator to actively move the microscope head to a predetermined position and / or orientation.

5. The auxiliary system includes an input unit for selecting a focus point in the surrounding image. The surgical assistance system according to claim 4, characterized in that the control unit is adapted to actively control the microscope arm and the microscope head via at least one actuator based on the selected focus point, such that the optical microscope axis is aligned with the selected focus point in the patient at a predetermined associated distance and / or a predetermined image angle.

6. The surgical assistance system according to claim 1, characterized in that the control unit is adapted to detect an object within the area of ​​the microscope head and the distance of the movable microscope head to the object via image analysis of the surrounding image and / or a sensor provided on the microscope head, and to emit an alarm signal and / or to limit at least the degree of freedom of movement of the microscope head (4) if there is a possibility of collision with the object or if the distance falls below a predetermined distance.

7. The surgical assistance system according to claim 1, characterized in that the control unit is adapted to correctly detect and continuously track the pose of a medical instrument and / or the microscope head using at least the ambient image and markers and / or patient features of the camera system.

8. Data containing the geometric relationships of at least one medical device, along with associated usage instructions, are stored in the memory unit. The surgical assistance system according to claim 1, characterized in that the control unit is adapted to detect surgical instruments in the surrounding image based on the geometric relationships of the stored data, and to output the relevant usage instructions to the user via the display device.

9. The surgical assistance system according to claim 1, characterized in that at least one of the display devices is an OR monitor and / or a head-mounted display and / or a binocular system having superimposed data.

10. The magnification of the optical system is at least 5x, and the optical system has a zoom function, and / or The surgical assistance system according to claim 1, characterized in that the magnification of the surrounding camera is up to 5x.

11. When executed by a computer, the computer will A step of targeting a region through an optical system located on a movable microscope head, A step of generating a microscope image at the magnification provided by the optical system using a microscope image unit located on the microscope head, A step of generating a surrounding image using a surrounding camera positioned on the microscope head, wherein the field of view of the surrounding camera includes the field of view of the optical system. A step of generating a composite view including the microscope image and the surrounding image, The steps of outputting the composite view using a display device, and The three-dimensional structure of the patient is spatially detected by the surrounding camera, and the detected three-dimensional structure is associated with the patient's three-dimensional image data in order to register the patient. A computer-readable storage medium equipped with instructions to perform the following steps.

12. An operating room, characterized by comprising the surgical support system described in claim 1.

13. The ambient camera of the camera system comprises an optical camera axis and a camera optical system, The surgical assistance system according to claim 1, characterized in that the microscope head is adapted such that the field of view of the peripheral camera includes the field of view of the optical system, via the orientation of the optical camera axis and the camera optical system.

14. The surgical assistance system according to claim 1, characterized in that the assistance system is adapted to spatially detect the patient's face as the patient's 3D structure.

15. The surgical assistance system according to claim 1, characterized in that, in order to register the patient, the detected 3D structure is associated with the 3D image data using only the surrounding camera.

16. The surgical assistance system according to claim 1, characterized in that the surrounding camera is securely attached to the housing.

17. The surgical assistance system according to claim 2, characterized in that the camera system is composed of the surrounding camera as the sole 2D or 3D camera of the camera system.

18. The auxiliary system is configured to detect the pose of the microscope head and provide it to the control unit. The control unit stores the pose of the microscope head in the storage unit as a registered pose and the associated surrounding image in the form of the detected 3D structure. The surgical assistance system according to claim 3, characterized in that when updating the surrounding image with the stored registered pose, the control unit determines that a misalignment exists if the overlapping portion of the stored surrounding image and the newly detected surrounding image is misaligned by more than a predetermined threshold.

Images