Method for providing different stereo image output signals using a medical visualisation system, medical visualisation system, and computer program product

Abstract

The invention relates to a method for providing different stereo image output signals (SBA1, SBA2) using a medical visualisation system (4), the method comprising the following steps: a. receiving at least one first stereo image signal (SB1) which has been generated by a stereo system (22) of a surgical microscope (10) and represents a three-dimensional image of an examination region (1); b. providing the first stereo image signal (SB1) as a first stereo image output signal (SBA1) or processing the first stereo image signal (SB1) to form a first stereo image output signal (SBA1); c. providing a further stereo image output signal (SBA2) which represents a further three-dimensional image of the examination region (1), wherein the first stereo image output signal (SBA1) is provided for the complete display of an image according to a first visualisation modality (VM1), and the further stereo image output signal (SBA2) is provided for the complete display of an image according to a further visualisation modality (VM2), wherein the first and the further visualisation modalities (VM1, VM2) are visualisation modalities (VM1, VM2) that are different from one another and are viewer-specific. The invention also relates to a medical visualisation system (4) and to a computer program product.

Description

[0001] Method for providing various stereo image output signals through a medical visualization system, medical visualization system and computer program product

[0002] The invention relates to a method for providing different stereo image output signals by means of a medical visualization system, a medical visualization system and a computer program product.

[0003] Surgical microscopes are used, among other things, to prepare for and perform medical operations on a patient. Such surgical microscopes are used by a user, e.g., a surgeon or an assistant, during a procedure to provide a magnified view of an area of ​​examination, particularly in or on the patient's surgical site. For this purpose, a surgical microscope may include an objective lens or a lens system to produce a true optical image of the area of ​​examination. The objective lens may include optical elements for beam guidance, shaping, and / or direction. An optical element may, in particular, be a lens.

[0004] Surgical microscopes are used in medical facilities, as well as in laboratories and industrial applications. Examples of medical applications include neurosurgery, ophthalmic surgery, otolaryngology (ENT), plastic and reconstructive surgery, and orthopedic surgery. This list is not exhaustive. Generally, they are used in all areas of surgery where a magnified, high-resolution view of the surgical field is required to perform precise procedures.

[0005] A distinction can be made between analog and digital surgical microscopes. Unlike digital surgical microscopes, analog surgical microscopes do not capture images that are then displayed, for example, on a screen to magnify the examination area. Instead, they offer the user a direct, visually perceptible magnification of the examination area. Here, radiation reflected or scattered from the area of ​​application passes through the objective lens into at least one beam path and to at least one output section, through or into which the user looks to visually perceive the radiation and thus also the typically magnified representation of the examination area. An exemplary embodiment of an output section is a so-called eyepiece, into or through which the user looks to optically perceive the examination area with at least one eye.

[0006] Digital surgical microscopes comprise, or at least include, an image acquisition device for microscopic imaging that captures radiation in a beam path of the surgical microscope to generate a magnified image. This image can be displayed to the user or multiple users on one or more display devices. This enables high-resolution visualization. The image can be generated as a transmittable image signal, which encodes or represents the image. Purely digital surgical microscopes, unlike analog surgical microscopes, do not have an output section for visually detectable radiation, specifically no eyepiece. The image signal can then be transmitted as a data signal, either wired or wirelessly.Digital surgical microscopes enable the capture, storage, and further processing of images and videos. By applying image processing techniques, contrast, brightness, and other parameters can be adjusted to optimize the image quality of the generated images. Hybrid surgical microscopes can include at least one image acquisition unit and at least one output section. For example, the radiation guided in a beam path of the surgical microscope can be split by a beam splitter, with one portion directed to the output section and another portion captured by the at least one image acquisition unit.

[0007] Stereoscopic surgical microscopes are also well-known. These typically include two separate beam paths for beam guidance and provide the user with a depth perception of the examination area. The beams guided in the two beam paths can be visually detected by the user via output sections. Digital surgical microscopes alternatively or additionally include two image acquisition units, each capturing the beams in one of the beam paths to generate an image. Based on these two images, which can also be referred to as corresponding images, a three-dimensional image is then provided to the user via a suitable display device. The image acquisition units are components of a stereo (camera) system.

[0008] The operating microscope can constitute a medical visualization system, or the medical visualization system can encompass the operating microscope. The components of the medical visualization system described below can be components of the operating microscope or components designed differently from the operating microscope.

[0009] Another known method is the provision of an augmented representation of the examination area to a user. An augmented representation can, in particular, be a representation of the real examination area that is enhanced by computer, especially by adding or overlaying at least one virtual object and / or other additional information onto the representation of the real examination area. The augmented representation can be displayed to a user as an augmented image on a display device or provided to them in a visually perceptible manner via an output section.

[0010] To generate the augmented image, the additional information can be introduced into the beam path, for example, by reflection. This information can be projected onto a projection element, such as a radiolucent disc, positioned in the beam path using a projection device of the operating microscope. The augmented image can then be generated by creating an image based on the beams into which the additional information has been introduced as described. Alternatively or additionally, the radiation representing the augmented image can also be provided via an output section for visual perception by a viewer.

[0011] Alternatively, an augmented image can be generated by computer-aided enhancement of an image of the real examination area, particularly through image processing. In this process, the additional information can be superimposed onto the image of the real examination area. With a stereoscopic operating microscope, it is possible to provide the user with two augmented representations. Generally, corresponding additional information can be introduced into each of the two beam paths of a stereoscopic operating microscope. With digital stereoscopic operating microscopes, for example, augmented images can be generated from the images produced by both image acquisition units. Thus, an augmented image with depth information—that is, an augmented three-dimensional representation—can be provided to the user on a suitable display device or via an output section.

[0012] Additional information can be provided in the form of data that represents or encodes a geometric description of a space, particularly a three-dimensional space, especially one containing objects arranged within it. Additional information can also be information generated from such data, for example, information produced by rendering. Rendering, or image synthesis, is the process of computer-implemented generation of a photorealistic or non-photorealistic image from a 2D or 3D model. Multiple models can be defined in a scene file, which contains objects in a defined language or data structure. The scene file can contain geometry, viewpoint, texture, lighting, and shading information that describes the virtual scene. The data contained in the scene file is then passed to a rendering program, which processes it and outputs it to a digital image or raster graphics file.A software application or component that performs the rendering is called a rendering engine, rendering system, graphics engine, or simply a renderer.

[0013] A key requirement for augmentation is that the viewer of the augmented image, particularly a surgeon, is not disturbed or distracted by the augmentation during their work. It is especially desirable that a surgeon can operate ergonomically even when viewing an area with superimposed augmentation. For example, it is crucial that an object, such as a tumor, located spatially behind a surface of the examination area along a certain line of sight, is represented in the augmented image in such a way that it does not mistakenly obscure objects located in front of the surface, as this can impair the viewer's spatial perception. Furthermore, this can distort the stereoscopic depth perception when viewed through a stereoscopic operating microscope.This can be particularly relevant when augmented objects are superimposed in a stereoscopically perceptible manner, as the resulting three-dimensional perception can be confusing for the viewer. If several people are present who wish to view an image of the examination area, e.g., a lead surgeon, a second surgeon, an assistant, and / or a surgical nurse, it may be desirable to provide a separate image for each person.

[0014] US Patent 2022 / 079675 A1 discloses devices and methods for performing a surgical step or procedure with visual guidance using an optical head-mounted display (OHMD) with focus plane adjustment. Further disclosed is a system configured for use with multiple OHMDs for multiple viewers. The system is configured so that all viewers view a single display through their OHMD. A disadvantage of the solution described in US Patent 2022 / 079675 A1 is that while a viewer can access individually desired information through a viewer's OHMD, this is only possible if they, like all other viewers, are viewing a common monitor / display through their OHMD. This results in the information displayed in the OHMD being superimposed on the image shown on the common display.The information displayed on the screen is identical for all viewers and is not personalized. Furthermore, the accuracy of information overlay in the OHMD display is limited, which can lead to inaccurate representations of augmented information in certain scenarios.

[0015] The technical problem therefore arises of creating a method for providing different stereo image output signals through a medical visualization system, as well as a medical visualization system and a computer program product, which enables an improved, in particular space-saving, display of information that is individualized for multiple viewers, while also ensuring high accuracy of the display.

[0016] The solution to the technical problem is provided by the objects with the features of the independent claims. Further advantageous embodiments of the invention are described in the dependent claims. A method for providing various stereo image output signals by a medical visualization system is proposed. The medical visualization system can be configured as described above. In particular, the medical visualization system can be or include a stereo operating microscope. Operating microscopes and their technical features have also been briefly described above. The operating microscope can also be configured as an endoscope, in particular a stereo endoscope.

[0017] An operating microscope can comprise a microscope body. The objective lens described above can be integrated into or attached to the microscope body, particularly in a detachable manner. The objective lens can be fixed in position relative to the microscope body. In addition to the objective lens, the microscope body can also have or incorporate at least one beam path for microscopic imaging and / or other optical elements for beam guidance, shaping, and / or deflection. In analog and hybrid operating microscopes, the microscope body can include at least one mounting interface for attaching an output element, such as an eyepiece, in particular in a detachable manner. The microscope body can comprise or form a housing, or be arranged within a housing. Components of the operating microscope, such as an image acquisition device for microscopic imaging, can be arranged in or on the housing.

[0018] The medical visualization system can include a stand for mounting the operating microscope. The operating microscope, in particular the microscope body, can be mechanically attached to the stand. The stand is designed to allow movement of the operating microscope in space, in particular with at least one degree of freedom, preferably with six degrees of freedom, where one degree of freedom can be translational or rotational. Furthermore, the stand can include at least one drive unit for moving the operating microscope. Such a drive unit can, for example, be a servo motor. Naturally, the stand can also include means for force / torque transmission, e.g., gear units.In particular, it is possible to control the at least one drive unit in such a way that the operating microscope performs a desired movement and thus a desired change of position in space, or assumes a desired position and / or orientation in space. For example, the at least one drive unit can be controlled in such a way that an optical axis of an objective lens of the operating microscope assumes a desired orientation. Furthermore, the at least one drive unit can be controlled in such a way that a reference point of the operating microscope, e.g., a focal point, is positioned at a desired position in space. A target position can be specified by a user or another higher-level system. Methods for controlling the at least one drive unit as a function of a target position and a kinematic structure of the stand are known to those skilled in the art.

[0019] Furthermore, the medical visualization system can include one or even several display devices for showing the images. The display device can be used to show two- or three-dimensional images. Typical display devices are screens, especially 3D screens, head-mounted displays (HMDs), or digital eyepieces, which can also be referred to as booms.

[0020] Furthermore, the medical visualization system, in particular the operating microscope, may include one or more of the following elements:

[0021] • at least one white light lighting device,

[0022] • at least one infrared lighting device,

[0023] • at least one fluorescence illumination device for excitation of

[0024] Fluorescence radiation,

[0025] • at least one beam filter to provide excitation radiation with wavelengths from a broader spectrum, e.g. the spectrum of the white light illumination device,

[0026] • at least one fluorescence detection device for detecting fluorescence radiation,

[0027] • at least one filter device for filtering radiation from a broader spectrum, e.g. for detection by an image acquisition device for microscopic imaging,

[0028] • at least one image acquisition device of an optical position detection device, which can also be referred to as a surrounding camera,

[0029] • at least one gaze direction detection device,

[0030] • at least one position detection device for determining a pose, i.e., a position and / or orientation, at least of the operating microscope • at least one input device for operation,

[0031] • at least one interface for data transmission to or from another system or facility,

[0032] • at least one device for determining depth information, in particular with regard to the elements arranged in the detection range of the operating microscope, which may, for example, be designed as a distance sensor,

[0033] • at least one storage device for storing signals and / or information, in particular in a retrievable manner,

[0034] • a device for identifying a viewer.

[0035] An image acquisition device may, in particular, include a CMOS or CCD sensor. The detection range of the ambient camera may fully or at least partially encompass the detection range of the surgical microscope. Alternatively, the detection range of the surgical microscope may fully or at least partially encompass the detection range of the ambient camera.

[0036] In a fluorescence visualization mode, a filter device can be inserted into an observation beam path, providing the viewer with a filtered representation of the examination area. This radiation can be captured by the at least one image acquisition device for microscopic imaging. Alternatively, fluorescence radiation can also be captured by a separate acquisition device, such as a spectral camera. The fluorescence mode advantageously enables intraoperative tissue differentiation. Tumor tissue, in particular, can be visualized using fluorescence-based images. Nerve tissue, in particular, can be visualized using polarization-contrast-based images.

[0037] Medical visualization systems can be operated, for example, by manually controlling a component, particularly the operating microscope, or a corresponding input device; by voice control; by gesture control; by eye-tracking; by image-based control; or by other operating methods. The medical visualization system or the operating microscope may include the necessary components. Image-based control may, in particular, include the generation of operating or control signals by evaluating at least one image produced by an image acquisition device for microscopic imaging or by an image acquisition device of an optical position detection system.

[0038] Adjustable operating parameters of the medical visualization system or the surgical microscope can be formed by one or more of the following parameters:

[0039] • Magnification factor or zoom factor,

[0040] • Working distance or focus position,

[0041] • Detection range,

[0042] • Light intensity,

[0043] • Illumination spectrum.

[0044] The method comprises the following steps: a) Receiving at least one first stereo image signal generated by a stereo system of the operating microscope, representing a three-dimensional image of an examination area. A three-dimensional image can, in particular, be or comprise a stereo image pair, wherein the images of this image pair are stereoscopic images. The image signal can be received via a suitable interface, in particular by an evaluation unit of the medical visualization system. A stereo image signal can, in particular, encode or represent two corresponding images, i.e., a stereo image pair, generated by the image acquisition devices of a stereo system of the medical visualization system. b) Providing the first stereo image signal as a first stereo image output signal or processing the first stereo image signal into a first stereo image output signal.A stereo image output signal encodes or represents a three-dimensional image, in particular a stereo image pair comprising two corresponding images of a stereo system intended for output or display by a 3D display device. A 3D display device can be, for example, a digital eyepiece, a 3D screen, or a head-mounted display (HMD). The stereo image output signal can, in particular, encode or represent two corresponding two-dimensional images, each displayed by a separate display device. It is conceivable that the corresponding images are output sequentially by the same display device or simultaneously on different display devices. The stereo image output signal can, in particular, represent a magnified three-dimensional image of the examination area.

[0045] Processing the initial stereo image signal can involve applying image processing techniques to generate the stereo output signal from the stereo image signal, potentially taking into account additional signals or information. Such an image processing technique could, for example, be a filtering method. Augmentation can also occur during processing; for instance, additional information can be superimposed on the three-dimensional image, and the resulting image can then be provided as the initial stereo output signal. As explained in more detail below, processing can also include generating a three-dimensional reconstruction of the imaged area and creating the initial stereo output signal based on this reconstruction.

[0046] In particular, a virtual (3D) image or an augmented (3D) image can depict visible and / or hidden objects or elements, thus enabling their visual representation. c) Providing an additional stereo image output signal that represents another three-dimensional image of the area under investigation. This additional three-dimensional image can, in particular, depict the same area under investigation and be represented by another stereo image signal.

[0047] In particular, a further stereo image signal can be provided as a further stereo image output signal. This further stereo image (output) signal can be described as a stereo image (output) signal corresponding to the first stereo image (output) signal. The further three-dimensional image represented by the further stereo image output signal can depict the area under investigation with the same properties as the three-dimensional image represented by the first stereo image output signal, in particular with the same magnification properties.

[0048] The additional three-dimensional image can be generated by image acquisition devices, such as those of another stereo system of the operating microscope. This additional three-dimensional image can also be a white light image, a fluorescence contrast image, a polarization contrast image, an image generated from additional information, or an image combined from at least two of these modalities, for example, through augmentation. An image generated through augmentation can be displayed, in particular, by an AR (Augmented Reality) display device. An image generated purely from additional information can be displayed, in particular, by a VR (Virtual Reality) display device. It is possible that such images are generated by the image acquisition devices of the stereo system or by different (image) acquisition devices of the medical visualization system, especially the operating microscope.

[0049] The further three-dimensional image can also be simulated. For example, it can be a virtual image generated by virtual image acquisition devices, which can be, in particular, optical models of the image acquisition devices of the stereo system. Specifically, the further stereo image output signal can be generated depending on additional information.

[0050] It is also possible to process the first stereo image signal into a further stereo image output signal, which is then provided as a further stereo image output signal. Such processing was explained previously.

[0051] The corresponding stereo image signal can be generated, for example, by another stereo system of the medical visualization system, particularly the operating microscope, and represents a three-dimensional image produced by this additional stereo system. The corresponding stereo image signal can also be generated from supplementary information, as previously explained. If the additional stereo image output signal is provided by processing the first stereo image signal, the method for providing it can differ from the method for providing the first stereo image signal from the first stereo image signal. If the same procedure is performed, it can be carried out in different ways, particularly with different adjustable parameters that determine the processing.

[0052] Image signals, in particular the image output signals, can then be transmitted to a display device, which is then controlled to output the image represented by the image (output) signal in a visually perceptible manner.

[0053] Additional information that can be displayed to a user in the form of an image or, for example, through augmentation, can include preoperatively generated information, such as preoperatively generated data, which can also be used for surgical planning. Such preoperatively generated data can be, in particular, volumetric data. Volumetric data can be provided as a point cloud, a voxel-based representation, or a mesh-based representation. The additional information can also be provided, in particular, as a transmittable signal.

[0054] Preoperative data can be generated, for example, using computed tomography or magnetic resonance imaging. Other imaging techniques, particularly ultrasound, X-ray, fluorescence, SPECT, or PET, can also be used. Such augmentation allows, for example, a tumor object or its contours, generated from preoperative information, to be superimposed on a white light image. Of course, it is also possible to display such preoperative information on its own, without augmentation.

[0055] Preoperative data generated using magnetic resonance imaging (MRI) can identify various tissue types, such as adipose tissue, muscle tissue, tumor tissue, as well as blood vessels and nerve pathways. Preoperative data generated using computed tomography (CT) can particularly depict bony structures.

[0056] Alternatively or additionally to using preoperatively generated information, intraoperative information—that is, information acquired or generated during treatment—can be used as supplementary information. For example, information can be collected and stored during surgery and subsequently used to display an image or to generate an augmented image. This is particularly advantageous when different visualization modalities are activated at different times. For instance, in a fluorescence visualization modality, fluorescence information can be displayed as an image or superimposed on a white light image.

[0057] The additional information can be assigned a reference coordinate system, which means that the additional information can also include spatial information. This reference coordinate system can also be called a world coordinate system.

[0058] The first stereo image output signal is provided for the complete display of a three-dimensional image according to a first visualization modality. The second stereo image output signal is provided for the complete display of a three-dimensional image according to a further visualization modality, wherein the first and the further visualization modalities are distinct and viewer-specific. A complete display can be the display of the complete three-dimensional image represented by the stereo image signal according to the viewer-specific visualization modality, in particular by a single display device. This can especially mean that no more than one display device is required for the complete display according to the corresponding visualization modality.

[0059] A visualization modality refers to the way in which an image is displayed and / or what information is represented in the image. In particular, a visualization modality can include a display with predetermined color properties and / or predetermined transparency properties.

[0060] According to a visualization modality, for example, a white light image, a fluorescence contrast image, a polarization contrast image, an image generated from additional information, or an image combined from these modalities, in particular an augmented image, can be displayed on a display device. A white light image (VIS image) is, in particular, an image generated on the basis of radiation with wavelengths in the range of 360 nm to 780 nm. A fluorescence contrast image is, in particular, an image generated with fluorescence-specific wavelengths, in particular with predetermined fluorescence-specific wavelengths or wavelength ranges, for example, with wavelengths of 400 nm or 560 nm. A polarization contrast image is, in particular, an image generated with radiation of a predetermined polarization.An image generated from additional information has already been explained in more detail and can, in particular, be an image generated from preoperative data information, especially when created with a virtual image acquisition device. An image combined from at least two modalities can, in particular, be an image in which an image according to a first visualization modality is superimposed on an image according to another visualization modality, especially in a partial area. Thus, a combined image can be an augmented image.

[0061] Images can also be generated in different ways, tailored to different viewer preferences, to suit various visualization modalities. This can be achieved by rendering using different rendering styles. These styles can include: transparent rendering (e.g., alpha-blending), animation-based rendering, texture-based rendering, and Fresnel-effect-based rendering.

[0062] To provide stereo image output signals, it may be necessary to identify different viewers. This can be achieved using a viewer identification device and appropriate identification methods. Alternatively, different viewers can register with the medical visualization system using their viewer-specific identity information, for example, by operating a suitable input device. Image-based viewer identification is also conceivable. Image-based identification can mean that viewer identification is performed by evaluating at least one image, where the image can depict the operating room or a part thereof, and the viewer is positioned within the depicted area.It is also possible to use marker-based identification, whereby a marker assigned to a viewer is detected, particularly in an optical manner. Specifically, a viewer identity can be assigned to a marker identity, for example, in the form of a computer-evaluable assignment.

[0063] A viewer identity can be assigned a viewer-specific visualization modality. Information about this assignment can be stored in a retrievable manner, particularly in a storage device of the medical visualization system or in a storage device of an external system. A viewer-specific visualization modality can be a visualization modality defined specifically for the viewer. This can be set, for example, by operating an input device, particularly in a manner associated with the viewer identity. This setting can be made preoperatively. However, it is also possible for the viewer-specific visualization modality to be set, particularly changed, intraoperatively. This can also be done by operating a corresponding input device.

[0064] To provide the stereo image output signals, it may be necessary to identify a viewer-specific display device on which the viewer-specific stereo image output signal is to be displayed. This can also be assigned to the viewer's identity. Suitable identification methods can also be used for this purpose. For example, it is conceivable to identify a viewer-specific display device depending on the viewer's pose and / or viewing direction, which can be determined as explained below. For example, depending on the pose and / or viewing direction, it can be determined which display device the viewer is currently looking at and identified as the viewer-specific display device. For this purpose, it may also be necessary to determine the pose of a display device, e.g., with a tracking system.

[0065] It is also possible for different users to log in to the medical visualization system with information about their user-specific display device, for example, by using a suitable input device. Information about the assignment of a user identity to a user-specific display device can also be stored in a retrievable manner. Depending on the user identity and, if necessary, by evaluating the assignment(s) described above, a user-specific visualization modality can thus be determined.

[0066] It is also possible to determine or record a viewer-specific pose, as well as a viewer-specific direction of gaze. This will be explained in more detail below.

[0067] Overall, this advantageously results in a simple and viewer-specific provision of various stereo image output signals, which can then be fully and individually output for a viewer by a common or by different, particularly viewer-specific, 3D display devices. In particular, it is not necessary to provide an individualized display by having the viewer look at a screen through an overhead display (OHMD). This also achieves improved display accuracy.

[0068] In another embodiment, a white light image, a fluorescence contrast image, a polarization contrast image, an image generated from additional information, or an image combined from these modalities is displayed according to a visualization modality. This has already been explained above. In particular, the different visualization modalities allow viewers to be provided with individually desired information. For example, a white light image enables good orientation, while a fluorescence contrast image is particularly suitable for improved visualization of, for example, tumorous tissue. A polarization contrast image is suitable for visualizing nerve tissue. An image generated from additional information also advantageously allows the display of preoperative or further intraoperative information.

[0069] In a further embodiment, a viewer's pose and / or viewing direction is determined, and a viewer-specific stereo image output signal is provided depending on the pose and / or viewing direction. The described position detection device, which can also be referred to as a tracking system, can be used to determine the pose. A tracking system can be optical, electromagnetic, or operate in another way. The tracking system can be a marker-based tracking system that detects active or passive markers. Markers can be arranged on objects or subjects whose pose is to be detected by the tracking system. An optical tracking system can, in particular, include optically detectable markers. An optical tracking system can, in particular, be a monoscopic position detection system.The pose of an object can be determined by evaluating a two-dimensional image, specifically a single two-dimensional image. In particular, the position can be determined by evaluating the intensity values ​​of pixels (picture elements) of the two-dimensional image. Furthermore, it is conceivable that the medical visualization system includes at least one image acquisition device or optical position detection device, which can be a component of the operating microscope. This can also be referred to as a peripheral camera and is particularly useful for monoscopic position detection. A tracking system can also be part of an input device, where, for example, gesture control or gaze direction control is implemented based on information generated by the tracking system.

[0070] The pose of a viewer can therefore be captured, in particular by a tracking system, whereby at least one marker can be placed on the viewer. It is also conceivable, for example, to determine the pose of a body part of such a user, e.g., a hand, an arm, or a head. The position detection device can be a so-called registered device. This can mean that the pose information can be determined with respect to a reference coordinate system, in particular a common reference coordinate system.

[0071] Naturally, it is possible for the position detection device, or another position detection device, to also detect the pose of at least one other object or part thereof. An object can be, in particular, another component of the medical visualization system, especially a display device. However, an object can also be an item that is not part of the medical visualization system, e.g., a piece of equipment such as an operating table or a medical instrument. This makes it possible to determine the pose of the additional subject or object within a desired reference coordinate system.

[0072] The gaze direction of a viewer can be determined using a gaze direction detection device, particularly image-based methods. For this purpose, the medical visualization system can include an image acquisition device, which is, for example, assigned to and / or designed to image the eyes of a viewer. The gaze direction can then be determined by evaluating the generated images. Such methods are known to those skilled in the art. The gaze direction detection device itself can also be a registered device; in particular, the gaze direction information can be determined in a reference coordinate system that differs from a device-specific coordinate system.

[0073] In particular, a stereo image output signal can be provided in such a way that the represented three-dimensional image is displayed with correct perspective, taking the microscopic image into account. In other words, the three-dimensional image can then be displayed as if the respective viewer, with their specific pose and / or viewing direction, were visually perceiving the examination area through the operating microscope. For this purpose, it may also be necessary to consider the pose of the operating microscope itself. Its determination will be explained in more detail below.

[0074] If a pose of the operating microscope is also determined, for example also with the tracking system, the viewer-specific stereo image output signal can be generated depending on the viewer's pose relative to the operating microscope.

[0075] For example, it is conceivable that different observers, from different positions and / or with different viewing angles, might wish to visually perceive the examination area magnified by the operating microscope. In other words, microscopic image information can be displayed depending on the observer's pose relative to the operating microscope.

[0076] The proposed approach then enables each viewer to see a perspective-correct representation of the three-dimensional images represented by the stereo image output signals, according to the viewer-specific visualization modality. It is also conceivable that, in different visualization modalities, viewers with different poses and / or viewing directions are each shown a pose- and / or viewing-direction-dependent image, as explained. In a further embodiment, the provision of the viewer-specific stereo image output signal includes the generation of virtual images by virtual image acquisition devices, which are an optical model of the image acquisition devices of the stereo system. Virtual images can be generated, in particular, based on additional information, especially preoperatively and / or intraoperatively generated additional information.A virtual image can encode a texture, especially with color and / or transparency information.

[0077] For example, virtual image acquisition devices can each generate an additional information signal, representing a virtual image created from or by the additional information. The virtual image acquisition devices can be optical models of the stereo system's image acquisition devices. This advantageously generates a perspectively correct and three-dimensionally perceptible representation of the additional information.

[0078] The additional information can be assigned a reference coordinate system, which means that the additional information can also include spatial information. This reference coordinate system can also be called a world coordinate system.

[0079] For this purpose, it may be necessary or useful to perform a registration between the reference coordinate system of the additional information and a reference coordinate system of the medical visualization system, in particular the operating microscope or an image acquisition device of the operating microscope. This registration can be carried out before the virtual images are generated. The registration establishes a spatial relationship between both the additional information and an image of the examination area, especially the three-dimensional image of the examination area, to a common reference coordinate system. This common reference coordinate system, which is also referred to below as the reference coordinate system, can be, in particular, the reference coordinate system of the additional information, the reference coordinate system of the medical visualization system, or a different reference coordinate system.Methods for registration are known to those skilled in the art. For example, model-based registration can be performed. In this case, features can be detected in an image that correspond to previously known features, e.g., geometric features in the additional information, and the registration can then be determined in a known manner depending on these corresponding features. The registration can, for example, be determined in the form of a transformation matrix that includes a rotation and / or translation component. An exemplary model-based registration can be an edge-based registration, where the corresponding features are formed, for example, by a property of at least one, preferably several, edges in both the image and the additional information. Topography-based registration can also be performed, particularly if a topography can be determined, e.g.,with a stereo system of an operating microscope. In this way, topographic information can be determined in at least one image, whereby corresponding features or points or sections are detected in the additional information as well as in this topographic information, which can then be used to determine the registration.

[0080] The additional information can therefore be registered information. For the purposes of this invention, the property "registered" can mean that a spatial reference to the reference coordinate system is known, in particular in the form of a transformation matrix. A registered device can generate signals whose spatial reference to the reference coordinate system is known.

[0081] The additional information can be generated, in particular, through rendering. For example, a virtual image can be created through rendering, which is then used to display the augmentation and can be superimposed on a live-recorded image of the real examination area. The virtual image can also be provided as an image signal, which encodes or represents the virtual image. The viewer-specific stereo image output signal can therefore consist of, or comprise, two image signals.

[0082] The virtual image can be generated using a virtual image acquisition device, which can be a mathematical or physical, and in particular computer-aided, optical model of an image acquisition device. Specifically, a computer-implemented calculation of the pixels of the virtual image can be performed. This virtual image depends, among other things, on parameters of the (modeled) image acquisition device. In particular, the virtual image can be generated based on the intrinsic parameters of the image acquisition device for microscopic imaging, especially with these parameters. The virtual images generated by a virtual stereo system can additionally be generated based on the extrinsic parameters of the two image acquisition devices of the (modeled) stereo system, especially with these parameters.

[0083] In other words, when evaluating the model to generate virtual images, the parameters of the stereo system's image acquisition device(s) used for microscopic imaging can be taken into account. This makes it possible to generate virtual images under the same conditions as real images.

[0084] The virtual image can also be generated based on a pose, i.e., a position and / or orientation, of the (modeled) image acquisition device of the stereo system. In particular, the pose of the stereo system's image acquisition device(s) can be taken into account when evaluating the model to generate the virtual images, utilizing the registration information described above. By considering the registration information, it is possible, for example, to determine which pose of the virtual image acquisition device corresponds to the actual pose of the (modeled) image acquisition device of the stereo system in the reference coordinate system of the additional information, which can also be referred to as the render coordinate system, and this information can then be used for the rendering process.In other words, a pose of at least one virtual image acquisition device can be identical to the pose of the modeled image acquisition device in the reference coordinate system. Thus, it is possible to create a virtual representation that corresponds to the modeled image acquisition device in terms of both parameters and capture pose.

[0085] For the provision of virtual images, it may also be necessary to determine the current pose of the stereo system, particularly its image acquisition devices. This pose can also be determined using the position detection device. A registration process can establish a relationship between the reference coordinate system of the position detection device and the previously described reference coordinate systems, especially the reference coordinate system of the additional information. Such registration methods are known to those skilled in the art. This makes it possible to determine the pose of the stereo system in a desired reference coordinate system, particularly the reference coordinate system.

[0086] When a pose of the stereo system or the operating microscope is determined, the pose of the objective's optical axis or the position of a focal point can also be determined. If the operating microscope is mounted on a stand with at least one joint, its pose can also be determined as a function of the joint position, which can be detected, for example, by a sensor. This, in turn, allows the determination of the viewing angle / direction of the operating microscope under which the area of ​​investigation was imaged.

[0087] The position of the stereo system and the additional information can define a previously described scene, i.e., a virtual spatial model that defines objects and their material properties, light sources, and the position and viewing direction of an observer, here the stereo system.

[0088] The described generation of virtual images advantageously enables a simple and perspectively correct representation of images generated from additional information in an individualized manner.

[0089] In another embodiment, the virtual images represent three-dimensional additional information. In particular, the virtual images can be corresponding images of a virtual stereo system. This also advantageously enables a simple and perspectively correct representation of additional information in an individualized manner.

[0090] In another embodiment, the three-dimensional additional information is generated from images of the stereo system's image acquisition devices. In particular, a reconstruction of the stereo system's detection range can be determined based on the images generated by the stereo system's image acquisition devices, and this reconstruction can be a three-dimensional, data-based representation of the detection range. For example, it is possible that corresponding images, i.e., stereo image pairs, are stored during operation of the operating microscope's stereo system and then used for the described reconstruction. For this purpose, it may be necessary to generate at least a predetermined number of corresponding images.

[0091] This reconstruction, in turn, can provide the additional information. This makes it possible, in a simple and reliable way, to present viewers in different poses with a perspectively correct image, for example, in a white light visualization mode.

[0092] In a further embodiment, the additional stereo image output signal represents another three-dimensional image, which was generated at least partially or completely by another stereo system of the medical visualization system. Thus, the medical visualization system, in particular the surgical microscope, can comprise another stereo system. At least one, but preferably both, image acquisition devices of the additional stereo system can be different from the image acquisition devices of the (first) stereo system. It is conceivable that a pose of the additional stereo system relative to the (first) stereo system can be changed. In particular, the additional stereo system can be movably mounted on / in the surgical microscope.It is also conceivable that the first and the second stereo system are arranged on different elements of the medical visualization system, for example on differently designed stands.

[0093] This advantageously enables the reliable and perspectively correct generation of images for different viewers, who, for example, each desire a representation of an image in a white light visualization modality, for different poses of these viewers.

[0094] For example, it is conceivable that the first stereo image signal is provided as a stereo image output signal and the subsequent stereo image output signal as a stereo image signal generated by the other stereo system.

[0095] In a further embodiment, the various stereo image output signals for displaying the respective represented images are provided by a common display device. On this common display device, the corresponding three-dimensional images can be displayed or output in the respective viewer-specific visualization mode for at least two different viewers.

[0096] Preferably, the shared display device is a 3D screen. As explained in more detail below, the three-dimensional representation can be achieved in various ways. The essential point is that the shared display device presents images to a first viewer in the viewer-specific first visualization modality, while the shared display also presents images to a second viewer according to a viewer-specific second visualization modality.

[0097] This advantageously saves space, as multiple viewers can use a single display to view images according to their individual visualization preferences. In contrast to head-mounted displays, this also allows the viewer to perceive their surroundings more effectively.

[0098] In a further embodiment, the first stereo image output signal for displaying the represented images is provided with a first polarization encoding, wherein the second stereo image output signal for displaying the represented images is provided with a further polarization encoding, the first and the further polarization encodings being different from each other. For example, it is possible that a first image encoded in the first stereo image output signal is provided for display with a first polarization, and a further image encoded by the first stereo image output signal is provided for display with a second polarization. Furthermore, a first image encoded by the further stereo image output signal can be provided for display with a third polarization, and a further image encoded by the further stereo image output signal can be provided for display with a fourth polarization.The first and subsequent images are images of a stereo image pair. In this embodiment, viewers may need to wear polarizing filter glasses through which the images are visually perceived on the common display device, with each lens of the polarizing filter glasses being transparent to only one of the polarizations. In this case, the first polarization encoding consists of providing the corresponding images encoded in the first stereo image output signal for display with the first and second polarizations, and the subsequent polarization encoding consists of providing the corresponding images encoded in the further stereo image output signal for display with the third and fourth polarizations. The images encoded by the stereo image output signals can be displayed simultaneously or sequentially by the common display device.This advantageously results in a simple and reliable display of the respective represented images by a common display device.

[0099] In an alternative embodiment, the additional stereo image output signal for displaying the represented images is provided at a time that is later than the time for displaying the images represented by the first stereo image output signal. In particular, it is possible to display a first image and a further image corresponding to the first image, represented by the first stereo image output signal, sequentially, especially with a predetermined time offset, using the common display device. Specifically, the first image can be displayed at a first time and the further, corresponding image at a second time.It is also possible to display the first and the subsequent, corresponding images, represented by the additional stereo image output signal, sequentially on the common display device, particularly with a predetermined time offset. Specifically, this first image can be displayed at a third point in time and the subsequent, corresponding image at a fourth point in time. The fourth point in time can be after the third, the third point in time after the second, and the second point in time after the first.In this embodiment, it may be necessary for viewers to wear so-called shutter glasses to view the shared display device. These glasses, as is known to those skilled in the art, allow viewing with the left eye while simultaneously preventing viewing with the right eye, and at a later time, prevent viewing with the left eye while allowing viewing with the right eye. Also as is known, the operation of such shutter glasses is synchronized with the display times. This advantageously results in a simple and reliable display of the respective images by the shared display device.

[0100] In a further alternative embodiment, the first stereo image output signal for displaying the represented images is provided with a first color encoding, wherein the second stereo image output signal for displaying the represented images is provided with a further color encoding, the first and the further color encodings being different from each other. Different color encodings can be based on different color bases.

[0101] A color base of a color space can be defined by a number, e.g. three, primary colors that span the color space, whereby each color point, i.e. each color, in the color space can be defined by a combination of these three primary colors.

[0102] Different color bases can preferably span the same color spaces. Two color spaces can be considered essentially the same if they overlap at least in one area, particularly in a large part of the area, for example in an area > 70%, > 80% or > 90%, and especially preferably in an area that includes visible light.

[0103] Different color bases can differ in at least one, preferably all, primary colors. The primary colors specific to each color base can be chosen such that they can be separated from one another by a suitable color filter or combination of color filters. For example, a first color filter can be transparent to the primary colors of a first color base and opaque to the primary colors of at least one other color base. A further color filter can be transparent to the primary colors of the second color base but opaque to at least the primary colors of the first color base.

[0104] For example, it is possible that a first image encoded in the first stereo image output signal is provided for display according to a first color base, and a further image encoded by the first stereo image output signal is provided for display according to a second color base. Furthermore, a first image encoded by the further stereo image output signal can be provided for display according to a third color base, and a further image encoded by the further stereo image output signal can be provided for display according to a fourth color base. The first and further images are again images of a stereo image pair. In this embodiment, it may be necessary for viewers to wear color filter glasses through which the images on the common display device are visually perceived, with the lenses of the color filter glasses being transparent only to the primary colors of each color base.Such color filter glasses can be, in particular, interference filter glasses.

[0105] In this case, the first color encoding consists of providing the corresponding images encoded in the first stereo image output signal for display according to the first and second color bases, and the subsequent color encoding consists of providing the corresponding images encoded in the further stereo image output signal for display according to the third and fourth color bases. The images encoded by the stereo image output signals can be displayed simultaneously or sequentially by the common display device. This advantageously results in a simple and reliable display of the respective represented images by a single display device.

[0106] In a further alternative embodiment, the first stereo image output signal for displaying the represented images is provided with a first angle encoding, and the second stereo image output signal for displaying the represented images is provided with a further angle encoding, wherein the first and the further angle encodings are different from each other. In this embodiment, the common display device can be configured to display a so-called lenticular image, i.e., as a display device with a lenticular grid. Alternatively, the common display device can be configured as a display device with a parallex barrier. These can be controlled in a manner known to those skilled in the art. Of course, other display techniques are also conceivable that output the respective encoded images at different angles.In particular, the shared display device can be designed as a so-called two-view display or multi-view display.

[0107] For example, the first image and the second, corresponding image encoded by the first stereo image output signal can be output with a suitably configured or adjusted lens array such that the first image can be perceived by the left eye and the second, corresponding image by the right eye of a viewer. Similarly, the first image and the second, corresponding image encoded by the second stereo image output signal can be output at angles such that the first image can be perceived by the left eye of another viewer and the second, corresponding image by the right eye of that viewer.In this embodiment, a pose and / or viewing direction of the respective viewer can preferably be determined, with the angle encoding then being adjusted depending on the pose and / or viewing direction. This also advantageously results in a simple and reliable display of the respective images by the common display device.

[0108] A further proposed medical visualization system comprises at least one interface for receiving at least one first stereo image signal, which is generated by a stereo system of an operating microscope of the medical visualization system and represents a three-dimensional image of an examination area. The visualization system further comprises at least one evaluation unit. The medical visualization system, in particular the evaluation unit, is configured to perform a method according to one of the embodiments described in this disclosure.

[0109] The evaluation unit can be configured as a computing unit or include one. A computing unit, in turn, can include at least one microcontroller and / or at least one integrated circuit, or be configured as such. The computing unit can, in particular, generate the virtual image. It is possible that the evaluation unit includes at least one graphics processing unit (GPU) for this purpose. The medical visualization system advantageously enables the execution of a method according to one of the embodiments described in this disclosure, with the advantages already explained.

[0110] In another embodiment, the medical visualization system comprises at least one device for identifying a viewer. Such a device, along with its corresponding technical advantages, has already been described previously.

[0111] In another embodiment, the medical visualization system includes an additional stereo system. This, along with its corresponding technical advantages, has already been explained previously.

[0112] In another embodiment, the medical visualization system comprises at least one device for determining a viewer's pose and / or gaze direction. Reference can also be made to the previously provided explanations and the corresponding technical advantages.

[0113] In another embodiment, the medical visualization system comprises at least one 3D display device, wherein the display device is configured as a digital eyepiece, a 3D screen, or a head-mounted display. This, along with its corresponding technical advantages, has already been explained. A digital eyepiece can also be referred to as a boom. Such a digital eyepiece can be a head-mounted eyepiece. It can include two display devices for the stereoscopic display or representation of images. Alternatively, such an eyepiece can also be attached to a support device, in particular a stand, which may differ from the stand of the surgical microscope.

[0114] A further proposal is a computer program product comprising a computer program, wherein the computer program includes software means for executing one, several, or all steps of the method according to one of the embodiments described in this disclosure, when the computer program is executed by or in a computer or an automation system. The computer or automation system may include the evaluation device described above. The computer program product may, in particular, include means for performing the rendering, i.e., a rendering engine. The computer program product advantageously enables the execution of a method according to one of the embodiments described in this disclosure with the advantages already explained. The invention is explained in more detail with reference to exemplary embodiments. The figures show:

[0115] Fig. 1 shows a schematic flowchart of a method according to the invention,

[0116] Fig. 2 shows a schematic flowchart of a method according to the invention in a further embodiment,

[0117] Fig. 3 shows a schematic flowchart of a method according to the invention in a further embodiment,

[0118] Fig. 4 shows a schematic flowchart of a method according to the invention in a further embodiment,

[0119] Fig. 5 shows a schematic top view of an application scenario.

[0120] Fig. 6 shows a schematic top view of another application scenario and

[0121] Fig. 7 shows a schematic block diagram of a medical visualization system according to the invention.

[0122] In the following, identical reference symbols denote elements with the same or similar technical characteristics.

[0123] Fig. 1 shows a schematic flowchart of a method according to the invention for providing various stereo image output signals SBA1, SBA2 by a medical visualization system 4 (see Fig. 7). In a receive step ES, a first stereo image signal SB1 is received, which was generated by a stereo system 3 of an operating microscope 10 (see Fig. 7) and which represents a three-dimensional image of an examination area. The first stereo image signal SB1 can, in particular, comprise a stereo image pair, wherein the images of this stereo image pair were generated by image acquisition devices 5, 5b of the stereo system 22. These images can be received via interfaces 21 of the medical visualization system 4. In a first provision step BS1, this first stereo image signal SB1 is provided as the first stereo image output signal SBA1.Alternatively, the first stereo image signal SB1 can be processed to create the first stereo image output signal SBA1; in particular, an augmented three-dimensional image can be provided as the first stereo image output signal SBA1.

[0124] In a further provisioning step BS2, another stereo image output signal SBA2 is provided, representing another three-dimensional image of the examination area. This additional three-dimensional image can, in particular, depict the same examination area and be represented by another stereo image signal. It can depict the examination area with the same properties as the three-dimensional image represented by the first stereo image output signal SBA1. Thus, the additional stereo image output signal SBA2 can also be referred to as a stereo image output signal corresponding to the first stereo image signal SB1. This signal can also comprise a stereo image pair.

[0125] The additional three-dimensional image can be generated by image acquisition devices, e.g., by image acquisition devices of another stereo system of the operating microscope 10. In this case, the additional three-dimensional image can be received in a further reception step (not shown). The additional three-dimensional image can also be generated as a white light image, as a fluorescence contrast image, or as a polarization contrast image by a corresponding acquisition device 23. The additional three-dimensional image can also be an image generated from additional information ZI or an image combined from at least two of these modalities, e.g., by augmentation. An image generated from additional information ZI can be simulated. For example, it can be a virtual image generated by virtual image acquisition devices, which can be, in particular, optical models of the image acquisition devices 5, 5b of the stereo system 22.Thus, the additional stereo image output signal SBA2 can be generated depending on additional information ZI.

[0126] The first stereo image signal SB1 can also be processed into a further stereo image signal, which is then provided as a further stereo image output signal SBA2. Such processing was explained previously. An optional storage device 6 of the medical visualization system 1 is shown with a dashed line. Additional information ZI, stored in this storage device 6, can be retrieved to provide the further stereo image output signal SBA2. A capture device 23 for generating the further three-dimensional image represented by the further stereo image output signal SBA2 is also shown. This capture device 23 can, for example, be the further stereo system described above. Alternatively, the further capture device 23 can be a fluorescence capture device 14 or a polarization contrast image capture device.

[0127] The stereo image output signals SBA1 and SBA2 are provided such that the first stereo image output signal SBA1 is used to fully display an image according to a first visualization modality VM1, and the second stereo image output signal SBA2 is used to fully display an image according to a further visualization modality VM2 (see Fig. 2), wherein the visualization modalities VM1 and VM2 are different and viewer-specific. This has been explained above. Thus, the stereo image output signals SBA1 and SBA2 enable the full display of an image according to a viewer-specific visualization modality VM1 or VM2 on a display device 20, 20a, or 20b (see Fig. 5 and Fig. 6).In particular, an image according to a viewer-specific visualization modality VM1, VM2 does not need to be displayed by different display devices, which in particular means that two image output signals do not need to be generated to control the different display devices.

[0128] Fig. 2 shows a schematic flowchart of a further embodiment of a method according to the invention. In this embodiment, an identification step IS is performed before the execution of the method steps ES, BS1, BS2 shown in Fig. 1. In this identification step IS, various viewers 24a, 24b (see, for example, Fig. 5) are identified. Such different viewers 24a, 24b can, for example, be identified image-based. Identification can also be carried out depending on the input of user identities via a suitable input device 18 of the medical visualization system 1 (see Fig. 7). A storage device 6 is also shown, which can store assignment information in a retrievable manner, wherein the assignment information represents an assignment of a viewer identity to a visualization modality VM1, VM2.Thus, as a result of the identification step IS, various visualization modalities VM1 , VM2, ... can be determined, according to which the stereo image output signals SBA1, SBA2 are to be generated.

[0129] In a pose determination step PBS, a pose and / or a viewing direction of a viewer 24a, 24b is determined. This pose information PI can then be used to generate the stereo image output signals SBA1, SBA2. In particular, the stereo image output signals SBA1, SBA2 can be provided in such a way that a perspectively correct display is enabled for the respective viewer 24a, 24b on a common display device 20 or on the viewer-specific display device 20a, 20b.

[0130] It is also possible to determine the pose of the stereo system 22 of the operating microscope 10, whereby the stereo image output signals SBA1, SBA2, and in particular the additional stereo image output signal SBA2, are provided depending on the pose of the stereo system 22. This is particularly relevant if the additional stereo image output signal SBA2 represents a three-dimensional image generated from supplementary information ZI, especially from virtual images. In this case, virtual images can be generated in such a way that they correspond to the real images of the stereo system 22. This may require the registration of the supplementary information ZI.

[0131] Fig. 3 shows a schematic flowchart of a further embodiment of a method according to the invention. It illustrates that the first stereo image output signal SBA1 is displayed on a first display device 20a (see also Fig. 6), and the further stereo image output signal SBA2 is displayed on a further display device 20b, which is different from the first display device 20a. The display devices 20a and 20b are each viewer-specific. This can mean that the correspondingly represented images are displayed by the display devices 20a and 20b.

[0132] Fig. 4 shows a schematic flowchart of a method according to a further embodiment of the invention. In contrast to the embodiment shown in Fig. 3, the stereo image output signals SBA1, SBA2 are displayed on a common display device 20, which can in particular be a 3D screen. Fig. 5 shows a schematic top view of an application scenario. Shown are an operating microscope 10, a common display device 20, and a tracking system 17, in particular an optical tracking system. The operating microscope 10, the common display device 20, and the tracking system 17 can be part of a medical visualization system 4 (see Fig. 7). Also shown are an operating table 25 and two viewers 24a, 24b, who view an examination area 1 (see Fig. 7) on the operating table 25 from different poses.The first stereo image output signal SBA1 can be provided such that, when the common display unit 20 is activated with the first stereo image output signal SBA1, a three-dimensional image corresponding to a white light visualization modality is displayed to a first viewer 24a. The second stereo image output signal SBA2 can be provided such that, when the common display unit 20 is activated with the second stereo image output signal SBA2, a three-dimensional image corresponding to a fluorescence contrast or polarization contrast visualization modality is displayed to a second viewer 24b. Alternatively, an image generated from additional information ZI can also be displayed to the second viewer 24b.For example, it is possible that a virtual 3D image generated from additional information ZI is displayed to the additional viewer 24b, which on the one hand corresponds to the 3D image generated by the operating microscope 20 and on the other hand is displayed to the additional viewer 24b in a perspectively correct manner according to their pose. For example, the image represented by the additional stereo image output signal SBA2 can depict the examination area 1 with the same magnification but with an angular offset of 90° relative to the 3D image generated by the operating microscope 10.

[0133] Exemplary methods for displaying the images encoded by the various stereo image output signals SBA1, SBA2 for the different viewers 24a, 24b have already been explained previously.

[0134] The pose of viewers 24a, 24b can be detected by the tracking system 10. Also shown is a device 16 for determining the viewing direction, which can, for example, be arranged on the common display unit 20. Such a device 16 can, for example, include at least one image capture device for imaging viewers 24a, 24b, in particular their eyes. Depending on the viewing direction, a specific area of ​​the display surface of the common display unit 20 can also be determined in which the viewer-specific 3D image is to be displayed.

[0135] Fig. 6 shows another exemplary application scenario. In contrast to the embodiment shown in Fig. 5, the medical visualization system 4 does not include a common display device 20 for displaying images in different visualization modalities VM1, VM2 for different viewers 24a, 24b. Instead, the medical visualization system 4 includes different viewer-specific display devices 20a, 20b, which can be configured, for example, as HMDs or as digital eyepieces. Images according to a first visualization modality VM1 are displayed on a first display device 20a, while images according to a further visualization modality VM2 are displayed on a further display device 20b.

[0136] Fig. 7 shows a schematic block diagram of a medical visualization system 4 and an examination area 1. Also shown are an instrument 2 and a tumor object 3, which are arranged within the detection range of an operating microscope 10 of the medical visualization system 4. The tumor object 3 can be a hidden object.

[0137] Optional elements of the medical visualization system 4 are shown here with dashed lines. The medical visualization system 4 comprises image acquisition devices 5, 5b for microscopic imaging of the examination area 1 (see Fig. 3). These image acquisition devices 5, 5b can be part of a surgical microscope 10, which may include an objective 19 with a lens. This surgical microscope 10 is designed as a stereo surgical microscope, whereby the image acquisition devices 5, 5b form a stereo system 22. Also shown is a storage device 6 in which additional information (ZI) and / or assignment information can be stored. The medical visualization system 4 further comprises an evaluation device 7, which can receive and evaluate an image generated by the image acquisition devices 5, 5b, which can be transmitted to the evaluation device 7 as image signals via an interface 21.The evaluation unit 7 can be used in particular to perform the receiving step ES explained in Fig. 1 as well as the provisioning steps BS1 , BS2.

[0138] The figure further shows that the medical visualization system 4 can include a device 8 for determining depth information. This device can, for example, be designed as a distance sensor or include one. Also shown is an interface 9 of the medical visualization system 4 for data transmission with other, especially higher-level, systems.

[0139] Furthermore, the medical visualization system can include 4:

[0140] • at least one white light lighting device 11 ,

[0141] • at least one infrared lighting device 12,

[0142] • at least one fluorescence illumination device 13 for excitation of fluorescence radiation,

[0143] • at least one fluorescence detection device 14 for detecting fluorescence radiation,

[0144] • at least one surround-view camera 15,

[0145] • at least one device 16 for detecting the gaze direction of a viewer,

[0146] • at least one tracking system 17,

[0147] • at least one input device 18 for operating or controlling the medical visualization system,

[0148] • at least one display device 20.

[0149] The elements of the medical visualization system 4 can be connected via data and / or signal technology.

[0150] Not shown are beam filters for providing excitation radiation with wavelengths from a broader spectrum, e.g. the spectrum of the white light illumination device, or for filtering radiation from a broader spectrum.

[0151] The environmental camera 15 can be part of a further tracking system, which serves in particular for the optical determination of the pose of instruments within the detection range of the environmental camera 15. The pose determination can be monoscopic. In particular, the determination can also be marker-based. Images from the environmental camera 15 can be evaluated, in particular, for object recognition in order to identify a sub-area with respect to which the size of the foreground area V and / or background area H is then changed.

[0152] List of reference symbols

[0153] 1 examination area

[0154] 2 Instrument

[0155] 3 Tumor object

[0156] 4 medical visualization system

[0157] 5 Image acquisition device of an operating microscope

[0158] 5b further image acquisition device of the operating microscope

[0159] 6 Storage setup

[0160] 7 Evaluation unit

[0161] 8 Device for determining depth information

[0162] 9 Interface

[0163] 10 Operating microscope

[0164] 11 White light lighting device

[0165] 12 Infrared lighting device

[0166] 13 Fluorescence lighting device

[0167] 14 Fluorescence detection device

[0168] 15 Surround camera

[0169] 16 Device for gaze direction detection

[0170] 17 Tracking system

[0171] 18 Input device

[0172] 19 Lens

[0173] 20, 20a, 20b Display device

[0174] 21 Interface

[0175] 22 Stereo system

[0176] 23 Recording device

[0177] 24a, 24b Viewer

[0178] 25 operating table

[0179] ES reception step

[0180] BS1, BS2 Determination step

[0181] IS identification step

[0182] PBS Pose Determination Step

[0183] SB1 first stereo image signal

[0184] SBA1 first stereo image output signal

[0185] SBA2 additional stereo image output signal

Claims

Patent claims 1. Method for providing different stereo image output signals (SBA1, SBA2) by a medical visualization system (4), comprising the steps of: a. Receiving at least one first stereo image signal (SB1) generated by a stereo system (22) of an operating microscope (10) and representing a three-dimensional image of an examination area (1), b. Providing a first stereo image output signal (SBA1), c.Providing a further stereo image output signal (SBA2) which represents a further three-dimensional image of the investigation area (1), wherein the first stereo image output signal (SBA1) is provided to fully display an image according to a first visualization modality (VM1) and the further stereo image output signal (SBA2) is provided to fully display an image according to a further visualization modality (VM2), wherein the first and the further visualization modality (VM1, VM2) are different and viewer-specific visualization modalities.

2. Method according to claim 1, characterized in that the first stereo image signal (SB1) is provided as the first stereo image output signal (SBA1).

3. Method according to claim 1, characterized in that the first stereo image signal (SB1) is processed to the first stereo image output signal (SBA1).

4. Method according to claim 3, characterized in that augmentation takes place through the processing.

5. Method according to one of the preceding claims, characterized in that a white light image, a fluorescence contrast image, a polarization contrast image, an image generated from additional information (SI) or an image combined from these modalities is displayed according to a visualization modality.

6. The method of claim 5, characterized in that the image displayed according to the visualization modality is a fluorescence contrast image, a A polarization contrast image is an image generated from preoperative supplemental information (ZI) or an image combined from at least two modalities of a set comprising these modalities and further comprising a white light image.

7. Method according to one of the preceding claims, characterized in that the further three-dimensional image is a white light image, a fluorescence contrast image, a polarization contrast image, an image generated from preoperative additional information or an image combined from at least two of these modalities, in particular by augmentation.

8. Method according to one of the preceding claims, characterized in that a pose and / or a viewing direction of a viewer (24a, 24b) is determined, wherein a viewer-specific stereo image output signal (SBA1, SBA2) is provided depending on the pose and / or the viewing direction.

9. Method according to claim 8, characterized in that a pose of the operating microscope (10) is determined and the viewer-specific stereo image output signal (SBA1, SBA2) is generated depending on the pose of the viewer (24a, 24b) relative to the operating microscope (10).

10. Method according to claim 8 or 9, characterized in that the provision of the viewer-specific stereo image output signal (SBA1, SBA2) comprises the generation of virtual images by virtual image acquisition devices which are an optical model of image acquisition devices (5, 5b) of the stereo system (22).

11. Method according to claim 10, characterized in that the pose of the image acquisition device(s) (5, 5b) of the stereo system (22) is taken into account when evaluating the model for generating the virtual images.

12. Method according to claim 10 or 11, characterized in that the virtual images depict three-dimensional additional information.

13. Method according to claim 12, characterized in that the three-dimensional additional information is preoperative additional information.

14. Method according to claim 12 or 13, characterized in that the three-dimensional additional information is generated from images of the image acquisition devices (5, 5b) of the stereo system (22).

15. Method according to claim 8 or 9, characterized in that the further stereo image output signal (SBA2) represents a further three-dimensional image that was generated at least partially by a further stereo system of the medical visualization system (4).

16. Method according to one of the preceding claims, characterized in that the various stereo image output signals (SBA1, SBA2) are provided by a common display device (20) for displaying the respective represented images.

17. Method according to claim 16, characterized in that the first stereo image output signal (SBA1) for displaying the represented images is provided with a first polarization encoding, wherein the further stereo image output signal (SBA2) for displaying the represented images is provided with a further polarization encoding, wherein the first and the further polarization encoding are different from each other.

18. Method according to claim 16, characterized in that the further stereo image output signal (SBA2) for displaying the represented images is provided at a time that is later in time than the time for displaying the images represented by the first stereo image output signal (SBA1).

19. Method according to claim 16, characterized in that the first stereo image output signal (SBA1) is provided for displaying the represented images with a first color encoding, wherein the further stereo image output signal (SBA2) is provided for displaying the represented images with a further color encoding, wherein the first and the further color encoding are distinct from each other. are different.

20. Method according to claim 16, characterized in that the first stereo image output signal (SBA1) for displaying the represented images is provided with a first angle encoding, wherein the further stereo image output signal (SBA2) for displaying the represented images is provided with a further angle encoding, wherein the first and the further angle encoding are different from each other.

21. Medical visualization system (4), comprising at least one interface (21) for receiving at least one first stereo image signal (SB1) generated by a stereo system (22) of an operating microscope (10) of the medical visualization system (4) and representing a three-dimensional image of an examination area (1), and at least one evaluation device (7), wherein the medical visualization system (4) is configured to perform a method comprising the steps according to any one of claims 1 to 20.

22. Medical visualization system according to claim 21, characterized in that the medical visualization system (4) comprises at least one device for identifying a viewer.

23. Medical visualization system according to claim 21, characterized in that image-based viewer identification takes place.

24. Medical visualization system according to one of claims 21 to 23, characterized in that the medical visualization system (4) comprises a further stereo system.

25. Medical visualization system according to one of claims 21 to 24, characterized in that the medical visualization system (4) comprises at least one device (16) for determining a pose and / or a viewing direction of a viewer.

26. Medical visualization system according to one of claims 21 to 25, characterized in that the medical visualization system (4) includes at least one 3D display device (20, 20a, 20b) comprising the display device (20, 20a, 20b) being configured as a digital eyepiece, as a 3D screen or as a head-mounted display.

27. Computer program product comprising a computer program, wherein the computer program comprises software means for performing one, several or all steps of the method according to any one of claims 1 to 20, when the computer program is executed by or in a computer or an automation system.

Images