Method for generating an augmented image using a medical visualisation system, medical visualisation system, and computer program product

Abstract

The invention relates to a method for generating an augmented image (AA) using a medical visualisation system (4), comprising the following steps: a. receiving at least one image signal which has been generated by at least one image capturing device (5) of a surgical microscope (10) and which represents an image (A1) of an examination region (1); b. dividing the image (A1) into a foreground region (V) and a background region (H); c. receiving at least one signal which represents or encodes supplementary information (ZI) for augmentation; d. generating the augmented image (AA) by i. superimposing the background region (H) of the image of the examination region (1), or a portion thereof, with the supplementary information (ZI), or ii. superimposing the image (A1) of the examination region (1), or a portion thereof, with the supplementary information (ZI), wherein the superimposition in the background region (H) and in the foreground region (V) is carried out according to superimposition modalities that are different from one another. The invention also relates to a medical visualisation system (4) and to a computer program product.

Description

[0001] Method for generating an augmented image using a medical device

[0002] Visualization system, medical visualization system and

[0003] Computer program product

[0004] The invention relates to a method for generating an augmented image using a medical visualization system, a medical visualization system and a computer program product.

[0005] Surgical microscopes are used, among other things, to prepare for and perform medical operations on a patient. Such surgical microscopes are used by a user, e.g., a surgeon or an assistant, during a procedure to provide a magnified view of an area of ​​examination, particularly in or on the patient's surgical site. For this purpose, a surgical microscope may include an objective lens or lens system to produce a true optical image of the area of ​​examination. The objective lens may include optical elements for beam guidance, shaping, and / or direction. An optical element may, in particular, be a lens.

[0006] Surgical microscopes are used in medical facilities, as well as in laboratories and industrial applications. Examples of medical applications include neurosurgery, ophthalmic surgery, otolaryngology (ENT), plastic or reconstructive surgery, and orthopedic surgery. This list is not exhaustive. Generally, they are used in all areas of surgery where a magnified, high-resolution view of the surgical field is required to perform precise procedures.

[0007] A distinction can be made between analog and digital surgical microscopes. Unlike digital surgical microscopes, analog surgical microscopes do not capture images that are then displayed, for example, on a screen to magnify the examination area. Instead, they offer the user a direct, visually perceptible magnification of the examination area. Here, radiation reflected or scattered from the area of ​​application passes through the objective lens into at least one beam path and to at least one output section, through or into which the user looks to visually perceive the radiation and thus also the typically magnified representation of the examination area. An exemplary embodiment of an output section is a so-called eyepiece, into or through which the user looks to optically perceive the examination area with at least one eye.

[0008] Digital surgical microscopes comprise, or at least include, an image acquisition device for microscopic imaging that captures radiation in a beam path of the surgical microscope to generate a magnified image. This image can be displayed to the user or multiple users on one or more display devices. This enables high-resolution visualization. The image can be generated as a transmittable image signal, which encodes or represents the image. Purely digital surgical microscopes, unlike analog surgical microscopes, do not have an output section for visually detectable radiation, specifically no eyepiece. The image signal can then be transmitted as a data signal, either wired or wirelessly.Digital surgical microscopes enable the capture, storage, and further processing of images and videos. By applying image processing techniques, contrast, brightness, and other parameters can be adjusted to optimize the image quality of the generated images. Hybrid surgical microscopes can incorporate at least one image acquisition unit and at least one output section. For example, the radiation guided in the optical path of the surgical microscope can be split by a beam splitter, with one portion directed to the output section and another portion captured by the at least one image acquisition unit.

[0009] Stereoscopic surgical microscopes are also well-known. These typically include two separate beam paths for beam guidance and provide the user with a depth perception of the examination area. The beams guided in the two beam paths can be visually detected by the user via output sections. Digital surgical microscopes alternatively or additionally include two image acquisition units, each capturing the beams in one of the beam paths to generate an image. Based on these two images, which can also be referred to as corresponding images, a three-dimensional image is then provided to the user via a suitable display device. The image acquisition units are components of a stereo (camera) system.

[0010] The operating microscope can constitute a medical visualization system, or the medical visualization system can encompass the operating microscope. The components of the medical visualization system described below can be components of the operating microscope or components designed differently from the operating microscope.

[0011] Another known method is the provision of an augmented representation of the examination area to a user. An augmented representation can, in particular, be a representation of the real examination area that is enhanced by computer, especially by adding or overlaying at least one virtual object and / or other additional information onto the representation of the real examination area. The augmented representation can be displayed to a user as an augmented image on a display device or provided in a visually perceptible manner via an output section.

[0012] Additional information can be provided in the form of data that represents or encodes a geometric description of a space, particularly a three-dimensional space, and especially of objects arranged within it. Additional information can also be information generated from such data, for example, information produced by rendering. Rendering, or image synthesis, is the process of computer-implemented generation of a photorealistic or non-photorealistic image from a 2D or 3D model. Multiple models can be defined in a scene file, which contains objects in a defined language or data structure. The scene file can contain geometry, viewpoint, texture, lighting, and shading information that describes the virtual scene. The data contained in the scene file is then passed to a rendering program, which processes it and outputs it to a digital image or raster graphics file.A software application or component that performs the rendering is called a rendering engine, rendering system, graphics engine, or simply a renderer.

[0013] A key requirement for augmentation is that the viewer of the augmented image, particularly a surgeon, is not disturbed or distracted by the augmentation during their work. It is especially desirable that a surgeon can operate ergonomically even when viewing an area with superimposed augmentation. For example, it is crucial that an object, such as a tumor, located spatially behind a surface of the examination area along a certain line of sight, is represented in the augmented image in such a way that it does not mistakenly obscure objects located in front of the surface, as this can impair the viewer's spatial perception. Furthermore, this can distort the stereoscopic depth perception when viewed through a stereoscopic operating microscope.This can be particularly the case when augmented objects are superimposed in a stereoscopically perceptible manner, as the corresponding three-dimensional perception can be confusing for the viewer.

[0014] If other objects, such as surgical instruments, are within the field of view of the medical visualization system, augmentation can be problematic if it obscures the representation of these objects. This can disrupt the viewer's perception of information regarding the relative position between the object and the area being examined. Furthermore, it can be problematic if augmentation covers areas where tissue is depicted, as a viewer, especially the aforementioned surgeon, needs to clearly see the depicted tissue in order to perform surgical procedures.

[0015] US2010 / 295931 A1 pertains to the technical field of medical navigation image output. Medical navigation is used in image-guided surgery and assists the surgeon in the optimal positioning of their instruments, for example, by referencing previously acquired image data of the patient. The treating physician thus has access to an image output, such as a monitor, on which they can see where their instrument or its functional component is located in relation to specific body regions of the patient.

[0016] US Patent 2015 / 221105 A1 discloses imaging systems, imaging devices, and imaging techniques that merge portions of a multidimensional reconstructed image with multidimensional visualizations of at least part of a surgical site. The imaging systems can generate multidimensional reconstructed images based on preoperative image data. In a selected section of the visualization, the imaging systems can display a portion of the multidimensional reconstructed image. The document M. Allan et al., 2017 Robotic Instrument Segmentation Challenge, https: / / arxiv.org / abs / 1902.06426, 2019, discloses semantic segmentation.

[0017] The technical problem therefore arises of creating a method for generating an augmented image using a medical visualization system, a medical visualization system, and a computer program product that increase the display quality of the augmented image to improve the perception of a depicted examination area during augmentation and thus overcome at least one of the disadvantages explained above.

[0018] The solution to the technical problem is provided by the articles with the features of the independent claims. Further advantageous embodiments of the invention are described in the dependent claims.

[0019] A method for generating an augmented image using a medical visualization system is proposed. This system includes, in particular, an operating microscope or can be generated by the operating microscope itself.

[0020] Surgical microscopes and their technical features were briefly explained earlier. A surgical microscope can be, in particular, a stereo microscope. It can also be designed as an endoscope.

[0021] An operating microscope can comprise a microscope body. The objective lens described above can be integrated into the microscope body or attached to it, particularly in a detachable manner. The objective lens can be fixed in position relative to the microscope body. In addition to the objective lens, the microscope body can also have or incorporate at least one beam path for microscopic imaging and / or other optical elements for beam guidance, shaping, and / or deflection. In analog and hybrid operating microscopes, the microscope body can include at least one mounting interface for attaching an output element, such as an eyepiece, in particular in a detachable manner. The microscope body can comprise or form a housing, or be arranged within a housing. Components of the operating microscope, such as an image acquisition device for microscopic imaging, can be arranged in or on the housing.

[0022] The medical visualization system can include a stand for mounting the operating microscope. The operating microscope, in particular the microscope body, can be mechanically attached to the stand. The stand is designed to allow movement of the operating microscope in space, in particular with at least one degree of freedom, preferably with six degrees of freedom, where one degree of freedom can be translational or rotational. Furthermore, the stand can include at least one drive unit for moving the operating microscope. Such a drive unit can, for example, be a servo motor. Naturally, the stand can also include means for transmitting force / torque, e.g., gear units.In particular, it is possible to control the at least one drive unit in such a way that the operating microscope performs a desired movement and thus a desired change of position in space, or assumes a desired position and / or orientation in space. For example, the at least one drive unit can be controlled in such a way that an optical axis of an objective lens of the operating microscope assumes a desired orientation. Furthermore, the at least one drive unit can be controlled in such a way that a reference point of the operating microscope, e.g., a focal point, is positioned at a desired position in space. A target position can be specified by a user or another higher-level system. Methods for controlling the at least one drive unit as a function of a target position and a kinematic structure of the stand are known to those skilled in the art.

[0023] Furthermore, the medical visualization system can include one or even several display devices for showing the images. The display device can be used to display two- or three-dimensional images. A three-dimensional image can, in particular, be or comprise a stereo image pair, wherein the images of this image pair are stereoscopic images. Typical display devices are screens, especially 3D screens, head-mounted displays (HMDs), or digital eyepieces, which can also be referred to as booms. Furthermore, the medical visualization system, in particular the

[0024] Operating microscope, including one or more of the following elements:

[0025] • at least one white light lighting device,

[0026] • at least one infrared lighting device,

[0027] • at least one fluorescence illumination device for exciting fluorescence radiation,

[0028] • at least one beam filter to provide excitation radiation with wavelengths from a broader spectrum, e.g. the spectrum of the white light illumination device,

[0029] • at least one fluorescence detection device for detecting fluorescence radiation,

[0030] • at least one filter device for filtering radiation from a broader spectrum, e.g. for detection by an image acquisition device for microscopic imaging,

[0031] • at least one image acquisition device of an optical position detection device, which can also be referred to as a surrounding camera,

[0032] • at least one gaze direction detection device,

[0033] • at least one position detection device for determining a pose, i.e. a position and / or orientation, at least of the operating microscope

[0034] • at least one input device for operation,

[0035] • at least one interface for data transmission to or from another system or facility,

[0036] • at least one device for determining depth information, in particular with regard to the elements arranged in the detection range of the operating microscope, which may, for example, be designed as a distance sensor,

[0037] • at least one storage device for storing signals and / or information, especially in a retrievable manner.

[0038] An image acquisition device may, in particular, comprise a CMOS or CCD sensor. The detection range of the ambient camera may encompass the detection range of the surgical microscope completely or at least partially. Alternatively, the detection range of the surgical microscope may encompass the detection range of the ambient camera completely or at least partially. In a fluorescence visualization mode, a filter device may be inserted into an observation beam path, thereby providing a viewer with a filtered representation of the examination area. This radiation may be detected by the at least one image acquisition device for microscopic imaging. Alternatively, fluorescence radiation may also be detected by a detection device different from the image acquisition device used for microscopic imaging, e.g., a spectral camera.The fluorescence mode advantageously enables intraoperative tissue differentiation. Tumor tissue, in particular, can be visualized using fluorescence-based images. Nerve tissue, in particular, can be visualized using polarization-contrast-based images.

[0039] Medical visualization systems can be operated, for example, by manually controlling a component, particularly the operating microscope, or a corresponding input device; by voice control; by gesture control; by eye-tracking; by image-based control; or by other operating methods. The medical visualization system or the operating microscope may include the necessary components. Image-based control may, in particular, include the generation of operating or control signals by evaluating at least one image produced by an image acquisition device for microscopic imaging or by an image acquisition device of an optical position detection system.

[0040] Adjustable operating parameters of the medical visualization system or the surgical microscope can be formed by one or more of the following parameters:

[0041] • Magnification factor or zoom factor,

[0042] • Working distance or focus position,

[0043] • Detection range

[0044] • Light intensity,

[0045] • Illumination spectrum.

[0046] The proposed method comprises the following steps: a) Receiving at least one image signal, which is generated by at least one

[0047] The image is generated by the operating microscope's image acquisition system and represents an image of the examination area. This image signal can be received via an interface of the medical visualization system. Specifically, the image signal can represent a two-dimensional image. The examination area can be a region of a patient's surgical site during an operation or during a diagnostic examination.

[0048] The received image signal can preferably represent a white light image (VIS image) generated with visible radiation, i.e., radiation from a wavelength range between 360 nm and 830 nm. It is also conceivable that the image is provided as a fluorescence contrast image, generated by radiation with predetermined fluorescence-specific wavelengths or wavelength ranges, for example, wavelengths of 400 nm or 560 nm. The image can also be provided as a polarization contrast image, generated by radiation with a predetermined polarization. The image acquisition device of the surgical microscope can therefore be an image acquisition device for microscopic, i.e., magnified, imaging of the examination area, or it can be one of several different image acquisition devices. b) Dividing the image into a foreground area and a background area.This division can also be referred to as segmentation. Specifically, pixels or image areas of the image are assigned to either the foreground or the background, or classified as either foreground or background. This division can be achieved, in particular, by including elements such as objects or structures in the foreground that should not be overlaid by augmentation; these elements can be called foreground elements. Foreground elements include, for example, instruments, especially surgical instruments, hands, or fingers, or sections thereof. Similarly, the background contains elements, also referred to as background elements, which can be overlaid by augmentation. Background elements include, in particular, tissue. However, background elements can also be instruments.In particular, so-called hybrid elements can exist, which can be background or foreground elements depending on the scenario. An example of a hybrid element could be a swab, which is classified as a background element, especially if it is static and / or unactivated within the area under investigation. However, the swab can also be classified as a foreground element, especially if it moves more than a predetermined amount and / or is actuated by a user. For the purposes of this invention, foreground elements are not limited to instruments. Furthermore, no object recognition is performed to detect foreground elements.

[0049] In particular, both parts of the image in which tissue is depicted and parts of the image in which an instrument is depicted can be classified as parts of the background area.

[0050] The division can be achieved, in particular, by creating an image mask that represents information about the foreground and background. Such an image mask will be explained in more detail below. The image mask can be represented or encoded by a transmittable signal. Of course, information about the division can also be provided in other ways.

[0051] It is particularly possible to identify foreground elements in an image, with the pixels in which the foreground elements are mapped being assigned to the foreground area. The remaining sub-areas can then be assigned to the background area. Alternatively, it is possible to identify background elements in an image, with the pixels in which the background elements are mapped being assigned to the background area. The remaining sub-areas can then be assigned to the foreground area. Similarly, both background and foreground elements can be identified. Pixels that represent neither a foreground nor a background can then be assigned to one of the areas, preferably the foreground area. c) Receiving at least one signal comprising additional information for augmentation.This signal, hereinafter referred to as the supplementary information signal, can be received via an interface of the medical visualization system. Examples of supplementary information have already been described. Preferably, the supplementary information signal represents a two-dimensional image of the examination area, which is provided based on preoperatively or intraoperatively generated information. The supplementary information signal can be generated by another component of the medical visualization system, such as an additional image acquisition device or a sensor. The supplementary information signal can also be retrieved from a storage device of the medical visualization system. It is also conceivable that the supplementary information signal could be retrieved from a higher-level system, such as a network.The additional information signal represents or encodes information to be superimposed on the image of the area under investigation. d) Generating the augmented image by: i. Superimposing the background area, or a portion thereof, with the additional information. This can also be referred to as background superimposition. In other words, the superimposition with additional information is performed exclusively in the background area or a portion thereof. ii. Superimposing the image of the area under investigation, or a portion thereof, with the superimposition in the background and foreground areas occurring according to different superimposition modalities. This can also be referred to as combined superimposition.

[0052] Both options i. and ii. have in common that the overlay does not occur across the entire image, but rather involves separate or different treatment or overlay of the background and foreground areas. In particular, the background and foreground areas are treated differently for augmentation.

[0053] Unless otherwise stated, the descriptions in this disclosure apply to both background overlay and combined overlay. The generated augmented image can then be transmitted, particularly as an image signal, to a display device, which is then controlled to output the image in a visually perceptible manner. In particular, a virtual (3D) image or an augmented (3D) image can depict visible and / or hidden objects or elements, which can then be represented in a visually perceptible manner.

[0054] In the case of a stereo operating microscope, image signals generated by the two image acquisition units of the stereo system, representing corresponding images of the examination area, can be received. Each of these corresponding images can then be divided into a foreground and a background area. Furthermore, after receiving at least one signal containing additional information, augmented images can be generated by superimposing the additional information in the background area onto each of the images. It is also possible to receive an image-specific additional information signal for augmentation for each of the corresponding images, which is then used to generate the augmented image. For example, with virtual image acquisition units, an additional information signal can be generated for each image, representing a virtual image composed of...generated from the additional information. The virtual image acquisition devices can be optical models of the stereo system's image acquisition devices. This advantageously generates a perspective-correct augmentation, particularly in a three-dimensional and consistent manner. A virtual image can encode a texture, especially with color and / or transparency information.

[0055] To generate the augmented image, the additional information can be introduced into the beam path, for example, by reflection. This information can be projected onto a projection element, such as a radiolucent disc, positioned in the beam path using a projection device of the operating microscope. The augmented image can then be generated by creating an image based on the beams into which the additional information has been introduced as described. Alternatively or cumulatively, the radiation representing the augmented image can also be provided via an output section for visual perception by a viewer. Alternatively, an augmented image can be generated by a computer-aided enhancement of an image of the real examination area, particularly through image processing. In this case, the additional information can be superimposed onto the image of the real examination area.With a stereoscopic operating microscope, it is possible to provide the user with two augmented images. Generally, corresponding additional information can be introduced into each of the two optical paths of a stereoscopic operating microscope. For example, with digital stereoscopic operating microscopes, augmented images can be generated from the images produced by both image acquisition units. Thus, an augmented image with depth information—that is, an augmented three-dimensional representation—can be provided to the user on a suitable display device or via an output section.

[0056] Additional information displayed to a user through augmentation can include, in particular, preoperatively generated information, such as preoperatively generated data, which can also be used for surgical planning. Such preoperatively generated data can be, in particular, volumetric data. Volumetric data can be provided as a point cloud, a voxel-based representation, or a mesh-based representation. The additional information can also be provided, in particular, as a transmittable signal.

[0057] Preoperative data can be generated, for example, using computed tomography (CT) or magnetic resonance imaging (MRI) methods. Other imaging techniques, particularly ultrasound, X-ray, fluorescence, SPECT (single-photon emission computed tomography), or PET (positron emission tomography) methods, can also be used. Such augmentation allows, for example, a tumor object or its contours, generated based on preoperative information, to be superimposed onto a white light image.

[0058] Preoperative data generated using magnetic resonance imaging (MRI) can identify various tissue types, such as adipose tissue, muscle tissue, tumor tissue, as well as blood vessels and nerve pathways. Preoperative data generated using computed tomography (CT) can particularly depict bony structures.

[0059] Alternatively or additionally to using preoperatively generated information to provide the augmented image, intraoperative information—that is, information acquired or generated during treatment—can be used as supplementary information for generating the augmented image. For example, information can be collected and stored during surgery and then subsequently used to generate an augmented image. This is particularly advantageous when different visualization modalities are activated at different times. For instance, fluorescence information can be displayed in a fluorescence visualization modality or superimposed on a white light image.

[0060] The additional information can be assigned a reference coordinate system, which means that the additional information can also include spatial information. This reference coordinate system can also be called a world coordinate system.

[0061] Augmentation typically requires registration between the reference coordinate system of the additional information and a reference coordinate system of the medical visualization system, particularly the operating microscope or its image acquisition unit. This registration can be performed prior to augmentation. The registration establishes a spatial relationship between both the additional information and the image to a common reference coordinate system, especially for the information in the image generated by the operating microscope's image acquisition unit. This common reference coordinate system, hereinafter also referred to as the reference coordinate system, can be, in particular, the reference coordinate system of the additional information, the reference coordinate system of the medical visualization system, or a different reference coordinate system altogether.

[0062] Various methods can be used for registration, for example, model-based registration. In this approach, features can be detected in an image that correspond to previously known features, such as geometric features in the supplementary information. The registration can then be determined based on these corresponding features. The registration can, for example, be defined as a transformation matrix that includes a rotation and / or translation component. An example of model-based registration is edge-based registration, where the corresponding features are formed, for example, by a property of at least one, preferably several, edges in both the image and the supplementary information. Topography-based registration is also possible, particularly if a topography can be determined, e.g.,with a stereo system of an operating microscope. In this way, topographic information can be determined in at least one image, whereby corresponding features or points or sections are detected in both the additional information and this topographic information, which can then be used to determine the registration.

[0063] The additional information can therefore be registered information. For the purposes of this invention, the property "registered" can mean that a spatial reference to the reference coordinate system is known, in particular in the form of a transformation matrix. A registered device can generate signals whose spatial reference to the reference coordinate system is known.

[0064] The additional information can be generated primarily through rendering, a process already described in the introduction. A virtual image can be created through rendering, which can then be used for augmentation and, for example, superimposed on a live-recorded image of the real-world area under investigation. The virtual image can also be provided as an image signal that encodes or represents the virtual image.

[0065] The virtual image can be generated using a virtual image acquisition device, which can be a mathematical or physical, and in particular computer-aided, optical model of an image acquisition device. Specifically, a computer-implemented calculation of the pixels of the virtual image can be performed. This virtual image depends, among other things, on parameters of the (modeled) image acquisition device. In particular, the virtual image can be generated for microscopic imaging based on the intrinsic parameters of the image acquisition device, especially with these parameters. If corresponding images of a virtual stereo system are generated, these can additionally be generated for microscopic imaging based on the extrinsic parameters of both image acquisition devices, especially with these parameters.

[0066] In other words, when evaluating the model to generate virtual images, the parameters of the operating microscope's image acquisition device(s) used for microscopic imaging can be taken into account. This makes it possible to generate virtual images under the same conditions as real images.

[0067] The virtual image can also be generated depending on the pose, i.e., the position and / or orientation, of the (modeled) image acquisition device of the operating microscope. In particular, when evaluating the model to generate the virtual images, the pose of the operating microscope's image acquisition device(s) used for microscopic imaging can be taken into account, utilizing the registration information described above. By considering the registration information, it is possible, for example, to determine which pose of the virtual image acquisition device corresponds to the actual pose of the (modeled) image acquisition device of the operating microscope in the reference coordinate system of the additional information, which can also be referred to as the render coordinate system, and this information can then be used for the rendering process.In other words, a pose of at least one virtual image acquisition device can be identical to the pose of the modeled image acquisition device in the reference coordinate system. This makes it possible to create a virtual image that corresponds to the modeled image acquisition device in terms of both parameters and acquisition pose. For example, an image of a tumor object to be superimposed can be generated by rendering and then transmitted as an image or video signal and used for augmentation.

[0068] For the provision of virtual images, it may be necessary to determine the current pose of the operating microscope, particularly the image acquisition device. This pose can be determined using a position detection device. Registration allows a relationship to be established between the position detection device's reference coordinate system and the previously described reference coordinate systems, especially the reference coordinate system for the additional information. This makes it possible to determine the pose of the operating microscope within a desired reference coordinate system, particularly the reference coordinate system. Depending on the position of the operating microscope, the pose of the objective's optical axis or the position of a focal point can then be determined.If the operating microscope is attached to a stand with at least one joint, the pose of the operating microscope can also be determined depending on a joint position, whereby the joint position can be detected, for example, by a detection device or a sensor.

[0069] The position of the operating microscope and the additional information can define a previously described scene, i.e., a virtual spatial model that defines objects and their material properties, light sources, and the position and viewing direction of an observer, here the operating microscope.

[0070] Naturally, it is possible for the position detection device, or another position detection device, to also detect the pose of at least one other subject or object, or a part thereof. A subject can be, in particular, a user of the medical visualization system, such as someone observing the examination area or a display device. For example, it is conceivable to determine the pose of a body part of such a user, such as a hand, arm, or head. An object can be, in particular, another component of the medical visualization system, especially a display device. However, an object can also be an item that is not part of the medical visualization system, such as a piece of equipment like an operating table or a medical instrument. This makes it possible to determine the pose of the other subject or object within a desired reference coordinate system.

[0071] Such a position detection device can also be called a tracking system. A tracking system can be optical, electromagnetic, or operate in another way. The tracking system can be marker-based, detecting active or passive markers. Markers can be attached to objects or subjects whose pose is to be detected by the tracking system. An optical tracking system can, in particular, include optically detectable markers. An optical tracking system can, in particular, be a monoscopic position detection system. Here, the pose of an object can be determined by evaluating a two-dimensional image, in particular, exactly one two-dimensional image. Specifically, the position can be determined by evaluating the intensity values ​​of pixels (picture elements) of the two-dimensional image.

[0072] It is further conceivable that the medical visualization system includes at least one image acquisition device of an optical position detection device, which may in particular be a component of the operating microscope. This can also be referred to as a peripheral camera and serves in particular for monoscopic position detection.

[0073] A tracking system can also be part of an input device, where, for example, gesture control or gaze direction control is performed depending on information generated by the tracking system.

[0074] In background overlay, additional information can be superimposed according to a predefined overlay modality. In particular, the additional information can be generated with a predefined rendering style.

[0075] Various rendering styles, from which the predefined rendering style can be selected, include, for example, transparent rendering, in particular alpha-blending-based rendering, animation-based rendering, texture-based rendering, or Fresnel-effect-based rendering. Another rendering style can be a wireframe rendering style, which only displays the edges or edge lines of an object. In such a rendering, no filled areas or textures are displayed. A rendering style can also be a style that sets at least one, preferably several or even all of the following effects: color, texture, transparency, animation, or Fresnel effect. With background overlay, the overlay with additional information can be applied exclusively to the background area or to a portion of the background area.

[0076] In combined overlay, the additional information is superimposed in a domain-specific manner. For example, background information can be overlaid using a background-specific overlay modality, while foreground information can be overlaid using a foreground-specific overlay modality. This domain-specific overlay ensures that additional information in the background is displayed differently in the augmented image than in the foreground. In other words, domain-specific overlay can create an augmented image in which background-specific information is displayed according to a first visualization modality, and foreground-specific information is displayed according to a second visualization modality that differs from the first.A visualization modality refers to the way in which an image is presented for visual perception and / or what information is represented in the image. Depending on the visualization modality, a representation may, in particular, have predetermined color properties and / or transparency properties. It may also be the case that only modality-specific areas, properties, or features of a (virtual) object are displayed, e.g., only surfaces and / or borders and / or edges.

[0077] For area-specific overlays, background-specific and foreground-specific additional information can also be generated using different rendering styles. These different rendering styles can vary in the number of effects, their specific parameters, and / or the combination of effects.

[0078] The additional information mentioned can also be generated using the same rendering style, although this is executed differently for generating background-specific additional information than for generating foreground-specific additional information, in particular with different (render) parameters.

[0079] In combined overlay, the elements, particularly pixels, of an image mask, each representing a pixel's position in the foreground or background, can also be assigned information about the background-specific or foreground-specific overlay modality. For example, pixels in the image mask representing a background position can be assigned a first transparency level for the overlay, and pixels representing a foreground position can be assigned a further transparency level for the overlay, where the further transparency level does not represent complete transparency but a higher level of transparency than the first.

[0080] In both background overlay and combined overlay, it is possible for an overlay modality in the edge regions of a foreground or background region, and / or in a transition region between a foreground and a background region, to be set differently from an overlay modality in the remaining sub-regions of the foreground or background region. Various overlay modalities and their provision have been explained previously. For example, a transition region can include pixels whose distance to the nearest pixel in the foreground region and distance to the nearest pixel in the background region is less than a predetermined distance threshold. In particular, the edge- or transition-region-specific overlay modality can change, especially depending on the distance to an edge / transition. For example,The degree of transparency can change, especially increasing towards the edge / transition. Additional information for the overlay in a transition area can also be generated using a predefined rendering style, such as the wireframe rendering style.

[0081] The method according to the invention advantageously enables improved display quality of an area of ​​investigation represented by an augmented image, since, according to the background overlay, the overlay being applied exclusively in the background area reliably prevents contradictory perceptions of information by the viewer, particularly since additional information, which, as explained above, lies behind a surface of the area of ​​investigation, is not erroneously displayed in the foreground. In other words, this occlusion-correct display ensures a correct 3D perception of the depth of augmented additional information.Combined overlay also results in improved display quality, as the selection of different overlay qualities ensures that a viewer can perceive an image with correct depth perception while still receiving information about augmented details in the foreground. Unlike pure object recognition for detecting foreground elements, the proposed method offers the advantage of increasing the information content of the augmented image, since the assignment to foreground or background is not based on object recognition. Rather, it allows an object to be assigned to the foreground or background depending on the scenario.

[0082] In a further embodiment, at least two image signals are received, generated sequentially by the at least one image acquisition device of the operating microscope, each representing an image of the examination area. Furthermore, information about temporal changes in the image information as a function of these at least two images is determined, and the division of the image, in particular the image generated earlier or later, into foreground and background regions is performed based on this information. The change in the image information can be determined, in particular, as a function of the change itself or as the change between the at least two images, e.g., as the difference between the images or dependent on this difference.

[0083] Information about temporal changes in image information can preferably be information about the optical flow, which can be determined depending on at least two images. The optical flow information represents motion information, specifically representing the apparent movement of brightness patterns in an image sequence. The optical flow of the described image sequence can, for example, be determined as a vector field of the velocity projected onto the image plane from visible points in object space, particularly within the coordinate system of the image acquisition device. Specifically, each pixel of the image can be assigned a quantity representing the optical flow. In this case, a pixel can be assigned to the foreground if its assigned quantity is greater than a predetermined threshold.If the size assigned to a pixel is less than or equal to the predetermined threshold, the pixel can be assigned to the background. This advantageously results in a simple and reliable division into foreground and background areas, enabling good image quality of the augmented image. As an alternative to determining optical flow information, other methods for determining information about temporal changes in image information can be used, such as methods for determining a motion field. The motion field provides motion information, particularly 3D motion, for each pixel in an image. Specifically, the motion field can represent the true, especially three-dimensional, motion at each point that is mapped onto the 2D image.

[0084] In a further embodiment, depth information is determined for at least one image of the examination area, with the division depending on this depth information. Depth information can, for example, represent the distance of the element imaged at a pixel from a reference point, a reference line, or a reference surface. This distance can be determined along a reference direction. In particular, the distance can be determined within the reference coordinate system. For example, the reference surface can be oriented perpendicular to the optical axis of the operating microscope, with the reference direction being parallel to the optical axis. The reference surface can, for example, include an intersection point between the optical axis and an end lens of the operating microscope.

[0085] Depth information can be determined using the described device for determining depth information, for example, a distance sensor. Such a distance sensor could be, for example, an OCT distance sensor, a lidar sensor, a time-of-flight sensor, or a triangulation sensor such as a fringe projection sensor. Of course, distance sensors that generate distance information according to other physical principles can also be used.

[0086] The device for determining depth information can be registered, meaning the depth information can be registered information.

[0087] It is also possible to determine, in particular, recorded reconstruction information from corresponding images of a stereo system. This information represents a three-dimensional reconstruction of the detection area, and the depth information is determined based on this reconstruction information. For example, it is possible to detect a surface of the examination area in the three-dimensional reconstruction and then determine its distance from the previously described reference point, line, or surface. Furthermore, it is possible to determine depth information from preoperatively generated data. For example, a skull surface or a dural surface can be detected in preoperative data, and then, as previously described, its distance from the previously described reference point, line, or surface can also be determined.

[0088] In particular, a depth map can be generated for the image of the area under investigation, assigning depth information to each pixel in the image. A pixel whose depth information is greater than a predetermined minimum threshold and / or less than a predetermined maximum threshold can then be assigned to the foreground. Otherwise, the pixel can be assigned to the background. It is also possible to determine a statistical parameter of the depth information generated in this way, such as a mean or median of all depth information assigned to the pixels. If the depth information assigned to a pixel is greater than or less than the median or mean, this pixel can be assigned to the foreground; otherwise, it is assigned to the background.This also advantageously results in an easily implementable and reliable division into foreground and background areas, which enables good display quality of the augmented image.

[0089] In a further embodiment, additional semantic information is determined for the image of the area under investigation, or at least a sub-area thereof, with the division being carried out depending on this additional semantic information. The additional semantic information can be information that is taken into account during the division in addition to other information, e.g., in addition to the previously described information about a temporal change in the image information and / or the depth information. In particular, in addition to information required for visually perceivable representation, such as color and / or transparency information, semantic information can be assigned to the image, especially to a pixel or a set of pixels.Semantic information can be, in particular, descriptive information about the type of depicted element, its relationship to other depicted elements, or to the environment. This information can be determined image-based, for example, using a semantic segmentation method. Semantic supplementary information can therefore be information about or derived from a semantic segmentation of a depicted scene, where the segmented scene can be divided into image areas, each assigned a class from a set of predefined classes. The set of predefined classes could, for example, include the class "fabric," the class "instrument," the class "cloth," the class "swab," and / or another class.

[0090] Semantic supplementary information can include, in particular, information that can be evaluated for the described division, i.e., division-relevant information. By evaluating semantic supplementary information, each pixel of the image can be classified as containing either a background or a foreground element. Specifically, the semantic supplementary information can differ from the information required for the visually perceptible representation of the image. Considering semantic supplementary information advantageously results in a very reliable division into foreground and background areas, which in turn improves the image quality.

[0091] In a further embodiment, contextual information is determined for the image of the area under investigation, or at least a sub-area thereof, with the division depending on this contextual information. Contextual information can, in particular, be information about a temporal or spatial context. Contextual information can also include information about the type of current user activity, the type of current operational phase, or the type of operation. In particular, the contextual information can differ from the information required for the visually perceptible representation of the image. By taking contextual information into account, a very reliable division into foreground and background areas is advantageously achieved, which in turn improves the image quality.

[0092] In another embodiment, the division is performed using a model generated by machine learning. The term machine learning here encompasses or refers to methods for determining the model based on training data. For example, the model can be determined using supervised learning methods, where the training data (i.e., a training dataset) comprises input data and output data. The input data can be images representing the area under investigation or the corresponding image signals, while the output data is the division of the respective image into foreground and background regions.

[0093] For example, input and output data for such training data can be generated by having a user manually select the foreground and background regions in the images. This allows the model to learn the relationship between images and their division into foreground and background regions. However, it is also conceivable that unsupervised learning methods could be used to determine the model.

[0094] Suitable mathematical algorithms for machine learning include: Decision Tree-based methods, Ensemble Methods (e.g., Boosting, Random Forest)-based methods, Regression-based methods, Bayesian Methods (e.g., Bayesian Belief Networks)-based methods, Kernel Methods (e.g., Support Vector Machines)-based methods, Instance (e.g., k-Nearest Neighbour)-based methods, Association Rule Learning-based methods, Boltzmann Machine-based methods, Artificial Neural Networks (e.g., Perceptron)-based methods, Deep Learning (e.g., Convolutional Neural Networks, Stacked Autoencoders)-based methods, Transformer-based methods, Dimensionality Reduction-based methods, and Regularization Methods-based methods.

[0095] After the model has been created, i.e., after the training phase, the parameterized model can be used in the so-called inference phase to generate the foreground and background partitioning from images. This results in a reliable and high-quality partitioning. The model can be trained once, preferably with a sufficiently large dataset (training phase), and then used as a model (inference phase).

[0096] In another embodiment, the partitioning is performed using a neural network. For example, the neural network can be configured as an autoencoder, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a neural transformer network, or a combination of at least two of the aforementioned networks. Such a neural network, particularly the autoencoder version, can be trained using the training data described above, after which the partitioning can be performed. The autoencoder configuration of the neural network advantageously minimizes the computational effort required for the partitioning, enabling it to be performed reliably and quickly, especially by embedded systems.

[0097] Training a CNN advantageously reduces network complexity, making it suitable for devices with limited computing power. This applies to both the training and inference phases. Furthermore, the training time required for CNNs is short, particularly compared to LSTM networks, which also require comparatively higher computing power. However, LSTM network training is especially well-suited for time series analysis because its architecture incorporates temporal dependencies. This results in a high-quality partitioning.

[0098] In a further embodiment, input variables for the model-based partitioning include, in addition to the at least one image of the examination area or the corresponding image signal, at least one of the following pieces of information: a) depth information for the at least one image of the examination area, b) at least one image of the examination area generated temporally prior to the image of the examination area, c) an image corresponding to the image of the examination area, which was generated by another image acquisition device of the operating microscope, d) information for classifying imaged objects, e) information for classifying a user activity, f) information for classifying a surgical phase, g) information for classifying a surgical type.

[0099] If the operating microscope includes a stereo system with two image acquisition units, and the image of the examination area was generated by one of the image acquisition units, the corresponding image described above can be generated by the remaining image acquisition unit of the stereo system. The information according to points d) and e) can constitute semantic (additional) information. The information according to points f) and g) can constitute contextual information. Thus, input variables for the model-based partitioning can be image information, e.g., in the form of a microscopic white-light image and, if applicable, an image according to point b) or c), and / or image processing information, e.g., information about a temporal change in the image information and / or information according to point a), and / or semantic (additional) information and / or contextual information.

[0100] Information for classifying depicted objects can be information about a class to which a depicted object is assigned. An example class could be "fabric" or "instrument".

[0101] Information for classifying user activity can be information about a class of user activity, for example, the class "cutting", "suction", "ablating", "coagulating", "retracting" or "swabbing".

[0102] Accordingly, information for the classification of an operational phase can be information about the class of the current operational phase, for example the class "opening", "exploration", "resection", "reconstruction", "wound closure", "wound healing", "planning of opening", "exposure".

[0103] Information regarding the classification of the surgical type can be a class of surgical type, for example, "neurosurgical surgery," "orthopedic surgery," or "ophthalmic surgery." Neurosurgical surgeries can also be classified into classes such as "tumor surgery," "vascular surgery," or "spinal surgery." Ophthalmic surgeries can also be classified into "anterior segment surgery" and "posterior segment surgery," with anterior segment surgery including procedures such as cataract treatment or Descemet membrane endothelial keratoplasty. Posterior segment surgery includes procedures such as (membrane) peeling or vitrectomy.

[0104] Depth information can be determined as previously explained. Classification information can be determined by a classification device, whereby the classification can be image-based, i.e., by evaluating at least one image, particularly from the image acquisition device of the operating microscope or a different image acquisition device. The classification information can also be predetermined. It is also possible for the classification information to be determined based on control signals generated for or within the medical visualization system. Such information can also be generated by operating an input device.

[0105] By taking into account one, several or all of the aforementioned information in addition to the at least one image of the area under investigation, it advantageously results in the reliability and accuracy of the division into foreground and background areas being improved, which in turn improves the user's perception of the augmented image.

[0106] In a further embodiment, a prediction of image positions in the foreground and / or background area, or a sub-area thereof, is performed, with the division occurring depending on a predicted image position. Through this prediction, the division into foreground and background areas for the image generated at the current time can be predicted from images generated at earlier times and then taken into account when generating the augmented image at the current time.

[0107] The division into foreground and background areas can also be predicted for a future image (i.e., a subsequent image) and then taken into account when generating the augmented image at that future time. Such a prediction of the division in a subsequent image can also consider the image generated at the current time.

[0108] In other words, prediction can be used to forecast the spatial division of an image into foreground and background areas. In particular, prediction can take into account the movement of a depicted element. Thus, prediction is preferably used to forecast the foreground or a portion thereof, especially when it contains a moving foreground element such as an instrument, a glove, a finger, a suction cup, etc. For example, prediction can forecast and / or take into account where the moving foreground object is moving or will move. Therefore, it is also possible to predict the image division of a subsequent image or sequence of subsequent images into foreground and background areas.According to the preceding explanations, by predicting an image position of the foreground area, an image position of the background area and vice versa can also be predicted, namely as the remaining part of the image.

[0109] This advantageously enables further improved display quality. In particular, latencies during overlaying, such as those caused by segmentation, can be avoided or reduced, and real-time or near-real-time overlaying is possible. Such latency can arise especially when segmentation requires processing time. This ensures high display quality even for moving foreground elements that are not augmented or are augmented in a different way.

[0110] It is also possible that, based on the division of an image generated at an earlier time into foreground and background areas, the sub-area(s) of a subsequent image can be identified that must be considered in the prediction, particularly for determining motion information. In other words, it is possible to determine from previously determined division results, which are available, for example, in the form of already generated image masks, which

[0111] • To analyze sub-areas of an image or subsequent image generated at the current time for prediction purposes, for example because the representation of moving elements is expected there, and / or

[0112] • Sub-areas of an image or subsequent image generated at the current time cannot be analyzed for prediction, for example because no representation of moving elements is expected there.

[0113] This can advantageously reduce the image processing effort required for prediction.

[0114] For prediction, several images relating to previous [data / data] can be used.

[0115] Information about the image positions of the (partial) area can be determined from previous images, i.e., those created before the current image is generated. This information can be stored in a retrievable format. It can include, in particular, information about the position, shape, and / or size of the foreground and / or background area (or the selected sub-areas). This information can be determined, for example, based on the image masks that were defined for images generated at earlier times.

[0116] Then, based on this information, image positions of a (sub)area in the image generated at the current time or in a subsequent image can be predicted using a prediction method, for example, in the form of an image mask. For instance, motion information for a (sub)area can be determined based on the information. This motion information can be information about a motion trajectory, such as the motion trajectory of a reference point (e.g., the geometric center of the (sub)area or a pixel), and / or information about motion parameters such as velocity. Then, based on this motion information, the image position in the current image and / or in a subsequent image (i.e., an image generated at a future time) can be predicted, for example, using an extrapolation method.Similarly, information about a change in shape and / or size can be determined.

[0117] It is possible that the division is carried out according to the predicted image position of the (sub)areas.

[0118] In general, it is also possible to fuse multiple pieces of partitioning information to determine the resulting partition. The set of partitioning information that can be fused can include the aforementioned prediction information, depth information, additional semantic information, contextual information, and information about temporal changes.

[0119] For example, prediction can be based on image masks, whereby, depending on image masks generated at earlier times (i.e., before the creation of an image mask for the current image), information about the coordinates of pixels representing foreground and / or background regions is determined within the image mask for the current image. In another embodiment, confidence information is determined during image splitting, with the splitting and / or superimposition being confidence-dependent.

[0120] The respective confidence information can, in particular, represent the confidence level in assigning a pixel or image area to one of the areas foreground or background. In other words, this information can represent how accurate the corresponding assignment is, or with what certainty the assignment is determined. If the division is performed using a machine learning model (MLM), the confidence level can be (co-)determined during processing by the MLM.

[0121] The confidence information can be provided as a confidence value, which can take values ​​from a predetermined range between a first limit representing maximum confidence and another limit representing minimum confidence.

[0122] For example, confidence-based partitioning can be achieved by assigning a pixel / image area to one of the areas only if the corresponding confidence value exceeds or falls below a predetermined confidence threshold. Otherwise, it can be assigned to the remaining area, or alternative methods can be used to assign these pixels / image areas to one of the areas.

[0123] In particular, pixels / image areas assigned to the background with a low confidence value can be reassigned to the foreground based on confidence, for example, if the confidence value for assigning them to the background is lower than a predetermined confidence threshold. This allows the foreground area to be enlarged based on confidence. This reduces the risk of additional information being erroneously superimposed on an area identified as the background with low confidence. This can advantageously improve the display quality.

[0124] It is also possible for pixels / image areas assigned to the foreground or background with a low confidence value to be assigned to a transition area based on confidence, for example, if a confidence value of the assignment represents a lower confidence level than a predetermined confidence threshold. This confidence-based assignment to the transition area preferably only occurs if at least one additional assignment criterion is met, for example, if the distance of the pixel / image area to the nearest foreground area, in particular to the nearest foreground pixel, is less than a predetermined distance threshold, and / or if the distance of the pixel / image area to the nearest background area, in particular to the nearest background pixel, is less than a predetermined distance threshold.If at least one additional criterion is not met, the explained assignment to the foreground area can take place.

[0125] The predetermined confidence thresholds for assigning a sample to the foreground region and the transition region can be the same or different. Preferably, the predetermined confidence threshold for assigning a sample to the foreground region is lower than the threshold for assigning it to the transition region.

[0126] For example, confidence-dependent overlaying can be achieved by applying a confidence-dependent overlay modality. Overlaying according to various overlay modalities has already been explained. In particular, coloring and / or transparency properties can be set based on confidence. For example, the lower the confidence, the higher the transparency, especially for pixels / image areas in the background. In other words, the additional information can be displayed in the relevant areas using a confidence-dependent visualization modality. Confidence-dependent rendering styles can also be applied to display the additional information. This allows viewers to easily assess the reliability of the overlay with additional information, which in turn improves the overall display quality.

[0127] Confidence values ​​representing low confidence can be generated, in particular, when assigning pixels in blurred images or in blurred image areas, which can occur during image acquisition with an operating microscope, especially during defocusing. This ensures good image quality even in such scenarios. Confidence-dependent overlay can also improve image quality, especially in peripheral areas and / or at transitions between foreground and background. If it is assumed that the assignment of pixels to one of the areas tends to occur with low confidence in peripheral areas / transitions, a viewer can easily identify such peripheral areas / transitions and the reliability of the overlay there with additional information, which in turn also improves image quality.

[0128] In a further embodiment, after the image of the area under investigation has been divided into foreground and background areas, but before the superimposition, the size of the foreground area is changed, particularly at least with respect to a sub-area of ​​the foreground. The size of the foreground area can be enlarged or reduced, in particular by assigning pixels from the background area surrounding a sub-area of ​​the foreground area to the foreground, or by assigning pixels from a sub-area of ​​the foreground area to the background. If the foreground area is enlarged, the background area is reduced. If the foreground area is reduced, the background area is enlarged accordingly. Such a change can be geometric; for example, a circular, ellipsoidal, rectangular, or square enlargement or reshaping can be achieved.The reduction in size occurs, whereby, for example, all pixels arranged in a geometric shape around a reference point of the sub-area are assigned to the foreground. Properties of the change, such as a geometric shape and / or the size of the change, can be predetermined.

[0129] Alternatively or cumulatively, the size of the background area is changed, in particular at least with respect to a sub-area of ​​the background area, especially by being enlarged or reduced. The explanations given for changing the foreground area apply accordingly.

[0130] By subsequently altering the size of the foreground and / or background area in this way, it advantageously results in a further improved display quality for the augmented image, particularly in certain application-dependent scenarios. For example, an image area that shows a part in contact with the tissue, such as the tip of an instrument, and which is classified as the foreground area, can be enlarged to allow the user to perceive the contacted tissue without augmentation interference. It is also possible to reduce the size of foreground sections containing little information. This further improves the display quality of the provided augmented image.

[0131] In a further embodiment, a sub-area of ​​the foreground or background to be modified is determined by an object detection method. The sub-area to be modified can designate a sub-area of ​​the foreground or background with respect to which the size of the foreground or background is changed. Alternatively, the sub-area to be modified can also designate a sub-area whose size is changed.

[0132] As explained above, the size of the foreground area can be increased or decreased, in particular by assigning pixels from the background area in a section adjacent to, or at least partially surrounding, a sub-area of ​​the foreground to the foreground, or by assigning pixels from a sub-area of ​​the foreground to the background. Similarly, the size of the background area can be increased or decreased, in particular by assigning pixels from the foreground area in a section adjacent to, or at least partially surrounding, a sub-area of ​​the background to the background, or by assigning pixels from a sub-area of ​​the background to the foreground.

[0133] The object recognition method can, in particular, be an image-based method, which assigns at least one pixel to an object by evaluating the at least one image. Such methods are known to those skilled in the art. For example, it is possible to identify specific sections, e.g., the tip of an instrument, in the image using an object recognition method. The size of the foreground area can then be enlarged with respect to this section. In this case, the properties of the change can be object-dependent, with predetermined properties of the change being assigned to an object. This advantageously results in a reliable identification of a section, which in turn improves the previously described image quality. In a further embodiment, a section to be changed is determined as a function of at least one optical parameter of the operating microscope.Optical parameters can include, in particular, the operating parameters described above. For example, an image area containing foreground elements that are not arranged within a predetermined interval around a set working distance of the operating microscope can be defined as a sub-area of ​​the foreground to be reduced in size. It can be assumed that such foreground elements are rendered blurry by the microscope, i.e., with insufficient image quality, where image quality can specifically represent image sharpness. It can be useful to enlarge or reduce such foreground sub-areas with insufficient image quality. In other words, an image area with insufficient image quality can be enlarged or reduced, with the image quality depending, for example, on the degree of blurriness of the elements rendered in that area.

[0134] By reducing the size of such a (foreground) area, an additional image area can be created for augmentation, but this additional image area only includes areas in which image information such as tissue or instruments are depicted with insufficient image quality.

[0135] Enlarging such a (foreground) area can prevent augmentation with sufficient image quality from occurring next to an area of ​​insufficient image quality, for example, by creating a sharp image area next to a blurry one. This type of representation can advantageously reduce viewer confusion. Such enlargement can be particularly useful when the image quality is below a predetermined level.

[0136] It is also conceivable that augmented information, particularly in the vicinity of an image area with insufficient image quality, could be displayed with the same insufficient image quality. Such a display of augmented information is particularly feasible when the image quality is better than or equal to a predetermined standard. This can advantageously also enable reliable identification of areas to be modified, which in turn improves the display quality described above.

[0137] In another embodiment, the change is carried out depending on a) a distance of the object or element depicted in the sub-area from a surface of the investigation area, b) sub-area-specific image information.

[0138] In particular, a property of the change can be chosen to depend on at least one of the aforementioned quantities. For example, the magnitude of the change can be positively or negatively correlated with the distance. The distance can be determined, in particular, depending on the depth information explained above. Of course, other methods of determining the distance are also conceivable.

[0139] Image information can be a property of the image and can be determined image-based. It can be, in particular, blur information, shadow information, or contrast information. Sub-area-specific image information can also be determined by a suitable device. In particular, this information can be determined image-based. For example, it is possible to perform a blur classification procedure, a shadow classification procedure, and / or a contrast classification procedure to identify image areas, especially in the foreground, that are blurred and / or shadowed by more than a predetermined degree and / or lack sufficient contrast. Such areas can then be reduced in size, or even completely removed from the foreground. It is also conceivable that such areas could be enlarged.

[0140] The image quality in a specific area can also be determined based on the image information, such as blur information, shadow information, and / or contrast information, whereby enlargement and / or reduction can then be adjusted according to the image quality. Reference can be made to the preceding explanations for further details.

[0141] This has the advantageous effect of allowing even parts of the foreground to be used for augmentation, areas that would otherwise provide little information to the user. This, in turn, improves the display quality and increases the information content of the augmented image.

[0142] In a further embodiment, a sub-area to be modified and / or the change in size is determined by means of a model generated by machine learning, in particular by means of a neural network. Reference can be made to the preceding explanations regarding the division of the image into a foreground and a background area by means of a model. The input data for a training dataset can consist of images and the division of these images into foreground and background areas, or the corresponding image signals. The output data can be information about selected sub-areas and / or the changed division into foreground and background areas. For example, input and output data for such training data can be generated by a user manually selecting the sub-areas and / or the changes to the foreground and background areas.This allows the model to learn the relationship between images and their division into foreground and background, as well as subsequent changes to this division. This results in a reliable and high-quality determination of the area to be changed and / or the change in size.

[0143] In a further embodiment, input variables for determining the sub-area to be modified and / or the change in size include at least one of the previously described pieces of information a) to g) for the model-based partitioning. This advantageously improves the reliability and accuracy of determining the sub-area to be modified and / or the change in size, which in turn improves the user's perception of the augmented image.

[0144] A further proposal is a medical visualization system, in particular comprising or consisting of an operating microscope, which includes at least one interface for receiving an image signal from an image acquisition device for generating an image of an examination area and at least one evaluation device. The medical visualization system, in particular the evaluation device, is configured to perform a method according to one of the embodiments described in this disclosure.As explained above, the medical visualization system may in particular include at least one of the following: a device for determining information for classifying depicted objects and / or a user activity and / or a surgical phase and / or a surgical type, a device for determining a distance of a depicted object from a surface of the examination area, a device for determining sub-area-specific blur and / or shadow information.

[0145] The evaluation unit can be configured as a computing unit or include one. A computing unit, in turn, can include at least one microcontroller and / or at least one integrated circuit, or be configured as such. The computing unit can, in particular, generate the virtual image. It is possible that the evaluation unit includes at least one graphics processing unit (GPU) for this purpose. The medical visualization system advantageously enables the execution of a method according to one of the embodiments described in this disclosure, with the advantages already explained.

[0146] A further proposal is a computer program product comprising a computer program, wherein the computer program includes software means for executing several or all steps, in particular steps b. and c., of the method according to one of the embodiments described in this disclosure, when the computer program is executed by or in a computer or an automation system. The computer or automation system may include the evaluation device described above. The computer program product may, in particular, include means for performing the rendering, i.e., a rendering engine. The computer program product advantageously enables the execution of a method according to one of the embodiments described in this disclosure with the advantages already explained.

[0147] The invention is explained in more detail using exemplary embodiments. The figures show:

[0148] Fig. 1a shows an exemplary representation of a superimposition without a division into foreground and background areas according to the invention,

[0149] Fig. 1b shows another exemplary representation of an augmented image without a division into foreground and background areas according to the invention,

[0150] Fig. 2 shows a schematic flowchart of a method according to the invention,

[0151] Fig. 3 shows an exemplary representation of an augmented image produced using the method according to the invention.

[0152] Fig. 4a shows a schematic representation of an examination area and an instrument,

[0153] Fig. 4b is a schematic representation of an image of the scene depicted in Fig. 4a,

[0154] Fig. 4c is a schematic representation of information on optical flow,

[0155] Fig. 4d a schematic representation of an image mask,

[0156] Fig. 5a shows a schematic representation of an investigation area and a

[0157] Instruments,

[0158] Fig. 5b is a schematic representation of an image of the scene depicted in Fig. 5a,

[0159] Fig. 5c is a schematic representation of depth information, Fig. 5d is a schematic representation of an image mask,

[0160] Fig. 6 shows a schematic flowchart of a method according to the invention in a further embodiment,

[0161] Fig. 7 shows a schematic flowchart of a method according to the invention in a further embodiment,

[0162] Fig. 8a shows a schematic representation of an image mask,

[0163] Fig. 8b shows an augmented image generated on the basis of the image mask shown in Fig. 8a,

[0164] Fig. 9a shows a schematic image mask with a locally enlarged foreground area,

[0165] Fig. 9b shows an augmented image generated on the basis of the image mask shown in Fig. 9a,

[0166] Fig. 10a a schematic representation of an operation scene,

[0167] Fig. 10b shows a schematic representation of the additional information to be augmented,

[0168] Fig. 10c shows an exemplary image mask,

[0169] Fig. 10d shows an augmented image generated on the basis of the image mask shown in Fig. 10c,

[0170] Fig. 10e shows another exemplary image mask,

[0171] Fig. 10f shows an augmented image generated on the basis of the image mask shown in Fig. 10e,

[0172] Fig. 10g shows another exemplary image mask,

[0173] Fig. 10h shows an augmented image generated on the basis of the image mask shown in Fig. 10g, Fig. 11 shows a schematic block diagram of a medical visualization system according to the invention and

[0174] Fig. 12 shows an exemplary representation of an augmented image produced using the inventive method in a further embodiment.

[0175] In the following, identical reference symbols denote elements with the same or similar technical characteristics.

[0176] Fig. 1a shows an exemplary white-light image of an examination area 1 with instruments 2. A virtual object 3, for example a tumor object 3, is superimposed onto the image. This superimposition is done without considering areas in the image that are suitable for augmentation. It can be seen that the tumor object 3 obscures parts of an instrument 2 and the tissue.

[0177] Fig. 1b shows the same scene as Fig. 1a, except that the tumor object 3 is depicted as more transparent than in Fig. 1a, in particular as semi-transparent. Despite the now possible visibility of parts of the instrument 2 that were obscured by the tumor object 3 in Fig. 1a, the perception of the tissue areas and instrument sections obscured by the tumor object 3 is nevertheless impaired. Furthermore, the spatial impression arises that the tumor object 3 is hovering above the tissue, which makes a spatially accurate perception of the scene difficult for the viewer.

[0178] Fig. 2 shows a schematic flowchart of a method according to the invention for generating an augmented image AA using a medical visualization system 4 (see Fig. 11). In a receive step ES1, an image signal is received which was generated by at least one image acquisition device 5 for microscopic imaging (see, for example, Fig. 11) and which represents an image of an examination area 1. This image A1 is divided in a partitioning step AS into a foreground area V and a background area H (see, for example, Fig. 4d). Exemplary partitioning possibilities are explained below. In a further receive step ES2, at least one signal comprising additional information ZI for augmentation, which is also referred to as the additional information signal, is received. In Fig.Figure 2 shows that the additional information signal is stored in a retrievable manner in a storage device 6, which may be part of the medical visualization system 4. However, it is also possible for the additional information ZI to be acquired intraoperatively and / or retrieved from a network via a suitable interface. The additional information signal can, in particular, also represent an image, preferably an image of the same size as the image A1 generated by the image acquisition device 5. The image represented by the additional information signal can, in particular, be a virtual image generated from preoperatively generated additional information ZI. As explained at the outset, such a virtual image can be generated with a virtual image acquisition device, which is an optical model of the image acquisition device 5.

[0179] In a generation step GS, the augmented image AA is then created by overlaying the additional information ZI onto the image A1 in the background area H or a part thereof.

[0180] In the partitioning step AS, an image mask M can be generated that encodes or represents information about the foreground region V and the background region H. The image mask M can be provided, in particular, as an image, especially a two-dimensional image, where the image size of the image mask M can correspond to the image size of the image A1. Each pixel of the image mask M can be classified as either a foreground region V or a background region H. The image mask M can, for example, be a binary image, where foreground region pixels are assigned the value 1 or 0, and background region pixels are assigned the remaining value.

[0181] In this case, the overlay of the additional information ZI can only occur for pixels of image A1 whose corresponding pixels in the image mask M are classified as background area pixels. A corresponding pixel can have the same pixel coordinates, which can refer to the same image coordinate systems. In particular, the overlay can be opaque or with a predetermined degree of transparency, especially semi-transparent. For example, an alpha blending method can be used for the overlay. In this case, each background area pixel of the image mask can be assigned an alpha value between 0 (inclusive) and 1 (inclusive), where the value 0 or 1 represents, for example, complete transparency and the value 1 or 0 represents complete opacity.If a background pixel in image mask M is assigned an alpha value representing complete opacity, the corresponding pixel in image A1 can be completely, i.e., opaquely, overlaid by additional information. If a background pixel in image mask M is assigned an alpha value that represents neither complete transparency nor complete opacity, the corresponding pixel in image A1 will not be completely, opaquely, overlaid by additional information. If a background pixel in image mask M is assigned an alpha value representing complete transparency, the corresponding pixel in image A1 cannot be overlaid by additional information.

[0182] It is also conceivable that at least one, but preferably several, foreground pixel(s) of the image mask are assigned an alpha value that does not represent complete transparency, where the minimum transparency level of all foreground pixels is greater than the maximum transparency level of all background pixels. Thus, an augmented image can be created in which an overlay in a foreground area is displayed more transparently than an overlay in a background area. In this way, a viewer can also perceive additional information in the foreground area; however, this information is less distracting due to the higher transparency.

[0183] It is possible to assign alpha values ​​between the measure of complete transparency (exclusive) and the measure of complete opacity (inclusive) to all background area pixels, while foreground area pixels are assigned an alpha value with the measure of complete transparency.

[0184] The augmented image AA can then be transmitted to a display device 20 (see Fig. 11), in particular as an image signal, to present it to a viewer in a visually perceptible manner. Fig. 3 shows an exemplary augmented image AA generated using the method described in Fig. 2. In the scene shown in Fig. 3, the additional information ZI represents the tumor object 3. It can be seen that, unlike the images shown in Fig. 1a and Fig. 1b, the tumor object 3 does not obscure the instruments 2. Thus, the instruments 2 are depicted in the foreground area V of image A1, while tissue areas of the examination area 1 are depicted in the background area H. The augmented image AA shown in Fig. 3 enables a viewer to perceive the scene more clearly and thus offers a higher display quality.

[0185] Fig. 4a shows an exemplary representation of an instrument 2 and an examination area 1, in particular a surgical surface. Also shown is an image acquisition device 5 of an operating microscope.

[0186] Fig. 4b shows an image A1 generated by the image acquisition device 5. The instrument 2 and the examination area 1 are depicted therein, with the instrument 2 obscuring parts of the examination area 1.

[0187] Fig. 4c shows, by way of example and indicated by arrows, information on the optical flow in image A1. This information on the optical flow can be generated by receiving at least two image signals, which were generated sequentially by the image acquisition device 5 and each represent an image A1 of the examination area 1. Methods for determining this information from the sequence of these images A1 are known to those skilled in the art. As indicated by the arrows, a quantity representing the optical flow, which is greater than a predetermined threshold, is assigned to a sub-region of image A1 in which the instrument 2 is imaged. The pixels of this sub-region can then be classified as foreground region V, i.e., foreground region pixels, while the remaining pixels of image A1 are classified as background region H, i.e., background region pixels.The resulting image mask M, consisting of foreground pixels and background pixels, is shown in Fig. 4d. Fig. 5a shows an exemplary representation of an instrument 2 and an examination area 1, in particular a surgical surface. Also shown is an image acquisition device 5 of an operating microscope.

[0188] Fig. 5b shows an image A1 generated by the image acquisition device 5. The instrument 2 and the examination area 1 are depicted therein, with the instrument 2 obscuring parts of the examination area 1.

[0189] Fig. 5c shows an example of depth information in image A1. This information can be generated, for example, with a distance sensor that detects the distance of the instrument and the uncovered part of the examination area from, for example, a reference plane oriented perpendicular to an optical axis of the operating microscope, in which an intersection of the optical axis with a lens of the operating microscope is located, along an optical axis of the operating microscope.

[0190] It is evident that distance information is assigned to a sub-area of ​​image A1, in which instrument 2 is depicted, representing a distance that is smaller than the distance assigned to the sub-area of ​​image A1 that depicts the unobstructed part of the examination area. In particular, this smaller distance is smaller than a predetermined distance.

[0191] The pixels of this instrument sub-area can then be classified as foreground area V, i.e., as foreground area pixels, while the remaining pixels of image A1 are classified as background area H, i.e., as background area pixels. The resulting image mask M, consisting of foreground area pixels and background area pixels, is shown in Fig. 5d.

[0192] Fig. 6 shows a schematic flowchart of a further embodiment of a method according to the invention. In contrast to the embodiment shown in Fig. 2, for the division step AS, in addition to the information necessary for the visually perceptible representation of the image A1, in particular color information, further information I is taken into account. Such further information I can be, in particular, additional semantic information or contextual information.

[0193] In particular, such semantic or contextual information can be the depth information for image A1 explained in relation to Fig. 5c. If the image acquisition device 5 with which image A1 was generated is part of a stereo system, the information I can also represent information about an image corresponding to image A1, which was generated by another image acquisition device 5b (see Fig. 11). The information I can also represent at least one image of the investigation area 1 generated prior to image A1.

[0194] Preferably, the information I also includes information about a classification of objects depicted in image A1. For this purpose, object recognition can be performed, whereby the recognized objects are then classified using a classification method. However, this classification does not provide a classification in the foreground and background areas V and H; in particular, for at least one object class from the set of all object classes, there is no unambiguous assignment to the foreground area V or background area H. For example, objects of the class "cloths" or "swabs" can be assigned to the background area, especially if they are static, e.g., fixed in position relative to the site or lying on the fabric of the site.Such objects of the class "cloths" or "swabs" can also be assigned to the foreground, especially if they are not statically arranged, particularly because they are being held or even moved by an instrument. Similarly, an object or area of ​​the class "fabric" can be assigned to the background, especially if there is no object or area of ​​the class "instrument" within that fabric area or within a predetermined area surrounding it. Alternatively, an object or area of ​​the class "fabric" can also be assigned to the foreground, especially if there is an object or area of ​​the class "instrument" within that fabric area or within a predetermined area surrounding it. In the latter case, it can be avoided that an augmentation would disturb a viewer when looking at areas with which an instrument is currently interacting.

[0195] The information can also include information about the classification of a user activity, an operation phase, or an operation type. These have already been explained previously.

[0196] It is possible that the partitioning step AS is performed using a model, or by evaluating a model, generated through machine learning, particularly through the evaluation of a neural network. An input to the model can be at least one image A1 of the investigation area 1. Another input can be the previously described information I. An output of the model can be the previously described image mask M.

[0197] Fig. 7 shows a schematic flowchart of a further embodiment of a method according to the invention. In contrast to the embodiment shown in Fig. 6, a modification step VS is performed after the partitioning step for generating the image mask M. In the modification step, a size of the foreground area V is changed, at least with respect to a sub-area of ​​the foreground area V. As a result of the modification step VS, a modified image mask MV is provided. This corresponds to the image mask M, but includes more or fewer foreground pixels and thus correspondingly fewer or more background pixels compared to the image mask M. The dashed lines indicate that the partitioning step AS and / or the modification step VS can optionally be performed depending on the additional information I explained above.

[0198] Fig. 8a shows a schematic image mask M with three foreground elements, which correspond, for example, to image sections of the image A1, into which instruments 2 (see Fig. 3) are depicted.

[0199] Fig. 8b shows the augmented image AA, which is generated based on this image mask M. It can be seen that an augmented tumor object 3 does not cover the depicted instruments 2.

[0200] Fig. 9a shows an exemplary modified or altered image mask MV. In contrast to the image mask M shown in Fig. 8a, it is evident that the foreground region V has been enlarged in the area of ​​a tip or free end of the instruments 2. Specifically, the enlargement of the foreground region V is achieved by defining a circular area with a predetermined diameter around a reference point of the sub-region, in this case, for example, around the geometric center of an image mask section classified as an instrument tip, and classifying all image points of the modified image mask MV within this circular area as foreground image points. Fig. 9b shows the augmented image generated based on the modified image mask MV. In contrast to the augmented image AA shown in Fig. 8b, it is apparent that a portion of the tissue around the instrument tips is not augmented by the tumor object 3.This allows a viewer to perceive the tissue actuated by the instruments without augmentation.

[0201] Fig. 10a shows an exemplary image A1, which was generated by an image acquisition device 5 of a medical visualization system 4 (see Fig. 11).

[0202] Fig. 10b shows an exemplary representation of a tumor object 3, which is represented by an additional information signal.

[0203] Fig. 10c shows an exemplary image mask M with foreground area V and background area H, which was created without the modification step VS shown in Fig. 7.

[0204] Fig. 10d shows the augmented image AA generated on the basis of this image mask M.

[0205] Fig. 10e shows an image mask MV modified compared to the image mask M shown in Fig. 10c, with a modified foreground and background area V, H. In the corresponding modification step VS (see Fig. 7), shadowed image areas vB of image A1 were detected, whereby the foreground area V of the image mask M shown in Fig. 10c was reduced by the image areas that are shadowed. Such shadowed image areas vB can be determined, for example, using shadow classification methods. One exemplary classification method compares the color values ​​of pixels in image A1 with predetermined threshold values ​​and classifies a pixel as a shadowed pixel depending on the comparison result.

[0206] Fig. 10f shows an augmented image AA, which was generated based on the modified image mask shown in Fig. 10e. It is evident that additional information ZI can also be superimposed on image A1 in the shaded image area vB.

[0207] Fig. 10g shows an image mask MV modified compared to the image mask M shown in Fig. 10c, with a modified foreground and background area V, H. In the corresponding modification step VS (see Fig. 7), in addition to the shadowed image areas vB, blurred image areas uB of the image A1 were detected, whereby the foreground area V of the image mask M shown in Fig. 10c was reduced by image areas that are blurred. Such blurred image areas uB can be determined, for example, using blur classification methods. An exemplary classification method for detecting blurred areas can include the application of Laplacian operators, in particular the application of Laplacian pyramids, and threshold operators.

[0208] Fig. 10h shows an augmented image AA, which was generated based on the modified image mask shown in Fig. 10g. It is evident that additional information ZI can also be superimposed on image A1 in the out-of-focus image area uB.

[0209] Fig. 11 shows a schematic block diagram of a medical visualization system 4 and an examination area 1. Also shown are an instrument 2 and a tumor object 3, which are arranged within the detection range of an operating microscope 10 of the medical visualization system 4. The tumor object 3 can be a hidden object.

[0210] Optional elements of the medical visualization system 4 are shown here with dashed lines. The medical visualization system 4 comprises at least one image acquisition device 5 for microscopic imaging of the examination area 1. The image acquisition device 5 can be part of the operating microscope 10, which may include an objective 19 with a lens. This operating microscope 10, in turn, may be configured as a stereo operating microscope, comprising a further image acquisition device 5b for microscopic imaging of the examination area 1, and the image acquisition devices 5 and 5b forming a stereo system. A storage device 6, in which additional information (ZI) may be stored, is also shown.The medical visualization system 4 further comprises an evaluation unit 7, which can receive and evaluate an image A1 generated by the image acquisition unit 5. The image A1 can be transmitted as an image signal to the evaluation unit 7 via an interface 21. The evaluation unit 7 can, in particular, perform the division step AS and the generation step GS. It is further shown that the medical visualization system 4 can include a device 8 for determining depth information. This device can, for example, be configured as a distance sensor or include one. Also shown is an interface 9 of the medical visualization system 4 for data transmission with other, in particular higher-level, systems.

[0211] Furthermore, the medical visualization system can include 4:

[0212] • at least one white light lighting device 11 ,

[0213] • at least one infrared lighting device 12,

[0214] • at least one fluorescence illumination device 13 for excitation of fluorescence radiation,

[0215] • at least one fluorescence detection device 14 for detecting fluorescence radiation,

[0216] • at least one surround-view camera 15,

[0217] • at least one device 16 for detecting the gaze direction of a viewer,

[0218] • at least one tracking system 17,

[0219] • at least one input device 18 for operating or controlling the medical visualization system,

[0220] • at least one display device 20.

[0221] The elements of the medical visualization system 4 can be connected via data and / or signal technology.

[0222] Not shown are beam filters for providing excitation radiation with wavelengths from a broader spectrum, e.g. the spectrum of the white light illumination device, or for filtering radiation from a broader spectrum.

[0223] The ambient camera 15 can be part of a further tracking system, which serves in particular for the optical determination of the pose of instruments within the detection range of the ambient camera 15. The pose determination can be monoscopic. In particular, the determination can also be marker-based. Images from the ambient camera 15 can be evaluated, in particular, for object recognition in order to identify a sub-area with respect to which the size of the foreground area V and / or background area H is then changed. Fig. 12 shows an exemplary augmented image AA, which was generated essentially like the augmented image AA shown in Fig. 3. In contrast to the augmented image AA shown in Fig. 3, however, the superimposition of image A1 with the tumor object 3 takes place in both the foreground area V and the background area H (see Fig. 12).4d), wherein the background area H is superimposed on the tumor object 3 according to a first superimposition modality represented by line hatching, and the foreground area V is superimposed on the tumor object 3 according to a further superimposition modality represented by dotted hatching. The superimposition modalities are different from each other.

[0224] It can be seen that, in contrast to the image shown in Fig. 3, the tumor object 3 overlays the instruments 2, which are shown in the foreground area V, differently than tissue areas which are shown in the background area H.

[0225] Reference symbol list

[0226] 1 examination area

[0227] 2 Instrument

[0228] 3 Tumor object

[0229] 4 medical visualization system

[0230] 5 Image acquisition device of an operating microscope

[0231] 5b further image acquisition device of the operating microscope

[0232] 6 Storage setup

[0233] 7 Evaluation unit

[0234] 8 Device for determining depth information

[0235] 9 Interface

[0236] 10 Operating microscope

[0237] 11 White light lighting device

[0238] 12 Infrared lighting device

[0239] 13 Fluorescence lighting device

[0240] 14 Fluorescence detection device

[0241] 15 Surround camera

[0242] 16 Device for gaze direction detection

[0243] 17 Tracking system

[0244] 18 Input device

[0245] 19 Lens

[0246] 20 Display device

[0247] 21 Interface

[0248] A1 image

[0249] M Image mask

[0250] MV modified image mask

[0251] I Information

[0252] ZI Additional Information

[0253] V Foreground area

[0254] H Background area

[0255] AA augmented image

[0256] ES1 receive step

[0257] AS division step

[0258] ES2 receive step GS generation step

[0259] VS change step

Claims

Patent claims 1. Method for generating an augmented image (AA) by a medical visualization system (4), comprising the steps of: a. Receiving at least one image signal generated by at least one image acquisition device (5) of a surgical microscope (10) and representing an image (A1) of an examination area (1), b. Splitting the image (A1) into a foreground area (V) and a background area (H), c. Receiving at least one signal representing or encoding additional information (SI) for augmentation, d. Generating the augmented image (AA) by i. superimposing the background area (H) of the image (A1) of the examination area (1) or a part thereof with the additional information (SI), or ii.Overlaying the image (A1) of the investigation area (1) or a part thereof with the additional information (ZI), wherein the overlay in the background area (H) and in the foreground area (V) is carried out according to different overlay modalities.

2. Method according to claim 1, characterized in that the overlay with additional information (ZI) is carried out exclusively in the background area (H) or in a part of the background area (H).

3. Method according to claim 1 or 2, characterized in that at least two image signals are received which were generated successively by the at least one image acquisition device (5) of the operating microscope (10) and which each represent an image (A1) of the examination area (1), wherein information about a temporal change of the image information depending on these at least two images (A1) is determined and the division of the image (A1) takes place depending on this information.

4. Method according to claim 3, characterized in that the information about a temporal change in the image information is information about the optical flow. is.

5. Method according to one of the preceding claims, characterized in that depth information is determined for the at least one image (A1) of the investigation area (1) and the division is carried out depending on this depth information.

6. Method according to one of the preceding claims, characterized in that additional semantic information is determined for the image (A1) of the investigation area (1) or at least a sub-area thereof, wherein the division is carried out depending on this additional semantic information.

7. Method according to one of the preceding claims, characterized in that context information is determined for the image (A1) of the investigation area (1) or at least a sub-area thereof, wherein the division is carried out depending on this context information.

8. Method according to one of the preceding claims, characterized in that the division is carried out using a model generated by machine learning, preferably using a neural network.

9. Method according to claim 6, characterized in that input variables for the model-based partitioning, in addition to the at least one image (A1) of the investigation area (1), comprise at least one of the following information: a. depth information for the at least one image of the investigation area, b. at least one temporally prior image of the investigation area. Image of the examination area, c. an image corresponding to the image of the examination area, which was generated by another image acquisition device (5b) of the operating microscope (10), d. information for the classification of imaged objects, e. information for the classification of a user activity, f. information for the classification of an operation phase, g. information for the classification of an operation type.

10. Method according to one of the preceding claims, characterized in that a prediction of image positions of the foreground area (V) and / or the background area (H) is carried out, wherein the division is performed depending on a predicted image position.

11. Method according to one of the preceding claims, characterized in that confidence information is determined when splitting the image (A1), wherein the splitting and / or superimposition is confidence-dependent.

12. Method according to one of the preceding claims, characterized in that after the division of the image (A1) of the investigation area (1) into the foreground area (V) and the background area (H) a. a size of the foreground area (V) and / or b. a size of the background area (H) is changed.

13. Method according to claim 12, characterized in that a partial area of ​​the foreground area (V) and / or the background area (H) to be changed is determined by a method for object recognition.

14. Method according to claim 12 or 13, characterized in that a partial area to be modified is determined depending on optical parameters of the operating microscope (10).

15. Method according to one of claims 12 to 14, characterized in that the change is carried out depending on a. a distance of the object depicted in the sub-area from a surface of the investigation area (1), b. a sub-area-specific image information.

16. Method according to one of claims 12 to 15, characterized in that a sub-area to be changed and / or the change in size is determined by means of a model which was generated by machine learning.

17. Method according to claim 16, characterized in that input variables for a model-based determination of the sub-area to be changed and / or the change in size comprise at least one of the following information: a. Depth information for the at least one image of the area under investigation, b. At least one image of the area under investigation generated prior to the image of the area under investigation, c. An image corresponding to the image of the area under investigation, which was generated by a further image acquisition device (5b) of the operating microscope (10), d. Information for classifying imaged objects, e. Information for classifying a user activity, f. Information for classifying a surgical phase, g. Information for classifying a surgical type.

18. Medical visualization system (4) comprising at least one interface (20) for receiving an image signal from an image acquisition device (5) for generating an image (A1) of an examination area (1) and at least one evaluation device (7), wherein the medical visualization system (4) is configured to perform a method comprising the steps according to any one of claims 1 to 17.

19. Computer program product comprising a computer program, wherein the computer program comprises software means for performing several or all steps of the method according to any one of claims 1 to 17, when the computer program is executed by or in a computer or an automation system.

Images