Method for generating an augmented image using a medical visualization system, medical visualization system and computer program product

Abstract

The invention relates to a method for generating an augmented image (AA) by a medical visualization system (4) comprising the steps: a. Receiving at least one image signal generated by at least one image acquisition device (5) of an operating microscope (10) and representing an image (A1) of an examination area (1), b. Dividing the image (A1) into a foreground area (V) and a background area (H), c. Receiving at least one signal that represents or encodes additional information (II) for augmentation, d. Generating the augmented image (AA) by overlaying the background area (H) of the image (A1) of the examination area (1) or part thereof with the additional information (ZI) as well as a medical visualization system (4) and a computer program product.

Description

[0001] The invention relates to a method for generating an augmented image using a medical visualization system, a medical visualization system and a computer program product.

[0002] Surgical microscopes are used, among other things, to prepare for and perform medical operations on a patient. These microscopes are used by a user, such as a surgeon or assistant, during a procedure to provide a magnified view of an area of ​​examination, particularly in or on the patient's surgical site. For this purpose, a surgical microscope may include an objective lens or lens system to produce a true optical image of the area of ​​examination. The objective lens may include optical elements for beam guidance, shaping, and / or direction. An optical element may, in particular, be a lens.

[0003] Surgical microscopes are used in medical facilities, as well as in laboratories and industrial applications. Examples of medical applications include neurosurgery, ophthalmic surgery, otolaryngology (ENT), plastic and reconstructive surgery, and orthopedic surgery. This list is not exhaustive. Generally, they are used in all areas of surgery where a magnified, high-resolution view of the surgical field is required to perform precise procedures.

[0004] A distinction can be made between analog and digital surgical microscopes. Unlike digital surgical microscopes, analog surgical microscopes do not capture images that are then displayed, for example, on a screen to magnify the examination area. Instead, they offer the user a direct, visually perceptible magnification of the examination area. Here, radiation reflected or scattered from the area of ​​application passes through the objective lens into at least one beam path and to at least one output section, through or into which the user looks to visually perceive the radiation and thus also the typically magnified representation of the examination area. An exemplary embodiment of an output section is a so-called eyepiece, into or through which the user looks to optically perceive the examination area with at least one eye.Digital surgical microscopes comprise, or at least include, an image acquisition device for microscopic imaging that captures radiation in a beam path of the surgical microscope to generate a magnified image. This image can be displayed to the user or multiple users on one or more display devices. This enables high-resolution visualization. The image can be generated as a transmittable image signal, which encodes or represents the image. Purely digital surgical microscopes, unlike analog surgical microscopes, do not have an output section for visually detectable radiation, specifically no eyepiece. The image signal can then be transmitted as a data signal, either wired or wirelessly.Digital surgical microscopes enable the capture, storage, and further processing of images and videos. By applying image processing techniques, contrast, brightness, and other parameters can be adjusted to optimize the image quality of the generated images. Hybrid surgical microscopes can incorporate at least one image acquisition unit and at least one output section. For example, the radiation guided in the optical path of the surgical microscope can be split by a beam splitter, with one portion directed to the output section and another portion captured by the at least one image acquisition unit.

[0005] Stereoscopic surgical microscopes are also well-known. These typically include two separate beam paths for beam guidance, providing the user with a depth perception of the examination area. The beams guided in the two paths can be visually detected by the user via output sections. Digital surgical microscopes alternatively or additionally include two image acquisition units, each capturing the beams in one of the beam paths to generate an image. Based on these two images, which can also be referred to as corresponding images, a three-dimensional image is then provided to the user via a suitable display device. The image acquisition units are components of a stereo (camera) system. The surgical microscope can constitute a medical visualization system, or the medical visualization system can encompass the surgical microscope.

[0006] The components of the medical visualization system described below may be components of the operating microscope or components designed differently from the operating microscope.

[0007] Another known method is the provision of an augmented representation of the examination area to a user. An augmented representation can, in particular, be a representation of the real examination area that is enhanced by computer, especially by adding or overlaying at least one virtual object and / or other additional information onto the representation of the real examination area. The augmented representation can be displayed to a user as an augmented image on a display device or provided in a visually perceptible manner via an output section.

[0008] Additional information can be provided in the form of data that represents or encodes a geometric description of a space, particularly a three-dimensional space, and especially of objects arranged within it. Additional information can also be information generated from such data, for example, information produced by rendering. Rendering, or image synthesis, is the process of computer-implemented generation of a photorealistic or non-photorealistic image from a 2D or 3D model. Multiple models can be defined in a scene file, which contains objects in a defined language or data structure. The scene file can contain geometry, viewpoint, texture, lighting, and shading information that describes the virtual scene. The data contained in the scene file is then passed to a rendering program, which processes it and outputs it to a digital image or raster graphics file.A software application or component that performs the rendering is called a rendering engine, rendering system, graphics engine, or simply a renderer.

[0009] A key requirement for augmentation is that the viewer of the augmented image, particularly a surgeon, is not disturbed or distracted by the augmentation during their work. It is especially desirable that a surgeon can operate ergonomically even when viewing an area with superimposed augmentation. For example, it is crucial that an object, such as a tumor, located spatially behind a surface of the examination area along a certain line of sight, is represented in the augmented image in such a way that it does not mistakenly obscure objects located in front of the surface, as this can impair the viewer's spatial perception. Furthermore, this can distort the stereoscopic depth perception when viewed through a stereoscopic operating microscope.This can be particularly the case when augmented objects are superimposed in a stereoscopically perceptible manner, as the corresponding three-dimensional perception can be confusing for the viewer.

[0010] If other objects, such as surgical instruments, are within the field of view of the medical visualization system, augmentation can be problematic if it obscures the representation of these objects. This can disrupt the viewer's perception of information regarding the relative position between the object and the area being examined. Furthermore, it can be problematic if augmentation covers areas where tissue is depicted, as a viewer, especially the aforementioned surgeon, needs to clearly see the depicted tissue in order to perform surgical procedures.

[0011] US2010 / 295931 A1 pertains to the technical field of medical navigation image output. Medical navigation is used in image-guided surgery and assists the surgeon in the optimal positioning of their instruments, for example, by referencing previously acquired image data of the patient. The treating physician thus has access to an image output, such as a monitor, on which they can see where their instrument or its functional component is located in relation to specific body regions of the patient.

[0012] US Patent 2015 / 221105 A1 discloses imaging systems, imaging devices, and imaging methods that merge portions of a multidimensional reconstructed image with multidimensional visualizations of at least a portion of a surgical site. The imaging systems can generate multidimensional reconstructed images based on preoperative image data. In a selected section of the visualization, the imaging systems can display a portion of the multidimensional reconstructed image.

[0013] The document M. Allan et al., 2017 Robotic Instrument Segmentation Challenge, https: / / arxiv.org / abs / 1902.06426, 2019 reveals a semantic segmentation.

[0014] The document Tian, ​​Yuan; Guan, Tao; Wang, Cheng: Real-time occlusion handling in augmented reality based on an object tracking approach. Sensors, 2010, Vol. 10, No. 4, pp. 2885-2900. DOI: https: / / doi.org / 10.3390 / s100402885 discloses a real-time occlusion handling method based on an object tracking approach.

[0015] The technical problem therefore arises of creating a method for generating an augmented image using a medical visualization system, a medical visualization system, and a computer program product that increase the display quality of the augmented image to improve the perception of a depicted examination area during augmentation and thus overcome at least one of the disadvantages explained above.

[0016] The solution to the technical problem is provided by the articles with the features of the independent claims. Further advantageous embodiments of the invention are described in the dependent claims.

[0017] A method for generating an augmented image using a medical visualization system is proposed. This system includes, in particular, an operating microscope or can be generated by the operating microscope itself.

[0018] Surgical microscopes and their technical features were briefly explained earlier. A surgical microscope can be, in particular, a stereo microscope. It can also be designed as an endoscope.

[0019] An operating microscope can comprise a microscope body. The objective lens described above can be integrated into the microscope body or attached to it, particularly in a detachable manner. The objective lens can be fixed in position relative to the microscope body. In addition to the objective lens, the microscope body can also have or incorporate at least one beam path for microscopic imaging and / or other optical elements for beam guidance, shaping, and / or deflection. In analog and hybrid operating microscopes, the microscope body can include at least one mounting interface for attaching an output element, such as an eyepiece, in particular in a detachable manner. The microscope body can comprise or form a housing, or be arranged within a housing. Components of the operating microscope, such as an image acquisition device for microscopic imaging, can be arranged in or on the housing.

[0020] The medical visualization system can include a stand for mounting the operating microscope. The operating microscope, in particular the microscope body, can be mechanically attached to the stand. The stand is designed to allow movement of the operating microscope in space, in particular with at least one degree of freedom, preferably with six degrees of freedom, where one degree of freedom can be translational or rotational. Furthermore, the stand can include at least one drive unit for moving the operating microscope. Such a drive unit can, for example, be a servo motor. Naturally, the stand can also include means for transmitting force / torque, e.g., gear units.In particular, it is possible to control the at least one drive unit in such a way that the operating microscope performs a desired movement and thus a desired change of position in space, or assumes a desired position and / or orientation in space. For example, the at least one drive unit can be controlled in such a way that an optical axis of an objective lens of the operating microscope assumes a desired orientation. Furthermore, the at least one drive unit can be controlled in such a way that a reference point of the operating microscope, e.g., a focal point, is positioned at a desired position in space. A target position can be specified by a user or another higher-level system. Methods for controlling the at least one drive unit as a function of a target position and a kinematic structure of the stand are known to those skilled in the art.

[0021] Furthermore, the medical visualization system can include one or more display devices for showing the images. The display device can be used to show two- or three-dimensional images. A three-dimensional image can, in particular, be or comprise a stereo image pair, wherein the images of this image pair are stereoscopic images. Typical display devices are screens, especially 3D screens, head-mounted displays (HMDs), or digital eyepieces, which can also be referred to as booms.

[0022] Furthermore, the medical visualization system, in particular the operating microscope, may include one or more of the following elements: • at least one white light lighting device, • at least one infrared lighting device, • at least one fluorescence illumination device for exciting fluorescence radiation, • at least one beam filter to provide excitation radiation with wavelengths from a broader spectrum, e.g. the spectrum of the white light illumination device, • at least one fluorescence detection device for detecting fluorescence radiation, • at least one filter device for filtering radiation from a broader spectrum, e.g. for detection by an image acquisition device for microscopic imaging, • at least one image acquisition device of an optical position detection device, which can also be referred to as a surrounding camera, • at least one gaze direction detection device, • at least one position detection device for determining a pose, i.e. a position and / or orientation, at least of the operating microscope • at least one input device for operation, • at least one interface for data transmission to or from another system or facility, • at least one device for determining depth information, in particular with regard to the elements arranged in the detection range of the operating microscope, which may be designed, for example, as a distance sensor, • at least one storage device for storing signals and / or information, especially in a retrievable manner.

[0023] An image acquisition device may, in particular, include a CMOS or CCD sensor. The detection range of the ambient camera may fully or at least partially encompass the detection range of the surgical microscope. Alternatively, the detection range of the surgical microscope may fully or at least partially encompass the detection range of the ambient camera.

[0024] In a fluorescence visualization mode, a filter device can be inserted into an observation beam path, providing the viewer with a filtered representation of the examination area. This radiation can be captured by the at least one image acquisition device for microscopic imaging. Alternatively, fluorescence radiation can also be captured by a separate acquisition device, such as a spectral camera. The fluorescence mode advantageously enables intraoperative tissue differentiation. Tumor tissue, in particular, can be visualized using fluorescence-based images. Nerve tissue, in particular, can be visualized using polarization-contrast-based images.

[0025] Medical visualization systems can be operated, for example, by manually controlling a component, particularly the operating microscope, or a corresponding input device; by voice control; by gesture control; by eye-tracking; by image-based control; or by other operating methods. The medical visualization system or the operating microscope may include the necessary components. Image-based control may, in particular, include the generation of operating or control signals by evaluating at least one image produced by an image acquisition device for microscopic imaging or by an image acquisition device of an optical position detection system.

[0026] Adjustable operating parameters of the medical visualization system or the surgical microscope can be formed by one or more of the following parameters: • Magnification factor or zoom factor, • Working distance or focus position, • Detection range • Light intensity, • Illumination spectrum.

[0027] The proposed procedure comprises the following steps: a) Receiving at least one image signal generated by at least one image acquisition device of the operating microscope, representing an image of an examination area. The image signal can be received via an interface of the medical visualization system. In particular, the image signal can represent a two-dimensional image. The examination area can be a region of a patient's surgical site during an operation or during a diagnostic examination. The received image signal can preferably represent a white light image (VIS image) generated with visible radiation, i.e., radiation with wavelengths between 360 nm and 830 nm. Alternatively, the image can be provided as a fluorescence contrast image, generated by radiation with predetermined fluorescence-specific wavelengths or wavelength ranges, for example, wavelengths of 400 nm or 560 nm. The image can also be provided as a polarization contrast image, generated by radiation with a predetermined polarization. The image acquisition device of the surgical microscope can therefore be an image acquisition device for microscopic, i.e., magnified, imaging of the examination area, or it can be one of several different image acquisition devices. b) Dividing the image into a foreground and a background. This division can also be referred to as segmentation. In particular, pixels or image areas of the image are assigned to either the foreground or the background, or classified as either foreground or background. The division can be carried out in such a way that elements such as objects or structures, which are not to be overlaid by augmentation, are depicted in the foreground; these elements can be referred to as foreground elements. Foreground elements include, for example, instruments, especially surgical instruments, hands or fingers, or sections thereof. Correspondingly, elements, also referred to as background elements, are depicted in the background, which can be overlaid by augmentation. Background elements include, in particular, tissue.Background elements can also be instruments. In particular, so-called hybrid elements can exist, which can be background or foreground elements depending on the scenario. An example of a hybrid element could be a swab, which is classified as a background element, especially if it is static and / or unactivated within the area under investigation. However, the swab can also be classified as a foreground element, especially if it moves more than a predetermined amount and / or is actuated by a user. For the purposes of this invention, foreground elements are not limited to instruments. Furthermore, no object recognition is performed to detect foreground elements. In particular, both parts of the image in which tissue is depicted and parts of the image in which an instrument is depicted can be classified as parts of the background area. The division can be achieved, in particular, by creating an image mask that represents information about the foreground and background. Such an image mask will be explained in more detail below. The image mask can be represented or encoded by a transmittable signal. Of course, information about the division can also be provided in other ways. c) Receiving at least one signal containing additional information for augmentation. This signal, hereinafter referred to as the additional information signal, can be received via an interface of the medical visualization system. Examples of additional information have already been described. Preferably, the additional information signal represents a two-dimensional image of the examination area, which is provided based on information generated preoperatively or intraoperatively. The additional information signal can be generated by another component of the medical visualization system, such as another image acquisition device or a sensor. The additional information signal can also be retrieved from a storage device of the medical visualization system.It is also conceivable that the additional information signal is retrieved from a higher-level system, such as a network. The additional information signal represents or encodes information that is to be superimposed on the image of the area under investigation. d) Generating the augmented image by overlaying the background area or a part thereof with the additional information. In one embodiment, the overlay with additional information is performed exclusively in the background area or in a part of the background area.

[0028] The generated augmented image can then be transmitted, particularly as an image signal, to a display device, which is then controlled to output the image in a visually perceptible manner. In particular, a virtual (3D) image or an augmented (3D) image can depict visible and / or hidden objects or elements, thus enabling their visual representation.

[0029] In the case of a stereo operating microscope, image signals generated by the two image acquisition units of the stereo system, representing corresponding images of the examination area, can be received. Each of these corresponding images can then be divided into a foreground and a background area. Furthermore, after receiving at least one signal containing additional information, augmented images can be generated by superimposing the additional information in the background area onto each of the images. It is also possible to receive an image-specific additional information signal for augmentation for each of the corresponding images, which is then used to generate the augmented image. For example, with virtual image acquisition units, an additional information signal can be generated for each image, representing a virtual image composed of...generated from the additional information. The virtual image acquisition devices can be optical models of the stereo system's image acquisition devices. This advantageously generates a perspective-correct augmentation, particularly in a three-dimensional and consistent manner. A virtual image can encode a texture, especially with color and / or transparency information.

[0030] To generate the augmented image, the additional information can be introduced into the beam path, for example, by reflection. This information can be projected onto a projection element, such as a radiolucent disc, positioned in the beam path using a projection device of the operating microscope. The augmented image can then be generated by creating an image based on the beams into which the additional information has been introduced as described. Alternatively or additionally, the radiation representing the augmented image can also be provided via an output section for visual perception by a viewer.

[0031] Alternatively, an augmented image can be generated by computer-aided enhancement of an image of the real examination area, particularly through image processing. In this process, additional information can be superimposed onto the image of the real examination area. With a stereoscopic operating microscope, it is possible to provide the user with two augmented representations. Generally, corresponding additional information can be introduced into each of the two beam paths of a stereoscopic operating microscope. For example, with digital stereoscopic operating microscopes, augmented images can be generated from the images produced by both image acquisition units. Thus, an augmented image with depth information—that is, an augmented three-dimensional representation—can be provided to the user on a suitable display device or via an output section.

[0032] Additional information displayed to a user through augmentation can include, in particular, preoperatively generated information, such as preoperatively generated data, which can also be used for surgical planning. Such preoperatively generated data can be, in particular, volumetric data. Volumetric data can be provided as a point cloud, a voxel-based representation, or a mesh-based representation. The additional information can also be provided, in particular, as a transmittable signal.

[0033] Preoperative data can be generated, for example, using computed tomography (CT) or magnetic resonance imaging (MRI) methods. Other imaging techniques, particularly ultrasound, X-ray, fluorescence, SPECT (single-photon emission computed tomography), or PET (positron emission tomography) methods, can also be used. Such augmentation allows, for example, a tumor object or its contours, generated based on preoperative information, to be superimposed onto a white light image.

[0034] Preoperative data generated using magnetic resonance imaging (MRI) can identify various tissue types, such as adipose tissue, muscle tissue, tumor tissue, as well as blood vessels and nerve pathways. Preoperative data generated using computed tomography (CT) can particularly depict bony structures.

[0035] Alternatively or additionally to using preoperatively generated information to provide the augmented image, intraoperative information—that is, information acquired or generated during treatment—can be used as supplementary information for generating the augmented image. For example, information can be collected and stored during surgery and then subsequently used to generate an augmented image. This is particularly advantageous when different visualization modalities are activated at different times. For instance, fluorescence information can be displayed in a fluorescence visualization modality or superimposed on a white light image.

[0036] The additional information can be assigned a reference coordinate system, which means that the additional information can also include spatial information. This reference coordinate system can also be called a world coordinate system.

[0037] Augmentation typically requires registration between the reference coordinate system of the additional information and a reference coordinate system of the medical visualization system, particularly the operating microscope or its image acquisition unit. This registration can be performed prior to augmentation. The registration establishes a spatial relationship between both the additional information and the image to a common reference coordinate system, especially for the information in the image generated by the operating microscope's image acquisition unit. This common reference coordinate system, hereinafter also referred to as the reference coordinate system, can be, in particular, the reference coordinate system of the additional information, the reference coordinate system of the medical visualization system, or a different reference coordinate system altogether.

[0038] Various methods can be used for registration, for example, model-based registration. In this approach, features can be detected in an image that correspond to previously known features, such as geometric features in the supplementary information. The registration can then be determined based on these corresponding features. The registration can, for example, be defined in the form of a transformation matrix that includes a rotation and / or translation component. An example of model-based registration is edge-based registration, where the corresponding features are formed, for example, by a property of at least one, preferably several, edges in both the image and the supplementary information. Topography-based registration is also possible, particularly if a topography can be determined, e.g.,with a stereo system of an operating microscope. In this way, topographic information can be determined in at least one image, whereby corresponding features or points or sections are detected in both the additional information and this topographic information, which can then be used to determine the registration.

[0039] The additional information can therefore be registered information. For the purposes of this invention, the property "registered" can mean that a spatial reference to the reference coordinate system is known, in particular in the form of a transformation matrix. A registered device can generate signals whose spatial reference to the reference coordinate system is known.

[0040] Additional information can be generated, in particular, through rendering. For example, a virtual image can be created through rendering, which can then be used for augmentation and superimposed on a live-recorded image of the real examination area. The virtual image can also be provided as an image signal that encodes or represents the virtual image.

[0041] The virtual image can be generated using a virtual image acquisition device, which can be a mathematical or physical, and in particular computer-aided, optical model of an image acquisition device. Specifically, a computer-implemented calculation of the pixels of the virtual image can be performed. This virtual image depends, among other things, on parameters of the (modeled) image acquisition device. In particular, the virtual image can be generated for microscopic imaging based on the intrinsic parameters of the image acquisition device, especially with these parameters. If corresponding images of a virtual stereo system are generated, these can additionally be generated for microscopic imaging based on the extrinsic parameters of both image acquisition devices, especially with these parameters.In other words, when evaluating the model to generate virtual images, the parameters of the operating microscope's image acquisition device(s) used for microscopic imaging can be taken into account. This makes it possible to generate virtual images under the same conditions as real images.

[0042] The virtual image can also be generated depending on the pose, i.e., the position and / or orientation, of the (modeled) image acquisition device of the operating microscope. In particular, when evaluating the model to generate the virtual images, the pose of the operating microscope's image acquisition device(s) used for microscopic imaging can be taken into account, utilizing the registration information described above. By considering the registration information, it is possible, for example, to determine which pose of the virtual image acquisition device corresponds to the actual pose of the (modeled) image acquisition device of the operating microscope in the reference coordinate system of the additional information, which can also be referred to as the render coordinate system, and this information can then be used for the rendering process.In other words, a pose of at least one virtual image acquisition device can be identical to the pose of the modeled image acquisition device in the reference coordinate system. This makes it possible to create a virtual image that corresponds to the modeled image acquisition device in terms of both parameters and acquisition pose. For example, an image of a tumor object to be superimposed can be generated by rendering and then transmitted as an image or video signal and used for augmentation.

[0043] For the provision of virtual images, it may be necessary to determine the current pose of the operating microscope, particularly the image acquisition device. This pose can be determined using a position detection device. Registration allows a relationship to be established between the position detection device's reference coordinate system and the previously described reference coordinate systems, especially the reference coordinate system for the additional information. This makes it possible to determine the pose of the operating microscope within a desired reference coordinate system, particularly the reference coordinate system. Depending on the position of the operating microscope, the pose of the objective's optical axis or the position of a focal point can then be determined.If the operating microscope is attached to a stand with at least one joint, the pose of the operating microscope can also be determined depending on a joint position, whereby the joint position can be detected, for example, by a detection device or a sensor.

[0044] The position of the operating microscope and the additional information can define a previously described scene, i.e., a virtual spatial model that defines objects and their material properties, light sources, and the position and viewing direction of an observer, here the operating microscope.

[0045] Naturally, it is possible for the position detection device, or another position detection device, to also detect the pose of at least one other subject or object, or a part thereof. A subject can be, in particular, a user of the medical visualization system, such as someone observing the examination area or a display device. For example, it is conceivable to determine the pose of a body part of such a user, such as a hand, arm, or head. An object can be, in particular, another component of the medical visualization system, especially a display device. However, an object can also be an item that is not part of the medical visualization system, such as a piece of equipment like an operating table or a medical instrument. This makes it possible to determine the pose of the other subject or object within a desired reference coordinate system.

[0046] Such a position detection device can also be called a tracking system. A tracking system can be optical, electromagnetic, or operate in another way. The tracking system can be marker-based, detecting active or passive markers. Markers can be attached to objects or subjects whose pose is to be detected by the tracking system. An optical tracking system can, in particular, include optically detectable markers. An optical tracking system can, in particular, be a monoscopic position detection system. Here, the pose of an object can be determined by evaluating a two-dimensional image, in particular, exactly one two-dimensional image. Specifically, the position can be determined by evaluating the intensity values ​​of pixels (picture elements) of the two-dimensional image.

[0047] It is further conceivable that the medical visualization system includes at least one image acquisition device of an optical position detection device, which may in particular be a component of the operating microscope. This can also be referred to as a peripheral camera and serves in particular for monoscopic position detection.

[0048] A tracking system can also be part of an input device, where, for example, gesture control or gaze direction control is performed depending on information generated by the tracking system.

[0049] The method according to the invention advantageously enables an improved display quality of an area under investigation that is represented by an augmented image, since the superimposition exclusively in the background area reliably avoids a contradictory perception of information by a viewer, in particular since additional information, which as explained above lies behind a surface of the area under investigation, is not erroneously displayed in the foreground.

[0050] Unlike pure object recognition for detecting foreground elements, the proposed method offers the advantage of increasing the information content of the augmented image, since the assignment to foreground or background is not based on object recognition. Instead, it allows an object to be assigned to the foreground or background depending on the scenario.

[0051] In a further embodiment, at least two image signals are received, generated sequentially by the at least one image acquisition device of the operating microscope, each representing an image of the examination area. Furthermore, information about temporal changes in the image information as a function of these at least two images is determined, and the division of the image, in particular the image generated earlier or later, into foreground and background regions is performed based on this information. The change in the image information can be determined, in particular, as a function of the change itself or as the change between the at least two images, e.g., as the difference between the images or dependent on this difference.

[0052] Information about temporal changes in image information can preferably be information about the optical flow, which can be determined depending on at least two images. The optical flow information represents motion information, specifically representing the apparent movement of brightness patterns in an image sequence. The optical flow of the described image sequence can, for example, be determined as a vector field of the projected velocity of visible points in object space, particularly in the coordinate system of the image acquisition device. Specifically, each pixel of the image can be assigned a quantity representing the optical flow. In this case, a pixel can be assigned to the foreground if its assigned quantity is greater than a predetermined threshold.If the size assigned to a pixel is less than or equal to the predetermined threshold, the pixel can be assigned to the background area. This advantageously results in a simple and reliable division into foreground and background areas, enabling good display quality of the augmented image.

[0053] As an alternative to determining information about optical flow, other methods can be used to determine information about temporal changes in image information, e.g., methods for determining a motion field. The motion field provides motion information, particularly 3D motion, for each pixel of an image. Specifically, the motion field can represent the true, especially three-dimensional, motion at each point that is mapped onto the 2D image.

[0054] In a further embodiment, depth information is determined for at least one image of the examination area, with the division depending on this depth information. Depth information can, for example, represent the distance of the element imaged at a pixel from a reference point, a reference line, or a reference surface. This distance can be determined along a reference direction. In particular, the distance can be determined within the reference coordinate system. For example, the reference surface can be oriented perpendicular to the optical axis of the operating microscope, with the reference direction being parallel to the optical axis. The reference surface can, for example, include an intersection point between the optical axis and an end lens of the operating microscope.

[0055] Depth information can be determined using the described device for determining depth information, for example, a distance sensor. Such a distance sensor could be, for example, an OCT distance sensor, a lidar sensor, a time-of-flight sensor, or a triangulation sensor such as a fringe projection sensor. Of course, distance sensors that generate distance information according to other physical principles can also be used.

[0056] The device for determining depth information can be registered, meaning the depth information can be registered information.

[0057] It is also possible to determine, in particular, recorded reconstruction information from corresponding images of a stereo system, which represents a three-dimensional reconstruction of the detection area, whereby the depth information is determined depending on this reconstruction information. For example, it is possible to detect a surface of the investigation area in the three-dimensional reconstruction and then determine the distance from the previously described reference point, line, or surface.

[0058] Furthermore, it is possible to determine depth information from preoperatively generated data. For example, a skull surface or a dural surface can be detected in preoperative data, and then, as explained above, a distance from the previously described reference point, line, or surface can also be determined.

[0059] In particular, a depth map can be generated for the image of the area under investigation, assigning depth information to each pixel in the image. A pixel can then be assigned to the foreground if its depth information is greater than a predetermined minimum threshold and / or less than a predetermined maximum threshold. Otherwise, the pixel can be assigned to the background. It is also possible to determine a statistical parameter for the depth information generated in this way, such as a mean or median of all depth information assigned to the pixels. If the depth information assigned to a pixel is greater than or less than the median or mean, this pixel can be assigned to the foreground; otherwise, it is assigned to the background.This also advantageously results in an easily implementable and reliable division into foreground and background areas, which enables good display quality of the augmented image.

[0060] In a further embodiment, additional semantic information is determined for the image of the area under investigation, or at least a sub-area thereof, with the division being carried out depending on this additional semantic information. The additional semantic information can be information that is taken into account during the division in addition to other information, e.g., in addition to the previously described information about a temporal change in the image information and / or the depth information. In particular, in addition to information required for visually perceivable representation, such as color and / or transparency information, semantic information can be assigned to the image, especially to a pixel or a set of pixels.Semantic information can be, in particular, descriptive information about the type of depicted element, its relationship to other depicted elements, or to the environment. This information can be determined image-based, for example, using a semantic segmentation method. Additional semantic information can therefore be information about or derived from a semantic segmentation of a depicted scene, where the segmented scene can be divided into image areas, each assigned a class from a set of predefined classes. The set of predefined classes could, for example, include the class "fabric," the class "instrument," the class "cloth," the class "swab," and / or another class.

[0061] Semantic supplementary information can include, in particular, information that can be evaluated for the described division, i.e., division-relevant information. By evaluating semantic supplementary information, each pixel of the image can be classified as containing either a background or a foreground element. Specifically, the semantic supplementary information can differ from the information required for the visually perceptible representation of the image. Considering semantic supplementary information advantageously results in a very reliable division into foreground and background areas, which in turn improves the image quality.

[0062] In a further embodiment, contextual information is determined for the image of the area under investigation, or at least a sub-area thereof, with the division depending on this contextual information. Contextual information can, in particular, be information about a temporal or spatial context. Contextual information can also include information about the type of current user activity, the type of current operational phase, or the type of operation. In particular, the contextual information can differ from the information required for the visually perceptible representation of the image. By taking contextual information into account, a very reliable division into foreground and background areas is advantageously achieved, which in turn improves the image quality.

[0063] In another embodiment, the division is performed using a model generated by machine learning. The term machine learning here encompasses or refers to methods for determining the model based on training data. For example, the model can be determined using supervised learning methods, where the training data (i.e., a training dataset) comprises input data and output data. The input data can be images representing the area under investigation or the corresponding image signals, while the output data is the division of the respective image into foreground and background regions. For example, input and output data for such training data can be generated by having a user manually select the foreground and background regions in the images. This allows the model to learn the relationship between images and their division into foreground and background regions. However, it is also conceivable that unsupervised learning methods could be used to determine the model.

[0064] Suitable mathematical algorithms for machine learning include: Decision Tree-based methods, Ensemble Methods (e.g., Boosting, Random Forest)-based methods, Regression-based methods, Bayesian Methods (e.g., Bayesian Belief Networks)-based methods, Kernel Methods (e.g., Support Vector Machines)-based methods, Instance (e.g., k-Nearest Neighbour)-based methods, Association Rule Learning-based methods, Boltzmann Machine-based methods, Artificial Neural Networks (e.g., Perceptron)-based methods, Deep Learning (e.g., Convolutional Neural Networks, Stacked Autoencoders)-based methods, Transformer-based methods, Dimensionality Reduction-based methods, and Regularization Methods-based methods.

[0065] After the model has been created, i.e., after the training phase, the parameterized model can be used in the so-called inference phase to generate the foreground and background partitioning from images. This results in a reliable and high-quality partitioning. The model can be trained once, preferably with a sufficiently large dataset (training phase), and then used as a model (inference phase).

[0066] In another embodiment, the partitioning is performed using a neural network. For example, the neural network can be configured as an autoencoder, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a neural transformer network, or a combination of at least two of the aforementioned networks. Such a neural network, particularly the autoencoder version, can be trained using the training data described above, after which the partitioning can be performed. The autoencoder configuration of the neural network advantageously minimizes the computational effort required for the partitioning, enabling it to be performed reliably and quickly, especially by embedded systems.

[0067] Training a CNN advantageously reduces network complexity, making it suitable for devices with limited computing power. This applies to both the training and inference phases. Furthermore, the training time required for CNNs is short, particularly compared to LSTM networks, which also require comparatively higher computing power. However, LSTM network training is especially well-suited for time series analysis because its architecture incorporates temporal dependencies. This results in a high-quality partitioning.

[0068] In a further embodiment, input variables for the model-based division include, in addition to the at least one image of the investigation area or the corresponding image signal, at least one of the following pieces of information: a) Depth information for at least one image of the investigation area, b) at least one image of the area under investigation that was generated prior to the image of the area under investigation, c) an image corresponding to the image of the examination area, which was generated by another image acquisition device of the operating microscope, d) Information on the classification of depicted objects, e) Information for classifying a user activity, f) Information on the classification of a surgical phase, g) Information on the classification of a type of operation.

[0069] If the operating microscope includes a stereo system with two image acquisition units, and the image of the examination area was generated by one of the image acquisition units, the corresponding image described above can be generated by the remaining image acquisition unit of the stereo system. The information according to points d) and e) can constitute semantic (additional) information. The information according to points f) and g) can constitute contextual information. Thus, input variables for the model-based partitioning can be image information, e.g., in the form of a microscopic white-light image and, if applicable, an image according to point b) or c), and / or image processing information, e.g., information about a temporal change in the image information and / or information according to point a), and / or semantic (additional) information and / or contextual information.

[0070] Information for classifying depicted objects can be information about a class to which a depicted object is assigned. An example class could be "fabric" or "instrument". Information for classifying user activity can be information about a class of user activity, for example, the class "cutting", "suction", "ablating", "coagulating", "retracting" or "swabbing".

[0071] Accordingly, information for the classification of an operational phase can be information about the class of the current operational phase, for example the class "opening", "exploration", "resection", "reconstruction", "wound closure", "wound healing", "planning of opening", "exposure".

[0072] Information regarding the classification of the surgical type can be a class of surgical type, for example, "neurosurgical surgery," "orthopedic surgery," or "ophthalmic surgery." Neurosurgical surgeries can also be classified into classes such as "tumor surgery," "vascular surgery," or "spinal surgery." Ophthalmic surgeries can also be classified into "anterior segment surgery" and "posterior segment surgery," with anterior segment surgery including procedures such as cataract treatment or Descemet membrane endothelial keratoplasty. Posterior segment surgery includes procedures such as (membrane) peeling or vitrectomy.

[0073] Depth information can be determined as previously explained. Classification information can be determined by a classification device, whereby the classification can be image-based, i.e., by evaluating at least one image, particularly from the image acquisition device of the operating microscope or a different image acquisition device. The classification information can also be predetermined. It is also possible for the classification information to be determined based on control signals generated for or within the medical visualization system. Such information can also be generated by operating an input device.

[0074] By taking into account one, several or all of the aforementioned information in addition to the at least one image of the area under investigation, it advantageously results in the reliability and accuracy of the division into foreground and background areas being improved, which in turn improves the user's perception of the augmented image.

[0075] According to the invention, after dividing the image of the area under investigation into foreground and background areas, but before superimposition, the size of the foreground area is changed, particularly at least with respect to a sub-area of ​​the foreground area. The size of the foreground area can be enlarged or reduced, in particular by assigning pixels from the background area surrounding a sub-area of ​​the foreground area to the foreground area, or by assigning pixels from a sub-area of ​​the foreground area to the background area. If the foreground area is enlarged, the background area is reduced. If the foreground area is reduced, the background area is enlarged accordingly. The change can be geometrically shaped; for example, a circular, ellipsoidal, rectangular, or square enlargement or reshaping can be achieved.The reduction in size occurs, whereby, for example, all pixels arranged in a geometric shape around a reference point of the sub-area are assigned to the foreground. Properties of the change, such as a geometric shape and / or the size of the change, can be predetermined.

[0076] Alternatively or cumulatively, the size of the background area is changed, in particular at least with respect to a sub-area of ​​the background area, especially by being enlarged or reduced. The explanations given for changing the foreground area apply accordingly.

[0077] By subsequently altering the size of the foreground and / or background area in this way, it advantageously enables further improved display quality for the augmented image, particularly in certain application-dependent scenarios. For example, an image area that shows a part in contact with the tissue, such as the tip of an instrument, and which is classified as the foreground area, can be enlarged to allow the user to perceive the contacted tissue without augmentation interference. It is also possible to reduce the size of foreground sections containing little information. This further improves the display quality of the provided augmented image.

[0078] In a further embodiment, a sub-area of ​​the foreground or background to be modified is determined by an object detection method. The sub-area to be modified can designate a sub-area of ​​the foreground or background with respect to which the size of the foreground or background is changed. Alternatively, the sub-area to be modified can also designate a sub-area whose size is changed.

[0079] As explained above, the size of the foreground area can be increased or decreased, in particular by assigning pixels from the background area in a section adjacent to, or at least partially surrounding, a sub-area of ​​the foreground to the foreground, or by assigning pixels from a sub-area of ​​the foreground to the background. Similarly, the size of the background area can be increased or decreased, in particular by assigning pixels from the foreground area in a section adjacent to, or at least partially surrounding, a sub-area of ​​the background to the background, or by assigning pixels from a sub-area of ​​the background to the foreground.

[0080] The object recognition method can, in particular, be an image-based method, which assigns at least one pixel to an object by evaluating at least one image. Such methods are known to those skilled in the art. For example, it is possible to identify specific sections, such as the tip of an instrument, in the image using an object recognition method. The size of the foreground area can then be increased with respect to this section. In this case, the properties of the change can be object-dependent, with predetermined properties of the change being assigned to an object. This advantageously results in a reliable identification of a section, which in turn improves the previously described image quality.

[0081] In another embodiment, a sub-area to be modified is determined depending on at least one optical parameter of the operating microscope. Optical parameters can be, in particular, the operating parameters described above. For example, an image area depicting foreground elements that are not arranged within a predetermined interval around a set working distance of the operating microscope can be defined as the sub-area of ​​the foreground to be reduced in size. It can be assumed that such foreground elements are rendered blurry by the microscope, i.e., with insufficient image quality, where image quality can, in particular, represent image sharpness. It can be advantageous to enlarge or reduce such foreground sub-areas with insufficient image quality.In other words, an image area with insufficient image quality can be enlarged or reduced, whereby the image quality may depend, for example, on a degree of blurriness of the elements depicted in the image area.

[0082] By reducing the size of such a (foreground) area, an additional image area can be created for augmentation, but this additional image area only includes areas in which image information such as tissue or instruments are depicted with insufficient image quality.

[0083] Enlarging such a (foreground) area can prevent augmentation with sufficient image quality from occurring next to an area of ​​insufficient image quality, for example, by creating a sharp image area next to a blurry one. This type of representation can advantageously reduce viewer confusion. Such enlargement can be particularly useful when the image quality is below a predetermined level.

[0084] It is also conceivable that augmented information, particularly in the vicinity of an image area with insufficient image quality, could be displayed with the same insufficient image quality. Such a display of augmented information is particularly possible if the image quality is better than or equal to a predetermined level.

[0085] This also allows for the advantageous and reliable identification of sub-areas that need to be changed, which in turn improves the previously explained presentation quality.

[0086] Furthermore according to the invention, the change depends on a) a distance of the object or element depicted in the sub-area from a surface of the investigation area, b) carried out the processing of sub-area-specific image information.

[0087] In particular, a property of the change can be chosen to depend on at least one of the aforementioned quantities. For example, the magnitude of the change can be positively or negatively correlated with the distance. The distance can be determined, in particular, depending on the depth information explained above. Of course, other methods of determining the distance are also conceivable.

[0088] Image information can be a property of the image and can be determined image-based. It can be, in particular, blur information, shadow information, or contrast information. Sub-area-specific image information can also be determined by a suitable device. In particular, this information can be determined image-based. For example, it is possible to perform a blur classification procedure, a shadow classification procedure, and / or a contrast classification procedure to identify image areas, especially in the foreground, that are blurred and / or shadowed by more than a predetermined degree and / or lack sufficient contrast. Such areas can then be reduced in size, or even completely removed from the foreground. It is also conceivable that such areas could be enlarged.

[0089] The image quality in a specific area can also be determined based on the image information, such as blur information, shadow information, and / or contrast information, whereby enlargement and / or reduction can then be adjusted according to the image quality. Reference can be made to the preceding explanations for further details.

[0090] This has the advantageous effect of allowing even parts of the foreground to be used for augmentation, areas that would otherwise provide little information to the user. This, in turn, improves the display quality and increases the information content of the augmented image.

[0091] In another embodiment, a sub-area to be modified and / or the change in size is determined by means of a model generated by machine learning, in particular by means of a neural network. Reference can be made to the preceding explanations regarding the division of the image into a foreground and a background area using a model. The input data for a training dataset can consist of images and the division of these images into foreground and background areas, or the corresponding image signals. The output data can be information about selected sub-areas and / or the changed division into foreground and background areas. For example, input and output data for such training data can be generated by a user manually selecting the sub-areas and / or the changes to the foreground and background areas.This allows the model to learn the relationship between images and their division into foreground and background, as well as subsequent changes to this division. This results in a reliable and high-quality determination of the area to be changed and / or the change in size.

[0092] In a further embodiment, input variables for determining the sub-area to be modified and / or the change in size include at least one of the previously described pieces of information a) to g) for the model-based partitioning. This advantageously improves the reliability and accuracy of determining the sub-area to be modified and / or the change in size, which in turn improves the user's perception of the augmented image.

[0093] A further proposed medical visualization system comprises, in particular, an operating microscope, which includes at least one interface for receiving an image signal from an image acquisition device for generating an image of an examination area, and at least one evaluation device. The medical visualization system, in particular the evaluation device, is configured to perform a method according to one of the embodiments described in this disclosure. As explained above, the medical visualization system may, in particular, further comprise at least one of the following: - a device for determining information for the classification of depicted objects and / or a user activity and / or an operation phase and / or an operation type, - a device for determining the distance of an imaged object from a surface of the investigation area, - a device for determining area-specific blur and / or shadow information.

[0094] The evaluation unit can be configured as a computing unit or include one. A computing unit, in turn, can include at least one microcontroller and / or at least one integrated circuit, or be configured as such. The computing unit can, in particular, generate the virtual image. It is possible that the evaluation unit includes at least one graphics processing unit (GPU) for this purpose. The medical visualization system advantageously enables the execution of a method according to one of the embodiments described in this disclosure, with the advantages already explained.

[0095] A further proposal is a computer program product comprising a computer program, wherein the computer program includes software means for executing several or all steps, in particular steps b. and c., of the method according to one of the embodiments described in this disclosure, when the computer program is executed by or in a computer or an automation system. The computer or automation system may include the evaluation device described above. The computer program product may, in particular, include means for performing the rendering, i.e., a rendering engine. The computer program product advantageously enables the execution of a method according to one of the embodiments described in this disclosure with the advantages already explained.

[0096] The invention is explained in more detail using exemplary embodiments. The figures show: Fig. 1a an exemplary representation of a superimposition without a division into foreground and background area according to the invention, Fig. 1b another exemplary representation of an augmented image without a division into foreground and background area according to the invention, Fig. 2 a schematic flowchart of a method according to the invention, Fig. 3 an exemplary representation of an augmented image produced using the method according to the invention, Fig. 4a a schematic representation of an investigation area and an instrument, Fig. 4b a schematic representation of an image of the in Fig. Scene 4a depicted, Fig. 4c a schematic representation of information on optical flow, Fig. 4d a schematic representation of an image mask, Fig. 5a a schematic representation of an investigation area and an instrument, Fig. 5b a schematic representation of an image of the in Fig. Scene 5a depicted, Fig. 5c a schematic representation of depth information, Fig. 5d a schematic representation of an image mask, Fig. 6 a schematic flowchart of a method according to the invention in a further embodiment, Fig. 7 a schematic flowchart of a method according to the invention in a further embodiment, Fig. 8a a schematic representation of an image mask, Fig. 8b based on the in Fig. augmented image generated by the image mask shown in 8a, Fig. 9a a schematic image mask with locally enlarged foreground area, Fig. 9b based on the in Fig. augmented image generated by the image mask shown in 9a, Fig. 10a a schematic representation of an operation scene, Fig. 10b a schematic representation of additional information to be augmented, Fig. 10c an exemplary image mask, Fig. 10d based on the in Fig. augmented image generated by the image mask shown in 10c, Fig. 10e another exemplary image mask, Fig. 10f based on the in Fig. 10e shown image mask, augmented image created, Fig. 10g another exemplary image mask, Fig. 10h one based on the in Fig. 10g displayed image mask augmented image created, Fig. 11 a schematic block diagram of a medical visualization system according to the invention.

[0097] In the following, identical reference symbols denote elements with the same or similar technical characteristics.

[0098] Fig. Figure 1a shows an exemplary white-light image of an examination area 1 with instruments 2. A virtual object 3, for example a tumor object 3, is superimposed on the image. This superimposition is done without considering areas in the image that are suitable for augmentation. It is evident that the tumor object 3 obscures parts of an instrument 2 and the tissue.

[0099] Fig. 1b shows the same scene as Fig. 1a, wherein the tumor object 3 is more transparent than in Fig. 1a, in particular semi-transparent, is shown. Despite the now possible perceptibility of in Fig. 1a Although the tumor object 3 obscures parts of the instrument 2, the perception of the tissue areas and instrument sections obscured by the tumor object 3 is nevertheless impaired. Furthermore, the spatial impression arises that the tumor object 3 is hovering above the tissue, which makes a spatially accurate perception of the scene difficult for an observer.

[0100] Fig. Figure 2 shows a schematic flowchart of a method according to the invention for generating an augmented image AA by a medical visualization system 4 (see Fig. 11). In a receiving step ES1, an image signal is received, which is used by at least one image acquisition device 5 for microscopic imaging (see e.g. Fig. 11) was generated and which represents an image of an investigation area 1. This image A1 is split in a partitioning step AS into a foreground area V and a background area H (see e.g. Fig. 4d) divided. Examples of possible divisions are explained below.

[0101] In a further reception step ES2, at least one signal containing additional information ZI for augmentation, also referred to as the additional information signal, is received. Fig. Figure 2 shows that the additional information signal is stored in a retrievable manner in a storage device 6, which may be part of the medical visualization system 4. However, it is also possible for the additional information ZI to be acquired intraoperatively and / or retrieved from a network via a suitable interface. The additional information signal can, in particular, also represent an image, preferably an image of the same size as the image A1 generated by the image acquisition device 5. The image represented by the additional information signal can, in particular, be a virtual image generated from preoperatively generated additional information ZI. As explained at the outset, such a virtual image can be generated with a virtual image acquisition device, which is an optical model of the image acquisition device 5.

[0102] In a generation step GS, the augmented image AA is then created by overlaying the additional information ZI onto the image A1 in the background area H or a part thereof.

[0103] In the partitioning step AS, an image mask M can be generated that encodes or represents information about the foreground region V and the background region H. The image mask M can be provided, in particular, as an image, especially a two-dimensional image, where the image size of the image mask M can correspond to the image size of the image A1. Each pixel of the image mask M can be classified as either a foreground region V or a background region H. The image mask M can, for example, be a binary image, where foreground region pixels are assigned the value 1 or 0, and background region pixels are assigned the remaining value.

[0104] In this case, the overlay of the additional information ZI can only occur for pixels of image A1 whose corresponding pixels in the image mask M are classified as background area pixels. A corresponding pixel can have the same pixel coordinates, which can refer to the same image coordinate systems. In particular, the overlay can be opaque or with a predetermined degree of transparency, especially semi-transparent. For example, an alpha blending method can be used for the overlay. In this case, each background area pixel of the image mask can be assigned an alpha value between 0 (inclusive) and 1 (inclusive), where the value 0 or 1 represents, for example, complete transparency and the value 1 or 0 represents complete opacity.If a background pixel in image mask M is assigned an alpha value representing complete opacity, the corresponding pixel in image A1 can be completely, i.e., opaquely, overlaid by additional information. If a background pixel in image mask M is assigned an alpha value that represents neither complete transparency nor complete opacity, the corresponding pixel in image A1 will not be completely, opaquely, overlaid by additional information. If a background pixel in image mask M is assigned an alpha value representing complete transparency, the corresponding pixel in image A1 cannot be overlaid by additional information.

[0105] It is also conceivable that at least one, but preferably several, foreground pixel(s) of the image mask are assigned an alpha value that does not represent complete transparency, where the minimum transparency level of all foreground pixels is greater than the maximum transparency level of all background pixels. Thus, an augmented image can be created in which an overlay in a foreground area is displayed more transparently than an overlay in a background area. In this way, a viewer can also perceive additional information in the foreground area; however, this information is less distracting due to the higher transparency.

[0106] It is possible to assign alpha values ​​between the measure of complete transparency (exclusive) and the measure of complete opacity (inclusive) to all background area pixels, while foreground area pixels are assigned an alpha value with the measure of complete transparency.

[0107] The augmented image AA can then be sent to a display device 20 (see Fig. 11) be transmitted, in particular as an image signal, in order to present it to a viewer in a visually perceptible manner.

[0108] Fig. Figure 3 shows an exemplary augmented image AA, which is shown in Fig. The process described in section 2 was used. Fig. In the scene depicted, the additional information ZI represents tumor object 3. It is evident that tumor object 3 differs from the ones in Fig. 1a and Fig. In the images shown in 1b, the instruments 2 are not obscured. Thus, the instruments 2 are depicted in the foreground area V of image A1, while tissue areas of the examination area 1 are depicted in the background area H. This is shown in Fig. The 3 augmented image AA shown allows a viewer an improved perception of the scene and thus offers a higher display quality.

[0109] Fig. Figure 4a shows an exemplary representation of an instrument 2 and an examination area 1, in particular a surgical surface. Also shown is an image acquisition device 5 of an operating microscope.

[0110] Fig. Figure 4b shows an image A1 generated by the image acquisition device 5. It depicts the instrument 2 and the examination area 1, with the instrument 2 obscuring parts of the examination area 1.

[0111] Fig. Figure 4c shows, by way of example and indicated by arrows, information on the optical flow in image A1. This information on the optical flow can be generated by receiving at least two image signals, which were generated sequentially by the image acquisition device 5 and each represent an image A1 of the area under investigation 1. Methods for determining this information from the sequence of these images A1 are known to those skilled in the art. As indicated by the arrows, a quantity representing the optical flow, which is greater than a predetermined threshold, is assigned to a sub-region of image A1 into which the instrument 2 is imaged. The pixels of this sub-region can then be classified as foreground region V, i.e., foreground region pixels, while the remaining pixels of image A1 are classified as background region H, i.e., background region pixels.The resulting image mask M, consisting of foreground pixels and background pixels, is in . Fig. Illustrated in 4D.

[0112] Fig. Figure 5a shows an exemplary representation of an instrument 2 and an examination area 1, in particular a surgical surface. Also shown is an image acquisition device 5 of an operating microscope.

[0113] Fig. Figure 5b shows an image A1 generated by the image acquisition device 5. It depicts the instrument 2 and the examination area 1, with the instrument 2 obscuring parts of the examination area 1.

[0114] Fig. Figure 5c shows an example of depth information in image A1. This information can be generated, for example, with a distance sensor that detects the distance of the instrument and the unobstructed part of the examination area from a reference plane oriented perpendicular to an optical axis of the operating microscope, where the intersection of the optical axis with a lens of the operating microscope is located. It can be seen that distance information is assigned to a sub-area of ​​image A1 in which instrument 2 is depicted. This distance represents a smaller distance than the distance assigned to the sub-area of ​​image A1 that depicts the unobstructed part of the examination area. In particular, this smaller distance is less than a predetermined distance. The pixels of this instrument sub-area can then be classified as the foreground area V, i.e., as foreground area pixels, while the remaining pixels of image A1 are classified as the background area H, i.e., as background area pixels. The resulting image mask M, consisting of foreground area pixels and background area pixels, is in Fig. 5D representation.

[0115] Fig. Figure 6 shows a schematic flowchart of a further embodiment of a method according to the invention. In contrast to the one in Fig. In the embodiment shown in section 2, in addition to the information necessary for the visually perceptible representation of image A1, in particular color information, further information I is taken into account for the division step AS. Such further information I can be, in particular, additional semantic information or contextual information. In particular, such semantic information or contextual information can relate to Fig. 5c explains the depth information for image A1. If the image acquisition device 5, with which image A1 was generated, is part of a stereo system, then the information I can also represent information about an image corresponding to image A1, which is generated by another image acquisition device 5b (see Fig. 11) was generated. The information I can also represent at least one image of the investigation area 1 that was generated before the image A1.

[0116] Preferably, the information I also includes information about a classification of objects depicted in image A1. For this purpose, object recognition can be performed, whereby the recognized objects are then classified using a classification procedure. However, this classification does not provide a classification in the foreground and background areas V and H; in particular, for at least one object class from the set of all object classes, there is no unambiguous assignment to the foreground area V or background area H. For example, objects of the class "cloths" or "swabs" can be assigned to the background area, especially if they are static, e.g., fixed in position relative to the site or lying on the fabric of the site.Such objects of the class "cloths" or "swabs" can also be assigned to the foreground, especially if they are not statically arranged, particularly because they are being held or even moved by an instrument. Similarly, an object or area of ​​the class "fabric" can be assigned to the background, especially if there is no object or area of ​​the class "instrument" within that fabric area or within a predetermined area surrounding it. Alternatively, an object or area of ​​the class "fabric" can also be assigned to the foreground, especially if there is an object or area of ​​the class "instrument" within that fabric area or within a predetermined area surrounding it. In the latter case, it can be avoided that an augmentation would disturb a viewer when looking at areas with which an instrument is currently interacting.

[0117] The information can also include information about the classification of a user activity, an operation phase, or an operation type. These have already been explained previously.

[0118] It is possible that the partitioning step AS is performed using a model, or by evaluating a model, generated through machine learning, particularly through the evaluation of a neural network. An input to the model can be at least one image A1 of the investigation area 1. Another input can be the previously described information I. An output of the model can be the previously described image mask M.

[0119] Fig. Figure 7 shows a schematic flowchart of a further embodiment of a method according to the invention. In contrast to the one in Fig. In the embodiment shown in Figure 6, a modification step VS is performed after the partitioning step for generating the image mask M. In the modification step, the size of the foreground area V is changed, at least with respect to a sub-area of ​​the foreground area V. As a result of the modification step VS, a modified image mask MV is provided. This corresponds to the image mask M, but includes more or fewer foreground pixels and thus correspondingly fewer or more background pixels compared to the image mask M. The dashed lines indicate that the partitioning step AS and / or the modification step VS can optionally be performed depending on the additional information I explained above.

[0120] Fig. Figure 8a shows a schematic image mask M with three foreground elements, which correspond, for example, to image sections of the image A1, into which instruments 2 (see Fig. 3) are shown.

[0121] Fig. Figure 8b represents the augmented image AA, which is generated based on this image mask M. It can be seen that an augmented tumor object 3 does not cover the depicted instruments 2.

[0122] Fig. Figure 9a shows an exemplary modified or altered image mask MV. Unlike the one in Fig. In the image mask M shown in Figure 8a, it is evident that the foreground area V has been enlarged in the region of a tip or free end of the instruments 2. Specifically, the enlargement of the foreground area V was achieved by defining a circular area with a predetermined diameter around a reference point of the sub-area, in this case, for example, around the geometric center of an image mask section classified as an instrument tip, and classifying all pixels of the modified image mask MV within this circular area as foreground pixels.

[0123] Fig. Figure 9b shows the augmented image generated based on the modified image mask MV. In contrast to the one in Fig. In the augmented image AA shown in Figure 8b, it is evident that a portion of the tissue surrounding the instrument tips is not augmented by the tumor object 3. This allows the viewer to perceive the tissue actuated by the instruments without augmentation.

[0124] Fig. Figure 10a shows an exemplary image A1, which was produced by an image acquisition device 5 of a medical visualization system 4 (see Fig. 11) was produced.

[0125] Fig. Figure 10b shows an exemplary representation of a tumor object 3, which is represented by an additional information signal.

[0126] Fig. Figure 10c shows an exemplary image mask M with foreground area V and background area H, which is shown without the one in Fig. The change step VS shown in section 7 was generated.

[0127] Fig. Figure 10d shows the augmented image AA generated on the basis of this image mask M.

[0128] Fig. 10e shows a comparison with the one in Fig. The image mask M shown in 10c is modified by the image mask MV with a resized foreground and background area V, H. In the corresponding modification step VS (see Fig. 7) Shadowed image sections vB of image A1 were detected, with the foreground area V of the in Fig. The image mask M shown in Figure 10c has been reduced by the number of shadowed image areas. Such shadowed image areas vB can be identified, for example, using shadow classification methods. One such classification method compares the color values ​​of pixels in image A1 with predetermined threshold values ​​and classifies a pixel as a shadowed pixel based on the comparison result.

[0129] Fig. Figure 10f shows an augmented image AA, which is based on the in Fig. The modified image mask shown in Figure 10e was created. It is evident that additional information (ZI) can also be superimposed on image A1 in the shaded image area (vB).

[0130] Fig. 10g shows a difference compared to the one in Fig. The image mask M shown in 10c is modified by the image mask MV with a resized foreground and background area V, H. In the corresponding modification step VS (see Fig. 7) In addition to the shadowed image areas vB, blurred image sections uB of image A1 were detected, with the foreground area V of the in Fig. The image mask M shown in Figure 10c has been reduced by the area of ​​the image that is out of focus. Such out-of-focus image areas uB can be determined, for example, using blur classification methods. An exemplary classification method for detecting out-of-focus areas can include the application of Laplacian operators, in particular the application of Laplacian pyramids, and threshold operators.

[0131] Fig. 10h shows an augmented image AA, which is based on the in Fig. The modified image mask shown in 10g was generated. It is evident that additional information (ZI) can also be superimposed on image A1 in the blurred image area.

[0132] Fig.Figure 11 shows a schematic block diagram of a medical visualization system 4 and an examination area 1. Also shown are an instrument 2 and a tumor object 3, which are arranged within the detection range of an operating microscope 10 of the medical visualization system 4. The tumor object 3 can be a hidden object.

[0133] Optional elements of the medical visualization system 4 are shown here with dashed lines. The medical visualization system 4 comprises at least one image acquisition device 5 for microscopic imaging of the examination area 1. The image acquisition device 5 can be part of the operating microscope 10, which may include an objective 19 with a lens. This operating microscope 10, in turn, may be configured as a stereo operating microscope, comprising a further image acquisition device 5b for microscopic imaging of the examination area 1, and the image acquisition devices 5 and 5b forming a stereo system. A storage device 6, in which additional information (ZI) may be stored, is also shown.The medical visualization system 4 further comprises an evaluation unit 7, which can receive and evaluate an image A1 generated by the image acquisition unit 5, whereby the image A1 can be transmitted as an image signal to the evaluation unit 7 via an interface 21. The evaluation unit 7 can, in particular, perform the splitting step AS and the generation step GS.

[0134] The figure further shows that the medical visualization system 4 can include a device 8 for determining depth information. This device can, for example, be designed as a distance sensor or include one. Also shown is an interface 9 of the medical visualization system 4 for data transmission with other, especially higher-level, systems.

[0135] Furthermore, the medical visualization system can include 4: • at least one white light lighting device 11, • at least one infrared lighting device 12, • at least one fluorescence illumination device 13 for excitation of fluorescence radiation, • at least one fluorescence detection device 14 for detecting fluorescence radiation, • at least one surround-view camera 15, • at least one device 16 for detecting the gaze direction of a viewer, • at least one tracking system 17, • at least one input device 18 for operating or controlling the medical visualization system, • at least one display device 20.

[0136] The elements of the medical visualization system 4 can be connected via data and / or signal technology.

[0137] Not shown are beam filters for providing excitation radiation with wavelengths from a broader spectrum, e.g. the spectrum of the white light lighting device, or for filtering radiation from a broader spectrum.

[0138] The environmental camera 15 can be part of a further tracking system, which serves in particular for the optical determination of the pose of instruments within the detection range of the environmental camera 15. The pose determination can be monoscopic. In particular, the determination can also be marker-based. Images from the environmental camera 15 can be evaluated, in particular, for object recognition in order to identify a sub-area with respect to which the size of the foreground area V and / or background area H is then changed. Reference symbol list 1 examination area 2 Instrument 3 Tumor object 4 medical visualization system 5 Image acquisition device of an operating microscope 5b further image acquisition device of the operating microscope 6 Storage setup 7 Evaluation unit 8 Device for determining depth information 9 Interface 10 Operating microscope 11 White light lighting device 12 Infrared lighting device 13 Fluorescence lighting device 14 Fluorescence detection device 15 Surround camera 16 Device for gaze direction detection 17 Tracking system 18 Input device 19 Lens 20 Display device 21 Interface A1 image M Image mask MV modified image mask I Information ZI Additional Information V Foreground area H Background area AA augmented image ES1 receive step AS division step ES2 receive step GS production step VS change step

Claims

[1] Method for generating an augmented image (AA) by a medical visualization system (4), comprising the steps: a. Receiving at least one image signal generated by at least one image acquisition device (5) of an operating microscope (10) and representing an image (A1) of an examination area (1), b. Dividing the image (A1) into a foreground area (V) and a background area (H), c. Receiving at least one signal that represents or encodes additional information (II) for augmentation, d. Generating the augmented image (AA) by superimposing the background area (H) of the image (A1) of the investigation area (1) or a part thereof with the additional information (ZI), wherein after dividing the image (A1) of the investigation area (1) into the foreground area (V) and the background area (H) - a size of the foreground area (V) and / or - the size of the background area (H) is changed, with the change depending on - a distance of an object mapped into a sub-area from a surface of the investigation area (1) and / or - is carried out using sub-area-specific image information. [2] Method according to claim 1, characterized by , that the overlay with additional information (ZI) is carried out exclusively in the background area (H) or in a part of the background area (H). [3] Method according to claim 1 or 2, characterized by, that at least two image signals are received which were generated successively by the at least one image acquisition device (5) of the operating microscope (10) and which each represent an image (A1) of the examination area (1), wherein information about a temporal change of the image information depending on these at least two images (A1) is determined and the division of the image (A1) depending on this information is carried out in relation to the optical flow. [4] Method according to any of the preceding claims, characterized by , that depth information is determined for at least one image (A1) of the investigation area (1) and the division is carried out depending on this depth information. [5] Method according to any of the preceding claims, characterized by, that for at least one image (A1) of the investigation area (1) or at least a sub-area thereof, additional semantic information is determined, whereby the division is carried out depending on this additional semantic information. [6] Method according to any of the preceding claims, characterized by , that for at least one image (A1) of the investigation area (1) or at least a sub-area thereof, context information is determined, whereby the division is carried out depending on this context information. [7] Method according to any of the preceding claims, characterized by that the division is carried out using a model generated by machine learning, preferably using a neural network. [8] Method according to claim 7, characterized by, that input variables for the model-based partitioning include at least one image (A1) of the investigation area (1) in addition to the at least one piece of information: a. Depth information for at least one image of the investigation area, b. at least one image of the area under investigation generated prior to the image of the area under investigation, c. an image corresponding to the image of the examination area, which was generated by another image acquisition device (5b) of the operating microscope (10), d. Information on the classification of depicted objects, e. Information for classifying user activity, f. Information on the classification of an operational phase, g. Information on the classification of a type of operation. [9] Method according to any of the preceding claims, characterized by, that a sub-area to be changed of the foreground area (V) and / or the background area (H) is determined by a method for object recognition and / or that a sub-area to be changed is determined depending on optical parameters of the operating microscope (10). [10] Method according to any of the preceding claims, characterized by , that a sub-area to be changed and / or the change in size is determined by means of a model which was generated by machine learning. [11] Method according to claim 10, characterized by , that input variables for a model-based determination of the sub-area to be changed and / or the change in size include at least one of the following pieces of information: a. Depth information for at least one image of the investigation area, b. at least one image of the area under investigation generated prior to the image of the area under investigation, c. an image corresponding to the image of the examination area, which was generated by another image acquisition device (5b) of the operating microscope (10), d. Information on the classification of depicted objects, e. Information for classifying user activity, f. Information on the classification of an operational phase, g. Information on the classification of a type of operation. [12] Medical visualization system (4) comprising at least one interface (20) for receiving an image signal from an image acquisition device (5) for generating an image (A1) of an examination area (1) and at least one evaluation device (7), wherein the medical visualization system (4) is configured to perform a method comprising the steps according to any one of claims 1 to 11. [13] Computer program product comprising a computer program, wherein the computer program comprises software means for performing several or all steps of the method according to any one of claims 1 to 11, when the computer program is executed by or in a computer or an automation system.

Images