Apparatus for an optical imaging system, optical imaging system, method and computer program

Abstract

Examples relate to an apparatus for an optical imaging system comprising one or more processors and one or more storage devices. The apparatus is configured to obtain sample data from a sensor of a microscope of an optical imaging system. The sample data is indicative of an image of the sample. Further, the apparatus is configured to determine segmented data indicative of a segmentation of the image of the sample into a plurality of regions and to obtain object data indicative of an object for guidance of the user of the optical imaging system. The apparatus is further configured to determine, based on the object data and the segmented data, output data indicative of a composed output image of the image of the sample and the object for guidance, such that an opaque region of the plurality of regions overlays the object for guidance. Further, the apparatus is configured to transmit the output data for displaying on a display device.

Description

[0001] Apparatus for an Optical Imaging System, Optical Imaging System, Method and Computer Program

[0002] Technical field

[0003] Examples relate to an apparatus for an optical imaging system, such as a surgical optical imaging system, an optical imaging system, a method, and a computer program.

[0004] Background

[0005] Surgical microscopes are often equipped with image-guided surgery (IGS) systems. IGS system is a technology that uses pre-operative imaging data, such as computer tomography, magnet resonance tomography, or ultrasound, to guide a surgeon during a surgical procedure. It allows the surgeon to see the patient's internal anatomy in real time and overlay it with preoperative imaging data. This allows for more accurate and precise surgical navigation, making it a useful tool for complex procedures such as brain surgery and spinal surgery.

[0006] The technical principle of IGS involves using a tracking system, such as infrared cameras or electromagnetic trackers, to determine the location and orientation of the surgical microscope in relation to the patient's internal anatomy. These tracking data are used to create a three- dimensional model of the patient's anatomy and overlay it with the pre-operative imaging data. This allows the surgeon to view the patient's internal anatomy in real-time, aligned with the perspective of the surgical microscope. As the microscope moves, the system dynamically updates the overlay, ensuring that the data remains aligned.

[0007] A Virtual Reality (VR) allows the user to experience a computer-generated simulation of a three-dimensional environment and can be used for IGS systems. VR immerses the user in the simulated world, allows the user to navigate within the virtual objects thus providing a more realistic perception and comprehension of the three-dimensional structure. However, if the virtual objects are displayed together with an image of the sample, there may be an overlap between different pieces of information, which may affect the availability of information. Thus, there may be a desire for an improved concept for providing an image composed of pre-operative imaging data and an image acquired of a sample by an optical imaging system.

[0008] Summary

[0009] This desire is addressed by the subject-matter of the independent claims.

[0010] The concept proposed in the present disclosure is based on the insight, that output data can be improved by composing an output image such that an opaque region of a plurality of regions overlays an object for guidance. The opaque region may be a region of a plurality of regions of an image of the sample. That is, the image of the sample may be segmented into the plurality of regions. Based on the segmented image of the sample an opaque region and / or a transparency region can be determined to determine the output data. For example, the opaque region may be overlaid, such that it overlaps the object for guidance. Optionally or alternatively, an opacity of the transparency region may be adjusted, such that it does not affect a perception of a displayed object for guidance.

[0011] Examples provide an apparatus for an optical imaging system comprising one or more processors and one or more storage devices. The apparatus is configured to obtain sample data from a sensor of a microscope of an optical imaging system. The sample data is indicative of an image of the sample. Further, the apparatus is configured to determine segmented data indicative of a segmentation of the image of the sample into a plurality of regions and to obtain object data indicative of an object for guidance of the user of the optical imaging system. The apparatus is further configured to determine, based on the object data and the segmented data, output data indicative of a composed output image of the image of the sample and the object for guidance, such that an opaque region of the plurality of regions overlays the object for guidance. Further, the apparatus is configured to transmit the output data for displaying on a display device. Segmenting the image of the sample into a plurality of regions may allow to determine different regions for adjusting an opacity. For example, the image of the sample may comprise regions of different interest to the user. An opaque region may be a region with interest to the user. That is, the opaque region may comprise a structure the user is interested in. For example, an opaque region can be determined to overlay the object for guidance. The opaque region may be opaque. That is, the opaque region may be non-transparent. In this way, information of the opaque region can overlap information of the object for guidance. Thus, a composed output image comprising the object for guidance and the opaque region overlapping the object for guidance can be displayed to the user. Segmenting the image of the sample may allow to provide information about the object for guidance and the image of the sample in an improved way. For example, a perception of the user can be improved by the composed output image.

[0012] In an example, the apparatus may be configured to determine the segmented data by determining a transparency region of the plurality of regions for reducing an opacity. The transparency region is determined based on the sample data. Further, the apparatus may be configured to determine the segmented data by generating a transparency image by reducing an opacity of the transparency region. Further, the apparatus is configured to determine the output data by overlaying the object for guidance with the transparency image. Reducing the opacity of the transparency region may allow to hide a structure that is of less or no interest to the user. Thus, the transparency image may comprise no region of less or no interest to the user, for example. In this way, the composed output image can easily be determined by overlaying the object for guidance with the transparency image. That is, the entire transparency image may overlap the object for guidance. Due to the reduced opacity of the transparency region, a visibility of the object for guidance can be increased. The transparency region may be a region of the plurality of regions for which an opacity can be adjusted to adjust a visibility of the object for guidance. For example, the transparency region may be region of the image of the sample with less or no interest to the user. The transparency region may be a region comprising no information to be displayed to the user, for example. Thus, an opacity of the transparency region can be reduced without affecting a user experience and / or perception of the image of the sample.

[0013] In an example, the apparatus may be configured to reduce the opacity of the transparency region such that the transparency region is transparent. That is, the opacity of the transparency region may be reduced to substantially 0. Thus, no structure of the transparency region may be displayed in the composed output image.

[0014] In an example, the apparatus may be configured to determine the level of opacity for the transparency region based on the sample data. For example, multiple transparency regions may comprise multiple different structures that are of different interest to the user. Thus, the level of opacity can be determined based on the different interest to the user. For example, a transparency region comprising a structure that is of no interest to the user may be adjusted, such that an opacity is lower than for a transparency region comprising a structure that is of low interest to the user. In this way, the opacity of the transparency region can be adjusted depending on an actual structure, i.e., interest to the user, of the transparency region.

[0015] In an example, the apparatus may be configured to determine an outer dimension of the transparency region based on the object data, such that the outer dimension of the transparency region is equal to or larger than an outer dimension of the object for guidance. Thus, it can be ensured that the object for guidance is only overlaid by a transparency region for which an opacity can be adjusted. Alternatively, the outer dimension of the transparency region may be smaller than the outer dimension of the object for guidance.

[0016] In an example, the apparatus may be configured to determine the output data by determining the opaque region and determine the output data by overlaying the object for guidance with the opaque region. That is, a region comprising a structure that is of interest to the user can be determined to overlap the object for guidance. For example, the apparatus may perform a determination of a structure that is of interest to the user to determine the opaque region. The opaque region may be a region comprising information to be displayed to the user. For example, the opaque region may be defined by a structure of interest to the user, e.g., arteries, veins, organs, tumors.

[0017] In an example, the apparatus may be configured to determine space allocation data indicative of a space allocation of at least one region of the plurality of regions. Further, the apparatus may be configured to obtain the object data by determining a dimension of the object based on the space allocation data. In this way, a footprint of the object for guidance can be adjusted to a region of the plurality of regions, e.g., to the opaque region and / or the transparency region.

[0018] In an example, the apparatus may be configured to determine the segmented data by colorbased segmentation, spectral unmixing and / or recognition of anatomical structure based on morphology. In this way, an appropriate process can be selected by the apparatus to segment the image. In an example, the object data may be indicative of a three-dimensional object for guidance of the user and the composed output image is a three-dimensional visualization. Thus, a three- dimensional image of the sample and a three-dimensional object for guidance can be displayed to the user.

[0019] In an example, the apparatus may be configured to obtain pose data indicative of a pose of the user and determine the output data based on the pose data. Obtaining pose data may allow to adjust the composed output image to an actual viewing angle of the user. For example, an orientation of the object for guidance may be adjusted to a viewing angle of the user. In this way, a user experience can be improved.

[0020] In an example, the apparatus may be configured to adjust a display angle of the output data based on the pose data. That is, an orientation of the composed output image can be adjusted. In this way, a perception of the user can be improved.

[0021] In an example, the image of the sample is a two-dimensional view of the sample and the apparatus may be configured to adjust the display angle by adjusting a rotation angle of the image. In this way, the displayed two-dimensional view of the sample, e.g., a white light view, a fluorescence view, a multispectral view, can be adjusted to a pose of the user.

[0022] In an example, the apparatus may be configured to determine the segmented data by determining the level of opacity for at least one region of the plurality of regions based on a pixel value of multiple pixels of the image of the sample. For example, based on a concentration of a spectral component a level of opacity may be determined for the at least one region. Optionally or alternatively, different levels of opacity may be assigned to a structure, i.e., an opaque region and / or transparency region. In this way, a display of the composed output image can be improved.

[0023] Examples provide an optical imaging system comprising an apparatus as described above.

[0024] Examples provide a method for an optical imaging system comprising obtaining sample data from a sensor of a microscope of an optical imaging system, the sample data indicative of an image of a sample. The method further comprises determining segmented data indicative of a segmentation of the image of the sample into a plurality of regions. The method further comprises obtaining object data indicative of an object for guidance of a user of the optical imaging system. The method further comprises determining, based on the object data and the segmented data, output data indicative of a composed output image of the image of the sample and the object for guidance, such that an opaque region of the plurality of regions overlays the object for guidance. The method further comprises transmitting the output data for displaying on a display device.

[0025] Various examples of the present disclosure relate to a corresponding computer program with a program code for performing the above method when the computer program is executed on a processor.

[0026] Short description of the Figures

[0027] Some examples of apparatuses and / or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which

[0028] Figs, la and lb show schematic diagrams of an example of an apparatus for an optical imaging system and of a corresponding optical imaging system comprising the apparatus;

[0029] Figs. 2a-2d show an example of generating a composed output image;

[0030] Figs. 3a-3f show another example of generating a composed output image;

[0031] Fig. 4 shows a flow chart of an example of a method for an optical imaging system that uses reflectance imaging; and

[0032] Fig. 5 shows a schematic diagram of a system comprising a microscope and a computer system.

[0033] Detailed Description

[0034] Various examples will now be described more fully with reference to the accompanying drawings in which some examples are illustrated. In the figures, the thicknesses of lines, layers and / or regions may be exaggerated for clarity. Figs, la and lb show schematic diagrams of an example of an apparatus 130 for an optical imaging system 100 and of a corresponding optical imaging system 100 comprising the apparatus 130. The apparatus 130 is tasked with controlling various aspects of a microscope 120 of the optical imaging system 100, which may be a surgical optical imaging system, and of the entire optical imaging system 100 and / or with processing various types of sensor data of the optical imaging system 100. Consequently, the apparatus 130 may be implemented as a computer system, which interfaces with the various components of the optical imaging system, e.g., the sensor 122.

[0035] The apparatus 130 comprises, as shown in Fig. la, one or more processors 134 and one or more storage devices 136. Optionally, the apparatus 130 further comprises one or more interfaces 132. The one or more processors 134 are coupled to the one or more storage devices 136 and to the optional one or more interfaces 132. In general, the functionality of the apparatus 130 may be provided by the one or more processors 134 (for determining the output data), in conjunction with the one or more interfaces 132 (for exchanging information, e.g., with the sensor 122 or a display device 180, e.g., to transmit the output data) and / or with the one or more storage devices 136 (for storing and / or retrieving information).

[0036] The apparatus 130 is configured to obtain sample data from a sensor of a microscope 120 of an optical imaging system 100. The sample data is indicative of an image of the sample 110. For example, the sample data may be obtained by receiving from the sensor, e.g., an optical imaging sensor 122 of the microscope 120 or from a frame buffer, e.g., part of the optical imaging system 100. Optionally or alternatively, the sample data may be obtained by measuring by the apparatus 130. For example, the optical imaging sensor 122 may be part of the apparatus 130. Thus, the apparatus 130 may control the optical imaging sensor 122 to measure the sample data. The sample data may be raw data from the sensor, i.e., the apparatus 130 may post process the sample data. Alternatively, the sample data may be post-processed data from the sensor, i.e., no further post-processing may be necessary by the apparatus 130.

[0037] The image of the sample 110 may be a white light image, a fluorescence image, a spectral or multispectral image and / or any combination or derivate of any of these images. For example, the image of the sample 110 may be acquired using white light imaging or fluorescence imaging. The image of the sample 110 may be a live view. Live view refers to a real-time display of the sample 110. For example, a live view may allow the user to see a continuous and up-to- date display of the sample 110 being acquired by the microscope.

[0038] A live view can be particularly useful in process control microscopy applications and / or during a surgery, as it allows the user to see an image of the sample 110 in real-time and make adjustments to the microscope or sample 110 as needed.

[0039] For example, a live view may be used to inspect and monitor the quality of a work piece during the manufacturing process, or to adjust the focus and positioning of the microscope to ensure that the live view is properly acquired or to perform a surgery.

[0040] The image of the sample 110 may comprise multiple regions, i.e., a plurality of regions. For example, the image of the sample 110 may comprise different structures such like arteries, veins, organs, bones, tumors. A user of the optical imaging system 100 may have different interests in the different structures. That is, not every region of the plurality of regions needs to be displayed to the user, for example. It is a finding of the inventors that an output image may be determined by overlapping an object for guidance with an opaque region of the plurality of regions.

[0041] Thus, the apparatus 130 is configured to determine segmented data indicative of a segmentation of the sample 110 into a plurality of regions. For example, the image of the sample 110 can be segmented by the apparatus 130 by a camera-based segmentation and / or spectral unmixing as described below. In this way, different regions of the image of the sample 110 can be determined by the apparatus 130. The image of the sample 110 may comprise different structures of different interest to the user. That is, the image of the sample 110 may be segmented based on the different structures of the image of the sample 110, for example. For example, the image of the sample 110 may comprise a first region defined by arteries, a second region defined by bones and a third region defined by a tumor. Optionally or alternatively, the image of the sample 110 may be segmented based on a concentration of a spectral component. That is, the segmentation into the plurality of regions may be based on the concentration of a spectral component in the image of the sample 110. In this way, the plurality of regions can be determined based on a concentration of blood and / or blood particles for example.

[0042] For example, the apparatus 130 may determine an opaque region in the image of the sample 110. The opaque region may be a region of interest to the user. That is, the opaque region may comprise a structure that the user is interested in. Therefore, the opaque region may overlay the object for guidance in the composed output image. In this way, the user can see the object for guidance and a relevant structure of the sample 110.

[0043] Optionally or alternatively the apparatus 130 may determine a transparency region in the image of the sample 110. The transparency region may be a region of less or no interest to the user. For example, the transparency region may comprise no structure that the user is interested in. Therefore, an opacity of the transparency region can be reduced, such that the visibility of an underlying objective for guidance can be increased. For example, when the transparency region comprises no structure that the user is interested in, the opacity of the transparency region can be reduced by the apparatus 130 to 0. That is, the transparency region may not affect a perception of the underlying object for guidance.

[0044] Thus, a composed output image may be obtained by the apparatus 130 by determining an opaque region and / or a transparency region. Depending on the determined opaque region and / or transparency region an opacity of the corresponding region can be adjusted by the apparatus 130. For example, the opacity can be decreased for a transparent region and / or increased for an opaque region.

[0045] To determine a composed output image the apparatus 130 need to obtain information about an object for guidance. Therefore, the apparatus 130 is configured to obtain object data indicative of an object for guidance of the user of the optical imaging system 100. The object for guidance may be pre-operative image of the sample 110 or a part of the sample 110. For example, the pre-operative image may show an organic tissue to be removed during a surgery. For example, the object for guidance may be a lateral and / or a medial image of the sample 110 or a part of the sample 110. Thus, the obj ect for guidance may provide a user of the optical imaging system 100 a guidance for a surgery. Optionally or alternatively, the object for guidance may be a navigation cue, a visualization of a surgical tool and / or a pathway of a surgical instrument. The object data may be obtained by retrieving from a storage device such as storage device 136 by the apparatus 130. Optionally or alternatively, the object data may be determined by the apparatus 130. For example, the apparatus 130 may comprise a sensor to measure an environment of the sample 110, e.g., a pose of the user of the optical imaging system 100. Thus, the apparatus 130 may determine the object data based on a pose of the user. For example, an orientation of an object for guidance may be determined by the apparatus 130 based on the pose of the user.

[0046] Using the object data and the sample data may allow to provide a user of the optical imaging system 100 both information about the object for guidance and the image of the sample 110. Thus, the apparatus 130 is configured to determine output data indicative of a composed output image of the image of the sample 110 and the object for guidance. The output data is determined based on the object data and the segmented data. The composed output image comprises an overlay of an opaque region of the plurality of regions and the object for guidance. That is, the opaque region may overlay the object for guidance. In this way, a structure of interest to the user of the image of the sample 110 can be displayed to the user in conjunction with the object for guidance, e.g., on the display device 180.

[0047] The opaque region and the object for guidance may be part of separate image layers. For example, the opaque region may be part of a sample layer and the object for guidance may be part of a guidance layer. That is, the sample layer can be arranged above or below the guidance layer. In this case, an adjustment of an opacity of the layers e.g., the sample layer, can be performed by the apparatus 130 to control a visual effect, but the pixel values of the separated layers won't be mixed. A perception of the user of the composed output image may depend on an order of the sample layer and the guidance layer.

[0048] Alternatively, the opaque region and the object for guidance may have two overlapping images and the pictures of these images may be on top of each other, also referred to as superimposed over late image. In this case, the pixel values of the two images are located at the same coordinates in space, and the pixels of the top layer can partially or fully cover those of the bottom layer. The final pixel color at that position may be a combination of the color values of the two overlapping pixels, depending on how the blending or mixing is set. Further, the apparatus 130 is configured to transmit the output data for displaying on a display device 180. For example, the output data is transmitted to the display device 180. Optionally or alternatively the output data is transmitted to a frame buffer, e.g., part of the optical imaging system 100.

[0049] In this way, the apparatus 130 can provide an overlay visualization of an image of the sample 110 and an object for guidance. Normally, an overlay visualization of pre-operative data, e.g., two-dimension or three-dimensional data, on a real time color image of the sample 110 acquired with a microscope is of very poor quality in terms of perception, especially three- dimensional perception, from the user. The image of the microscope shows the surface of the sample 110, e.g., a tissue, while the preoperative data are always located in deep tissue layers. For example, the visualization of a vessel inside a tissue is a counterintuitive picture (see also Fig. 2).

[0050] For example, pre-operative data used for ISG such as computer tomography, magnet resonance tomography data, are not inherently and readily in a form that can be perceived by a human. The main limitation may be that a whole volume of the human body is typically full of tissue and a superficial layer may hide a structure within the volume. For example, a human body part may include many anatomical components, and it may be not possible to visualize everything in a single visualization, e.g., in the image of the sample 110, in conjunction with an object for guidance. It is a finding of the inventors that an object for guidance can be combined with an image of a sample 110 acquired by a microscope by segmenting the image of the sample 110 to improve a visualization of a composed output image. For example, multiple anatomical components such as skin, muscle and bones are omitted to increase visibility for other components to be visible, e.g., arteries, veins, lungs. In this way, the composed output image can be restricted to structures of interest to a user. For example, the image of the sample 110 can be segmented to determine an opaque region to be overlaid on the object for guidance. An information overhead can be reduced and / or a perception of the object for guidance can be improved by segmenting the image of the sample 110.

[0051] Thus, determining the output data by the apparatus 130 may allow that the user can perceive the object for guidance in a natural and / or more precise way. The apparatus 130 may provide a faster comprehension of the relative position, arrangement and / or size of the object for guidance can be achieved. The apparatus 130 can avoid information overflow. That may help the user to focus on the important elements by omitting displaying of unnecessary anatomical details of the image of the sample 100. The apparatus 130 can streamline the observation of pre-operative information. For example, in cases with complex crucial anatomical structures which are tangled with the pathological tissue, today’s microscope IGS system is deemed insufficient, and the surgeon needs to use additional IGS hardware to interrogate the object for guidance. This could be avoided with the apparatus 130. The apparatus 130 can be utilized to achieve a combined visualization of an image of the sample 110, e.g., a real-time (i.e., live view) white image and an object for guidance, e.g., a three-dimensional preoperative image.

[0052] The proposed concept may be built around two main components - the microscope 120, which comprises the optical components, and the apparatus 130, which may be used to control the optical imaging system 100, process sensor data of the microscope 120, e.g., the optical imaging sensor 122, and / or to determine the output data.

[0053] In general, a microscope, such as the microscope 120, is an optical instrument that is suitable for examining objects that are too small to be examined by the human eye (alone). For example, a microscope 120 may provide an optical magnification of a sample, such as a sample 110 shown in Fig. la. In modern microscopes, the optical magnification is often provided for a camera or an imaging sensor, such as the optical imaging sensors 122 of the microscope 120. The microscope 120 may further comprise one or more optical magnification components that are used to magnify a view of the sample 110, such as an objective.

[0054] There are a variety of different types of optical imaging systems. If the optical imaging system 100 is used in the medical or biological fields, the sample 110 may be a sample of organic tissue, e.g., arranged within a petri dish or present in a part of a body of a patient. In some examples of the present disclosure, e.g., as shown in Fig. lb, the optical imaging system 100 may be a surgical optical imaging system, e.g., an optical imaging system that is to be used during a surgical procedure, such as an oncological surgical procedure or during tumor surgery. However, the proposed concept may also be applied to other types of microscopy, e.g., microscopy in a laboratory or microscopy for the purpose of material inspection.

[0055] As is evident, the optical imaging system 100 comprises a number of components, such as the apparatus 130, the microscope 120 with the at least one optical imaging sensors 122, an optional main pair of ocular displays 140, an optional secondary pair of ocular displays 145 and a display device such as a head-mounted display 180 or a display. Fig. lb shows a schematic diagram of an example of a surgical optical imaging system 100 comprising the microscope 120 and the apparatus 130. In general, a (surgical) optical imaging system is a system that comprises a microscope 120 and additional components, which are operated together with the microscope 120. In other words, a (surgical) optical imaging system is a system that comprises the microscope 120 and one or more additional components, such as the apparatus 130 (which may be a computer system being adapted to control the microscope 120 and, for example, determine the output data), an illumination system (which is used to illuminate a sample being imaged by the microscope 120), additional sensors, displays etc.

[0056] The surgical optical imaging system 100 shown in Fig. lb comprises a number of optional components, such as a base unit 105 (which may comprise the apparatus 130) with a stand, ocular displays 140; 145 that are arranged at the microscope 120, a head-mounted display 180, and a (robotic or manual) arm 160 which holds the microscope 120 in place, and which is coupled to the base unit 105 and to the microscope 120. In general, these optional and non- optional components may be coupled to the apparatus 130, which may be configured to control and / or interact with the respective components.

[0057] In an example, the apparatus 130 may be configured to determine the segmented data by determining a transparency region of the plurality of regions for reducing an opacity. The transparency region is determined based on the sample data. That is, the image of the sample 110 may comprise different regions of different interest to the user. The apparatus 130 may determine a region of the image of the sample 110 with less or no interest to the user as transparency region. Since the transparency region may be of less or no interest to the user, an opacity of the transparency region can be reduced. In this way, a visibility of the object for guidance hided by the image of the sample 110, i.e., by the transparency region, can be improved and / or a perception of the object for guidance can be increased.

[0058] Thus, the apparatus 130 may be configured to determine the segmented data by generating a transparency image by reducing an opacity of the transparency region. Reducing the opacity of the transparency region may allow to straightforward overlay the object for guidance with the transparency image. For example, less interesting or uninteresting regions of the image of the sample 110 can be set transparent by the apparatus 130. In this way, an overlay of the object for guidance and the transparency image (derived from the image of the sample 110) can be performed by the apparatus 130 with less computational effort. Therefore, the apparatus 130 is configured to determine the output data by overlaying the object for guidance with the transparency image.

[0059] For example, the transparency image may be identical to the image of the sample 110 excluding the opacity of the transparency region. Therefore, overlaying the object for guidance with the transparency region instead of the image of the sample 110 may reduce an occlusion of the object for guidance.

[0060] An example is overlaying the object for guidance with data derived from the sample data, e.g., an opaque region and / or a transparency image. In other examples, the object for guidance may overlay the data derived from the sample data. That is, the object for guidance may be in an image layer above an image layer comprising the data device from the sample data. For example, the object for guidance may overlay a transparency image and / or an opaque region. In this case, a perception of the composed output image can also be increased. For example, an object for guidance may be semitransparent and by reducing an opacity of a transparency region overlapped by the object for guidance, a perception of the object for guidance overlapping the transparency region can be improved. This may be especially of interest for two different image layers, e.g., sample layer and guidance layer, as described above.

[0061] In an example, the apparatus 130 may be configured to reduce the opacity of the transparency region such that the transparency region is transparent. That is, the opacity of the transparency region may be reduced to substantially 0. Thus, no structure of the transparency region may be displayed on a display device that displays the composed output image.

[0062] In an example, the apparatus 130 may be configured to determine the level of opacity for the transparency region based on the sample data. For example, the apparatus 130 may determine a structure, e.g., an artery, a vein, an organ, a tumor, in the image of the sample 110. The determined structure may be assigned as the transparency region. Based on the determined structure the apparatus 130 may determine the level of opacity. For example, a vein may have a different opacity than a tumor. Optionally, a user can define the level of opacity, for example by defining an opacity for a structure. For example, the definition of the user can be received by a user input from an input device, such like a touch display, a keyboard. Optionally or alternatively, the level of opacity can be determined based on a concentration of a spectral component of the image of the sample 110. That is, the apparatus 130 may adjust the level of opacity of the transparency region based on a concentration of a spectral component. Optionally, the opaque region can be adjusted similarly to the transparency region. That is, the apparatus 130 may be configured to determine the level of opacity for the opaque region based on sample data and / or a user input as described with reference to the transparency region.

[0063] In an example, the apparatus 130 may be configured to determine an outer dimension of the transparency region based on the object data, such that the outer dimension of the transparency region is equal to or larger than an outer dimension of the object for guidance. That is, the dimension of the transparency region may depend on the dimension of the object for guidance. For example, the dimension of the transparency region can be determined such that a footprint of the transparency region may be at least as large as a footprint of the object for guidance. In this way, a perception of the object for guidance can be improved since the transparency region of the image of the sample 110 overlapping the object for guidance can have at least the same footprint (or base area).

[0064] In an example, the apparatus 130 may be configured to determine the output data by determining the opaque region and determine the output data by overlaying the object for guidance with the opaque region. That is, a region comprising a structure that is of interest to the user can be determined to overlap the object for guidance. Determining the opaque region can be performed by the apparatus 130 in conjunction with determining the transparency region or alternatively. That is, the apparatus 130 can determine a transparency region, an opaque region and / or a combination thereof.

[0065] The opaque region may comprise a structure of interest to the user. That is, the opaque region may comprise data to be displayed to the user during use of the optical imaging system 100. For example, the opaque region may comprise a tumor to be removed during a surgery and / or an artery that is not to be touched during surgery.

[0066] Therefore, determining the segment data may comprise determining a transparency region and / or determining an opaque region. In this way, structures in the image of the sample 110 can be determined based on different parameters. For example, structures that should be displayed to the user can be determined to define the opaque region and / or structures that should not be displayed to the user can be determined to define the transparency region. Optionally, an opacity of the opaque region can be set by the apparatus 130. For example, an opacity of the opaque region can be increased to increase a perception of the opaque region. Optionally, different opaque regions may have different opacities.

[0067] In an example, the apparatus 130 may be configured to determine space allocation data indicative of a space allocation of at least one region of the plurality of regions. Further, the apparatus 130 may be configured to obtain the object data by determining a dimension of the object based on the space allocation data. The space allocation of the at least one region of the plurality of regions may be a footprint (or base area) or dimension of the at least one region of the plurality of regions. That is, a footprint or a dimension of the object for guidance can be adjusted to the footprint or dimension of the at least one region. For example, the at least one region may be a transparency region, e.g., with an opacity of 0. In this case, a footprint or a dimension of the object for guidance can be set such that it substantially match the footprint of the transparency region. In this way, a perception of the object for guidance can be improved.

[0068] In an example, the apparatus 130 may be configured to determine the segmented data by color-based segmentation, spectral unmixing and / or recognition of anatomical structure based on morphology, e.g., vessels. In this way, an appropriate process can be selected by the apparatus 130 to segment the image. For example, color-based segmentation may comprise color space conversion, thresholding, filtering and post processing, connected component analysis and / or region-based segmentation. For example, spectral unmixing may comprise hyperspec- tral imaging, and member identification, linear unmixing and / or abundance maps. For example, recognition of anatomical structure may comprise feature extraction, morphological operations, object detection and / or classification. The choice of method may depend on the nature of images. Color-based segmentation can be suitable for images where objects of interest have distinct and separable colors. Spectral unmixing can be used in hyperspectral imagery to identify materials with unique spectral signatures. Recognition of anatomical structures based on morphology can be common in medical imaging, where structures like cells, organs, or vessels are characterized by their shapes and textures.

[0069] In an example, the apparatus 130 may be configured to determine the segmented data by determining a level of opacity for at least one region of the plurality of regions. The level of opacity may be determined based on a pixel value of multiple pixels of the image of the sample 110. For example, the concentration of a spectral component of the image of the sample 110 can be used by the apparatus 130 to determine the level of opacity. Thus, the apparatus 130 may be configured to adjust the opacity proportional to a concentration in a structure. For example, the opacity may be proportional to a blood concentration in a vein.

[0070] Optionally or alternatively, the level of opacity may vary in the at least one region, e.g., the opaque region or the transparency region. For example, a central part of a vein may have a higher opacity than an edge of the vein. Thus, the apparatus 130 may determine multiple level of opacities for a structure, i.e., the opaque region and / or the transparency region. In this way, a perception of the structure can be improved.

[0071] In an example, the object data may be indicative of a three-dimensional object for guidance of the user and the composed output image is a three-dimensional visualization. Thus, a three- dimensional image of the sample 110 and a three-dimensional object for guidance can be displayed to the user.

[0072] In an example, the apparatus 130 may be configured to obtain pose data indicative of a pose of the user and determine the output data based on the pose data. The pose data may be obtained by receiving from an environmental sensor, such as camera and / or from the microscope 120, for example. Optionally or alternatively, the pose data may be measured by the apparatus 130. For example, the apparatus 130 may comprise the microscope 120 and may control the optical imaging sensor 122 to measure the pose data. For example, the microscope 120 may be an exoscope.

[0073] Two factors may determine an angle of view of a person: a head or facial orientation (i.e., face pose or face direction) and eye orientation (i.e., eye gaze direction or angle of view). The head orientation can be used to determine a global direction of the gaze. The eye orientation can be used to determine a local direction of the gaze, i.e., the angle of view. When a head of the person is level and he or she looks straight ahead, then the line of sight of the eye gaze and the facial orientation are straight ahead with 0° azimuth and 0° elevation and 0° tilt. This head position can also be described as a neutral head orientation position. The position of the eyes of the user, e.g., an angle of view of the user, may be used by the apparatus 130 to determine the output data. For example, the angle of view of the user may indicate the viewing angle of the user or may be identical, which could be used to determine the output data. The position of the head of the user, e.g., an orientation of the head, relative to the microscope 100 can be used by the apparatus 130 to determine the output data. For example, the orientation of the head may be indicative of the viewing angle of the user, which could be used to determine the output data. That is, a visualization of the composed optical image can be adjusted to an actual angle of view of the user.

[0074] The apparatus 130 may adjust the image of the sample 110 and / or the object for guidance based on the pose data. For example, the apparatus 130 can automatically adjust the image of the sample 110 and / or the object for guidance based on the pose data to improve a perception of the composed output image. For example, the image of the sample 110 may be a two- dimensional image, but the image of the sample 110 may be captured using a stereoscopic microscope. Thus, a disparity map can be used to rotate the two-dimensional image to account for a pose of the user. Optionally or alternatively, the object for guidance can be rotated to account for a pose of the user. For example, the apparatus 130 may retrieve the object for guidance from a storage device, such as a storage device 136, may receive the pose data and may generate an adjusted object for guidance by rotating the retrieved object for guidance based on the pose data.

[0075] In an example, the apparatus 130 may be configured to adjust a display angle of the output data based on the pose data. That is, an orientation of the composed output image can be adjusted. In this way, a perception of the user can be improved.

[0076] In an example, the image of the sample 110 is a two-dimensional view of the sample 110 and the apparatus 130 may be configured to adjust the display angle by adjusting a rotation angle of the image. In this way, the displayed two-dimensional view of the sample 110, e.g., a white light view, a fluorescence view, a multispectral view, can be adjusted to a pose of the user.

[0077] As shown in Fig. la the optional one or more interfaces 132 is coupled to the respective one or more processors 134 at the apparatus 130. In examples the one or more processors 134 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software. Similar, the described functions of the one or more processors 134 may as well be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a microcontroller, etc. The one or more processors 134 is capable of controlling the one or more interfaces 132, so that any data transfer that occurs over the one or more interfaces 132 and / or any interaction in which the one or more interfaces 132 may be involved may be controlled by the one or more processors 134.

[0078] In an embodiment the apparatus 130 may comprise a memory, e.g., the one or more storage devices 136 and at least one or more processors 134 operably coupled to the memory and configured to perform the method described below.

[0079] In examples the one or more interfaces 132 may correspond to any means for obtaining, receiving, transmitting or providing analog or digital signals or information, e.g., any connector, contact, pin, register, input port, output port, conductor, lane, etc. which allows providing or obtaining a signal or information. The one or more interfaces 132 may be wireless or wireline and it may be configured to communicate, e.g., transmit or receive signals, information with further internal or external components.

[0080] The apparatus 130 may be a computer, processor, control unit, (field) programmable logic array ((F)PLA), (field) programmable gate array ((F)PGA), graphics processor unit (GPU), application-specific integrated circuit (ASICs), integrated circuits (IC) or system-on-a-chip (SoCs) system.

[0081] More details and aspects are mentioned in connection with the examples described below. The example shown in Fig. 1 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described below (e.g., Fig. 2 - 5).

[0082] Figs. 2a-2d show an example of generating a composed output image. Figs. 2a and 2b show an example of generating a transparency image. Figs. 2c and 2d show the example of Figs. 2a and 2b in conjunction with an object for guidance. In principle, Figs. 2a and 2c can be identified as prior art. Fig. 2a shows an image of a sample. The image of the sample may be acquired using an optical imaging system. The image of the sample may comprise different (anatomical) structures, such as tissue vasculature 210 and certain tissue parts 220, 230, 240. Using the apparatus as described above (e.g., with reference to Fig. 1) the image of the sample can be segmented into a plurality of regions. For example, a first region may be defined by the tissue parts 220, a second region may be defined by the tissue part 230, a third region may be defined by the tissue part 240 and a fourth region may be defined by the tissue vasculature 210. The structure of interest to the user may be the fourth region, i.e., the tissue vasculature may be of interest to the user.

[0083] Thus, the apparatus may be configured to keep the structure of interest, i.e., the fourth region. For example, an aim may be to keep important anatomical structures which allow the user, e.g., a surgeon, to maintain a clinical awareness orientation. At the same time, the rest of the image of the sample, i.e., the first region, the second region and the third region, which are of less or no interest to the user, can be set to transparent (or an opacity can be reduced). In this way, the user can look through the structures of less or no interest (see Fig. 2b).

[0084] For example, the apparatus may convert the image of the sample shown in Fig. 2a to a transparency image shown in Fig. 2b by use of a transparency mask. Fig. 2a shows today’s visualization for an IGS system, and Fig. 2b shows a part of the composed output image, e.g., a sample layer. The image of the sample may be a live image from a microscope of the optical imaging system. Therefore, also the output data may comprise a live image. The image of the sample may be processed by the apparatus to extract the desired channel of anatomical elements (i.e., the fourth region), that will be visualized, and the rest of the image of the sample is set to transparent or an opacity is reduced. In the example shown in Fig. 2b, the preserved channel or structure is the total hemoglobin channel which consists primarily of the tissue vasculature 210, while the other tissue parts 220, 230, 240 are set transparent as shown in Fig. 2b.

[0085] Figs. 2a and 2b show only the image without the IGS data, i.e., the object for guidance 250. Figs. 2c and 2d include the IGS data, e.g., an IGS visualization (or simulation). Fig. 2c shows a prior art IGS visualization with a semitransparent object for guidance 250 overlaying the image of the sample. Fig. 2d shows a composed output image generated by an apparatus as described above. As can be seen a virtual impression of certain tissue parts being transparent can be generated, such that the user can see the underlying object for guidance 250 in an improved way. That is, a perception of the user of the object for guidance 250 can be improved. The IGS visualizations, i.e., the object for guidance 250, may be located below the tissue surface shown in the image of the sample captured by the microscope. Alternatively, the IGS visualization may be located above tissue surface.

[0086] In contrast to current IGS visualizations, which are typically a semitransparent overlay (see Fig. 2c), the apparatus may allow to visualize the object for guidance 250 to be seen through transparent holes of an overlayed transparency image (see Fig. 2d). As described above, e.g., with reference to Fig. 1, the transparency image can be generated by reducing an opacity of a transparency region. Alternatively, the transparency image can be generated by selecting an opaque region of the image of the sample. That is, the opaque region is overlaid on the object for guidance 250.

[0087] The example Fig. 2 discloses two-dimensional implementation. Optionally, the apparatus can also be used to generate a three-dimensional composed output image. This described in more detail in Fig. 3.

[0088] More details and aspects are mentioned in connection with the examples described above and / or below. The example shown in Fig. 2 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g., Fig. 1) and / or below (e.g., Fig. 3 - 5).

[0089] Figs. 3a-3f show another example of generating a composed output image. Figs. 3a, 3c and 3e show prior art implementation and Figs. 3b, 3d and 3f show examples of composed output images. Figs. 3a and 3b are identical to Figs. 2a and 2b, respectively. That is, Figs. 3a and 3b show the perspective of the microscope without any rotation (in three dimensions).

[0090] Figs. 3c-3f show examples with an applied three-dimensional rotation. Figs. 3d and 3f show examples of the proposed visualization i.e., different examples of composed output images, while Figs. 3c and 3e show the equivalent visualization with typical state of the art IGS systems. For example, the apparatus as described above may apply on the image of the sample the same three-dimensional rotation as on the IGS data allowing an easier and more precise perception of the three-dimensional constellation of visible anatomical structures and objects for guidance. That is, Figs. 3c, 3d and 3e, 3f may be a rotated visualization of Figs. 3a, 3b.

[0091] Therefore, it is an aspect of the invention that a three-dimensional rotation of the composed output image, i.e., combined data of sample data and object data, can be performed by the apparatus. The proposed visualization approach applies on the image of the sample, e.g., a white light image, the same three-dimensional rotation as on the object for guidance 250, which may allow an easier and more precise perception of the three-dimensional constellation of visible anatomical structures and objects for guidance.

[0092] To generate the examples shown in Fig. 2 and Fig. 3 the apparatus may consider an image of the sample as a two-dimensional, flat image. However, optical imaging systems are often stereoscopic, thus allowing the calculation of the sample three-dimensional surface. This would allow a more precise and realistic result. Moreover, it would be possible to utilize the three-dimensional scanning technology such as structured light to provide an even more precise and robust three-dimensional scan of the surgical cavity. That is, the apparatus can also be used to generate a three-dimensional composed output image comprising a three-dimensional image of the sample.

[0093] For example, an important element for the three-dimensional perception by humans is the observation of multiple different angles. The parallax effect of a continuously rotating scene is exploited by the human brain and lead to a very natural and precise comprehension of the scene. Thus, a real time rotation of the visualized data may be of great value. Some ways this could be done are use of a typical computer interface with mouse pointer control (e.g., to receive a user input), use gesture control as replacement for a pointing device(e.g., to receive a user input), detection of head movement to simulate different observation angle (e.g., obtain post data), e.g., looking a monitor from different angle would change the visualized perspective, use of a digital viewer with appropriate sensors to measure the head position (e.g., obtain post data) and / or use of the microscope handles to control the perspective. In this way, an alignment of the angle of view of the user and the composed output image displayed on the display device can be provided. More details and aspects are mentioned in connection with the examples described above and / or below. The example shown in Fig. 3 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g., Fig. 1 - 2) and / or below (e.g., Fig. 4 - 5).

[0094] Fig. 4 shows an example of a method 400 for an optical imaging system. The method 400 comprises obtaining 410 sample data from a sensor of a microscope of an optical imaging system, the sample data indicative of an image of a sample. The method 400 further comprises determining 420 segmented data indicative of a segmentation of the image of the sample into a plurality of regions. The method 400 further comprises obtaining 430 object data indicative of an object for guidance of a user of the optical imaging system. The method 400 further comprises determining 440, based on the object data and the segmented data, output data indicative of a composed output image of the image of the sample and the object for guidance, such that an opaque region of the plurality of regions overlays the object for guidance. The method 400 further comprises transmitting 450 the output data for displaying on a display device. The method may be performed by an apparatus as described above, e.g., with reference to Fig. 1.

[0095] More details and aspects are mentioned in connection with the examples described above and / or below. The example shown in Fig. 4 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g., Fig. 1 - 3) and / or below (e.g., Fig. 5).

[0096] Some embodiments relate to a microscope comprising an apparatus as described in connection with Fig. 1. Alternatively, a microscope may comprise of or can be communicatively connected to an apparatus as described in connection with Fig. 1. Fig. 5 shows a schematic illustration of a system 500, e.g., an optical imaging system, configured to perform a method described herein, e.g., with reference to Fig. 4. The system 500 comprises a microscope 510 and a computer system 520. The microscope may comprise the apparatus as described above, e.g., with reference to Fig. 1. The microscope 510 is configured to take images and is connected to the computer system 520. The computer system 520 is configured to execute at least a part of a method described herein. The computer system 520 may be configured to execute a machine learning algorithm. The computer system 520 and microscope 510 may be separate entities but can also be integrated together in one common housing. The computer system 520 may be part of a central processing system of the microscope 510 and / or the computer system 520 may be part of a subcomponent of the microscope 510, such as a sensor, an actor, a camera or an illumination unit, etc. of the microscope 510.

[0097] The computer system 520 may be a local computer device (e.g., personal computer, laptop, tablet computer or mobile phone) with one or more processors and one or more storage devices or may be a distributed computer system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 520 may comprise any circuit or combination of circuits. In one embodiment, the computer system 520 may include one or more processors which can be of any type. As used herein, processor may mean any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), multiple core processor, a field programmable gate array (FPGA), for example, of a microscope or a microscope component (e.g., camera) or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 520 may be a custom circuit, an application-specific integrated circuit (ASIC), or the like, such as, for example, one or more circuits (such as a communication circuit) for use in wireless devices like mobile telephones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 520 may include one or more storage devices, which may include one or more memory elements suitable to the particular application, such as a main memory in the form of random access memory (RAM), one or more hard drives, and / or one or more drives that handle removable media such as compact disks (CD), flash memory cards, digital video disk (DVD), and the like. The computer system 520 may also include a display device, one or more speakers, and a keyboard and / or controller, which can include a mouse, trackball, touch screen, voice-recognition device, or any other device that permits a system user to input information into and receive information from the computer system 520.

[0098] More details and aspects are mentioned in connection with the examples described above. The example shown in Fig. 5 may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g., Fig. 1 - 4).

[0099] Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a processor, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

[0100] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a non- transitory storage medium such as a digital storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

[0101] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0102] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

[0103] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0104] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0105] A further embodiment of the present invention is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitionary. A further embodiment of the present invention is an apparatus as described herein comprising a processor and the storage medium.

[0106] A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.

[0107] A further embodiment comprises a processing means, for example, a computer or a programmable logic device, configured to, or adapted to, perform one of the methods described herein.

[0108] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0109] A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0110] In some embodiments, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

[0111] If some aspects have been described in relation to a device or system, these aspects should also be understood as a description of the corresponding method and vice versa. For example, a block, device or functional aspect of the device or system may correspond to a feature, such as a method step, of the corresponding method. Accordingly, aspects described in relation to a method shall also be understood as a description of a corresponding block, a corresponding element, a property or a functional feature of a corresponding device or a corresponding system.

[0112] The following claims are hereby incorporated in the detailed description, wherein each claim may stand on its own as a separate example. It should also be noted that although in the claims a dependent claim refers to a particular combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent or independent claim. Such combinations are hereby explicitly proposed, unless it is stated in the individual case that a particular combination is not intended. Further- more, features of a claim should also be included for any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.

[0113] The aspects and features described in relation to a particular one of the previous examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the features into the further example.

[0114] List of reference Signs

[0115] 100 optical imaging system

[0116] 105 base

[0117] 110 sample

[0118] 120 microscope

[0119] 122 optical imaging sensor

[0120] 130 apparatus

[0121] 132 interface

[0122] 134 processor

[0123] 136 storage device

[0124] 140, 145 ocular display

[0125] 160 arm

[0126] 180 display device

[0127] 182 head-mounted display

[0128] 210 region of interest

[0129] 220, 230, 240 region of less or no interest

[0130] 250 object for guidance

[0131] 400 method

[0132] 410 obtaining sample data

[0133] 420 determining segmented data

[0134] 430 obtaining object data

[0135] 440 determining output data

[0136] 450 transmitting the output data

[0137] 500 system

[0138] 510 microscope

[0139] 520 computer system

Claims

Claims1. An apparatus (130) for an optical imaging system (100), comprising one or more processors (134) and one or more storage devices (136), wherein the apparatus (130) is configured to: obtain sample data from a sensor (122) of a microscope of an optical imaging system (100), the sample data indicative of an image of a sample (110); determine segmented data indicative of a segmentation of the image of the sample into a plurality of regions; obtain object data indicative of an object for guidance of a user of the optical imaging system (100); determine, based on the object data and the segmented data, output data indicative of a composed output image of the image of the sample and the object for guidance, such that an opaque region of the plurality of regions overlays the object for guidance; and transmit the output data for displaying on a display device (180).

2. The apparatus (130) according to claim 1, wherein the apparatus (130) is configured to determine the segmented data by: determining, based on the sample data, a transparency region of the plurality of regions for reducing an opacity; generating a transparency image by reducing an opacity of the transparency region; and determine the output data by overlaying the object for guidance with the transparency image.

3. The apparatus (130) according to claim 2, wherein the apparatus (130) is configured to reduce the opacity such that the transparency region is transparent.

4. The apparatus (130) according to claim 2, wherein the apparatus (130) is configured to determine a level of opacity for the transparency region based on the sample data.

5. The apparatus (130) according to any one of claims 2-4, wherein the apparatus (130) is configured to: determine an outer dimension of the transparency region based on the object data, such that the outer dimension of the transparency region is equal to or larger than an outer dimension of the object for guidance.

6. The apparatus (130) according to claim 1, wherein the apparatus (130) is configured to determine the output data by: determining the opaque region of the plurality of regions; and determine the output data by overlaying the object for guidance with the opaque region.

7. The apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is configured to: determine space allocation data indicative of a space allocation of at least one region of the plurality of regions; and obtain the object data by determining a dimension of the object based on the space allocation data.

8. The apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is configured to determine the segmented data by at least one of color-based segmentation, spectral unmixing and recognition of anatomical structure based on morphology.

9. The apparatus (130) according to any one of the preceding claims, wherein the object data is indicative of a three-dimensional object for guidance of the user and the composed output image is a three-dimensional visualization.

10. The apparatus (130) according to claim 9, wherein the apparatus (130) is configured to: obtain pose data indicative of a pose of the user; and determine the output data based on the pose data.

11. The apparatus (130) according to claim 10, wherein the apparatus (130) is configured to adjust a display angle of the output data based on the pose data.

12. The apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is configured to determine the segmented data by determining a level of opacity for at least one region of the plurality of regions based on a pixel value of multiple pixels of the image of the sample (110).

13. An optical imaging system (100), comprising an apparatus (130) according to any one of the preceding claims.

14. A method (400) for an optical imaging system, comprising: obtaining (410) sample data from a sensor of a microscope of an optical imaging system, the sample data indicative of an image of a sample; determining (420) segmented data indicative of a segmentation of the image of the sample into a plurality of regions; obtaining (430) object data indicative of an object for guidance of a user of the optical imaging system; determining (440), based on the object data and the segmented data, output data indicative of a composed output image of the image of the sample and the object for guidance, such that an opaque region of the plurality of regions overlays the object for guidance; and transmitting (450) the output data for displaying on a display device.

15. A computer program with a program code for performing the method according to claim 14 when the computer program is executed on a processor.

Images