Apparatus for an optical imaging system, optical imaging system, method and computer program

Abstract

Examples relate to an apparatus for an optical imaging system, the apparatus comprising one or more processors and one or more storage devices. The apparatus is configured to obtain sample data from a sensor of a microscope of the optical imaging system. The sample data is indicative of an image of the sample. Further, the apparatus is configured to determine segmentation data indicative of a segmentation of the image of the sample into a plurality of regions, and to obtain object data indicative of an object for guiding a user of the optical imaging system. The apparatus is further configured to determine, based on the object data and the segmentation data, output data indicative of a composite output image of the image of the sample and the object for guiding, such that opaque regions of the plurality of regions are superimposed on the object for guiding. Further, the apparatus is configured to transmit the output data for display on a display device.

Description

Technical Field

[0001] Examples relate to an apparatus for an optical imaging system (e.g., a surgical optical imaging system), an optical imaging system, a method, and a computer program. Background Technology

[0002] Surgical microscopes are typically equipped with image-guided surgery (IGS) systems. An IGS system is a technique that guides the surgeon during surgical procedures using preoperative imaging data, such as computed tomography (CT), magnetic resonance imaging (MRI), or ultrasound. It allows the surgeon to see the patient's internal anatomy in real time and overlay it with the preoperative imaging data. This makes surgical navigation more accurate and precise, making it a useful tool for complex surgeries such as brain and spinal surgeries.

[0003] The technical principle of IGS involves using a tracking system (such as an infrared camera or an electromagnetic tracker) to determine the position and orientation of the surgical microscope relative to the patient's internal anatomy. This tracking data is used to create a three-dimensional model of the patient's anatomy and overlay it onto preoperative imaging data. This allows the surgeon to observe the patient's internal anatomy in real time, aligned with the surgical microscope's viewpoint. As the microscope moves, the system dynamically updates the overlay, ensuring the data remains aligned.

[0004] Virtual reality (VR) allows users to experience computer-generated 3D environment simulations and can be used in IGS systems. VR immerses users in a simulated world, allowing them to navigate within virtual objects, thus providing a more realistic perception and understanding of 3D structures. However, if virtual objects are displayed alongside images of samples, overlap may exist between different information fragments, potentially affecting the usability of the information.

[0005] Therefore, it may be desirable to have an improved concept for providing an image composed of preoperative imaging data and images of the sample acquired by an optical imaging system. Summary of the Invention

[0006] This expectation is addressed through the subject matter of the independent claims.

[0007] The concept presented in this disclosure is based on the insight that output data can be improved by synthesizing an output image in which opaque regions from multiple regions are superimposed on an object used for guidance. The opaque regions can be areas within multiple regions of an image of a sample. That is, the image of the sample can be segmented into multiple regions. Based on the segmented image of the sample, opaque and / or transparent regions can be determined to determine the output data. For example, opaque regions can be superimposed such that they overlap with the object used for guidance. Optionally or alternatively, the opacity of the transparent regions can be adjusted so that it does not affect the perception of the displayed object used for guidance.

[0008] An example provides an apparatus for an optical imaging system, the apparatus including one or more processors and one or more storage devices. The apparatus is configured to acquire sample data from a sensor of a microscope in the optical imaging system. The sample data indicates an image of the sample. Furthermore, the apparatus is configured to determine segmentation data indicating the segmentation of the image of the sample into multiple regions, and to obtain object data indicating an object for guiding a user of the optical imaging system. The apparatus is also configured to determine output data based on the object data and the segmentation data, indicating a composite output image of the image of the sample and the object for guidance, such that opaque regions in the multiple regions are superimposed on the object for guidance. Furthermore, the apparatus is configured to transmit the output data for display on a display device. Segmenting the image of the sample into multiple regions allows for the determination of different regions for adjusting opacity. For example, the image of the sample may include regions of different interest to the user. Opaque regions may be regions of interest to the user. That is, opaque regions may include structures of interest to the user. For example, opaque regions may be determined to be superimposed on the object for guidance. Opaque regions may be opaque. That is, opaque regions may be non-transparent. In this way, information about the opaque regions can overlap with information about the object for guidance. Therefore, a synthetic output image can be displayed to the user, including the object used for guidance and the opaque areas overlapping with it. Segmenting the sample image allows for the provision of information about the object used for guidance and the sample image in an improved manner. For example, the user's perception can be enhanced through the synthetic output image.

[0009] In the example, the device can be configured to determine segmentation data by identifying transparent regions among multiple regions to reduce opacity. The transparent regions are determined based on sample data. Furthermore, the device can be configured to determine segmentation data by generating a transparent image by reducing the opacity of the transparent regions. Additionally, the device is configured to determine output data by overlaying the guided object onto the transparent image. Reducing the opacity of the transparent regions allows for hiding structures that are less interesting or uninteresting to the user. Therefore, for example, the transparent image may not contain regions that are less interesting or uninteresting to the user. In this way, the composite output image can be easily determined by overlaying the guided object onto the transparent image. That is, the entire transparent image can overlap with the guided object. Due to the reduced opacity of the transparent regions, the visibility of the guided object can be increased. The transparent region can be one of multiple regions for which the opacity can be adjusted to adjust the visibility of the guided object. For example, the transparent region can be a region in the sample image that is less interesting or uninteresting to the user. For example, the transparent region can be a region that does not contain information to be displayed to the user. Therefore, the opacity of the transparent regions can be reduced without affecting the user experience and / or the perception of the sample image.

[0010] In the example, the device can be configured to reduce the opacity of the transparent region, making the transparent region truly transparent. That is, the opacity of the transparent region can be reduced to essentially zero. Therefore, the structure of the transparent region may not be displayed in the synthesized output image.

[0011] In this example, the device can be configured to determine the opacity level of the transparent areas based on sample data. For instance, multiple transparent areas may include multiple different structures that the user has different interests in. Therefore, the opacity level can be determined based on the user's different interests. For example, the opacity of a transparent area including structures that the user is not interested in can be adjusted to be lower than the opacity of a transparent area including structures that the user has low interest in. In this way, the opacity of the transparent areas can be adjusted according to the actual structure of the transparent areas (i.e., the user's interests).

[0012] In the example, the device can be configured to determine the outer size of the transparent area based on object data, such that the outer size of the transparent area is equal to or greater than the outer size of the object used for guidance. Therefore, it can be ensured that the object used for guidance is only overlaid with a transparent area whose opacity can be adjusted. Alternatively, the outer size of the transparent area can be smaller than the outer size of the object used for guidance.

[0013] In the example, the device can be configured to determine output data by identifying opaque regions and by overlaying the object used for guidance onto the opaque regions. That is, a region including structures of interest to the user can be determined to overlap with the object used for guidance. For example, the device can perform the determination of structures of interest to the user to define opaque regions. The opaque regions can be areas that include information to be displayed to the user. For example, the opaque regions can be defined by structures of interest to the user (e.g., arteries, veins, organs, tumors).

[0014] In the example, the device can be configured to determine spatial allocation data indicating the spatial allocation of at least one of a plurality of regions. Furthermore, the device can be configured to obtain object data by determining the size of an object based on the spatial allocation data. In this way, the coverage of the object used for guidance can be adjusted to areas within the plurality of regions, for example, to opaque and / or transparent areas.

[0015] In this example, the device can be configured to determine segmentation data through color-based segmentation, spectral unmixing, and / or morphology-based identification of anatomical structures. In this way, the device can select an appropriate process to segment the image.

[0016] In the example, object data can indicate 3D objects used to guide the user, and the synthesized output image is a 3D visualization. Therefore, a 3D image of the sample and a 3D object used for guidance can be displayed to the user.

[0017] In the example, the device can be configured to acquire posture data indicating the user's pose and determine output data based on the posture data. Acquiring posture data allows the synthesized output image to be adjusted to the user's actual viewing angle. For example, the orientation of the object used for guidance can be adjusted to the user's viewing angle. In this way, the user experience can be improved.

[0018] In this example, the device can be configured to adjust the display angle of the output data based on attitude data. That is, the orientation of the synthesized output image can be adjusted. This improves the user's perception.

[0019] In this example, the image of the sample is a two-dimensional view of the sample, and the device can be configured to adjust the display angle by adjusting the rotation angle of the image. In this way, the displayed two-dimensional view of the sample can be adjusted for the user's posture, such as a white light view, a fluorescence view, or a multispectral view.

[0020] In the example, the device can be configured to determine segmentation data by determining the opacity level of at least one region among multiple regions based on pixel values ​​of multiple pixels in an image of the sample. For example, the opacity level of at least one region can be determined based on the concentration of spectral components. Optionally or alternatively, different opacity levels, i.e., opaque regions and / or transparent regions, can be assigned to the structure. In this way, the display of the synthesized output image can be improved.

[0021] The example provides an optical imaging system that includes the device as described above.

[0022] An example provides a method for an optical imaging system, the method comprising acquiring sample data from a sensor of a microscope of the optical imaging system, the sample data indicating an image of a sample. The method further comprises determining segmentation data indicating that the image of the sample is segmented into multiple regions. The method further comprises obtaining object data indicating an object for guiding a user of the optical imaging system. The method further comprises determining output data based on the object data and the segmentation data, indicating a composite output image of the image of the sample and the object for guidance, such that opaque regions of the multiple regions are superimposed on the object for guidance. The method further comprises transmitting the output data for display on a display device.

[0023] The various examples disclosed herein relate to corresponding computer programs having program code for performing the methods described above when the computer program is executed on a processor. Attached Figure Description

[0024] The following description will use only examples of devices and / or methods, with reference to the accompanying drawings, in which...

[0025] Figure 1a and 1b A schematic diagram of an example of a device for an optical imaging system and a corresponding optical imaging system including the device is shown;

[0026] Figures 2a-2d An example of generating a synthetic output image is shown;

[0027] Figures 3a-3f Another example of generating a synthetic output image is shown;

[0028] Figure 4 A flowchart illustrating an example of a method for an optical imaging system using reflective imaging is shown; and

[0029] Figure 5 A schematic diagram of a system including a microscope and a computer system is shown. Detailed Implementation

[0030] Various examples will now be described more fully with reference to the accompanying drawings, some of which illustrate certain aspects. For clarity, the thickness of lines, layers, and / or regions may be exaggerated in the drawings.

[0031] Figure 1a and 1b A schematic diagram is shown of a device 130 for an optical imaging system 100 and an example of a corresponding optical imaging system 100 including device 130. The task of device 130 is to control the microscope 120 of the optical imaging system 100 (which may be a surgical optical imaging system) and various aspects of the entire optical imaging system 100, and / or to process various types of sensor data from the optical imaging system 100. Therefore, device 130 can be implemented as a computer system that interfaces with various components of the optical imaging system (e.g., sensor 122).

[0032] like Figure 1a As shown, device 130 includes one or more processors 134 and one or more storage devices 136. Optionally, device 130 also includes one or more interfaces 132. The one or more processors 134 are coupled to one or more storage devices 136 and optionally one or more interfaces 132. Typically, the functionality of device 130 may be provided by one or more processors 134 (for determining output data) in combination with one or more interfaces 132 (for exchanging information, such as exchanging information with sensor 122 or display device 180, for example, to transmit output data) and / or one or more storage devices 136 (for storing and / or retrieving information).

[0033] Device 130 is configured to acquire sample data from a sensor of microscope 120 of optical imaging system 100. The sample data indicates an image of sample 110. For example, the sample data can be acquired by receiving it from a sensor (e.g., optical imaging sensor 122 of microscope 120) or from a frame buffer (e.g., part of optical imaging system 100). Optionally or alternatively, the sample data can be acquired by measurement by device 130. For example, optical imaging sensor 122 can be part of device 130. Therefore, device 130 can control optical imaging sensor 122 to measure sample data. The sample data can be raw data from the sensor, i.e., the sample data can be post-processed by device 130. Alternatively, the sample data can be post-processed data from the sensor, i.e., further post-processing by device 130 may not be required.

[0034] Images of sample 110 may be white light images, fluorescence images, spectral or multispectral images and / or any combination of these images or derivatives of any of these images. For example, white light imaging or fluorescence imaging may be used to acquire images of sample 110.

[0035] The image of sample 110 can be a live view. A live view refers to the real-time display of sample 110. For example, a live view allows a user to see a continuous and up-to-date display of sample 110 being acquired by the microscope.

[0036] Real-time viewing can be particularly useful in process-controlled microscopy applications and / or during surgery, as it allows users to see an image of sample 110 in real time and adjust the microscope or sample 110 as needed.

[0037] For example, real-time views can be used to inspect and monitor the quality of workpieces during the manufacturing process, or to adjust the focus and positioning of a microscope to ensure that the real-time view is correctly acquired, or to perform surgery.

[0038] The image of sample 110 can include many regions, i.e., multiple regions. For example, the image of sample 110 can include different structures, such as arteries, veins, organs, bones, and tumors. The user of optical imaging system 100 can have different interests in different structures. That is, for example, it is not necessary to show every single region among the multiple regions to the user. The inventors have discovered that the output image can be determined by overlaying the object used for guidance with opaque areas among the multiple regions.

[0039] Therefore, device 130 is configured to determine segmentation data instructing the segmentation of sample 110 into multiple regions. For example, an image of sample 110 can be segmented by device 130 through camera-based segmentation and / or spectral unmixing as described below. In this way, different regions of the image of sample 110 can be determined by device 130. The image of sample 110 may include different structures of different interest to the user. That is, for example, the image of sample 110 can be segmented based on different structures of the image of sample 110. For example, the image of sample 110 may include a first region defined by an artery, a second region defined by bone, and a third region defined by a tumor. Optionally or alternatively, the image of sample 110 can be segmented based on the concentration of spectral components. That is, segmentation into multiple regions can be based on the concentration of spectral components in the image of sample 110. In this way, multiple regions can be determined, for example, based on the concentration of blood and / or blood particles.

[0040] For example, device 130 can identify opaque regions in an image of sample 110. These opaque regions can be regions of interest to the user. That is, the opaque regions can include structures of interest to the user. Therefore, in the synthesized output image, the opaque regions can be superimposed on the guiding object. In this way, the user can see the guiding object and the relevant structures of sample 110.

[0041] Optionally or alternatively, device 130 may identify transparent regions in an image of sample 110. Transparent regions may be areas of little or no interest to the user. For example, transparent regions may not include structures of interest to the user. Therefore, the opacity of transparent regions can be reduced to improve the visibility of underlying objects used for guidance. For example, when a transparent region does not include structures of interest to the user, the opacity of the transparent region may be reduced to 0 by device 130. That is, transparent regions may not affect the perception of underlying objects used for guidance.

[0042] Therefore, device 130 can obtain a composite output image by determining opaque and / or transparent regions. Based on the determined opaque and / or transparent regions, device 130 can adjust the opacity of the corresponding regions. For example, for transparent regions, the opacity can be reduced and / or for opaque regions, the opacity can be increased.

[0043] To determine the synthesized output image, device 130 needs to obtain information about the object used for guidance. Therefore, device 130 is configured to obtain object data indicating the object used to guide the user of optical imaging system 100. The object used for guidance may be a preoperative image of sample 110 or a portion of sample 110. For example, a preoperative image may show organic tissue to be removed during surgery. For example, the object used for guidance may be an lateral and / or medial image of sample 110 or a portion of sample 110. Therefore, the object used for guidance can provide surgical guidance to the user of optical imaging system 100. Optionally or alternatively, the object used for guidance may be navigation prompts, visualization of surgical tools, and / or the path of surgical instruments.

[0044] Object data can be obtained by device 130 by retrieving it from a storage device (such as storage device 136). Optionally or alternatively, object data can be determined by device 130. For example, device 130 may include sensors for measuring the environment of sample 110 (e.g., the posture of the user of optical imaging system 100). Thus, device 130 can determine object data based on the user's posture. For example, device 130 can determine the orientation of an object for guidance based on the user's posture.

[0045] Using object data and sample data allows the user of the optical imaging system 100 to be provided with information about both the object used for guidance and an image of the sample 110. Therefore, the device 130 is configured to determine output data indicating a composite output image of the image of the sample 110 and the object used for guidance. The output data is determined based on the object data and segmentation data. The composite output image includes an overlay of opaque regions and the object used for guidance in multiple regions. That is, opaque regions can be overlaid on the object used for guidance. In this way, structures of interest to the user in the image of the sample 110 can be displayed to the user along with the object used for guidance, for example, on a display device 180.

[0046] The opaque area and the object used for guidance can be part of different image layers. For example, the opaque area can be part of the sample layer, and the object used for guidance can be part of the guide layer. That is, the sample layer can be placed above or below the guide layer. In this case, the device 130 can perform adjustments to the opacity of each layer (e.g., the sample layer) to control the visual effect, but the pixel values ​​of the separate layers will not be mixed. The user's perception of the synthesized output image can depend on the order of the sample layer and the guide layer.

[0047] Alternatively, the opaque area and the object used for guidance can have two overlapping images, and these images can be superimposed on each other, also known as overlaying on the latter image. In this case, the pixel values ​​of the two images are located at the same coordinates in space, and the pixels of the top layer can partially or completely cover the pixels of the bottom layer. Depending on how the blending or mixing is set, the final pixel color at that location can be a combination of the color values ​​of the two overlapping pixels.

[0048] Furthermore, device 130 is configured to transmit output data for display on display device 180. For example, output data is transmitted to display device 180. Optionally or alternatively, output data is transmitted to a frame buffer, such as a portion of optical imaging system 100.

[0049] In this way, device 130 can provide an overlay visualization of an image of sample 110 and an object used for guidance. Typically, overlay visualization of preoperative data (e.g., two-dimensional or three-dimensional data) on a real-time color image of sample 110 acquired with a microscope is of very poor quality in terms of user perception, especially three-dimensional perception. The microscope image shows the surface of sample 110, such as tissue, while the preoperative data is always located in deep tissue. For example, visualization of blood vessels inside tissue is a counterintuitive picture (see also Figure 2).

[0050] For example, preoperative data used for ISG, such as computed tomography (CT) and magnetic resonance imaging (MRI) data, are not inherently and readily available in a form perceptible to humans. A major limitation may be that the entire volume of the human body is typically filled with tissue, and the surface may conceal structures within the volume. For example, a human body part may include many anatomical components, and in a single visualization (e.g., in an image of sample 110), it may not be possible to visualize all of them in conjunction with the object used for guidance. The inventors discovered that by segmenting an image of sample 110 acquired by a microscope, the object used for guidance can be combined with the image of sample 110 to improve the visualization of the synthetic output image. For example, to increase the visibility of other components (e.g., arteries, veins, lungs), multiple anatomical components (such as skin, muscles, and bones) are omitted. In this way, the synthetic output image can be limited to the structure of interest to the user. For example, the image of sample 110 can be segmented to determine opaque areas to be superimposed on the object used for guidance. By segmenting the image of sample 110, information overhead can be reduced and / or the perception of the object used for guidance can be improved.

[0051] Therefore, the output data determined by device 130 allows the user to perceive the object used for guidance in a natural and / or more precise manner. Device 130 can facilitate a faster understanding of the relative position, arrangement, and / or size of the object used for guidance. Device 130 can avoid information overflow. This can help the user focus on important elements by omitting unnecessary anatomical details in the image displaying sample 100. Device 130 can simplify the observation of preoperative information. For example, in cases where complex critical anatomical structures are entangled with pathological tissues, current microscopic IGS systems are considered inadequate, and surgeons need to use additional IGS hardware to probe the object used for guidance. This situation can be avoided using device 130. Device 130 can be used to achieve combined visualization of images of sample 110 (e.g., real-time (i.e., live view) white light images) and the object used for guidance (e.g., three-dimensional preoperative images).

[0052] The proposed concept can be built around two main components: microscope 120 (which includes optical components) and device 130 (which can be used to control optical imaging system 100, process sensor data of microscope 120 (e.g., optical imaging sensor 122) and / or determine output data).

[0053] Typically, microscopes such as microscope 120 are optical instruments suitable for examining objects too small to be examined (by the human eye alone). For example, microscope 120 can provide samples (such as...) Figure 1aOptical magnification of sample 110 (shown in the image). In modern microscopes, optical magnification is commonly used in cameras or imaging sensors, such as the optical imaging sensor 122 of microscope 120. Microscope 120 may also include one or more optical magnification components for magnifying a view of sample 110 (such as an object).

[0054] Various types of optical imaging systems exist. If the optical imaging system 100 is used in the medical or biological field, the sample 110 can be a sample of organic tissue, for example, arranged in a petri dish or present in a part of a patient's body. In some examples of this disclosure, such as... Figure 1b As shown, the optical imaging system 100 can be a surgical optical imaging system, such as an optical imaging system that will be used during or during surgical procedures (such as tumor surgery). However, the proposed concept can also be applied to other types of microscopy, such as laboratory microscopy or microscopy for materials examination purposes.

[0055] It is evident that the optical imaging system 100 includes a number of components, such as device 130, microscope 120 having at least one optical imaging sensor 122, optional primary eyepiece display pair 140, optional auxiliary eyepiece display pair 145, and display device (such as head-mounted display 180 or display).

[0056] Figure 1b A schematic diagram of an example of a surgical optical imaging system 100 including a microscope 120 and an instrument 130 is shown. Generally, a (surgical) optical imaging system is a system that includes a microscope 120 and additional components that operate in conjunction with the microscope 120. In other words, a (surgical) optical imaging system is a system that includes a microscope 120 and one or more additional components, such as an instrument 130 (which may be a computer system adapted to control the microscope 120 and, for example, determine output data), an illumination system (for illuminating the sample imaged by the microscope 120), additional sensors, a display, etc.

[0057] Figure 1b The surgical optical imaging system 100 shown includes a number of optional components, such as a base unit 105 with a support (which may include device 130), eyepiece displays 140, 145 disposed at a microscope 120, a head-mounted display 180, and a (robotic or artificial) arm 160 that holds the microscope 120 in place and is coupled to the base unit 105 and the microscope 120. Typically, these optional and non-optional components can be coupled to device 130, which can be configured to control and / or interact with the respective components.

[0058] In the example, device 130 can be configured to determine segmentation data by identifying transparent regions among multiple regions to reduce opacity. The transparent regions are determined based on sample data. That is, the image of sample 110 may include different regions of different interest to the user. Device 130 can identify regions in the image of sample 110 that the user is less interested in or less interested in as transparent regions. Since the user may be less interested in or less interested in the transparent regions, their opacity can be reduced. In this way, the visibility of guiding objects occluded by the image of sample 110 (i.e., by transparent regions) can be improved, and / or the perception of guiding objects can be enhanced.

[0059] Therefore, device 130 can be configured to determine segmentation data by generating a transparent image through reducing the opacity of transparent regions. Reducing the opacity of transparent regions allows for direct overlay of the guiding object onto the transparent image. For example, less interesting or uninteresting regions in the image of sample 110 can be made transparent by device 130. In this way, device 130 can perform the overlay of the guiding object onto the transparent image (derived from the image of sample 110) with less computational effort. Therefore, device 130 is configured to determine output data by overlaying the guiding object onto the transparent image.

[0060] For example, except for the opacity of the transparent areas, the transparent image can be the same as the image of sample 110. Therefore, overlaying the object used for guidance with the transparent areas instead of with the image of sample 110 may reduce occlusion of the object used for guidance.

[0061] The example involves overlaying the guiding object onto data derived from sample data (e.g., opaque regions and / or transparent images). In other examples, the guiding object can be overlaid on top of data derived from the sample data. That is, the guiding object can be located in an image layer above an image layer containing data derived from the sample data. For example, the guiding object can be overlaid on transparent images and / or opaque regions. In this case, the perception of the synthesized output image can also be improved. For example, the guiding object can be semi-transparent, and by reducing the opacity of the transparent regions overlapping the guiding object, the perception of the guiding object overlapping with the transparent regions can be improved. This may be particularly relevant for two distinct image layers, such as a sample layer and a guiding layer as described above.

[0062] In the example, device 130 can be configured to reduce the opacity of the transparent area, making the transparent area transparent. That is, the opacity of the transparent area can be reduced to essentially zero. Therefore, any structure of the transparent area cannot be displayed on the display device displaying the composite output image.

[0063] In the example, device 130 can be configured to determine the opacity level of transparent regions based on sample data. For example, device 130 can identify structures in an image of sample 110, such as arteries, veins, organs, or tumors. The identified structures can be assigned as transparent regions. Based on the identified structures, device 130 can determine the opacity level. For example, a vein may have a different opacity than a tumor. Optionally, a user can define the opacity level, for example, by defining the opacity of the structure. For example, the user's definition can be received via user input from an input device such as a touch display or keyboard. Optionally or alternatively, the opacity level can be determined based on the concentration of spectral components of the image of sample 110. That is, device 130 can adjust the opacity level of transparent regions based on the concentration of spectral components. Optionally, opaque regions can be adjusted similarly to transparent regions. That is, device 130 can be configured to determine the opacity level of opaque regions based on sample data and / or user input as described with reference to transparent regions.

[0064] In the example, device 130 can be configured to determine the outer dimensions of the transparent region based on object data, such that the outer dimensions of the transparent region are equal to or greater than the outer dimensions of the object used for guidance. That is, the size of the transparent region can depend on the size of the object used for guidance. For example, the size of the transparent region can be determined such that its coverage area is at least as large as the coverage area of ​​the object used for guidance. In this way, the perception of the object used for guidance can be improved because the transparent regions of the image of sample 110 overlapping with the object used for guidance can have at least the same coverage area (or substrate area).

[0065] In the example, device 130 can be configured to determine output data by identifying opaque regions and by overlaying the object used for guidance onto the opaque regions. That is, regions including structures of interest to the user can be identified to overlap with the object used for guidance. Determining opaque regions can be performed by device 130 in conjunction with determining transparent regions, or alternatively. That is, device 130 can determine transparent regions, opaque regions, and / or combinations thereof.

[0066] The opaque area may include structures of interest to the user. That is, the opaque area may include data to be displayed to the user during the use of the optical imaging system 100. For example, the opaque area may include tumors to be removed during surgery and / or arteries that should not be touched during surgery.

[0067] Therefore, determining segmented data may include determining transparent regions and / or determining opaque regions. In this way, structures in the image of sample 110 can be determined based on different parameters. For example, structures that should be displayed to the user can be determined to define opaque regions, and / or structures that should not be displayed to the user can be determined to define transparent regions. Optionally, device 130 can set the opacity of opaque regions. For example, the opacity of opaque regions can be increased to enhance the perception of opaque regions. Optionally, different opaque regions can have different opacities.

[0068] In the example, device 130 can be configured to determine spatial allocation data indicating the spatial allocation of at least one region among a plurality of regions. Furthermore, device 130 can be configured to obtain object data by determining the size of an object based on the spatial allocation data. The spatial allocation of at least one region among the plurality of regions can be the coverage area (or base area) or size of at least one region among the plurality of regions. That is, the coverage area or size of the object used for guidance can be adjusted to the coverage area or size of at least one region. For example, at least one region can be a transparent region, such as one with an opacity of 0. In this case, the coverage area or size of the object used for guidance can be set such that it substantially matches the coverage area of ​​the transparent region. In this way, the perception of the object used for guidance can be improved.

[0069] In the example, device 130 can be configured to determine segmentation data through color-based segmentation, spectral unmixing, and / or morphology-based identification of anatomical structures (e.g., blood vessels). In this way, device 130 can select appropriate processes to segment the image. For example, color-based segmentation may include color space transformation, thresholding, filtering and post-processing, connectivity analysis, and / or region-based segmentation. For example, spectral unmixing may include hyperspectral imaging and member identification, linear unmixing, and / or abundance mapping. For example, identification of anatomical structures may include feature extraction, morphological operations, object detection, and / or classification. The choice of method can depend on the nature of the image. Color-based segmentation is suitable for images where objects of interest have different and separable colors. Spectral unmixing can be used in hyperspectral images to identify materials with unique spectral characteristics. Morphology-based identification of anatomical structures can be common in medical imaging, where structures such as cells, organs, or blood vessels are characterized by their shape and texture.

[0070] In the example, device 130 can be configured to determine segmentation data by determining the opacity level of at least one of a plurality of regions. The opacity level can be determined based on pixel values ​​of multiple pixels in an image of sample 110. For example, the concentration of spectral components of an image of sample 110 can be used by device 130 to determine the opacity level. Therefore, device 130 can be configured to adjust the opacity proportionally to the concentration in the structure. For example, the opacity can be proportional to the blood concentration in a vein.

[0071] Optionally or alternatively, the opacity level can vary in at least one region (e.g., an opaque region or a transparent region). For example, the central portion of a vein may have higher opacity than the edges of the vein. Thus, the device 130 can determine multiple opacity levels for the structure (i.e., opaque and / or transparent regions). In this way, the perception of the structure can be improved.

[0072] In the example, object data can indicate 3D objects used to guide the user, and the synthesized output image is a 3D visualization. Therefore, a 3D image of sample 110 and the 3D objects used for guidance can be displayed to the user.

[0073] In this example, device 130 may be configured to acquire posture data indicating a user's posture and determine output data based on the posture data. The posture data may be acquired, for example, from an environmental sensor such as a camera and / or from microscope 120. Optionally or alternatively, the posture data may be measured by device 130. For example, device 130 may include microscope 120 and may control optical imaging sensor 122 to measure the posture data. For example, microscope 120 may be an exoscope.

[0074] Two factors determine a person's visual angle: head or face orientation (i.e., facial posture or facial direction) and eye orientation (i.e., the direction of eye gaze or visual angle). Head orientation can be used to determine the global direction of the gaze. Eye orientation can be used to determine the local direction of the gaze, i.e., the visual angle. When a person's head is level and he or she is looking straight ahead, the line of sight and facial orientation of the gaze are both straight ahead, with an azimuth of 0°, an elevation of 0°, and a tilt of 0°. This head position can also be described as a neutral head orientation position.

[0075] Device 130 can determine output data using the user's eye position (e.g., the user's viewing angle). For example, the user's viewing angle can indicate or be the same as the user's observation angle, which can be used to determine output data. Device 130 can also determine output data using the position of the user's head relative to microscope 100 (e.g., head orientation). For example, head orientation can indicate the user's observation angle, which can be used to determine output data. In other words, the visualization of the synthesized optical image can be adjusted to the user's actual viewing angle.

[0076] Device 130 can adjust the image of sample 110 and / or the object used for guidance based on attitude data. For example, device 130 can automatically adjust the image of sample 110 and / or the object used for guidance based on attitude data to improve perception of the synthesized output image. For example, the image of sample 110 may be a two-dimensional image, but an image of sample 110 may be captured using a stereomicroscope. Therefore, a parallax map can be used to rotate the two-dimensional image to take into account the user's attitude. Optionally or alternatively, the object used for guidance may be rotated to take into account the user's attitude. For example, device 130 may retrieve the object used for guidance from a storage device (such as storage device 136), may receive attitude data, and may generate an adjusted object for guidance by rotating the retrieved object for guidance based on the attitude data.

[0077] In the example, device 130 can be configured to adjust the display angle of the output data based on the pose data. That is, the orientation of the synthesized output image can be adjusted. In this way, the user's perception can be improved.

[0078] In the example, the image of sample 110 is a two-dimensional view of sample 110, and device 130 can be configured to adjust the display angle by adjusting the rotation angle of the image. In this way, the displayed two-dimensional view of sample 110 can be adjusted according to the user's posture, such as a white light view, a fluorescence view, or a multispectral view.

[0079] like Figure 1aAs shown, one or more optional interfaces 132 are coupled to corresponding one or more processors 134 at device 130. In the example, the one or more processors 134 may be implemented using one or more processing units, one or more processing devices, or any means for processing, such as a processor, computer, or programmable hardware component that can operate with appropriately adapted software. Similarly, the functionality of the described one or more processors 134 may also be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. The one or more processors 134 are capable of controlling one or more interfaces 132 such that any data transfers occurring through the one or more interfaces 132 and / or any interactions that may be involved in the one or more interfaces 132 can be controlled by the one or more processors 134.

[0080] In an implementation, device 130 may include a memory, such as one or more storage devices 136, and at least one or more processors 134 operatively coupled to the memory and configured to perform the methods described below.

[0081] In the example, one or more interfaces 132 may correspond to any means for acquiring, receiving, transmitting, or providing analog or digital signals or information, such as any connector, contact, pin, register, input port, output port, conductor, channel, etc., that allows the provision or acquisition of signals or information. One or more interfaces 132 may be wireless or wired, and may be configured to communicate with other internal or external components, such as transmitting or receiving signals or information.

[0082] Device 130 may be a computer, processor, control unit, (Field) Programmable Logic Array ((F) PLA), (Field) Programmable Gate Array ((F) PGA), Graphics Processing Unit (GPU), Application-Specific Integrated Circuit (ASIC), Integrated Circuit (IC), or System-on-Chip (SoC) system.

[0083] Further details and aspects are mentioned in conjunction with the examples described below. The examples shown in Figure 1 may include one or more optional additional features corresponding to one or more aspects mentioned in conjunction with the proposed concepts or one or more examples described below (e.g., Figures 2–5).

[0084] Figures 2a-2d An example of generating a synthetic output image is shown. Figure 2a and 2b An example of generating a transparent image is shown. Figure 2c and Figure 2d It shows the combination with the object used for bootstrapping. Figure 2a and Figure 2b Examples. In principle, Figure 2a and Figure 2c It can be considered as existing technology.

[0085] Figure 2a An image of the sample is shown. An optical imaging system can be used to acquire an image of the sample. The image of the sample may include different (anatomical) structures, such as the tissue vascular system 210 and certain tissue portions 220, 230, 240. Using the device described above (e.g., referring to Figure 1), the image of the sample can be segmented into multiple regions. For example, a first region may be defined by tissue portion 220, a second region may be defined by tissue portion 230, a third region may be defined by tissue portion 240, and a fourth region may be defined by the tissue vascular system 210. The structure of interest to the user may be the fourth region; that is, the tissue vascular system may be of interest to the user.

[0086] Therefore, the device can be configured to preserve the structure of interest, i.e., the fourth region. For example, the aim could be to preserve important anatomical structures, allowing the user (e.g., a surgeon) to maintain a clinically conscious orientation. Simultaneously, the remaining portions of the sample image, i.e., the first, second, and third regions that the user is less interested in or less interested in, can be made transparent (or their opacity can be reduced). In this way, the user can view through the less interesting or less interesting structures (see...). Figure 2b ).

[0087] For example, the device can use a transparent mask to... Figure 2a The image of the sample shown is converted to Figure 2b The transparent image shown. Figure 2a The current visualization of the IGS system is shown, and Figure 2b A portion of the synthesized output image is shown, such as a sample layer. The image of the sample can be a real-time image from a microscope of an optical imaging system. Therefore, the output data can also include real-time images. The image of the sample can be processed by the device to extract the desired channels (i.e., the fourth region) of the anatomical features to be visualized, and the rest of the image of the sample is set to transparent or its opacity reduced. Figure 2b In the example shown, the preserved channel or structure is the total hemoglobin channel, which is mainly composed of the tissue vascular system 210, while other tissue components 220, 230, 240, etc. Figure 2b The area shown is set to transparent.

[0088] Figure 2a and Figure 2b Only images without IGS data (i.e., object 250 used for boots) are shown. Figure 2c and Figure 2dThis includes IGS data, such as IGS visualizations (or simulations). Figure 2c An IGS visualization of the prior art is shown, in which a semi-transparent object 250 for guidance is superimposed on an image of the sample. Figure 2d A synthetic output image generated by the device described above is shown. It can be seen that a virtual impression of certain tissue portions being transparent can be generated, allowing the user to see the underlying guiding object 250 in an improved manner. That is, the user's perception of the guiding object 250 can be improved. The IGS visualization, i.e., the guiding object 250, can be located below the tissue surface shown in the image of the sample captured by the microscope. Alternatively, the IGS visualization can be located above the tissue surface.

[0089] With the current IGS visualization (which is usually a semi-transparent overlay) (see...) Figure 2c Unlike other devices, the device allows the object 250 used for guidance to be visualized as being seen through a transparent hole in an overlaid transparent image (see...). Figure 2d As described above, referring to Figure 1 for example, a transparent image can be generated by reducing the opacity of the transparent areas. Alternatively, a transparent image can be generated by selecting opaque areas of an image of the sample. That is, the opaque areas are superimposed on the object 250 used for guidance.

[0090] Exemplary Figure 2 discloses a two-dimensional implementation. Optionally, the device can also be used to generate a three-dimensional synthetic output image. This is described in more detail in Figure 3.

[0091] Further details and aspects are mentioned in conjunction with the examples described above and / or below. The examples shown in Figure 2 may include one or more optional additional features corresponding to one or more aspects mentioned in conjunction with the proposed concepts or one or more examples described above (e.g., Figure 1) and / or below (e.g., Figures 3–5).

[0092] Figures 3a-3f Another example of generating a synthetic output image is shown. Figure 3a , Figure 3c and Figure 3e The prior art implementation is shown, while Figure 3b , Figure 3d and Figure 3f An example of the synthesized output image is shown. Figure 3a and Figure 3b respectively with Figure 2a and Figure 2b Same. That is to say, Figure 3a and Figure 3b The image shows a perspective view of the microscope without any rotation (in three dimensions).

[0093] Figure 3c-3fAn example of the applied three-dimensional rotation is shown. Figure 3d and Figure 3f Examples of the proposed visualizations are shown, namely different examples of the synthesized output images, while Figure 3c and Figure 3e Equivalent visualization using a typical existing IGS system is shown. For example, the device described above can apply the same three-dimensional rotation to an image of the sample as it would to IGS data, allowing for easier and more precise perception of visible anatomical structures and the three-dimensional constellation of objects used for guidance. That is, Figure 3c , Figure 3d and Figure 3e , Figure 3f It can be Figure 3a , Figure 3b Rotation visualization.

[0094] Therefore, an aspect of the invention is that the device can perform a three-dimensional rotation on the synthesized output image (i.e., a combination of sample data and object data). The proposed visualization method applies the same three-dimensional rotation to the image of the sample (e.g., a white light image) as it does to the object 250 used for guidance, which allows for easier and more accurate perception of visible anatomical structures and the three-dimensional constellation of the object used for guidance.

[0095] To generate the examples shown in Figures 2 and 3, the device can treat the image of the sample as a two-dimensional planar image. However, optical imaging systems are typically stereoscopic, thus allowing for the calculation of the three-dimensional surface of the sample. This allows for more accurate and realistic results. Furthermore, even more accurate and robust three-dimensional scans of the surgical cavity can be provided using three-dimensional scanning techniques such as structured light. That is, the device can also be used to generate three-dimensional synthetic output images that include a three-dimensional image of the sample.

[0096] For example, a key element of human three-dimensional perception is observation from multiple different angles. The human brain utilizes the parallax effect of continuously rotating scenes, and this parallax effect leads to a very natural and accurate understanding of the scene. Therefore, real-time rotation of visualization data can be of great value. Some ways to achieve this include: using a typical computer interface with mouse pointer control (e.g., to receive user input), using gesture control instead of pointing devices (e.g., to receive user input), detecting head movements to simulate different viewing angles (e.g., to obtain posture data), for example, viewing a monitor from different angles changes the visualization perspective, using a digital observer with appropriate sensors to measure head position (e.g., to obtain posture data), and / or using a microscope handle to control the perspective. In this way, alignment between the user's perspective and the synthetic output image displayed on the display device can be provided.

[0097] Further details and aspects are mentioned in conjunction with the examples described above and / or below. The examples shown in Figure 3 may include one or more aspects mentioned in conjunction with the presented concepts or in conjunction with the preceding (e.g., Figures 1–2) and / or below (e.g., Figure 4 –5) One or more optional additional features corresponding to one or more examples described above.

[0098] Figure 4 An example of a method 400 for an optical imaging system is shown. Method 400 includes acquiring 410 sample data from a sensor of a microscope in the optical imaging system, the sample data indicating an image of the sample. Method 400 further includes determining 420 segmentation data, the segmentation data indicating the segmentation of the image of the sample into multiple regions. Method 400 further includes acquiring 430 object data, the object data indicating an object used to guide a user of the optical imaging system. Method 400 further includes determining, based on the object data and segmentation data, output data indicating a composite output image of the image of the sample and the object used for guidance, such that opaque regions in the multiple regions are superimposed on the object used for guidance. Method 400 further includes transmitting 450 output data for display on a display device. The method can be performed by a device as described above, for example, with reference to FIG1.

[0099] Further details and aspects will be mentioned in conjunction with the examples described above and / or below. Figure 4 The examples shown may include one or more aspects mentioned in conjunction with the proposed concept or the preceding (e.g., Figures 1–3) and / or the following (e.g., Figure 5 One or more optional additional features corresponding to one or more examples described above.

[0100] Some embodiments relate to a microscope that includes the device as described in conjunction with Figure 1. Alternatively, the microscope may include the device as described in conjunction with Figure 1, or may be communicatively connected to said device. Figure 5 This shows the configuration to execute this article, for example, reference. Figure 4A schematic diagram of a system 500 (e.g., an optical imaging system) described herein. System 500 includes a microscope 510 and a computer system 520. The microscope may include devices as described above, for example, with reference to FIG1. ​​Microscope 510 is configured to capture images and is connected to computer system 520. Computer system 520 is configured to perform at least a portion of the methods described herein. Computer system 520 may be configured to execute machine learning algorithms. Computer system 520 and microscope 510 may be separate entities, but may also be integrated together in a common housing. Computer system 520 may be part of the central processing system of microscope 510, and / or computer system 520 may be part of sub-components of microscope 510 (such as sensors, actuators, cameras, or illumination units of microscope 510).

[0101] Computer system 520 may be a local computer device (e.g., a personal computer, laptop computer, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in various locations (e.g., distributed at local clients and / or one or more remote server farms and / or data centers)). Computer system 520 may include any circuitry or combination of circuitry. In one embodiment, computer system 520 may include one or more processors of any type. As used herein, a processor may mean any type of computing circuitry, such as, but not limited to, a microscope or microscope component (e.g., a camera), such as, but not limited to, a microprocessor, microcontroller, complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, graphics processor, digital signal processor (DSP), multi-core processor, field-programmable gate array (FPGA), or any other type of processor or processing circuitry. Other types of circuitry that may be included in computer system 520 may be custom circuitry, application-specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radio components, and similar electronic systems. Computer system 520 may include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for disposing of removable media (such as optical discs (CDs), flash memory cards, digital video discs (DVDs), etc.). Computer system 520 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input information into and receive information from computer system 520.

[0102] In conjunction with the examples mentioned above, further details and aspects will be discussed. Figure 5 The examples shown may include one or more optional additional features corresponding to one or more aspects mentioned in conjunction with the proposed concept or one or more examples described above (e.g., Figures 1–4).

[0103] Some or all of the method steps may be performed by (or using) hardware devices (such as processors, microprocessors, programmable computers, or electronic circuits). In some embodiments, one or more of the most important method steps may be performed by such devices.

[0104] Depending on certain specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be executed using a non-transitory storage medium (such as a digital storage medium, e.g., floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory) that stores electronically readable control signals thereon, which cooperate with (or are capable of cooperating with) a programmable computer system to cause the corresponding method to be executed. Therefore, the digital storage medium can be computer-readable.

[0105] Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to enable the execution of one of the methods described herein.

[0106] Generally, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operable to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0107] Other implementations include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0108] Therefore, in other words, an embodiment of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is run on a computer.

[0109] Therefore, another embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) including a computer program stored thereon for performing one of the methods described herein when executed by a processor. Data carriers, digital storage media, or recording media are generally tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, including a processor and a storage medium.

[0110] Therefore, another embodiment of the invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0111] Another embodiment includes a processing means (e.g., a computer or programmable logic device) configured or adapted to perform one of the methods described herein.

[0112] Another embodiment includes a computer on which a computer program is installed for performing one of the methods described herein.

[0113] Another embodiment of the invention includes an apparatus or system configured to transmit, for example, electronically or optically to a receiver, a computer program for performing one of the methods described herein. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transmitting the computer program to the receiver.

[0114] In some implementations, programmable logic devices (e.g., field-programmable gate arrays) may be used to perform some or all of the functions of the methods described herein. In some implementations, field-programmable gate arrays may cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0115] If certain aspects of an apparatus or system have been described, those aspects should also be understood as a description of the corresponding method, and vice versa. For example, a block, device, or functional aspect of an apparatus or system may correspond to a feature of the corresponding method, such as a method step. Therefore, aspects of a method description should also be understood as a description of the corresponding block, element, characteristic, or functional feature of the corresponding apparatus or system.

[0116] The appended claims are incorporated herein by reference in the detailed description, wherein each claim may be taken independently as a separate example. It should also be noted that while in the claims a dependent claim refers to a specific combination with one or more other claims, other examples may also include combinations of dependent claims with the subject matter of any other dependent or independent claim. Such combinations are explicitly stated herein unless it is stated in individual cases that a particular combination is not intended to be used. Furthermore, for any other independent claim, the features of the claim should also be included, even if the claim is not directly defined as dependent on the other independent claim.

[0117] Aspects and features related to a specific example in the previous examples can also be combined with one or more other examples to replace the same or similar features of the other examples, or to introduce features additionally into the other examples. List of reference numerals 100 Optical Imaging System 105 Base 110 samples 120 microscope 122 Optical Imaging Sensor 130 equipment 132 interface 134 processor 136 storage devices 140 and 145 eyepiece displays 160 arms 180 display device 182 Head-mounted Display 210 Area of ​​Interest Areas 220, 230, and 240: Areas of little or no interest 250 objects used for bootstrapping 400 methods 410 Obtaining Sample Data 420 Determine the segmentation data 430 Obtaining Object Data 440 Determine the output data 450 Transmit Output Data 500 system 510 Microscope 520 Computer System

Claims

1. An apparatus (130) for an optical imaging system (100), comprising one or more processors (134) and one or more storage devices (136), wherein the apparatus (130) is configured to: Sample data is obtained from the sensor (122) of the microscope in the optical imaging system (100), the sample data indicating an image of the sample (110); The determination indicates that the image of the sample is segmented into multiple regions using segmentation data. Obtain object data indicating the object used by the user to guide the optical imaging system (100); Based on the object data and the segmentation data, output data is determined for an image indicating the sample and a composite output image of the object for guidance, such that opaque regions in the plurality of regions are superimposed on the object for guidance; as well as The output data is transmitted for display on the display device (180).

2. The device (130) according to claim 1, wherein the device (130) is configured to The segmented data is determined through the following operations: Based on the sample data, transparent areas are identified among the plurality of regions to reduce opacity; A transparent image is generated by reducing the opacity of the transparent areas; as well as The output data is determined by overlaying the object used for guidance onto the transparent image.

3. The device (130) according to claim 2, wherein the device (130) is configured to Reduce the opacity so that the transparent area is transparent.

4. The device (130) according to claim 2, wherein the device (130) is configured to The opacity level of the transparent region is determined based on the sample data.

5. The device (130) according to any one of claims 2-4, wherein the device (130) is configured to: The outer dimensions of the transparent region are determined based on the object data, such that the outer dimensions of the transparent region are equal to or greater than the outer dimensions of the object used for guidance.

6. The device (130) according to claim 1, wherein the device (130) is configured to The output data is determined by the following steps: Determine the opaque regions among the plurality of regions; and The output data is determined by overlaying the object used for guidance onto the opaque area.

7. The device (130) according to any one of the preceding claims, wherein the device (130) is configured to: Determine spatial allocation data indicating the spatial allocation of at least one of the plurality of regions; and The object data is obtained by determining the object's dimensions based on the space allocation data.

8. The device (130) according to any one of the preceding claims, wherein the device (130) is configured to The segmentation data is determined by at least one of color-based segmentation, spectral unmixing, and morphology-based identification of anatomical structures.

9. The device (130) according to any one of the preceding claims, wherein The object data indicates a 3D object used to guide the user, and The synthesized output image is a three-dimensional visualization.

10. The device (130) according to claim 9, wherein the device (130) is configured to: Obtain gesture data indicating the user's posture; and The output data is determined based on the attitude data.

11. The device (130) according to claim 10, wherein the device (130) is configured to The display angle of the output data is adjusted based on the attitude data.

12. The device (130) according to any one of the preceding claims, wherein the device (130) is configured to The segmentation data is determined by determining the opacity level of at least one region among the plurality of regions based on the pixel values ​​of a plurality of pixels in the image of the sample (110).

13. An optical imaging system (100), comprising: The device (130) according to any one of the preceding claims.

14. A method (400) for an optical imaging system, comprising: (410) Sample data is obtained from the sensor of the microscope in the optical imaging system, the sample data indicating an image of the sample; Determine (420) the segmentation data for dividing the image of the sample into multiple regions; Obtain (430) object data indicating the object used to guide the user of the optical imaging system; Based on the object data and the segmentation data, output data is determined (440) indicating the image of the sample and the composite output image of the object for guidance, such that the opaque regions in the plurality of regions are superimposed on the object for guidance; as well as The output data is transmitted (450) for display on a display device.

15. A computer program having program code for executing the method of claim 14 when the computer program is executed on a processor.

Images