Microscope system and corresponding method

Abstract

The embodiment relates to a microscope system, e.g., a surgical microscope system, and a corresponding method. The microscope system (100) includes a microscope (120) including a first optical imaging sensor (122) that forms first imaging sensor data based on light having a first wavelength spectrum, a second optical imaging sensor (124) that forms second imaging sensor data based on light having a second wavelength spectrum, and a multispectral iris (130) configured to form an aperture with a first numerical aperture for the light having the first wavelength spectrum and an aperture with a second numerical aperture for the light having the second wavelength spectrum, the second numerical aperture being different from the first numerical aperture. The microscope system includes one or more processors (114) that are configured to form a composite image based on the first imaging sensor data and the second imaging sensor data.

Description

[Technical field]

[0001] The embodiments relate to a microscope system, such as a surgical microscope system, and a corresponding method. [Background technology]

[0002] A microscope, such as those used in surgical microscope systems, is an optical system that includes various optical components. One optical component of a microscope is the iris, which is an adjustable opening that controls how much light reaches the microscope's eyepiece or optical imaging sensor. Opening and closing the iris affects the image resolution and depth of field. If the iris opening is larger, more light passes through the iris to the eyepiece or optical imaging sensor. This generally increases the resolution of the view or image, but decreases the depth of field, which is the distance interval at which the observed or acquired view of the specimen appears sharp. If the iris opening is smaller, less light passes through. This increases depth perception, but reduces resolution. Typically, surgical microscopes are set to have a fixed iris in a "sweet spot" to provide an acceptable balance of inherent tradeoffs.

[0003] Some surgical microscopes use a technology that can reduce the trade-off between resolution and depth of field. Its mode of operation is to use different iris sizes for the left and right eye in the optical eyepieces. As a result, the user sees two images (right / left) with different qualities, one with higher resolution and the other with a larger depth of field. The brain processes the information from both eyes perceived by the user, i.e. an image with both qualities of high resolution and large depth of field. Summary of the Invention [Problem to be solved by the invention]

[0004] It may be desirable to further address the tradeoff between resolution and depth of field in microscopy. [Means for solving the problem]

[0005] These desires are addressed by the subject matter of the respective independent claims.

[0006] The concept proposed in this disclosure is based on the discovery that a microscope, particularly one used as part of a surgical microscope system, can include multiple optical imaging sensors. For example, a surgical microscope can use separate optical imaging sensors for reflection imaging and fluorescence imaging. However, the sensors used for fluorescence imaging often cover one or more fluorescence emission wavelength bands, i.e. well-defined spectral subsets. Different sizes of iris apertures can be formed for two separate optical imaging sensors of the microscope, for example by using a so-called multispectral iris as shown in EP 3285116, to form different sizes of iris apertures for two separate wavelength spectra recorded by each optical imaging sensor. Image processing can then be used to combine the imaging sensor data of the two optical imaging sensors to obtain a composite image with the increased resolution provided by the larger iris aperture and the increased depth of field provided by the smaller iris aperture.

[0007] Various examples of the present disclosure relate to a microscope system, e.g., a surgical microscope system, including a microscope and one or more processors. The microscope includes a first optical imaging sensor that forms a first imaging sensor data based on light having a first wavelength spectrum. The microscope further includes a second optical imaging sensor that forms a second imaging sensor data based on light having a second wavelength spectrum. The microscope further includes a multispectral iris configured to form an aperture with a first numerical aperture for light having the first wavelength spectrum and an aperture with a second numerical aperture for light having the second wavelength spectrum. The first numerical aperture is different from the second numerical aperture. The one or more processors are configured to form a composite image based on the first imaging sensor data and the second imaging sensor data. By using a multispectral sensor, different wavelength spectra can be formed from differently sized iris apertures. By using two separate optical imaging sensors that sense light in different spectra, two imaging sensor data sets can be formed, one with increased resolution and the other with increased depth of field. In this case, the two imaging sensor data sets can be combined into a composite image to form a composite image that has both increased resolution and increased depth perception.

[0008] In microscopy, particularly in surgical microscopy, different optical imaging sensors are used for reflection imaging and fluorescence imaging. This inherent separation can be used to realize the proposed concept. For example, a first wavelength spectrum can be suitable for performing reflection imaging and a second wavelength spectrum can be suitable for performing fluorescence imaging. Alternatively, both the first and second optical imaging sensors can be used for reflection imaging or fluorescence imaging. In other words, the first and second wavelength spectra can be suitable for performing reflection imaging or the first and second wavelength spectra can be suitable for performing fluorescence imaging.

[0009] In various embodiments, the first numerical aperture is greater than the second numerical aperture. Thus, the first imaging sensor data can include imaging sensor data having a higher resolution, while the second imaging sensor data can include imaging sensor data covering a larger depth of field. In this application, the term "resolution" does not refer to the amount or density of pixels, but rather to optical resolution, generally measured in line pairs per millimeter.

[0010] The first imaging sensor data and the second imaging sensor data are combined into a composite image, thereby using, for example, both the increased resolution and the increased depth of field of the respective imaging sensor data. Both the increased resolution and the increased depth of field can be manifested in terms of increased image sharpness, where the increased resolution increases image sharpness where the respective images are in focus, and the increased depth of field increases image sharpness in areas where the high resolution images are out of focus. For example, the one or more processors can be configured to determine one or more portions of the first imaging sensor data that have higher image sharpness than one or more corresponding portions of the second imaging sensor data. The one or more processors can be configured to form a composite image using the one or more portions of the first imaging sensor data that have higher image sharpness, with the remaining portions of the composite image being based on the second imaging sensor data. In other words, from the first imaging sensor data, which may be based on a larger iris opening and therefore have higher resolution in focus, the sharper parts can be selected at higher resolution, and the remaining image can be based on the second imaging sensor data (which has lower resolution but greater depth of field).

[0011] Various techniques can be used to determine the sharp portions of the respective imaging sensor data. For example, the one or more processors can be configured to determine one or more portions of the first imaging sensor data that have higher image sharpness than one or more corresponding portions of the second imaging sensor data based on contrast and / or the presence of spatial frequencies above a predetermined spatial frequency threshold in the respective one or more portions.

[0012] In some embodiments, a grid-based approach can be used to determine the sharpness and combine the two imaging sensor data sets. For example, the one or more processors can be configured to subdivide the first imaging sensor data and the second imaging sensor data into a two-dimensional grid of pixel blocks, each including a plurality of pixels. The one or more processors can be configured to determine, on a pixel block basis, one or more portions of the first imaging sensor data that have a higher image sharpness than the corresponding one or more corresponding portions of the second imaging sensor data. For example, on a pixel block basis, a respective pixel block can be selected from one of the imaging sensor data sets to be used for the combined image.

[0013] In another approach (or in addition to the above), the two imaging sensor data sets can be merged at another level by using color information and spatial features from one imaging sensor data set (e.g., the first imaging sensor data) and (only) spatial features from the other imaging sensor data set (e.g., the second imaging sensor data). For example, the one or more processors can be configured to extract spatial features of the second imaging sensor data. The one or more processors can be configured to merge the spatial features from the second imaging sensor data with the spatial features and color information of the first imaging sensor data to form a composite image. This applies in particular when the second optical imaging sensor is generally used for fluorescence imaging, because the wavelength spectrum sensed by a fluorescence sensor is limited, making the color information contained in the second imaging sensor data of such a fluorescence sensor less useful than the color information contained in the first imaging sensor data.

[0014] Such separation of color information and spatial features can be performed by converting the respective imaging sensor data to an HSB (hue, saturation, brightness) color representation. For example, the one or more processors can be configured to process the first imaging sensor data and the second imaging sensor data in a hue, saturation, and brightness color representation. The one or more processors can be configured to merge the saturation and brightness components of the second imaging sensor data with the hue, saturation, and brightness components of the first imaging sensor data. For example, the hue component can include color information, and the saturation and brightness components can represent spatial features.

[0015] In the proposed concept, two separate optical imaging sensors are used to form the respective imaging sensor data. The respective imaging sensor data frames can be time-synchronized such that at a given time, a frame of the first imaging sensor data can be combined with a frame of the second imaging sensor data captured at the same time. Thus, the one or more processors can be configured to form a composite image based on the frames of the first imaging sensor data and the frames of the second imaging sensor data captured at the same time. Thus, the frame rate and synchronization of the two imaging sensor data sets are not affected by the proposed concept.

[0016] Microscope systems, particularly surgical microscope systems, generally include an illumination system that is used to illuminate an object (i.e., a sample) to be imaged. Thus, a microscope system may comprise an illumination system configured to illuminate an object to be imaged using the microscope system. The emission wavelength spectrum of the illumination system may include a first wavelength spectrum and a second wavelength spectrum. For example, the first wavelength spectrum may include a first wavelength band that is excluded from the second wavelength spectrum, and the second wavelength spectrum may include a second wavelength band that is excluded from the first wavelength spectrum. The emission spectrum of the illumination system may include a first wavelength band and a second wavelength band. In fact, suitable illumination may be provided to simultaneously form first and second imaging sensor data.

[0017] The proposed concept uses a multispectral iris, which is an iris with different numerical apertures for different wavelength bands. In various embodiments, such an iris can be implemented using bandpass filters that affect the perceived iris aperture in one wavelength band but not in another. For example, the multispectral iris can include at least one bandpass filtering component configured to block a second wavelength spectrum.

[0018] In some embodiments, the microscope can operate with one or more fixed aperture multispectral irises, for example, the microscope can include a filter wheel having a single multispectral iris with a fixed numerical aperture or multiple multispectral irises with fixed numerical apertures.

[0019] Alternatively, the first and / or second numerical apertures may be adjustable via motors and movable components. For example, a multispectral iris may include a plurality of (first) bandpass filtering components configured to block the second wavelength spectrum. The microscope may include a motor to adjust the relative position of the plurality of bandpass filtering components relative to one another. One or more processors may be configured to control the motors of the microscope system to adjust the relative position of the plurality of bandpass filtering components relative to one another, thereby adjusting the second numerical aperture. For example, the plurality of bandpass filtering components may form an aperture, or the plurality of bandpass filtering components may correspond to two L-shaped bandpass filtering components. For example, a plurality of bandpass filtering components may be transparent (i.e. non-obstructing) to the first wavelength spectrum that may be usable without affecting the first numerical aperture, to adjust the second numerical aperture.

[0020] To adjust the (larger) first numerical aperture, a second set of components that blocks light in both wavelength spectrums can be used. For example, a multispectral iris can include a plurality of second components configured to block the first and second wavelength spectrums. The microscope can include a second motor that adjusts the relative position of the plurality of second components with respect to each other. The one or more processors can also be configured to control the second motor to adjust the relative position of the plurality of second components with respect to each other, thereby adjusting the first numerical aperture. Thus, two sets of components can be used to control the first and second numerical apertures. Similarly, the plurality of second components can form an aperture, or the plurality of second components can correspond to two L-shaped bandpass filtering components.

[0021] For example, the one or more processors can be configured to control at least one motor of the microscope system to adjust at least one of the first and second numerical apertures, e.g., the at least one motor can be used to adjust a component set or to rotate a filter wheel.

[0022] Various embodiments of the present disclosure relate to a method for a corresponding microscope system. The method includes forming, with a multispectral iris, an aperture having a first numerical aperture for light having a first wavelength spectrum and an aperture having a second numerical aperture for light having a second wavelength spectrum. The first numerical aperture is different from the second numerical aperture. The method includes forming, with a first optical imaging sensor, first imaging sensor data based on the light having the first wavelength spectrum. The method includes forming, with a second optical imaging sensor, second imaging sensor data based on the light having the second wavelength spectrum. The method includes forming a composite image based on the first imaging sensor data and the second imaging sensor data.

[0023] Some embodiments of the apparatus and / or methods are now described, by way of example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0024] [Figure 1a] FIG. 1 is a schematic diagram illustrating an embodiment of a microscope system. [Figure 1b] FIG. 1 is a schematic diagram showing an embodiment of a surgical microscope system. [Figure 1c] FIG. 2 is a schematic diagram illustrating an embodiment of a multispectral iris. [Figure 1d] FIG. 2 is a schematic diagram illustrating an embodiment of a multispectral iris. [Figure 1e] FIG. 2 is a schematic diagram illustrating an embodiment of a multispectral iris. [Figure 1f]FIG. 2 is a schematic diagram illustrating an embodiment of a multispectral iris. [Diagram 2] 1 is a flow chart illustrating one embodiment of a method for a microscope system. [Diagram 3] FIG. 1 is a schematic diagram showing an embodiment of the digital fusion optical system concept. [Figure 4] FIG. 1 is a schematic diagram illustrating one embodiment of a system including a microscope and a computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Various embodiments will now be generally described with reference to the accompanying drawings in which several examples are shown, in which line thicknesses, layer thicknesses and / or area sizes may be exaggerated for clarity.

[0026] 1a is a schematic diagram illustrating an embodiment of a microscope system 100. The microscope system 100 includes a microscope 120, which includes a first optical imaging sensor 122 that forms a first imaging sensor data based on light having a first wavelength spectrum, and a second optical imaging sensor 124 that forms a second imaging sensor data based on light having a second wavelength spectrum. The microscope system 100 further includes a multispectral iris 130 configured to form an aperture with a first numerical aperture for the light having the first wavelength spectrum and an aperture with a second numerical aperture for the light having the second wavelength spectrum. The first numerical aperture is different from the second numerical aperture. Optionally, the microscope system 100 further includes an illumination system 140 configured to illuminate an object 10 to be imaged using the microscope system 100.

[0027] The optical imaging sensor 122; 124 and the iris 130 are part of the microscope 120 of the (surgical) microscope system 100, as shown in Fig. 1a and Fig. 1b. In general, a microscope, such as the microscope 120, is an optical instrument suitable for inspecting objects that are too small for a human to inspect (only) with the naked eye. For example, the microscope may provide optical multiplication of a sample, such as the sample 10 shown in Fig. 1a, Fig. 1b and Fig. 3. In modern microscopes, optical multiplication is often provided for a camera or an imaging sensor, such as the optical imaging sensor 122; 124 of the microscope 120. The microscope 120 may further include one or more magnification optics used to magnify the view on the sample, such as an objective lens (i.e., a lens).

[0028] Besides optical components, such as the microscope 120 with the optical imaging sensor 122; 124, the iris 130 and (optionally) an illumination system, the microscope system further includes a computer system with one or more processors 114 configured to perform image processing in the microscope system 100 and / or to control the microscope system 100. For example, as shown in Fig. 1a and Fig. 1b, the one or more processors 114 may be part of a (computer) system 110, which may further include one or more interfaces 112 and / or one or more storage devices 116 coupled to the one or more processors 114. In general, the functionality of the system is provided by one or more processors in cooperation with one or more interfaces (e.g., for exchanging information with the optical imaging sensor 122; 124 of the microscope, the multispectral iris 130 or a display device of the surgical microscope system) and / or one or more storage devices (for storing and / or retrieving information). The one or more processors 114, or more generally the system 110, are configured to form a composite image based on the first imaging sensor data and the second imaging sensor data.

[0029] In general, a microscope system is a system that includes a microscope 120 and additional components that operate therewith. In other words, a microscope system is a system that includes a microscope and one or more additional components, such as system 110 (a computer system adapted to control, e.g., process, the imaging sensor data of the microscope), illumination system 140 (used to illuminate the object being imaged by the microscope), additional sensors, a display, etc.

[0030] There are different types of microscopes. When a microscope is used in the medical or biological field, the object 10 observed through the microscope may be a sample of organic tissue, for example placed in a petri dish or present in a part of a patient's body. For example, the microscope 120 may be a microscope of a surgical microscope system, i.e. a microscope used during a surgical procedure, such as tumor surgery or during cancer surgery. The object observed through the microscope and shown in the composite image may therefore be a sample of the patient's organic tissue, in particular a surgical site where a surgeon will perform an operation during a surgical procedure.

[0031] 1b shows a schematic diagram of an embodiment of a surgical microscope system 100 including a system 110 (with an optical imaging sensor and a multispectral iris) and a microscope 120. The surgical microscope system 100 shown in FIG. 1b includes a number of optical components, such as a base unit 105 (including the system 110) with a (rotating) stand, an eyepiece display 150a disposed on the microscope 120, an auxiliary display 150b disposed on the base unit 105, and an arm 160 (robotic or manual) coupled to the base unit 105 and the microscope 120 to hold the microscope 120 in place. In general, these optional and non-optional components can be coupled to the system 110 and thus to one or more processors 114 that can be configured to control and / or interact with the respective components.

[0032] The proposed concept focuses on three components: a multispectral iris (to be precise), which produces different numerical apertures in two different wavelength spectrums, an optical imaging sensor used to sense light in the two different wavelength spectrums, and one or more processors (e.g., systems) configured to form a composite image based on the light sensed in the two different wavelength spectrums.

[0033] The proposed system is built around a multispectral iris 130, which is used in combination with the definition of two wavelength spectra to provide different numerical apertures for the two optical imaging sensors. While there are some approaches where the microscope system has separate irises for the different optical imaging sensors, i.e. the light passing through the first iris is directed towards the first sensor (only) and the light passing through the second iris is directed towards the second sensor (only), in this case the light passing through the multispectral iris is directed towards both optical imaging sensors. In other words, the light reaching the first optical imaging sensor and the second optical imaging sensor is the light passing through the same iris. However, due to the optical properties of the multispectral iris, the numerical apertures perceived by the different wavelength spectra are different. For example, the light reaching the two optical imaging sensors can be passed through the same objective lens and the same focusing system can be used.

[0034] In general terms, the multispectral iris 130 is configured to form an aperture with a first numerical aperture for light having a first wavelength spectrum and an aperture with a second numerical aperture for light having a second wavelength spectrum. The first numerical aperture of the aperture for light having a first wavelength spectrum is different from the second numerical aperture of the aperture for light having a second wavelength spectrum. Thus, the size of the aperture perceived by the light of each spectrum is different. This can be achieved by using a component that blocks one spectrum and passes the other wavelength spectrum. In other words, this can be achieved by one or more bandpass filters configured to block one of the spectrums. In the following, it is assumed that the first numerical aperture is greater than the second numerical aperture. Thus, the multispectral iris can include at least one bandpass filtering component (i.e., at least one optical bandpass filter) configured to block the second wavelength spectrum. On the other hand, the at least one bandpass filtering component does not block the first wavelength spectrum, i.e., allows it to pass through the at least one bandpass filtering component. As a result, the aperture perceived by the second wavelength spectrum may be smaller than the aperture perceived by the first wavelength spectrum (because the at least one bandpass filtering component limits the size of the aperture of the second wavelength spectrum).

[0035] 1c-1f are schematic diagrams of examples of multispectral irises. In a basic example, as shown in FIG. 1c, a multispectral iris can provide two fixed numerical apertures for two different wavelength spectrums. The multispectral iris 130 shown in FIG. 1c includes two components: a bandpass filtering component 132 (configured to reject a second wavelength spectrum) and an aperture 134. The bandpass filtering component 132 has a fixed size and is surrounded by a rim. In FIG. 1c, the first numerical aperture is defined by the size of the bandpass filtering component 132 (defined by the rim) and the second numerical aperture is defined by the size of the aperture 134.

[0036] In some embodiments, it may be desirable to change the first and / or second numerical aperture. If an iris with a fixed numerical aperture is used, such a change can be made by exchanging it for another iris, for example via a filter wheel. In other words, the microscope can include a filter wheel (or iris wheel) with multiple multispectral irises with fixed numerical apertures. An example of this is shown in FIG. 1d, where a filter wheel 136 includes four multispectral irises 130, each with a different first and / or second numerical aperture.

[0037] Typically, the filter wheel can be rotated manually. However, to allow the system to automatically control the numerical aperture, the filter wheel can be operated via a servo motor included in the microscope that is used to rotate the filter wheel to an appropriate position. One or more processors (e.g., the system) can be configured to control at least one motor of the microscope system to cause the motor to adjust at least one of the first numerical aperture and the second numerical aperture, for example by rotating the filter wheel.

[0038] However, the application of motors in microscopes is not limited to rotating filter wheels: instead, motors can be used to adjust the relative positions of multiple (bandpass filtering) components with respect to each other in order to adjust the second or first numerical aperture.

[0039] For example, as shown in Figures 1e and 1f, the multispectral iris can include multiple components 138 and a motor to adjust the position of each component 138. The components 138 shown in Figures 1e and 1f can be bandpass filtering components 138 configured to block (only) light of the second wavelength spectrum, or components 138 (e.g., made of metal or opaque plastic) configured to block both light of the first and second wavelength spectrums. In the former case, the components can be used to adjust the second numerical aperture (in which case the first numerical aperture is not affected by light of the first wavelength spectrum passing through the bandpass filtering components). In the latter case, the components can be used to adjust the first numerical aperture (in which case the second numerical aperture, which is smaller than the first numerical aperture, is not affected). For example, two such irises can be combined to form a multispectral iris (passing light reaching the optical imaging sensor through both irises in sequence). For example, a first iris including a plurality of components configured to block light of both a first wavelength spectrum and a second wavelength spectrum, and a second iris including a plurality of bandpass filtering components configured to block light of the second wavelength spectrum (but not the first wavelength spectrum) can be disposed in an optical path between an object and two optical imaging sensors, e.g., between the object and a beam splitter dichroic mirror configured to split a portion of light directed toward one of the optical imaging sensors (wherein light passing through the beam splitter dichroic mirror is directed toward the other optical imaging sensor).

[0040] The multispectral iris (e.g., a second iris of the multispectral iris) may include a plurality of (first) bandpass filtering components configured to reject a second wavelength spectrum. The microscope may include a motor to adjust the relative position of the plurality of (first) bandpass filtering components with respect to one another. The one or more processors (e.g., a system) may be configured to control the motors of the microscope system to adjust the relative position of the plurality of (first) bandpass filtering components with respect to one another, thereby adjusting the second numerical aperture. Similarly, the multispectral iris (e.g., a first iris of the multispectral iris) may include a plurality of second components configured to reject a first wavelength spectrum and a second wavelength spectrum. The microscope may include a second motor to adjust the relative position of the plurality of second components with respect to one another. The one or more processors (e.g., a system) may be configured to control the second motor to adjust the relative position of the plurality of second components with respect to one another, thereby adjusting the first numerical aperture.

[0041] In the example shown in Fig. 1e-1f, the positions of the multiple first and / or second components 138 are adjusted relative to each other to adjust the respective numerical apertures. In practice, the size of the aperture formed by each (bandpass filtering) component is adjusted by moving each (bandpass filtering) component relative to each other, thereby adjusting the respective numerical apertures. For example, as shown in Fig. 1e, each (bandpass filtering) component may correspond to two L-shaped components (e.g., two L-shaped components or two bandpass filtering components that block both the first and second wavelength spectra) 138. By moving the two L-shaped components toward the common center of gravity, the aperture formed by the two L-shaped components becomes smaller, thereby decreasing the numerical aperture. By moving the two L-shaped components away from the common center of gravity, the aperture formed by the two L-shaped components becomes larger, thereby increasing the numerical aperture.

[0042] Alternatively, as shown in Fig. 1f, an aperture can be formed by a respective (bandpass filtering) component 138. The component 138 can be rotated inwards, thereby decreasing the size of the aperture 132 (and therefore decreasing the numerical aperture), or rotated outwards, thereby increasing the size of the aperture 132.

[0043] Component 138 of Fig. 1f is used to adjust the first numerical aperture while keeping the second numerical aperture fixed according to the size of aperture 134 in the bandpass filter. The same scheme can also be applied to Fig. 1e, for example by combining a bandpass filter with a fixed size aperture with two L-shaped components of Fig. 1e. Alternatively, in both cases, instead of using a bandpass filter with a fixed size aperture, multiple (first) bandpass filtering components can be used to form an adjustable iris aperture for the second wavelength spectrum.

[0044] As outlined above, the (surgical) microscope system may include an illumination system 140 configured to illuminate an object to be imaged using the microscope system. Since both a first wavelength spectrum and a second wavelength spectrum are recorded using respective sensors, the illumination system may emit light across both wavelength spectrums. In other words, the emission wavelength spectrum of the illumination system may include (e.g., at least partially or completely cover) the first wavelength spectrum and the second wavelength spectrum. For example, the first wavelength spectrum may include a first wavelength band that is excluded from the second wavelength spectrum, and the second wavelength spectrum includes a second wavelength band that is excluded from the first wavelength spectrum, and the emission spectrum of the illumination system may include the first wavelength band and the second wavelength band.

[0045] In the proposed microscope system, a first optical imaging sensor and a second optical imaging sensor are used to sense (and measure) light of respective wavelength spectrums. In particular, the first optical imaging sensor 122 is configured to form a first imaging sensor data based on light having a first wavelength spectrum, and the second optical imaging sensor 124 is configured to form a second imaging sensor data based on light having a second wavelength spectrum. In the proposed concept, a multispectral iris is used to pass light with different effective numerical apertures towards the sensor. To avoid sensor sensing light with two (largely) different numerical apertures, the first and second wavelength bands can be different, e.g., non-overlapping. It is noted that the optical imaging sensor is not limited to either wavelength band. For example, the first optical imaging sensor can be configured to sense light with the first wavelength spectrum and, optionally, at least a portion of light with the second wavelength spectrum. Similarly, the second optical imaging sensor can be configured to sense light having the second wavelength spectrum and, optionally, at least a portion of the light having the first wavelength spectrum. If the optical imaging sensor senses light of the "other" respective wavelength spectrum, its effect can be computationally removed (e.g., by the system or one or more processors) from the respective imaging sensor data. For example, the one or more processors (e.g., the system) can be configured to remove, from the first imaging sensor data, the portion of the first imaging sensor data based on the light having the second wavelength spectrum. Similarly, the one or more processors (e.g., the system) can be configured to remove, from the second imaging sensor data, the portion of the second imaging sensor data based on the light having the first wavelength spectrum.

[0046] This is particularly important when the proposed concept is applied to a surgical microscope system. For example, in a surgical microscope system, a first optical imaging sensor can typically be used for reflectance imaging and a second optical imaging sensor can be used for fluorescence imaging. In fluorescence imaging, light having a wavelength that matches the fluorescence excitation wavelength band of a fluorophore is emitted towards an object observed by the microscope. The fluorophore, which may be a chemical substance injected into the patient's blood vessels or tissue, is excited by light in the fluorescence excitation wavelength band and emits light in a fluorescence emission wavelength band, which is sensed by at least one optical imaging sensor used for fluorescence imaging. In many cases, a surgical microscope system supports a limited selection of fluorophores, and one or more optical imaging sensors are used for fluorescence imaging, tuned to the fluorescence emission wavelengths of the selected fluorophore. During surgery, the reflectance image (showing the surgical site in "natural" colors) and the fluorescence image (as a pseudo-color overlay) can be combined into a further composite image that can be viewed by the surgeon. Thus, one or more processors (e.g., a system) can be configured to form a composite image in a first operating mode (i.e., an operating mode used to form a composite image based on both high resolution images and large depth of field images) and to form a second composite image based on reflectance imaging and fluorescence imaging in a second operating mode (a combined reflectance imaging and fluorescence imaging mode).

[0047] During reflection imaging, the second optical imaging sensor may not be used for other purposes. In the proposed concept, an optical imaging sensor that is normally used for fluorescence imaging can be repurposed for recording light of a second wavelength spectrum. The processing system can therefore be configured to perform reflection imaging using the first optical imaging sensor and to perform fluorescence imaging in a second operation mode using the second optical imaging sensor. In other words, the first optical imaging sensor can be generally used for reflection imaging in a microscope system and the second optical imaging sensor can be generally used for fluorescence imaging in a microscope system. As a result, the second optical imaging sensor can be configured to sense, e.g. only to sense, a limited spectrum (i.e. a fluorescence emission wavelength band), e.g. by a bandpass filter arranged between the second optical imaging sensor and the object. The second wavelength spectrum can therefore include, i.e. can be based on or consist of, a fluorescence emission wavelength band of one or more fluorophores. The first wavelength spectrum can therefore be suitable for performing reflection imaging and the second wavelength spectrum is suitable for performing fluorescence imaging. In this case, the first wavelength spectrum can exclude the fluorescent emission bands contained in the second wavelength spectrum, or can include fluorescent emission bands whose effects have been removed from the first imaging sensor data by one or more processors (i.e., systems).

[0048] However, the proposed concept is also applicable to other scenarios, for example, when both the first and second optical imaging sensors are suitable and / or configured to perform reflection imaging, or when both the first and second optical imaging sensors are suitable and / or configured to perform fluorescence imaging. Thus, the first and second wavelength spectra may be suitable for performing reflection imaging. Alternatively, the first and second wavelength spectra may be suitable for performing fluorescence imaging. For example, a wavelength spectrum may be suitable for performing reflection imaging if it covers at least 50% (or at least 60%, or at least 70%, or at least 80%) of the visible light spectrum. A wavelength spectrum may be suitable for performing fluorescence imaging if it covers one or more fluorescence emission wavelength bands of one or more fluorophores.

[0049] The proposed concept is based on combining the first and second imaging sensor data in a way that uses both the higher resolution of the first imaging sensor data and the larger depth of field of the second imaging sensor data. In the following, the influence of the first and second wavelength bands and thus the effective numerical aperture for the first and second imaging sensor data is briefly described.

[0050] The resolving power of a microscope in the focal plane of the microscope objective is limited by the diffraction of light, which is itself determined by the numerical aperture of the microscope. In particular, the maximum resolving power and therefore the resolution are proportional to the numerical aperture. In practice, high resolution depends on a high numerical aperture, which can be obtained by increasing the size of the iris opening.

[0051] However, the numerical aperture has a large effect on the microscope's depth of field, i.e., the clarity of parts of the imaged object that lie outside the focal plane: a large numerical aperture results in a shallow depth of field, and therefore more of the object will appear out of focus due to variations in the distance between the imaged object and the microscope's objective lens and sensor.

[0052] In the proposed concept, two imaging sensor data sets are formed, one with a larger numerical aperture resulting in an increased resolution (and a smaller depth of field) (i.e., the first imaging sensor data) and the other with a smaller numerical aperture resulting in an increased depth of field (and a lower resolution) (i.e., the second imaging sensor data). Both imaging data sets can have the same field of view. One or more processors (e.g., systems) are configured to combine the two imaging sensor data sets to form a composite image having both an increased resolution (in the image parts that are in focus in the first imaging sensor data) and an increased depth of field based on the second imaging sensor data (to increase the image parts that appear sharp). In the following, two techniques for forming the composite image are presented. Both techniques have in common that they both operate on the basis of preferably synchronized imaging sensor data. In other words, one or more processors (e.g., systems) can be configured to form a composite image based on frames of the first imaging sensor data and frames of the second imaging sensor data that are taken at the same time, i.e., time-synchronized. In particular, the first imaging sensor data and the second imaging sensor data may not be captured consecutively (e.g., based on imaging sensor data recorded using two different numerical aperture settings of an iris capable of rapid switching between the two numerical aperture settings).

[0053] In the following, a first approach is presented. In this approach, two imaging sensor data sets are analyzed and a portion that appears sharper in one of the imaging sensor data sets is determined. In other words, one or more processors (e.g., a system) can be configured to determine one or more portions of the first imaging sensor data that have a higher image sharpness than one or more corresponding portions of the second imaging sensor data. In general, if a portion of the first imaging sensor data is in focus, i.e., if the portion is in the depth of field around the focal plane of the first optical imaging sensor, a higher image sharpness based on a high resolution can be provided. However, since the depth of field of the first optical imaging sensor is shallower than the depth of field of the second optical imaging sensor, many portions of the first imaging sensor data may be out of focus, reducing the sharpness of each portion. Therefore, the portion that appears sharper in the first imaging sensor data, i.e., the portion that is in focus in the first imaging sensor data, may be included in the composite image, and the remaining portion may be obtained from the second imaging sensor data. In other words, the one or more processors (e.g., systems) can be configured to form a composite image using one or more portions of the first imaging sensor data having a higher image definition, while basing the remaining portion of the composite image on the second imaging sensor data. In practice, the one or more processors (e.g., systems) can be configured to form a composite image using one or more portions of the first imaging sensor data having a higher image definition in the first imaging sensor data and one or more other portions of the second imaging sensor data having a higher definition in the second imaging sensor data.

[0054] To compare the sharpness of the corresponding portions of the first imaging sensor data and the second imaging sensor data, the first imaging sensor data and the second imaging sensor data can be subdivided into a grid of pixel blocks, where the corresponding pixel blocks of the first imaging sensor data and the second imaging sensor data are based on the corresponding portions of the field of view. Thus, the one or more processors (e.g., a system) can be configured to subdivide the first imaging sensor data and the second imaging sensor data into a two-dimensional grid of pixel blocks each including a plurality of pixels. For example, the one or more processors (e.g., a system) can be configured to subdivide the first imaging sensor data and the second imaging sensor data into a grid of m×n blocks of o×p pixels, where m, n, o, p are positive integers equal to or greater than 2. The one or more processors (e.g., a system) can be configured to determine on a pixel block basis one or more portions of the first imaging sensor data that have a higher image sharpness than the corresponding one or more portions of the second imaging sensor data. In other words, for each block of pixels in the grid, one or more processors (e.g., a system) can be configured to compare the sharpness of the block of the first imaging sensor data with the sharpness of a corresponding block of the second imaging sensor data.

[0055] In the present disclosure, the first and second imaging sensor data portions are compared by evaluating image sharpness. The sharpness of the different portions can be determined based on the contrast of the respective images and / or based on the proportion of high spatial frequencies in the respective images. For example, one or more processors (e.g., a system) can be configured to determine one or more portions of the first imaging sensor data that have higher image sharpness than one or more corresponding portions of the second imaging sensor data based on the contrast and / or based on the presence of spatial frequencies above a predetermined spatial frequency threshold in each one or more portions (e.g., pixel blocks).

[0056] For example, the system can be configured to determine the contrast of each imaging sensor data portion (e.g., pixel block), e.g., by determining the ratio of the standard deviation to the mean value of the pixels of the image, or by performing a kernel-based comparison between each pixel and its neighboring pixels (i.e., adjacent pixels). The sharper an image portion appears, generally, the higher the contrast of the image.

[0057] The system can also be configured to determine the spatial frequency distribution of each portion (e.g., pixel block) of the imaging sensor data, for example, by performing a 2D Fourier transform of the two imaging sensor data sets. The higher the proportion of high spatial frequencies in each portion of the imaging sensor data, the more fine-grained structures are visible in that portion of the imaging sensor data, which is the case when each portion of the image containing such fine-grained structures is perceived as sharp. By determining the integral of the portion of the spatial frequency distribution above a predetermined frequency, a quantitative measure that can be used to compare the presence of fine-grained structures across the imaging sensor data, for example a quantitative measure that can be used to compare the sharpness of two corresponding portions of the field of view of the first imaging sensor data and the second imaging sensor data, can be determined.

[0058] In a second approach, which can be combined with the first approach, two imaging sensor data sets are merged. In particular, spatial features of the second imaging sensor data are merged with spatial features and color information of the first imaging sensor data. In other words, one or more processors (e.g., systems) can be configured to extract spatial features of the second imaging sensor data and merge the spatial features from the second imaging sensor data with the spatial features and color information of the first imaging sensor data to form a composite image. The color information of the second imaging sensor data can be excluded, especially if the second optical imaging sensor is typically used for fluorescence imaging and thus contains limited color information. To combine the second approach with the first approach, the merging can be limited to parts (i.e., pixel blocks) of the second imaging sensor data that have higher definition than the corresponding parts of the first imaging sensor data. Alternatively, the merging can be performed over the entire field of view.

[0059] To perform such merging, the first imaging sensor data and the second imaging sensor data can be converted into different color representations known as HSB (hue, saturation, brightness), HSV (hue, saturation, value) or HSL (hue, saturation, lightness). Generally, HSB and HSV are two terms used for the same representation, and HSL is similar, where the maximum lightness in HSL is pure white, in contrast to HSB or HSV, where the maximum value or maximum brightness increases the color intensity. However, for the purpose of merging two imaging sensor data sets, both HSB or HSV and HSL can be used. In the following, the term "hue, saturation, brightness" is used to mean any of HSB, HSV and HSL. One or more processors (e.g., a system) can be configured to process the first imaging sensor data and the second imaging sensor data in a hue, saturation and brightness color representation (e.g., into one of HSB or HSV or HSL). The one or more processors (e.g., systems) can be configured to merge the saturation and brightness or value or lightness components of the second imaging sensor data with the hue, saturation and brightness or value or lightness components of the first imaging sensor data. In this context, merging can be performed by determining the average between the two components to be merged (per pixel) or by using the maximum or minimum value of the two components to be merged (per pixel). After merging, the imaging sensor data can be converted back to the original color representation. For example, the first imaging sensor data and the second imaging sensor data can be obtained as RGB (red, green, blue) imaging sensor data from an optical imaging sensor, converted to an HSB color representation or an HSV color representation or an HSL color representation for merging, and converted back to an RGB color representation to obtain a composite image.

[0060] The composite image may be viewed by a user of the microscope system, e.g., a surgeon. For that purpose, the composite image may be provided to a display, e.g., an auxiliary display 150b or an eyepiece display 150a of the microscope system. The processing system is therefore configured to form a display signal based on the composite image for the display device 150 of the microscope system. For example, the display signal may be a signal for driving (e.g., controlling) the display device 150. For example, the display signal may include video data and / or control instructions for driving the display. For example, the display signal may be provided via one of one or more interfaces 112 of the system. The system 110 may therefore include a video interface 112 suitable for providing a video signal to the display 150 of the microscope system 100.

[0061] In the proposed microscope system, the first imaging sensor data and the second imaging sensor data are formed using optical imaging sensors. Thus, the first imaging sensor 122 and the second optical imaging sensor 124 are configured to form the first imaging sensor data and the second imaging sensor data, respectively. For example, the optical imaging sensor 122;124 of the microscope 120 may include or be an APS (Active Pixel Sensor)-based or CCD (Charge-Coupled-Device)-based imaging sensor 122;124. For example, in an APS-based imaging sensor, a photodetector and an active amplifier of the pixel are used to record the light at each pixel. APS-based imaging sensors are often based on CMOS (Complementary Metal Oxide Semiconductor) technology or S-CMOS (Scientific CMOS) technology. In a CCD-based imaging sensor, the incident photons are converted into electronic charges at the semiconductor-oxide interface, which are then transferred between capacitive bins in the imaging sensor by the imaging sensor circuitry for imaging. Processing system 110 is configurable to obtain (i.e., receive or read) the respective imaging sensor data from the respective optical imaging sensors. The respective imaging sensor data may be obtained by receiving the imaging sensor data from the respective optical imaging sensors (e.g., via interface 112), or by reading the respective imaging sensor data from a memory of the respective optical imaging sensor (e.g., via interface 112), or by reading the imaging sensor data from storage device 116 of system 110 after the imaging sensor data has been written to the storage device 116, for example, by the respective optical imaging sensor or another system or processor.As shown in Figures 1a, 1c, 3 and 4, the microscope may include a beam splitter that guides light reflected by or emitted from the object toward a first optical imaging sensor and a second optical imaging sensor.

[0062] The one or more interfaces 112 of the system 110 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values ​​according to a specified code, within a module, between modules or between modules of different entities respectively. For example, the one or more interfaces 112 may comprise an interface circuit configured to receive and / or transmit information. The one or more processors 114 of the system 110 may be implemented using one or more processing units, one or more processing devices, any processing means capable of operating with correspondingly adapted software, such as a processor, a computer or a programmable hardware component. In other words, the functions described for the one or more processors 114 may also be implemented in software, which may then be executed in one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. The one or more storage devices 116 of the system 110 may include at least one element of the following group: a computer-readable storage medium, such as a magnetic or optical storage medium, e.g., a hard disk drive, a flash memory, a floppy disk, a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electronically erasable programmable read-only memory (EEPROM), or network storage.

[0063] Further details and aspects of the (surgical) microscope system are referred to in relation to the proposed concept or one or more of the embodiments described above or below (e.g. in Figures 2-4). The (surgical) microscope system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more of the embodiments described above or below.

[0064] 2 illustrates a flow chart of a method for a microscope system. The method includes forming 210, with a multispectral iris, an aperture having a first numerical aperture for light having a first wavelength spectrum and an aperture having a second numerical aperture for light having a second wavelength spectrum, the second numerical aperture being different from the first numerical aperture. The method includes forming 220, with a first optical imaging sensor, first imaging sensor data based on the light having the first wavelength spectrum. The method includes forming 230, with a second optical imaging sensor, second imaging sensor data based on the light having the second wavelength spectrum. The method includes forming 240 a composite image based on the first imaging sensor data and the second imaging sensor data.

[0065] For example, the method can be implemented by a microscope system as introduced in relation to one of Figures 1a to 1f. Features introduced in relation to the (surgical) microscope system of Figures 1a to 1f can likewise be included in the corresponding method.

[0066] Further details and aspects of the method for the microscope system are mentioned in relation to the proposed concept or one or more of the embodiments described above or below (e.g., in Figures 1a-1c, 3-4). The method for the microscope system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more of the embodiments described above or below.

[0067] Various embodiments relate to a concept for a digital version of a "fusion optics", i.e. the simultaneous use of different numerical aperture settings based on multispectral imaging, thereby providing a concept for a digital fusion optics.

[0068] In the proposed concept, the operation method of the "fusion optics" is translated into digital imaging to form a two-dimensional image (not a stereoscopic image) with both increased resolution and increased depth of field. In the proposed concept, two images (i.e., the first imaging sensor data and the second imaging sensor data) are digitally fused into one image without relying on the visual perception of the human brain. Instead of using at least two imaging sensors with two different irises, which increases cost, complexity and size, the proposed concept can be realized by combining existing reflective and fluorescent sensors with a multispectral iris.

[0069] In particular, multispectral imaging systems, such as systems that provide a pseudo-color overlay based on fluorescence emission, use multiple imaging sensors that can be reused for fusion optical imaging. In systems that provide a pseudo-color fluorescence overlay, each imaging channel (right / left) uses two sensors, one sensor used for white light and the other sensor used for fluorescence imaging. In various embodiments of the proposed concept, the multispectral capabilities of such systems or any other multispectral system can be used to form a fusion optical image using a multispectral iris. As outlined above, a multispectral iris is a single element that acts as a closed iris for part of the spectrum and an open iris for the rest of the spectrum. This allows the two sensors of the multispectral imaging system to capture two images, each with a different iris setting. In effect, the proposed concept combines the use of a multispectral iris with a multispectral sensor to achieve a fusion optical system, combining resolution and depth of focus.

[0070] In some examples, the metal iris can be replaced with a filter that allows some spectral bands (e.g., a first wavelength spectrum) to pass in their entirety and other bands (e.g., a second wavelength spectrum) to have a much smaller pass area. The two images can then be fused to form a composite image.

[0071] A schematic diagram of the digital fusion optical system concept is shown in Fig. 3. In Fig. 3, a microscope system 300 and a sample 10 are shown. The microscope of the microscope system 300 includes a first optical imaging sensor 122 (forming a first imaging sensor data having a high resolution), a second optical imaging sensor 124 (forming a second imaging sensor data having a larger depth of field), a multispectral iris 130, and a beam splitter 330. The multispectral iris includes an effective iris 130a with a first numerical aperture in a first wavelength band sensed by the first optical imaging sensor 122 and a resulting wider aperture (and a corresponding shallower depth of field 310), and an effective iris 130b with a second numerical aperture in a second wavelength band sensed by the second optical imaging sensor 124 and a resulting narrower aperture (and a corresponding deeper depth of field 320). First imaging sensor data 340 (i.e., a higher resolution image with a shallower depth of field 310) formed by a first optical imaging sensor is combined with second imaging sensor data 350 (i.e., an image with a lower resolution and a deeper depth of field 320) of a second optical imaging sensor to form a composite image 360 ​​using real-time image fusion. The composite image (i.e., the fused image) has an increased depth of field and increased resolution.

[0072] In some embodiments, the spectral sensitivity of both sensors can include sufficient color rendering bands so that they can be calibrated to form a suitable color image. In some embodiments, where one sensor is tuned for white light reflectance and the other sensor is tuned for fluorescent imaging of medically approved fluorophores (5-ALA, fluorescein, and ICG), the color content of the FL sensor may not be sufficient to form a suitable color image. In this case, only the spatial features of the image formed by the fluorescent sensor can be extracted without the full color content, and this information can be used to improve the spatial features of the other sensor that captures color.

[0073] The fusion of the two images can be done digitally. In a simple embodiment, the local contrast can be measured in both images, for example by measuring high spatial frequencies, to identify the best focused areas of each image. Then, for each image region, a selection can be made to take the region from the image with the higher contrast.

[0074] Further details and aspects of the digital fusion concept are mentioned in relation to the proposed concept or one or more of the embodiments described above or below (e.g., in Figs. 1a-2, 4). The digital fusion concept may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more of the embodiments described above or below.

[0075] As used in this specification, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0076] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0077] Some embodiments relate to a microscope including a system as described in relation to one or more of the figures of Figures 1 to 3. Alternatively, the microscope may be part of a system as described in relation to one or more of the figures of Figures 1 to 3 or may be connected to a system as described in relation to one or more of the figures of Figures 1 to 3. Figure 4 shows a schematic diagram of a system 400 configured to perform the methods described herein. The system 400 includes a microscope 410 and a computer system 420. The microscope 410 is configured to take images and is connected to the computer system 420. The computer system 420 is configured to perform at least some of the methods described herein. The computer system 420 may be configured to perform machine learning algorithms. The computer system 420 and the microscope 410 may be separate entities, but may be integrated in one common housing. The computer system 420 may be part of a central processing system of the microscope 410 and / or the computer system 420 may be part of a subordinate part of the microscope 410, such as a sensor, actor, camera or lighting unit of the microscope 410.

[0078] The computer system 420 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, such as local clients and / or one or more remote server farms and / or data centers). The computer system 420 may include any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors, which may be of any type. As used herein, a processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit, for example, of a microscope or a microscope component (e.g., a camera). Other types of circuits that may be included in computer system 420 may be custom circuits, application specific integrated circuits (ASICs), such as one or more circuits (such as communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 420 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives handling removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), and the like.Computer system 420 may also include a display device, one or more speakers and a keyboard and / or controller which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that enables a user of the system to input information to and receive information from computer system 420.

[0079] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the crucial steps may be performed by such an apparatus.

[0080] Depending on certain implementation requirements, the embodiments of the present invention can be implemented in hardware or software. The implementation can be performed by a non-transitory recording medium, such as a digital recording medium, for example a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM and EPROM, an EEPROM or a FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to execute the respective methods. Thus, the digital recording medium can be computer readable.

[0081] Some embodiments of the present invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause any of the methods described herein to be performed.

[0082] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code which is operable to perform any of the methods when the computer program product is run on a computer, the program code may for example be stored on a machine readable carrier.

[0083] Another embodiment comprises the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0084] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein, when the computer program runs on a computer.

[0085] Therefore, another embodiment of the present invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein, including a processor and a recording medium.

[0086] A further embodiment of the invention is therefore a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or the sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0087] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0088] Another embodiment comprises a computer having the computer program installed thereon for performing any of the methods described herein.

[0089] Another embodiment of the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server to transfer the computer program to the receiver.

[0090] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus. [Explanation of symbols]

[0091] 10. Object 100 (Surgery) Microscope System 110 System 105 Stand 112 One or more interfaces 114 One or more processors 116 One or more storage devices 120 Microscope 122 First Optical Imaging Sensor 124 Second Optical Imaging Sensor 130 Multispectral Iris, Iris 132 Bandpass Filtering Components 134 Aperture 136 Filter Wheel 138 Components 140 Lighting System 150a eyepiece display 150b Auxiliary Display 160 Arm 210 Forming an aperture having a first numerical aperture and an aperture having a second numerical aperture 220 Forming first imaging sensor data 230 Forming second imaging sensor data 240 Forming a composite image 300 Microscope System 310,320 depth of field 330 Beam splitter 340 First Imaging Sensor Data 350 Second Imaging Sensor Data 360 Composite Image 400 Systems 410 Microscope 420 Computer Systems

Claims

1. A microscope system (100) comprising a microscope (120) and one or more processors (114), The microscope (120) a first optical imaging sensor (122) that forms first imaging sensor data based on light having a first wavelength spectrum; a second optical imaging sensor (124) that forms second imaging sensor data based on light having a second wavelength spectrum; a multispectral iris (130) configured to form an aperture having a first numerical aperture for light having the first wavelength spectrum and an aperture having a second numerical aperture for light having the second wavelength spectrum, the second numerical aperture being different from the first numerical aperture; Including, the one or more processors (114) are configured to form a composite image based on the first imaging sensor data and the second imaging sensor data. A microscope system (100).

2. the first wavelength spectrum is suitable for performing reflectance imaging and the second wavelength spectrum is suitable for performing fluorescence imaging; or the first wavelength spectrum and the second wavelength spectrum are suitable for performing reflectance imaging; or the first wavelength spectrum and the second wavelength spectrum are suitable for performing fluorescence imaging; The microscope system according to claim 1 .

3. the first numerical aperture is greater than the second numerical aperture; 3. The microscope system according to claim 1.

4. the one or more processors are configured to determine one or more portions of the first imaging sensor data having a higher image definition than one or more corresponding portions of the second imaging sensor data, and form a composite image using the one or more portions of the first imaging sensor data having the higher image definition, whereby a remaining portion of the composite image is based on the second imaging sensor data. The microscope system according to claim 3 .

5. the one or more processors are configured to determine one or more portions of the first imaging sensor data having greater image sharpness than one or more corresponding portions of the second imaging sensor data based on contrast and / or based on the presence of spatial frequencies above a predetermined spatial frequency threshold in the respective one or more portions.

5. The microscope system according to claim 4.

6. the one or more processors are configured to subdivide the first imaging sensor data and the second imaging sensor data into a two-dimensional grid of pixel blocks, each including a plurality of pixels, and determine one or more portions of the first imaging sensor data having a higher image definition than a corresponding one or more corresponding portions of the second imaging sensor data on a pixel block basis.

5. The microscope system according to claim 3.

7. the one or more processors are configured to extract spatial features of the second imaging sensor data and merge the spatial features from the second imaging sensor data with the spatial features and color information of the first imaging sensor data to form a composite image.

7. The microscope system according to claim 1.

8. the one or more processors are configured to process the first imaging sensor data and the second imaging sensor data in a color representation of hue, saturation and brightness and to merge saturation and brightness components of the second imaging sensor data with the hue, saturation and brightness components of the first imaging sensor data. The microscope system according to claim 7.

9. the microscope system comprises an illumination system (140) configured to illuminate an object to be imaged using the microscope system; an emission wavelength spectrum of the illumination system comprising the first wavelength spectrum and the second wavelength spectrum; 9. The microscope system according to claim 1.

10. the first wavelength spectrum includes a first wavelength band that is excluded from the second wavelength spectrum; the second wavelength spectrum includes a second wavelength band that is excluded from the first wavelength spectrum; an emission spectrum of the illumination system comprising the first wavelength band and the second wavelength band; The microscope system according to claim 9.

11. the multi-spectral iris includes at least one bandpass filtering component configured to reject the second wavelength spectrum. The microscope system according to any one of claims 1 to 10.

12. the multi-spectral iris includes a plurality of bandpass filtering components configured to reject the second wavelength spectrum; the microscope includes a motor for adjusting the relative positions of the bandpass filtering components with respect to one another; the one or more processors are configured to control the motors of the microscope system to adjust the relative positions of the plurality of bandpass filtering components with respect to one another, thereby adjusting the second numerical aperture. The microscope system according to claim 11.

13. the plurality of bandpass filtering components form an aperture; or the plurality of band-pass filtering components correspond to two L-shaped band-pass filtering components; The microscope system according to claim 12.

14. the multispectral iris comprises a plurality of second components configured to reject the first wavelength spectrum and the second wavelength spectrum; the microscope includes a second motor that adjusts the relative positions of the second components with respect to one another; the one or more processors are configured to control the second motor to adjust relative positions of the plurality of second components with respect to one another, thereby adjusting the first numerical aperture. The microscope system according to any one of claims 11 to 13.

15. the one or more processors are configured to control at least one motor of the microscope system to cause the motor to adjust at least one of the first numerical aperture and the second numerical aperture.

15. The microscope system according to claim 1.

16. The microscope system includes a filter wheel having a plurality of multispectral irises having a fixed numerical aperture. A microscope system according to any one of claims 1 to 11 or claim 15.

17. 1. A method for a microscope system, the method comprising: forming (210) an aperture with a first numerical aperture for light having a first wavelength spectrum and an aperture with a second numerical aperture for light having a second wavelength spectrum, the second numerical aperture being different from the first numerical aperture, with a multispectral iris; forming (220) first imaging sensor data based on light having the first wavelength spectrum with a first optical imaging sensor; forming (230) second imaging sensor data based on light having the second wavelength spectrum with a second optical imaging sensor; forming (240) a composite image based on the first imaging sensor data and the second imaging sensor data; The method includes:

Images