System, method and computer program for a microscope system and for determining a transformation function - Patents.com

Abstract

The present example relates to a system, method, and computer program for determining a conversion function for a microscope system, and a corresponding microscope system. The system for a microscope system includes one or more processors and one or more storage devices. The system is configured to acquire first image sensor data from a first image sensor of a microscope of the microscope system and second image sensor data from a second image sensor of the microscope. The first image sensor data includes sensor data based on light sensed in a first plurality of mutually separated wavelength bands. The second image sensor data includes sensor data based on light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The system is configured to generate a composite color image based on the first image sensor data and the second image sensor data. The composite color image is based on multiple color channels. The composite color image is generated using a transformation function that defines a transformation performed between the imaging sensor data and the composite color image, whereby the composite color image is generated using sensor data based on light sensed in each wavelength band of the first and second plurality of mutually separated wavelength bands.

Description

[Technical field]

[0001] The present example relates to a system, method and computer program for a microscope system and for determining a transformation function, and a corresponding microscope system. [Background technology]

[0002] Surgeons often use tissue (e.g., brain tissue) color to distinguish suspicious tissue (e.g., lesions). However, subtle differences in tissue color are often only visible to surgeons with years of experience and a trained eye, which can be difficult to master. Multispectral reflectance imaging can capture very small or even invisible color differences using measurements of very small spectral differences. However, known concepts of multispectral imaging may require additional hardware in addition to the multiple sensors of modern microscopes. Summary of the Invention [Problem to be solved by the invention]

[0003] There may be a need for improved concepts for providing color images in which subtle differences between different types of tissue are visible. [Means for solving the problem]

[0004] This need is solved by the subject matter of the independent claims.

[0005] An embodiment of the present disclosure is based on the discovery that multispectral imaging can be performed by combining image sensor data from an image sensor used primarily to perform fluorescence imaging and image sensor data from an image sensor used primarily for reflection imaging. Accordingly, an embodiment of the present disclosure provides a system for a microscope system. The system includes one or more processors and one or more storage devices. The system is configured to acquire first image sensor data from a first image sensor of a microscope of the microscope system and second image sensor data from a second image sensor of the microscope. The first image sensor data includes sensor data based on light sensed in a first plurality of mutually separated wavelength bands. The second image sensor data includes sensor data based on light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The system is configured to generate a composite color image based on the first image sensor data and based on the second image sensor data. The composite color image is based on a plurality of color channels. The composite color image is generated using a transformation function that defines a transformation performed between the imaging sensor data and the composite color image, whereby the composite color image is generated using sensor data based on light sensed in each wavelength band of the first and second multiple mutually separated wavelength bands.

[0006] An embodiment of the present disclosure further provides a microscope system including a system having a first and second imaging sensor and a microscope, wherein one of the first and second imaging sensors is an imaging sensor adapted to provide a fluorescence imaging function of the microscope system.

[0007] Through the use of two sets of imaging sensor data, a more accurate transformation function can be used to construct a color image, which can therefore show subtle differences between different types of tissue. At the same time, reusing the fluorescence imaging sensor for reflectance imaging can enable the use of an approach that does not create the need to include additional sensors in the microscope system.

[0008] In various embodiments, the conversion is based on a set of conversion coefficients, each of which defines a conversion performed between image sensor data based on light sensed in a wavelength band and a color channel of the composite color image, for example, the set of conversion coefficients provides a one-to-one conversion between the intensity of light measured in one of the wavelength bands and a color channel.

[0009] For example, the set of conversion coefficients may include one conversion coefficient for each combination of wavelength band and color channel. Thus, a one-to-one conversion may be applied, for example, using matrix multiplication. In other words, the set of conversion coefficients may provide a conversion between the image sensor data for each wavelength band of the first and second plurality of mutually separated wavelength bands and each of the color channels of the composite color image.

[0010] For example, the composite color image may include three color channels (e.g., red, green, and blue). Each of these color channels may be generated based on a transformation of the image sensor data of each wavelength band of the first and second plurality of mutually separated wavelength bands. The transformation may be performed using a conversion function. By using the image sensor data of each wavelength band of each of these color channels, even subtle color differences may be included in the composite color image.

[0011] In various embodiments, the transformation function may be implemented by a transformation matrix. The system may be configured to transform the image sensor data for each wavelength band of the first and second plurality of mutually separated wavelength bands using the transformation matrix. The transformation matrix provides a computationally efficient implementation of the transformation function.

[0012] In various embodiments, the system is configured to provide a display signal to a display of the microscope system using an interface of the system to cause the display to display the composite color image, so that the composite color image may be shown to a user of the microscope system, for example, a surgeon.

[0013] An embodiment of the present disclosure further provides a system for determining a conversion function. The system includes one or more processors and one or more storage devices. The system is configured to acquire first image sensor data of a reference object from a first image sensor of a microscope and second image sensor data of the reference object from a second image sensor of the microscope. The first image sensor data includes sensor data based on light sensed in a first plurality of mutually separated wavelength bands. The second image sensor data includes sensor data based on light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The system is configured to acquire a composite reference image of the reference object. The composite reference image includes a plurality of color channels. The system is configured to determine a conversion function by determining a set of conversion coefficients that provide an approximate conversion between the image sensor data of each wavelength band of the first and second plurality of mutually separated wavelength bands and each color channel of the composite reference image. The conversion function is based on the set of conversion coefficients. Through the use of two sets of imaging sensor data, a more accurate transformation function can be used to construct a color image, which can therefore show subtle differences between different types of tissue.

[0014] In some embodiments, the system is configured to identify a set of transform coefficients that results in a lower mismatch between the synthetic reference image and the transformed image generated based on the set of transform coefficients than at least one other set of transform coefficients. In other words, the system may be configured to iteratively search for transform coefficients that reduce the mismatch between the synthetic reference image and the transformed image.

[0015] For example, the composite reference image of the reference image may define a plurality of colors of a plurality of portions of the reference object. The plurality of colors may include a first predetermined subset color and a second predetermined subset color. The system may be configured to identify a set of conversion coefficients in which a mismatch between the composite reference image and the conversion image generated based on the set of conversion coefficients is lower than at least one other set of conversion coefficients for the first predetermined subset color. In other words, conversion coefficients that reduce the mismatch for the first subset color that may be of particular interest may be identified. For example, the first predetermined subset color may be a color that exists as a color of organic tissue in a surgical setting. An enhanced sharpness of the first subset color over other colors may be more beneficial.

[0016] In various embodiments, the system is configured to identify a set of transform coefficients that, compared to at least one other set of transform coefficients, reduces a mismatch value representing a mismatch between the composite reference images. The mismatch value may be calculated for a plurality of colors. A mismatch for a color of a predetermined first subset color may have a higher influence on the mismatch value than a mismatch for a color of a second subset color. Thus, the first subset color may receive a higher weighting than other colors in determining the transform coefficients.

[0017] An embodiment of the present disclosure further provides a method for a microscope system. The method includes acquiring first image sensor data from a first image sensor of a microscope of a microscope system and second image sensor data from a second image sensor of the microscope. The first image sensor data includes sensor data based on light sensed in a first plurality of mutually separated wavelength bands. The second image sensor data includes sensor data based on light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The method includes generating a composite color image based on the first image sensor data and based on the second image sensor data. The composite color image is based on a plurality of color channels. The composite color image is generated using a conversion function that defines a conversion performed between the image sensor data and the composite color image, whereby the composite color image is generated using sensor data based on light sensed in each wavelength band of the first and second plurality of mutually separated wavelength bands.

[0018] An embodiment of the present disclosure further provides a method for determining a conversion function. The method includes acquiring first image sensor data of a reference object from a first image sensor of a microscope and second image sensor data of the reference object from a second image sensor of the microscope. The first image sensor data includes sensor data based on light sensed in a first plurality of mutually separated wavelength bands. The second image sensor data includes sensor data based on light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The method includes acquiring a composite reference image of the reference object. The composite reference image includes a plurality of color channels. The method includes determining a conversion function by determining a set of conversion coefficients that provide an approximate conversion between the image sensor data of each wavelength band of the first and second plurality of mutually separated wavelength bands and each color channel of the composite reference image. The conversion function is based on the set of conversion coefficients.

[0019] An embodiment of the present disclosure further provides a computer program having a program code for performing at least one method, when the computer program is run on a processor.

[0020] Some examples of apparatus and / or methods are now described, by way of example only, and with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0021] [Figure 1a] FIG. 1 is a schematic diagram showing a system for a microscope system. [Figure 1b] FIG. 1 is a schematic diagram showing a microscope system including the present system. [Diagram 2] 1 is a flow chart illustrating a method for a microscope system. [Diagram 3] FIG. 1 is a schematic diagram illustrating a system for determining a transformation function. [Figure 4]1 is a flow chart illustrating a method for determining a transformation function. [Figure 5a] FIG. 2 is a schematic diagram illustrating different colors present in an image frame. [Figure 5b] FIG. 2 is a schematic diagram illustrating the intensity of different colors perceived in different frequency bands. [Figure 5c] FIG. 2 is a schematic diagram illustrating the intensity of different colors perceived in different frequency bands. [Figure 5d] FIG. 2 illustrates an example transformation matrix. [Figure 6] FIG. 1 is a schematic diagram showing a microscope and illumination system. [Figure 7a] FIG. 2 is a schematic diagram illustrating an exemplary color table. [Figure 7b] FIG. 2 illustrates an exemplary system of equations. [Figure 8] FIG. 1 is a schematic diagram showing a microscope system including a microscope and a computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Various examples will now be described more fully with reference to the accompanying drawings, in which several examples are shown and in which the thicknesses of lines, layers and / or regions may be exaggerated for clarity.

[0023] 1a and 1b show schematic diagrams of a system 110 for and including a microscope system 100. The system 110 includes one or more processors 114 and one or more storage devices 116. Optionally, the system further includes an interface 112. The one or more processors are coupled to the interface and coupled to the one or more storage devices. In general, the functionality of the system is provided by the one or more processors, e.g., in association with an optional interface or one or more storage devices. For example, the system may be configured to obtain image sensor data via the interface and / or store conversion coefficients of a conversion function using one or more storage devices.

[0024] The system is configured to acquire first image sensor data from a first image sensor 122 of a microscope 120 of a microscope system and second image sensor data from a second image sensor 124 of the microscope. The first image sensor data includes sensor data based on light sensed in a first plurality of mutually separated wavelength bands. The second image sensor data includes sensor data based on light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The system is configured to generate a composite color image based on the first image sensor data and based on the second image sensor data. The composite color image is based on a plurality of color channels. The composite color image is generated using a conversion function that defines a conversion performed between the image sensor data and the composite color image, whereby the composite color image is generated using sensor data based on light sensed in each wavelength band of the first and second plurality of mutually separated wavelength bands.

[0025] FIG. 1b shows a block diagram of a microscope system including a microscope 120 and a system 110. The microscope system shown in FIG. 1b is a surgical microscope system that may be used at a surgical site by a surgeon. This surgical microscope system shown in FIG. 1b includes several optional components, such as a base unit 105 (including the system 110) with a (rotating) stand, an auxiliary display 140a, an illumination system 130, an arm 160 (robotic or manual) that holds the microscope 120 in place and is coupled to the base unit 105 and the microscope 120, and a steering handle 150 attached to the microscope 120. In addition to the first and second imaging sensors 122;124, the microscope 120 may include an optional eyepiece display 140b and an optional auxiliary display. In the context of this application, the term "(surgical) microscope system" is used to cover parts of the system that are not part of the actual microscope (including the optical components) but are used in conjunction with the microscope, such as the display and illumination system. One of the first and second imaging sensors is an imaging sensor adapted to provide a fluorescence imaging function of the microscope system.

[0026] Various embodiments of the present disclosure relate to systems, methods and computer programs for a microscope system. In general, a microscope is an optical instrument suitable for the inspection of objects that are too small to be inspected by the naked eye (alone). For example, a microscope can provide optical magnification of an object. In modern microscopes, optical magnification is often provided for a camera or an imaging sensor, such as the first and second imaging sensors 122; 124 of the microscope 120 of FIG. 1a. The microscope 120 can further include one or more optical magnification components used to expand the field of view on a sample.

[0027] There are various types of microscopes. When the microscope system is used in the medical or biological fields, the object viewed through the microscope may be, for example, a sample of organic tissue placed in a petri dish or present in a part of a patient's body. For example, the microscope system 100 may be a microscope system for use in a laboratory, such as a microscope used to inspect a sample of organic tissue in a petri dish. Alternatively, the microscope 120 may be part of a surgical microscope system 100, such as a microscope used during a surgical procedure. Such a system is shown, for example, in FIG. 1b. Although the embodiments are described in relation to a microscope system, they may also be applied in a more general manner to any optical device.

[0028] The system is configured to acquire first and second imaging sensor data from the first and second imaging sensors 122;124 of the microscope. For example, the first and second imaging sensors 122;124 may include or be APS (Active Pixel Sensor) or CCD (Charge-Coupled-Device) based imaging sensors. For example, in APS-based imaging sensors, light is recorded at each pixel using a photodetector and an active amplifier in the pixel. APS-based imaging sensors are often based on CMOS (Complementary Metal Oxide Semiconductor) or S-CMOS (Scientific CMOS) technology. In CCD-based imaging sensors, incident photons are converted into electronic charges at the semiconductor-oxide interface, and these electronic charges are then transferred between capacitive bins in the imaging sensor module by the control circuitry of the imaging sensor module to perform imaging. The first and second image sensor data may be obtained by receiving the respective image sensor data from the image sensor (e.g., via interface 112), by reading the respective image sensor data from the memory of the respective image sensor (e.g., via interface 112), or by reading the respective image sensor data from storage device 116 of system 110, for example, after the image sensor data has been written to storage device 116 by the respective image sensor or by another system or processor.

[0029] The first image sensor data is acquired from a first image sensor, and the second image sensor data is acquired from a second image sensor. In other words, the first image sensor data and the second image sensor data are acquired from different sensors. Thus, the first image sensor is different from the second image sensor.

[0030] In various embodiments, as already pointed out, one of the imaging sensors may be a typical sensor used for fluorescence imaging. For example, if fluorescence imaging is not used, each imaging sensor may be used to provide additional sensor data, for example to improve the color accuracy of the composite color image. The embodiments may take advantage of the fact that a surgical microscope equipped for fluorescence microscopy has two imaging systems, namely, one imaging system (which may be a conventional RGB camera) with a known response band (e.g., a first plurality of mutually separated wavelength bands) for generating a visual (reflectance) image, and also a fluorescence imaging system with specifically defined wavelength bands (e.g., a second plurality of mutually separated wavelength bands of about 560 nm, 630 nm, 800 nm for fluorescein, PPIX (protoporphyrin IX), and ICG (indocyanine green), respectively). For example, the second imaging sensor may be adapted to provide the fluorescence imaging function of the microscope system. For example, the system may be configured to use a first optical imaging sensor for reflectance imaging in a first operating state of the microscope 120 and a second optical imaging sensor for fluorescence imaging in a second operating state of the microscope 120, and the system may be configured to use the first and second imaging sensors for performing reflectance imaging by generating a composite color image. In other words, the composite color image may be a reflectance image, i.e., not based on fluorescence imaging. Thus, the wavelength bands of the second plurality of mutually separated wavelength bands may be wavelength bands used for fluorescence imaging (i.e., emission wavelength bands used for fluorescence imaging). Thus, the wavelength bands of the first plurality of mutually separated wavelength bands may be wavelength bands used for reflectance imaging, i.e., wavelength bands across the entire visible color spectrum. However, in some embodiments, the first plurality of mutually separated wavelength bands may exclude wavelength bands used for fluorescence imaging. Thus, the wavelength bands of the first plurality of mutually separated wavelength bands and the second plurality of mutually separated wavelength bands do not have to overlap with each other.In other words, a wavelength band may be covered by either a first plurality of mutually separated wavelength bands or a second plurality of mutually separated wavelength bands, in various embodiments, both the first and second plurality of mutually separated wavelength bands include exactly three (consecutive) wavelength bands.

[0031] In general, a microscope system may include an illumination system (i.e., an illumination system) configured to illuminate a sample viewed through the microscope. In an embodiment, the illumination system may be used to assist reflectance imaging and fluorescence imaging performed using the illumination system. To generate a composite color image that is a reflectance image, the sample may be illuminated with each of a first and a second plurality of mutually independent wavelength bands. Thus, the system may be configured to control the illumination system (e.g., in a second operating state) such that the sample is illuminated with each of the first and second plurality of mutually independent wavelength bands. If fluorescence imaging is performed (in addition to reflectance imaging), the system may be configured to control the illumination system such that the sample is illuminated with each of the first plurality of mutually separated wavelength bands (and not with a wavelength band of the second plurality of mutually separated wavelength bands), as well as with one or more excitation wavelength bands of a fluorescent material used for the sample (if not already included in the first plurality of mutually separated wavelength bands). In a microscope system, light from the emission wavelength band may be (logically) removed from the illumination light and may be directed only to the fluorescence camera. If the system is not used for microscopy, additional information from reflectance imaging of the fluorescence emission bands or each fluorescence emission band, including the NIRICG (near infrared indocyanine green) band, can be used to provide a more accurate reconstructed color image. Thus, an illumination system may be operated by system 110 to illuminate the site / sample with the emission wavelength bands.

[0032] In general, both the illumination emission by the illumination system and the wavelength bands sensed by the imaging sensor may be defined by filters (see, for example, filters 620-640 in FIG. 6) mounted in the optical path of the sensor or illumination system. For example, the filter mounted in the optical path of the first imaging sensor may be a bandpass filter adapted to filter out light in wavelength bands outside the first plurality of mutually separated wavelength bands, e.g., such that (only) light having wavelengths within the first plurality of mutually separated wavelength bands is incident on the first imaging sensor. Accordingly, the filter mounted in the optical path of the second imaging sensor may be a bandpass filter adapted to filter out light in wavelength bands outside the second plurality of mutually separated wavelength bands, e.g., such that (only) light having wavelengths within the second plurality of mutually separated wavelength bands is incident on the second imaging sensor. The filter mounted in the optical path of the illumination system may be adapted to pass all light of the first and second plurality of mutually separated wavelength bands.

[0033] In various embodiments, as shown in Figures 1a and / or 1b, a beam splitter 126 is used to direct light reflected or emitted by the sample to the first and second imaging sensors. For example, the beam splitter may be a polychroic mirror configured to split the light such that light of a given wavelength is directed to either the first imaging sensor or the second imaging sensor. For example, the polychroic mirror may be adapted to direct light having wavelengths within a first plurality of mutually separated wavelength bands (only) to the first imaging sensor and light having wavelengths within a second plurality of mutually separated wavelength bands (only) to the second imaging sensor.

[0034] The system is configured to generate a composite color image based on the first and second imaging sensor data. In other words, the system is configured to generate a color image based on the imaging sensor data provided by the two imaging sensors. Thus, the composite color image is a color image generated based on the imaging sensor data provided by the two different imaging sensors. For example, the composite color image may be a multispectral color image generated by using imaging sensor data from two imaging sensors, where the imaging sensor data of each sensor represents light of a plurality of mutually separated wavelength bands. As a result, the composite color image is a multispectral color image generated based on light sensed in a plurality of mutually separated wavelength bands, where the plurality of mutually separated wavelength bands are sensed by two different imaging sensors (e.g., one is used for reflectance imaging and the other is used for reflectance and fluorescence imaging). For example, the composite color image may be generated based on the first and second imaging sensor data to improve the color accuracy of the composite color image.

[0035] The composite color image is generated using a conversion function that defines a conversion performed between the imaging sensor data and the composite color image. In contrast to other approaches, the conversion function is configured such that the composite color image is generated using sensor data based on light sensed in each wavelength band of the first and second plurality of mutually separated wavelength bands. In other words, the light sensed in all of the first and second plurality of mutually separated wavelength bands may be combined to generate the composite color image to obtain a composite color image with, for example, improved color accuracy. In general, the conversion function may be considered as a set of instructions used for conversion between the first and second imaging sensor data on the one hand and the composite color image on the other hand.

[0036] In general, the first and second image sensor data may each include a plurality of pixels generated by a plurality of sensor pixels of each image sensor (e.g., after demosaicing). Each image sensor data may include, for each pixel of the plurality of pixels, a plurality of numerical values ​​representing the light sensed in the plurality of modulated wavelength bands by wavelength band. For example, if light is sensed in three mutually separated wavelength bands by the first and / or second image sensor, the first and / or second image sensor data may include three numerical values ​​(respectively) per pixel representing the light sensed in the three mutually separated wavelength bands. The plurality of pixels of the first and second image sensor data may be in a predetermined relationship. Ideally, the plurality of pixels of the two image sensors may be generated such that each pixel of the first image sensor data represents the same point on the sample as the corresponding pixel of the second image sensor data.

[0037] For each pixel of the plurality of pixels, the system may be configured to input the numerical value of the pixel of the first image sensor data and the numerical value of the corresponding pixel of the second image sensor data into a conversion function to calculate a numerical value representing the pixel in a plurality of color channels of the composite color image. For example, if both the first and second plurality of mutually separated wavelength bands include three wavelength bands (for a total of six wavelength bands, and therefore a total of six numerical values ​​representing light in the six wavelength bands), and if the composite color image is based on three color channels (e.g., red, green and blue), the conversion function may specify a total of 18 (6×3) conversions to be performed between the image sensor data and the channels of the composite color image. These conversions may be defined as multiplication coefficients that may be multiplied by the numerical values ​​of the individual pixels of the first and second image sensor data. Thus, the conversion function may be based on a set of conversion coefficients each defining a conversion to be performed between the image sensor data based on light sensed in a wavelength band and a color channel (e.g., one of the three color channels) of the composite color image (i.e., between one of the plurality of numerical values ​​of the pixel). In the above example, the set of conversion coefficients may include 18 conversion coefficients. In other words, the set of conversion coefficients may include one conversion coefficient for each combination of wavelength band (i.e., pixel numerical value representing light sensed in the wavelength band) and color channel (i.e., color channel numerical value for the pixel). For example, the set of conversion coefficients may provide a conversion between the image sensor data of each wavelength band of the first and second multiple mutually separated wavelength bands and each color channel of the composite color image. The conversion function, and thus the set of conversion coefficients, may be applied (separately) to each pixel of the first and second image sensor data.

[0038] To improve performance, the transformation function may be defined as a matrix that can be multiplied by a vector containing the first and second imaging sensor data. In other words, the transformation function may be implemented by a transformation matrix. The entries of the transformation matrix may be defined by a set of transformation coefficients. The system may be configured to transform the imaging sensor data for each wavelength band of the first and second multiple mutually separated wavelength bands using the transformation matrix. For example, an example transformation matrix is ​​shown in FIG. 5d.

[0039] As already pointed out, the composite color image may include three color channels, e.g., a red channel, a blue channel, and a green channel (RGB). RGB is a channel model that is frequently used to represent color images, e.g., color images displayed on a display. Each color channel of the composite color image may be generated based on a transformation of the image sensor data of each wavelength band of the first and second plurality of mutually separated wavelength bands. In other words, the numerical values ​​of each of the first and second plurality of mutually separated wavelength bands may be used to calculate the numerical values ​​of each color channel. The transformation may be performed, for example, using a transformation function as shown above.

[0040] In various embodiments, the system is configured to provide a display signal to a display 140 of the microscope system using the interface 112 of the system 110 to cause the display to display the composite color image. In other words, the composite color image may be displayed on a display of the microscope system, for example, on an eyepiece display of the microscope system or on an auxiliary display of the microscope system.

[0041] The interface 112 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values ​​according to a specified code, within a module, between modules, or between modules of different entities. For example, the interface 112 may include an interface circuit configured to receive and / or transmit information. In an embodiment, the one or more processors 114 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer, or a programmable hardware component operating with appropriately adapted software. In other words, the described functions of the one or more processors 114 may also be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. In at least some embodiments, the one or more storage devices 116 may include at least one element of a group of computer-readable storage media, such as, for example, a magnetic or optical storage medium, e.g., a hard disk drive, a flash memory, a floppy disk, a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or network storage.

[0042] Further details and aspects of the system and microscope system are mentioned in connection with the proposed concept or one or more examples described above or below (e.g., FIGS. 2-8). The system and microscope system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0043] FIG. 2 shows a flow chart of an embodiment of a corresponding method for a microscope system. The method includes a step 210 of acquiring first image sensor data from a first image sensor of a microscope of a microscope system and second image sensor data from a second image sensor of the microscope. The first image sensor data includes sensor data based on light sensed in a first plurality of mutually separated wavelength bands. The second image sensor data includes sensor data based on light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The method includes a step 220 of generating a composite color image based on the first image sensor data and based on the second image sensor data. The composite color image is based on a plurality of color channels. The composite color image is generated using a conversion function that defines a conversion performed between the image sensor data and the composite color image, whereby the composite color image is generated using sensor data based on light sensed in each wavelength band of the first and second plurality of mutually separated wavelength bands.

[0044] As indicated above, the features described in relation to the system 110 and microscope system 100 of FIG. 1a and / or FIG. 1b may be applied to the method of FIG. 2 as well.

[0045] Further details and aspects of the method are mentioned in relation to the proposed concept or one or more examples described above or below (e.g., Figures 1a-1b, 3-8). The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0046] While Figures 1a-2 relate to the application of transformation functions in generating a composite color image, Figures 3 and 4 below relate to the generation of transformation functions. Thus, the transformation functions used in Figures 1a-2 may be generated using the systems, methods and / or computer programs presented in Figures 3 or 4.

[0047] 3 shows a schematic diagram of a system for determining a transformation function. The system 310 includes one or more processors 314 and one or more storage devices 316. Optionally, the system further includes an interface 312. The one or more processors are coupled to the interface and coupled to the one or more storage devices. In general, the functionality of the system is provided by the one or more processors, e.g., in association with an optional interface or one or more storage devices. For example, the system may be configured to obtain image sensor data via the interface and / or store conversion coefficients of the transformation function and / or a composite reference image using one or more storage devices.

[0048] The system is configured to acquire first image sensor data of the reference object 300 from the first image sensor 122 of the microscope 120 and second image sensor data of the reference object from the second image sensor 124 of the microscope. The first image sensor data includes sensor data based on light sensed in a first plurality of mutually separated wavelength bands, and the second image sensor data includes sensor data based on light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The system is configured to acquire a composite reference image of the reference object. The composite reference image includes a plurality of color channels. The system is configured to determine a conversion function by determining a set of conversion coefficients that provide an approximate conversion between the image sensor data of each wavelength band of the first and second plurality of mutually separated wavelength bands and each color channel of the composite reference image (i.e., a conversion in which the mismatch between the generated composite color image and the composite reference image is smaller than the mismatch resulting from using another set of conversion coefficients or is smaller than a threshold value). The transform function is based on a set of transform coefficients.

[0049] For example, the system of FIG. 3 may be implemented, for example, by the same system as the system 110 of FIG. 1a and / or FIG. 1b. Thus, the system 110 and / or the system 310 may be configured to provide the functionality of the respective other system 310;110. Thus, the microscope 120 may be the microscope 120 of the microscope system 100 of FIG. 1a and / or FIG. 1b. Thus, the first and second image sensor data may also be implemented similarly to the first and second image sensor data of FIG. 1a and / or FIG. 1b. Furthermore, the conversion function may be implemented similarly to the conversion function of FIG. 1a-FIG. 2.

[0050] Some embodiments of the present disclosure relate to a system, method and computer program for determining a conversion function. As already pointed out in relation to Figs. 1a-2, the conversion function defines a conversion performed between the imaging sensor data and a composite color image based on the imaging sensor data. In general, the conversion function may be considered as a set of instructions used for conversion between the first and second imaging sensor data on the one hand and the composite color image on the other hand. The embodiment shown in relation to Figs. 3 and 4 provides an approach for generating such a conversion function using a composite reference image and using the imaging sensor data provided by two imaging sensors.

[0051] In general, the composite reference image may be implemented similarly to the composite color image introduced in relation to Fig. 1a and / or Fig. 1b, which is a color image and is based on multiple color channels (e.g., three color channels - red, green, blue). The composite reference image may differ from the composite color image in that the numerical values ​​representing the color channels of the individual pixels are reference values, i.e. values ​​that define the numerical values ​​at which the determined transformation function is provided when applied to the first and second imaging sensor data representing the reference object. In other words, the first and second imaging sensor data represent the reference object and the reference composite image represents a composite color image that is the desired result of the transformation function applied to the first and second imaging sensor data. Preferably, the composite reference image includes a plurality of predefined colors with corresponding predefined numerical values ​​representing the color channels of the individual pixels. For example, the reference object may be a color chart or color table, i.e. a table showing a plurality of colors each having a predefined representation in the multiple channels of the composite reference image. Since the color of the reference object is known from the composite reference image, the system can use the composite reference image as the desired result of the transformation function applied to the imaging sensor data. Thus, the system may be configured to identify a set of conversion coefficients defining a conversion performed between the image sensor data and the composite color image such that the difference between the composite color image and the composite reference image is reduced. In contrast to other systems, a set of conversion coefficients is identified for the image sensor data of both image sensors, so that, for example, a first subset of conversion coefficients for providing a conversion between the first image sensor data and the composite color image and a second subset of conversion coefficients for providing a conversion between the first image sensor data and the composite color image have the potential to be more than twice the conversion coefficients such that, for example, the difference between the composite color image and the composite reference image is reduced compared to other sets of conversion coefficients. In other words, the system may be configured to identify a set of conversion coefficients for which the mismatch (i.e. difference) between the composite reference image and the converted image (i.e. composite color image) generated based on the set of conversion coefficients is lower than at least one other set of conversion coefficients.For example, the system may be configured to determine the set of conversion coefficients by defining a set of equations using the first and second imaging sensor data and the composite reference image, and solving the set of equations for a set of potential conversion coefficients that provide a conversion between the first and second imaging sensor data and the composite reference image. This may be performed for a number of different colors, e.g., for a number of colors of the reference object.

[0052] For example, the reference object may include a plurality of predefined colors (e.g., a color table), and the composite color image may include a plurality of portions representing the plurality of predefined colors. In other words, the composite reference image of the reference image may define a plurality of colors of a plurality of portions of the reference object, e.g., a plurality of predefined colors of a color table. The system may be configured to determine a set of conversion coefficients for the plurality of (predefined) colors, i.e., by identifying, for each of the plurality of colors, a set of conversion coefficients that provides an approximate conversion between the first and second imaging sensor data of the portion representing the (predefined color) as represented by the plurality of channels of the composite reference image, and the predefined color.

[0053] In a surgical microscope application, some colors may be more useful to the surgeon. For example, some colors may indicate diseased tissue, but the difference between the color of the diseased tissue and the color of healthy tissue may be small. In some embodiments, these colors may be treated with priority in determining the set of conversion coefficients. For example, the plurality of (predefined) colors may include a first subset of predefined colors and a second subset of predefined colors. For example, the first subset of predefined colors may be colors that are predefined as important for a particular application of the microscope. The second subset of predefined colors may be other colors, i.e., colors that are not important for a particular application of the microscope. For example, the first subset of predefined colors may be colors that exist as colors of organic tissue in a surgical setting, e.g., colors that indicate healthy or diseased tissue. The second subset of predefined colors may be colors that are less relevant in identifying healthy or diseased tissue. Thus, the system may be configured to identify a set of transform coefficients for which a mismatch between the composite reference image and the transformed image generated based on the set of transform coefficients is lower than a set of at least one other transform coefficient for the predetermined first subset color. In some embodiments, the second subset color may not be considered in identifying the set of transform coefficients. However, in some other embodiments, the second subset color may be considered, albeit with a lower priority or weighting than the predetermined first subset color. In other words, the system may be configured to identify a set of transform coefficients that reduces a mismatch value representing a mismatch between the composite reference image compared to the set of at least one other transform coefficients. This mismatch value may be calculated for colors of a plurality of colors. A mismatch for a color of the predetermined first subset color may have a higher influence on the mismatch value than a mismatch for a color of the second subset color.In other words, discrepancies in color for the first predetermined subset of colors may receive a higher weighting in identifying the set of transform coefficients than discrepancies in color for the second predetermined subset of colors.

[0054] An embodiment of the present disclosure further provides a microscope system 100 (which may be implemented similarly to the microscope system of Figures 1a and / or 1b) including system 310 (and optionally system 110 if the systems are implemented separately) and a microscope 120 having first and second imaging sensors 122, 124. One of the first and second imaging sensors may be an imaging sensor adapted to provide a fluorescence imaging function of the microscope system.

[0055] The interface 312 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values ​​according to a specified code, within a module, between modules, or between modules of different entities. For example, the interface 312 may include an interface circuit configured to receive and / or transmit information. In an embodiment, the one or more processors 314 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer, or a programmable hardware component operating with appropriately adapted software. In other words, the described functions of the one or more processors 314 may also be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. In at least some embodiments, the one or more storage devices 316 may include at least one element of a group of computer-readable storage media, such as, for example, a magnetic or optical storage medium, e.g., a hard disk drive, a flash memory, a floppy disk, a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or network storage.

[0056] Further details and aspects of the system and microscope system are mentioned in connection with the proposed concept or one or more examples described above or below (e.g., FIGS. 1a-2, 4-8). The system and microscope system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0057] FIG. 4 shows a flow chart of an embodiment of a method for determining a conversion function. The method includes a step 410 of acquiring first image sensor data of a reference object from a first image sensor of a microscope and second image sensor data of a reference object from a second image sensor of the microscope. The first image sensor data includes sensor data based on light sensed in a first plurality of mutually separated wavelength bands. The second image sensor data includes sensor data based on light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The method includes a step 420 of acquiring a composite reference image of the reference object, the composite reference image including a plurality of color channels. The method includes a step 430 of determining a conversion function by determining a set of conversion coefficients that provide an approximate conversion between the image sensor data of each wavelength band of the first and second plurality of mutually separated wavelength bands and each color channel of the composite reference image. The conversion function is based on the set of conversion coefficients.

[0058] As indicated above, the features described in connection with the system 310 of FIG. 3 may be applied to the method of FIG.

[0059] Further details and aspects of the method are mentioned in relation to the proposed concept or one or more examples described above or below (e.g., Figures 1a-3, 5a-8). The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0060] Various embodiments of the present disclosure are based on the use of a fluorescence camera in a microscope system for multispectral reflectance imaging.

[0061] As already noted, surgeons often use tissue (brain tissue) color to distinguish suspicious tissues for pathology. However, subtle differences in tissue color may only be visible to surgeons with years of experience and a trained eye, which can be difficult to master.

[0062] Multispectral reflectance imaging can capture very small or even invisible color differences using measurements of very small spectral differences, although traditional multispectral imaging may require additional hardware in addition to the multiple sensors of modern microscopes, typically equipped with 3D and fluorescence cameras.

[0063] The disclosed embodiments can use the existing fluorescent sensor of a surgical microscope system for multispectral reflectance imaging when the fluorescent mode is not activated, allowing for software-based functionality without the added cost, size and complexity.

[0064] Various embodiments can use multispectral imaging to generate accurate color tissue imaging. This can be done by measuring more than three spectral bands, for example six spectral bands (e.g., three spectral bands from each of the first and second imaging sensor data), and then digitally recombining the spectral information for calculation of RGB values ​​(i.e., a combined color image). This may result in more accurate color capture.

[0065] For imaging systems using multiple sensors for reflectance and fluorescence imaging, multispectral reflectance imaging can be performed by using the fluorescence band in reflectance mode, which can be enabled by providing illumination in the fluorescence band.

[0066] Subtle color differences with diagnostic importance may be digitally amplified and therefore may be easily visible even to an untrained surgeon.

[0067] Figure 5a shows a schematic diagram of the different colors present in an image frame. In Figure 5a, there are three regions 510-530 of organic tissue (e.g., brain tissue is shown) where each region has a specific color, where region 510 has a bluish color, region 520 has a greenish color, and region 530 has a reddish color.

[0068] 5b and 5c show schematic diagrams of the intensities of different colors in different regions 510-530 sensed by two sensors (indexed 1 and 2) at different frequency bands 1-6, each sensor sensing light in three wavelength bands (A-C), thereby generating the image sensor data S A1 , S A2 , S B1 , S B2 , S C1 and S C2 As can be seen in Figure 5c, the sensors sense light in narrower frequency bands than shown in Figure 5b, allowing the frequency bands to be separated from one another. The different sensors are highlighted using different background patterns.

[0069] The sensor data generated in the mutually separated frequency bands may be used to generate an RGB image, for example, via a transformation matrix that includes a set of transformation coefficients. FIG. 5d shows a set of 18 transformation coefficients F 11 ~F 36 3 shows an example transformation matrix of size 3 x 6, including (row and column indices). The transformation matrix is ​​used to transform the sensor data S to obtain the RGB values ​​of the composite color image (as a 3 x 1 matrix). A1 , S A2 , S B1 , S B2 , S C1 and S C2 It is multiplied by a (6-by-1) matrix containing

[0070] FIG. 6 shows a schematic diagram of the microscope and the illumination system. As already pointed out, the sensor data of two imaging sensors are used: a first imaging sensor (e.g., a CCD sensor) 122 used for reflectance imaging, and a second imaging sensor (e.g., a CCD sensor) 124 used mainly for fluorescence imaging. The first imaging sensor may be configured to provide a color reflectance image, and the second imaging sensor provides a fluorescence image (in a first operating mode) and also a reflectance image (in a second operating mode, e.g., in three bands). Both imaging sensors are combined with mirrors 630; 640, which are able to filter light outside the frequency bands sensed by each imaging sensor. A polychroic mirror 650 is used to split the light into different wavelength bands and direct each wavelength band to a sensor. An illumination system 610 is used (in combination with a filter 620, which filters the light emitted by the illumination system) to illuminate the object to be imaged by the imaging sensor. FIG. 6 further illustrates light of different wavelength bands (with different line styles) where visible light 660 is incident on a first imaging sensor and light in an emission wavelength band is incident on a second imaging sensor, and light in the emission wavelength band is either emitted 670 by an illumination system or emitted by fluorescent material 680 of an object that is excited in a fluorescence excitation wavelength band (e.g., by visible light 660).

[0071] In Fig. 7a and Fig. 7b, a concept for generating a conversion function is shown. As has been described in relation to Fig. 3 and Fig. 4, the conversion function may be determined by using a reference color table with "known" RGB values ​​of the color samples of the color table. Fig. 7a shows a schematic diagram of an exemplary color table including color fields 1 to 20. This color table may be recorded by two imaging sensors and the corresponding sensor data may be determined for each mutually separated wavelength band. The known RGB values ​​and the sensor data of the individual wavelength bands may be input into a system of equations, which may be solved such that the difference between the resulting color and the known RGB values ​​is reduced or minimized for the different color samples. Fig. 7b is a diagram of an exemplary system of equations with a column of sample numbers, a column of sensor data of different wavelength bands and a column of known RGB values.

[0072] Further details and aspects of the concept are mentioned in connection with the proposed concept or one or more examples described above or below (e.g., Figures 1a-4, 8). The concept may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0073] Some embodiments relate to a microscope including a system as described in relation to one or more of Figures 1 to 7b. Alternatively, the microscope may be part of a system as described in relation to one or more of Figures 1 to 7b or may be connected to a system as described in relation to one or more of Figures 1 to 7b. Figure 8 shows a schematic diagram of a microscope system including a microscope and a computer system. Figure 8 shows a schematic diagram of a system 800 configured to perform the methods described herein. The system 800 includes a microscope 810 and a computer system 820. The microscope 810 is configured to capture images and is connected to the computer system 820. The computer system 820 is configured to perform at least a portion of the methods described herein. The computer system 820 may be configured to execute a machine learning algorithm. The computer system 820 and the microscope 810 may be separate entities, but may be integrated in one common housing. The computer system 820 may be part of a central processing system of the microscope 810 and / or the computer system 820 may be part of a subordinate component of the microscope 810, such as a sensor, actor, camera or lighting unit of the microscope 810.

[0074] The computer system 820 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, such as local clients and / or one or more remote server farms and / or data centers). The computer system 820 may include any circuit or combination of circuits. In one embodiment, the computer system 820 may include one or more processors, which may be of any type. As used herein, a processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit, for example, of a microscope or a microscope component (e.g., a camera). Other types of circuits that may be included in computer system 820 may be custom circuits, application specific integrated circuits (ASICs), such as one or more circuits (such as communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 820 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives handling removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), and the like.Computer system 820 may also include a display device, one or more speakers and a keyboard and / or controller which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that enables a user of the system to input information to and receive information from computer system 820.

[0075] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the crucial steps may be performed by such an apparatus.

[0076] Depending on certain implementation requirements, the embodiments of the present invention can be implemented in hardware or software. The implementation can be performed by a non-transitory recording medium, such as a digital recording medium, for example a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM and EPROM, an EEPROM or a FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Thus, the digital recording medium can be computer readable.

[0077] Some embodiments of the present invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to perform any of the methods described herein.

[0078] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code which is operable to perform any of the methods when the computer program product is run on a computer, the program code may for example be stored on a machine readable carrier.

[0079] Another embodiment comprises the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0080] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein, when the computer program runs on a computer.

[0081] Therefore, another embodiment of the invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, including a processor and a recording medium.

[0082] A further embodiment of the invention is therefore also a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or the sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0083] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0084] Another embodiment comprises a computer having the computer program installed thereon for performing any of the methods described herein.

[0085] Another embodiment of the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server to transfer the computer program to the receiver.

[0086] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus.

[0087] As used in this specification, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0088] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus. [Explanation of symbols]

[0089] 100 Microscope System 105 Base Unit 110 System 112 Interface 114 One or more processors 116 One or more storage devices 120 Microscope 122 First image sensor 124 Second Image Sensor 126 Beam splitter / polychroic mirror 130 Lighting System 140a / b Display 150 Steering wheel 160 Arm 210 Acquire first and second image sensor data 220 Generate a composite color image 300 Reference Objects 310 System 312 Interface 314 One or more processors 316 One or more storage devices 410 Acquire first and second image sensor data 420 Acquire a synthetic reference image 430 Determine the transformation function 510~530 Areas with different colors 610 Lighting System 620~640 Filter 650 Polychroic Mirror 660 visible light 670 Light emitted by an illumination system and reflected by an object 680 Light emitted by an object 800 Systems 810 Microscope 820 Computer Systems

Claims

1. A system (310) for determining a transformation function, comprising: The system includes one or more processors (314) and one or more storage devices (316); The system is configured to acquire first image sensor data of a reference object (300) from a first image sensor (122) of a microscope (120) and second image sensor data of the reference object (300) from a second image sensor (124) of the microscope, the first image sensor data comprising sensor data based on light sensed in a first plurality of mutually separated wavelength bands and the second image sensor data comprising sensor data based on light sensed in a second plurality of mutually separated wavelength bands, the first plurality of mutually separated wavelength bands being wavelength bands used for reflectance imaging and a wavelength band of the second plurality of mutually separated wavelength bands being wavelength bands used for fluorescence imaging; the system is configured to acquire a synthetic reference image of the reference object, the synthetic reference image including a plurality of color channels; The system is configured to determine the conversion function by determining a set of conversion coefficients that provide an approximation conversion between the imaging sensor data for each wavelength band of the first and second plurality of mutually separated wavelength bands and each color channel of the composite reference image, the conversion function being based on the set of conversion coefficients: the system is configured to identify a set of transform coefficients that results in a lower mismatch between the synthetic reference image and a transformed image generated based on the set of transform coefficients than at least one other set of transform coefficients; the composite reference image defines a plurality of colors of a plurality of portions of the reference object, the plurality of colors including a first predetermined subset of colors and a second predetermined subset of colors, and the system is configured to identify a set of transform coefficients for which a mismatch between the composite reference image and a transformed image generated based on the set of transform coefficients is lower than at least one other set of transform coefficients for the first predetermined subset of colors. system.

2. the first predetermined subset of colors being colors present as organic tissue colors in a surgical setting; The system of claim 1 .

3. the system is configured to identify a set of transform coefficients that, in comparison with at least one other set of transform coefficients, reduces a mismatch value representing a mismatch between the composite reference images, the mismatch value being calculated for colors of the plurality of colors, and a mismatch for colors of the first predetermined subset colors has a higher influence on the mismatch value than a mismatch for colors of the second predetermined subset colors.

3. The system according to claim 1 or 2.

4. 1. A method for determining a transformation function, the method comprising: acquiring (410) first image sensor data of a reference object from a first image sensor of a microscope and second image sensor data of the reference object from a second image sensor of the microscope, the first image sensor data comprising sensor data based on light sensed in a first plurality of mutually separated wavelength bands and the second image sensor data comprising sensor data based on light sensed in a second plurality of mutually separated wavelength bands, the first plurality of mutually separated wavelength bands being wavelength bands used for reflectance imaging and a wavelength band of the second plurality of mutually separated wavelength bands being wavelength bands used for fluorescence imaging; acquiring (420) a synthetic reference image of the reference object, the synthetic reference image including multiple color channels; determining (430) a conversion function by determining a set of conversion coefficients that provide an approximation conversion between the imaging sensor data for each wavelength band of the first and second plurality of mutually separated wavelength bands and each color channel of the composite reference image, the conversion function being based on the set of conversion coefficients; Including, The method includes identifying a set of transform coefficients that results in a lower mismatch between the synthetic reference image and a transformed image generated based on the set of transform coefficients than at least one other set of transform coefficients; the synthetic reference image defines a plurality of colors of a plurality of portions of the reference object, the plurality of colors including a predetermined first subset of colors and a second subset of colors, the method including identifying a set of transform coefficients for which a mismatch between the synthetic reference image and a transformed image generated based on the set of transform coefficients is lower than at least one other set of transform coefficients for the predetermined first subset of colors. method.

5. 5. A computer program comprising a program code for performing at least one method according to claim 4, when the computer program is executed on a processor.

Images