System, method and computer program for a microscope system and for determining a transfer function

Abstract

Examples relate to a microscope system and a system, method and computer program for determining a transformation function, and a corresponding microscope system. The system for a microscope system comprises one or more processors and one or more storage devices. The system is configured to acquire first imaging sensor data from a first imaging sensor of a microscope of the microscope system and second imaging sensor data from a second imaging sensor of the microscope. The first imaging sensor data comprises sensor data of light perceived in a first plurality of mutually separate wavelength bands. The second imaging sensor data comprises sensor data of light perceived in a second plurality of mutually separate wavelength bands. A wavelength band of the first plurality of mutually separate wavelength bands or the second plurality of mutually separate wavelength bands is a wavelength band for fluorescence imaging. The system is configured to generate a composite color image based on the first imaging sensor data and based on the second imaging sensor data. The composite color image is based on a plurality of color channels. The composite color image is generated using a transformation function, the transformation function defining a transformation performed between the imaging sensor data and the composite color image, such that the composite color image is generated using sensor data of light perceived in each wavelength band of the first and second plurality of mutually separate wavelength bands.

Description

[0001] This application is a divisional application of Leica Instruments (Singapore) Pte Ltd, filed on March 8, 2021, with application number 202180039647.3, entitled "System, method and computer program for microscope system and for determining transformation function". Technical Field

[0002] Examples relate to microscope systems and systems, methods, and computer programs for determining transformation functions, as well as corresponding microscope systems. Background Technology

[0003] Surgeons often use the color of tissues (such as brain tissue) to distinguish suspicious tissues (such as lesions). However, in many cases, subtle differences in tissue color can only be seen by surgeons with extensive experience and trained visual acuity, making them difficult to learn. Multispectral reflectance imaging can capture very small or even invisible color differences by measuring very small spectral variations. However, for the known concept of multispectral imaging, additional hardware may be required beyond the multiple sensors of modern microscopes. Summary of the Invention

[0004] The desired concept is to provide improved color images that make subtle differences between different types of tissues visible.

[0005] This expectation is addressed by the subject matter of the independent claims.

[0006] Embodiments of the present invention are based on the discovery that multispectral imaging can be performed by combining imaging sensor data from an imaging sensor primarily used for fluorescence imaging with imaging sensor data from an imaging sensor primarily used for reflectance imaging. Therefore, embodiments of the present invention provide a system for a microscope system. The system includes one or more processors and one or more storage devices. The system is configured to acquire first imaging sensor data from a first imaging sensor of the microscope system and second imaging sensor data from a second imaging sensor of the microscope. The first imaging sensor data includes sensor data regarding light sensed in a first plurality of mutually separate wavelength bands. The second imaging sensor data includes sensor data regarding light sensed in a second plurality of mutually separate wavelength bands. The wavelength bands of either the first plurality of mutually separate wavelength bands or the second plurality of mutually separate wavelength bands are wavelength bands used for fluorescence imaging. The system is configured to generate a composite color image based on the first imaging sensor data and the second imaging sensor data. The composite color image is based on multiple color channels. The composite color image is generated using a transformation function that defines a transformation performed between the imaging sensor data and the composite color image, thereby generating the composite color image using sensor data of light sensed in each of the first and second plurality of mutually separate wavelength bands.

[0007] Embodiments of the present invention further provide a microscope system, including a system and a microscope having first and second imaging sensors. One of the first and second imaging sensors is an imaging sensor adapted to provide fluorescence imaging functionality of the microscope system.

[0008] By using data from two imaging sensors, a more precise transformation function can be used to construct color images, revealing subtle differences between different tissue types. Simultaneously, reusing fluorescence imaging sensors for reflectance imaging allows this method to be used without including additional sensors in the microscope system.

[0009] In various embodiments, the transformation is based on a set of transformation factors that define the transformations performed between imaging sensor data of light sensed in a wavelength band and the color channels of a composite color image. For example, the set of transformation factors can provide a one-to-one transformation between the intensity of light measured in a wavelength band and the color channels.

[0010] For example, the transform factor set may include a transform factor for each combination of wavelength bands and color channels. Therefore, a one-to-one transformation can be applied, for example, using matrix multiplication. In other words, the transform factor set can provide a transformation between imaging sensor data for each of the first and second sets of mutually separate wavelength bands and each color channel of the composite color image.

[0011] For example, a composite color image can include three color channels (e.g., red, green, and blue). Each color channel can be generated based on a transformation of imaging sensor data for each wavelength band of a first and second plurality of mutually separate wavelength bands. The transformation can be performed using a transformation function. By using imaging sensor data for each wavelength band for each color channel, even subtle color differences can be included in the composite color image.

[0012] In various embodiments, the transformation function can be implemented using a transformation matrix. The system can be configured to transform imaging sensor data in each of first and second, or more, mutually separate wavelength bands, using the transformation matrix. The transformation matrix provides a computationally efficient implementation of the transformation function.

[0013] In various embodiments, the system is configured to provide display signals to the microscope system's display using the system's interface, causing the display to show a composite color image. Therefore, the composite color image can be shown to the microscope system user, such as a surgeon.

[0014] Embodiments of the present invention further provide a system for determining a transformation function. The system includes one or more processors and one or more storage devices. The system is configured to acquire first imaging sensor data of a reference object from a first imaging sensor of a microscope, and second imaging sensor data of the reference object from a second imaging sensor of a microscope. The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separated wavelength bands. The second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The system is configured to acquire a composite reference image of the reference object. The composite reference image includes multiple color channels. The system is configured to determine a transformation function by determining a set of transformation factors that provides an approximate transformation between the imaging sensor data of each wavelength band of the first and second plurality of mutually separated wavelength bands and each color channel of the composite reference image. The transformation function is based on the set of transformation factors. By using two sets of imaging sensor data, a more accurate transformation function can be used to construct a color image, thereby revealing subtle differences between different types of tissue.

[0015] In some embodiments, the system is configured to identify a set of transform factors that produce a lower mismatch between the composite reference image and the transformed image generated based on the set of transform factors than at least one other set of transform factors. In other words, the system can be configured to iteratively search for transform factors that reduce the mismatch between the composite reference image and the transformed image.

[0016] For example, a composite reference image of reference images can define multiple colors for multiple parts of a reference object. These multiple colors may include a predefined first subset of colors and a second subset of colors. The system can be configured to identify a set of transformation factors for which, for the predefined first subset of colors, the mismatch generated between the composite reference image and the transformed image generated based on the transformation factor set is less than that of at least one other set of transformation factors. In other words, transformation factors that reduce the mismatch of the first subset of colors can be identified, which can be particularly beneficial. For example, the predefined first subset of colors might be colors that appear as organic tissue colors in surgical settings. Improving sensitivity within the first subset of colors may be more beneficial than within other colors.

[0017] In various embodiments, the system is configured to identify a set of transform factors that reduces mismatch values ​​representing mismatches with a composite reference image compared to at least one other set of transform factors. Mismatch values ​​can be calculated for colors of multiple colors. Mismatches in a predefined first subset of colors may have a greater impact on the mismatch values ​​than mismatches in a second subset of colors. Therefore, colors in the first subset may receive higher weights than other colors when determining transform factors.

[0018] Embodiments of the present invention further provide a method for a microscope system. The method includes acquiring first imaging sensor data from a first imaging sensor of a microscope in the microscope system and acquiring second imaging sensor data from a second imaging sensor of the microscope. The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separate wavelength bands. The second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separate wavelength bands. The wavelength bands of the first plurality of mutually separate wavelength bands or the second plurality of mutually separate wavelength bands are wavelength bands used for fluorescence imaging. The method includes generating a composite color image based on the first imaging sensor data and based on the second imaging sensor data. The composite color image is based on a plurality of color channels. The composite color image is generated using a transformation function that defines a transformation performed between the imaging sensor data and the composite color image, thereby generating the composite color image using sensor data of light sensed in each of the first and second plurality of mutually separate wavelength bands.

[0019] Embodiments of the present invention further provide a method for determining a transformation function. The method includes acquiring first imaging sensor data of a reference object from a first imaging sensor of a microscope and second imaging sensor data of the reference object from a second imaging sensor of the microscope. The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separated wavelength bands. The second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separated wavelength bands. The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. The method includes acquiring a composite reference image of the reference object. The composite reference image includes multiple color channels. The method includes determining a transformation function by determining a set of transformation factors, the set of transformation factors providing an approximate transformation between the imaging sensor data of each wavelength band of the first and second plurality of mutually separated wavelength bands and each color channel of the composite reference image. The transformation function is based on the set of transformation factors.

[0020] Embodiments of the present invention further provide a computer program having program code for performing at least one method when the computer program is run on a processor. Attached Figure Description

[0021] The following will only illustrate some examples of apparatus and / or methods, with reference to the accompanying drawings, wherein...

[0022] Figure 1a and 1b A schematic diagram of a system for a microscope system is shown, as well as a schematic diagram of a microscope system including the system;

[0023] Figure 2 A flowchart of a method for a microscope system is shown;

[0024] Figure 3 A schematic diagram of the system used to determine the transformation function is shown;

[0025] Figure 4 A flowchart is shown for the method used to determine the transformation function;

[0026] Figure 5a A schematic diagram showing the different colors present in an image frame is shown;

[0027] Figure 5b and 5c A schematic diagram showing the intensity of different colors perceived in different frequency bands is shown;

[0028] Figure 5d An example transformation matrix is ​​shown;

[0029] Figure 6 A schematic diagram of the microscope and illumination system is shown;

[0030] Figure 7a A schematic diagram of an exemplary color table is shown;

[0031] Figure 7b A diagram of an exemplary system of equations is shown; and

[0032] Figure 8 A schematic diagram of a microscope system, including a microscope and a computer system, is shown. Detailed Implementation

[0033] Various examples will now be described more fully with reference to the accompanying drawings, some of which are illustrated in the drawings. In the drawings, the thickness of lines, layers, and / or areas may be exaggerated for clarity.

[0034] Figure 1a and 1b A schematic diagram of system 110 for microscope system 100 is shown, as well as a schematic diagram of microscope system 100 including system 100. System 110 includes one or more processors 114 and one or more storage devices 116. Optionally, the system further includes an interface 112. One or more processors are coupled to the interface and one or more storage devices. Typically, the functionality of the system is provided by one or more processors, for example, in conjunction with the optional interface or with one or more storage devices. For example, the system may be configured to acquire imaging sensor data via the interface and / or store transformation factors of transformation functions using one or more storage devices.

[0035] The system is configured to acquire first imaging sensor data from a first imaging sensor 122 of a microscope 120 of a microscope system, and second imaging sensor data from a second imaging sensor 124 of the microscope. The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separate wavelength bands. The second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separate wavelength bands. The wavelength bands of either the first plurality of mutually separate wavelength bands or the second plurality of mutually separate wavelength bands are wavelength bands used for fluorescence imaging. The system is configured to generate a composite color image based on the first imaging sensor data and the second imaging sensor data. The composite color image is based on multiple color channels. The composite color image is generated using a transformation function that defines a transformation performed between the imaging sensor data and the composite color image, thereby generating the composite color image using sensor data of light sensed in each of the first and second plurality of mutually separate wavelength bands.

[0036] Figure 1b A block diagram of a microscope system including microscope 120 and system 110 is shown. Figure 1bThe microscope system shown is a surgical microscope system that surgeons can use at the surgical site. Figure 1b The surgical microscope system shown includes a number of optional components, such as a base unit 105 (including system 110) with a (rolling) support, an auxiliary display 140a, an illumination system 130, a (robotic or manual) arm 160 that fixes the microscope 120 and is coupled to the base unit 105 and the microscope 120, and a steering handle 150 attached to the microscope 120. In addition to the first and second imaging sensors 122; 124, the microscope 120 may include an optional eye display 140b and an optional auxiliary display. In the context of this application, the term "(surgical) microscope system" is used to cover parts of a system that are not part of the actual microscope (including optical components) but are used with the microscope, such as a display or illumination system. One of the first and second imaging sensors is an imaging sensor adapted to provide fluorescence imaging capabilities for the microscope system.

[0037] Various embodiments of this disclosure relate to systems, methods, and computer programs for use in microscope systems. Generally, a microscope is an optical instrument suitable for examining objects too small to be examined by the naked eye (alone). For example, a microscope can provide optical magnification of an object. In modern microscopes, optical magnification is often provided to a camera or imaging sensor, such as... Figure 1a The microscope 120 includes first and second imaging sensors 122 and 124. The microscope 120 may further include one or more optical magnification components for magnifying views on a sample.

[0038] There are various types of microscopes. If the microscope system is used in the medical or biological field, the object observed through the microscope can be a sample of organic tissue, such as that arranged in a petri dish or present in a part of a patient's body. For example, microscope system 100 can be a laboratory microscope system, such as a microscope used to examine organic tissue samples in a petri dish. Alternatively, microscope 120 can be part of surgical microscope system 100, such as a microscope used during surgery. For example, such a system... Figure 1b As shown in the illustration. Although the embodiments are described in conjunction with a microscope system, they can also be applied in a more general way to any optical device.

[0039] The system is configured to acquire first and second imaging sensor data from first and second imaging sensors 122; 124 of a microscope. For example, the first and second imaging sensors 122; 124 may include or be based on APS (Active Pixel Sensor) or CCD (Charge Coupled Device) imaging sensors. For example, in an APS-based imaging sensor, light is recorded at each pixel using a photodetector and an active amplifier. APS-based imaging sensors are typically based on CMOS (Complementary Metal-Oxide-Semiconductor) or S-CMOS (Scientific CMOS) technology. In a CCD-based imaging sensor, incoming photons are converted into electron charges at a semiconductor-oxide interface and then moved between capacitor cells in the imaging sensor module via control circuitry to perform imaging. The first and second imaging sensor data can be acquired by receiving corresponding imaging sensor data from the imaging sensor (e.g., via interface 112), reading corresponding imaging sensor data from the memory of the corresponding imaging sensor (e.g., via interface 112), or reading corresponding imaging sensor data from storage device 116 of system 110, for example, after the corresponding imaging sensor or another system or processor has written imaging sensor data to storage device 116.

[0040] The first imaging sensor data is acquired from the first imaging sensor, and the second imaging sensor data is acquired from the second imaging sensor. In other words, the first imaging sensor data and the second imaging sensor data are acquired from different sensors. Therefore, the first imaging sensor is different from the second imaging sensor.

[0041] In various embodiments, as noted above, one of the imaging sensors can be a sensor typically used for fluorescence imaging. For example, when fluorescence imaging is not used, a corresponding imaging sensor can be used to provide additional sensor data, for example, to improve the color accuracy of a composite color image. Embodiments can take advantage of the fact that a surgical microscope equipped with a fluorescence microscope has two imaging systems: one with known response bands (e.g., a first plurality of mutually separate wavelength bands) for generating visual materials (reflected images, which could be a conventional RGB camera), and the other a fluorescence imaging system with specifically defined wavelength bands (e.g., approximately 560 nm, 630 nm, and 800 nm for fluorescein, PPIX (protoporphyrin IX), and ICG (indocyanine green), respectively, a second plurality of mutually separate wavelength bands). For example, the second imaging sensor can be adapted to provide fluorescence imaging capabilities for the microscope system. For example, in a first operating state of the microscope 120, the system can be configured to perform reflective imaging using a first optical imaging sensor and fluorescence imaging using a second optical imaging sensor, and in a second operating state of the microscope 120, the system can be configured to perform reflective imaging using a first optical imaging sensor and fluorescence imaging using a second optical imaging sensor. A second imaging sensor performs reflective imaging to generate a composite color image. In other words, the composite color image can be a reflective image, i.e., it may not be based on fluorescence imaging. Therefore, the wavelength bands of the second plurality of mutually separated wavelength bands can be wavelength bands used for fluorescence imaging (i.e., emission wavelength bands used for fluorescence imaging). Therefore, the wavelength bands of the first plurality of mutually separated wavelength bands can be wavelength bands used for reflective imaging, such as across the visible color spectrum. However, in some embodiments, the first plurality of mutually separated wavelength bands may exclude wavelength bands used for fluorescence imaging. Therefore, the wavelength bands of the first plurality of and the second plurality of mutually separated wavelength bands may not overlap. In other words, the wavelength bands can be covered by either the first or the second plurality of mutually separated wavelength bands. In various embodiments, both the first and second plurality of mutually separated wavelength bands comprise exactly three (consecutive) wavelength bands.

[0042] Generally, a microscope system may include an illumination system (i.e., a light system) configured to illuminate a sample observed through the microscope. In embodiments, the illumination system may be used to support reflectance imaging and fluorescence imaging performed using the illumination system. To generate a composite color image as a reflectance image, the sample may be illuminated in each of a first and second plurality of mutually independent wavelength bands. Thus, the system may be configured to control the illumination system such that the sample is illuminated in each of the first and second plurality of mutually independent wavelength bands (e.g., in a second operating state). If fluorescence imaging (in addition to reflectance imaging) is performed, the system may be configured to control the illumination system such that the sample is illuminated in each of the first plurality of mutually separate wavelength bands (rather than in the wavelength bands of the second plurality of mutually separate wavelength bands), and (if not already included in the first plurality of mutually separate wavelength bands), in one or more excitation wavelength bands of the fluorescent material used for the sample. In the microscope system, light from the emission wavelength bands may (logically) be removed from the illumination light and may also be separately directed to the fluorescence camera. When the system is not used for microscopy, additional information from reflectance imaging or each fluorescence emission band (including the NIR ICG (near-infrared indocyanine green) band) can be used to provide a more accurate reconstructed color image. Therefore, the illumination system can be operated by system 110 to illuminate the field / sample using emission wavelengths.

[0043] Generally, the emission of illumination from a lighting system and the wavelength band sensed by an imaging sensor can be defined by filters installed in the optical path of the sensor or lighting system (see, for example...). Figure 6 Filters 620-640). For example, a filter installed in the optical path of a first imaging sensor can be a bandpass filter adapted to filter out light in wavelength bands other than a first plurality of mutually separated wavelength bands, such that light having wavelengths within the first plurality of mutually separated wavelength bands is allowed to enter the first imaging sensor. Correspondingly, a filter installed in the optical path of a second imaging sensor can be a bandpass filter adapted to filter out light in wavelength bands other than a second plurality of mutually separated wavelength bands, such that light having wavelengths within the second plurality of mutually separated wavelength bands is allowed to enter the second imaging sensor. A filter installed in the optical path of an illumination system can be adapted to allow light in all first and second plurality of mutually separated wavelength bands to pass through.

[0044] In various embodiments, such as Figure 1aAs shown in 1b, a beam splitter 126 is used to direct light reflected or emitted from the sample to the first and second imaging sensors. For example, the beam splitter may be a polychromatic mirror configured to split the light, directing light of a given wavelength toward either the first or second imaging sensor. For instance, the polychromatic mirror may be adapted to direct light having wavelengths within a first plurality of mutually separated wavelength bands (only) to the first imaging sensor and light having wavelengths within a second plurality of mutually separated wavelength bands (only) to the second imaging sensor.

[0045] The system is configured to generate a composite color image based on data from first and second imaging sensors. In other words, the system is configured to generate a color image based on imaging sensor data provided by two imaging sensors. Therefore, the composite color image is a color image generated based on imaging sensor data provided by two different imaging sensors. For example, the composite color image could be a multispectral color image generated using imaging sensor data from two imaging sensors, where the imaging sensor data represents light in multiple mutually separate wavelength bands. Thus, the composite color image is a multispectral color image generated based on light sensed in multiple mutually separate wavelength bands by two different imaging sensors (e.g., one for reflection imaging and another for reflection and fluorescence imaging). For example, a composite color image can be generated based on data from first and second imaging sensors to improve the color accuracy of the composite color image.

[0046] A composite color image is generated using a transform function, which defines the transformation performed between imaging sensor data and the composite color image. In contrast to other methods, the transform function is configured such that a composite color image is generated using sensor data of light sensed in each of a first and second plurality of mutually separate wavelength bands. In other words, light sensed in all the first and second plurality of mutually separate wavelength bands can be combined to generate a composite color image, for example, to obtain a composite color image with improved color accuracy. Typically, the transform function can be viewed as a set of instructions for converting between the first and second imaging sensor data (one side) and the composite color image (the other side).

[0047] Generally, the first and second imaging sensor data may each include multiple pixels generated by multiple sensor pixels of the respective imaging sensor (e.g., after de-mosaicing). For each of the multiple pixels, the corresponding imaging sensor data may include multiple values ​​representing light sensed in multiple modulated wavelength bands in wavelength bands. For example, if light is sensed in three mutually separate wavelength bands by the first and / or second imaging sensors, then the first and / or second imaging sensor data may (each) include three values ​​for each pixel representing the light sensed in the three mutually separate wavelength bands. The multiple pixels of the first and second imaging sensor data may be in a predefined relationship. Ideally, the multiple pixels of the two imaging sensors may be generated such that each pixel of the first imaging sensor data represents the same point on the sample as the corresponding pixel of the second imaging sensor data.

[0048] For each pixel of a plurality of pixels, the system can be configured to input the pixel value from the first imaging sensor data and the corresponding pixel value from the second imaging sensor data into a transformation function and calculate the pixel value representing the plurality of color channels of the composite color image. For example, if the first and second plurality of mutually separated wavelength bands each comprise three wavelength bands (a total of six wavelength bands, thus a total of six values ​​representing light in the six wavelength bands), and if the composite color image is based on three color channels (e.g., red, green, and blue), the transformation function can specify performing a total of 18 (6 x 3) transformations between the imaging sensor data and the channels of the composite color image. These transformations can be defined as multiplication factors that can be multiplied by the values ​​of a single pixel in the first and second imaging sensor data. Therefore, the transformation function can be based on a set of transformation factors, each defining a transformation performed between the imaging sensor data of light sensed in a wavelength band (i.e., one of the plurality of pixel values) and the color channels of the composite color image (e.g., one of the three color channels). In the example above, the set of transformation factors could include 18 transformation factors. In other words, the transform factor set may include one transform factor for each combination of wavelength bands (i.e., the values ​​of pixels representing light sensed in a wavelength band) and color channels (i.e., the values ​​of the color channels used for pixels). For example, the transform factor set may provide a transformation between imaging sensor data for each of the first and second plurality of mutually separate wavelength bands and each color channel of a composite color image. The transform function, and therefore the transform factor set, may be applied (respectively) to each pixel of the first and second imaging sensor data.

[0049] To improve performance, the transformation function can be defined as a matrix that can be multiplied by a vector comprising data from both the first and second imaging sensors. In other words, the transformation function can be implemented using a transformation matrix. The terms of the transformation matrix can be defined by a set of transformation factors. The system can be configured to transform imaging sensor data in each of a plurality of mutually separate wavelength bands using the transformation matrix. For example, in Figure 5d An example of a transformation matrix is ​​given in the text.

[0050] As previously mentioned, a composite color image may include three color channels, such as a red channel, a blue channel, and a green channel (RGB). RGB is a channel model commonly used to represent color images, such as those intended to be displayed on a monitor. Each color channel of a composite color image can be generated based on a transformation of imaging sensor data from each of a first and second plurality of mutually separate wavelength bands. In other words, the values ​​of each of the first and second plurality of mutually separate wavelength bands can be used to calculate the value of each color channel. The transformation can be performed using a transformation function, as shown above, for example.

[0051] In various embodiments, the system is configured to provide a display signal to the microscope system's display 140 using the system 110's interface 112, so that the display shows a composite color image. In other words, the composite color image can be displayed on the microscope system's display, for example, on the microscope system's eye monitor or on the microscope system's auxiliary monitor.

[0052] Interface 112 may correspond to one or more inputs and / or outputs for receiving and / or sending information, which may be received and / or sent in digital (bit) values ​​according to specified codes within a module, between modules, or between modules of different entities. For example, interface 112 may include interface circuitry configured to receive and / or send information. In embodiments, one or more processors 114 may be implemented using one or more processing units, one or more processing devices, or any processing means, such as a processor, a computer, or a programmable hardware component operable by appropriately adapted software. In other words, the functionality of the one or more processors 114 described may also be implemented in software and then executed on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. In at least some embodiments, one or more storage devices 116 may include at least one element from the group consisting of: computer-readable storage media, such as magnetic or optical storage media, such as hard disk drives, flash memory, floppy disks, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or network storage.

[0053] Further details and aspects of the system and microscope system are described in conjunction with the proposed concept or one or more examples described above or below (e.g. Figures 2 to 8 The system and microscope system may include one or more additional optional features, corresponding to one or more aspects of the proposed concept, or one or more examples described above or below.

[0054] Figure 2 A flowchart illustrating an embodiment of a corresponding method for a microscope system is shown. The method includes acquiring 210 first imaging sensor data from a first imaging sensor of the microscope system and second imaging sensor data from a second imaging sensor of the microscope. The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separate wavelength bands. The second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separate wavelength bands. The wavelength bands of either the first plurality of mutually separate wavelength bands or the second plurality of mutually separate wavelength bands are wavelength bands used for fluorescence imaging. The method includes generating 220 a composite color image based on the first imaging sensor data and the second imaging sensor data. The composite color image is based on multiple color channels. The composite color image is generated using a transformation function that defines a transformation performed between the imaging sensor data and the composite color image, thereby generating the composite color image using sensor data of light sensed in each of the first and second plurality of mutually separate wavelength bands.

[0055] As described above, the features described with respect to system 110 and microscope system 10 in Figures 1a and / or 1b can also be applied in the same way. Figure 2 The method.

[0056] Further details and aspects of the method are proposed regarding the concepts presented or one or more examples described above or below (e.g., Figures 1a to 1b , Figures 3 to 8 The method may include one or more additional optional features, corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0057] Figures 1a to 2 illustrate the application of transformation functions in the generation of composite color images. The following... Figure 3 and Figure 4 This involves the generation of transformation functions. Therefore, the transformation functions used in Figures 1a to 2 can be used... Figure 3 or Figure 4 The system, method, and / or computer program shown are generated.

[0058] Figure 3A schematic diagram of a system for determining a transform function is shown. System 310 includes one or more processors 314 and one or more storage devices 316. Optionally, the system further includes an interface 312. One or more processors are coupled to the interface and one or more storage devices. Generally, the functionality of the system is provided by one or more processors, for example, in conjunction with the optional interface or with one or more storage devices. For example, the system may be configured to acquire imaging sensor data via the interface and / or store transform factors of the transform function and / or composite reference images using one or more storage devices.

[0059] The system is configured to acquire first imaging sensor data of a reference object 300 from a first imaging sensor 122 of a microscope 120, and second imaging sensor data of the reference object from a second imaging sensor 124 of the microscope. The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separate wavelength bands, and the second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separate wavelength bands. The wavelength bands of the first plurality of mutually separate wavelength bands or the second plurality of mutually separate wavelength bands are wavelength bands used for fluorescence imaging. The system is configured to acquire a composite reference image of the reference object. The composite reference image includes multiple color channels. The system is configured to determine a transformation function by determining a set of transformation factors that provides an approximate transformation between the imaging sensor data of each wavelength band of the first and second plurality of mutually separate wavelength bands and each color channel of the composite reference image (i.e., producing a mismatch between the generated composite color image and the composite reference image that is less than the mismatch produced using another set of transformation factors or less than a threshold). The transformation function is based on the set of transformation factors.

[0060] For example, Figure 3 The system can be implemented as similar to Figure 1a And / or system 110 of 1b, for example, through the same system. Accordingly, system 110 and / or system 310 can be configured to provide the functions of the corresponding other system 310; 110. Thus, microscope 120 can be Figure 1a And / or the microscope 120 of the microscope system 100 of 1b. Therefore, the data from the first and second imaging sensors can also be implemented in a similar manner. Figure 1a And / or 1b of the first and second imaging sensor data. Furthermore, the transform function can be implemented as similar to... Figures 1a to 2 The transformation function.

[0061] Some embodiments of this disclosure relate to systems, methods, and computer programs for determining transformation functions. As previously stated in Figures 1a to 2As noted in the document, a transformation function defines the transformation performed between imaging sensor data and a composite color image based on the imaging sensor data. Generally, a transformation function can be viewed as a set of instructions for converting between first and second imaging sensor data (one side) and the composite color image (the other side). Figure 3 and Figure 4 The illustrated embodiment provides a method for generating such a transformation function using a composite reference image and imaging sensor data provided by two imaging sensors.

[0062] Generally, composite reference images can be implemented similarly to combining... Figure 1a and / or Figure 1bThe composite color image introduced is a color image based on multiple color channels (e.g., three color channels—red, green, and blue). A composite reference image can differ from the composite color image in that the values ​​representing the color channels of each pixel are reference values, i.e., values ​​that define the transform function to be determined when applied to first and second imaging sensor data displaying a reference object. In other words, the first and second imaging sensor data represent the reference object, while the reference composite image represents the composite color image, which is the desired result of the transform function applied to the first and second imaging sensor data. Preferably, the composite reference image includes multiple predefined colors, whose corresponding predefined values ​​represent the color channels of each pixel. For example, the reference object can be a color chart or color table, i.e., a table showing multiple colors, each with a predefined representation in multiple channels of the composite reference image. Since the colors of the reference object are known from the composite reference image, the system can use the composite reference image as the desired result of applying the transform function to the imaging sensor data. Therefore, the system can be configured to identify a set of transform factors that define the transforms performed between the imaging sensor data and the composite color image, thereby reducing the differences between the composite color image and the composite reference image. Unlike other systems, this system identifies a set of transform factors for the imaging sensor data from two imaging sensors, potentially generating more than twice the number of transform factors. For example, it identifies a first subset of transform factors providing a transform between the first imaging sensor data and the composite color image, and a second subset of transform factors providing a transform between the first imaging sensor data and the composite color image, thereby reducing the difference between the composite color image and the composite reference image compared to other sets of transform factors. In other words, the system can be configured to identify a set of transform factors that produces a lower mismatch (i.e., difference) between the composite reference image and the transformed image (i.e., the composite color image) generated based on the transform factor set than at least one other set of transform factors. For example, the system can be configured to determine the set of transform factors by defining a set of equations using the first and second imaging sensor data and the composite reference image, and solving these equations to obtain a set of potential transform factors providing a transform between the first and second imaging sensor data and the composite reference image. This can be performed for multiple different colors, such as multiple colors for a reference object.

[0063] For example, a reference object may include multiple predefined colors (e.g., a color table), and a composite color image may include multiple portions representing the multiple predefined colors. In other words, a composite reference image of reference images may define multiple colors for multiple portions of the reference object, such as multiple predefined colors from a color table. The system may be configured to determine a set of transformation factors for the multiple (predefined) colors, i.e., a set of transformation factors identified by first and second imaging sensor data providing representations (predefined colors) for each of the multiple colors, and approximate transformations between the predefined colors represented by multiple channels of the composite reference image.

[0064] In applications such as surgical microscopy, some colors may be more useful to surgeons. For example, some colors may be markers of pathological tissue, but the difference between the color of pathological tissue and the color of healthy tissue may be small. In some embodiments, these colors may be prioritized when determining the set of transformation factors. For example, multiple (predefined) colors may include a predefined first subset of colors and a second subset of colors. For example, the predefined first subset of colors may be colors that are very important for the specific application of the microscope. The second subset of colors may be other colors, i.e., colors that are not important for the specific application of the microscope. For example, the predefined first subset of colors may be colors that appear as the color of organic tissue in surgical settings, such as colors that indicate healthy or pathological tissue. The colors of the second subset of colors may be colors that are less relevant to identifying healthy or pathological tissue. Accordingly, the system may be configured to identify a set of transformation factors for the predefined first subset of colors, where the mismatch produced by the set of transformation factors between the composite reference image and the transformed image generated based on the set of transformation factors is less than at least one other set of transformation factors. In some embodiments, the second subset of colors may not be considered when identifying the set of transformation factors. However, in some other embodiments, the second subset of colors may be considered, although its priority or weight is lower than that of the predefined first subset of colors. In other words, the system can be configured to identify a set of transform factors that, compared to at least one other set of transform factors, reduces the mismatch values ​​representing the mismatch with the composite reference image. Mismatch values ​​can be calculated for multiple colors. Mismatches in a predefined first subset of colors may have a greater impact on the mismatch values ​​than mismatches in a second subset of colors. In other words, when identifying the set of transform factors, mismatches in a predefined first subset of colors may receive higher weight than mismatches in a predefined second subset.

[0065] Embodiments of the present invention further provide a microscope system 100 (which can be implemented as similar to...) Figure 1aThe microscope system (and / or 1b) includes system 310 (and optionally system 110 if the system is implemented separately) and microscope 120 having a first imaging sensor 122 and a second imaging sensor 124. One of the first and second imaging sensors may be an imaging sensor adapted to provide fluorescence imaging functionality for the microscope system.

[0066] Interface 312 may correspond to one or more inputs and / or outputs for receiving and / or sending information, which may be received and / or sent in digital (bit) values ​​according to specified codes within a module, between modules, or between modules of different entities. For example, interface 312 may include interface circuitry configured to receive and / or send information. In embodiments, one or more processors 314 may be implemented using one or more processing units, one or more processing devices, or any processing means, such as a processor, a computer, or a programmable hardware component operable by appropriately adapted software. In other words, the described functionality of one or more processors 314 may also be implemented in software and then executed on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. In at least some embodiments, one or more storage devices 316 may include at least one element from the group consisting of: computer-readable storage media, such as magnetic or optical storage media, such as hard disk drives, flash memory, floppy disks, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or network storage.

[0067] Further details and aspects of the system and microscope system are illustrated in conjunction with the proposed concept or one or more examples described above or below (e.g., Figures 1a to 2, 4 to 8). The system and microscope system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0068] Figure 4A flowchart illustrating an embodiment of a method for determining a transformation function is shown. The method includes acquiring 410 first imaging sensor data of a reference object from a first imaging sensor of a microscope and second imaging sensor data of a reference object from a second imaging sensor of the microscope. The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separate wavelength bands. The second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separate wavelength bands. The wavelength bands of either the first plurality of mutually separate wavelength bands or the second plurality of mutually separate wavelength bands are wavelength bands used for fluorescence imaging. The method includes acquiring 420 a composite reference image of the reference object, the composite reference image including a plurality of color channels. The method includes determining 430 a transformation function by determining a set of transformation factors, the set of transformation factors providing an approximate transformation between the imaging sensor data of each wavelength band of the first and second plurality of mutually separate wavelength bands and each color channel of the composite reference image. The transformation function is based on the set of transformation factors.

[0069] As mentioned above, with Figure 3 The features related to system 310 can also be applied in the same way. Figure 4 The method.

[0070] With respect to the proposed concept or one or more examples described above or below (e.g. Figures 1a to 3 , Figures 5a to 8 This section elaborates on further details and aspects of the method. The method may include one or more additional optional features, corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0071] Various embodiments of the present invention are based on a fluorescence camera using a microscope system for multispectral reflectance imaging.

[0072] As noted earlier, surgeons often use the color of tissue (brain) to distinguish tissues suspected of being related to lesions. However, subtle differences in tissue color may only be visible to surgeons with extensive experience and trained visual acuity, and are difficult to learn.

[0073] Multispectral reflectance imaging can capture very small, even invisible, color differences by measuring extremely small spectral variations. However, for conventional multispectral imaging, additional hardware may be required beyond the multiple sensors typically found in modern microscopes equipped with 3D and fluorescence cameras.

[0074] Embodiments of the present invention enable multispectral reflectance imaging using existing fluorescence sensors in surgical microscope systems when fluorescence modes are ineffective. This can be enabled via software without increasing cost, scale, or complexity.

[0075] Various embodiments can use multispectral imaging to produce accurate color tissue images. This can be accomplished by measuring three or more spectral bands, such as six spectral bands (e.g., three spectral bands from data from first and second imaging sensors, respectively), and then digitally reconstructing the spectral information to calculate RGB values ​​(i.e., a combined color image). Therefore, color capture can be more accurate.

[0076] For imaging systems that use multiple sensors for reflection and fluorescence imaging, multispectral reflectance imaging can be achieved by utilizing the fluorescence band in the reflection mode. This can be achieved by providing illumination in the fluorescence band.

[0077] Subtle color differences that are of diagnostic importance can be enhanced by digital technology, making them easily visible even to untrained surgeons.

[0078] Figure 5a This diagram illustrates the different colors present in an image frame. Figure 5a In the diagram, the organic tissue (e.g., the brain tissue shown) has three regions 510-530, each with a specific color: region 510 is blue, region 520 is green, and region 530 is red.

[0079] Figure 5b and 5c The diagram illustrates the intensity of different colors in different regions 510-530, as perceived by two sensors (index 1 and 2) in different frequency bands 1-6. Each sensor senses light in three wavelength bands (AC), thereby generating imaging sensor data S. A1 S A2 S B1 S B2 S C1 and S C2 .like Figure 5c As shown, the sensor can sense a greater than Figure 5b The light in the narrower frequency band shown makes the frequency bands separate from each other. Different sensors are highlighted with different background patterns.

[0080] Sensor data generated in mutually separate frequency bands can be used to generate RGB images, for example, via a transformation matrix that includes a set of transformation factors. Figure 5d An exemplary transformation matrix is ​​shown, which is 3×6 in size and includes 18 transformation factors F. 11 -F 36 (Indices indicate rows and columns). Combine this transformation matrix with the sensor data S. A1 S A2 S B1 S B2 S C1 and SC2 Multiply by (6×1) matrices to obtain the RGB values ​​of the composite color image (as a 3×1 matrix).

[0081] Figure 6 A schematic diagram of the microscope and illumination system is shown. As previously described, sensor data from two imaging sensors are used: a first imaging sensor (e.g., a CCD sensor) 122 for reflective imaging and a second imaging sensor (e.g., a CCD sensor) 124 primarily for fluorescence imaging. The first imaging sensor can be configured to provide a color reflective image, and the second imaging sensor provides a fluorescence image (in a first operating mode) and a reflective image (e.g., in three bands, in a second operating mode). The two imaging sensors are combined with mirrors 630 and 640, which filter out light outside the frequency bands that the respective imaging sensors can perceive. A polychromatic mirror 650 is used to separate light into different wavelength bands and direct the corresponding wavelength bands to the sensor. An illumination system 610 (used in combination with a filter 620, which filters the light emitted by the illumination system) is used to illuminate the object imaged by the imaging sensors. Figure 6 Further illustrations show light in different wavelength bands (in different linear forms), wherein visible light 660 is allowed to enter the first imaging sensor and light in the emission wavelength band is allowed to enter the second imaging sensor, the light in the emission wavelength band being emitted either by the illumination system 670 or by the fluorescent material 680 of the object excited in the fluorescence excitation wavelength band (e.g., by visible light 660).

[0082] exist Figure 7a and Figure 7b The concept of generating transformation functions is illustrated in the diagram. As combined with... Figure 3 and Figure 4 The transformation function described above can be determined by using a reference color table with "known" RGB values ​​of color samples from the color table. Figure 7a A schematic diagram of an exemplary color table is shown, including color fields 1 to 20. The color table can be recorded by two imaging sensors, and corresponding sensor data can be determined for individual, mutually separate wavelength bands. Known RGB values ​​and sensor data for each wavelength band can be input into a system of equations, and the equations can be solved to reduce or minimize the difference between the obtained color and the known RGB values ​​for different color samples. Figure 7b A schematic diagram of an exemplary set of equations is shown, with columns representing the number of samples, columns representing sensor data at different wavelength bands, and columns representing known RGB values.

[0083] With respect to the proposed concept or one or more examples described above or below (e.g., Figures 1a to 1b) Figure 4, 8) together elaborate on more details and aspects of the concept. A concept may include one or more additional optional features, corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0084] Some embodiments relate to microscopes, including systems described in conjunction with one or more of the figures 1 through 7b. Alternatively, the microscope may also be part of or connected to the systems described in conjunction with one or more of the figures 1 through 7b. Figure 8 A schematic diagram of a microscope system, including a microscope and a computer system, is shown. Figure 8 A schematic diagram of a system 800 configured to perform the methods described herein is shown. The system includes a microscope 810 and a computer system 820. The microscope 810 is configured to capture images and is connected to the computer system 820. The computer system 820 is configured to perform at least a portion of the methods described herein. The computer system 820 may be configured to execute machine learning algorithms. The computer system 820 and the microscope 810 may be separate entities, but may also be integrated together in a common enclosure. The computer system 820 may be part of the central processing system of the microscope 810, and / or the computer system 820 may be part of a sub-component of the microscope 810, such as a sensor, actuator, camera, or illumination unit of the microscope 810.

[0085] Computer system 820 may be a local computer device (e.g., a personal computer, laptop, tablet, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in various locations, such as local clients and / or one or more remote server farms and / or data centers). Computer system 820 may include any circuitry or combination of circuitry. In one embodiment, computer system 820 may include one or more processors that can be of any type. As used herein, a processor may refer to any type of computing circuitry, such as, for example, a microscope or microscope component (e.g., a camera) or any other type of processor or processing circuitry, but not limited to, microprocessors, microcontrollers, complex instruction set computing (CISC) microprocessors, simplified instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, graphics processors, digital signal processors (DSPs), multi-core processors, and field-programmable gate arrays (FPGAs). Other types of circuitry that may be included in computer system 820 may be custom circuitry, application-specific integrated circuits (ASICs), or similar circuitry, such as, for example, one or more circuitry (e.g., communication circuitry) for wireless devices such as mobile phones, tablets, laptops, two-way radios, and similar electronic systems. Computer system 820 may include one or more storage devices, including one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for processing removable media, such as optical discs (CDs), flash memory cards, digital video disks (DVDs), etc. Computer system 820 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input information into and receive information from computer system 820.

[0086] Some or all of the method steps may be performed by (or using) hardware devices, such as processors, microprocessors, programmable computers, or electronic circuits. In some embodiments, some or more of the most important method steps may be performed by such devices.

[0087] Depending on the specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can be performed using non-transitory storage media, such as digital storage media like floppy disks, DVDs, Blu-ray discs, CDs, ROMs, PROMs, EPROMs, EEPROMs, or FLASH memories, storing electronically readable control signals that cooperate (or are capable of cooperating with) a programmable computer system to execute the corresponding methods. Therefore, the digital storage media may be computer-readable.

[0088] Some embodiments of the invention include a data carrier having electronically readable control signals, which is capable of cooperating with a programmable computer system to perform one of the methods described in the invention.

[0089] Generally, embodiments of the present invention can be implemented as a computer program product having program code, which, when run on a computer, can be used to execute one of the methods. For example, the program code can be stored on a machine-readable medium.

[0090] Other embodiments include a computer program for performing one of the methods described herein, stored on a machine-readable medium.

[0091] In other words, therefore, when a computer program is run on a computer, an embodiment of the present invention is a computer program having program code for performing one of the methods described in the present invention.

[0092] Therefore, a further embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) including a computer program stored thereon, which, when executed by a processor, is used to perform one of the methods described herein. Data carriers, digital storage media, or recorded media are generally tangible and / or non-transitory. A further embodiment of the invention is an apparatus comprising a processor and a storage medium as described herein.

[0093] Therefore, a further embodiment of the present invention represents a data stream or signal sequence for performing one of the methods described in the present invention. For example, the data stream or signal sequence may be configured to be transmitted via a data communication connection, such as via the Internet.

[0094] Further embodiments include processing means, such as a computer or programmable logic device configured or adapted to perform one of the methods described herein.

[0095] A further embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

[0096] Further embodiments of the invention include means or systems configured to transmit, for example, a computer program for performing one of the methods described herein to a receiver (e.g., electronically or optically). For example, the receiver may be a computer, mobile device, storage device, or similar device. For example, the means or system may include a file server for transmitting the computer program to the receiver.

[0097] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0098] The term “and / or” as used in this article includes any and all combinations of one or more related listed items, which may be abbreviated as “ / ”.

[0099] Although some aspects have been described in the context of the apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of the corresponding apparatus.

[0100] List of reference numerals

[0101] 100 Microscope System

[0102] 105 basic units

[0103] 110 system

[0104] 112 interface

[0105] 114 or more processors

[0106] 116 One or more storage devices

[0107] 120 microscope

[0108] 122 First Imaging Sensor

[0109] 124 Second Imaging Sensor

[0110] 126 beam splitter / polychromatic mirror

[0111] 130 lighting system

[0112] 140a / b monitor

[0113] 150 steering handle

[0114] 160 arms

[0115] 210 Acquire data from the first and second imaging sensors

[0116] 220 Generate composite color image

[0117] 300 reference objects

[0118] 310 system

[0119] 312 interface

[0120] 314 One or more processors

[0121] 316 One or more storage devices

[0122] 410 Acquire data from the first and second imaging sensors

[0123] 420 Acquisition of Composite Reference Image

[0124] 430 Determine the transformation function

[0125] Areas 510-530 with different colors

[0126] 610 Lighting System

[0127] 620-640 filter

[0128] 650 multicolor mirrors

[0129] 660 visible light

[0130] 670 Light emitted by the lighting system and reflected by the object

[0131] 680 Light reflected from the object

[0132] 800 system

[0133] 810 microscope

[0134] 820 Computer System

Claims

1. A system (110) for a microscope system (100), the system (110) comprising one or more processors (114) and one or more storage devices (116), wherein the system is configured to: First imaging sensor data is acquired from a first imaging sensor (122) of the microscope (120) of the microscope system, and second imaging sensor data is acquired from a second imaging sensor (124) of the microscope. The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separate wavelength bands, and the second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separate wavelength bands, wherein... The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging; and A composite color image is generated based on the data from the first imaging sensor and the data from the second imaging sensor. The composite color image is based on multiple color channels. The composite color image is generated using a transformation function defined as a transformation performed between imaging sensor data and the composite color image, thereby generating the composite color image using sensor data of light sensed at each of the first plurality of mutually separated wavelength bands and the second plurality of mutually separated wavelength bands.

2. The system of claim 1, wherein the transformation is based on a set of transformation factors, each transformation factor defining a transformation performed between imaging sensor data of light sensed in a wavelength band and the color channels of the composite color image.

3. The system of claim 2, wherein the set of transformation factors comprises one transformation factor for each combination of wavelength bands and color channels. and / or among them, The set of transformation factors provides a transformation between the imaging sensor data of each wavelength band of the first plurality of mutually separated wavelength bands and the second plurality of mutually separated wavelength bands and each color channel of the composite color image.

4. The system according to any one of claims 1 to 3, wherein the composite color image comprises three color channels, wherein each color channel is generated based on a transformation of imaging sensor data for each wavelength band of the first plurality of mutually separated wavelength bands and the second plurality of mutually separated wavelength bands, and the transformation is performed using the transformation function.

5. The system according to any one of claims 1 to 4, wherein the transformation function is implemented by a transformation matrix, wherein the system is configured to transform imaging sensor data for each of the first plurality of mutually separated wavelength bands and the second plurality of mutually separated wavelength bands using the transformation matrix.

6. The system according to any one of claims 1 to 5, wherein the system is configured to provide a display signal to a display (140) of the microscope system using an interface (112) of the system (110) to cause the display to display the composite color image.

7. A system (310) for determining a transformation function, the system comprising one or more processors (314) and one or more storage devices (316), wherein the system is configured to: First imaging sensor data of the reference object (300) is acquired from the first imaging sensor (122) of the microscope (120), and second imaging sensor data of the reference object is acquired from the second imaging sensor (124) of the microscope. The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separated wavelength bands, and the second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separated wavelength bands, wherein... The wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. Obtain a composite reference image of the reference object, the composite reference image including multiple color channels; as well as The transformation function is determined by determining a set of transformation factors, which provides an approximate transformation between imaging sensor data for each wavelength band of the first plurality of mutually separated wavelength bands and the second plurality of mutually separated wavelength bands and each color channel of the composite reference image, and the transformation function is based on the set of transformation factors.

8. The system according to claim 7, wherein, The system is configured to identify a set of transform factors that produce less mismatch between the composite reference image and the transformed image generated based on the set of transform factors than at least one other set of transform factors.

9. The system of claim 8, wherein the composite reference image of the reference image defines multiple colors for multiple portions of the reference object, the multiple colors including a predefined first subset of colors and a second subset of colors, wherein, The system is configured to identify a set of transformation factors that, for the predefined first color subset, produce a mismatch between the composite reference image and the transformed image generated based on the set of transformation factors that is less than at least one other set of transformation factors.

10. The system of claim 9, wherein the predefined first subset of colors is colors that appear as the color of organic tissue in surgical settings.

11. The system according to any one of claims 9 or 10, wherein the system is configured to identify a set of transform factors, the set of transform factors reducing mismatch values ​​representing mismatches with the composite reference image compared to at least one other set of transform factors, wherein the mismatch values ​​are calculated for the colors of the plurality of colors, wherein, The color mismatch of the predefined first color subset has a greater impact on the mismatch value than the color mismatch of the second color subset.

12. A microscope system (100) comprising a system (110) according to any one of claims 1 to 6 and a microscope (120) having a first imaging sensor (122) and a second imaging sensor (124), wherein one of the first imaging sensor and the second imaging sensor is an imaging sensor adapted to provide fluorescence imaging functionality of the microscope system.

13. A method for a microscope system, the method comprising: First imaging sensor data is acquired from a first imaging sensor of the microscope system, and second imaging sensor data (210) is acquired from a second imaging sensor of the microscope. The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separated wavelength bands, and the second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separated wavelength bands, wherein the wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging; and A composite color image (220) is generated based on the data from the first imaging sensor and the data from the second imaging sensor. The composite color image is based on multiple color channels. The composite color image is generated using a transformation function defined as a transformation performed between imaging sensor data and the composite color image, thereby generating the composite color image using sensor data of light sensed at each of the first plurality of mutually separated wavelength bands and the second plurality of mutually separated wavelength bands.

14. A method for determining a transformation function, the method comprising: First imaging sensor data of a reference object is acquired from a first imaging sensor of a microscope, and second imaging sensor data of a reference object is acquired from a second imaging sensor of a microscope (410). The first imaging sensor data includes sensor data of light sensed in a first plurality of mutually separated wavelength bands, and the second imaging sensor data includes sensor data of light sensed in a second plurality of mutually separated wavelength bands, wherein the wavelength bands of the first plurality of mutually separated wavelength bands or the second plurality of mutually separated wavelength bands are wavelength bands used for fluorescence imaging. Acquire a composite reference image (420) of the reference object, the composite reference image including multiple color channels; and A transformation function (430) is determined by determining a set of transformation factors, which provides an approximate transformation between imaging sensor data for each wavelength band of the first plurality of mutually separated wavelength bands and the second plurality of mutually separated wavelength bands and each color channel of the composite reference image, and the transformation function is based on the set of transformation factors.

15. A computer program having program code that, when the computer program is run on a processor, is used to perform at least one of the methods according to claim 13 or 14.

Images