System and method for digital pathology color calibration

Abstract

The present disclosure provides a system and method for color calibration of digital pathology. The system for color calibration of digital pathology includes: a non-transitory computer-readable medium storing instructions; a processor configured to obtain a first digital image of a specimen captured by a first imaging system in XYZ color, the first digital image including a plurality of XYZ pixels, each of which has an XYZ color value; obtain a second digital image of the specimen captured by a second imaging system in RGB color, the second digital image including a plurality of RGB pixels, each of which has an RGB color value; identify a plurality of first groups of image pixels in the first digital image based on the XYZ color values ​​of the XYZ pixels in the first digital image; identify a plurality of second groups of image pixels in the second digital image that respectively correspond to the first groups of image pixels in the first digital image; and generate a lookup table to associate each of the plurality of XYZ color values ​​from the first digital image with one of the plurality of RGB color values ​​from the second digital image.

Description

[0001] This application is a divisional application of the invention patent application with the application date of April 20, 2017, application number 201780024621.5 (international application number PCT / US2017 / 028532), and invention name “Digital Pathology Color Calibration and Verification”.

[0002] Related applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 325,330, filed April 20, 2016, which is incorporated herein by reference in its entirety. Technical Field

[0004] The present invention relates generally to digital pathology, and more particularly to systems and methods for calibrating color management in digital pathology systems and validating digital color data in conjunction with a corresponding physical color spectrum. Background Art

[0005] Accurately representing color information is a persistent challenge in many industries. The most common approach to addressing this challenge is to create specially constructed test targets that represent specific colors. In the photography industry, for example, these specially constructed test targets consist of arrays of rows and columns, with each cell having a constant color. For reflected light applications, these specially constructed test targets take the form of paper, while for transmitted light applications, they take the form of film. For nature photography, for example, color patches are spectrally matched to blue skies, green foliage, brown skin, and so on.

[0006] In the digital pathology industry, there is a significant problem: the color presented on the display is not the same as the color of the stained specimen. The lack of availability of any specially constructed test targets exacerbates this problem. This is generally because paper and / or membranes are inadequate substitutes for tissue as a medium for carrying stains containing color information. Attempts to use materials to produce such test targets to approximate tissue have been largely unsuccessful. For example, special biopolymer strips have been hand-made and stained with standard pathology stains. However, the manual production of these targets is complicated and expensive. In addition, the specially constructed biopolymer strips, when stained, generally cannot provide an accurate color match to the color of the same stain applied to the tissue. Therefore, the unresolved issue is how to construct a test target that approximates tissue and provides an accurate color match when the stain is applied to the test target.

[0007] Therefore, what is needed is a system and method that overcomes these significant problems found in conventional systems such as those described above. Summary of the Invention

[0008] To address the issues discussed above, described herein are systems and methods for calibrating a digital pathology slide scanning system so that the colors of the scanned specimen presented on a display are substantially the same as the colors of the stained or unstained specimen on a physical slide.

[0009] The inventors have recognized that because the spectral properties of a stain change when bound to tissue, a test target constructed to faithfully approximate the characteristics of tissue with or without the stain will always have significant flaws. This is because an adequate solution requires that the spectral properties of the test target match the spectral properties of the object being imaged. Therefore, in this description, the inventors provide a solution that employs one or more standard slides as test targets for successfully calibrating an imaging system.

[0010] In one aspect, a standard slide with a specimen is prepared. A single stain or a combination of multiple stains or no stain can be applied to the specimen. A grid is overlaid or superimposed on the slide to divide the specimen into discrete segments. A digital image of the specimen on the slide is then obtained using a hyperspectral imaging system (referred to herein as a "hyperspectral image," "H image," "HYP image," or "XYZ image"). Hyperspectral imaging systems are typically image tiling systems, and, for example, a single field of view of the hyperspectral imaging system can be captured for each cell in the grid overlay. The resulting hyperspectral image for each cell includes a stack of images ranging between 400 nm and 750 nm (visual spectrum). The number of images in the stack can vary, for example, with one image spaced every 10 nm. The hyperspectral image stack is then processed to produce an XYZ color image having multiple individual picture elements ("pixels"), each pixel having an XYZ color value. The hyperspectral image is then registered to the grid by mapping the upper left corner pixel to the upper left corner of the grid. Individual pixels in an XYZ color image can be combined to produce superpixels, which advantageously reduces pixel location errors when relating XYZ pixels produced by a hyperspectral imaging system to RGB pixels produced by a digital pathology system.

[0011] Next, the same slide with the grid overlay is scanned using a digital pathology system with color imaging capabilities. The resulting digital image (referred to herein as a "pathology image," "P-image," "PATH image," or "RGB image") has red, green, and blue ("RGB") values ​​for each pixel. The pathology image is then registered to the grid by mapping the top-left pixel to the top-left corner of the grid. Individual pixels in RGB can also be combined to create superpixels and the pixel size of the hyperspectral image can be matched to allow direct color comparison between XYZ values ​​and RGB values. Furthermore, the pixel size of the hyperspectral and pathology images can be reduced or enlarged to optimize pixel size matching.

[0012] Next, a lookup table ("LUT") is generated that correlates the XYZ color information of a single pixel in the hyperspectral image with the RGB color information of the same pixel in the pathology image. The LUT correlates the XYZ and RBG color information for all pixels in the pathology image. Advantageously, registering the hyperspectral image to the pathology image (including image pixel size mapping) results in a one-to-one pixel correlation in the LUT. However, it is also possible to include in the LUT the average RGB value of a combined number of pixels from the pathology image and the XYZ value of a single pixel from the hyperspectral image, or vice versa.

[0013] Once the LUT is generated, the display module can use the LUT to render the colors of a scanned specimen in a digital image file having RGB color data using the corresponding hyperspectral XYZ colors on the display, such that the displayed colors are substantially identical to the colors of the specimen on the physical slide. The displayed colors can be measured by a color measurement device such as a colorimeter or spectrophotometer.

[0014] In an alternative embodiment, a hyperspectral imaging system as described above is used to scan a standard slide containing a specimen to produce an XYZ image. The same slide is also scanned using a digital pathology system to produce an RGB image. The pixels of the XYZ and RGB images are registered with one another to align the respective images. If the individual pixels of the image sensors in the hyperspectral imaging system and the digital pathology imaging system differ in size, pixel binning or reduction can be employed to facilitate proper alignment of the XYZ and RGB images and proper image pixel size matching.

[0015] After scanning, one of the XYZ image or the RGB image is indexed to identify a small number of colors to which each pixel in the indexed image can be assigned, while also minimizing error values ​​for each association of image pixels to colors. For example, when a camera sensor may be capable of sensing millions of colors, the indexing process may advantageously reduce the number of colors in the XYZ image to ten and assign each pixel in the XYZ image to one of the ten colors. In one embodiment, during indexing, all pixels of approximately the same color are averaged into a single color of an indexed color palette. This process is performed iteratively until all pixels have been assigned to the averaged single color value, wherein error values ​​for the original pixel color compared to the averaged single color are minimized across all pixels in the XYZ image. In one embodiment, root mean square analysis may be used to minimize error values.

[0016] Once the XYZ image has been indexed, the result is a set of N pixel groups, where N is the index value (ten in the above example), and combining the pixels in the N pixel groups produces the complete XYZ image. Pixel groupings are also referred to herein as "indexes." After the XYZ image and the RGB image have been registered, the pixels in the RGB image can similarly be associated into the same N pixel groups based on the pixel registration between the XYZ image and the RGB image. Each of the N pixel groups from the RGB image is analyzed to calculate an average color value for each of the N pixel groups. The result is that each index in the XYZ image has an average color value, and each index in the RGB image has an average color value, and these average color values ​​are used to generate the LUT.

[0017] Once the LUT has been generated, the display module can use the LUT so that the colors of the scanned specimen in a digital image file having RGB color data can be presented on the display using the corresponding hyperspectral XYZ colors, so that the displayed colors (e.g., as measured by a colorimeter or spectrophotometer) are substantially the same as the colors of the specimen on the physical slide.

[0018] Other features and advantages of the present invention will become more readily apparent to those skilled in the art after reviewing the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The structure and operation of the present invention will be understood by reviewing the following detailed description and accompanying drawings, in which like reference numerals refer to like parts, and in which:

[0020] Figure 1 is a plan view illustrating an exemplary digital pathology slide with a barcode and a grid overlay according to an embodiment of the present invention;

[0021] Figure 2 is a block diagram illustrating an exemplary prior art hyperspectral imaging system according to an embodiment of the present invention;

[0022] Figure 3A is a block diagram illustrating an exemplary hyperspectral imaging stack produced by a hyperspectral imaging system according to an embodiment of the present invention;

[0023] Figure 3B is a graph illustrating exemplary color matching for use with a hyperspectral imaging stack according to an embodiment of the present invention;

[0024] Figure 3C is a block diagram illustrating an exemplary hyperspectral image in XYZ colors according to an embodiment of the present invention;

[0025] Figure 3D is a flow chart illustrating an exemplary process for converting a hyperspectral image stack into a single XYZ color image according to an embodiment of the present invention;

[0026] Figure 4A is a block diagram illustrating an exemplary image processor apparatus according to an embodiment of the present invention;

[0027] Figure 4B is a block diagram illustrating an exemplary superpixel of a hyperspectral image in XYZ color according to an embodiment of the present invention;

[0028] Figure 5 is a flow chart illustrating an exemplary process for calibrating color values ​​produced by a digital pathology scanning device using a superpixel process according to an embodiment of the present invention;

[0029] Figure 6 is a flow chart illustrating an exemplary process for verifying color values ​​produced by a digital pathology scanning device using a superpixel process according to an embodiment of the present invention;

[0030] Figure 7A 、 Figure 7B and Figure 7C is a graph illustrating an exemplary comparison of color values ​​of a specimen scanned by a hyperspectral imaging system relative to superpixelated color values ​​of the same specimen scanned by a digital pathology imaging system and presented on a display and measured by a color measurement device in accordance with an embodiment of the present invention;

[0031] Figure 8 is a block diagram illustrating an exemplary set of pixel groups forming a composite XYZ image according to an embodiment of the present invention;

[0032] Figure 9is a flow chart illustrating an exemplary process for calibrating color values ​​produced by a digital pathology scanning device using an indexing process according to an embodiment of the present invention;

[0033] Figure 10 is a flow chart illustrating an exemplary process for validating color values ​​produced by a digital pathology scanning device using an indexing process according to an embodiment of the present invention;

[0034] Figure 11A 、 Figure 11B and Figure 11C is a graph illustrating an exemplary comparison of color values ​​of a specimen scanned by a hyperspectral imaging system relative to indexed color values ​​of the same specimen scanned by a digital pathology imaging system and presented on a display and measured by a color measurement device in accordance with an embodiment of the present invention;

[0035] Figure 12A is a block diagram illustrating an exemplary processor-enabled device 550 that may be used in conjunction with various embodiments described herein;

[0036] Figure 12B is a block diagram illustrating an exemplary line scan camera having a single linear array;

[0037] Figure 12C is a block diagram illustrating an exemplary line scan camera having three linear arrays; and

[0038] Figure 12D is a block diagram illustrating an exemplary line scan camera having multiple linear arrays. DETAILED DESCRIPTION

[0039] Certain embodiments disclosed herein provide color calibration for digital pathology systems. For example, one embodiment disclosed herein allows a pathology slide to be scanned by a hyperspectral imaging system and the color of the resulting digital hyperspectral image is compared with the color of a digital image of the same slide scanned by the digital pathology system. The comparison generates a lookup table that translates RGB values ​​into XYZ values ​​so that when the digital slide image scanned by the digital pathology system is presented on a display, the presented color matches the XYZ values ​​corresponding to the true color of the physical specimen on the slide. After reading this description, those skilled in the art will understand how to implement the present invention in various alternative embodiments and alternative applications. However, although various embodiments of the present invention will be described herein, it should be understood that these embodiments are presented by way of example only and not by way of limitation. Therefore, this detailed description of the various alternative embodiments should not be understood as limiting the scope or breadth of the present invention, which is set forth in the appended claims.

[0040] Figure 1FIG2 is a plan view illustrating an exemplary digital pathology slide 10 having a barcode 20 and a grid overlay 30 according to an embodiment of the present invention. In the illustrated embodiment, the grid overlay 30 is positioned over the sample on the slide 10. The grid overlay 30 is used to facilitate registration of the hyperspectral digital image with the digital pathology digital image. An alternative way to register the images is to use image pattern matching of the two digital images to align the hyperspectral digital image and the digital pathology digital image. Digital images having the same magnification facilitate successful image pattern matching.

[0041] Figure 2 is a block diagram illustrating an exemplary prior art hyperspectral imaging system 50 according to an embodiment of the present invention. In the illustrated embodiment, the hyperspectral imaging system 50 includes a 2D pixel array monochrome camera 60, a microscope 70, a slide 80 for supporting a specimen, and a narrowband filter wheel 90. In one embodiment, the monochrome camera 60 of the hyperspectral imaging system 50 is a monochrome line scan camera. Preferably, the monochrome line scan camera 60 in the hyperspectral imaging system 50 has the same characteristics as the color line scan cameras used in digital pathology imaging systems. Employing line scan cameras with the same characteristics (e.g., pixel size) in hyperspectral and digital pathology imaging systems advantageously simplifies image registration by reducing or eliminating the need for pixel matching or allowing for easy resampling of pixels, such as by downsampling pixels into larger superpixels, to simplify registration of XYZ and RGB images.

[0042] Figure 3A is a block diagram illustrating an exemplary hyperspectral imaging stack 100 generated by a hyperspectral imaging system according to an embodiment of the present invention. Skilled artisans will appreciate that a hyperspectral imaging system generates a set of individual images, each of which is captured using light of a different wavelength, for example, by employing a narrowband filter wheel and capturing an image of the same region using each filter of the filter wheel. This set of images is referred to herein as a hyperspectral stack or spectral stack. Spectral stacks can be generated for individual regions of a sample or for the entire sample / slide. For example, in one embodiment, a hyperspectral imaging system employing a line scan camera can capture a full-slide image using each filter on the filter wheel to generate a full-slide image spectral stack.

[0043] Figure 3B is a graph illustrating exemplary color matching for use with a hyperspectral imaging stack according to an embodiment of the present invention. In the illustrated embodiment, a color matching function can be applied to a full-slide image spectral stack at multiple wavelengths of light (110, 120, and 130) to produce a full-slide hyperspectral image 140 in XYZ colors. Figure 3Cis a block diagram illustrating an exemplary whole-slide hyperspectral image 140 in XYZ colors, according to an embodiment of the present invention.

[0044] Figure 3D is a flow chart illustrating an exemplary process for converting a whole-slide hyperspectral image stack 100 into a single hyperspectral image 140 in XYZ colors according to an embodiment of the present invention. Initially, a hyperspectral imaging system captures whole-slide images at each different wavelength of light to produce the hyperspectral image stack 100. Subsequently, a color matching function is applied to each digital image in the spectral stack 100 at multiple wavelengths of light (110, 120, 130) to produce a whole-slide hyperspectral digital image 140 of the entire specimen in XYZ colors.

[0045] Figure 4A is a block diagram illustrating an exemplary image processor device 260 according to an embodiment of the present invention. In the illustrated embodiment, the image processor device 260 is a device having processor functionality, having a processor 267 and a non-transitory data storage area 265 for storing information and instructions that can be executed by the processor 267. For example, the data storage area 260 can store multiple hyperspectral XYZ images and multiple digital pathology RGB images and multiple instructions for processing such images. As shown in the illustrated embodiment, the image processor device 260 includes a registration module 270, a superpixel module 280, an index module 290, and a LUT module 295. In one embodiment, the superpixel module 280 and the index module 290 can be combined into a color module 285. The image processor device 260 can also be communicatively coupled to an integrated or external display device 576. In one embodiment, a color measurement device 577 can be configured to read color information from the display device 576 and translate the color information into one or more XYZ values.

[0046] The registration module 270 is configured to register two digital images with each other. For example, the registration module 270 is configured to register a hyperspectral XYZ image to a digital pathology RGB image. The registration module 270 registers the two digital images by aligning the image data of the two images in XY so that the two images achieve XY alignment (e.g., by pattern matching of features in the image data). The registration module 260 also registers the digital images by adjusting the two digital images so that they have common characteristics. For example, the registration model 270 can evaluate and adjust image characteristics, including image pixel size and spatial alignment of translation, rotation, and magnification. In addition, the registration module 270 can also take into account optical distortions between two separate systems, which can be detected at the pixel level.

[0047] The superpixel module 280 is configured to identify adjacent image pixels having the same or similar color values ​​and to combine those pixels into a single superpixel. The superpixel module 280 is also configured to determine the color value of a superpixel by averaging the color values ​​of all individual image pixels in the superpixel. Averaging is important for reducing noise due to possible measurement and registration errors. The average color value of the superpixel can be determined, for example, by summing the color values ​​of the image pixels in the superpixel and dividing the sum by the number of pixels in that superpixel. The superpixel module 280 can advantageously identify multiple superpixels in a digital image and determine the color value of each of the multiple superpixels.

[0048] Indexing module 290 is configured to identify individual image pixels having the same or similar color values ​​and assign these individual image pixels to one of a plurality of color indices. Indexing module 290 is also configured to determine a color value for each color index by averaging the color values ​​of all individual image pixels in the corresponding color index. Averaging is important for reducing noise due to possible measurement and registration errors. The average color value for an index can be determined, for example, by summing the color values ​​of the image pixels in the index and dividing the sum by the number of pixels in that index.

[0049] The LUT module 295 is configured to generate one or more lookup tables that relate XYZ color values ​​to RGB color values.

[0050] Figure 4B is a block diagram illustrating an exemplary registered whole slide image 170 according to an embodiment of the present invention. In the illustrated embodiment, the whole slide image 170 can be a hyperspectral image or a digital pathology image. During the registration process, image data from the hyperspectral digital image and the pathology digital image are analyzed to achieve XY alignment. For example, the image data can be analyzed to identify features in the image data that can be matched and aligned to register the hyperspectral digital image to the pathology digital image in XY. Advantageously, pattern matching can be used to associate common features between the hyperspectral digital image and the pathology digital image to facilitate XY alignment.

[0051] Additionally, during the registration process, the image data of the hyperspectral digital image and the pathology digital image are converted to common characteristics. This is because the imaging hardware of the hyperspectral scanning system and the pathology scanning system are unlikely to produce digital image data that is identical in terms of, for example, magnification and image pixel size within the digital image data. Therefore, during the registration process, the image data of the hyperspectral digital image and the pathology digital image are adjusted to have common characteristics. For example, magnification adjustment may be required, and adjustment to the common image pixel size is almost always required. In the illustrated embodiment, image pixels 180 are image pixels having a common image pixel size.

[0052] Figure 5 is a flow chart illustrating an exemplary process for calibrating color values ​​produced by a digital pathology scanning device using a superpixel process according to an embodiment of the present invention. Certain steps of the illustrated process may be replaced by, for example, those previously described with respect to Figure 4A The image processor device described is performed. Initially, in step 300, one or more test slides are prepared. A test slide is any type of slide that will be scanned by the hyperspectral imaging system and the digital pathology imaging system and used for calibration / validation purposes. Therefore, no special type of slide is required to serve as a test slide, and the test slide does not have special characteristics. Any slide with any stain can be used as a test slide to calibrate the digital pathology scanning device. This essentially deviates from all previous color calibration attempts and eliminates the problem of trying to produce test slides or test color samples. Using slides prepared in the normal manner using specimens and stains also allows the use of actual stains to calibrate the scanning device, and the actual stains are modified by applying them to the tissue that will be encountered during the production scan. This provides significant advantages. In addition, using multiple test slides can produce a combined LUT of color values ​​including multiple stains.

[0053] Additionally, in one embodiment, a registration grid may be overlaid on the slide, preferably over the portion of the slide that includes the sample. The registration grid, if present, may later be used as a marker in the hyperspectral digital image and the digital pathology digital image to register the hyperspectral digital image to the digital pathology digital image by aligning the image data in XY.

[0054] Next, in step 310, a hyperspectral image is scanned and stored. The hyperspectral image can be scanned as individual image tiles using tiling system hardware, or as a full-slide image using line scan system hardware. Advantageously, the present color calibration and verification system and method is hardware agnostic.

[0055] After scanning, the native hyperspectral image comprises one or more spectral stacks having a plurality of individual images, each of which is processed using a color matching function to produce a single digital image in XYZ color. Next, in step 320, a color digital pathology image is scanned and stored. The scanned digital pathology image is in RGB color. The digital pathology image can also be scanned as individual image tiles using a tiling system, or as a full-slide image using a line scan system. While there are advantages to using cameras with the same or very similar characteristics, such as pixel size and number of pixels, in the hyperspectral imaging system and the digital pathology imaging system, these advantages primarily serve to simplify the image registration process and make it more robust.

[0056] Next, in step 330, the hyperspectral image and the digital pathology image are registered with each other. Image registration includes XY alignment, for example, by pattern matching, and conversion to common characteristics, such as magnification and image pixel size. In some embodiments, the registration process includes generating resampled image pixels (larger or smaller) so that the individual image pixel sizes from the separate scanning systems are substantially the same. In some embodiments, the registration process can also include local changes in translation to account for optical distortions.

[0057] Once the hyperspectral image and the digital pathology image are registered with each other, in step 340, the color groups in the hyperspectral image are determined. In a simple embodiment, each individual image pixel in the hyperspectral image is its own color group. However, this will produce significant noise because the individual image pixels are very small and therefore the sample size of each color is also very small. To reduce noise, in step 340, adjacent individual image pixels of the same or very similar colors are combined into larger superpixels, such as in Figure 4B 190 is shown in FIG. The larger the superpixel, the larger the sample size, which has the advantage of reduced noise. However, a disadvantage of including more pixels in a superpixel is that this reduces the range of colors across all superpixels. Once a superpixel has been identified, the color of the superpixel determines the color group. In one embodiment, the color values ​​of all image pixels included in the superpixel are averaged together to determine an average color value, and the average color value is determined as the color of that color group, as shown in step 350.

[0058] Advantageously, the color groups in the hyperspectral image encompass every image pixel in the hyperspectral image, and each color group has an XY perimeter. Therefore, because the hyperspectral image and the digital pathology image have already been registered with each other, the XY perimeters of the color groups from the hyperspectral image can be applied to the digital pathology image, as shown in step 360, to associate individual image pixels of the digital pathology image with the same color groups as the hyperspectral image. Thus, the color values ​​of the individual image pixels of each color group in the digital pathology image can similarly be averaged in step 370 to determine an average color value for each color group in the digital pathology image.

[0059] Once the average color values ​​for each color group in the hyperspectral image and the average color values ​​for the same color group in the digital pathology image have been established, these color values ​​can be correlated to each other in a lookup table that correlates XYZ color values ​​with their associated RGB color values, as shown in step 380. In one embodiment, the lookup table can be embedded in the data structure containing the digital pathology image. In one embodiment, the correlation of XYZ color data with RGB color data can be included in the digital pathology image data structure as part of an International Color Consortium (ICC) profile. For example, as previously mentioned, the lookup table can be embedded in the digital pathology image data structure, or alternatively, the information in the lookup table can be converted into a mathematical model or formula or a set of executable instructions, and the model or formula or set of instructions can be embedded in the digital pathology image data structure. An advantage of embedding the model or formula or set of instructions is that the data size of the model or formula or set of instructions is smaller, thereby reducing the size of the digital pathology image data structure. Another advantage is that the model or formula or set of instructions serves to average out small differences and discrepancies in the correlation of XYZ color data to RGB color data that may be introduced due to metamerism. Metamerism is when two colors that are not actually the same (i.e., they reflect different wavelengths of light) appear the same under certain lighting conditions.

[0060] In one embodiment, a single combined lookup table is generated over time from multiple slides having multiple different stains. Advantageously, a single combined lookup table can be generated and optimized over time so that a single combined lookup table can be used for any type of digital pathology slide having any type of staining profile.

[0061] Figure 6 is a flow chart illustrating an exemplary process for verifying color values ​​generated by a digital pathology scanning device using a superpixel process according to an embodiment of the present invention. Certain steps of the illustrated process may be replaced by, for example, those previously described with respect to Figure 4AIn the illustrated embodiment, a test slide is initially prepared in step 400. As previously discussed, the test slide can be any slide prepared in the normal manner using a specimen and zero or more stains. Next, in step 410, for example, using the image processor apparatus previously described, Figure 5 The process described above is used to generate a lookup table. The lookup table can contain color values, such as those shown in the hyperspectral XYZ columns and the associated digital pathology RGB columns of Table 1 below, where each row represents the same color group (e.g., superpixel) in the hyperspectral digital image and the digital pathology digital image.

[0062] Table 1

[0063] Superpixel Hyperspectral XYZ Digital Pathology RGB Digital Pathology XYZ 1 0.777,0.631,0.551 225,147,169 0.770,0.625,0.556 2 0.743,0.574,0.500 219,130,154 0.750,0.570,0.510 3 0.712,0.426,0.454 213,117,140 0.719,0.431,0.450 4 0.683,0.485,0.413 208,105,128 0.678,0.481,0.418

[0064] Table 1 illustrates the correlation between hyperspectral image data in XYZ color values ​​and digital pathology image data in RGB color values ​​and digital pathology image data in XYZ color values.

[0065] Next, in step 420, the XYZ values ​​of the digital pathology image are determined. This can be accomplished in at least two ways. First, the XYZ values ​​of the digital pathology image can be calculated from the RGB values ​​of the digital pathology image, for example, using a lookup table or formula. Second, the digital pathology image can be presented on a display and the color emitted from the display can be measured using a color measurement device (e.g., a colorimeter or spectrophotometer) that measures the color in the XYZ values.

[0066] Finally, the calculated or measured XYZ values ​​for a particular region (e.g., a superpixel) of the digital pathology image are compared with the XYZ values ​​of the hyperspectral image for the same region in step 430. In this way, the color information generated by the digital pathology device and presented on the display screen can be verified.

[0067] Figure 7A 、 Figure 7B and Figure 7CGraphs illustrating an exemplary comparison of the color values ​​of a specimen scanned by a hyperspectral imaging system relative to the superpixelated color values ​​of the same specimen scanned by a digital pathology imaging system, according to an embodiment of the present invention. XYZ color values ​​from the digital pathology imaging system are obtained by presenting a digital slide image on a display and measuring the superpixel region using a color measurement device. Alternatively, because the superpixel region can be very small and therefore difficult to measure using a color measurement device, the entire display can be filled with the superpixel color values, and then the region can be measured using a color measurement device. As demonstrated by the graph, the measured off-screen color values ​​are very close to the true color values ​​measured by the hyperspectral imaging system, with an average difference of 2.18, which is less than a just noticeable difference.

[0068] exist Figure 7A In FIG, a graph 200 shows a comparison of the system hyperspectral value of brightness with the digital pathology display value of brightness. Figure 7B In FIG, graph 220 shows a comparison of the system hyperspectral values ​​for green / red with the digital pathology display values ​​for green / red. Similarly, in FIG. Figure 7C , graph 240 shows a comparison of the system hyperspectral values ​​for blue / yellow with the digital pathology display values ​​for blue / yellow. Furthermore, in each of graphs 200, 220, and 240, it is apparent that there are a very large number of individual comparisons. Notably, each individual comparison corresponds to a separate superpixel. After the registration process, although the number of image pixels in the hyperspectral image and the digital pathology image is still very large, the consequence of having a large number of superpixels is that the sample size (i.e., the number of image pixels) for each superpixel is small, and thus noise increases in the dataset.

[0069] Figure 8 6 is a block diagram illustrating an exemplary set of pixel groups that form a composite XYZ image 650 according to an embodiment of the present invention. In the illustrated embodiment, there are ten indices: index 1,600, index 2,605, index 3,610, index 4,615, index 5,620, index 6,25, index 7,630, index 8,635, index 9,640, and index 10,645. Each of the indices represents a separate color group of the underlying digital image. When combined, the ten indices produce the composite image 650. An index color palette 660 represents each of the color values ​​corresponding to each individual index.

[0070] As can be seen in the individual index images, each index image represents a dispersion of all image pixels in the base digital image that have the same color value within a certain threshold. The indexing process can be applied to either an XYZ digital image or an RGB digital image. Advantageously, the indexing process analyzes the color value of each pixel in the digital image and identifies all pixels, regardless of XY position, that belong to a single color value. In the illustrated embodiment, the entire digital pathology digital image, whether produced by a hyperspectral imaging system or a pathology imaging system, can be indexed into approximately ten (10) color values. A significant advantage of indexing all image pixels in a digital image into a relatively small number of color values ​​is that the sample size of each color value is significantly increased, which significantly reduces noise. Another advantage of indexing all image pixels in a digital image into a relatively small number of color values ​​is that the widest range of average color values ​​is provided by a minimum number of indices.

[0071] Figure 9 is a flow chart illustrating an exemplary process for calibrating color values ​​generated by a digital pathology scanning device using an indexing process according to an embodiment of the present invention. Certain steps of the illustrated process may be replaced by, for example, those previously described with respect to Figure 4A The image processor apparatus described herein is used for execution. Initially, in step 700, a slide is obtained. As previously discussed, any type of slide containing a specimen having zero or more stains is suitable. Next, in step 710, a hyperspectral image is scanned and stored as an XYZ image. As previously discussed, any type of scanning system hardware can be used to scan the hyperspectral image into individual image tiles. Next, in step 720, a color digital pathology image is scanned and stored as an RGB image.

[0072] Next, in step 730, the hyperspectral image and the digital pathology image are registered with each other, as previously described. Image registration includes XY alignment and conversion to common characteristics, such as image pixel size. Once the hyperspectral image and the digital pathology image are registered with each other, in step 740, the hyperspectral image is indexed to identify a set of colors to which each image pixel in the hyperspectral image can be assigned. In a simple embodiment, the indexing process receives an index value (i.e., the total number of indices) and sorts individual pixels into that number of color groups in a manner that minimizes the error associated with assigning each pixel to an index defined by a color value that differs from the color value of the corresponding pixel assigned to that index. For example, the indexing module 290 can be configured to use an index value of ten, fifteen, or twenty for any digital image. In a more complex embodiment, the indexing module 290 can be configured to analyze the digital image data to determine the optimal index value that assigns each pixel to the minimum number of indices, thereby minimizing error across the entire digital image.

[0073] Once all individual image pixels have been assigned to an index of the hyperspectral image, a color value for the corresponding index is determined by averaging the color values ​​of all individual image pixels in the corresponding index to determine an average color value, and the average color value is determined to be the color of the corresponding index of the hyperspectral image, as shown in step 750.

[0074] Advantageously, the combined index in the hyperspectral image includes every image pixel in the hyperspectral image. Therefore, because the hyperspectral image and the digital pathology image have already been registered with one another, each index from the hyperspectral image can be applied to the digital pathology image in step 760 so that the same individual image pixels included in the hyperspectral image index are grouped together in the corresponding index of the digital pathology image. This is possible because the hyperspectral image and the digital pathology image were previously registered with one another and their respective image pixel sizes were adjusted to be the same.

[0075] Once all individual image pixels of a digital pathology image have been assigned to an index, a color value for each respective index of the digital pathology image is determined by averaging the color values ​​of all individual image pixels in the respective index to determine an average color value, and determining the average color value to be the color of the respective index, as shown in step 770.

[0076] Once the average color value for each index in the hyperspectral image and the average color value for the same index have been established, these color values ​​can be correlated to each other in a lookup table that correlates XYZ color values ​​with their associated RGB color values, as shown in step 780. In one embodiment, the lookup table can be embedded in the data structure containing the digital pathology image. In one embodiment, the XYZ color data can be included in the digital pathology image data structure as part of the International Color Consortium (ICC) profile. As previously mentioned, the lookup table or mathematical model or formula or set of instructions can be embedded in the digital pathology image data structure.

[0077] As previously described, a single combined lookup table can advantageously be generated over time from multiple slides having multiple different stains. Advantageously, a single combined lookup table can be generated and optimized over time so that a single combined lookup table can be used for any type of digital pathology slide having any type of staining profile.

[0078] Figure 10 is a flow chart illustrating an exemplary process for verifying color values ​​generated by a digital pathology scanning device using an indexing process according to an embodiment of the present invention. Certain steps of the illustrated process may be replaced by, for example, those previously described with respect to Figure 4A Initially, a test slide is prepared in step 800. As previously discussed, the test slide can be any slide prepared in the normal manner using a specimen and zero or more stains. Next, in step 810, for example, using the image processor apparatus previously described Figure 9 The process described above is used to generate a lookup table. The lookup table may contain color values, such as those shown in the hyperspectral XYZ columns and the associated digital pathology RGB columns of Table 2 below, where each row represents a single color value (i.e., an index) in the hyperspectral digital image and the digital pathology digital image.

[0079] Table 2

[0080]

[0081]

[0082] Table 2 illustrates the correlation between the hyperspectral image data in XYZ color values ​​and the digital pathology image data in RGB color values ​​and the digital pathology image data in XYZ color values.

[0083] Next, the XYZ values ​​of the digital pathology image are determined in step 820. As previously discussed, this can be accomplished by calculating the XYZ values ​​of the digital pathology image based on its RGB values, or by presenting color values ​​across a display and measuring the color emitted from the display using a color measurement device that measures the color in the XYZ values.

[0084] Finally, in step 830, the calculated or measured XYZ values ​​for a particular color value (e.g., index) of the digital pathology image are compared with the XYZ values ​​of the hyperspectral image for the same index. In this way, the color information generated by the digital pathology device and presented on the display screen can be verified against the true color measured by the hyperspectral imaging system.

[0085] Figure 11A 、 Figure 11B and Figure 11C is a graph illustrating an exemplary comparison of indexed color values ​​of a specimen scanned by a hyperspectral imaging system, according to an embodiment of the present invention, relative to indexed color values ​​of the same specimen scanned by a digital pathology imaging system. The XYZ color values ​​from the digital pathology imaging system were obtained by presenting each indexed color value on a display and measuring a portion of the display using a color measurement device. As demonstrated by the graph, the measured off-screen color values ​​closely approximate the true color values ​​measured by the hyperspectral imaging system, with the average difference being less than a just noticeable difference.

[0086] exist Figure 11A In FIG, graph 210 shows a comparison of the system hyperspectral value of brightness with the digital pathology display value of brightness. Figure 11B In FIG, graph 230 shows a comparison of the system hyperspectral values ​​for green / red with the digital pathology display values ​​for green / red. Figure 11C In Figure 250, graph 250 shows a comparison of the system hyperspectral values ​​for blue / yellow with the digital pathology display values ​​for blue / yellow. Furthermore, in each of graphs 210, 230, and 250, it is apparent that there are very few individual comparisons. It is noteworthy that each individual comparison corresponds to a separate index. Advantageously, having a small number of indices results in a large number of pixels in each index, which therefore reduces noise in the dataset. Figure 11A 、 Figure 11B and Figure 11C and Figure 7A 、 Figure 7B and Figure 7C When compared, there are fewer measurements, but much less scatter due to noise. This demonstrates the advantage of having a very large number of image pixels in each index that form the basis for determining the average color.

[0087] Figure 12A 5 is a block diagram illustrating an exemplary processor-enabled device 550 that can be used in conjunction with the various embodiments described herein. A skilled artisan will appreciate that alternative forms of device 550 can also be used. In the illustrated embodiment, device 550 is presented as a digital imaging device (also referred to herein as a scanner system or scanning system) that includes: one or more processors 555; one or more memories 565; one or more motion controllers 570; one or more interface systems 575; one or more movable stages 580, each of which supports one or more glass slides 585 having one or more samples 590; one or more illumination systems 595 that illuminate the samples; one or more objectives 600, each of which defines an optical path 605 that travels along an optical axis; one or more objective positioners 630; one or more optional epi-illumination systems 635 (e.g., included in a fluorescence scanner system); one or more focusing optics 610; one or more line scan cameras 615 and / or one or more area scan cameras 620, each of which defines a separate field of view 625 on the sample 590 and / or the glass slide 585. The various elements of the scanner system 550 are communicatively coupled via one or more communication buses 560. Although there may be one or more of each of the various elements of the scanner system 550, in the following description, for simplicity, these elements will be described individually unless multiple descriptions are required to convey appropriate information.

[0088] The one or more processors 555 may include, for example, a central processing unit ("CPU") and a separate graphics processing unit ("GPU") capable of processing instructions in parallel, or the one or more processors 555 may include a multi-core processor capable of processing instructions in parallel. Additional separate processors may also be provided to control specific components or perform specific functions, such as image processing. For example, the additional processors may include auxiliary processors for managing data input, auxiliary processors for performing floating-point mathematical operations, specialized processors with an architecture suitable for quickly executing signal processing algorithms (e.g., digital signal processors), slave processors (e.g., back-end processors) at a level below the main processor, additional processors for controlling the line scan camera 615, the stage 580, the objective lens 225, and / or a display (not shown). Such additional processors may be separate discrete processors or may be integrated with the processor 555.

[0089] The memory 565 provides storage for data and instructions for programs that can be executed by the processor 555. The memory 565 may include one or more volatile and persistent computer-readable storage media that store data and instructions, such as random access memory, read-only memory, a hard drive, a removable storage drive, etc. The processor 555 is configured to execute the instructions stored in the memory 565 and communicate with the various elements of the scanner system 550 via the communication bus 560 to perform the overall functions of the scanner system 550.

[0090] The one or more communication buses 560 may include a communication bus 560 configured to carry analog electrical signals and may include a communication bus 560 configured to carry digital data. Thus, communications from the processor 555, motion controller 570, and / or interface system 575 via the one or more communication buses 560 may include both electrical signals and digital data. The processor 555, motion controller 570, and / or interface system 575 may also be configured to communicate with one or more of the various elements of the scanning system 550 via a wireless communication link.

[0091] The motion control system 570 is configured to precisely control and coordinate the XYZ movement of the stage 580 and the objective 600 (e.g., via the objective positioner 630). The motion control system 570 is also configured to control the movement of any other moving parts in the scanner system 550. For example, in a fluorescence scanner embodiment, the motion control system 570 is configured to coordinate the movement of optical filters, etc., in the epi-illumination system 635.

[0092] The interface system 575 allows the scanner system 550 to interface with other systems and human operators. For example, the interface system 575 can include a user interface to provide information directly to the operator and / or allow direct input from the operator. The interface system 575 is also configured to facilitate communication and data transfer between the scanner system 550 and one or more external devices (e.g., a printer, removable storage media) connected directly or connected to the scanner system 550 via a network (not shown), such as an image server system, an operator station, a user station, and a management server system. In one embodiment, the color measurement device 577 can be configured to read color information from the user interface 575 and translate the color information into one or more XYZ values.

[0093] The illumination system 595 is configured to illuminate a portion of the sample 590. The illumination system may include, for example, a light source and illumination optics. The light source may be a variable intensity halogen light source having a concave reflector to maximize light output and a KG-1 filter to suppress heat. The light source may also be any type of arc lamp, laser, or other light source. In one embodiment, the illumination system 595 illuminates the sample 590 in a transmission mode, such that the line scan camera 615 and / or the area scan camera 620 senses optical energy transmitted through the sample 590. Alternatively, or in combination, the illumination system 595 may also be configured to illuminate the sample 590 in a reflection mode, such that the line scan camera 615 and / or the area scan camera 620 senses optical energy reflected from the sample 590. In general, the illumination system 595 is configured to be suitable for investigating the microscopic sample 590 in any known mode of optical microscopy.

[0094] In one embodiment, the scanner system 550 optionally includes an epi-illumination system 635 to optimize the scanner system 550 for fluorescence scanning. Fluorescence scanning involves scanning a sample 590 that includes fluorescent molecules, which are photon-sensitive molecules that absorb light of a specific wavelength (excitation). These photon-sensitive molecules also emit light of a higher wavelength (emission). Because the efficiency of this photoluminescence phenomenon is very low, the amount of emitted light is often very low. This low amount of emitted light often hinders conventional techniques for scanning and digitizing the sample 590 (e.g., transmission mode microscopy). Advantageously, in the optional fluorescence scanner system embodiment of the scanner system 550, a line scan camera 615 comprising multiple linear sensor arrays (e.g., a time delay integration ("TDI") line scan camera) is used to increase the sensitivity of the line scan camera to light by exposing the same area of ​​the sample 590 to each of the multiple linear sensor arrays of the line scan camera 615. This is particularly useful when scanning fluorescent samples that have low emitted light.

[0095] Therefore, in the fluorescence scanner system embodiment, the line scan camera 615 is preferably a monochrome TDI line scan camera. Advantageously, monochrome images are ideal in fluorescence microscopy because they provide a more accurate representation of the actual signal from each channel present on the sample. Those skilled in the art will understand that a variety of fluorescent dyes emitting light of different wavelengths (also referred to as "channels") can be used to label the fluorescent sample 590.

[0096] Furthermore, because the low-end and high-end signal levels of various fluorescent samples exhibit a broad spectrum of wavelengths to be sensed by the line scan camera 615, it is desirable that the low-end and high-end signal levels that the line scan camera 615 can sense are similarly broad. Therefore, in the fluorescence scanner embodiment, the line scan camera 615 used in the fluorescence scanning system 550 is a monochrome 10-bit 64-bit linear array TDI line scan camera. It should be noted that a variety of bit depths of the line scan camera 615 can be employed for use with the fluorescence scanner embodiment of the scanning system 550.

[0097] The movable stage 580 is configured to perform precise XY movement under the control of the processor 555 or the motion controller 570. The movable stage can also be configured to move in Z under the control of the processor 555 or the motion controller 570. The movable stage is configured to position the sample at a desired location during image data capture by the line scan camera 615 and / or the area scan camera. The movable stage is also configured to accelerate the sample 590 to a substantially constant velocity in the scan direction and subsequently maintain the substantially constant velocity during image data capture by the line scan camera 615. In one embodiment, the scanner system 550 can employ a highly precise and tightly coordinated XY grid to assist in positioning the sample 590 on the movable stage 580. In one embodiment, the movable stage 580 is a linear motor-based XY stage with high-precision encoders employed on the X and Y axes. For example, highly precise nanometer encoders can be employed on axes in the scan direction, axes perpendicular to the scan direction, and in the same plane as the scan direction. The stage is also configured to support a glass slide 585 on which a sample 590 is disposed.

[0098] Sample 590 can be anything that can be investigated by an optical microscope. For example, glass microscope slides 585 are frequently used as viewing substrates for specimens including tissues and cells, chromosomes, DNA, proteins, blood, bone marrow, urine, bacteria, droplets, biopsy materials, or any other type of biological material or substance that is dead or alive, stained or unstained, labeled or unlabeled. Sample 590 can also be an array of DNA or DNA-related materials such as cDNA or RNA or proteins deposited on any type of slide or other substrate, including any and all samples commonly referred to as microarrays. Sample 590 can be a microtiter plate, such as a 96-well plate. Other examples of sample 590 include integrated circuit boards, electrophoresis records, culture dishes, membranes, semiconductor materials, forensic identification materials, or machined parts.

[0099] The objective lens 600 is mounted on an objective lens positioner 630, which, in one embodiment, can employ a very precise linear motor to move the objective lens 600 along the optical axis defined by the objective lens 600. For example, the linear motor of the objective lens positioner 630 can include a 50 nanometer encoder. The relative positions of the stage 580 and the objective lens 600 in the XYZ axes are coordinated and controlled in a closed-loop manner using a motion controller 570 under the control of a processor 555 that employs a memory 565 for storing information and instructions comprising computer-executable programmed steps for the operation of the entire scanning system 550.

[0100] In one embodiment, objective 600 is a plan apochromatic ("APO") infinity-corrected objective having a numerical aperture corresponding to the highest desired spatial resolution, wherein objective 600 is suitable for transmitted-mode illumination microscopy, reflected-mode illumination microscopy, and / or epi-illumination fluorescence microscopy (e.g., an Olympus 40X, 0.75NA or 20X, 0.75NA). Advantageously, objective 600 is corrected for chromatic and spherical aberrations. Because objective 600 is infinity-corrected, focusing optics 610 can be placed in optical path 605 above objective 600, where the light beam passing through the objective becomes a collimated beam. Focusing optics 610 focuses the optical signal captured by objective 600 onto the photoresponsive elements of line scan camera 615 and / or area scan camera 620 and can include optical components such as filters, a magnification converter lens, etc. The combination of objective 600 and focusing optics 610 provides the total magnification of scanning system 550. In one embodiment, focusing optics 610 may include a tube lens and an optional 2X magnification changer. Advantageously, the 2X magnification changer allows a native 20X objective 600 to scan sample 590 at a 40X magnification.

[0101] The line scan camera 615 includes at least one linear array of picture elements ("pixels"). The line scan camera can be monochrome or color. Color line scan cameras typically have at least three linear arrays, while monochrome line scan cameras can have a single linear array or multiple linear arrays. Any type of single linear array or multiple linear array can also be used, whether packaged as part of the camera or custom integrated into the imaging electronics module. For example, a 3-linear array ("red-green-blue" or "RGB") color line scan camera or a 96-linear array monochrome TDI can also be used. TDI line scan cameras typically provide a substantially better signal-to-noise ratio ("SNR") in the output signal by summing the intensity data from previously imaged areas of the specimen, resulting in an increase in SNR proportional to the square root of the number of integrated stages. TDI line scan cameras include multiple linear arrays, for example, TDI line scan cameras with 24, 32, 48, 64, 96 or even more linear arrays are available. The scanner system 550 also supports linear arrays manufactured in a variety of formats, including some with 512 pixels, some with 1024 pixels, and others with up to 4096 pixels. Similarly, linear arrays with a variety of pixel sizes can also be used in the scanner system 550. A significant requirement for selecting any type of line scan camera 615 is that the motion of the stage 580 can be synchronized with the line rate of the line scan camera 615 so that the stage 580 can be moved relative to the line scan camera 615 during the capture of a digital image of the sample 590.

[0102] The image data generated by the line scan camera 615 is stored as part of the memory 565 and processed by the processor 555 to generate a contiguous digital image of at least a portion of the sample 590. The processor 555 may further process the contiguous digital image and may also store a revised contiguous digital image in the memory 565.

[0103] In an embodiment having two or more line scan cameras 615, at least one of the line scan cameras 615 can be configured to act as a focus sensor that operates in combination with at least one of the line scan cameras that is configured to act as an image sensor. The focus sensor can be logically located on the same optical path as the image sensor, or the focus sensor can be logically located before or after the image sensor relative to the scanning direction of the scanner system 550. In this embodiment in which at least one line scan camera 615 acts as a focus sensor, image data generated by the focus sensor is stored as part of the memory 565 and processed by the one or more processors 555 to generate focus information, thereby allowing the scanner system 550 to adjust the relative distance between the sample 590 and the objective lens 600 during scanning to maintain focus on the sample.

[0104] In operation, the various components of the scanner system 550 and the program modules stored in the memory 565 enable automated scanning and digitization of a sample 590 disposed on a glass slide 585. The glass slide 585 is securely positioned on the movable stage 580 of the scanner system 550 for scanning the sample 590. Under the control of the processor 555, the movable stage 580 accelerates the sample 590 to a substantially constant speed for sensing by the line scan camera 615, wherein the stage speed is synchronized with the line rate of the line scan camera 615. After scanning a stripe of image data, the movable stage 580 decelerates and brings the sample 590 to a substantially complete stop. The movable stage 580 then moves orthogonally to the scanning direction to position the sample 590 for scanning subsequent stripes of image data, e.g., adjacent stripes. Additional stripes are then scanned until an entire portion of the sample 590 or the entire sample 590 has been scanned.

[0105] For example, during a digital scan of sample 590, a contiguous digital image of sample 590 is acquired as a plurality of contiguous fields of view, which are combined together to form an image strip. Similarly, a plurality of contiguous image strips are combined together to form a contiguous digital image of a portion or the entire sample 590. Scanning sample 590 may include acquiring vertical image strips or horizontal image strips. The scanning of sample 590 may be from top to bottom, from bottom to top, or both (bidirectional) and may begin at any point on the sample. Alternatively, the scanning of sample 590 may be from left to right, from right to left, or both (bidirectional) and may begin at any point on the sample. In addition, the image strips need not be acquired in a contiguous or contiguous manner. Furthermore, the resulting image of sample 590 may be an image of the entire sample 590 or only a portion of the sample 590.

[0106] In one embodiment, computer-executable instructions (e.g., program modules and software) are stored in memory 565 and, when executed, enable the scanning system 550 to perform the various functions described herein. In this description, the term "computer-readable storage medium" is used to refer to any medium for storing computer-executable instructions and providing the computer-executable instructions to the scanning system 550 for execution by the processor 555. Examples of such media include memory 565 and any removable or external storage media (not shown) that is communicatively coupled to the scanning system 550, either directly or indirectly, such as via a network (not shown).

[0107] Figure 12B A line scan camera is illustrated having a single linear array 640, which can be implemented as a charge coupled device ("CCD") array. The single linear array 640 includes a plurality of individual pixels 645. In the illustrated embodiment, the single linear array 640 has 4096 pixels. In alternative embodiments, the linear array 640 can have more or fewer pixels. For example, common formats for linear arrays include 512, 1024, and 4096 pixels. The pixels 645 are arranged in a linear manner to define a field of view 625 of the linear array 640. The size of the field of view varies depending on the magnification of the scanner system 550.

[0108] Figure 12C A line scan camera is illustrated having three linear arrays, each of which can be implemented as a CCD array. The three linear arrays are combined to form a color array 650. In one embodiment, each individual linear array in the color array 650 detects a different color intensity, such as red, green, or blue. The color image data from each individual linear array in the color array 650 is combined to form a single field of view 625 of color image data.

[0109] Figure 12D A line scan camera is illustrated having multiple linear arrays, each of which can be implemented as a CCD array. The multiple linear arrays are combined to form a TDI array 655. Advantageously, a TDI line scan camera can provide a substantially better SNR in its output signal by summing intensity data from previously imaged regions of a specimen, resulting in an increase in SNR proportional to the square root of the number of linear arrays (also referred to as integrated stages). A TDI line scan camera can include a wider variety of numbers of linear arrays, for example, common formats for TDI line scan cameras include 24, 32, 48, 64, 96, 120, or even more linear arrays.

[0110] Exemplary embodiments

[0111] The disclosure of the present application may be embodied in a system comprising: a non-transitory computer-readable medium configured to store data and executable program modules; at least one processor communicatively coupled to the non-transitory computer-readable medium and configured to execute instructions stored thereon; a registration module stored in the non-transitory computer-readable medium and configured to be executed by the processor, the registration module being configured to: obtain a first digital image of a specimen in XYZ color, the first digital image having a plurality of image pixels having a first image pixel size; obtain a second digital image of the specimen in RGB color, the second digital image having a plurality of image pixels having a second image pixel size; convert the first digital image and the second digital image to a common image pixel size; and align the converted image pixels of the first digital image with corresponding converted image pixels of the second digital image. Such a system may be implemented as a device having processor functionality, such as previously described with respect to Figure 4A and Figures 12A-12D The digital imaging device or image processing device described.

[0112] The disclosure of the present application can also be embodied in a system, the system comprising: a non-transitory computer-readable medium configured to store data and executable program modules; at least one processor communicatively coupled to the non-transitory computer-readable medium and configured to execute instructions stored thereon; a registration module stored in the non-transitory computer-readable medium and configured to be executed by the processor, the registration module being configured to: obtain a first digital image of a specimen in XYZ color, the first digital image having a plurality of image pixels having a first image pixel size; obtain a second digital image of the specimen in RGB color, the second digital image having a plurality of image pixels having a second image pixel size. image pixels; aligning image data of the first digital image with corresponding image data of the second digital image; and converting the first digital image and the second digital image to a common image pixel size, wherein the converted image pixels of the first digital image are aligned with corresponding converted image pixels of the second digital image; a lookup table module stored in the non-transitory computer-readable medium and configured to be executed by the processor, the lookup table module configured to: generate a lookup table to associate XYZ color values ​​of a plurality of converted image pixels of the first digital image with RGB color values ​​of a plurality of corresponding converted image pixels of the second digital image. Such a system can be implemented as a device having processor functionality, such as previously described with respect to Figure 4A and Figures 12A-12D The digital imaging device or image processing device described.

[0113] The disclosure of the present application may be embodied in a system, the system comprising: a non-transitory computer-readable medium configured to store data and executable program modules; at least one processor communicatively coupled to the non-transitory computer-readable medium and configured to execute instructions stored thereon; a registration module stored in the non-transitory computer-readable medium and configured to be executed by the processor, the registration module configured to: obtain a first digital image of a specimen in XYZ color, the first digital image having a plurality of image pixels having a first image pixel size; obtain a second digital image of the specimen in RGB color, the second digital image having a plurality of image pixels having a second image pixel size; align image data of the first digital image with corresponding image data of the second digital image; and convert the first digital image and the second digital image to a common image pixel size, wherein the converted image pixels of the first digital image are aligned with corresponding converted image pixels of the second digital image; a color module storing In the non-transitory computer-readable medium and configured to be executed by the processor, the color module is configured to: identify a first group of image pixels in the first digital image, wherein each image pixel in the first group of image pixels in the first digital image has substantially the same XYZ color value; determine an average XYZ color value of the first group of image pixels in the first digital image; identify a second group of image pixels in the second digital image, wherein each image pixel in the second group of image pixels in the second digital image corresponds to an image pixel in the first group of image pixels in the first digital image; and determine an average RGB color value of the second group of image pixels in the second digital image; a lookup table module, stored in the non-transitory computer-readable medium and configured to be executed by the processor, the lookup table module is configured to: generate a lookup table to associate the average XYZ color value of the first group of image pixels in the first digital image with the average RGB color value of the corresponding second group of image pixels in the second digital image. Such a system can be implemented as a device having processor functionality, such as previously described with respect to Figure 4A and Figures 12A-12D The digital imaging device or image processing device described.

[0114] Any of the three system embodiments described above may further embody wherein each of the image pixels in the first set of image pixels in the first digital image is adjacent to at least one other image pixel in the first set of image pixels in the first digital image.

[0115] Alternatively, any of the three system embodiments described above may further embody the invention wherein at least some of the image pixels in the first group of image pixels are non-adjacent, and further wherein the color module is further configured to identify a plurality of first groups of image pixels in the first digital image, wherein each image pixel in the first group of image pixels in the first digital image has substantially the same XYZ color value.

[0116] The disclosure of the present application can also be embodied in a technical system, the technical system comprising: a non-transitory computer-readable medium configured to store an executable program module; and at least one processor communicatively coupled to the non-transitory computer-readable medium, the at least one processor configured to execute instructions to perform steps comprising: scanning a specimen using a hyperspectral imaging system to generate a first digital image of the specimen in XYZ color; scanning the same specimen using a digital pathology imaging system to generate a second digital image of the specimen in RGB color; registering the first digital image to the second digital image to align the image data in the first digital image and the second digital image; and generating a lookup table that associates the XYZ color of the first digital image with the RGB color of the second digital image. Such a system can be implemented as a device having processor functionality, such as the one previously described with respect to Figure 4A and Figures 12A-12D The digital imaging device or image processing device described.

[0117] This system embodiment may further include providing the XYZ color data to a display module for presenting the second digital image on a display.

[0118] This system embodiment may further include storing the XYZ color data as part of the second digital image.

[0119] This system embodiment may further include using pattern matching to register the first digital image to the second digital image.

[0120] This system embodiment may further include overlaying a grid over the specimen prior to generating the first and second digital images, and using the grid in the first and second digital images to register the first digital image to the second digital image.

[0121] This system embodiment may further include combining pixels in one or more of the first digital image and the second digital image such that a pixel size in the first digital image is substantially the same as the pixel size in the second digital image.

[0122] This system embodiment may also include generating a single lookup table for a single stain.

[0123] This system embodiment may also include generating a single lookup table for multiple stains.

[0124] The disclosure of the present application can also be embodied in a technology system, the technology system comprising: a non-transitory computer-readable medium configured to store executable program modules; and at least one processor communicatively coupled to the non-transitory computer-readable medium, the at least one processor configured to execute instructions to perform steps including: obtaining a first digital image of a specimen scanned by a first imaging system to generate the first digital image of the specimen in XYZ color, the first digital image having a plurality of image pixels having a first image pixel size; obtaining a second digital image of the specimen scanned by a second imaging system to generate the second digital image of the specimen in RGB color, the second digital image having a plurality of image pixels having a second image pixel size; registering the first digital image to the second digital image so that image data in the first digital image and the second digital image are aligned; presenting the second digital image on a display; using a color measurement device to measure XYZ values ​​of the color presented in a first area on the display; and comparing the measured XYZ values ​​of the first area with the XYZ values ​​of the first digital image of the first area to validate a digital pathology system. Such a system can be implemented as a device having processor functionality, such as previously described with respect to Figure 4A and Figures 12A-12D The digital imaging device or image processing device described.

[0125] The disclosure of the present application can also be embodied in a method, the method comprising: obtaining a first digital image of a specimen scanned by a first imaging system to generate the first digital image of the specimen in XYZ color, the first digital image having a plurality of image pixels having a first image pixel size; obtaining a second digital image of the specimen scanned by a second imaging system to generate the second digital image of the specimen in RGB color, the second digital image having a plurality of image pixels having a second image pixel size; generating a lookup table to associate the XYZ color of the first digital image with the RGB color of the second digital image. This method can be implemented by a system, for example, as previously described with respect to Figure 4A and Figures 12A-12D The digital imaging device or image processing device described.

[0126] This method embodiment may further include aligning image data in the first digital image with image data in the second digital image; and generating a lookup table based on the alignment to associate XYZ colors of the first digital image with corresponding RGB colors of the second digital image.

[0127] This method embodiment may also include, wherein the first digital image includes a plurality of image pixels having a first image pixel size and the second digital image includes a plurality of image pixels having a second image pixel size, converting the image pixels of the first digital image and the image pixels of the second digital image to a common image pixel size, and aligning the image pixels of the converted first digital image with corresponding image pixels of the converted second digital image.

[0128] This method embodiment may also include: identifying a first group of image pixels in the first digital image, wherein each image pixel in the first group of image pixels in the converted first digital image has substantially the same XYZ color value; determining an average XYZ color value of the first group of image pixels in the converted first digital image; identifying a second group of image pixels in the converted second digital image, wherein each image pixel in the second group of image pixels in the converted second digital image corresponds to an image pixel in the first group of image pixels in the converted first digital image; determining an average RGB color value of the second group of image pixels in the converted second digital image; and generating a lookup table to associate the average XYZ color value of the first group of image pixels in the converted first digital image with the corresponding average RGB color value of the second group of image pixels in the converted second digital image.

[0129] The disclosure of the present application can also be embodied in a method comprising: obtaining a first digital image of a specimen scanned by a first imaging system to generate the first digital image of the specimen in XYZ color, the first digital image having a plurality of image pixels having a first image pixel size; obtaining a second digital image of the specimen scanned by a second imaging system to generate the second digital image of the specimen in RGB color, the second digital image having a plurality of image pixels having a second image pixel size; aligning image data in the first digital image with image data in the second digital image; and generating a lookup table based on the alignment to associate the XYZ colors of the first digital image with corresponding RGB colors of the second digital image. This method can be implemented by a system, for example, as previously described with respect to Figure 4A and Figures 12A-12D The digital imaging device or image processing device described.

[0130] This method embodiment may also include, wherein the first digital image includes a plurality of image pixels having a first image pixel size and the second digital image includes a plurality of image pixels having a second image pixel size, converting the first digital image and the second digital image to a common image pixel size, and aligning the converted image pixels of the first digital image with corresponding converted image pixels of the second digital image.

[0131] This method embodiment may also include: identifying a plurality of first groups of image pixels in the converted first digital image, wherein each image pixel in each of the plurality of first groups of image pixels in the converted first digital image has substantially the same XYZ color value; determining an average XYZ color value for each of the plurality of first groups of image pixels in the converted first digital image; identifying a corresponding plurality of second groups of image pixels in the converted second digital image, wherein each image pixel in each of the plurality of second groups of image pixels in the converted second digital image corresponds to an image pixel in the converted first digital image; determining an average RGB color value for each of the plurality of second groups of image pixels in the converted second digital image; and generating a lookup table to associate the average XYZ color value of each of the first group of image pixels in the converted first digital image with the corresponding average RGB color value of the corresponding second group of image pixels in the converted second digital image.

[0132] The disclosure of the present application can also be embodied in a method comprising: obtaining a first digital image of a specimen scanned by a first imaging system to generate the first digital image of the specimen in XYZ color, the first digital image having a plurality of image pixels having a first image pixel size; obtaining a second digital image of the specimen scanned by a second imaging system to generate the second digital image of the specimen in RGB color, the second digital image having a plurality of image pixels having a second image pixel size; registering the first digital image to the second digital image so that the image data in the first digital image and the second digital image are aligned; and generating a lookup table that associates the XYZ color of the first digital image with the RGB color of the second digital image. This method can be implemented by a system, for example, as previously described with respect to Figure 4A and Figures 12A-12D The digital imaging device or image processing device described.

[0133] This method embodiment may further include providing the XYZ color data to a display module for presenting the second digital image on a display.

[0134] This method embodiment may further include storing the XYZ color data as part of the second digital image file.

[0135] This method embodiment may further include using pattern matching to register the first digital image to the second digital image.

[0136] This method embodiment may further include overlaying a grid over the specimen prior to generating the first and second digital images, and using the grid in the first and second digital images to register the first digital image to the second digital image.

[0137] This method embodiment may further include combining pixels in one or more of the first digital image and the second digital image such that a first image pixel size in the first digital image is substantially the same as a second image pixel size in the second digital image.

[0138] This method embodiment may further comprise generating a single lookup table for a single stain.

[0139] This method embodiment may further include generating a single lookup table for multiple stains.

[0140] The disclosure of the present application can also be embodied in a method comprising: obtaining a first digital image of a specimen scanned by a first imaging system to generate the first digital image of the specimen in XYZ color, the first digital image having a plurality of image pixels having a first image pixel size; obtaining a second digital image of the specimen scanned by a second imaging system to generate the second digital image of the specimen in RGB color, the second digital image having a plurality of image pixels having a second image pixel size; registering the first digital image to the second digital image so that image data in the first digital image and the second digital image are aligned; presenting the second digital image on a display; using a color measurement device to measure XYZ values ​​of the color presented in a first area on the display; and comparing the measured XYZ values ​​of the first area with the XYZ values ​​of the first digital image of the first area to validate a digital pathology system. This method can be implemented by a system, for example, as previously described with respect to Figure 4A and Figures 12A-12D The digital imaging device or image processing device described.

[0141] The disclosure of the present application can also be embodied in a method comprising: obtaining a first digital image of a specimen scanned by a first imaging system to produce the first digital image of the specimen in XYZ color, the first digital image having a plurality of image pixels having a first image pixel size; obtaining a second digital image of the specimen scanned by a second imaging system to produce the second digital image of the specimen in RGB color, the second digital image having a plurality of image pixels having a second image pixel size; registering the first digital image to the second digital image so that image data in the first digital image and the second digital image are aligned; converting the first digital image and the second digital image to a common image pixel size; identifying a plurality of first groups of image pixels in the converted first digital image, wherein each image pixel in each group of the plurality of first groups of image pixels in the converted first digital image has substantially the same XYZ color value; and determining that each group of the plurality of first groups of image pixels in the converted first digital image has a substantially identical XYZ color value. the average XYZ color value of each of the first group of image pixels in the converted first digital image; identifying a corresponding plurality of second groups of image pixels in the converted second digital image, wherein each image pixel in each of the plurality of second groups of image pixels in the converted second digital image corresponds to an image pixel in the converted first digital image; determining an average RGB color value for each of the plurality of second groups of image pixels in the converted second digital image; and generating a lookup table to associate the average XYZ color value of each of the first group of image pixels in the converted first digital image with the corresponding average RGB color value of the corresponding second group of image pixels in the converted second digital image; presenting a first average RGB color value from the lookup table on a first region of a display; measuring XYZ values ​​from the first region of the display using a color measurement device; and comparing the measured XYZ values ​​from the first region of the display to the average XYZ values ​​corresponding to the first average RGB color value in the lookup table. This method can be implemented by a system, such as the one previously described with respect to Figure 4A and Figures 12A-12D The digital imaging device or image processing device described.

[0142] The above description of the disclosed embodiments is provided to enable those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein may be applied to other embodiments without departing from the spirit or scope of the present invention. Therefore, it will be understood that the description and drawings presented herein represent the presently preferred embodiments of the present invention, and therefore represent the subject matter widely contemplated by the present invention. It should be further understood that the scope of the present invention fully encompasses other embodiments that may be contemplated by those skilled in the art, and therefore the scope of the present invention is not limited.

Claims

1. A system for digital pathology color calibration, the system comprising: a non-transitory computer-readable medium configured to store instructions; at least one processor communicatively coupled to the non-transitory computer-readable medium and configured to: obtaining a first digital image of a specimen captured by a first imaging system in XYZ color, wherein the first digital image comprises a plurality of XYZ pixels, and wherein each of the plurality of XYZ pixels has an XYZ color value, obtaining a second digital image of the specimen captured by the second imaging system in RGB color, wherein the second digital image comprises a plurality of RGB pixels, and wherein each of the plurality of RGB pixels has an RGB color value, identifying a first plurality of image pixels in the first digital image based on XYZ color values ​​of XYZ pixels in the first digital image, identifying a plurality of second groups of image pixels in the second digital image, the plurality of second groups of image pixels respectively corresponding to the first groups of image pixels in the first digital image, and generating a lookup table to associate each of a plurality of XYZ color values ​​from the first digital image with one of a plurality of RGB color values ​​from the second digital image, wherein the number of pixel groups of the plurality of XYZ pixels and the number of pixel groups of the plurality of RGB pixels are limited to a minimum number that assigns each of the plurality of XYZ pixels to one of the plurality of XYZ color values ​​and assigns each of the plurality of RGB pixels to one of the plurality of RGB color values ​​while minimizing an error between the corresponding color values ​​of the pixels and the corresponding color values ​​to which the pixels are assigned.

2. The system of claim 1, wherein: The at least one processor is further configured to: aligning image data of the first digital image with corresponding image data of the second digital image, and The first digital image and the second digital image are converted to a common image pixel size, wherein converted image pixels of the first digital image are aligned with corresponding converted image pixels of the second digital image.

3. The system of claim 2, wherein: Each image pixel in each first group of image pixels in the first digital image has an XYZ color value that is within a threshold value of the other image pixels in the same first group of image pixels, and Each second group of image pixels in the second digital image corresponds to one of the plurality of first groups of image pixels in the first digital image based on the alignment.

4. The system of claim 2, wherein: The at least one processor is further configured to: assigning each first group of image pixels in the first digital image to one of a plurality of XYZ color indices, determining an average XYZ color value for each XYZ color index by averaging the XYZ color values ​​of all image pixels assigned to the corresponding XYZ color index, assigning each second group of image pixels in the second digital image to one of a plurality of RGB color indices, and determining an average RGB color value for each RGB color index by averaging the RGB color values ​​of all image pixels assigned to the corresponding RGB color index, and The lookup table further associates the average XYZ color value of each XYZ color index with the average RGB color value of the corresponding RGB color index.

5. The system of claim 1, wherein: Generating a lookup table involves: registering the first digital image and the second digital image to a common grid, and The plurality of XYZ pixels are mapped to the plurality of RGB pixels according to the common grid.

6. The system of claim 1, wherein: The at least one processor is further configured to: One or both of the first digital image and the second digital image are upsampled or downsampled to a common pixel size.

7. The system of claim 1, wherein: Each image pixel of each first group of image pixels in the first digital image is contiguous with at least one other image pixel of the first group of image pixels in the first digital image.

8. The system of claim 1, wherein: At least some of the image pixels in each first group of image pixels in the first digital image are non-contiguous.

9. The system of claim 1, wherein: The number of pixel groups of the multiple XYZ pixels and the number of pixel groups of the multiple RGB pixels are both limited to a predetermined number, so that the number of the multiple XYZ color values ​​and the number of the multiple RGB color values ​​in the lookup table are also limited to the predetermined number.

10. The system of claim 9, wherein: The predetermined number is ten.

11. The system of claim 1, wherein: Root mean square analysis was used to minimize the error.

12. The system of claim 1, wherein: The first digital image of the specimen captured via the first imaging system in XYZ color includes: capturing a hyperspectral image stack comprising images of the specimen at different wavelengths of light; and The first digital image is generated from the hyperspectral image stack.

13. The system of claim 1, wherein: The at least one processor is further configured to: The lookup table is embedded in a data structure comprising the second digital image.

14. The system of claim 13, wherein: The lookup table is contained in an International Color Consortium ICC profile in the data structure.

15. The system of claim 1, wherein: The at least one processor is further configured to: converting the lookup table into a model, and The model is embedded in a data structure comprising the second digital image.

16. A method for digital pathology color calibration, the method comprising: obtaining a first digital image of the specimen captured by the first imaging system in XYZ color, wherein the first digital image comprises a plurality of XYZ pixels, and wherein each of the plurality of XYZ pixels has an XYZ color value; obtaining a second digital image of the specimen captured by the second imaging system in RGB color, wherein the second digital image comprises a plurality of RGB pixels, and wherein each of the plurality of RGB pixels has an RGB color value; identifying a first plurality of image pixels in the first digital image based on XYZ color values ​​of XYZ pixels in the first digital image; identifying a plurality of second groups of image pixels in the second digital image, the plurality of second groups of image pixels respectively corresponding to the first groups of image pixels in the first digital image; as well as generating a lookup table to associate each of a plurality of XYZ color values ​​from the first digital image with one of a plurality of RGB color values ​​from the second digital image, wherein the number of pixel groups of the plurality of XYZ pixels and the number of pixel groups of the plurality of RGB pixels are limited to a minimum number that assigns each of the plurality of XYZ pixels to one of the plurality of XYZ color values ​​and assigns each of the plurality of RGB pixels to one of the plurality of RGB color values ​​while minimizing an error between the corresponding color values ​​of the pixels and the corresponding color values ​​to which the pixels are assigned.

17. The method of claim 16, further comprising: aligning image data of the first digital image with corresponding image data of the second digital image, and The first digital image and the second digital image are converted to a common image pixel size, wherein converted image pixels of the first digital image are aligned with corresponding converted image pixels of the second digital image.

18. The method of claim 17, wherein: Each image pixel in each first group of image pixels in the first digital image has an XYZ color value that is within a threshold value of the other image pixels in the same first group of image pixels, and Each second group of image pixels in the second digital image corresponds to one of the plurality of first groups of image pixels in the first digital image based on the alignment.

19. The method of claim 17, further comprising: assigning each first group of image pixels in the first digital image to one of a plurality of XYZ color indices, determining an average XYZ color value for each XYZ color index by averaging the XYZ color values ​​of all image pixels assigned to the corresponding XYZ color index, assigning each second group of image pixels in the second digital image to one of a plurality of RGB color indices, and determining an average RGB color value for each RGB color index by averaging the RGB color values ​​of all image pixels assigned to the corresponding RGB color index, and The lookup table further associates the average XYZ color value of each XYZ color index with the average RGB color value of the corresponding RGB color index.

Images