Image sensing system and method for imaging - Patents.com

Abstract

The embodiments described herein relate to a system and method for imaging. The imaging system may include a castellated optical element, a CMOS image sensor, and a color filtering element. The CMOS image sensor may include a focal region, a line scan region, a 2D imaging region, and a look-ahead gap. The imaging system may be configured to scan and capture bidirectional forward-focused brightfield and fluorescent images of one or more slides containing at least one biological material.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 329,673, filed April 11, 2022, which is incorporated by reference in its entirety for all purposes.

[0002] FIELD OF THEINVENTION The present invention relates generally to systems and methods for imaging, and more particularly to systems and methods for imaging biological materials. [Background technology]

[0003] Digital pathology scanners generate images of the prepared biological material slides that can be used by pathologists to inform diagnostics and guide treatment decisions. Some digital pathology scanners include multiple line scan sensors. However, these digital pathology scanners suffer from drawbacks. For example, line scan sensors are typically designed for and used in industrial applications, resulting in a limited number of manufacturers with widely different sensor designs. Furthermore, the design of many line scan sensors is application driven, making it unlikely that line scan sensors supplied by different manufacturers are optically and electronically interchangeable with each other. Furthermore, due to the limited number of line scan sensor manufacturers, a manufacturer offering a particular line scan sensor may not offer that particular line scan sensor for the planned life of the digital pathology scanner. As another example, digital pathology scanners are subject to errors in alignment due to physical separation between the multiple line scan sensors. As yet another example, digital pathology scanners are not configured to perform both bright field imaging and fluorescent imaging. The present invention overcomes these drawbacks. Summary of the Invention

[0004] The embodiments described herein relate to systems and methods for imaging. According to some embodiments, an imaging system may include a castellated optical element and a CMOS image sensor having pixels arranged in rows of pixels. The rows of pixels may include a first set of rows that may include a focal region, a second set of rows that may include a line scan region, and a third set of rows that may include a two-dimensional imaging region. The imaging system may also include a plurality of color filtering elements arranged across the pixels of the two-dimensional imaging region, and may include one or more first color filtering elements, one or more second color filtering elements, and one or more third color filtering elements.

[0005] The castellated optic may be formed from sections, each section having a different refractive index.The castellated optic may comprise dense flint glass.

[0006] One section of the castellated optical element may be positioned across a portion of the first set of rows such that light may be focused onto a first focal plane a first distance away from the object plane, and another section of the objective lens may be positioned across another portion of the first set of rows such that light may be focused onto a second focal plane a second distance away from the object plane that is less than the first distance.

[0007] The first focal region was focused on a first focal plane, and the second focal region was focused on a second focal plane.

[0008] A third section of the castellated optical element may be positioned across a third portion of the first set of rows such that light may be focused to a third focal plane a third distance away from the object plane that is less than the first and second distances.

[0009] The first focal region may receive light focused at a first focal plane, the second focal region may receive light focused at a second focal plane, and the third focal region may receive light focused at a third focal plane.

[0010] Each color filtering element may include one or more materials dyed with at least an organic dye. In some embodiments, the color filtering element may include at least one dielectric stack. In some embodiments, the color filtering element may be disposed across pixels of a line scan region. In some embodiments, the color filtering element may be disposed across pixels of a two-dimensional imaging region.

[0011] The color filtering element may filter light into a first red wavelength band and a second red wavelength band, filter light into a first green wavelength band and a second green wavelength band, and filter light into a first blue wavelength band and a second blue wavelength band.

[0012] The line scan region may be disposed between the first focal region and the second focal region, and the two-dimensional imaging region may be disposed between the second focal region and the third focal region. In some embodiments, the two-dimensional imaging region may capture bright field images. In some embodiments, the two-dimensional imaging region may capture fluorescent images. In some embodiments, the focal region and the line scan region may perform bidirectional forward-looking dynamic focusing on the tissue sample.

[0013] A method for imaging may include acquiring a focused signal of a tissue sample using a focal region of a CMOS image sensor such that a first focal region may receive light that has passed through the castellated optical element from a first focal plane, a second focal region may also receive light that has passed through the castellated optical element from the second focal plane, and a third focal region may also receive light that has passed through the castellated optical element from the third focal plane, scanning the tissue sample using a line scan region of the CMOS image sensor, and capturing an image of the tissue sample using a two-dimensional imaging region of the CMOS image sensor.

[0014] The castellated optical element may be formed from sections, with each section having a different refractive index, hi some embodiments, the castellated optical element may comprise dense flint glass.

[0015] The first focal plane may be located at a first distance from the object plane, the second focal plane may be located at a second distance from the object plane, and the third focal plane may be located at a third distance from the object plane, in some embodiments, the first distance may be greater than the second distance, and the second distance may be greater than the third distance.

[0016] The color filtering element may be disposed across the pixels of the two-dimensional imaging region. The color filtering element may include one or more materials dyed with at least an organic dye. The color filtering element may comprise at least one dielectric stack. The color filtering element may filter light into a first red wavelength band and a second red wavelength band, filter light into a first green wavelength band and a second green wavelength band, and filter light into a first blue wavelength band and a second blue wavelength band.

[0017] The line scan region may be disposed between the first focal region and the second focal region, and the two-dimensional imaging region may be disposed between the second focal region and the third focal region. Capturing the image may include capturing a bright field image. Capturing the image may include capturing a fluorescent image. The method may include performing bidirectional forward looking dynamic focusing on the tissue sample based on the acquired focusing signal and the scanned tissue sample.

[0018] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0019] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0020] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein.

[0021] The terms and expressions used are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions to exclude any equivalents of the features or portions thereof shown and described, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, although the invention described in the claims has been specifically disclosed by embodiments and optional features, it is to be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. [Brief description of the drawings]

[0022] BRIEF DESCRIPTION OF THE DRAWINGS Aspects and features of various embodiments will become more apparent from the following detailed description of the invention, taken in conjunction with the accompanying drawings, in which: FIG.

[0023] [Figure 1] 1 shows an example of a digital pathology scanner. [Diagram 2] 1 illustrates an example of a complementary metal oxide semiconductor (CMOS) image sensor configuration according to some embodiments of the present invention. [Diagram 3] 2 illustrates another example of the configuration of a CMOS image sensor according to an embodiment of the present invention. [Figure 4] 1 illustrates examples of configurations of castellation type optical elements according to some embodiments of the present invention. [Diagram 5] 1 illustrates an exemplary digital pathology scanner in accordance with some embodiments of the present invention. [Figure 6] 1 shows a flowchart of an exemplary process for imaging, according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes the similar components. If only a first reference label is used in this specification, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label.

[0025] Digital pathology typically involves the acquisition, management, and interpretation of data related to pathology. Some aspects of digital pathology involve capturing digital images that pathologists can use to inform diagnoses and guide therapeutic decisions. The digital images may be of various objects, and in some examples, the digital images may be of one or more slides containing one or more biological materials. The biological materials may include materials of human origin, materials of animal origin, and / or microorganisms. The biological materials may also include muscle tissue, organ tissue, blood, blood derivatives, urine, feces, saliva, cells, cultures, and / or other materials. In some examples, the biological materials may be acquired from living and dead organisms. In some examples, the biological materials may be acquired from biobanks, biorepositories, and / or other entities that acquire and store biological materials. In some examples, the digital pathology scanner may perform dynamic focusing using line scan and focusing sensors while capturing the digital images.

[0026] As shown in FIG. 1, an example digital pathology scanner 100 includes a line scan sensor 110, an image capture objective lens 111, a line scan focus sensor 120, and a castellated optics 124. A signal from the line scan focus sensor 120 is used to determine a focus position of the line scan sensor 110 based on an image of one or more slides 123 captured through the castellated optics 124. The line scan sensor 110 is focused by adjusting the height (z-height) of the image capture objective lens 111 according to the determined focus position. The focus position is determined from a contrast-based focus score determined for a far focus position 121 and a contrast-based focus score determined for a near focus position 122 using a contrast-based focusing method. The far focus position 121 and the near focus position 122 are established by the castellated optics 124. The castellated optics 124 is provided with periodically spaced air through holes 125 formed by removing material from a portion of the castellated optics 124, such that light passing through the air through holes 125 converges to a near focus position 122, and light passing through the material of the castellated optics 124 converges to a far focus position 121. The nominal focus position 112 of the line scan sensor 110 is established by the image capture objective lens 111 and is located at a position between the near focus position 121 and the far focus position 121. Based on the determined focus position, the height (z-height) of the image capture objective lens 111 is adjusted within a predetermined range. Thus, as one or more slides 123 pass in front of the line scan focusing sensor 120, the focus position can be dynamically adjusted at high frequency.

[0027] However, in such digital pathology scanners, the physical separation between the line scan sensor and the line scan focusing sensor may cause errors in the alignment between the portion of the slide or slides relied upon to perform focusing and the portion of the slide or slides relied upon to capture the image. Furthermore, the digital pathology scanner is not configured to perform both bright field imaging and fluorescent imaging on one or more slides. Furthermore, line scan sensors are typically designed for and used in industrial applications, so there is a limited number of manufacturers involved in widely different sensor designs. Furthermore, because the design of many line scan sensors is application driven, it is unlikely that line scan sensors supplied by different manufacturers are optically and electronically interchangeable with each other. Furthermore, because there is a limited number of manufacturers of line scan sensors, a manufacturer offering a particular line scan sensor may not offer that particular line scan sensor for the planned life of the digital pathology scanner.

[0028] To compensate for these limitations, systems and methods are provided for imaging one or more slides containing one or more biological materials with a two-dimensional (2D) complementary metal oxide semiconductor (CMOS) image sensor configured to perform bidirectional forward-looking dynamic focusing. CMOS image sensors offer unique advantages over line-scan sensors in that 2D CMOS image sensors are typically available in a variety of different designs and are manufactured using standard semiconductor manufacturing techniques. Replacing line-scan and line-scan focused sensors with 2D CMOS image sensors eliminates potential alignment errors caused by incorporating multiple sensors and increases the possibility of sourcing optically and electronically replaceable sensors over the intended lifespan of the digital pathology scanner. Additionally, as described in more detail herein, the 2D CMOS image sensor can capture images of one or more slides containing one or more biological materials using bright-field and fluorescent imaging techniques.

[0029] FIG. 2 illustrates an example of a configuration of a CMOS image sensor according to an embodiment of the present invention. As illustrated in FIG. 2, portions of the CMOS image sensor 200 may be configured for line scanning (e.g., line scan area 210), 2D imaging (e.g., 2D imaging area 215), and focusing (e.g., focal area 1, 212, focal area 2, 213, and focal area 3, 214). In some examples, the portion of the CMOS image sensor 200 configured as the line scan area 210 may include one or more columns of pixels configured with red color filtering elements (red pixels), one or more columns of pixels configured with green color filtering elements (green pixels), and one or more columns of pixels configured with blue color filtering elements (blue pixels). Additionally, the CMOS image sensor 200 may be configured with one or more columns of pixels without color filtering elements (gray pixels) included between one or more columns of red pixels and one or more columns of green pixels, and one or more other columns of gray pixels included between one or more columns of green pixels and one or more columns of blue pixels.

[0030] As further shown in FIG. 2, the portion of the CMOS image sensor 200 configured as a 2D imaging area 215 may include a plurality of columns of pixels arranged in a matrix. In some examples, the pixels of the 2D imaging area 215 may be configured with red, green, and blue color filtering elements. The red, green, and blue color filtering elements may be arranged in a Bayer pattern or other color filtering pattern. In some examples, the pixels of the 2D imaging area 215 may not be overlaid with any color filtering elements.

[0031] In some examples, each of the red, green, and blue color filtering elements may be formed from one or more materials dyed with at least an organic dye. In some examples, each of the red, green, and blue color filtering elements may be formed from at least one dielectric stack. In some configurations, the red pixel receives light in a wavelength band in a range of about 620 nm to about 750 nm filtered by the red color filtering element, the green pixel receives light in a wavelength band in a range of about 495 nm to about 570 nm filtered by the green color filtering element, and the blue pixel receives light in a wavelength band in a range of about 450 nm to about 495 nm filtered by the blue color filtering element. In some examples, the red pixels receive light filtered by the red color filtering element into one or more sub-bands within a wavelength band ranging from about 620 nm to about 750 nm, the green pixels receive light filtered by the green color filtering element into one or more sub-bands within a wavelength band ranging from about 495 nm to about 570 nm, and the blue pixels receive light filtered by the blue color filtering element into one or more sub-bands within a wavelength band ranging from about 450 nm to about 495 nm.

[0032] The aforementioned color filtering elements and arrangements are merely exemplary. Other color filtering elements and arrangements are encompassed by the present invention. For example, the line scan area 210 may scan an image based on a complementary color scheme, where one or more columns of pixels have a cyan color filtering element associated therewith, one or more columns of pixels have a magenta color filtering element associated therewith, one or more columns of pixels have a yellow color filtering element associated therewith, and one or more columns of pixels have a green color filtering element associated therewith. In another example, the 2D imaging area 215 may capture an image based on a complementary color scheme, where it may include pixels overlaid with a cyan color filtering element, a magenta color filtering element, a yellow color filtering element, and a green color filtering element. In some examples, each pixel of the line scan area 210 and each pixel of the 2D imaging area 215 may receive light filtered by one or more color filtering elements of one or more wavelength bands. In some examples, each pixel of the line scan area 210 and each pixel of the 2D imaging area 215 may be formed from vertically stacked photodiodes of respective spectral sensitivities. In some examples, the CMOS image sensor 200 may be configured with tunable color filters and plasmonic-based color filters.

[0033] In some examples, the line scan region 210 may scan a bright field image of one or more slides. In some examples, the 2D imaging region 215 may capture a 2D bright field image of one or more slides. In some examples, the 2D imaging region 215 may capture a 2D fluorescent image of one or more slides. In some examples, the line scan region 210 may scan one or more bright field images of one or more slides and the 2D imaging region 215 may capture one or more 2D bright field images of one or more slides. In some examples, the line scan region 210 may scan one or more bright field images of one or more slides and the 2D imaging region 215 may capture one or more 2D fluorescent images of one or more slides.

[0034] As shown in FIG. 2, the portions of the CMOS image sensor 200 configured as focal regions 1, 212 may include one or more columns of pixels, focal regions 2, 213 may include one or more columns of pixels, and focal regions 3, 214 may include one or more columns of pixels. In some examples, focal regions 1, 212 and focal regions 2, 213 may be disposed on either side of the line scan region 210. In some examples, focal regions 2, 213 and focal regions 3, 214 may be disposed on either side of the 2D imaging region 215. In some examples, signals output from focal regions 1, 212, focal regions 213, and focal regions 3, 214 may be used to estimate the focal planes of the line scan region 210 and the 2D imaging region 215. As shown in FIG. 2, the portions of the CMOS image sensor 200 may be configured as look-ahead gaps 211. In some examples, the portions of the CMOS image sensor 200 configured as look-ahead gaps 211 may include one or more columns of pixels. In some examples, signals output from focus area 1, 212, focus area 2, 213, focus area 3, 214, and lookahead gap 211 enable CMOS image sensor 200 to perform bidirectional forward-looking dynamic focusing on one or more slides.

[0035] In some examples, the signals output from focal region 1, 212, focal region 2, 213, and focal region 3, 214 may be used to estimate the focal plane using a contrast-based autofocus technique or a phase-difference based autofocus technique. In some examples, the signals output from focal region 1, 212, focal region 2, 213, and focal region 3, 214 may be used to estimate the focal plane of line scan region 210 and 2D imaging region 215 based on a combination of contrast-based autofocus technique and phase-difference based autofocus technique. In some examples, the signals output from focal region 1, 212, focal region 2, 213, and focal region 3, 214 may be used to estimate the focal plane of line scan region 210 and 2D imaging region 215 based on a combination of passive focusing technique, active ranging technique, and / or focusing technique.

[0036] The above-mentioned arrangements of the focal regions and the look-ahead gap are merely exemplary, and other arrangements are encompassed by the present invention. For example, FIG. 3 illustrates another example of a configuration of a CMOS image sensor according to an embodiment of the present invention. As shown in FIG. 3, portions of the CMOS image sensor 300 may be configured as a line scan region 310, a 2D imaging region 314, a focal region 1, 312, a focal region 2, 313, and a look-ahead gap 311. In this example, the focal region 1, 312 and the focal region 2, 313 may be disposed on one side of the line scan region 310, and the look-ahead gap 311 may be disposed between the focal region 1, 312 and the focal region 2, 313, and on both sides of the line scan region 310. Other features and operations of the CMOS image sensor 300 are similar to those of the CMOS image sensor 200 described above, and will not be repeated herein.

[0037] In some examples, signals output from focal region 1, 212, focal region 2, 213, and focal region 3, 214 of CMOS image sensor 200 may be used to estimate the focal plane of line scan region 210 and 2D imaging region 215, and signals output from focal region 1, 312 and focal region 2, 313 of CMOS image sensor 300 may be used to estimate the focal plane of line scan region 310 and 2D imaging region 314. To estimate the focal plane, CMOS image sensor 200 or 300 may be coupled with a castellated optical element.

[0038] FIG. 4 illustrates an exemplary configuration of a castellated optical element 415 that may be coupled with a CMOS image sensor 200, 300 according to some embodiments of the present invention. As illustrated in FIG. 4, the castellated optical element 415 may be configured with portions of different refractive indexes. For example, the castellated optical element 415 may be configured with a first portion 416 having a first refractive index that focuses light to a far focal plane 413, a second portion 417 having a second refractive index that focuses light to a near focal plane 2, 411, and a third portion 418 having a third refractive index that focuses light to a near focal plane 1, 410. In some examples, the far focal plane 413 may be located at a first distance from an object plane (not shown), the near focal plane 2, 411 may be located at a second distance from the object plane (not shown) that is less than the first distance, and the near focal plane 1, 410 may be located at a third distance from the object plane (not shown) that is less than the first and second distances. In some examples, the third portion 418 may include one or more air through holes formed by removing material from the castellated optical element 415. In some examples, the third portion 418 may be comprised of one or more materials having a refractive index lower than the first and second refractive indices. The castellated optical element 415 may further be comprised of an image capture portion 419 having an image capture refractive index that focuses light to the nominal focal plane 412. In some examples, the first portion 416 may be thicker than the second portion 417, the third portion 418, and the image capture portion 419, respectively. In some examples, the third portion 418 and the image capture portion 419 may each be thicker than the second portion 417. In some examples, the castellated optical element 415 may be comprised of high density flint glass. In some examples, the castellated optical element 415 may be comprised of acrylic plastic. In some examples, the castellated optical element 415 may be comprised of a combination of high density flint glass and acrylic plastic. In some examples, first portion 416, second portion 417, third portion 418, and image capture portion 419 may each be composed of different materials having different refractive indices.In some examples, image capture portion 419 may be at least 3.6 mm thick, first portion 416 may be at least 7 mm thick, and second portion 417 and third portion 418 may each range in thickness between 3.6 mm and 7 mm.

[0039] With respect to the CMOS image sensor 200, in some examples, focal region 1, 212 may receive light focused at the far focal plane 413, focal region 2, 213 may receive light focused at the near focal plane 2, 411, focal region 3, 214 may receive light focused at the near focal plane 1, 410, and the line scan region 210 and the 2D imaging region 215 may each receive light focused at the nominal focal plane 412.

[0040] As mentioned above, in some examples, the signals output from focal region 1, 212, focal region 2, 213, and focal region 3, 214 may be used to estimate the focal plane of the line scan region 210 and the 2D imaging region 215. In some examples, the castellated optical element 415 may be placed at an initial position. In some examples, when the castellated optical element 415 is positioned at the initial position, the nominal focal plane 412 may be located between the far focal plane 413 and the near focal plane 2, 411, and the near focal plane 2, 411 may be located between the near focal plane 1, 410 and the nominal focal plane 412. In some examples, the castellated optical element 415 may shift from its initial position in a direction perpendicular to the object plane (z-height) based on the estimated focal plane. In some examples, the castellated optical element 415 may shift about 1 μm from its initial position toward the object plane or about 1 μm from its initial position toward the imaging plane (not shown).

[0041] The focal arrangements described above with respect to the CMOS image sensor 200 of Figure 2 may be similarly applied to the CMOS image sensor 300 of Figure 3. For example, in some arrangements, focal region 1, 312 and focal region 2, 313 of the CMOS image sensor 300 may be arranged to receive light that converges to either a far focal plane 413, a near focal plane 2, 411, and a near focal plane 1, 410, respectively. The above arrangements are merely exemplary, and other arrangements utilizing one or more focal regions and focal planes are encompassed by the present invention.

[0042] In some examples, digital pathology involves the acquisition, management, and interpretation of data related to pathology. Some embodiments of digital pathology may involve capturing images of one or more slides containing one or more biological materials using a CMOS image sensor and castellated optical elements according to the configurations shown in Figures 2, 3, and / or 4, performing computer-based analysis of the captured images, and outputting the results of the analysis for interpretation, diagnosis, and treatment decision-making. In some examples, the biological material may include human origin material, animal origin material, and / or microorganisms. In some examples, the biological material may also include muscle tissue, organ tissue, blood, blood derivatives, urine, feces, saliva, cells, cultures, and / or other materials. In some examples, the biological material may be obtained from living and dead organisms. In some examples, the biological material may be obtained from biobanks, biorepositories, and / or other entities that acquire and store biological material.

[0043] In some examples, one or more biological materials may be fixed / embedded on one or more slides. For example, in the case of tissue slices (e.g., samples of one or more parts of a tumor), the tissue slices may be sliced ​​using a fixing / embedding agent to obtain multiple tissue samples, with each sample having a specific dimension and fixed / embedded on one or more glass slides. In some examples, slicing each tissue slice may involve cooling the tissue slice and slicing the cooled tissue slice in a warm water bath. The fixing / embedding process renders the cells of each sample substantially transparent, so each sample may be stained with an agent to make the cellular structure more visible. In some examples, different samples may be stained with one or more different stains to represent different characteristics of each sample. In some examples, different samples may be exposed to different predetermined volumes of stains for a predetermined time. In some examples, the staining may involve a histochemical stain. In some examples, the stains may include hematoxylin, trichrome, periodic acid Schiff, Giemsa, reticulin, and / or toluidine blue. Additionally, in some instances, staining may involve direct or indirect immunohistochemical staining. The foregoing examples are non-limiting and other methods for fixing / embedding biological material are encompassed by the present invention.

[0044] In some examples, images of one or more slides including one or more biological materials may be captured based on whole slide imaging techniques, tile-based scanning techniques, and / or line-based scanning techniques. In some examples, images may be captured based on bright field detection techniques, fluorescent detection techniques, and / or multispectral detection techniques. In some examples, the captured images of one or more slides may be analyzed by an image analysis algorithm configured for digital pathology. In some examples, the image analysis algorithm may detect, characterize, and / or quantify biological elements of interest in the captured images (e.g., tumor cells, tumors, immune cells, etc.). In some examples, the image analysis algorithm may localize and delineate tumors in the captured images. In some examples, the image analysis algorithm may determine a quantity, quantity, and / or size (e.g., cell count) associated with one or more biological materials. In some examples, the image analysis algorithm may generate one or more detections, segmentations, bounding boxes, labels, identifications, classifications, annotations, highlighting, frames, and / or contours of one or more objects of interest in the captured images. In some examples, the image analysis algorithm may be configured using machine learning and deep learning techniques. In some examples, the image analysis algorithm may include and / or use one or more neural networks. In some examples, the output of the image analysis algorithm may be provided to a human pathologist for further analysis and / or interpretation. In some examples, using a viewing device, the human pathologist may annotate the captured image to train the image analysis algorithm to detect, analyze, and classify biological objects of interest in the captured image. In some examples, the annotated image and the output of the image analysis algorithm may assist a physician in diagnosing a subject and / or guide therapeutic decision-making.

[0045] FIG. 5 illustrates an exemplary digital pathology scanner 500 according to some embodiments of the present invention. As shown in FIG. 5, the digital pathology scanner 500 may include a scanning platform 510, one or more processors 520, a RAM 530, a network interface 540, a CMOS image sensor and castellated optics 550, one or more memories 560, one or more storage devices 570, and a display 580. The scanning platform 510 may be configured to pass one or more slides containing one or more biological materials in front of the CMOS image sensor and castellated optics 550 for imaging. In some examples, the imaging may be based on whole slide imaging techniques, tile-based scanning techniques, and / or line-based scanning techniques. In some examples, capturing the image may involve bright field detection techniques, fluorescent detection techniques, and multispectral detection techniques. In some examples, one or more slides may be created as described above. In some examples, the CMOS image sensor and castellated optics 550 may be configured according to the configurations shown in FIG. 2, FIG. 3, and / or FIG. 4. Other configurations are within the scope of the present invention.

[0046] The one or more memories 560 are configured to store one or more programs for imaging and analyzing the captured images. The one or more processors 520 are configured to read and execute the one or more programs from the one or more memories 560 using the RAM 530. In some examples, the one or more programs may include programs for executing image analysis algorithms. In some examples, the image analysis algorithms may detect, characterize, and / or quantify biological elements of interest in the captured images (e.g., tumor cells, tumors, immune cells, etc.). In some examples, the image analysis algorithms may localize and delineate tumors in the captured images. In some examples, the image analysis algorithms may determine quantities, quantities, and / or sizes (e.g., cell counts) associated with one or more biological materials. In some examples, the image analysis algorithms may generate one or more detections, segmentations, bounding boxes, labels, identifications, classifications, annotations, highlights, frames, and / or contours of one or more objects of interest in the captured images. In some examples, the image analysis algorithms may be configured using machine learning and deep learning techniques. In some examples, the image analysis algorithm may include and / or use one or more neural networks.

[0047] One or more storage devices 570 may be configured to store the captured images and / or the output of the image analysis algorithm. In some examples, the output of the image analysis algorithm may be provided to a human pathologist for further analysis and / or interpretation. In some examples, the network interface 540 may output the captured images and / or the output of the image analysis algorithm to a network, server, or other device. In some examples, the display 580 may display the captured images and / or the output of the image analysis algorithm. In some examples, using the display 580, the human pathologist may annotate the captured images to train the image analysis algorithm to detect and classify biological objects of interest in the captured images. In some examples, the annotated images may assist a physician in diagnosing a subject and / or guide treatment decisions.

[0048] FIG. 6 shows a flow chart of an exemplary process 600 for imaging according to some embodiments of the present invention. In some examples, the process may be implemented by a digital pathology scanning system based on the digital pathology scanner 500 according to FIG. 5. In some examples, the process may be implemented in software or hardware or any combination thereof. In some examples, a processor or computer system may be configured to execute the process. For example, in block 610, the system acquires signals from one or more focal regions of a CMOS image sensor, such as the CMOS image sensor 200 of FIG. 2 or the CMOS image sensor 300 of FIG. 3. In block 620, the system performs focusing on one or more slides based on the acquired signals. In some examples, the focusing may include bidirectional forward-looking dynamic focusing. In some examples, the signals output from the one or more focal regions may be used to estimate a focal plane of the 2D imaging region of the CMOS image sensor. In some examples, focusing may be performed based on a contrast-based autofocusing technique, a phase-difference-based autofocusing technique, or a combination of contrast-based autofocusing technique and phase-difference-based autofocusing technique. In some examples, focusing may be performed based on a combination of passive focusing techniques, active ranging techniques, and / or focusing techniques.

[0049] In some examples, a castellated optical element, such as castellated optical element 415 of FIG. 4, may be placed at an initial position. In some examples, when the castellated optical element is positioned at the initial position, the nominal focal plane may be located between the far focal plane and the near focal plane 2, and the near focal plane 2 may be located between the near focal plane 1 and the nominal focal plane. In some examples, the castellated optical element may be shifted from its initial position in a direction perpendicular to the object plane (z-height) based on the estimated focal plane. In some examples, the castellated optical element may be shifted from its initial position toward the object plane by about 1 μm, or toward the imaging plane (not shown) by about 1 μm.

[0050] At blocks 630 and 640, the system scans one or more slides in a line scan region and captures one or more images of the one or more slides in a 2D imaging region. In some examples, the scanning and capture may be based on whole slide imaging techniques, tile-based scanning techniques, and line-based scanning techniques. In some examples, the scanning and capture may be based on bright field detection techniques, fluorescent detection techniques, and / or multispectral detection techniques. In some examples, the line scan region may scan one or more bright field images and / or one or more fluorescent images of the one or more slides. In some examples, the 2D imaging region may capture one or more 2D bright field images and / or one or more 2D fluorescent images of the one or more slides.

[0051] At block 650, the system analyzes one or more captured images of one or more slides using an image analysis algorithm configured for digital pathology stored in memory and executed by the system's processor. In some examples, the image analysis algorithm may detect, characterize, and / or quantify biological elements of interest in the captured image (e.g., tumor cells, tumors, immune cells, etc.). In some examples, the image analysis algorithm may localize and delineate tumors in the captured image. In some examples, the image analysis algorithm may determine a quantity, quantity, and / or size (e.g., cell count) associated with one or more biological materials. In some examples, the image analysis algorithm may generate one or more detections, segmentations, bounding boxes, labels, identifications, classifications, annotations, highlights, frames, and / or contours of one or more objects of interest in the captured image. In some examples, the image analysis algorithm may be configured using machine learning and deep learning techniques. In some examples, the image analysis algorithm may include and / or use one or more neural networks (e.g., convolutional neural networks). At block 660, the system stores, displays, and / or outputs the captured image and / or the output of the image analysis algorithm. In some examples, the output of the image analysis algorithm may be provided to a human pathologist via the network interface 540 and / or the display 580 for further analysis and / or interpretation. In some examples, using a viewing device, the human pathologist may annotate the captured image to train the image analysis algorithm to detect, analyze, and classify biological objects of interest in the captured image. In some examples, the annotated image and the output of the image analysis algorithm may assist a physician in diagnosing a subject and / or guide therapeutic decision-making.

[0052] The systems and methods of the present disclosure may be implemented using hardware, software, firmware, or a combination thereof, and may be implemented in one or more computer systems or other processing systems. Some embodiments of the present disclosure include a system including one or more processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more processors, cause the one or more processors to perform a part or all of one or more methods and / or a part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause the one or more processors to perform a part or all of one or more methods and / or a part or all of one or more processes disclosed herein.

[0053] The terms and expressions used are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude any equivalents of the features or portions thereof shown and described, but it is recognized that various modifications are possible within the scope of the claimed invention. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be employed by those skilled in the art, and such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0054] The description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the description of the preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0055] In the following description, specific details are given to provide a comprehensive understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. 1. An imaging system comprising: A castellation type optical element; a complementary metal oxide semiconductor (CMOS) image sensor; The CMOS image sensor comprises: a plurality of pixels arranged in a plurality of pixel columns, a first set of columns of the plurality of pixel columns comprising a plurality of focal regions, a second set of columns of the plurality of pixel columns comprising a line scan region, and a third set of columns of the plurality of pixel columns comprising a two-dimensional imaging region; a plurality of color filtering elements disposed across at least one of pixels of the line scan region or pixels of the two-dimensional imaging region, the plurality of color filtering elements comprising one or more first color filtering elements, one or more second color filtering elements, and one or more third color filtering elements; An imaging system comprising:

2. 10. The imaging system of claim 1, wherein the castellated optical element is formed from a plurality of portions, each portion of the plurality of portions having a different refractive index.

3. 2. The imaging system of claim 1, wherein the castellated optical element is formed from a plurality of portions, a first portion of the plurality of portions is disposed across a first portion of the first set of rows of the plurality of pixels and focuses light onto a first focal plane a first distance away from an object plane, and a second portion of the plurality of portions of the castellated optical element is disposed across a second portion of the first set of rows of the plurality of pixels and focuses light onto a second focal plane a second distance away from the object plane, the second distance being less than the first distance.

4. 2. The imaging system of claim 1, wherein a first focal region of the plurality of focal regions receives light focused onto a first focal plane a first distance away from an object plane, and a second focal region of the plurality of focal regions receives light focused onto a second focal plane a second distance away from the object plane, the second distance being less than the first distance.

5. 2. The imaging system of claim 1 , wherein the castellation type optical element is formed from a plurality of portions, a first portion of the plurality of portions being disposed across a first portion of the first set of rows of the plurality of columns of pixels, a second portion of the plurality of portions of the castellation type optical element being disposed across a second portion of the first set of rows of the plurality of columns of pixels, and a third portion of the plurality of portions of the castellation type optical element being disposed across a third portion of the first set of rows of the plurality of columns of pixels.

6. 2. The imaging system of claim 1, wherein a first focal region of the plurality of focal regions receives light focused at a first focal plane, a second focal region of the plurality of focal regions receives light focused at a second focal plane, and a third focal region of the plurality of focal regions receives light focused at a third focal plane.

7. 2. The imaging system of claim 1 , wherein each of the one or more first color filtering elements, the one or more second color filtering elements, and the one or more third color filtering elements comprises one or more materials dyed with at least an organic dye.

8. The imaging system of claim 1 , wherein the plurality of color filtering elements are disposed across pixels of the line scan region.

9. The imaging system of claim 1 , wherein the plurality of color filtering elements are disposed across pixels of the two-dimensional imaging area.

10. 10. The imaging system of claim 1, wherein each of the one or more first color filtering elements, the one or more second color filtering elements, and the one or more third color filtering elements comprises at least one dielectric stack.

11. 2. The imaging system of claim 1 , wherein the one or more first color filtering elements filters light into a first red wavelength band and a second red wavelength band, the one or more second color filtering elements filters light into a first green wavelength band and a second green wavelength band, and the one or more third color filtering elements filters light into a first blue wavelength band and a second blue wavelength band.

12. 2. The imaging system of claim 1, wherein the line scan region is disposed between a first focal region of the plurality of focal regions and a second focal region of the plurality of focal regions, and the two-dimensional imaging region is disposed between the second focal region of the plurality of focal regions and a third focal region of the plurality of focal regions.

13. The imaging system of claim 1 , wherein the line scan region is disposed between the plurality of focal regions and the two-dimensional imaging region.

14. The imaging system of claim 1 , wherein the CMOS image sensor is configured to capture bright field images.

15. The imaging system of claim 1 , wherein the CMOS image sensor is configured to capture a fluorescent image.

16. The imaging system of claim 1 , wherein the castellated optical element comprises high density flint glass.

17. 2. The imaging system of claim 1, wherein the CMOS image sensor is configured to perform bidirectional forward-looking dynamic focusing on one or more slides containing at least biological material based on signals acquired from the multiple focal regions.

18. 1. A method for imaging comprising: acquiring a plurality of focused signals of one or more slides containing at least one biological material having a plurality of focal regions of a CMOS image sensor, wherein a first focal region of the plurality of focal regions receives light that has passed through a castellated optical element from a first focal plane, a second focal region of the plurality of focal regions receives light that has passed through the castellated optical element from a second focal plane, and a third focal region of the plurality of focal regions receives light that has passed through the castellated optical element from a third focal plane; scanning the one or more slides with a line scan area of ​​the CMOS image sensor; capturing an image of the one or more slides at a two-dimensional imaging area of ​​the CMOS image sensor; A method comprising:

19. 20. The method for imaging of claim 18, wherein the castellated optical element is formed from a plurality of portions, each portion of the plurality of portions having a different refractive index.

20. 20. The method for imaging of claim 18, wherein the first focal plane is located at a first distance from an object plane, the second focal plane is located at a second distance from the object plane, and the third focal plane is located at a third distance from the object plane, the first distance being greater than the second distance, and the second distance being greater than the third distance.

21. 20. The method for imaging of claim 18, wherein the plurality of color filtering elements disposed across at least one of the pixels of the line scan region or the pixels of the two-dimensional imaging region comprises one or more first color filtering elements, one or more second color filtering elements, and one or more third color filtering elements.

22. 20. The method for imaging of claim 18, wherein a plurality of color filtering elements disposed across at least one of the pixels of the line scan region or the pixels of the two-dimensional imaging region include one or more substances dyed with at least an organic dye.

23. 20. The method for imaging of claim 18, wherein a plurality of color filtering elements disposed across at least one of the pixels of the line scan region or the pixels of the two-dimensional imaging region comprises at least one dielectric stack.

24. 20. The method for imaging of claim 18, wherein the one or more first color filtering elements filter light into a first red wavelength band and a second red wavelength band, the one or more second color filtering elements filter light into a first green wavelength band and a second green wavelength band, and the one or more third color filtering elements filter light into a first blue wavelength band and a second blue wavelength band.

25. 20. The method for imaging of claim 18, wherein the line scan region is disposed between a first focal region of the plurality of focal regions and a second focal region of the plurality of focal regions, and the two-dimensional imaging region is disposed between the second focal region of the plurality of focal regions and a third focal region of the plurality of focal regions.

26. 20. The method for imaging of claim 18, wherein the line scan region is disposed between the plurality of focal regions and the two-dimensional imaging region.

27. 20. The method for imaging of claim 18, wherein the step of capturing an image comprises capturing a bright field image.

28. 20. The method for imaging of claim 18, wherein the step of capturing an image comprises capturing a fluorescent image.

29. 20. The method for imaging of claim 18, further comprising performing bidirectional forward-looking dynamic focusing on the one or more slides based on the plurality of focusing signals and the one or more slides.

Images