Image sensing system and method for imaging

Through the combination of CMOS image sensor and castle-shaped optical components, the limitations of the design of line scanning sensors in digital pathology scanners are solved, and efficient bidirectional forward dynamic focus and a variety of imaging technologies are achieved, improving imaging quality and maintainability of equipment sex.

CN118985137APending Publication Date: 2024-11-19VENTANA MEDICAL SYSTEMS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380033385.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-11
Filing Date
2023-03-23
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing digital pathology scanners are not interchangeable due to line scanning sensor design applications driven by limited manufacturers, and are unable to provide specific sensors within the planned service life and fail to perform bright field and fluorescence simultaneously. Imaging.

Method used

The CMOS image sensor is used to combine castle-shaped optical elements and multi-color filtering components to achieve bidirectional forward-view dynamic focus and two-dimensional imaging, support bright field and fluorescence imaging, and optimize image capture and analysis through data processors and computer program products.

Benefits of technology

Reduces registration error, increases the interchangeability and service life of sensors, improves the imaging capabilities of digital pathology scanners, and supports the application of a variety of imaging technologies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118985137A_ABST
    Figure CN118985137A_ABST
Patent Text Reader

Abstract

Embodiments described herein relate to systems and methods for imaging. An imaging system may include a pheasanted optical element, a CMOS image sensor, and a color filter element. The CMOS image sensor may include a focus region, a line scan region, a 2D imaging region, and a foresight gap. The imaging system may be configured to scan and capture bi-directionally forward focused bright field images and fluorescence images of one or more slides comprising at least one biological material.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 329,673 filed on April 11, 2022, the contents of which are hereby incorporated by reference in their entirety for all purposes. Technical Field

[0003] The present invention generally relates to a system and method for imaging. In particular, the present invention relates to a system and method for imaging biological material. Background Art

[0004] Digital pathology scanners generate images of prepared slides of biological materials that pathologists can use to inform diagnostic results and guide treatment decisions. Some digital pathology scanners include multiple line scan sensors. However, these digital pathology scanners have defects. For example, because line scan sensors are generally designed for and used in industrial applications, only a limited number of manufacturers have widely different sensor designs. In addition, because many line scan sensor designs are application-driven, line scan sensors from different manufacturers are unlikely to be optically and electronically interchangeable with each other. In addition, because there are a limited number of manufacturers of line scan sensors, there is a risk that a manufacturer that provides a specific line scan sensor may not be able to provide the specific line scan sensor within the planned service life of the digital pathology scanner. As another example, due to the physical separation between multiple line scan sensors, digital pathology scanners are prone to registration errors. As yet another example, digital pathology scanners are not configured to perform both bright field imaging and fluorescence imaging. The present invention overcomes these defects. Summary of the invention

[0005] Embodiments described herein relate to systems and methods for imaging. According to some embodiments, an imaging system may include a castellated optical element, a CMOS image sensor having pixels arranged in pixel columns. The pixel columns may include a first group of columns that may include a focus area, a second group of columns that may include a line scan area, and a third group of columns that may include a two-dimensional imaging area. The imaging system may also include a plurality of color filter components that are disposed above the pixels of the two-dimensional imaging area and may include one or more first color filter elements, one or more second color filter elements, and one or more third color filter elements.

[0006] The castellated optic may be formed from a plurality of segments such that each segment may have a different refractive index. The castellated optic may comprise high density flint glass.

[0007] A section of the castellated optical element can be disposed over a portion of the first group so that light can converge on a first focal plane separated from the object plane by a first distance, and another section of the objective lens can be disposed over another portion of the first group so that light can converge on a second focal plane separated from the object plane by a second distance, the second distance being less than the first distance.

[0008] The first focal region converges on the first focal plane and the second focal region converges on a second focal plane.

[0009] A third section of the castellated optical element may be disposed over a third portion of the first array such that light may be focused on a third focal plane spaced a third distance from the object plane that is less than the first distance and the second distance.

[0010] The first focal region may receive light focused on a first focal plane, the second focal region may receive light focused on a second focal plane, and the third focal region may receive light focused on a third focal plane.

[0011] Each color filter element in the color filter element may include one or more materials dyed with at least an organic dye. In some embodiments, the color filter element may include at least one dielectric stack. In some embodiments, the color filter element may be arranged above the pixels of the line scan area. In some embodiments, the color filter element may be arranged above the pixels of the two-dimensional imaging area.

[0012] The color filter element may filter light into first and second red wavelength bands, filter light into first and second green wavelength bands, and filter light into first and second blue wavelength bands.

[0013] The line scan area can be disposed between the first focal area and the second focal area, and the two-dimensional imaging area can be disposed between the second focal area and the third focal area. In some embodiments, the two-dimensional imaging area can capture a bright field image. In some embodiments, the two-dimensional imaging area can capture a fluorescent image. In some embodiments, the focal area and the line scan area can perform a two-way forward-looking dynamic focusing on the tissue sample.

[0014] The method for imaging may include acquiring a focus signal of a tissue sample using a focusing area of ​​a CMOS image sensor, such that a first focusing area can receive light from a first focal plane that has passed through a castellated optical element, a second focusing area can also receive light from a second focal plane that has passed through the castellated optical element, and a third focusing area can also receive light from a third focal plane that has passed through the castellated optical element, thereby scanning the tissue sample using a line scanning area of ​​the CMOS image sensor and capturing an image of the tissue sample using a two-dimensional imaging area of ​​the CMOS image sensor.

[0015] The castellated optic may be formed from multiple segments such that each segment may have a different refractive index. In some embodiments, the castellated optic may comprise high density flint glass.

[0016] The first focal plane may be located at a first distance from the object plane, the second focal plane may be located at a second distance from the object plane, and the third focal plane may be located at a third distance from the object plane. In some embodiments, the first distance may be greater than the second distance, and the second distance may be greater than the third distance.

[0017] The color filter element may be disposed above the pixels of the two-dimensional imaging area. The color filter element may include one or more materials dyed with at least an organic dye. The color filter element may include at least one dielectric stack. The color filter element may filter light into a first red wavelength band and a second red wavelength band, filter light into a first green wavelength band and a second green wavelength band, and filter light into a first blue wavelength band and a second blue wavelength band.

[0018] The line scan area may be disposed between the first focus area and the second focus area, and the two-dimensional imaging area may be disposed between the second focus area and the third focus area. Capturing an image may include capturing a bright field image. Capturing an image may include capturing a fluorescence image. The method may include performing bidirectional forward-looking dynamic focusing on the tissue sample based on the acquired focus signal and the scanned tissue sample.

[0019] In some embodiments, a system is provided, comprising one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0020] In some embodiments, a computer program product is provided, the computer program product being tangibly embodied in a non-transitory machine-readable storage medium and comprising instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0021] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions, which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product, which is tangibly embodied in a non-transitory machine-readable storage medium, including instructions, which are configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein.

[0022] The terms and expressions that have been adopted are used as terms of description rather than limitation, and when such terms and expressions are used, there is no intention to exclude any equivalents of the features shown and described or parts thereof, but it should be recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed through embodiments and optional features, modifications and variations of the concepts disclosed herein may be adopted by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Various aspects and features of the various embodiments will become more apparent by describing examples with reference to the accompanying drawings, in which:

[0024] Figure 1 An example of a digital pathology scanner is shown.

[0025] Figure 2 An example of a configuration of a complementary metal oxide semiconductor (CMOS) image sensor according to some embodiments of the present invention is shown.

[0026] Figure 3 Another example of the configuration of a CMOS image sensor according to some embodiments of the present invention is shown.

[0027] Figure 4 Exemplary configurations of castellated optical elements according to some embodiments of the present invention are shown.

[0028] Figure 5 An exemplary digital pathology scanner according to some embodiments of the present invention is shown.

[0029] Figure 6 A flow chart showing an exemplary process for imaging according to some embodiments of the present invention is shown.

[0030] In the drawings, similar components and / or features may have the same reference number. Further, various components of the same type may be distinguished by following the reference number with a dash and a second reference number that distinguishes the similar components. If only the first reference number is used in the specification, the description applies to any similar component having the same first reference number, regardless of the second reference number. DETAILED DESCRIPTION

[0031] Digital pathology generally involves the collection, management, and interpretation of pathology-related data. Some aspects of digital pathology involve capturing digital images that pathologists can use to inform diagnosis and guide treatment decisions. The digital images can be of various objects, and in some instances, the digital images can be one or more slides containing one or more biological materials. The biological material can include materials of human origin, materials of animal origin, and / or microorganisms. The biological material can also include muscle tissue, organ tissue, blood, blood derivatives, urine, feces, saliva, cells, cultures, and / or other materials. In some instances, the biological material can be obtained from living and dead organisms. In some instances, the biological material can be obtained from a biobank, a biorepository, and / or other entities that obtain and store biological materials. In some instances, a digital pathology scanner can perform dynamic focusing using a line scan sensor and a focus sensor while capturing a digital image.

[0032] like Figure 1As shown, an example of a digital pathology scanner 100 includes a line scan sensor 110, an image capture objective 111, a line scan focus sensor 120, and a castellated optical element 124. Signals from the line scan focus sensor 120 are used to determine a focal position of the line scan sensor 110 based on an image of one or more slides 123 captured by the castellated optical element 124. The line scan sensor 110 is focused by adjusting the height (z-height) of the image capture objective 111 according to the determined focal position. Using a contrast-based focusing method, the focal position is determined according to a contrast-based focus score determined for a far focal position 121 and a contrast-based focus score determined for a near focal position 122. The far focal position 121 and the near focal position 122 are established by the castellated optical element 124. The castellated optical element 124 is provided with periodically spaced air holes 125 formed by removing material from portions of the castellated optical element 124, such that light passing through the air holes 125 is focused at a near focus position 122, and light passing through the material of the castellated optical element 124 is focused at a far focus position 121. The nominal focus position 112 of the line scan sensor 110 is established by the image capture objective 111 and is located at a position between the near focus position 121 and the far focus position 121. Based on the determined focus position, the height (z-height) of the image capture objective 111 is adjusted within a predetermined range. Therefore, the focus position can be dynamically adjusted at a high frequency when one or more slides 123 pass in front of the line scan focus sensor 120.

[0033] However, in such digital pathology scanners, the physical separation between the line scan sensor and the line scan focus sensor may result in registration errors between the portions of the one or more slides on which the focusing is performed and the portions of the one or more slides on which the image is captured. In addition, the digital pathology scanner is not configured to perform both bright field imaging and fluorescence imaging on one or more slides. In addition, since line scan sensors are typically designed for and used in industrial applications, only a limited number of manufacturers have widely different sensor designs. In addition, since many line scan sensor designs are application driven, line scan sensors from different manufacturers are unlikely to be optically and electronically interchangeable with each other. In addition, since there are a limited number of manufacturers of line scan sensors, there is a risk that a manufacturer that provides a particular line scan sensor may not be able to provide that particular line scan sensor within the planned service life of the digital pathology scanner.

[0034] To remedy these limitations, systems and methods are provided for imaging one or more slides containing one or more biological materials using a two-dimensional (2D) complementary metal oxide semiconductor (CMOS) image sensor configured to perform bidirectional forward-looking dynamic focusing. The unique advantage provided by CMOS image sensors over line scan sensors is that 2D CMOS image sensors are generally available in a variety of different designs and are manufactured using standard semiconductor manufacturing techniques. By replacing line scan sensors and line scan focus sensors with 2D CMOS image sensors, potential registration errors caused by combining multiple sensors can be avoided, and the possibility of procuring optically and electronically interchangeable sensors during the planned service life of the digital pathology scanner is increased. In addition, as discussed in more detail herein, the 2D CMOS image sensor can capture images of one or more slides containing one or more biological materials using bright field imaging techniques and fluorescence imaging techniques.

[0035] Figure 2 2 shows an example of a configuration of a CMOS image sensor according to some embodiments of the present invention. Figure 2 As shown, a portion of the CMOS image sensor 200 may be configured for line scanning (e.g., line scanning area 210), 2D imaging (e.g., 2D imaging area 215), and focusing (e.g., focusing area 1 212, focusing area 2 213, and focusing area 3 214). In some examples, the portion of the CMOS image sensor 200 configured as the line scanning area 210 may include one or more pixel columns configured with red color filter elements (red pixels), one or more pixel columns configured with green color filter elements (green pixels), and one or more pixel columns configured with blue color filter elements (blue pixels). In addition, the CMOS image sensor 200 may be configured with one or more pixel columns (gray pixels) without any color filter elements included between one or more red pixel columns and one or more green pixel columns, and one or more additional gray pixel columns included between one or more green pixel columns and one or more blue pixel columns.

[0036] like Figure 2 As further shown, the portion of the CMOS image sensor 200 configured as the 2D imaging area 215 may include a plurality of pixel columns arranged in a matrix. In some examples, the pixels of the 2D imaging area 215 may be configured with a red color filter element (red pixel), a green color filter element (green pixel), and a blue color filter element (blue pixel). The red color filter element, the green color filter element, and the blue color filter element may be arranged in a Bayer pattern or other color filter pattern. In some examples, the pixels of the 2D imaging area 215 may not overlap with any color filter element.

[0037] In some instances, each of the red color filter element, the green color filter element, and the blue color filter element can be formed by one or more materials dyed with at least an organic dye. In some instances, each of the red color filter element, the green color filter element, and the blue color filter element can be formed by at least one dielectric stack. In some configurations, the red pixel receives the light of the wavelength band range from about 620nm to about 750nm filtered by the red color filter element, the green pixel receives the light of the wavelength band range from about 495nm to about 570nm filtered by the green color filter element, and the blue pixel receives the light of the wavelength band range from about 450nm to about 495nm filtered by the blue color filter element. In some instances, the red pixels receive light filtered by the red color filter element into one or more sub-bands within a wavelength band ranging from about 620 nm to about 750 nm, the green pixels receive light filtered by the green color filter element into one or more sub-bands within a wavelength band ranging from about 495 nm to about 570 nm, and the blue pixels receive light filtered by the blue filter element into one or more sub-bands within a wavelength band ranging from about 450 nm to about 495 nm.

[0038] The aforementioned color filter elements and arrangements are merely illustrative. The present invention encompasses other filter elements and arrangements. For example, the line scan area 210 can scan an image based on a complementary color scheme, wherein one or more pixel columns have a cyan filter element associated therewith, one or more pixel columns have a magenta color filter element associated therewith, one or more pixel columns have a yellow color filter element associated therewith, and one or more pixel columns have a green color filter element associated therewith. In another example, the 2D imaging area 215 can capture an image based on a complementary color scheme and include pixels covered with a cyan filter element, a magenta color filter element, a yellow color filter element, and a green color filter element. In some examples, each of the pixels of the line scan area 210 and each of the pixels of the 2D imaging area 215 can receive light in one or more wavelength bands filtered by one or more color filter elements. In some examples, each of the pixels of the line scan area 210 and each of the pixels of the 2D imaging area 215 can be formed by a vertically stacked photodiode with a corresponding spectral sensitivity. In some examples, CMOS image sensor 200 can be configured with tunable color filters and plasmon-based color filters.

[0039] In some instances, the line scan region 210 may scan a bright field image of one or more slides. In some instances, the 2D imaging region 215 may capture a 2D bright field image of one or more slides. In some instances, the 2D imaging region 215 may capture a 2D fluorescent image of one or more slides. In some instances, the line scan region 210 may scan one or more bright field images of one or more slides, and the 2D imaging region 215 may capture one or more 2D bright field images of one or more slides. In some instances, the line scan region 210 may scan one or more bright field images of one or more slides, and the 2D imaging region 215 may capture one or more 2D fluorescent images of one or more slides.

[0040] like Figure 2 As shown, the portion of the CMOS image sensor 200 configured as the focus area 1 212 may include one or more pixel columns, the focus area 2 213 may include one or more pixel columns, and the focus area 3 214 may include one or more pixel columns. In some instances, the focus area 1 212 and the focus area 2 213 may be arranged on both sides of the line scan area 210, respectively. In some instances, the focus area 2 213 and the focus area 3 214 may be arranged on both sides of the 2D imaging area 215, respectively. In some instances, the signals output from the focus area 1 212, the focus area 213, and the focus area 3 214 may be used to estimate the focal planes of the line scan area 210 and the 2D imaging area 215. Figure 2 As shown, portions of the CMOS image sensor 200 may be configured as a forward-looking gap 211. In some instances, the portion of the CMOS image sensor 200 configured as the forward-looking gap 211 may include one or more pixel columns. In some instances, signals output from focus region 1 212, focus region 2 213, focus region 3 214, and the forward-looking gap 211 enable the CMOS image sensor 200 to perform bidirectional forward-looking dynamic focusing on one or more slides.

[0041] In some examples, the signals output from the focus area 1 212, the focus area 2 213, and the focus area 3 214 may be used to estimate the focal plane using a contrast-based autofocus technique or a phase-difference-based autofocus technique. In some examples, the signals output from the focus area 1 212, the focus area 2 213, and the focus area 3 214 may be used to estimate the focal plane of the line scan area 210 and the 2D imaging area 215 based on a combination of the contrast-based autofocus technique and the phase-difference-based autofocus technique. In some examples, the signals output from the focus area 1 212, the focus area 2 213, and the focus area 3 214 may be used to estimate the focal plane of the line scan area 210 and the 2D imaging area 215 based on a combination of a passive focusing technique, a ranging technique, and / or a focusing technique.

[0042] The foregoing arrangement of the focal region and the forward-looking gap is merely illustrative, and the present invention encompasses other arrangements. For example, Figure 3 Another example of the configuration of a CMOS image sensor according to some embodiments of the present invention is shown. Figure 3 As shown, the various parts of the CMOS image sensor 300 may be configured as a line scan area 310, a 2D imaging area 314, a focus area 1 312, a focus area 2 313, and a forward-looking gap 311. In this example, the focus area 1 312 and the focus area 2 313 may be arranged on one side of the line scan area 310, and the forward-looking gap 311 may be arranged between the focus area 1 312 and the focus area 2 313 and on both sides of the line scan area 310. Other features and operations of the CMOS image sensor 300 are similar to those of the CMOS image sensor 200, which have been described above and are not repeated here.

[0043] In some examples, signals output from focus area 1 212, focus area 2 213, and focus area 3 214 of CMOS image sensor 200 may be used to estimate focal planes of line scan area 210 and 2D imaging area 215, and signals output from focus area 1 312 and focus area 2 313 of CMOS image sensor 300 may be used to estimate focal planes of line scan area 310 and 2D imaging area 314. To estimate the focal plane, CMOS image sensor 200 or 300 may be coupled to a castellated optical element.

[0044] Figure 4 2 shows an exemplary configuration of a castellated optical element 415 that may be coupled to a CMOS image sensor 200, 300 according to some embodiments of the present invention. Figure 4As shown, the castellated optical element 415 can be configured with portions having different refractive indices. For example, the castellated optical element 415 can be configured with a first portion 416 having a first refractive index that converges light on a far focal plane 413, a second portion 417 having a second refractive index that converges light on a near focal plane 2 411, and a third portion 418 having a third refractive index that converges light on a near focal plane 1 410. In some examples, the far focal plane 413 can be located at a first distance from an object plane (not shown), the near focal plane 2 411 can be located at a second distance from the object plane (not shown), the second distance being less than the first distance, and the near focal plane 1 410 can be located at a third distance from the object plane (not shown), the third distance being less than the first distance and the second distance. In some examples, the third portion 418 can include one or more air holes formed by removing material from the castellated optical element 415. In some examples, the third portion 418 can be configured with one or more materials having a refractive index less than the first and second refractive indices. The castellated optical element 415 can be further configured with an image capture portion 419 having an image capture refractive index that focuses light on the nominal focal plane 412. In some examples, the first portion 416 can be thicker than the second portion 417, the third portion 418, and the image capture portion 419, respectively. In some examples, the third portion 418 and the image capture portion 419 can be thicker than the second portion 417, respectively. In some examples, the castellated optical element 415 can be constructed of high-density flint glass. In some examples, the castellated optical element 415 can be constructed of acrylic plastic. In some examples, the castellated optical element 415 can be constructed of a combination of high-density flint glass and acrylic plastic. In some examples, the first portion 416, the second portion 417, the third portion 418, and the image capture portion 419 can be constructed of different materials having different refractive indices, respectively. In some examples, the image capture portion 419 can be at least 3.6 mm thick, the first portion 416 can be at least 7 mm thick, and the thickness of the second portion 417 and the third portion 418 can be between 3.6 mm and 7 mm, respectively.

[0045] For the CMOS image sensor 200, in some instances, focus area 1 212 can receive light that converges on a far focal plane 413, focus area 2 213 can receive light that converges on a near focal plane 2411, focus area 3 214 can receive light that converges on a near focal plane 1 410, and the line scan area 210 and the 2D imaging area 215 can each receive light that converges on a nominal focal plane 412.

[0046] As previously discussed, in some examples, the signals output from the focal region 1 212, the focal region 2 213, and the focal region 3 214 may be used to estimate the focal planes of the line scan region 210 and the 2D imaging region 215. In some examples, the castellated optical element 415 may be positioned in an initial position. In some examples, when the castellated optical element 415 is in the initial position, the nominal focal plane 412 may be located between the far focal plane 413 and the near focal plane 2 411, and the near focal plane 2 411 may be located between the near focal plane 1 410 and the nominal focal plane 412. In some examples, the castellated optical element 415 may be offset (z-height) from its initial position in a direction perpendicular to the object plane based on the estimated focal plane. In some examples, the castellated optical element 415 may be offset from its initial position toward the object plane by about 1 μm, or offset from its initial position toward the imaging plane (not shown) by about 1 μm.

[0047] and Figure 2 The aforementioned focus arrangement associated with the CMOS image sensor 200 may be similarly applied to Figure 3 CMOS image sensor 300. For example, in some arrangements, focus region 1 312 and focus region 2 313 of CMOS image sensor 300 may be arranged to receive light focused on any one of far focus, near focus 2, and near focus 1 planes 413, 411, and 410, respectively. The foregoing arrangements are merely exemplary, and the present invention encompasses other arrangements utilizing one or more focus regions and focus planes.

[0048] In some instances, digital pathology involves the acquisition, management, and interpretation of pathology-related data. Some aspects of digital pathology may involve the use of Figure 2 , Figure 3 , and / or Figure 4 The CMOS image sensor and castellated optical element of the configuration shown in capture images of one or more slides containing one or more biological materials; perform computer-based analysis on the captured images; and output the analysis results for interpretation, diagnosis, and treatment decisions. In some instances, the biological material may include materials of human origin, materials of animal origin, and / or microorganisms. In some instances, the biological material may also include muscle tissue, organ tissue, blood, blood derivatives, urine, feces, saliva, cells, cultures, and / or other materials. In some instances, the biological material can be obtained from living and dead organisms. In some instances, the biological material can be obtained from a biobank, a biorepository, and / or other entities that obtain and store biological materials.

[0049] In some instances, one or more biological materials can be fixed / embedded to one or more slides. For example, in the case of a tissue block (e.g., a sample of one or more parts of a tumor), the tissue block can be sliced ​​to obtain multiple tissue samples, each sample having a specific size and fixed / embedded to one or more slides using a fixing / embedding agent. In some instances, slicing each tissue block can involve cooling the tissue block and slicing the cooled tissue block in a warm water bath. Since the fixing / embedding process makes the cells in each sample almost transparent, each sample in the sample can be stained with an agent to make the cell structure more visible. In some instances, different samples can be stained with one or more different stains to express the different characteristics of the corresponding samples. In some instances, different samples can be exposed to a predefined volume of stain for a predefined time. In some instances, staining can involve histochemical staining. In some instances, staining agents can include hematoxylin, trichrome, periodic acid Schiff, Giemsa, reticulin, and / or toluidine blue. In addition, in some instances, staining can involve direct or indirect immunohistochemical staining. The foregoing examples are non-limiting, and the present invention encompasses other methods for fixation / embedding of biological materials.

[0050] In some instances, the image of one or more slides containing one or more biological materials can be captured based on full slide imaging technology, block-based scanning technology, and / or line-based scanning technology. In some instances, images can be captured based on bright field detection, fluorescence detection, and / or multispectral detection technology. In some instances, the captured images of one or more slides can be analyzed by an image analysis algorithm configured for digital pathology. In some instances, the image analysis algorithm can detect, characterize and / or quantify the biological elements of interest (e.g., tumor cells, each tumor, immune cells, etc.) within the captured image. In some instances, the image analysis algorithm can locate and outline the outline of the tumor within the captured image. In some instances, the image analysis algorithm can determine the amount, quantity and / or size (e.g., cell count) associated with one or more biological materials. In some instances, the image analysis algorithm can generate one or more detections, segmentation, bounding boxes, labels, identifications, classifications, annotations, highlights, frames, and / or outlines of one or more objects of interest within the captured image. In some instances, machine learning and deep learning techniques can be used to configure the image analysis algorithm. In some instances, the image analysis algorithm may include and / or use one or more neural networks. In some instances, the output of the image analysis algorithm may be provided to a human pathologist for further analysis and / or interpretation. In some instances, using a viewing device, a human pathologist may annotate the captured images to train the image analysis algorithm to detect, analyze, and classify biological objects of interest within the captured images. In some instances, the annotated images and the output of the image analysis algorithm may help a physician diagnose a subject and / or guide treatment decisions.

[0051] Figure 5 An exemplary digital pathology scanner 500 according to some embodiments of the present invention is shown. Figure 5 As shown, the digital pathology scanner 500 may include a scanning platform 510, one or more processors 520, a RAM 530, a network interface 540, a CMOS image sensor and a castellated optical element 550, one or more memories 560, one or more storage devices 570, and a display 580. The scanning platform 510 may be configured to pass one or more slides containing one or more biological materials in front of the CMOS image sensor and the castellated optical element 550 for imaging. In some instances, the imaging may be based on whole slide imaging technology, tile-based scanning technology, and / or line-based scanning technology. In some instances, capturing images may involve brightfield detection, fluorescence detection, and multispectral detection technology. In some instances, the one or more slides may be prepared as described above. In some instances, the CMOS image sensor and the castellated optical element 550 may be based on Figure 2 , Figure 3 and / or Figure 4 Other arrangements are also within the scope of the present invention.

[0052] One or more memories 560 are configured to store one or more programs for imaging and analyzing the captured images. One or more processors 520 are configured to read one or more programs from one or more memories 560 and execute them using RAM 530. In some instances, one or more of the programs may include a program for executing an image analysis algorithm. In some instances, the image analysis algorithm may detect, characterize and / or quantify biological elements of interest (e.g., tumor cells, each tumor, immune cells, etc.) within the captured image. In some instances, the image analysis algorithm may locate and outline the contours of the tumor within the captured image. In some instances, the image analysis algorithm may determine the amount, quantity and / or size (e.g., cell count) associated with one or more biological materials. In some instances, the image analysis algorithm may generate one or more detections, segmentations, bounding boxes, labels, identifications, classifications, annotations, highlights, frames, and / or contours of one or more objects of interest within the captured image. In some instances, the image analysis algorithm may be configured using machine learning and deep learning techniques. In some instances, the image analysis algorithm may include and / or use one or more neural networks.

[0053] One or more storage devices 570 may be configured to store the captured images and / or the output of the image analysis algorithm. In some instances, the output of the image analysis algorithm may be provided to a human pathologist for further analysis and / or interpretation. In some instances, the network interface 540 may output the captured images and / or the output of the image analysis algorithm to a network, server, or other device. In some instances, a display 580 may display the captured images and / or the output of the image analysis algorithm. In some instances, using the display 580, a human pathologist may annotate the captured images to train the image analysis algorithm to detect and classify the biological objects of interest within the captured images. In some instances, annotated images may help doctors diagnose subjects and / or guide treatment decisions.

[0054] Figure 6 is a flow chart of an exemplary process 600 for imaging according to some embodiments of the present invention. In some examples, the process may be based on Figure 5The process may be implemented in a digital pathology scanning system of the digital pathology scanner 500. In some examples, the process may be implemented in software or hardware or any combination thereof. In some examples, a processor or computer system may be configured to perform the process. For example, at block 610, the system obtains a CMOS image sensor (such as Figure 2 CMOS image sensor 200 or Figure 3 The system acquires signals from one or more focus areas of the CMOS image sensor 300). At block 620, the system performs focusing on one or more slides based on the acquired signals. In some instances, focusing may include bidirectional forward-looking dynamic focusing. In some instances, the signals output from the one or more focus areas may be used to estimate the focal plane of the 2D imaging area of ​​the CMOS image sensor. In some instances, focusing may be performed based on a contrast-based autofocus technique, a phase-difference-based autofocus technique, or a combination of a contrast-based autofocus technique and a phase-difference-based autofocus technique. In some instances, focusing may be performed based on a passive focusing technique, an active ranging technique, and / or a combination of focusing techniques.

[0055] In some examples, a castellated optical element (such as Figure 4 The castellated optical element 415) can be positioned in an initial position. In some examples, when the castellated optical element is in the initial position, the nominal focal plane can be located between the far focal plane and the near focal plane 2, and the near focal plane 2 can be located between the near focal plane 1 and the nominal focal plane. In some examples, the castellated optical element can be offset from its initial position in a direction perpendicular to the object plane (z-height) based on the estimated focal plane. In some examples, the castellated optical element can be offset from its initial position toward the object plane by about 1 μm, or from its initial position toward an imaging plane (not shown).

[0056] At boxes 630 and 640, the system scans one or more slides with a line scan area and captures one or more images of the one or more slides with a 2D imaging area. In some instances, the scanning and capturing can be based on full slide imaging technology, tile-based scanning technology, and line-based scanning technology. In some instances, the scanning and capturing can be based on bright field detection, fluorescence detection, and / or multi-spectral detection technology. In some instances, the line scan area can scan one or more bright field images and / or one or more fluorescence images of one or more slides. In some instances, the 2D imaging area can capture one or more 2D bright field images and / or one or more 2D fluorescence images of one or more slides.

[0057] At box 650, the system uses an image analysis algorithm configured for digital pathology stored in a memory and executed by a processor of the system to analyze one or more captured images of one or more slides. In some instances, the image analysis algorithm can detect, characterize and / or quantify biological elements of interest (e.g., tumor cells, each tumor, immune cells, etc.) within the captured image. In some instances, the image analysis algorithm can locate and outline the tumor within the captured image. In some instances, the image analysis algorithm can determine the amount, quantity and / or size (e.g., cell count) associated with one or more biological materials. In some instances, the image analysis algorithm can generate one or more detections, segmentations, bounding boxes, labels, identifications, classifications, annotations, highlights, frames, and / or outlines of one or more objects of interest within the captured image. In some instances, the image analysis algorithm can be configured using machine learning and deep learning techniques. In some instances, the image analysis algorithm can include and / or use one or more neural networks (e.g., convolutional neural networks). At box 660, the system stores, displays and / or outputs the captured image and / or the output of the image analysis algorithm. In some instances, the output of the image analysis algorithm can be provided to a human pathologist via the network interface 540 and / or the display 580 for further analysis and / or interpretation. In some instances, using a viewing device, a human pathologist can annotate the captured images to train the image analysis algorithm to detect, analyze, and classify biological objects of interest within the captured images. In some instances, the annotated images and the output of the image analysis algorithm can assist a physician in diagnosing a subject and / or guiding treatment decisions.

[0058] The systems and methods of the present disclosure may be implemented using hardware, software, firmware, or a combination thereof, and may be implemented in one or more computer systems or other processing systems. Some embodiments of the present disclosure include a system comprising one or more processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions, which, when executed on the one or more processors, cause the one or more processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein.

[0059] The terms and expressions that have been adopted are used as descriptive rather than limiting terms, and when such terms and expressions are used, there is no intention to exclude any equivalents of the features shown and described or parts thereof, but it should be recognized that various modifications are possible within the scope of the invention claimed. Therefore, it should be understood that although the claimed invention is specifically disclosed through embodiments and optional features, those skilled in the art may make modifications and changes to the concepts disclosed herein, and such modifications and changes are considered to be within the scope of the invention defined by the appended claims.

[0060] This description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the description of the preferred exemplary embodiments will provide a feasible description for implementing various embodiments for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.

[0061] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components can be shown as parts in block diagram form to avoid obscuring the embodiments in unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details to avoid obscuring the embodiments.

Claims

1. An imaging system, comprising: Crenellated optical elements; and A complementary metal oxide semiconductor (CMOS) image sensor, wherein the CMOS image sensor comprises: a plurality of pixels arranged in a plurality of pixel columns, wherein a first group of the plurality of pixel columns comprises a plurality of focus regions, a second group of the plurality of pixel columns comprises a line scan region, and a third group of the plurality of pixel columns comprises a two-dimensional imaging region; and A plurality of color filter components are arranged above at least one of the pixels of the line scanning area or the pixels of the two-dimensional imaging area, wherein the plurality of color filter components include one or more first color filter elements, one or more second color filter elements, and one or more third color filter elements.

2. The imaging system of claim 1, wherein the crenellated optical element is formed from a plurality of sections, each section of the plurality of sections having a different refractive index.

3. The imaging system of claim 1 , wherein the crenellated optical element is formed of a plurality of portions, wherein a first portion of the plurality of portions is disposed over a first portion of the first group of the plurality of pixel columns and causes light to converge on a first focal plane that is a first distance from an object plane, and a second portion of the plurality of portions of the crenellated optical element is disposed over a second portion of the first group of the plurality of pixel columns and causes light to converge on a second focal plane that is a second distance from the object plane, the second distance being less than the first distance.

4. The imaging system of claim 1 , wherein a first focal region of the plurality of focal regions receives light that converges on a first focal plane that is a first distance from an object plane, and a second focal region of the plurality of focal regions receives light that converges on a second focal plane that is a second distance from the object plane, the second distance being less than the first distance.

5. The imaging system of claim 1 , wherein the crenellated optical element is formed of a plurality of portions, wherein a first portion of the plurality of portions is disposed over a first portion of the first group of the plurality of pixel columns, a second portion of the plurality of portions of the crenellated optical element is disposed over a second portion of the first group of the plurality of pixel columns, and a third portion of the plurality of portions of the crenellated optical element is disposed over a third portion of the first group of the plurality of pixel columns.

6. The imaging system of claim 1 , wherein a first focus area of ​​the plurality of focus areas receives light that converges on a first focal plane, a second focus area of ​​the plurality of focus areas receives light that converges on a second focal plane, and a third focus area of ​​the plurality of focus areas receives light that converges on a third focal plane.

7. The imaging system of claim 1, wherein each of the one or more first color filter elements, the one or more second color filter elements, and the one or more third color filter elements comprises one or more materials dyed with at least an organic dye.

8. The imaging system of claim 1, wherein the plurality of color filter members are disposed over pixels of the line scan area.

9. The imaging system of claim 1, wherein the plurality of color filter members are disposed above pixels of the two-dimensional imaging area.

10. The imaging system of claim 1, wherein each of the one or more first color filter elements, the one or more second color filter elements, and the one or more third color filter elements comprises at least one dielectric stack.

11. The imaging system of claim 1 , wherein the one or more first color filter elements filter light into a first red wavelength band and a second red wavelength band, wherein the one or more second color filter elements filter light into a first green wavelength band and a second green wavelength band, and wherein the one or more third color filter elements filter light into a first blue wavelength band and a second blue wavelength band.

12. The imaging system of claim 1 , wherein the line scanning area is disposed between a first focus area among the plurality of focus areas and a second focus area among the plurality of focus areas, and the two-dimensional imaging area is disposed between the second focus area among the plurality of focus areas and a third focus area among the plurality of focus areas.

13. The imaging system according to claim 1, wherein the line scanning area is disposed between the plurality of focusing areas and the two-dimensional imaging area.

14. The imaging system of claim 1, wherein the CMOS image sensor is configured to capture bright field images.

15. The imaging system of claim 1, wherein the CMOS image sensor is configured to capture a fluorescence image.

16. The imaging system of claim 1, wherein the crenellated optical element comprises high density flint glass.

17. The imaging system of claim 1, wherein the CMOS image sensor is configured to perform bidirectional forward-looking dynamic focusing on one or more slides containing at least biological material based on signals acquired from the plurality of focus regions.

18. A method for imaging, the method comprising: acquiring a plurality of focus signals of one or more slides containing at least one biological material with a plurality of focus regions of a CMOS image sensor, wherein a first focus region of the plurality of focus regions receives light transmitted through a crenellated optical element from a first focus plane, a second focus region of the plurality of focus regions receives light transmitted through the crenellated optical element from a second focus plane, and a third focus region of the plurality of focus regions receives light transmitted through the crenellated optical element from a third focus plane; scanning the one or more slides with a line scan area of ​​the CMOS image sensor; as well as An image of the one or more slides is captured using the two-dimensional imaging area of ​​the CMOS image sensor.

19. The method for imaging according to claim 18, wherein the crenellated optical element is formed of a plurality of sections, each section of the plurality of sections having a different refractive index.

20. The method for imaging of claim 18, wherein the first focal plane is located at a first distance from an object plane, the second focal plane is located at a second distance from the object plane, and the third focal plane is located at a third distance from the object plane, wherein the first distance is greater than the second distance and the second distance is greater than the third distance.

21. A method for imaging according to claim 18, wherein the multiple color filter components arranged above at least one of the pixels of the line scanning area or the pixels of the two-dimensional imaging area include one or more first color filter elements, one or more second color filter elements, and one or more third color filter elements.

22. The method for imaging according to claim 18, wherein a plurality of color filter members disposed over at least one of the pixels of the line scan area or the pixels of the two-dimensional imaging area comprise one or more materials dyed with at least an organic dye.

23. The method for imaging according to claim 18, wherein the plurality of color filter members disposed over at least one of the pixels of the line scan area or the pixels of the two-dimensional imaging area include at least one dielectric stack.

24. A method for imaging according to claim 18, wherein one or more first color filter elements filter light into a first red wavelength band and a second red wavelength band, one or more second color filter elements filter light into a first green wavelength band and a second green wavelength band, and one or more third color filter elements filter light into a first blue wavelength band and a second blue wavelength band.

25. The method for imaging according to claim 18, wherein the line scanning area is set between a first focus area among the plurality of focus areas and a second focus area among the plurality of focus areas, and the two-dimensional imaging area is set between the second focus area among the plurality of focus areas and a third focus area among the plurality of focus areas.

26. The method for imaging according to claim 18, wherein the line scanning area is disposed between the plurality of focus areas and the two-dimensional imaging area.

27. The method for imaging of claim 18, wherein capturing the image comprises capturing a bright field image.

28. The method for imaging of claim 18, wherein capturing the image comprises capturing a fluorescence image.

29. The method for imaging according to claim 18, further comprising: Based on the plurality of focus signals and the one or more slides, bidirectional forward-looking dynamic focusing is performed on the one or more slides.