Computational model for analyzing images of biological specimens

Abstract

A method of analyzing images of a biological specimen using a computational model is described, the method including processing a cell image of the biological specimen and a phase contrast image of the biological specimen using the computational model to generate output data. The cell image is a composite of a first brightfield image of the biological specimen at a first focal plane and a second brightfield image of the biological specimen at a second focal plane. The method further includes performing a comparison of the output data and reference data, and improving the computational model based on the comparison of the output data and the reference data. The method further includes thereafter processing additional image pairs according to the computational model to further improve the computational model based on a comparison of additional output data generated by the computational model and additional reference data.

Description

Technical Field

[0001] This application is an international application claiming priority to U.S. Application No. 16 / 950,368, filed November 17, 2020, which is incorporated herein by reference. Also incorporated by reference are U.S. Application No. 16 / 265,910, filed February 1, 2019, and U.S. Application No. 17 / 099,983, filed November 17, 2020. Background Technology

[0002] Deep artificial neural networks (ANNs), typically convolutional neural networks (CNNs), can be used to analyze labeled or unlabeled images of biological specimens. Fluorescent labeling is often used to provide detailed biological insights, such as labeling specific proteins, subcellular compartments, or cell types. However, this labeling can also disrupt biology and cause phototoxic effects due to the long exposure times required for fluorescence. In label-free analysis, the use of ANNs typically involves analyzing a single microscopic image of a given biological specimen. Summary of the Invention

[0003] In an example, this disclosure includes a method for analyzing biological specimen images using a computational model, the method comprising: processing a cell image and a phase-contrast image of the biological specimen using the computational model to generate output data, wherein the cell image is a composite of a first brightfield image of the biological specimen at a first focal plane and a second brightfield image of the biological specimen at a second focal plane; performing a comparison of the output data and reference data; improving the computational model based on the comparison of the output data and reference data; and subsequently processing additional image pairs according to the computational model to further improve the computational model based on a comparison of the additional output data generated by the computational model with additional reference data.

[0004] In another example, the invention includes a non-transitory data memory storing instructions that, when executed by a computing device, cause the computing device to perform a function of analyzing images of a biological specimen using a computational model. The function includes: processing cell images and phase-contrast images of the biological specimen using the computational model to generate output data, wherein the cell images are a composite of a first brightfield image of the biological specimen at a first focal plane and a second brightfield image of the biological specimen at a second focal plane; performing a comparison of the output data and reference data; improving the computational model based on the comparison of the output data and reference data; and subsequently processing additional image pairs according to the computational model to further improve the computational model based on a comparison of the additional output data generated by the computational model with additional reference data.

[0005] In yet another example, the present invention includes a system for analyzing biological specimens, the system comprising: an optical microscope; one or more processors; and a non-transitory data memory storing instructions which, when executed by the one or more processors, cause the system to perform the following functions: capturing a first bright-field image of the biological specimen at a first focal plane and a second bright-field image of the biological specimen at a second focal plane via the optical microscope; generating a cell image of the biological specimen by performing pixel-level mathematical operations on the first and second bright-field images; processing the cell image and the phase-contrast image of the biological specimen using a computational model to generate output data; performing a comparison of the output data and reference data; improving the computational model based on the comparison of the output data and reference data; and subsequently processing additional image pairs according to the computational model to further improve the computational model based on a comparison of the additional output data generated by the computational model with additional reference data.

[0006] The features, functions, and advantages already discussed can be implemented independently in various examples or combined in other examples, and further details can be seen in the following description and figures. Attached Figure Description

[0007] Figure 1 This is a functional block diagram of an environment based on an exemplary implementation scheme;

[0008] Figure 2 A block diagram depicts a computing device and a computer network according to an exemplary embodiment;

[0009] Figure 3 A flowchart of a method according to an exemplary implementation is shown;

[0010] Figure 4 Images of biological specimens according to an exemplary embodiment are shown;

[0011] Figure 5 An image of another biological specimen according to an exemplary embodiment is shown;

[0012] Figure 6A Experimental results of cell-by-cell segmentation masks generated according to an exemplary embodiment are shown, representing the cell image response 24 hours after the time course of apoptosis in HT1080 fibrosarcoma following camptothecin (CPT, cytotoxic) treatment.

[0013] Figure 6B It shows according to Figure 6A The implementation scheme is based on cell subpopulations classified by red (Nuclight Red, a cell health indicator, "NucRed") and green fluorescence (Caspase 3 / 7, an apoptosis indicator);

[0014] Figure 6C It shows according to Figure 6A The implementation scheme showed that after CPT treatment, the red population decreased (indicating loss of live cells), red and green fluorescence increased (indicating early apoptosis), and green fluorescence increased after 24 hours (indicating late apoptosis);

[0015] Figure 6D It shows according to Figure 6A The implementation scheme, concentration response time process of the early apoptotic population (percentage of total cells showing red and green fluorescence);

[0016] Figure 6E Experimental results of cell-by-cell segmentation masks generated according to an exemplary embodiment are shown, representing the cell image response 24 hours after the time course of apoptosis in HT1080 fibrosarcoma following cyclohexylamide (CHX, which inhibits cell growth).

[0017] Figure 6F It shows according to Figure 6E The implementation scheme is based on cell subpopulations classified by red (Nuclight Red, a cell health indicator, "NucRed") and green fluorescence (Caspase 3 / 7, an apoptosis indicator);

[0018] Figure 6G It shows according to Figure 6E The implementation scheme showed that CHX treatment resulted in a lack of apoptosis, but a decrease in cell count;

[0019] Figure 6H It shows according to Figure 6E The implementation scheme, concentration response time process of the early apoptotic population (percentage of total cells showing red and green fluorescence);

[0020] Figure 7A A cell-by-cell segmentation mask is shown applied to a phase-contrast image for label-free cell counting of adherent cells using cell-by-cell segmentation analysis generated according to an exemplary embodiment. Both label-free cell-by-cell analysis and erythrocyte nuclear counting analysis are used to analyze A549 cells labeled with NucLight Red reagent at different densities to verify label-free counts over time;

[0021] Figure 7B This shows the situation where there is no contrasting image in the background. Figure 7A Cell-by-cell segmentation mask;

[0022] Figure 7C It shows according to Figure 7A The implementation scheme, the time process of phase counting and NucRed counting data with different densities;

[0023] Figure 7DIt shows according to Figure 7A The implementation scheme, the correlation of count data over 48 hours, and the R² value of 1 when the slope is 1;

[0024] Figure 8 This is a schematic diagram of the three focal planes;

[0025] Figure 9 This is a schematic diagram of the environmental operation;

[0026] Figure 10 This is a diagram illustrating the output and reference data;

[0027] Figure 11 This is a diagram illustrating the output and reference data;

[0028] Figure 12 It is a flowchart of the method;

[0029] Figure 13 It is a flowchart of the method;

[0030] Figure 14 Images related to image classification are shown; and

[0031] Figure 15 The results of the computational model are shown.

[0032] The accompanying drawings are for illustrative purposes, but it should be understood that the invention is not limited to the arrangements and means shown in the drawings. Detailed Implementation

[0033] I. Overview

[0034] Embodiments of the methods described herein can be used to segment phase-contrast images of one or more cells in a biological specimen using defocused bright-field images, thereby allowing for single-cell and subpopulation analysis with rapid processing times. The disclosed exemplary methods also advantageously achieve real-time label-free (i.e., fluorescence-free) cell counting, avoiding the effects of fluorescent labeling that can impair the viability and function of living cells. Another advantage of the disclosed example methods is the ability to detect the boundaries of individual cells, regardless of the complexity of cell morphology, including flattened cells such as HUVECs.

[0035] II. Exemplary Architecture

[0036] Figure 1 This is a block diagram illustrating an operating environment 100, which includes or relates to, for example, an optical microscope 105 and a biological specimen 110 having one or more cells. The following description... Figures 3 to 5 Method 300 illustrates an embodiment of a method that can be implemented within the operating environment 100.

[0037] Figure 2This is a block diagram illustrating an example of a computing device 200 according to an exemplary embodiment, configured to interact directly or indirectly with an operating environment 100. The computing device 200 can be used to perform... Figures 3 to 5 The functions of the methods shown and described below. Specifically, computing device 200 may be configured to perform one or more functions, including, for example, an image generation function based in part on images obtained by optical microscope 105. Computing device 200 has a processor 202 and also has a communication interface 204, a data memory 206, an output interface 208, and a display 210, each coupled to a communication bus 212. Computing device 200 may also include hardware to enable communication within computing device 200 and between computing device 200 and other devices (e.g., devices not shown). For example, the hardware may include a transmitter, a receiver, and an antenna.

[0038] Communication interface 204 may be a wireless interface and / or one or more wired interfaces, allowing short-range and long-range communication to one or more networks 214 or one or more remote computing devices 216 (e.g., tablet 216a, personal computer 216b, laptop 216c, and mobile computing device 216d). Such a wireless interface may provide communication under one or more wireless communication protocols, such as Bluetooth, WiFi (e.g., IEEE 802.11), LTE, cellular communication, Near Field Communication (NFC), and / or other wireless communication protocols. Such a wired interface may include an Ethernet interface, a Universal Serial Bus (USB) interface, or a similar interface for communication to a wired network via wires, twisted pairs, coaxial cables, optical links, fiber optic links, or other physical connections. Therefore, communication interface 204 may be configured to receive input data from one or more devices and may also be configured to send output data to other devices.

[0039] The communication interface 204 may also include user input devices, such as a keyboard, keypad, touch screen, touchpad, computer mouse, trackball and / or other similar devices.

[0040] Data storage 206 may include or take the form of one or more computer-readable storage media that can be read or accessed by processor 202. The computer-readable storage medium may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices, which may be integrated wholly or partially with processor 202. Data storage 206 is considered a non-transitory computer-readable medium. In some examples, data storage 206 may be implemented using a single physical device (e.g., a single optical, magnetic, organic, or other memory or disk storage unit), while in other examples, data storage 206 may be implemented using two or more physical devices.

[0041] Data storage 206 is therefore a non-transitory computer-readable storage medium, and executable instructions 218 are stored on the non-transitory computer-readable storage medium. Instructions 218 include computer-executable code. When instructions 218 are executed by processor 202, processor 202 performs functions. Such functions include, but are not limited to, receiving bright-field images from optical microscope 105 and generating phase-contrast images, confluence masks, cell images, seed masks, cell-by-cell segmentation masks, and fluorescence images.

[0042] Processor 202 may be a general-purpose processor or a special-purpose processor (e.g., a digital signal processor, an application-specific integrated circuit, etc.). Processor 202 may receive input from communication interface 204 and process the input to generate output stored in data memory 206 and output to display 210. Processor 202 may be configured to execute executable instructions 218 (e.g., computer-readable program instructions) stored in data memory 206 and executable to provide the functionality of computing device 200 described herein.

[0043] Output interface 208 outputs information to display 210, or to other components. Therefore, output interface 208 can be similar to communication interface 204, and can be a wireless interface (e.g., a transmitter) or a wired interface. For example, output interface 208 can send commands to one or more controllable devices.

[0044] Figure 2 The computing device 200 shown may also represent a local computing device 200a in the operating environment 100, for example, communicating with the optical microscope 105. This local computing device 200a can perform one or more steps of the method 300 described below, and can receive input from the user and / or send image data and user input to the computing device 200 to perform all or some steps of method 300. Furthermore, in an optional exemplary embodiment, Incucyte ® The platform can be used to perform method 300 and includes the combined functions of computing device 200 and optical microscope 105.

[0045] Figure 3 A flowchart is shown of an exemplary method 300 for performing cell-by-cell segmentation of one or more cells of a biological specimen 110 according to an exemplary embodiment. For example, Figure 3 The method 300 shown presents a way to interact with Figure 2 An example of a method used with computing device 200. Furthermore, the device or system can be used or configured to perform... Figure 3The logical functions are illustrated. In some cases, components of a device and / or system can be configured to perform functions, such that the components are configured and constructed with hardware and / or software to achieve this performance. Components of a device and / or system can be arranged such that, when operated in a particular manner, they are suited, capable of, or adapted to perform functions. Method 300 may include one or more operations, functions, or actions as illustrated in one or more of the boxes 305 to 330. Although these boxes are shown in a sequential order, some of these boxes may also be performed in parallel, and / or in an order different from that described herein. Furthermore, different boxes may be combined into fewer boxes, split into additional boxes, and / or deleted based on the desired implementation.

[0046] It should be understood that, for the processes and methods disclosed herein, as well as other processes and methods, the flowchart illustrates the function and operation of one possible implementation of this example. In this respect, each box may represent a module, segment, or portion of program code, which includes one or more instructions executable by a processor to implement a specific logical function or step in the process. The program code may be stored on any type of computer-readable medium or data storage, such as storage devices including disks or hard disk drives. Furthermore, the program code may be encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art. Computer-readable media may include non-transitory computer-readable media or storage, such as computer-readable media that store data for short periods, such as register memory, processor cache, and random access memory (RAM). Computer-readable media may also include non-transitory media, such as secondary or permanent long-term storage devices, such as read-only memory (ROM), optical disks or magnetic disks, and optical disc read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage system. Computer-readable media can be considered, for example, tangible computer-readable storage media.

[0047] also, Figure 3 Each box in the present disclosure, as well as each box in other processes and methods disclosed herein, may represent a circuit that is wired to perform a specific logical function of the process. Alternative implementations are included within the scope of the examples of this disclosure, wherein, depending on the function involved, the function may not be performed in the order shown or discussed, including a substantially simultaneous order or in reverse order, as will reasonably be understood by those skilled in the art.

[0048] III. Exemplary Method

[0049] As used in this article, a "bright-field image" refers to an image obtained by microscopy based on a biological sample illuminated from below (so that light waves pass through the transparent parts of the biological sample). Thus, varying levels of brightness are captured in a bright-field image.

[0050] As used herein, a “phase-contrast image” refers to an image obtained directly or indirectly via a microscope based on a biological sample illuminated from below, capturing the phase shift of light passing through the biological sample due to differences in refractive index across different parts of the sample. For example, as light waves pass through a biological specimen, the amplitude (i.e., brightness) and phase of the light waves change in a manner dependent on the characteristics of the biological specimen. As a result, a phase-contrast image has brightness intensity values ​​associated with pixels, the variations of which render denser areas with higher refractive indices darker in the resulting image, while sparser areas with lower refractive indices are rendered brighter. Phase-contrast images can be generated using various techniques, including generation from a Z-stack of bright-field images.

[0051] As used in this article, "Z-stack" or "Z-scan" of brightfield images refers to a digital image processing method that combines multiple images taken at different focal lengths to provide a composite image with a greater depth of field (i.e., the thickness of the focal plane) compared to any single source brightfield image.

[0052] As used in this article, the "focal plane" refers to a plane perpendicular to the lens axis of an optical microscope, on which biological specimens can be observed at the optimal focal point.

[0053] As used in this article, “defocus” refers to the distance above or below the focal plane that allows a biological specimen to be observed in a defocused state.

[0054] As used in this article, a “merging mask” refers to a binary image in which pixels are identified as belonging to one or more cells in a biological specimen, such that pixels corresponding to one or more cells are assigned a value of 1, and the remaining pixels corresponding to the background are assigned a value of 0, and vice versa.

[0055] As used in this article, a “cell image” refers to an image generated based on at least two bright-field images obtained in different planes to enhance the contrast of cells relative to the background.

[0056] As used in this article, a “seed mask” refers to a binary pixelated image generated based on a set pixel intensity threshold.

[0057] As used herein, a “cell-by-cell segmentation mask” refers to an image with binary pixelation (i.e., the processor assigns a value of 0 or 1 to each pixel) such that the cells of biosample 110 are each displayed as distinct regions of interest. A cell-by-cell segmentation mask can advantageously allow for label-free counting of the cells displayed therein, allow for determination of the entire area of ​​a single adherent cell, allow for analysis based on cell texture metrics and cell shape descriptors, and / or allow for detection of the boundaries of individual cells, including for adherent cells that tend to form sheets, where each cell can contact many other neighboring cells in biosample 110.

[0058] As used in this paper, "region growing iteration" refers to a single step in an iterative image segmentation method in which the region of interest ("ROI") is iteratively expanded by acquiring one or more initially identified single or multiple sets of pixels (i.e., "seeds") and adding neighboring pixels to those sets. The processor uses a similarity metric to determine which pixels to add to the growing region, and a stopping criterion is defined for the processor to determine when region growing is complete.

[0059] Now for reference Figures 3 to 5 ,use Figures 1 to 2 The computing device illustrates method 300. Method 300 includes, at block 305, processor 202 generating at least one phase-contrast image 400 of a biological specimen 110, the biological specimen comprising one or more cells centered on a focal plane of the biological specimen 110. Then, at block 310, processor 202 generates a confluence mask 410 in the form of a binary image based on the at least one phase-contrast image 400. Next, at block 315, processor 202 receives a first bright-field image 415 of one or more cells in the biological specimen 110 at a defocused amount above the focal plane, and receives a second bright-field image 420 of one or more cells in the biological specimen 110 at a defocused amount below the focal plane. Then, at block 320, processor 202 generates a cell image 425 of the one or more cells in the biological specimen based on the first bright-field image 415 and the second bright-field image 420. At block 325, processor 202 generates a seed mask 430 based on the cell image 425 and the at least one phase-contrast image 400. In box 330, processor 202 generates an image of one or more cells in a biological specimen based on seed mask 430 and confluence mask 410, the image showing cell-by-cell segmentation mask 435.

[0060] like Figure 3 As shown, in block 305, the processor 202 generating at least one phase-contrast image 400 of the biological specimen 110 (including one or more cells of the biological specimen 110 centered on the focal plane) includes: the processor 202 receiving a Z-scan of a bright-field image, and then generating at least one phase-contrast image 400 based on the Z-scan of the bright-field image. In various embodiments, the biological specimen 110 may be dispersed in multiple wells in a well plate representing an experimental group.

[0061] In an alternative embodiment, the method includes processor 202 receiving at least one fluorescence image and then calculating the fluorescence intensity of one or more cells in a biosample 110 within a cell-by-cell segmentation mask 435. In this embodiment, the fluorescence intensity corresponds to the level of a protein of interest, such as an antibody labeling a cell surface marker (e.g., CD20) or an annexin-V reagent that induces fluorescence corresponding to cell death. Furthermore, determining the fluorescence intensity within individual cell boundaries can enhance subpopulation identification and allow for the calculation of subpopulation-specific measures (e.g., the average area and eccentricity of all dead cells, as defined by the presence of annexin-V).

[0062] In another embodiment, at block 310, the processor 202 generates a confluence mask 410 in binary image form based on at least one contrasting image 400, including applying one or more local texture filters or brightness filters to enable the identification of pixels belonging to one or more cells in the biological specimen 110. Example filters may include, but are not limited to, local range filters, local entropy filters, local standard deviation filters, local brightness filters, and Gabor wavelet filters. Figure 4 and Figure 5 Example confluence mask 410 is shown.

[0063] In yet another alternative embodiment, the optical microscope 105 determines the focal plane of the biological specimen 110. Furthermore, in various embodiments, the defocus amount can range from 20 µm to 60 µm. The optimal defocus amount is determined based on the optical characteristics of the objective lens used, including its magnification and working distance.

[0064] exist Figure 5 In another embodiment shown, at block 320, the processor 202 generates a cell image 425 based on a first bright-field image 415 and a second bright-field image 420 by: the processor 202 enhancing the first bright-field image 415 and the second bright-field image 420 based on a third bright-field image 405 centered on the focal plane, utilizing at least one of a plurality of pixel-level mathematical operations or feature detection. An example of pixel-level mathematical operations includes addition, subtraction, multiplication, division, or any combination of these operations. The processor 202 then calculates transformation parameters to align the first bright-field image 415 and the second bright-field image 420 with at least one contrasting image 400. Next, the processor 202 combines the brightness level of each pixel in the aligned second bright-field image 420 with the brightness level of the corresponding pixel in the aligned first bright-field image 415 to form the cell image 425. The combination of brightness levels of each pixel can be achieved by any of the mathematical operations described above. The technical effect of generating the cell image 425 is to remove bright-field artifacts (e.g., shadows) and enhance image contrast to increase cell detection of the seed mask 430.

[0065] In yet another alternative embodiment, at block 320, the processor 202 generates a cell image 425 of one or more cells in the biological specimen 110 based on the first bright-field image 415 and the second bright-field image 420, including: the processor 202 receiving one or more user-defined parameters that determine one or more threshold levels and one or more filter sizes. The processor 202 then applies one or more smoothing filters to the cell image 425 based on the one or more user-defined parameters. The technical effect of the smoothing filters is to further increase the accuracy of cell detection in the seed mask 430 and increase the likelihood that each cell will be assigned a seed. The smoothing filter parameters are selected to accommodate different adherent cell morphologies, such as flat vs. round, protruding cells, clustered cells, etc.

[0066] In another alternative embodiment, at block 325, the processor 202 generates a seed mask 430 based on the cell image 425 and at least one contrasting image 400, comprising: the processor 202 modifying the cell image 425 such that each pixel at or above a threshold pixel intensity is identified as a cell seed pixel, thereby producing a seed mask 430 with binary pixelation. The technical effect of binary pixelation of the seed mask is that it allows comparison with the corresponding binary pixelation of the confluence mask. The binary pixelation of the seed mask is also used as the starting point for the region growing iterations discussed below. For example, in yet another alternative embodiment, the seed mask 430 may have multiple seeds, each seed corresponding to a single cell in the biological specimen 110. In this embodiment, method 300 further includes, before processor 202 generates an image showing one or more cells in a biological specimen illustrating cell-by-cell segmentation mask 435, processor 202 compares seed mask 430 and confluence mask 410, and eliminates one or more regions from seed mask 430 that are not arranged in confluence mask 410, and eliminates one or more regions from confluence mask 410 that do not contain one of a plurality of seeds from seed mask 430. The technical effect of these eliminated regions is to exclude small, bright objects (e.g., cell debris) that generate seeds, and to increase the recognition of seeds used in the region growth iterations described below.

[0067] In another alternative embodiment, at block 330, the processor 202 generates an image of one or more cells in a biological specimen 110 showing a cell-by-cell segmentation mask 435 based on the seed mask 430 and the confluence mask 410, comprising: the processor 202 performing a region growth iteration for each of the active seed sets. The processor 202 then repeats the region growth iteration for each seed in the active seed set until the growth region of a given seed reaches one or more boundaries of the confluence mask 410 or overlaps with the growth region of another seed. The processor 202 selects the active seed set for each iteration based on attributes of corresponding pixel values ​​in the cell image. Furthermore, the technical effect of using at least one phase-contrast image 400 and a first bright-field image 415, a second bright-field image 420, and a third bright-field image 405 is that the seeds correspond to bright spots in the cell image 425 and high-texture regions in the phase-contrast image 400 (i.e., the overlap between the confluence mask 410 and the seed mask 430 will be described in more detail below). Another technical effect of using the confluence mask 410, at least one phase-contrast image, and the first bright-field image 415, the second bright-field image 420, and the third bright-field image 405 is the increased accuracy in identifying the location of individual cells and cell boundaries in the cell-by-cell segmentation mask 435. As an example, this advantageously allows for the quantification of features such as cell surface protein expression.

[0068] In yet another alternative embodiment, method 300 may include processor 202 applying one or more filters in response to user input to remove objects based on one or more cell texture metrics and cell shape descriptors. Processor 202 then modifies an image of a biological specimen showing a cell-by-cell segmentation mask in response to the application of the one or more filters. Example cell texture metrics and cell shape descriptors include, but are not limited to, cell size, perimeter, eccentricity, fluorescence intensity, aspect ratio, solidity, Feret's diameter, phase contrast entropy, and phase contrast standard deviation.

[0069] In another alternative embodiment, method 300 may include processor 202 determining a cell count of the biosample 110 based on an image of one or more cells in the biosample 110 illustrating a cell-by-cell segmentation mask 435. As a result of the defined cell boundaries shown in the cell-by-cell segmentation mask 435, the aforementioned cell counting is advantageously permitted, for example... Figure 4As shown. In one alternative embodiment, one or more cells in the biological specimen 110 are one or more of adherent and non-adherent cells. In another embodiment, adherent cells may include: one or more different cancer cell lines, including human lung cancer cells, fibroblast cells, breast cancer cells, and ovarian cancer cells; or human microvascular cell lines, including human umbilical vein cells. In an alternative embodiment, processor 202 performs region growth iterations such that different smoothing filters are applied to non-adherent cells (including human immune cells, such as PMBC and Jurkat cells) instead of adherent cells to improve the approximation of cell boundaries.

[0070] As described above, the non-transitory computer-readable medium stores program instructions thereon, which, when executed by the processor 202, can perform any of the functions of the aforementioned method.

[0071] As an example, a non-transitory computer-readable medium stores program instructions thereon that, when executed by processor 202, perform a set of actions including that processor 202 generates at least one phase-contrast image 400 of the biological specimen 110, comprising one or more cells, based on at least one third bright-field image 405 centered on the focal plane of the biological specimen 110. Then, processor 202 generates a confluence mask 410 in the form of a binary image based on the at least one phase-contrast image 400. Next, processor 202 receives a first bright-field image 415 of one or more cells in the biological specimen 110 at a defocused amount above the focal plane and a second bright-field image 420 of one or more cells in the biological specimen 110 at a defocused amount below the focal plane. Then, processor 202 generates a cell image 425 of one or more cells based on the first bright-field image 415 and the second bright-field image 420. Processor 202 also generates a seed mask 430 based on the cell image 425 and the at least one phase-contrast image 400. Furthermore, the processor 202 generates images of one or more cells in the biological specimen 110 based on the seed mask 430 and the confluence mask 410, the images showing the cell-by-cell segmentation mask 435.

[0072] In an alternative embodiment, the non-transitory computer-readable medium further includes enabling the processor 202 to receive at least one fluorescence image and enabling the processor 202 to calculate the fluorescence intensity of one or more cells in a biological specimen within a cell-by-cell segmentation mask.

[0073] In another alternative embodiment, the non-transitory computer-readable medium further includes enabling processor 202 to generate a seed mask 430 based on cell image 425 and at least one contrasting image 400. The non-transitory computer-readable medium also includes enabling processor 202 to modify cell image 425 such that each pixel at or above a threshold pixel intensity is identified as a cell seed pixel, thereby causing seed mask 430 to have binary pixelation.

[0074] In yet another alternative embodiment, the seed mask 430 has a plurality of seeds, each seed corresponding to a single cell. The non-transitory computer-readable medium also includes, before the processor 202 generates an image showing one or more cells in the biological specimen 110 of the cell-by-cell segmentation mask 435, causing the processor 202 to compare the seed mask 430 and the confluence mask 410, and to eliminate one or more regions from the seed mask 430 that are not arranged in the confluence mask 410, and to eliminate one or more regions from the confluence mask 410 that do not contain one of the plurality of seeds of the seed mask 430.

[0075] In another alternative embodiment, program instructions causing processor 202 to generate an image showing one or more cells in a biological specimen 110 illustrating a cell-by-cell segmentation mask 435 based on seed mask 430 and confluence mask 410 include: processor 202 performing a region growth iteration for each of the active seed sets. The non-transitory computer-readable medium then further includes causing processor 202 to repeat the region growth iteration for each seed in the active seed sets until the growth region of a given seed reaches one or more boundaries of confluence mask 410 or overlaps with the growth region of another seed.

[0076] The non-transitory computer-readable medium also includes enabling processor 202 to apply one or more filters in response to user input to remove objects based on one or more cell texture metrics and cell shape descriptors. Furthermore, processor 202 modifies an image of biological specimen 110 showing a cell-by-cell segmentation mask 435 in response to the application of the one or more filters.

[0077] IV. Experimental Results

[0078] Exemplary implementations allow for tracking cell health across subpopulations over time. For example... Figure 6A Experimental results are shown, generated according to an exemplary embodiment, of a cell-by-cell segmentation mask representing the phase-contrast image response 24 hours after the time course of apoptosis in HT1080 fibrosarcoma following camptothecin (CPT, cytotoxic) treatment. (Using Incucyte...) ® NucLightRed (nuclear active label) and interference-free Incucyte ®Multiple readings of the Caspase 3 / 7 green reagent (apoptosis indicator) are used to determine cell health. Figure 6B It shows according to Figure 6A The implementation plan uses Incucyte ® Cell-by-cell analysis software tool, classifying cell subpopulations based on red and green fluorescence. Figure 6C It shows according to Figure 6A According to the implementation plan, after CPT treatment, the red population decreased (indicating loss of live cells), red and green fluorescence increased (indicating early apoptosis), and green fluorescence increased after 24 hours (indicating late apoptosis). Figure 6D It shows according to Figure 6A The implementation scheme, concentration response time process of the early apoptotic population (percentage of total cells showing red and green fluorescence). The values ​​shown are the mean ± SEM of 3 wells.

[0079] In another example, Figure 6E Experimental results are shown using a cell-by-cell segmentation mask generated according to an exemplary embodiment, representing the cell image response 24 hours after the time course of apoptosis in HT1080 fibrosarcoma following cyclohexylamide (CHX, which inhibits cell growth). (Using Incucyte) ® NucLight Red (nuclear active labeling) and interference-free Incucyte ® Multiple readings of the Caspase 3 / 7 green reagent (apoptosis indicator) are used to determine cell health. Figure 6F It shows according to Figure 6E The implementation plan uses Incucyte ® Cell-by-cell analysis software tool, classifying cell subpopulations based on red and green fluorescence. Figure 6G It shows according to Figure 6E The implementation scheme showed that apoptosis was absent after CHX treatment, but cell count was reduced (data not shown). Figure 6H It shows according to Figure 6E The implementation scheme, concentration response time process of the early apoptotic population (percentage of total cells showing red and green fluorescence). The values ​​shown are the mean ± SEM of 3 wells.

[0080] Figure 7A The use of Incucyte according to an exemplary implementation is shown. ® Software-generated cell-by-cell segmentation analysis was performed using a cell-by-cell segmentation mask applied to a phase-contrast image for label-free cell counting of adherent cells. Both label-free cell-by-cell analysis and erythrocyte nuclear counting analysis were used to analyze A549 cells labeled with NucLight Red reagent at different densities to validate label-free counts over time. Figure 7BThis shows the situation where there is no contrasting image in the background. Figure 7A Cell-by-cell segmentation mask. Figure 7C It shows according to Figure 7A The implementation scheme, the time process of phase count and red count data with different densities. Figure 7D It shows according to Figure 7A The implementation scheme was described, the correlation of count data over 48 hours was analyzed, and an R² value of 1 was shown when the slope was 1. This was replicated across a range of cell types. The values ​​shown are the mean ± SEM values ​​from 4 wells.

[0081] V. Other Examples and Experimental Data

[0082] For example, the following functions can be performed by operating environment 100. (See reference) Figure 4 The optical microscope 105 captures a first bright-field image 415 of the biological specimen 110 at a first focal plane and a second bright-field image 420 of the biological specimen 110 at a second focal plane. Next, the operating environment 100 generates a cell image 425 by performing pixel-level mathematical operations on the first bright-field image 415 and the second bright-field image 420.

[0083] Figure 8 This is a schematic diagram of three focal planes. The first focal plane 611 is located at a defocus point 617 above the third focal plane 615, at which biological specimens can be observed at an improved focal point relative to the first and second focal planes 611 and 613. The second focal plane 613 is located at a defocus point 617 below the third focal plane 615. In some examples, the defocus point ranges from 20 μm to 60 μm.

[0084] Figure 9 This is a schematic diagram of the operation of the operating environment 100 executed by the processor 202, which executes instructions stored in the data memory 206. For example, the processor 202 executes a computational model 601, which may take the form of, for example, an artificial neural network (ANN) or a convolutional neural network (CNN).

[0085] An ANN or CNN consists of artificial neurons called nodes. Each node can transmit data to other nodes. The receiving node then processes the data and can send it to the nodes connected to it. The data typically consists of numbers, and the output of each node is computed as a function of the sum of its inputs (e.g., non-linear). These connections are called edges. Nodes and edges typically have weights that adjust as learning progresses. Weights are increased or decreased in strength and determine the direction of data at the connection. Nodes may have thresholds, such that data is only sent when the aggregated data exceeds the threshold. Typically, nodes are aggregated into layers. Different layers can perform different transformations on their inputs. Data may travel from the first layer (input layer) to the last layer (output layer), possibly traversing multiple layers multiple times.

[0086] Operating environment 100 uses computational model 601 to process cell image 425 and phase-contrast image 400 of biological specimen 110 to generate output data 609. Cell image 425 is a composite of a first bright-field image 415 of biological specimen 110 at a first focal plane 611 and a second bright-field image 420 of biological specimen 110 at a second focal plane 613. For example, operating environment 100 processes cell image 425 and phase-contrast image 400 according to nodes, connections, and weights defined by computational model 601.

[0087] In some examples, when the operating environment 100 is input into the computational model 601, it immediately processes the synthesis of the cell image 425 and the contrast image 400 into two corresponding channels of image information that overlap with each other.

[0088] In other examples, operating environment 100 processes cell image 425 via a first channel (e.g., input channel) of computational model 601 and contrast image 400 via a second channel (e.g., different input channel) of computational model 601. Thus, in this example, operating environment 100 processes a first output of the first channel and a second output of the second channel to generate output data 609 or to generate intermediate data for obtaining output data 609.

[0089] Output data 609 typically includes information about biological specimen 110. For example, output data 609 may estimate the location and / or extent (e.g., boundaries, area, or volume) of one or more cells in biological specimen 110. Other examples of output data 609 are described below.

[0090] Next, the operating environment 100 compares the output data 609 with the reference data 615. The reference data 615 is typically “real data” generated by a human that represents information about the biological specimen 110. For example, the output data 609 may include human-generated markers indicating the location and / or extent of cells in the biological specimen 110. In some examples, the operating environment 100 generates a computer-implemented transformation 619 of the human-generated data, which may also be incorporated as part of the reference data 615. Examples of computer-implemented transformations include rotation, magnification, translation, and / or resolution changes. Therefore, the operating environment 100 is able to improve the computational model 601 in a self-supervised or semi-supervised manner.

[0091] Then, the operating environment 100 improves the calculation model 601 based on the comparison between the output data 609 and the reference data 615. For example, the operating environment 100 may calculate the pixel-wise brightness and / or color difference between the output data 609 and the reference data 615, and adjust the nodes, connections, and / or weights of the calculation model 601 so that the output data 609 generated by the calculation model 601 better matches the reference data 615.

[0092] Subsequently, the operating environment 100 processes the additional image pairs according to the computational model 601 to further improve the computational model 601 based on a comparison of the additional output data generated by the computational model with the additional reference data. The additional image pairs include cell images 425 and phase-contrast images 400, which correspond to other biological specimens 110 or other views of the same biological specimens 110 described above. The additional reference data 615 corresponds to the additional biological specimens 110 or other views of the same biological specimens 110 described above.

[0093] More specifically, the operating environment 100 may improve the computational model 601 (e.g., adjust the nodes, connections, and / or weights of the computational model 601) to reduce the sum of the corresponding differences between the additional output data and the additional reference data. Therefore, the operating environment 100 may adjust the nodes, connections, and / or weights of the computational model 601 so that the collective output data optimally matches the collective reference data overall.

[0094] like Figure 10 As shown, output data 609 can represent an estimate of the location and / or extent of cell 621 within biological specimen 110. In this case, reference data 615 correctly defines the location and / or extent of cell 621. Although Figure 10 The diagram shows that output data 609 and reference data 615 are the same, but this is not usually the case.

[0095] In some examples, if the biological specimen has a fluorescent label, output data 609 represents an estimate of the appearance of the biological specimen 110. In this case, reference data 615 is generated from or includes an actual image of the biological specimen 110 with the fluorescent label.

[0096] In some examples, output data 609 represents an estimate of the location and / or extent of one or more cell nuclei within biosample 110. In this case, reference data 615 correctly defines the location and / or extent of one or more cell nuclei within biosample 110. Additionally or alternatively, reference data 615 represents processed fluorescence data corresponding to biosample 110. This fluorescence data can be processed to identify cell nuclei.

[0097] refer to Figure 11 Output data 609 can represent an estimate of the classification of the first part 631 of biological specimen 110 into a first category and the classification of the second part 633 of biological specimen 110 into a second category (e.g., live cells versus dead cells; stem cells versus lineage-specific cells; undifferentiated cells versus differentiated cells; epithelial cells versus mesenchymal cells; wild-type cells versus mutant cells; cells expressing a specific protein of interest versus cells not expressing that specific protein of interest; etc.). Other categories are also possible. In this case, reference data 615 correctly defines the classification of the first part 631 of biological specimen 110 into the first category and the classification of the second part 633 of biological specimen 110 into the second category.

[0098] In other examples, output data 609 indicates that the whole of biological specimen 110 is classified into a single category. In this case, reference data 615 correctly classifies the whole of biological specimen 110 into a single category of two or more categories (e.g., healthy vs. unhealthy, malignant vs. benign, wild type vs. mutant).

[0099] Figure 12 and Figure 13 These are block diagrams for methods 501 and 701, respectively. For example, methods 501 and 701, along with their related functions, can be executed by the operating environment 100. Figure 12 and Figure 13 As shown, methods 501 and 701 include one or more operations, functions, or actions as illustrated in boxes 503, 505, 507, 509, 702, 704, 706, 708, 710, and 712. Although these boxes are shown in a sequential order, they may also be executed in parallel and / or in an order different from that described herein. Furthermore, multiple boxes may be combined into fewer boxes, split into additional boxes, and / or removed based on the target implementation.

[0100] In block 503, method 501 includes processing cell image 425 and phase-contrast image 400 of biological specimen 110 using computational model 601 to generate output data 609. Cell image 425 is a composite of a first bright-field image 415 of biological specimen 110 at a first focal plane 611 and a second bright-field image 420 of biological specimen 110 at a second focal plane 613.

[0101] In box 505, method 501 includes performing a comparison of output data 609 and reference data 615.

[0102] In box 507, method 501 includes improving computational model 601 based on a comparison of output data 609 and reference data 615.

[0103] In box 509, method 501 includes subsequently processing additional image pairs according to computational model 601 to further improve computational model 601 based on a comparison of additional output data generated by computational model 601 with additional reference data.

[0104] In block 702, method 701 includes capturing a first bright-field image 415 of the biological specimen 110 at a first focal plane 611 and a second bright-field image 420 of the biological specimen 110 at a second focal plane 613 via an optical microscope 105.

[0105] In box 704, method 701 includes generating a cell image 425 of biological specimen 110 by performing pixel-level mathematical operations on a first brightfield image 415 and a second brightfield image 420.

[0106] In box 706, method 701 includes processing cell image 425 and phase-contrast image 400 of biological specimen 110 using computational model 601 to generate output data 609.

[0107] In box 708, method 701 includes performing a comparison of output data 609 and reference data 615.

[0108] In box 710, method 701 includes improving computational model 601 based on a comparison of output data 609 and reference data 615.

[0109] In box 712, method 701 includes subsequently processing additional image pairs according to computational model 601 to further improve computational model 601 based on a comparison of additional output data generated by computational model 601 with additional reference data.

[0110] Figure 14Images related to the generation of cell-by-cell segmentation masks using a computational model are shown. Incorporating cell image 425 (e.g., information from the first brightfield image 415 and the second brightfield image 420) improves the performance of computational model 601. Computational model 601 is better at decomposing clusters (e.g., dense groups of cells). Models that only include phase-contrast images as input are more likely to produce false-positive cell identifications due to plate texture.

[0111] Figure 15 The results of computational model 601 (e.g., phase + cell) and models that only include phase-contrast images as input (e.g., phase only) are shown. Both the reference box mAP (mean average precision) metric and the mask mAP metric are used. Computational model 601 produces a higher score than the "phase only" model, indicating that computational model 601 improves cell recognition or classification. Computational model 601 is generally more robust than other models because the addition of cell image data provides more robust training information.

[0112] The mean average precision (mAP) score is used to compare the cell-by-cell segmentation mask of the computational model with a manually annotated reference cell-by-cell segmentation mask. Box mAP refers to the score computed on the cell-by-cell bounding box, while mask mAP refers to the score computed on the cell-by-cell mask.

[0113] Various advantageous arrangements have been described for purposes of illustration and description, but are not intended to be exhaustive or limited to the forms presented. Many modifications and variations will be apparent to those skilled in the art. Furthermore, different advantageous examples may describe different advantages compared to other advantageous examples. The selection and description of one or more examples are intended to best explain the principles and practical applications of the examples, and to enable those skilled in the art to understand the various examples of this disclosure and the various modifications suitable for the particular intended use.

Claims

1. A method for analyzing images of biological specimens using a computational model, the method comprising: The computational model is used to process the cell images and phase-contrast images of the biological specimen to generate output data, wherein the cell images are a composite of a first bright-field image of the biological specimen at a first focal plane and a second bright-field image of the biological specimen at a second focal plane. Perform a comparison between the output data and the reference data; The computational model is improved based on the comparison between the output data and the reference data; as well as Subsequently, additional image pairs are processed according to the computational model to further improve the computational model based on a comparison between additional output data generated by the computational model and additional reference data.

2. The method according to claim 1, further comprising generating the cell image by performing pixel-level mathematical operations on the first bright-field image and the second bright-field image.

3. The method of claim 1, wherein the first focal plane is located at a defocus amount above the third focal plane, at which the biological specimen can be observed at an improved focal point relative to the first focal plane and the second focal plane, and wherein the second focal plane is located at the defocus amount below the third focal plane.

4. The method according to claim 3, wherein the defocusing amount is in the range of 20 μm to 60 μm.

5. The method according to any one of claims 1 to 4, wherein the reference data comprises computer-implemented transformations of human-generated data.

6. The method according to any one of claims 1 to 4, further comprising generating a computer-implemented transformation of human-generated data, wherein the reference data includes the computer-implemented transformation.

7. The method according to any one of claims 1 to 4, wherein improving the computational model comprises: The computational model is improved using a self-supervised or semi-supervised approach.

8. The method according to any one of claims 1 to 4, wherein the output data represents an estimate of the location and extent of cells within the biological specimen.

9. The method of claim 8, wherein the reference data correctly defines the location and extent of the cell.

10. The method according to any one of claims 1 to 4, wherein if the biological specimen has a fluorescent label, the output data represents an estimate of the appearance of the biological specimen.

11. The method of claim 10, wherein the reference data is generated from an actual image of the biological specimen having a fluorescent label.

12. The method according to any one of claims 1 to 4, wherein the output data represents an estimate of the location and extent of cell nuclei in the biological specimen.

13. The method of claim 12, wherein the reference data correctly defines the location and extent of the cell nucleus within the biological specimen.

14. The method of claim 13, wherein the reference data represents processed fluorescence data corresponding to the biological specimen.

15. The method according to any one of claims 1 to 4, wherein the output data represents an estimate of classifying a first portion of the biological specimen into a first category and a second portion of the biological specimen into a second category.

16. The method of claim 15, wherein the reference data correctly defines classifying the first portion of the biological specimen as the first category and classifying the second portion of the biological specimen as the second category.

17. The method according to any one of claims 1 to 4, wherein the output data represents classifying the entire biological specimen into a category.

18. The method of claim 17, wherein the reference data correctly classifies the entire biological specimen into the category.

19. The method according to any one of claims 1 to 4, wherein processing the additional image pair according to the computational model to further improve the computational model comprises: The computational model is improved to reduce the sum of the corresponding differences between the additional output data and the additional reference data.

20. The method according to any one of claims 1 to 4, wherein processing the cell image of the biological specimen and the phase-contrast image of the biological specimen using the computational model comprises: The cell images and phase-contrast images of the biological specimen are processed using an artificial neural network.

21. The method according to any one of claims 1 to 4, wherein processing the cell image of the biological specimen and the phase-contrast image of the biological specimen using the computational model comprises: The cell images and phase-contrast images of the biological specimen are processed using a convolutional neural network.

22. The method according to any one of claims 1 to 4, wherein processing the cell image of the biological specimen and the phase-contrast image of the biological specimen using the computational model comprises: The synthesis of the cell image and the phase-contrast image is processed.

23. The method according to any one of claims 1 to 4, wherein processing the cell image of the biological specimen and the phase-contrast image of the biological specimen using the computational model comprises: The cell image is processed via the first channel of the computational model, and the phase-contrast image is processed via the second channel of the computational model.

24. The method of claim 23, wherein processing the cell image and the phase-contrast image of the biological specimen using the computational model further comprises: The first output of the first channel and the second output of the second channel are processed to generate the output data.

25. A non-transitory data memory for storing instructions, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 24.

26. A system for analyzing biological specimens, the system comprising: Optical microscope; One or more processors; as well as A non-transitory data memory that stores instructions, which, when executed by the one or more processors, cause the system to perform the method according to any one of claims 1 to 24.

27. A system for analyzing biological specimens, the system comprising: Optical microscope; One or more processors; as well as A non-transitory data memory that stores instructions, which, when executed by the one or more processors, cause the system to perform functions including: The optical microscope captures a first bright-field image of the biological specimen at a first focal plane and a second bright-field image of the biological specimen at a second focal plane; Cell images of the biological specimen are generated by performing pixel-level mathematical operations on the first bright-field image and the second bright-field image; The cell images and phase-contrast images of the biological specimens are processed using a computational model to generate output data; Perform a comparison between the output data and the reference data; The computational model is improved based on the comparison between the output data and the reference data; as well as Subsequently, additional image pairs are processed according to the computational model to further improve the computational model based on a comparison between additional output data generated by the computational model and additional reference data.

Images