Computational models for analyzing images of biological specimens

Abstract

A method for analyzing images of a biological specimen using a computational model is described, the method including processing a cellular image of the biological specimen and a phase-contrast image of the biological specimen using the computational model to generate output data. The cellular image is a composite of a first bright-field image of the biological specimen at a first focal plane and a second bright-field image of the biological specimen at a second focal plane. The method further includes performing a comparison of the output data and reference data and refining the computational model based on the comparison of the output data and the reference data. The method further includes thereafter processing additional image pairs according to the computational model to further refine the computational model based on a comparison of additional output data generated by the computational model with additional reference data.

Description

Technical Field

[0001] Cross - reference to Related Applications

[0001] This application is an international application claiming priority to U.S. Patent Application No. 16 / 950,368, filed on November 17, 2020, which is incorporated herein by reference. Also incorporated by reference are U.S. Patent Application No. 16 / 265,910, filed on February 1, 2019, and U.S. Patent Application No. 17 / 099,983, filed on November 17, 2020.

Background Art

[0002] Background

[0002] To analyze labeled or unlabeled images of biological specimens, deep artificial neural networks (ANNs), commonly convolutional neural networks (CNNs), can be used. Fluorescent labels are commonly used to mark, for example, specific proteins, intracellular compartments, or cell types, which gives detailed insights into the organism. However, such labeling can be detrimental to the organism and can also cause phototoxic effects due to the long exposure times required for fluorescence. In unlabeled analysis, the use of ANNs typically requires the analysis of a single microscopic image of a given biological specimen.

Summary of the Invention

Means for Solving the Problems

[0003] Summary

[0003] In an example, the disclosure includes a method for analyzing images of a biological specimen using a computational model, which involves processing cell images and phase-contrast images of the biological specimen using the computational model to generate output data, wherein the cell images are a composite of a first bright-field image of the biological specimen at a first focal plane and a second bright-field image of the biological specimen at a second focal plane; performing a comparison of the output data and reference data; refining the computational model based on the comparison of the output data and reference data; and then processing additional image pairs according to the computational model to further refine the computational model based on a comparison of additional output data generated by the computational model and additional reference data.

[0004]

[0004] In another example, the disclosure includes a non-temporary data storage device that stores instructions causing a computing device to perform a function to analyze images of a biological specimen using a computational model, the function being to process cell images and phase-contrast images of the biological specimen using a computational model to generate output data, wherein the cell images are a composite of a first bright-field image of the biological specimen at a first focal plane and a second bright-field image of the biological specimen at a second focal plane; to perform a comparison of the output data and reference data; to refine the computational model based on the comparison of the output data and reference data; and thereafter to process additional image pairs according to the computational model to further refine the computational model based on a comparison of additional output data and additional reference data generated by the computational model.

[0005]

[0005] In another example, the disclosure includes a system for certifying a biological specimen, comprising: an optical microscope; one or more processors; and a non-temporary data storage device that stores instructions causing the system to perform functions including, if performed by one or more processors, acquiring a first bright-field image of the biological specimen at a first focal plane and a second bright-field image of the biological specimen at a second focal plane via the optical microscope; generating a cellular image of the biological specimen by performing mathematical operations on the pixels of the first and second bright-field images; processing the cellular image and phase-contrast image of the biological specimen using a computational model to generate output data; performing a comparison of the output data and reference data; refining the computational model based on the comparison of the output data and reference data; and then processing additional image pairs according to the computational model to further refine the computational model based on a comparison of additional output data and additional reference data generated by the computational model.

[0006]

[0006] The features, functions, and benefits described can be achieved independently in various examples, or they may be combined in other examples with further details which can be seen by referring to the following description and drawings. [Brief explanation of the drawing]

[0007] Brief explanation of the drawing [Figure 1]

[0007] This is a functional block diagram of the environment, representing one example of an implementation configuration. [Figure 2]

[0008] Block diagrams of computing devices and computer networks are shown, illustrating examples of implementation configurations. [Figure 3]

[0009] A flowchart of the method is shown, illustrating the implementation configuration. [Figure 4]

[0010] Images of biological specimens are shown as examples of implementation configurations. [Figure 5]

[0011] An image of another biological specimen is shown, illustrating an example of its implementation. [Figure 6A]

[0012] This paper presents experimental results of cell-by-cell splitting masks generated by implementation examples for the cell image response 24 hours after apoptosis of HT1080 fibrosarcoma following camptothecin (CPT, cytotoxic) treatment. [Figure 6B]

[0013] Figure 6A shows cell subsets classified based on the implementation configuration, using red (NucLight Red, cell health indicator, "NucRed") and green fluorescence (caspase 3 / 7, apoptosis indicator). [Figure 6C]

[0014] Figure 6A shows that the implementation configuration resulted in a decrease in the red group after CPT treatment, indicating the loss of viable cells; an increase in red and green fluorescence, indicating early apoptosis; and an increase in green fluorescence after 24 hours, indicating late apoptosis. [Figure 6D]

[0015] Figure 6A shows the concentration response time course of the initial apoptotic population (percentage of total cells exhibiting red and green fluorescence) based on the implementation configuration. [Figure 6E]

[0016] This paper presents experimental results of cell-by-cell splitting masks generated according to implementation examples for cell image responses 24 hours after HT1080 fibrosarcoma apoptosis following cyclohexaamide (CHX, cell proliferation inhibitory). [Figure 6F]

[0017] Figure 6E shows cell subsets classified based on their implementation configuration, using red (NucLight Red, cell health indicator, "NucRed") and green fluorescence (caspase 3 / 7, apoptosis indicator). [Figure 6G]

[0018] Figure 6E shows that, with the implementation configuration, there was no apoptosis after CHX treatment, although there was a decrease in the number of cells. [Figure 6H]

[0019] Figure 6E shows the concentration response time course of the initial apoptotic population (percentage of total cells exhibiting red and green fluorescence) based on the implementation configuration. [Figure 7A]

[0020] The cell-by-cell segmentation mask superimposed on the phase contrast image for label-free cell counting of adherent cells using cell-by-cell segmentation analysis generated by an example embodiment is shown. Various densities of A549 cells labeled with the NucLight Red reagent were analyzed in both label-free cell-by-cell analysis and red nucleus counting analysis to confirm label-free counting over time. [Figure 7B]

[0021] The cell-by-cell segmentation mask according to FIG. 7A without a phase contrast image in the background is shown. [Figure 7C]

[0022] The time course of phase number and NucRed number data over density according to the embodiment of FIG. 7A is shown. [Figure 7D]

[0023] The correlation of number data over 48 hours according to the embodiment of FIG. 7A is shown, showing an R2 value of 1 with a slope of 1. [Figure 8]

[0024] It is a schematic diagram of three focal planes. [Figure 9]

[0025] It is a schematic diagram of the operation of the environment. [Figure 10]

[0026] It is a schematic diagram of output data and reference data. [Figure 11]

[0027] It is a schematic diagram of output data and reference data. [Figure 12]

[0028] It is a block diagram of a method. [Figure 13]

[0029] It is a block diagram of a method. [Figure 14]

[0030] Images related to image classification are shown. [Figure 15]

[0031] The results of the calculation model are shown.

[0008]

[0032] The drawings are for illustrative examples and the present invention is not limited to the configurations and means shown in the drawings.

Embodiments for Carrying Out the Invention

[0009] Detailed Description I. Overview

[0033] Embodiments of the methods described herein can be used to split phase-contrast images of one or more cells in a biological specimen using out-of-focus bright-field imaging, thereby enabling individual cell and subpopulation analysis in high processing times. Furthermore, the examples of the method disclosure beneficially enable real-time label-free (i.e., non-fluorescent) counting of cells, avoiding the influence of fluorescent markers which can compromise the viability and function of living cells. A further advantage of the examples of the method disclosure is the detection of individual cell boundaries, including squamous cells such as HUVECs, regardless of the complexity of cell morphology.

[0010] II. Examples of Architectures

[0034] Figure 1 is a block diagram showing an operating environment 100 that includes, for example, an optical microscope 105 and a biological specimen 110 having one or more cells. The method 300 shown in Figures 3 to 5 below illustrates an embodiment of the method that can be carried out within this operating environment 100.

[0011]

[0035] Figure 2 is a block diagram showing an example of a computing device 200, configured to interface directly or indirectly with the operating environment 100. The computing device 200 may be used to perform the functions of the methods shown in Figures 3 to 5, which will be described later. In particular, the computing device 200 may be configured to perform one or more functions, including, for example, an image generation function that is partially based on images obtained by an optical microscope 105. The computing device 200 has a processor 202, a communication interface 204, a data storage device 206, an output interface 208, and a display 210, each connected to a communication bus 212. Furthermore, the computing device 200 may include hardware that enables communication within the computing device 200 and between the computing device 200 and other devices (e.g., not shown). The hardware may include, for example, a transmitter, a receiver, and an antenna.

[0012]

[0036] The communication interface 204 may be a wireless interface and / or one or more wired interfaces that enable both short-range and long-range communication with one or more networks 214 or one or more telecomputing devices 216 (e.g., a tablet 216a, a personal computer 216b, a laptop computer 216c, and a mobile computing device 216d). Such a wireless interface may provide communication under one or more wireless communication protocols, such as Bluetooth®, WiFi (e.g., IEEE 802.11 protocol), Long-Term Evolution (LTE), cellular communication, near-field communication (NFC), and / or other wireless communication protocols. Such a wired interface may include an Ethernet interface, a Universal Serial Bus (USB) interface, or a similar interface that communicates via wire, twisted pair wire, coaxial cable, optical link, fiber optic link, or other physical connection to a wired network. Accordingly, the communication interface 204 may be configured to receive input data from one or more devices and to transmit output data to other devices.

[0013]

[0037] Furthermore, the communication interface 204 may include user input devices, such as a keyboard, keypad, touchscreen, touchpad, computer mouse, trackball, and / or other similar devices.

[0014]

[0038] The data storage device 206 may include, or take the form of, one or more computer-readable storage media that are readable or accessible by the processor 202. The computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices, that can be integrated in whole or in part with the processor 202. The data storage device 206 is considered a non-transient computer-readable medium. In some examples, the data storage device 206 can be implemented using a single physical device, such as one optical, magnetic, organic, or other memory or disk storage unit, while in other examples, the data storage device 206 can be implemented using two or more physical devices.

[0015]

[0039] Therefore, the data storage device 206 is a non-temporary computer-readable storage medium and stores executable instructions 218 in the data storage device 206. The instructions 218 include computer-executable code. When the instructions 218 are executed by the processor 202, the processor 202 performs a function. Such functions include, but are not limited to, receiving bright-field images from the optical microscope 105 and generating phase-contrast images, confluence masks, cell images, seed masks, cell-by-cell masks and fluorescence images.

[0016]

[0040] The processor 202 may be a general-purpose processor or a dedicated processor (e.g., a digital signal processor, an application-specific integrated circuit, etc.). The processor 202 may receive input from the communication interface 204, process the input, and generate output that is stored in the data storage device 206 and output to the display 210. The processor 202 may be configured to execute executable instructions 218 (e.g., computer-readable program instructions) that are stored in the data storage device 206 and are executable to provide the functionality of the computing device 200 described herein.

[0017]

[0041] The output interface 208 outputs information to the display 210 or other components. Therefore, the output interface 208 may be similar to the communication interface 204, and may be a wireless interface (e.g., a transmitter) or even a wired interface. The output interface 208 may, for example, send commands to one or more controllable devices.

[0018]

[0042] Furthermore, the computing device 200 shown in Figure 2 may represent, for example, a local computing device 200a in an operating environment 100 that is communicating with an optical microscope 105. This local computing device 200a may perform one or more of the steps of Method 300 described later, may receive input from a user, and / or may transmit image data and user input to the computing device 200 and perform all or part of the steps of Method 300. Furthermore, in one optional example of the embodiment, the Incucyte® platform may be used to perform Method 300 and includes a coupling function between the computing device 200 and the optical microscope 105.

[0019]

[0043] Figure 3 shows a flowchart of an example of a method 300 that achieves cell-to-cell splitting of one or more cells in a biological specimen 110, by example of an implementation. The method 300 shown in Figure 3 is an example of a method that can be used, for example, with the computing device 200 in Figure 2. Furthermore, the device or system may be used or configured to perform the logical functions shown in Figure 3. In some cases, components of the device and / or system may be configured to perform functions in order to configure and build the components in hardware and / or software to enable such performance. Components of the device and / or system may be arranged to be adapted to perform functions, capable of performing functions, or suitable for performing functions, for example, when operated in a particular way. The method 300 may include one or more operations, functions, or actions as illustrated by one or more blocks from blocks 305 to 330. Although the blocks are illustrated in a sequential order, some of these blocks may be executed in parallel and / or in an order different from the order described herein. Furthermore, various blocks may be combined into fewer blocks, divided into further blocks, and / or removed based on the desired implementation.

[0020]

[0044] For this and other processing and methods disclosed herein, the flowcharts should be understood to illustrate the function and operation of one possible implementation of the example. In this regard, each block may represent a module, segment, or portion of program code containing one or more instructions executable by a processor to perform a particular logical function or step in the processing. The program code may be stored, for example, on any type of computer-readable medium or data storage device (e.g., a storage device including a disk or hard drive). Furthermore, the program code may be encoded in a machine-readable format on a computer-readable storage medium, or on other non-temporary medium or product. The computer-readable medium may include, for example, non-temporary computer-readable medium or memory, such as computer-readable medium for short-term storage of data (e.g., register memory, processor cache, and random access memory (RAM)). The computer-readable medium may also include non-temporary medium, such as read-only memory (ROM), optical or magnetic disks, and secondary or persistent long-term storage devices such as compact disk read-only memory (CD-ROM). Furthermore, the computer-readable medium may be any other volatile or non-volatile storage system. The computer-readable medium may be considered, for example, a tangible computer-readable storage medium.

[0021]

[0045] In addition, in other processes and methods disclosed herein, each block in Figure 3 may represent a circuit wired to perform a particular logical function in the process. As will be understood by those skilled in the art, alternative implementations in which the functions can be performed in a sequence deviating from the illustrated or described sequence, including substantially simultaneous or reversed sequences, are included within the scope of the examples of this disclosure, depending on the functions involved.

[0022] III. Examples of Methods

[0046] As used herein, “bright-field image” means an image obtained via a microscope based on a biological specimen illuminated from below so that light waves pass through the transparent parts of the biological specimen. Various brightness levels are then incorporated into the bright-field image.

[0023]

[0047] As used herein, “phase contrast image” means an image obtained directly or indirectly via a microscope based on a biological specimen illuminated from below, capturing the phase shift of light passing through the biological specimen due to differences in refractive index of different parts of the biological specimen. For example, as a light wave travels through a biological specimen, the amplitude (i.e., luminance) and phase of the light wave change in a manner dependent on the properties of the biological specimen. As a result, the phase contrast image has luminance intensity values ​​associated with pixels that change so that denser areas with a higher refractive index appear darker in the resulting image, and thinner areas with a lower refractive index appear brighter in the resulting image. Phase contrast images can be generated via many techniques, including from a Z-stack of brightfield images.

[0024]

[0048] As used herein, “Z-stacking” or “Z-sweeping” of brightfield images means a digital image processing method that combines a number of images taken at different focal lengths to give a composite image having a greater depth of field (i.e., thickness of the focal plane) than any of the individual source brightfield images.

[0025]

[0049] As used herein, “focal plane” means a plane positioned perpendicular to the axis of an optical microscope lens from which a biological specimen can be observed at optimal focus.

[0026]

[0050] As used herein, “defocusing distance” means the distance beyond or below the focal plane at which a biological specimen is observable outside the focal point.

[0027]

[0051] As used herein, “confluence mask” means a binary image that identifies pixels as belonging to one or more cells in a biological specimen, such that pixels corresponding to one or more cells are assigned a value of 1, and the remaining pixels corresponding to the background are assigned a value of 0, or vice versa.

[0028]

[0052] As used herein, “cell image” means an image generated based on at least two bright-field images obtained in different planes to enhance the contrast of cells against the background.

[0029]

[0053] As used herein, “seed mask” means an image having a binary pixelation generated based on a set pixel intensity threshold.

[0030]

[0054] As used herein, “cell-by-cell splitting mask” means an image having binary pixelation (i.e., each pixel is assigned a value of 0 or 1 by the processor) so as to display each cell of the biological specimen 110 as a different region of interest. Advantageously, the cell-by-cell splitting mask allows for label-free counting of the displayed cells, determination of the entire area of ​​individual adherent cells, analysis based on cell texture quantification and cell shape descriptors, and / or detection of individual cell boundaries, including adherent cells that tend to form in a sheet, where each cell may be in contact with many other neighboring cells in the biological specimen 110.

[0031]

[0055] As used herein, “region expansion iteration” means a single step in an iterative image segmentation method that defines a region of interest (“ROI”) by taking one or more initial identified individual pixels or sets of pixels (i.e., seeds) and iteratively expanding those seeds by adding adjacent pixels to the set. The processor uses similarity metrics to determine which pixels to add to the expanded region, and stopping criteria are defined for the processor to determine when the region expansion is complete.

[0032]

[0056] Now, let's explain Figures 3 to 5. Using the computing devices of Figures 1 and 2, we illustrate Method 300. In block 305, Method 300 includes a processor 202 that generates at least one phase-contrast image 400 of the biological specimen 110, which includes one or more cells centered on the focal plane of the biological specimen 110. Next, in block 310, the processor 202 generates a confluence mask 410 in the form of a binary image based on at least one phase-contrast image 400. Next, in block 315, the processor 202 receives a first bright-field image 415 of one or more cells in the biological specimen 110 at a defocus distance beyond the focal plane, and a second bright-field image 420 of one or more cells in the biological specimen 110 at a defocus distance less than the focal plane. Next, in block 320, the processor 202 generates a cell image 425 of one or more cells in the biological specimen based on the first bright-field image 415 and the second bright-field image 420. In block 325, the processor 202 generates a seed mask 430 based on the cell image 425 and at least one phase-contrast image 400. Furthermore, in block 330, the processor 202 generates an image of one or more cells in a biological specimen showing a cell-by-cell splitting mask 435 based on the seed mask 430 and the confluence mask 410.

[0033]

[0057] As shown in Figure 3, in block 305, the processor 202 that generates at least one phase-contrast image 400 of the biological specimen 110, which includes one or more cells centered on the focal plane for the biological specimen 110, includes a processor 202 that receives a Z-sweep of the bright-field image and then generates at least one phase-contrast image 400 based on the Z-sweep of the bright-field image. In various embodiments, the biological specimen 110 may be dispersed in a plurality of depressions in a depression plate representing the experimental set.

[0034]

[0058] In one optional embodiment, method 300 includes a processor 202 that receives at least one fluorescence image within a cell-by-cell split mask 435 and then calculates the fluorescence intensity of one or more cells in a biological specimen 110. In this embodiment, the fluorescence intensity corresponds to the level of a protein of interest, e.g., an antibody that labels a cell surface marker such as CD20, or an annexin V reagent that induces fluorescence corresponding to cell death. Furthermore, determining fluorescence intensity within individual cell boundaries increases subpopulation discrimination and enables the calculation of subpopulation-specific metrics (e.g., the average area and eccentricity of all dysentery cells, as defined by the presence of annexin V).

[0035]

[0059] In another embodiment, the processor 202 in block 310 that generates a confluence mask 410 in the form of a binary image based on at least one phase difference image 400 includes a processor 202 that applies one or more local texture filters or luminance filters to enable identification of pixels belonging to one or more cells in the biological specimen 110. Examples of filters may include, but are not limited to, local range filters, local entropy filters, local standard deviation filters, local luminance filters, and Gabor wavelet filters. Examples of confluence masks 410 are shown in Figures 4 and 5.

[0036]

[0060] In another optional embodiment, the optical microscope 105 determines the focal plane of the biological specimen 110. Furthermore, in various embodiments, the defocusing distance may be in the range of 20 μm to 60 μm. The optimal defocusing distance is determined based on the optical characteristics of the objective lens used, including the magnification and working distance of the objective lens.

[0037]

[0061] In a further embodiment shown in Figure 5, in block 320, the processor 202 that generates a cell image 425 based on a first bright-field image 415 and a second bright-field image 420 includes a processor 202 that improves the first bright-field image 415 and the second bright-field image 420 based on a third bright-field image 405 centered on the focal plane by utilizing at least one of a mathematical operation on a plurality of pixels or feature detection. One example of a mathematical operation on pixels includes addition, subtraction, multiplication, division, or any combination thereof. Next, the processor 202 calculates transformation parameters to align the first bright-field image 415 and the second bright-field image 420 to at least one phase-difference image 400. Next, the processor 202 combines the luminance levels for each pixel in the aligned second bright-field image 420 with the luminance levels of the corresponding pixels in the aligned first bright-field image 415, thereby forming the cell image 425. The combination of luminance levels for each pixel can be achieved through any of the mathematical operations described above. The technical effect of generating cell images 425 is to increase image contrast in order to remove bright-field artifacts (e.g., shadows) and increase cell detection against the seed mask 430.

[0038]

[0062] In another optional embodiment, in block 320, a processor 202 that generates a cell image 425 of one or more cells in a biological specimen 110 based on a first bright-field image 415 and a second bright-field image 420 includes a processor 202 that receives one or more user-defined parameters for determining one or more threshold levels and one or more filter sizes. The processor 202 then applies one or more smoothing filters to the cell image 425 based on one or more user-defined parameters. The technical effect of the smoothing filters is to further increase the accuracy of cell detection in the species mask 430 and to increase the possibility of assigning one species to each cell. Smoothing filter parameters are selected to suit different adherent cell morphologies (e.g., flat vs. round shapes, protruding cells, cluster cells, etc.).

[0039]

[0063] In a further optional embodiment, block 325 includes a processor 202 that generates a seed mask 430 based on a cell image 425 and at least one phase-contrast image 400, which identifies each pixel at or above a threshold pixel intensity as a cell seed pixel, thereby modifying the cell image 425 to have a seed mask 430 with binarization. The technical effect of binarizing the seed mask is that it allows for comparison with the corresponding binarization of the confluence mask. Furthermore, the binarization of the seed mask is used as a starting point for the region expansion iterations described later. For example, in yet another optional embodiment, the seed mask 430 may have multiple species, each corresponding to a single cell in the biological specimen 110. In this embodiment, Method 300 further includes comparing a species mask 430 and a confluence mask 410 before a processor 202 generates an image of one or more cells in a biological specimen showing a cell-by-cell mask 435, removing one or more regions from the species mask 430 that are not located in the area of ​​the confluence mask 410, and removing one or more regions from the confluence mask 410 that do not contain one of the multiple species in the species mask 430. The technical effect of these removed regions is to eliminate small bright objects (e.g., cell debris) that generate species and to enhance species identification for use in the region expansion repeats described later.

[0040]

[0064] In a further optional embodiment, in block 330, a processor 202 that generates images of one or more cells in a biological specimen 110 showing a cell-by-cell splitting mask 435 based on a species mask 430 and a confluence mask 410 includes a processor 202 that performs region expansion iterations for each of the species activity sets. The processor 202 then repeats the region expansion iterations for each species in the species activity set until the expanded region for a given species reaches one or more boundaries of the confluence mask 410 or overlaps with the expanded region of another species. The species activity sets are selected by the processor 202 for each iteration based on the characteristics of the corresponding pixel values ​​in the cell image. Furthermore, a technical effect of using not only bright-field images 415, 420, 405 but also at least one phase-contrast image 400 is that a species corresponds to both a bright spot in the cell image 425 and a high-texture area in the phase-contrast image 400 (i.e., overlap of the confluence mask 410 with the species mask 430, described in more detail later). Another technical effect resulting from using not only bright-field images 415, 420, and 405, but also a confluence mask 410 and at least one phase-contrast image, is, as one example, an increased accuracy in identifying individual cell locations and cell boundaries in the cell-by-cell splitting mask 435, which can favorably quantify features such as cell surface protein expression.

[0041]

[0065] In yet another optional embodiment, method 300 may include a processor 202 that applies one or more filters in response to user input to remove objects based on one or more cell texture metric and cell shape descriptors. The processor 202 then modifies an image of the biological specimen showing a cell-by-cell split mask in response to the application of one or more filters. Examples of cell texture metric and cell shape descriptors include, but are not limited to, cell size, perimeter, eccentricity, fluorescence intensity, aspect ratio, solidity, Ferret diameter, phase difference entropy, and phase difference standard deviation.

[0042]

[0066] In a further optional embodiment, method 300 may include a processor 202 that determines a cell count for a biological specimen 110 based on images of one or more cells in the biological specimen 110 showing a cell-per-segment mask 435. Advantageously, the above cell count is recognized as a result of clear cell boundaries shown in the cell-per-segment mask 435, for example, shown in Figure 4. In one optional embodiment, the one or more cells in the biological specimen 110 are one or more cells of adherent and non-adherent cells. In a further embodiment, adherent cells may include one or more cell lines of various cancer cell lines, including human lung cancer cells, fibrocarcinoma cells, breast cancer cells, ovarian cancer cells, or human microvascular cell lines, including human umbilical vein cells. In an optional embodiment, the processor 202 performs region expansion iterations in such a manner that it applies a different smoothing filter to non-adherent cells, including human immune cells such as PMBCs and Jurkat cells, than the filter applied to adherent cells, thereby improving the approximation of cell boundaries.

[0043]

[0067] As described above, a non-temporary computer-readable medium that stores program instructions available at runtime by processor 202 in order to provide the performance of any of the functions of the above-described method.

[0044]

[0068] As one example, a non-temporary computer-readable medium storing program instructions that, when executed by the processor 202, brings a set of operations including a processor 202 that generates at least one phase-contrast image 400 of a biological specimen 110 containing one or more cells based on at least one bright-field image 405 centered on the focal plane of the biological specimen 110. Next, the processor 202 generates a confluence mask 410 in the form of a binary image based on at least one phase-contrast image 400. Next, the processor 202 receives a first bright-field image 415 of one or more cells in the biological specimen 110 at a defocus distance beyond the focal plane, and a second bright-field image 420 of one or more cells in the biological specimen 110 at a defocus distance less than the focal plane. Next, the processor 202 generates a cell image 425 of one or more cells based on the first bright-field image 415 and the second bright-field image 420. Furthermore, the processor 202 generates a seed mask 430 based on the cell image 425 and at least one phase-contrast image 400. Furthermore, the processor 202 generates an image of one or more cells in the biological specimen 110, showing a cell-by-cell splitting mask 435 based on the seed mask 430 and the confluence mask 410.

[0045]

[0069] In one optional embodiment, the non-temporary computer-readable medium further includes a processor 202 that receives at least one fluorescence image, and a processor 202 that calculates the fluorescence intensity of one or more cells in a biological specimen within a cell-by-cell split mask.

[0046]

[0070] In another optional embodiment, the non-temporary computer-readable medium further includes a processor 202 that generates a seed mask 430 based on a cell image 425 and at least one phase-contrast image 400. Furthermore, the non-temporary computer-readable medium further includes a processor 202 that identifies each pixel at or above a threshold pixel intensity as a cell seed pixel, thereby modifying the cell image 425 to form a seed mask 430 having binary pixelation.

[0047]

[0071] In a further optional embodiment, the seed mask 430 has multiple species, each corresponding to a single cell. Furthermore, the non-temporary computer-readable medium further includes a processor 202 that compares the seed mask 430 and the confluence mask 410 before a processor 202 that generates an image of one or more cells in a biological specimen 110 showing a cell-by-cell division mask 435, and removes one or more regions from the seed mask 430 that are not located in the area of ​​the confluence mask 410, and removes one or more regions from the confluence mask 410 that do not contain one of the multiple species of the seed mask 430.

[0048]

[0072] In yet another optional embodiment, a program instruction causing a processor 202 to generate an image of one or more cells in a biological specimen 110 showing a cell-by-cell splitting mask 435 based on a species mask 430 and a confluence mask 410 includes a processor 202 that performs region expansion iterations for each of the species' active sets. The non-temporary computer-readable medium then further includes a processor 202 that repeats the region expansion iterations for each of the species' active sets until the expanded region for a given species reaches one or more boundaries of the confluence mask 410 or overlaps with the expanded region of another species.

[0049]

[0073] The non-temporary computer-readable medium further includes a processor 202 that applies one or more filters in response to user input to remove objects based on one or more cell texture quantifiers and cell shape descriptors. Furthermore, the processor 202 modifies an image of a biological specimen 110 showing a cell-by-cell splitting mask 435 in response to the application of one or more filters.

[0050] IV. Experimental Results

[0074] An example of the implementation allows for tracking cell health in subpopulations over time. For example, Figure 6A shows experimental results of cell-by-cell segmentation masks generated by an example implementation for phase-contrast image responses 24 hours after HT1080 fibrosarcoma apoptosis following camptothecin (CPT, cytotoxic) treatment. Cell health was determined by multiplexed readouts of Incucyte® NucLight Red (nuclear viability marker) and non-perturbed Incucyte® caspase 3 / 7 green reagent (apoptosis indicator). Figure 6B shows cell subsets classified based on red and green fluorescence using the implementation in Figure 6A with the Incucyte® cell-by-cell analysis software tool. Figure 6C shows the decrease in the red population after CPT treatment indicating loss of viable cells, the increase in red and green fluorescence indicating early apoptosis, and the increase in green fluorescence after 24 hours indicating late apoptosis, all using the implementation in Figure 6A. Figure 6D shows the concentration response time course of the initial apoptotic population (percentage of total cells showing red and green fluorescence) based on the implementation configuration of Figure 6A. The values ​​shown are the ±SEM averages of the three depressions.

[0051]

[0075] In another example, Figure 6E shows experimental results of cell-by-cell segmentation masks generated by an implementation example for cell image response 24 hours after HT1080 fibrosarcoma apoptosis following cyclohexaamide (CHX, cell proliferation inhibitory). Cell health was determined by multiplexed readouts of Incucyte® NucLight Red (nuclear viability marker) and non-perturbed Incucyte® caspase 3 / 7 green reagent (apoptosis indicator). Figure 6F shows cell subsets classified based on red and green fluorescence using the implementation in Figure 6E with the Incucyte® cell-by-cell analysis software tool. Figure 6G shows the absence of apoptosis after CHX treatment, but with a decrease in cell number (data not shown), using the implementation in Figure 6E. Figure 6H shows the concentration response time course of the initial apoptotic population (percentage of total cells showing red and green fluorescence) using the implementation in Figure 6E. The values ​​shown are the ±SEM average of the three depressions.

[0052]

[0076] Figure 7A shows a cell-to-cell division mask pressed onto a phase-contrast image of unlabeled cell counts of adherent cells using cell-to-cell division analysis generated by an example implementation via Incucyte® software. Various densities of A549 cells labeled with NucLight Red reagent were analyzed using both unlabeled cell-to-cell analysis and red nucleus count analysis to confirm unlabeled counts over time. Figure 7B shows the cell-to-cell division mask from Figure 7A without a phase-contrast image in the background. Figure 7C shows the time course of phase count and red count data across densities using the implementation from Figure 7A. Figure 7D shows the correlation of count data over 48 hours using the implementation from Figure 7A, showing an R² value of 1 with a slope of 1. This is repeated across various cell types. The values ​​shown are the ±SEM average of the four depressions.

[0053] V. Additional Examples and Experimental Data

[0077] The functions described later can be performed, for example, by environment 100. Figure 4 is explained below. The optical microscope 105 captures a first bright-field image 415 of the biological specimen 110 at the first focal plane and a second bright-field image 420 of the biological specimen 110 at the second focal plane. Next, environment 100 generates a cell image 425 by performing mathematical operations on the pixels of the first bright-field image 415 and the second bright-field image 420.

[0054]

[0078] Figure 8 is a schematic diagram of three focal planes. The first focal plane 611 is at a defocusing distance 617 beyond the third focal plane 615, where the biological specimen is observable with improved focus relative to the first and second focal planes 611 and 613. The second focal plane 613 is at a defocusing distance 617 less than that of the third focal plane 615. In some examples, the defocusing distances are in the range of 20 μm to 60 μm.

[0055]

[0079] Figure 9 is a schematic diagram of the operation of the environment 100, which is executed by the processor 202 that executes instructions stored in the data storage device 206. For example, the processor 202 executes a computational model 601, which can take the form of, for example, an artificial neural network (ANN) or a convolutional neural network (CNN).

[0056]

[0080] An ANN or CNN contains artificial neurons called nodes. Each node can transmit data to other nodes. The receiving node can then process the data and send it to the connected nodes. The data typically contains numbers, and the output of each node is calculated by a (e.g., nonlinear) function of the sum of the inputs. The connections are called edges. Nodes and edges usually have weights that are adjusted as learning progresses. The weights increase or decrease the intensity and determine the direction of the data at the connections. Nodes may have thresholds so that they only send data if the aggregated data exceeds a threshold. Typically, nodes are aggregated into layers. Different layers can perform different transformations on the layer's inputs. Data progresses from the first layer (input layer) to the last layer (output layer), possibly traversing many layers many times.

[0057]

[0081] Environment 100 processes cell images 425 and phase-contrast images 400 of the biological specimen 110 using computational model 601 to generate output data 609. Cell image 425 is a composite of a first bright-field image 415 of the biological specimen 110 at a first focal plane 611 and a second bright-field image 420 of the biological specimen 110 at a second focal plane 613. For example, environment 100 processes cell images 425 and phase-contrast images 400 according to nodes, connections, and weights as defined by computational model 601.

[0058]

[0082] In some cases, the environment 100 processes the composite of the cell image 425 and the phase-contrast image 400 as two channels of superimposed image information, depending on the input to the computational model 601.

[0059]

[0083] In another example, environment 100 processes cell images 425 through a first channel (e.g., an input channel) of computational model 601 and processes phase-contrast images 400 through a second channel (e.g., a different input channel) of computational model 601. As such, in this example, environment 100 processes the first output of the first channel and the second output of the second channel to generate output data 609, or generates intermediate data used to obtain the output data 609.

[0060]

[0084] The output data 609 generally contains information about the biological specimen 110. For example, the output data 609 can estimate the location and / or extent (e.g., boundaries, regions, or volume) of one or more cells in the biological specimen 110. Other examples of the output data 609 are described below.

[0061]

[0085] Next, environment 100 performs a comparison of output data 609 and reference data 615. Reference data 615 is typically “truth data” representing information about a human-generated biological specimen 110. For example, output data 609 may include human-generated markings indicating the location and / or extent of cells in the biological specimen 110. In some examples, environment 100 generates computer-implemented transformations 619 of the human-generated data, which may also be included as part of the reference data 615. Examples of computer-implemented transformations include rotation, scaling, translation, and / or resolution changes. Thus, environment 100 can refine the computational model 601 in a self-managed or semi-managed manner.

[0062]

[0086] Next, environment 100 refines the computational model 601 based on a comparison of the output data 609 and the reference data 615. For example, environment 100 can calculate the pixel-by-pixel luminance and / or the color difference between the output data 609 and the reference data 615, and adjust the nodes, connections, and / or weights of the computational model 601 so that the output data 609 generated by the computational model 601 better matches the reference data 615.

[0063]

[0087] Subsequently, environment 100 processes additional image pairs according to computational model 601 in order to further refine computational model 601 based on a comparison of additional output data 609 generated by the computational model with additional reference data 615. The additional image pairs include cell images 425 and phase-contrast images 400 corresponding to other biological specimens 110 or other fields of view of the same biological specimen 110. The additional reference data 615 corresponds to the additional biological specimens 110 or other fields of view of the same biological specimen 110.

[0064]

[0088] More specifically, environment 100 can refine the computational model 601 (e.g., by adjusting the nodes, connections, and / or weights of the computational model 601) in order to reduce the sum of each difference between the additional output data 609 and the additional reference data 615. Thus, environment 100 can adjust the nodes, connections, and / or weights of the computational model 601 so that the collected output data 609 as a whole best matches the collected reference data 615.

[0065]

[0089] As shown in Figure 10, output data 609 can represent an estimate of the location and / or extent of cell 621 within the biological specimen 110. In this context, reference data 615 precisely defines the location and / or extent of cell 621. Although Figure 10 shows output data 609 and reference data 615 as identical, in practice, they are generally not identical.

[0066]

[0090] In some examples, output data 609 represents an estimate of the appearance of the biological specimen 110 if the biological specimen has a fluorescent label. In this context, reference data 615 is generated from actual images of the biological specimen 110 having a fluorescent label, or includes actual images of the biological specimen 110 having a fluorescent label.

[0067]

[0091] In some examples, output data 609 represents an estimate of the location and / or extent of one or more cell nuclei within the biological specimen 110. In this context, reference data 615 precisely specifies the location and / or extent of one or more cell nuclei within the biological specimen 110. Alternatively, reference data 615 represents processed fluorescence data corresponding to the biological specimen 110. Such fluorescence data can be processed to identify cell nuclei.

[0068]

[0092] Figure 11 is explained. Output data 609 can represent estimations of the classification of the first part 631 of the biological specimen 110 into a first category and the classification of the second part 633 of the biological specimen 110 into a second category (e.g., viable cells vs dead cells, stem cells vs lineage-specific cells, undifferentiated cells vs differentiated cells, epithelial cells vs mesenchymal cells, wild-type cells vs mutant cells, cells expressing a specific protein of interest vs cells not expressing that specific protein of interest, etc.). Other categories are also possible. In this regard, reference data 615 precisely defines the classification of the first part 631 of the biological specimen 110 into a first category and the classification of the second part 633 of the biological specimen 110 into a second category.

[0069]

[0093] In another example, output data 609 represents the classification of all biological specimens 110 into a single category. In this context, reference data 615 accurately classifies all biological specimens 110 into a single category of two or more categories (e.g., healthy vs. unhealthy, malignant vs. benign, wild-type vs. variant).

[0070]

[0094] Figures 12 and 13 are block diagrams of methods 501 and 701, respectively. Methods 501 and 701 and related functions can be performed, for example, by environment 100. As shown in Figures 12 and 13, methods 501 and 701 include one or more operations, functions, or actions as illustrated by blocks 503, 505, 507, 509, 702, 704, 706, 708, 710, and 712. Although the blocks are illustrated in a sequential order, these blocks may be performed in parallel and / or in an order different from that described herein. Furthermore, various blocks may be combined into fewer blocks, divided into further blocks, and / or removed based on a desired implementation.

[0071]

[0095] In block 503, method 501 includes processing a cell image 425 of a biological specimen 110 and a phase-contrast image 400 of a biological specimen 110 using a computational model 601 to generate output data 609. The cell image 425 is a composite of a first bright-field image 415 of the biological specimen 110 at a first focal plane 611 and a second bright-field image 420 of the biological specimen 110 at a second focal plane 613.

[0072]

[0096] In block 505, method 501 includes performing a comparison between output data 609 and reference data 615.

[0073]

[0097] In block 507, method 501 includes refining the calculation model 601 based on a comparison of output data 609 and reference data 615.

[0074]

[0098] In block 509, method 501 then includes processing additional image pairs according to computational model 601 and further refining computational model 601 based on a comparison of additional output data 609 generated by computational model 601 with additional reference data 615.

[0075]

[0099] In block 702, method 701 includes acquiring a first bright-field image 415 of the biological specimen 110 at a first focal plane 611 and a second bright-field image 420 of the biological specimen 110 at a second focal plane 613 via an optical microscope 105.

[0076] [000100] In block 704, method 701 includes generating a cell image 425 of a biological specimen 110 by performing mathematical operations on the pixels of a first bright-field image 415 and a second bright-field image 420.

[0077] [000101] In block 706, method 701 includes processing cell images 425 and phase contrast images 400 of the biological specimen 110 using a computational model 601 to generate output data 609.

[0078] [000102] In block 708, method 701 includes performing a comparison between output data 609 and reference data 615.

[0079] [000103] In block 710, method 701 includes refining the calculation model 601 based on a comparison of output data 609 and reference data 615.

[0080] [000104] In block 712, method 701 then includes processing additional image pairs according to computational model 601 and further refining computational model 601 based on a comparison of additional output data 609 generated by computational model 601 with additional reference data 615.

[0081] [000105] Figure 14 shows images related to generating a cell-by-cell split mask using the computational model. Including cell images 425 (e.g., information from the first bright-field image 415 and the second bright-field image 420) improves the performance of computational model 601. Computational model 601 is even better when clusters (e.g., rigid groups of cells) are disrupted. Models that only include phase-contrast images as input are likely to have false-positive cell recognition due to plate texture.

[0082] [000106] Figure 15 shows a comparison of the results of computational model 601 (e.g., phase + cell) and a model that includes only phase-contrast images as input (e.g., phase only). Box mAP metric and mask mAP metric are described. Computational model 601 yields a higher score than the "phase only" model, which shows an improvement in cell identification or classification by computational model 601. Computational model 601 is generally more robust than the other models because the addition of cell images provides more robust training information.

[0083] [000107] The Intermediate Mean Precision (mAP) score is used to compare the cell-to-cell split mask of the computational model with a manually annotated reference cell-to-cell split mask. Box mAP refers to the score calculated using the cell-to-cell bounding box, while mask mAP refers to the score calculated using the cell-to-cell mask.

[0084] [000108] Descriptions of different advantageous configurations are presented for illustrative and explanatory purposes and are not intended to be exhaustive or to be limited to the examples in the form of disclosure. Many modifications and changes will be apparent to those skilled in the art. Furthermore, different advantageous examples may describe different advantages compared to other advantageous examples. Selected examples or a selection of selected examples are described so that those skilled in the art can understand the disclosure to various examples that best illustrate the principles of the examples, their practical applications and have various modifications to suit specific conceivable uses.

Claims

1. A method for analyzing images of biological specimens using a computational model, The method involves generating a cellular image of a biological specimen by performing mathematical calculations on a pixel-by-pixel basis on a first bright-field image of the biological specimen at a first focal plane and a second bright-field image of the biological specimen at a second focal plane. To generate output data, the calculation model is used to process the cell image of the biological specimen and the phase-contrast image of the biological specimen, wherein the cell image is a composite of the first bright-field image and the second bright-field image. The output data and the reference data are compared, and the output data represents an estimation that the first part of the biological specimen is classified into the first category and the second part of the biological specimen is classified into the second category, and the reference data accurately defines that the first part of the biological specimen is classified into the first category and the second part of the biological specimen is classified into the second category. The calculation model is refined based on the comparison of the output data and the reference data. Subsequently, in order to further refine the calculation model based on a comparison between the additional output data generated by the calculation model and the additional reference data, additional image pairs are processed according to the calculation model. A method that includes this.

2. The method according to claim 1, wherein the first focal plane is at a defocus distance above a third focal plane on which the biological specimen is observable with improved focus relative to the first and second focal planes, and the second focal plane is at the defocus distance below the third focal plane.

3. The method according to claim 1 or 2, wherein the output data represents an estimate of the location and extent of cells within the biological specimen.

4. The method according to claim 3, wherein the reference data accurately defines the position and range of the cells.

5. The method according to any one of claims 1 to 4, wherein the output data represents an estimate of the appearance of the biological specimen if the biological specimen has a fluorescent label.

6. The method according to claim 5, wherein the reference data is generated from actual images of the biological specimen having a fluorescent label.

7. The method according to any one of claims 1 to 6, wherein the output data represents an estimate of the location and extent of cell nuclei within the biological specimen.

8. The method according to claim 7, wherein the reference data accurately defines the position and range of the cell nuclei within the biological specimen.

9. The method according to any one of claims 1 to 8, wherein the output data represents an estimation of the classification of the first portion of the biological specimen into the first category and the classification of the second portion of the biological specimen into the second category.

10. The method according to claim 9, wherein the reference data accurately defines the classification of the first portion of the biological specimen into the first category and the classification of the second portion of the biological specimen into the second category.

11. The method according to any one of claims 1 to 10, wherein processing the additional image pairs according to the calculation model in order to further refine the calculation model includes refining the calculation model in order to reduce the sum of each difference between the additional output data and the additional reference data.

12. The method according to any one of claims 1 to 11, wherein processing the cell images of the biological specimen and the phase-contrast images of the biological specimen using the computational model includes processing the cell images of the biological specimen and the phase-contrast images of the biological specimen using an artificial neural network.

13. The method according to any one of claims 1 to 12, wherein processing the cell images of the biological specimen and the phase-contrast images of the biological specimen using the computational model includes processing the cell images of the biological specimen and the phase-contrast images of the biological specimen using a convolutional neural network.

14. The method according to any one of claims 1 to 13, wherein processing the cell image of the biological specimen and the phase contrast image of the biological specimen using the computational model includes processing a composite of the cell image and the phase contrast image.

15. A non-temporary data storage device that, when executed by a computing device, stores instructions causing the computing device to perform the method according to any one of claims 1 to 14.

16. A system for testing biological specimens, Optical microscope and, One or more processors, When executed by one or more of the aforementioned processors, a non-temporary data storage device that stores instructions causing the system to execute the method according to any one of claims 1 to 14, and A system that includes this.

17. A system for testing biological specimens, Optical microscope and, One or more processors, When executed by one or more of the aforementioned processors, The optical microscope captures a first bright-field image of the biological specimen at a first focal plane and a second bright-field image of the biological specimen at a second focal plane. The cell image of the biological specimen is generated by performing mathematical operations on a pixel-by-pixel basis on the first bright-field image and the second bright-field image. To generate output data, a computational model is used to process the cell images and phase-contrast images of the biological specimen, The output data and the reference data are compared, and the output data represents an estimation that the first part of the biological specimen is classified into the first category and the second part of the biological specimen is classified into the second category, and the reference data accurately defines that the first part of the biological specimen is classified into the first category and the second part of the biological specimen is classified into the second category. The calculation model is refined based on the comparison of the output data and the reference data. Subsequently, in order to further refine the calculation model based on a comparison between the additional output data generated by the calculation model and the additional reference data, additional image pairs are processed according to the calculation model. A non-temporary data storage device that stores instructions for causing the system to execute a function including the above, A system that includes this.

Images