Distance-based determination of tissue status

Abstract

The present invention relates to an image analysis method for determining the biomedical state of a tissue sample. The method comprises: - receiving (202) a digital image (118) of a tissue sample; - identifying (204) the number and location of type A cells (504) and type B cells (506); - obtaining (206) an observed relative distribution (708, 706); - obtaining (208) a reference relative distribution (710, 714) of an expected distance between a reference type A cell and a reference type B cell; - calculating (210) a proximity score (718, 720) as a difference between the reference relative distribution and the observed relative distribution; - calculating (212) a combined score (912), the combined score comprising the proximity score and the density of the type A cells and / or the density of the type B cells; - using (214) the combined score to determine the biomedical state of the tissue sample and / or outputting (216) the combined score.

Description

Technical Field

[0001] The present invention relates to the field of image analysis, and in particular to the field of image-based classification of biomedical states of tissue samples. Background Art

[0002] Today, a variety of different image analysis methods are used to automatically determine biomedical features of tissue samples. For example, image analysis is used to automatically identify cells, to segment digital pathology images into background and tissue regions, and to automatically identify and / or quantify biomedical conditions, such as specific diseases.

[0003] One example is the detection of biomarkers, such as tumor markers, for example in the diagnosis of leukemia, breast cancer, colon cancer and other types of cancer. It involves labeling cells with antibodies against proteins that are specifically expressed or overexpressed in a particular type of cell, such as tumor cells or other types of cells. By choosing the right antibodies, different types of cells can be accurately determined.

[0004] Immunophenotyping is a technique commonly used in basic science research and laboratory diagnostics. It uses antibodies against specific target molecules to identify the "phenotype" of cells based on the expression levels of one or more proteins. The cellular phenotype of one or more cells in a tissue can also be used to gain insights into the biomedical condition of the tissue, such as whether the tissue is healthy tissue, primary tumor tissue, or metastatic tissue.

[0005] A common problem associated with many of the image analysis algorithms currently used to analyze digital pathology images is that the expression levels of only certain proteins may allow for the correct identification of cells of a specific cell type but may not be sufficient to accurately, reproducibly, and automatically determine the biomedical status of the tissue.

[0006] Steele KE et al., "Measuring multiple parameters of CD8+ tumor-infiltrating lymphocytes in human cancers by image analysis" (J Immunother Cancer. 2018 Mar 6;6(1):20. doi:10.1186 / s40425-018-0326-x) characterized the immune response to cancer based on the number and location of CD8+ tumor-infiltrating lymphocytes (TILs). The strength of the immune response is considered to be a parameter with high prognostic, pharmacodynamic, and predictive potential.

[0007] However, the ability of known methods to provide precise quantitative data while also resolving spatial aspects of cellular distribution remains limited. Furthermore, the subjectivity and diversity of methods used to account for spatial correlations performed by multiple laboratories hinders the ability to compare histopathological results. Summary of the invention

[0008] An object of the present invention is to provide an improved method and image analysis system for determining the biomedical status of a tissue sample according to the present invention. This specification gives embodiments of the present invention. If the embodiments of the present invention are not mutually exclusive, they can be freely combined with each other.

[0009] In one aspect, the present invention relates to an image analysis method for determining the biomedical state of a tissue sample. The method is implemented in an image analysis system. The method comprises:

[0010] - receiving a digital image of a tissue sample;

[0011] - analyzing the received image to identify the number and location of type A cells and type B cells observed in the area of ​​the received image, wherein the type A cells and the type B cells are different cell types;

[0012] - analyzing the positions of the type A and type B cells in the region to obtain an observed relative distribution indicating an observed distance between the type A cells and the type B cells in the region;

[0013] - obtaining a reference relative distribution indicating an expected distance between a reference A-type cell and a reference B-type cell;

[0014] - calculating a closeness score as the difference between the reference relative distribution and the observed relative distribution;

[0015] - calculating a combined score comprising the proximity score and comprising the density of the type A cells and / or the density of the type B cells;

[0016] - using the combined score to determine the biomedical status of the tissue sample and / or outputting the combined score to enable a user to determine the biomedical status of the tissue sample.

[0017] Described feature may be advantageous, because contrary to prior art methods, the spatial distribution of different cell types is not determined subjectively by pathologists.On the contrary, observation relative distribution is compared with reference to relative distribution, i.e., compared with the distribution of the expected distance between reference A-type cells and reference B-type cells in the region representing. Expected distance can be, for example, assuming that A-type cells and / or B-type cells are distributed and expected distance according to specific predefined mathematical distribution (such as Poisson distribution). Alternatively, it is possible to empirically determine expected distribution based on the analysis of A-type and B-type cells in one or more other tissue samples with known biomedical status. By defining reference to relative distribution and by calculating the proximity score as the difference of reference to relative distribution and observation relative distribution, a reproducible measurement of the relative spatial position of two different cell types in tissue sample is provided. The proximity score can provide a reproducible quantitative measurement of the deviation of the relative distribution / distance of observed A-type and B-type cells and the expected relative distribution / distance of A-type and B-type cells.

[0018] The proximity score can be considered to represent the "delta" between the observed relative distribution and the reference relative distribution. The proximity score integrates the spatial difference in the A-type cell-B-type cell distance represented in the observed relative distribution and the reference relative distribution. Preferably, the proximity score is a single digital data value that integrates the distance information obtained from multiple A-type and B-type cells. For example, the "delta" can be calculated as a radius difference as shown in Figure 7 or an area difference as shown in Figure 8.

[0019] In another beneficial aspect, the proximity score can be a highly integrated data value because the proximity score value can be a single data value obtained by processing multiple observed and expected distances of type A cells and type B cells. Therefore, information encoded in the spatial distribution of hundreds and potentially thousands of cells can now be considered and used to reproducibly determine the biomedical state of a particular tissue. The information encoded in the spatial locations of cells is complex, and their interpretation is also complex. By providing a single score value calculated as a function of observed and expected A cell-B cell distances, a parameter value is provided that encodes a large amount of biomedical information and therefore may have high predictive relevance for various prediction tasks (e.g., determination of the biomedical state of a tissue or tissue region).

[0020] In another beneficial aspect, the proximity score is combined with additional information (i.e., the density of type A and / or type B cells in the image region) to provide a final combined score. The combined score may be advantageous because it contains both "relative" information (i.e., the deviation of the observed A cell-B cell distance from the expected A cell-B cell distance) and "absolute" information, i.e., the density of type A cells or type B cells in the examined image region. The combination of absolute and relative information may be beneficial because it may also address differences in biomedical states of tissue samples that cannot be identified based solely on the proximity score or based solely on the density of one of the cell types. For example, there may be several different biomedical tissue states that have the same proximity score, but can be distinguished based on differences in the absolute density of type B or type A cells. Additionally, or alternatively, there may be several different biomedical tissue states that have the same density of type B cells (or type A cells), but can be distinguished based on differences in proximity scores.

[0021] Embodiments of the present invention can allow the assumption that a particular biomedical state (e.g., a particular tumor type and stage) drives the observed cell distribution away from an expected distribution (e.g., a random distribution such as a Poisson distribution), and allows the assumption to be easily tested by calculating a proximity score. If the proximity score is zero or close to zero, there is substantially no difference between the observed relative distribution and the expected reference relative distribution, and the assumption can be accepted. For example, it is assumed that in the case where the tissue sample is a tumor tissue sample taken from a tumor of a particular type and stage, the tumor microenvironment drives the observed relative distribution of tumor cells (type A cells) and immune cells (type B cells) away from a random reference distribution.

[0022] Thus, the combined score can provide a highly integrated parameter with high predictive power and can allow reproducible characterization of spatial information related to the relative distribution of type A and type B cells in a tissue. The proximity score and the combined score can have the further advantage that they can be used to characterize the relative distribution of any two different cell types and can therefore be used to automatically determine biomedical states related to many different biomedical questions. Thus, a highly reproducible, non-subjective but universal score value is provided that can be used to automatically, reproducibly and accurately determine the biomedical state of a tissue sample.

[0023] Embodiments of the present invention may allow for avoiding inconsistencies when comparing results of IHC biomarker analyses performed by multiple laboratories and allow for better coordination between laboratories.

[0024] Furthermore, it has been observed that the combined score is able to capture more complex histopathological information for determining biomedical tissue status compared to state-of-the-art methods based on image analysis.

[0025] According to an embodiment, cells of one of the two cell types having a higher cell density in the image region are referred to as type A cells, and cells of the other cell type are referred to as type B cells.

[0026] According to a preferred embodiment, the distance between type A cells and type B cells is measured starting from type A cells, and the density used to calculate the combined score is the density of type B cells in the image area.

[0027] According to an embodiment, the density of the reference A-type cells is the same or similar to the density of the A-type cells observed in the image region. For example, the reference A-type cells can be simulated in the same image region where the A-type cells are observed, whereby the number of the simulated reference A-type cells is the same or similar to the number of the A-type cells observed in the image region (e.g., the same + / - 10%). In addition, the density of the reference B-type cells is the same or similar to the density of the B-type cells observed in the image region. For example, the reference B-type cells can be simulated in the same image region where the B-type cells are observed, whereby the number of the simulated reference B-type cells is the same or similar to the number of the B-type cells observed in the image region (e.g., the same + / - 10%).

[0028] Alternatively, the reference type A cells and the reference type B cells are cells that have been observed in the reference tissue sample. The reference tissue sample is selected so that the number and density of type A cells and type B cells in the reference tissue sample can be expected to be similar to the density of type A cells and type B cells in the tissue sample depicted in the received image. This can ensure that the difference in the observed relative distribution and the reference relative distribution is caused by factors other than cell density.

[0029] According to an embodiment, obtaining the observed relative distribution includes, for each identified type A cell (e.g., tumor cell) observed in the image region:

[0030] a) selecting the type A cell as the center of a circle with a radius of 0;

[0031] b) increasing the radius by one step to produce an enlarged circle;

[0032] c) determining the number of type B cells contained in the circle generated in step b);

[0033] d) storing the current radius of the circle associated with the number of type B cells determined in step b) in a storage medium, and repeating steps b), c) and d) until a termination criterion is reached; and

[0034] e) selecting one of the unselected type A cells and continuing step a) using the newly selected type A cells until all type A cells are selected.

[0035] The associated radius and number of observed type B cells obtained for each identified type A cell and radius were stored as the observed relative distribution.

[0036] For example, the termination criterion may be a maximum radius, a maximum number of steps, reaching a maximum execution time, or a maximum number of repetitions. The feature may be advantageous because a stepwise increase in radius may allow a wide range of intercellular distances to be covered with a relatively low computational effort. For example, the stepping may be selected such that the radius increases in size by 1–100 μm in each step, e.g., 10 μm. The radius may be increased, e.g., until a predetermined maximum radius is reached, e.g., about 500 μm or about 200 μm. By selecting the size, and thereby the number of steps and cell counting operations, a discrete data set may be obtained that reflects the observed relative spatial distribution of type A and type B cells.

[0037] According to an embodiment, the observed relative distribution is calculated based on the binary Ripley K(t) function of the type A cells and type B cells observed in the image region.

[0038] For example, Philip M. Dixon describes the Ripley K function in Encyclopedia of Environmetrics (ISBN 0471 899976), edited by Abdel H. El-Shaarawi and Walter W. Pigorsch, John Wiley & Sons, Ltd, Chichester, 2002, Vol. 3, pp. 1796-1803. In the document, the Ripley K(t) function is generalized to more than one type of point (multivariate space point process) in equation (11) according to the following formula:

[0039]

[0040] This can be advantageous because the Ripley K(t) function allows the relative spatial distribution of two or more point patterns in two-dimensional space to be recorded and simulated. It is relatively fast and is an integral part of some computational tools, such as mathematical software like "R".

[0041] According to an embodiment, the observed relative distribution is calculated as K according to the following formula i，j (r):

[0042]

[0043] - K i，j (r) is the binary Ripley K(t) function,

[0044] - where i is the occurrence of object type "observed type A cells";

[0045] - where j is the occurrence of object type "observed B-type cells";

[0046] - where λ j is the density of observed B-type cells (number per image area);

[0047] -Where X (i) is the total number of observed type A cells identified within the image area;

[0048] -Where X j is the total number of observed B-type cells identified within the image area;

[0049] - where u∈X (i) Indicates that the cell (also referred to as the "currently examined cell") is an observed type A cell;

[0050] -where r is the radius of the step increase centered at the observed A-type cell;

[0051] - where t is u, r and X j A function of , and counting the number of type B cells observed within a circle with a radius of r surrounding the observed type A cell u in the image region;

[0052] - where "|u∈X (i) ” means “on all observed A-type cells u”;

[0053] -where E is the expected value of t obtained over all u.

[0054] In general, the "expected value" E of t obtained over all u describes the expected value of a variable (in this case t). For example, the expected value E can be calculated as an average, such as the probability-weighted average of all possible values ​​of t obtained over all u. Intuitively, E is a bit like an "average outcome". For example, E of t obtained over all u can be calculated as the average t obtained over all u.

[0055] As used herein, an "average" is a single number that represents a series of numbers. For example, the "average" may be the arithmetic mean, which is the sum of the numbers divided by the number of numbers being averaged. Alternatively, the average may be the median or the mode. The average is a measure of the central tendency of a group of numbers.

[0056] According to an embodiment, the observed relative distribution is calculated based on the derivative of the binary Ripley K(t) function for the type A cells and type B cells observed in the image region.

[0057] According to one example, the derivative binary Ripley K(t) function is a linear or nonlinear transformation of the binary Ripley K(t) function. In particular, the derivative binary Ripley K(t) function can be a linear transformation of the binary Ripley K(t) function. The linear transformation can be, for example, K calculated according to the following formula: i，j Divide by the square root of π:

[0058]

[0059] L i，j (r) is also called "L function" or "Besag L function". When the process is a Poisson process, the linear transformation approximates the variance of the stable estimate K and also i,j (r) = πr 2 Transform to L i,j (r) = r.

[0060] According to an embodiment, the reference relative distribution is calculated based on a binary Ripley K(t) function of simulated type A cells and simulated type B cells in the image region.

[0061] According to an embodiment, said obtaining of said reference relative distribution comprises:

[0062] - computationally simulating the distribution of the simulated reference A-type cells, the density of the simulated reference A-type cells being the same as the density of the A-type cells observed in the image region;

[0063] - computationally simulating a distribution of simulated reference B-type cells in the region, the density of the simulated reference B-type cells being the same as the density of B-type cells observed in the image region;

[0064] - calculating the reference relative distribution as a function of the computationally simulated distributions of simulated reference type A cells and simulated reference type B cells, the reference relative distribution being indicative of the distance between the simulated reference type A cells and the simulated reference type B cells in the region.

[0065] The computational generation of simulated reference A-type cells and simulated reference B-type cells in an image region (a "simulation" of A-type and B-type cells and their distribution based on a predefined expected distribution) can have the following advantages, namely, no additional (usually scarce) tissue is required as a reference, and the distribution of the reference A-type and reference B-type cells is guaranteed to accurately represent the expected distribution as long as the number of simulated reference A-type and reference B-type cells is high enough. Therefore, the simulation of a "virtual" tissue image region including an expected (e.g., Poisson) distribution of simulated reference A-type and reference B-type cells is computationally economical and does not require the collection and staining of one or more reference tissue samples to provide an empirical reference basis and corresponding reference images. In addition, the simulation of reference A-type and B-type cells is very flexible and allows arbitrary modification of the simulated distribution. This may allow "fine-tuning" of biomedical assumptions and may allow the precise definition of complex binary co-distribution patterns that do not follow a Poisson distribution. For example, a complex co-distribution pattern may correspond to a high proportion of reference A-type cells having reference B-type neighboring cells in exactly two concentric "bands" surrounding the reference A-type cells. While such complex patterns may be difficult to observe in nature, simulations can allow for the generation of “atypical,” complex distributions that may be difficult to observe in their “ideal” form in real tissue samples, but which can nonetheless help formulate and examine biological hypotheses related to the spatial distribution of cells in tissue samples.

[0066] According to an embodiment, the reference relative distribution is derived from simulated distributions of reference type A and reference type B cells, whereby the simulated reference type A cell distribution and / or the simulated reference type B cell distribution may be, for example, a Poisson distribution.

[0067] According to a preferred embodiment, the distribution of the simulated reference type A cells is a Poisson distribution, and the distribution of the simulated reference type B cells is a Poisson distribution.

[0068] The Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed time or space interval if those events occur at a known constant rate and are independent of the time since the last event. The Poisson distribution can also be used for the number of events in other specified intervals, such as distance, area, or volume. If the occurrence of any particular type of event does not affect the temporal probability of future events, that is, if the observed events occur independently of each other, then it is reasonable to assume that the number of events occurring in a day follows a Poisson distribution. Therefore, by generating a set of Poisson distributions for simulated type A cells in an image region and generating a set of Poisson distributions for simulated type B cells in an image region, and by determining a reference relative distribution of simulated reference type A cells and reference type B cells, the hypothesis that the distribution of type A cells is completely independent of the distribution of reference type B cells and vice versa can be simulated and tested for the tissue sample currently being examined.

[0069] According to an embodiment, the density of the simulated reference type A cells in the region is the same as or highly similar to the density of the type A cells observed in the image region. The term "highly similar" herein means that the number of simulated reference type A cells in the image region is equal to + / - 10% of the number of type A cells observed in the image region. The number of simulated reference type B cells in the region is the same as or highly similar to the number of type B cells observed in the image region.

[0070] This may be advantageous because it ensures that the total number and density of simulated reference type A and type B cells in the image region are the same or highly similar to the corresponding number and density of observed cells. Therefore, any deviation of the observed relative distribution from the reference relative distribution is not caused by any deviation in the number or density of simulated cells, but only by the difference in the relative spatial distribution of cells of the two cell types. Therefore, although the reference relative distribution can be based on simulated reference type A and type B cells, the simulation is strictly limited to the density of type A and type B cells observed in the tissue sample depicted in the image region. This can allow tissue sample specific simulations to be performed that allow any deviations in the relative spatial distribution of cells of the two cell types in the tissue sample to be accurately determined compared to the expected simulated distribution. The difference in spatial distribution from the simulation-based reference distribution is not affected by any other interfering factors, which may be relevant when using a reference tissue sample to empirically create a reference relative distribution.

[0071] According to an embodiment, said calculation of said reference relative distribution as a function of said computationally simulated distribution of reference type A and reference type B cells comprises:

[0072] For each randomly distributed simulated reference type A cell (e.g., tumor cell), perform the following steps:

[0073] a) selecting the simulated reference A-type cell as the center of a circle with a radius of 0;

[0074] b) increasing the radius by one step to produce an enlarged circle;

[0075] c) determining the number of simulated reference B-type cells (e.g., immune cells) contained in the circle generated in step b);

[0076] d) storing the current radius of the circle associated with the number of simulated reference B-type cells determined in step b) in a storage medium, and repeating steps b), c) and d) until a termination criterion is met; and

[0077] e) selecting one of the unselected simulated reference A-type cells, and continuing step a) using the newly selected simulated reference A-type cell until all simulated reference A-type cells are selected.

[0078] The method further includes providing an associated radius and number of simulated reference B-type cells as the reference relative distribution.

[0079] This stepwise calculation of the reference B-type cell count within a stepwise increasing circle around each reference A-type cell may take advantage of the similar determination of the radius and number of B-type cells already described for storage during the calculation of the observed relative distribution.

[0080] According to an embodiment, the reference relative distribution is calculated as K according to the following formula i，j (r):

[0081]

[0082] - K i，j (r) is the binary Ripley K(t) function,

[0083] - where i is the occurrence of object type "Simulated Reference A-Type Cell";

[0084] - where j is the occurrence of object type "Simulated Reference B-cell";

[0085] - where λ j is the density of simulated reference B-type cells (number per image area);

[0086] -Where X (i) is the total number of simulated reference A-type cells in the image region;

[0087] -Where X j is the total number of simulated reference B-type cells in the image region;

[0088] - where u∈X (i) Indicates that the cell is a simulated reference type A cell;

[0089] -where r is the radius of the step increase centered on the simulated reference A-type cell;

[0090] - where t is u, r and X j A function of , and counting the number of simulated reference B-type cells in a circle with a radius of r surrounding the simulated reference A-type cell u in the image region;

[0091] - where "|u∈X (i) ” means “on all simulated reference A-type cells u”;

[0092] -where E is the expected value of t for all u.

[0093] As defined above, the "expected value" E of t obtained over all u describes the expected value of a variable (in this case t). For example, the expected value E can be calculated as an average, such as a probability-weighted average of all possible values ​​of t obtained over all u. For example, E of t obtained over all u can be calculated as the average t obtained over all u.

[0094] According to an embodiment, the reference relative distribution is calculated based on the derivative of the binary Ripley K(t) function of the simulated type A cells and the simulated type B cells. According to one example, the derivative binary Ripley K(t) function is a linear or nonlinear transformation of the binary Ripley K(t) function. In particular, the derivative binary Ripley K(t) function can be a linear transformation of the binary Ripley K(t) function. For example, the linear transformation can be an "L function" or "Besag's L function", which is calculated as K according to the following formula: i，j Divide by the square root of π:

[0095]

[0096] According to an embodiment, the reference relative distribution and / or the observed relative distribution is calculated as a version of the Ripley K(t) function as described in equation (12) in the above-mentioned publication by Philip M. Dixon according to the following formula:

[0097]

[0098] Where w(i k , j l ) is a circle located in the study area with its center at the kth position of process I and a radius of d ik,jl part of the circumference of a circle.

[0099] According to an embodiment, the obtaining of the reference relative distribution comprises calculating a plurality of initial reference relative distributions according to any one of the embodiments and examples of the present invention described herein. Then, the method comprises calculating an average reference relative distribution from the plurality of initial reference relative distributions and using the average reference relative distribution as the reference relative distribution.

[0100] Preferably, the proximity score is calculated as the difference ("delta") between the average reference relative distribution and the observed relative distribution. This includes the difference between the parameter values ​​associated with the average reference relative distribution and the observed relative distribution. For example, the difference may be the radius difference associated with the two distributions shown in FIG7 , or the area difference shown in FIG8 .

[0101] For example, n initial reference relative distributions may be calculated, where n is a number preferably greater than 10, more preferably greater than 30, such as 39. Then, the "average relative distribution" is calculated by calculating, for each of a plurality of radii used to count B-type cells in a type A cell spatial neighborhood defined by a circle with the corresponding radius, the average number of B-type cells observed in a circle of the corresponding radius. For example, in the case where n=5, and the four radii examined will be 10 μm, 30 μm, 60 μm and 100 μm, then a first average number of B cells for a radius of 10 μm will be calculated based on the 5 values ​​obtained from the corresponding n initial relative distributions, a second average number of B cells for a radius of 30 μm will be calculated based on the 5 values ​​obtained from the corresponding n initial relative distributions, and so on, until an average B-type cell count is calculated for each radius examined.

[0102] According to some embodiments, for each radius (or each of a plurality of other predetermined distances defining the spatial neighborhood of type A cells used to calculate the relative distribution), not only is the average reference type B cell count calculated (or the reference type A cell count if the number of type A cells in the spatial neighborhood of the reference type B cells is examined), but also the minimum and maximum reference type B cell counts in any one of the initial reference relative distributions and any one of the plurality of radii / distances are examined. The average reference relative distribution is used to calculate the proximity score, and the minimum / maximum cell counts are used to calculate confidence bands, where the minimum / maximum values ​​represent the boundaries of the confidence bands, and where the width of the bands shows the "significant region" in the plot. If the observed relative distribution is within the confidence band, then the observed relative distribution is considered to be the same as, or at least not significantly different from, the reference relative distribution.

[0103] For example, an initial reference relative distribution of n=39 can be calculated based on simulations of Poisson distribution reference A-type and B-type cells, respectively, whereby the density of the simulated reference A-type cells is equal to the density of the "real" A-type cells observed in the image region of the tissue sample, and where the cell density of the simulated reference B-type cells is equal to the density of the "real" B-type cells observed in the image region. 39 simulations are performed for A-type and B-type cells of a given density to obtain a "p-value" for the combined observed relative distribution 1 / (39+1)=0.025=α / 2, where α is a predetermined significance level. Then, a 2D plot is created, where one dimension represents the cell distance and one dimension represents the cell count. Then, the average reference relative distribution obtained by averaging the 39 initial reference distributions is plotted in the 2D plot. In addition, a "confidence band" is plotted, where the lower boundary of the band is calculated as the minimum cell count observed for a specific distance (e.g., radius) in all 39 initial reference relative distributions, and where the upper boundary of the band is calculated as the maximum cell count observed for a specific distance (e.g., radius) in all 39 initial reference relative distributions. Thus, for each distance / radius r around cells of a particular cell type (e.g., type A cells), the minimum, maximum, and average observed cell counts of the corresponding other cell types (e.g., type B cells) derived from all 39 simulations are plotted. In addition, the observed relative distribution is plotted in a 2D plot. This may be beneficial because it allows a human user to intuitively assess whether the observed relative distribution is inside or outside the confidence band around the average reference relative distribution. If the observed reference relative distribution is outside the confidence band, the observed reference relative distribution is different from the average reference relative distribution at a predefined significance level α, whereby α may be, for example, 0.05 (with 0.025 on both sides).

[0104] The 2D drawing mentioned above may also be referred to as a "saliency drawing". Figure 7 provides an example of a saliency drawing. The saliency drawing may include a first dimension and a second dimension, the first dimension representing the radius of a circle drawn around each tumor cell and its nearest immune cell in the image area, and the second dimension representing the number of immune cells located within the circle, which is normalized by the density of immune cells and tumor cells. The drawing may include a confidence band around a reference relative distribution, which indicates the confidence interval of the number of immune cells observed for each radius value according to the drawn reference relative distribution. According to a preferred embodiment, the confidence band represents a 95% confidence band from 39 simulations (it can be shown that these 39 simulations for a specific type of biomedical state determination task correspond to 95% confidence). The observed relative distribution is also plotted in the saliency drawing. The observed relative distribution can be graphically represented as a curve that intuitively indicates the number of identified immune cells that have been observed in the corresponding circle around the observed tumor cell, and the number is normalized by the density of immune cells and tumor cells.

[0105] According to other embodiments, the reference relative distribution is not obtained by simulating a specific cell distribution, but is empirically obtained from one or more other tissue samples of known biomedical states and corresponding digital pathology images of said samples.

[0106] According to other embodiments, said obtaining of said reference relative distribution comprises:

[0107] - receiving a further digital image of each of the one or more further tissue samples, each further tissue sample originating from tissue of a known biomedical state; the further digital images may also be referred to as "reference images" and the further tissue samples may be referred to as "reference samples";

[0108] - analyzing each received further image to identify the number and location of reference type A cells and observed reference type B cells in a region of said received further image;

[0109] - analyzing said positions of observed reference type A cells and observed reference type B cells in said region of each further image to obtain an observed reference relative distribution, said observed reference relative distribution indicating an observed distance between said reference type A cells and said reference type B cells observed in the region of said further received image; and

[0110] - using said observed reference relative distribution as said reference relative distribution.

[0111] Therefore, the generation of the reference relative distribution described in the above-mentioned embodiments is not based on computational simulation, but on an evaluation of the relative spatial distribution of reference type A and reference type B cells in one or more other tissue samples with known biomedical states. This may be advantageous in the absence of suitable simulation software, or in the case where a mathematical function for generating a specific relative distribution representing a known biomedical state has not yet been defined and implemented in the software. For example, if a biomedical state, such as the cancer type and stage or immune cell infiltration level of a specific tumor tissue, has been clearly identified by one or more human pathologists, it may be more convenient to provide a reference relative distribution using empirical information implicitly contained in such further tissue samples than to perform computational simulation. For example, in the case where the relative spatial distribution of two different cell types in a tissue sample with a known biomedical state does not follow a Poisson distribution, it may be easier to simply empirically determine the relative distribution of cells of the two cell types in order to obtain a reference relative distribution, which represents a "hypothesis" that a tissue sample with a similar relative distribution also has the biomedical state of one or more other tissue samples, from which an empirically determined reference relative distribution can be derived.

[0112] The empirically determined reference relative distribution may be obtained, for example, from a tissue sample of the same "currently examined" organism as the tissue sample from which the observed relative distribution was obtained using the image analysis method. For example, the "other tissue sample" may be a tissue sample adjacent to the tissue sample depicted in the digital image from which the observed relative distribution was obtained. Alternatively, the other tissue sample may be from another organism different from the "currently examined" organism, whereby the other organism is preferably from the same species or a close species as the "currently examined" organism. Only the biomedical state of the other organism needs to be known, as this ensures that the reference relative distribution obtained from the other organism represents a specific, known biomedical state.

[0113] It is also possible to empirically determine an initial reference relative distribution from each of a plurality of other tissue samples having the same known biomedical state. Then, by calculating the average of each empirically determined initial reference relative distribution, an average reference relative distribution is obtained. According to some embodiments, obtaining the average reference relative distribution includes performing curve fitting.

[0114] The reference relative distribution or initial reference relative distribution may be obtained from one or more other tissue samples by image analysis in the same manner as the observed relative distribution is obtained.

[0115] According to one embodiment, a pathologist can manually annotate the biomedical status of a reference tissue sample to an image of the reference tissue sample of the patient. A reference relative distribution is obtained based on image analysis of the image of the reference tissue sample. A plurality of tissue samples adjacent to the reference tissue sample are obtained from the patient. From each of the plurality of adjacent other tissue samples, a corresponding digital image can be acquired, which is automatically analyzed to calculate the observed relative distribution. By comparing the observed relative distribution of each adjacent tissue sample with the reference relative distribution of the reference tissue sample, it can be determined whether the patient tissue from which the plurality of tissue samples including the reference sample are derived shows a homogeneous or heterogeneous spatial relationship and distribution of type A and type B cells.

[0116] According to an embodiment, the calculation of a closeness score as a difference between the reference relative distribution and the observed relative distribution comprises:

[0117] - providing a predetermined number (e.g., "1") representing a predetermined minimum number of type B cells;

[0118] - Identify the observation radius r within the observed relative distribution o-min , where the observation radius r o-min is a radius that, if drawn around each of said observed type A cells, would define a circle that, on average, includes a predetermined number of said observed type B cells;

[0119] - identifying a reference radius r within the reference relative distribution r-min , where the reference radius r r-min is a radius that, if drawn around each type A cell simulated or observed in a further digital image of a further tissue sample used to provide said reference relative distribution, would define a circle that, on average, includes a predetermined number of type B cells simulated or observed in a further digital image of a further tissue sample used to provide said reference relative distribution;

[0120] - Calculated as the observation radius r o-min and the reference radius r r-min Preferably, the function is the difference between the observation radius and the reference radius.

[0121] For example, the predefined number representing the predefined minimum number of type B cells may be any number greater than 0. Preferably, the predefined minimum number is 1.

[0122] The feature may be advantageous because the complexity of the information contained in the observed relative distribution and the reference relative distribution is reduced in a first step to two specific distance metrics, i.e. two specific radii, and then in another step to a proximity score value. In the simplest case, the proximity score is calculated as the difference between the two radii. However, the calculation of the proximity score may also be more complex and include, for example, additional normalization steps, steps for performing error correction, steps for multiplying the difference between the two radii by an error correction factor, etc. The proximity score may be a numerical value.

[0123] According to an alternative embodiment, said calculation of a closeness score as a difference between said reference relative distribution and said observed relative distribution comprises:

[0124] - defining a first distance threshold (t1); for example, the first difference threshold may have a value between 10 and 20 μm, such as 15 μm;

[0125] - defining a second distance threshold (t2); for example, the second difference threshold may have a value between 30 and 40 μm, for example 35 μm;

[0126] - plotting the reference relative distribution and the observed relative distribution, respectively, in a plot in the form of a curve, wherein a first dimension of the plot represents a radius r used to determine the observed relative distribution and the reference relative distribution, and wherein a second dimension of the plot represents the number of type B cells that are closer to type A cells than r (or vice versa);

[0127] - identifying a first region in the plot, the first region being defined by the first threshold and the second threshold, a baseline corresponding to a second dimension value of zero, and an observed relative distribution curve segment between the first threshold and the second threshold;

[0128] - identifying a second region in the plot, the second region being defined by the first threshold and the second threshold, the baseline corresponding to a second dimension value of zero, and a reference relative distribution curve segment between the first threshold and the second threshold;

[0129] - calculating a proximity score as a function of the first region and the second region, the function being in particular a difference between the first region and the second region.

[0130] In some embodiments, the proximity score is normalized within a predefined scale range. This can allow the proximity score obtained for the currently examined tissue sample to be mapped on a predefined proximity score scale. The scale with the score values ​​can be displayed to the user, for example, via a display of the image analysis system, and can allow the user to quickly and intuitively assess how much the relative distribution of type A and type B cells in the currently examined tissue sample deviates from a reference relative distribution of type A and type B cells, whereby the reference relative distribution represents a random distribution or any other type of expected relative distribution.

[0131] The difference in radius at which a predefined minimum B-type cell count is observed, respectively, indicates the deviation of the observed distribution from the expected (eg, random) relative distribution of A-type and B-type cells.

[0132] According to some embodiments, the Ripley's K of the observed relative distribution is multiplied by the cell density of type B cells observed in the image area. In addition, the Ripley's K of the reference relative distribution is multiplied by the cell density of type B cells observed in the image area of ​​the same size in the simulated or reference tissue sample image to obtain a reference cell count of type B cells near type A cells. Then, the difference between the observed and reference type B cell counts is calculated.

[0133] According to some embodiments, the method includes graphically representing the observed relative distribution as an observed distribution curve in a 2D plot whose first dimension represents the radius r(Ro) and whose second dimension represents the number of observed type B cells associated with the radius. The 2D plot is displayed on a display device of the image analysis system.

[0134] Additionally, or alternatively, the method comprises graphically representing the reference relative distribution as a reference distribution curve in a 2D plot whose first dimension represents the radius r (Rs) and whose second dimension represents the number of simulated B-type cells associated with the radius.

[0135] Preferably, the observed profile and the reference profile are plotted as the same 2D plot, as this facilitates comparison of the two profiles by the user.

[0136] Then, a 2D plot including the observed distribution curve and / or the reference distribution curve is displayed on a display device of the image analysis system. Preferably, confidence bands are also calculated from the minimum and maximum values ​​of a plurality of simulated or empirically obtained initial reference relative distributions, and the confidence bands are also drawn into the 2D plot such that the confidence bands surround the reference distribution curve.

[0137] This can be advantageous because the 2D plot allows even users with limited statistical knowledge to intuitively and quickly assess whether an observed relative distribution is significantly different from a given reference relative distribution.

[0138] According to an embodiment, type A cells are tumor cells and / or type B cells are immune cells. Preferably, the relative distribution is obtained by counting the number of immune cells observed or simulated in the spatial neighborhood of each tumor cell. This method may be preferred for most tumor types in which the density of tumor cells is greater than that of immune cells, because this method can ensure that a sufficient number of "count results" are obtained. The number of "count results" corresponds to the number of tumor cells, and in the case where immune cells are very rare in the tissue, it can usually be "zero".

[0139] According to other embodiments, the B-type cells are tumor cells and / or the A-type cells are immune cells. Preferably, the relative distribution is obtained by counting in the spatial neighborhood of each immune cell the number of tumor cells observed or simulated in the spatial neighborhood. This approach may be preferred for certain tumor types where immune cells have a greater density than tumor cells, or for other reasons where this is a better starting point for determining the number of cells of other cell types in their respective neighborhoods.

[0140] According to an embodiment, the tissue sample is a tumor tissue sample, and the determination of the biomedical state of the tissue sample comprises determining an infiltration state of tumor tissue depicted in a region of the received digital image having immune cells.

[0141] This may be advantageous because the determination of the infiltration state is an important prognostic parameter for characterizing tumors and selecting appropriate treatment options. The density of immune cells and tumor cells alone is not sufficient to distinguish, for example, tumor tissues that display an infiltration pattern of "blocking" and tumor tissues that display an infiltration pattern of "infiltration". Similarly, the spatial proximity of tumors and immune cells alone may not be sufficient to distinguish, for example, tumor tissues that display an infiltration pattern of "blocking" and tumor tissues that display an infiltration pattern of "desert". However, a combined score including density and spatial proximity information allows many different infiltration states to be distinguished, such as "infiltration", "blocking" and "desert". Therefore, the calculation of the combined score described herein is particularly useful in the context of determining the immune cell infiltration state of tumor tissue. The immune set point before immunotherapy may be very helpful in achieving truly personalized treatment. The combined score can better describe the characteristics of tumor patients in order to predict susceptibility to drugs with different modes of action, evaluate the response to a given immunotherapy, and provide decision support for continued treatment, each time switching to another drug with another mode of action The need for additional drugs.

[0142] The proximity value can be a numerical value represented by a specific point on a continuous scale, thereby providing more granularity and sensitivity than the manually defined classifications currently used. This can be very beneficial because most cases cannot be clearly classified into a specific extreme state such as "infiltrated", "desert" or "blocked", but rather fall into a moderately inflammatory state, and the difference between a minimally inflammatory and a minimally blocked tumor can be difficult to identify and characterize in a clear, reproducible and non-subjective way.

[0143] For example, tumor cells can be identified in a digital image of a tissue sample by staining the tissue sample with one or more tumor marker specific stains (e.g., stains that selectively stain cytokeratin, BRAC1, etc.) and applying image analysis methods to identify cells expressing the biomarkers as tumor cells. Optionally, the tissue can be stained with immune cell specific stains and image analysis methods performed so that selective non-immune cells expressing tumor markers or proliferation markers are identified as tumor cells. For example, cells expressing the biomarker KI67 and not expressing the lymphocyte biomarker CD8 (or CD3) are identified as tumor cells.

[0144] In some cases, it has been observed that not all tumor cells express Ki 67. However, it has been observed that Ki67+ cells provide a representative sample of tumor cells and provide accurate results.

[0145] According to an embodiment, immune cells are identified as cells expressing a biomarker that is selectively expressed in immune cells. For example, cytotoxic T cells can be identified as CD8A+ cells.

[0146] According to some embodiments, the region in which cells are identified in the received digital image includes a sub-region that is manually or automatically annotated as a "tumor region". In this case, preferably, only cells expressing proliferation biomarkers such as Ki67 and / or tumor markers are identified as "tumor cells", and the cells are additionally located in the "tumor region". And only cells expressing immune cell-specific biomarkers are identified as "tumor cells", and the immune cell-specific biomarkers are additionally located in the "tumor region". This ensures that regardless of whether the spatial relative distribution of tumor cells and immune cells is the same as the expected relative distribution, the biomedical characteristics of tissues outside the tumor region will not affect the analysis results.

[0147] According to some embodiments, the method further includes graphically representing the combined score as a symbol in a 2D score drawing. The first dimension of the 2D score drawing represents the proximity score, and the second dimension of the 2D score drawing represents the density of the B-type cells or the density of the A-type cells. Preferably, the density of the B-type cells is drawn, whereby the B-type cells are immune cells, and the A-type cells are tumor cells. The 2D score drawing is output on a display of an image analysis system to enable humans to determine the biomedical state of the tissue sample. In addition, or alternatively, the biomedical state of the tissue sample is determined by automatically identifying the biomedical state of the tissue sample within a limited set of predetermined biomedical states or within a predetermined continuous range of biomedical states. Then, the identified biomedical state is output (e.g., displayed) as the determined biomedical state.

[0148] Preferably, the density represented by one dimension of the 2D score plot is the observed density of one of the two types of cells having the lower density, such as type B cells. Preferably, type B cells are one of the two cell types counted in the neighborhood of each type A cell.

[0149] The determined biomedical state of the tissue may be identified in or mapped to a continuous range of biomedical states. Additionally, or alternatively, the biomedical state may be one of a finite set of predefined, discrete biomedical tissue states.

[0150] According to some embodiments, determining the immune cell infiltration state of the tumor sample includes classifying the tumor into different predetermined infiltration states, such as "desert," "barrier," or "inflammation." For example, a threshold value of the combined score value can be obtained from empirical data of multiple tissue samples, and the threshold value of the obtained combined score value can be used to classify the tissue sample depicted in the image region into one of the predetermined infiltration states.

[0151] According to other embodiments, determination of the infiltration state includes identifying the infiltration state of the tissue sample on a predetermined continuous range representing the level of inflammation in the tumor. The extremes of the range represent highly inflamed tumors, rather than tumors completely devoid of immune cells. In this case, rather than identifying one of a predetermined, limited set of infiltration states, a continuous score is provided that allows mapping of the tumor in the depicted image region on a range between the most extreme infiltration states.

[0152] The infiltration of tumors by immune cells is triggered by the presence of mutated tumor antigens in the context of the MHC on the surface of tumor cells. Tumor antigens are considered non-self and are eliminated primarily by natural killer cells and cytotoxic T cells. This elimination relies on complex signaling pathways that can be altered or impeded by the tumor. This balance between tumor progression and attempts by the immune system to kill tumor cells is described as the elimination-equilibrium-escape mechanism. Current developments in immunotherapy have great potential to change the poor prognosis of many cancer patients by altering the tumor immune microenvironment in favor of the tumor cell elimination side of the balance.

[0153] The field has seen many proposals for tumor immune phenotypes to measure and describe the state of the elimination-equilibrium-escape balance, but no consensus has been reached. The main obstacle to overcome is the lack of a deeper understanding of how the theoretical continuum between elimination and escape translates into biologically meaningful information presented in histological tumor samples. Manual scoring systems suggest dividing the range into a varying number of categories, and in general, there is no consensus on how to find thresholds or objective definitions for these categories. Embodiments of the present invention allow these obstacles to be overcome and allow for the provision of automated, calibrated, and discrete measurements of tumor infiltration status based on the presence and distribution of tumor-infiltrating immune cells. Thus, embodiments of the present invention allow for the capture of information about biological trends, such as if the balance of the immune system is biased towards elimination of tumor escape, and if the immune system as a whole is activated or silenced.

[0154] According to an embodiment, the image analysis method further includes: receiving a digital image of a tissue sample from each of a plurality of different patients; calculating a combined score for each of the patients using the corresponding received images; and graphically representing the biomedical state of each patient on a 2D score plot by a corresponding biomedical state-specific symbol, the position of the symbol for each patient in the plot depending on the B-cell density and the proximity score calculated for the patient.

[0155] This can be advantageous because a user is provided with a plot that allows the biomedical status (e.g., immune cell infiltration status) of a tissue sample to be quickly and intuitively determined based on the x and y coordinates of a point in the plot, whereby the point represents the tissue sample and the x and y coordinates of the plot represent the proximity score and immune observed B-type cell density in the tissue sample.

[0156] Immune infiltration in a tumor is the host's response to a foreign body in the body. It is triggered by tumor antigens (mutated proteins) present on tumor cells. If the immune system is suppressed or the tumor downregulates the expression of tumor antigens, the tumor may lack immune cells. In fact, the amount of immune infiltration is seen as a spectrum from none to severe infiltration.

[0157] According to an embodiment, the identified biomedical state is selected from the group consisting of:

[0158] - "Inflammation", wherein "Inflammation" indicates an immune tissue state in which immune cells have a significantly increased cell density, which indicates that the tumor tissue is heavily infiltrated with immune cells in all compartments of the tumor, wherein the higher the density of immune cells observed in the image area and the higher the proximity score, the higher the possibility that the tissue sample is classified as "Inflammation";

[0159] - "blocked", wherein "blocked" indicates an immune tissue state in which immune cells are present in the tissue sample but are blocked from close contact with tumor cells, whereby immune cells are concentrated in the invasion margin and / or intratumoral stroma but separated from tumor cells, wherein the higher the density of immune cells observed in the image area and the lower the proximity score, the higher the probability that the tissue sample is classified as "blocked";

[0160] - "Desert", where "Desert" indicates an immune tissue state in which the cell density of immune cells in the tumor tissue region is zero or close to zero, where the lower the immune cell density observed in the image area, the higher the probability that the tissue sample is classified as "Desert", regardless of the height of the proximity score.

[0161] Applicants have observed that many tumors display images that do not fit neatly into any of the above extreme categories. Therefore, the calculation of a combined score value is advantageous because it is a combination of two continuous numerical score values ​​(proximity score value, cell density), thus allowing for an accurate prediction / determination of the biomedical state of the tissue based on two important features: Density indicates whether the organism is capable of eliciting an immune response. The proximity score indicates whether the tumor is immunogenic and suitable for attracting immune cells. If not, the tumor may have developed ways to avoid immune cells (e.g., HLA-G upregulation or cytokine production, which would render immune cells dormant or stimulate regulatory immune subsets).

[0162] According to an embodiment, when the biomedical state of the tissue sample is determined to be the infiltration state "inflammation", the image analysis method includes outputting a treatment recommendation via a user interface to prescribe a drug used as a checkpoint inhibitor, such as anti-PD-L1 / PD-1, CTL4.

[0163] Additionally, or alternatively, in the case where the biomedical state of the tissue sample is determined to be a "blocking" infiltration state, the image analysis method includes outputting a treatment recommendation via a user interface to prescribe a drug suitable for attracting immune cells closer to tumor cells. For example, such a drug can be a bispecific antibody against tumor cells. An example of such a bispecific antibody is CEA-TCB, which simultaneously binds to CD3 on T cells with one arm and to CEA on tumor cells with two arms, thereby bringing T cells into close proximity with cancer cells. This results in T cell activation and subsequent tumor cell killing. In the case of a "blocking" infiltration state, another example of a recommended suitable drug is a drug that destroys tumor escape mechanisms, such as drugs that block HLA-G or upregulate MHC I.

[0164] Additionally or alternatively, in case the biomedical state of the tissue sample is determined to be an infiltration state "desert", the image analysis method comprises outputting a treatment recommendation via the user interface to prescribe a drug suitable for generally enhancing the immune system, such as IL-2.

[0165] According to an embodiment, the shapes of tumor cells and immune cells within the image region are stored in the form of points in a spatial database. The simulated distance and the measured distance are identified using spatial database operations provided by the spatial DBMS, particularly in the WITHIN_DISTANCE() and DISTANCE() operations. This may be advantageous because the calculation of the relative distribution can be greatly accelerated. This is particularly useful if a large number of initial reference relative distributions must be simulated in order to provide a confidence band of the desired width. For example, the SDO_GEOM function of the Oracle Spatial DBMS may be used.

[0166] According to other embodiments, the score is calculated outside of DB using R

[0167] According to an embodiment, the expected distribution indicates the distance between the reference type A cells and the reference type B cells expected under the assumption of a Poisson distribution of type A cells and type B cells in the region. Preferably, the assumption is further based on the observed same density of type A cells and reference type A cells and the observed same density of type B cells and reference type B cells.

[0168] In another aspect, the present invention relates to a method for determining the efficacy of a drug for treating a specific type of cancer. The method comprises:

[0169] - receiving a plurality of first digital images, each first image depicting a tissue sample of a respective organism having a specific type of cancer before treatment of said organism with said drug;

[0170] - calculating a first combined score from the image area of ​​each of the first digital images, wherein the biomedical state is the immune cell infiltration state of the tumor, the type A cells are tumor cells, and the type B cells are cancer cells;

[0171] - receiving a plurality of second digital images, each second image depicting a tissue sample of a respective organism having a specific type of cancer after treatment of said organism with said drug;

[0172] - calculating a second combined score from the image area of ​​each of said second digital images, wherein the biomedical state is the infiltration state, the type A cells are tumor cells, and the type B cells are cancer cells;

[0173] - displaying a 2D score plot on a display device, wherein each first combined score is represented by a first symbol in said 2D score plot, wherein each second combined score is represented by a second symbol in said 2D score plot, wherein the positions of said first and second symbols in said plot depend on said immune cell density observed in said image region of said corresponding image and said proximity score calculated for said image region of said corresponding image, wherein first and second symbols representing the same organism are visually linked in said 2D score plot for visualizing any offset in x and / or y coordinates from any one of said first symbols to its corresponding linked second symbol.

[0174] This may be advantageous because the above method allows for a visual representation of the efficacy of a particular drug tested on multiple different patients, thereby visualizing the effect of the drug on the spatial distribution of immune cells and tumor cells. Visual representation of modifications to the relative spatial distribution of cells in a single patient has been a challenge. Here, the response of multiple patients to a particular drug in terms of changing the relative spatial distribution of tumor and immune cells is visualized, allowing medical professionals to easily assess whether the drug has an effect and to assess which type of biomedical state (e.g., tumor infiltration state) the drug promotes or inhibits. Visual linking can be performed, for example, by connecting symbols (e.g., circles, triangles, etc.) originating from the same patient with a line.

[0175] In another aspect, the present invention relates to a storage medium comprising computer-interpretable instructions which, when executed by a processor, cause the processor to perform an image analysis method according to any one of the embodiments and examples described herein.

[0176] In another aspect, the present invention relates to an image analysis system for determining a biomedical state of a tissue sample. The system comprises a processor and computer-interpretable instructions configured to cause the processor executing the instructions to perform a method comprising:

[0177] - receiving a digital image of a tissue sample;

[0178] - analyzing the received image to identify the number and location of type A cells and type B cells observed in the area of ​​the received image, wherein the type A cells and the type B cells are different cell types;

[0179] - analyzing the positions of the type A and type B cells in the region to obtain an observed relative distribution indicating an observed distance between the type A cells and the type B cells in the region;

[0180] - obtaining a reference relative distribution indicating an expected distance between a reference A-type cell and a reference B-type cell;

[0181] - calculating a closeness score as the difference between the reference relative distribution and the observed relative distribution;

[0182] - calculating a combined score comprising the proximity score and comprising the density of the type A cells and / or the density of the type B cells;

[0183] - using the combined score to determine the biomedical status of the tissue sample and / or outputting the combined score to enable a user to determine the biomedical status of the tissue sample.

[0184] "Relative distribution" as used herein is a function that describes the spatial discreteness between two types of objects. According to a preferred embodiment, "relative distribution" is a function that captures the distance of a first type of object to a second type of object. According to a preferred embodiment, "relative distribution" quantifies spatial discreteness in a manner that is invariant to rotation and mirroring. Several simple spatial discreteness measures can be defined using, for example, the covariance matrix of the object coordinates. The trace, determinant, and maximum eigenvalue of the covariance matrix can be used as measures of spatial discreteness. An example of a spatial discreteness measure that is not based on a covariance matrix is ​​the average distance between nearest neighbors. A further preferred measure of spatial discreteness is the Ripley K function.

[0185] "Reference relative distribution" used herein is a kind of relative distribution, which captures the spatial discreteness between two types of objects (such as reference A-type cells and reference B-type cells), and is used as a reference. "Reference" is a kind of "standard" or "baseline", and another kind of information (such as another relative distribution of two types of objects) can be compared with it. Reference relative distribution can be derived by computational simulation based on the expected distribution of object (such as based on Poisson distribution), or can be derived from the relative distribution of the object observed in one or more reference samples according to experience. Reference relative distribution can represent "baseline". It allows the relative distribution of different cell types in the tissue sample currently inspected to be compared with the relative distribution of the same type of cells in one or more reference tissue samples with known biomedical status. Observing the comparison of relative distribution with reference to relative distribution can allow determining whether the relative distribution of cells in the tissue sample significantly deviates from the relative distribution of the type of cells described in the reference relative distribution determined by simulation or experience.

[0186] As used herein, "observed relative distribution" captures the relative distribution of spatial dispersion between two types of objects (eg, two types of cells) within a tissue sample depicted in the digital image currently being analyzed.

[0187] "Proximity score" used here is a score value calculated as a function of observed relative distribution and reference relative distribution. Preferably, the proximity score is calculated by calculating the difference between one or more distances or distance metrics encoded in the observed relative distribution and the reference relative distribution. According to an embodiment of the present invention, the score value is a digital data value. According to an embodiment of the present invention, the score value is calculated as the difference (increment) between the reference distribution and the observed distribution. For example, the score value can be calculated as the difference between two distances r1, r2, wherein r1 can be the distance from the A-type cell at which a specific number of B-type cells are observed on average according to the observed relative distribution, and wherein r2 can be the distance from the A-type cell at which a specific number of (simulated or empirically determined) B-type cells are observed on average according to the reference relative distribution. Alternatively, the difference can be the difference between two regions, wherein one region is defined by two predefined distance thresholds, a baseline for "0 observed events occur" and a section of the reference relative distribution curve between the two distance thresholds, and a second region is defined by two predefined distance thresholds, a baseline and a section of the observed relative distribution curve between the two distance thresholds.

[0188] As used herein, "tumor immune phenotype" or "spatial immune infiltration phenotype" is a method of dividing tumors into different groups or mapping them on a continuous or discrete scale according to the grade of tumor inflammation. The extremes of the range would represent highly inflamed tumors versus tumor tissue completely devoid of immune cells.

[0189] "Ripley's K" as used herein is a spatial descriptive statistic that can be used to generate a reference relative distribution and / or an observed relative distribution. More specifically, it is a measure of the spatial discreteness between two types of objects. For example, Philip M. Dixon described the Ripley K function in Encyclopedia of Environmetrics (ISBN 0471 899976), edited by Abdel H. El-Shaarawi and Walter W. Pigorsch, John Wiley & Sons, Ltd, Chichester in 2002, Vol. 3, pp. 1796-1803. The Ripley K (t) function, also known as the "K function" or "Ripley's K", is a mathematical representation of the expected value of the distance between type A cells and type B cells.

[0190] As used herein, a "biomedical state" is any disease, metabolic, genomic, proteomic-related or tissue morphological state known in a biological and / or medical context or having predictive or therapeutic use. For example, the "biomedical state" of a tissue can be related to the degree to which a tumor tissue is infiltrated by immune cells, it can be related to the degree of differentiation of a tumor tissue or a tissue of a young, developing organism. BRIEF DESCRIPTION OF THE DRAWINGS

[0191] FIG1 is a block diagram of an image analysis system;

[0192] FIG2 is a flow chart of an image analysis method for determining the biomedical status of a tissue sample;

[0193] FIG3 depicts images of three tumor tissue samples with different degrees of immune cell infiltration and corresponding schematic diagrams of cell distribution;

[0194] FIG4 depicts the density of immune cells and tumor cells in the three tissue samples of FIG3 , respectively;

[0195] Figure 5 depicts the use of radius to define spatial cell neighborhoods and calculate the relative distribution of cells of different cell types;

[0196] Figure 6 depicts the relative distribution of observations obtained from images showing the “wetting” and “blocking” wetting states;

[0197] 7A and 7B depict a 2D plot including the observed relative distribution, the reference relative distribution, the confidence bands, and the "delta" used as the proximity score, respectively;

[0198] FIG8 depicts an alternative, region-based analysis of a 2D plot for determining a "delta" to be used as a proximity score;

[0199] FIG9 depicts fluorescent images of two tumor tissue samples with either “blocking” or “infiltrating” immune cell infiltration states;

[0200] FIG10 depicts a 2D score plot including clusters of combined scores, each combined score representing a corresponding one of a plurality of different tissue samples;

[0201] FIG11 depicts a logarithmic 2D score plot including points representing combined score values ​​obtained from multiple patients;

[0202] FIG12 depicts a logarithmic 2D score plot comprising paired connection points representing combined score values ​​obtained from multiple patients before and after treatment;

[0203] FIG13 depicts a logarithmic 2D score plot including combined score values ​​for three clusters obtained from multiple patients; and

[0204] FIG. 14 depicts a logarithmic 2D score plot including combined score values ​​of five clusters (labeled 1-5, respectively) obtained from multiple patients. DETAILED DESCRIPTION

[0205] In the following embodiments of the invention, purely by way of example, the invention is explained in more detail with reference to the accompanying drawings, in which:

[0206] Fig. 1 is a block diagram of an image analysis system 100 according to an embodiment of the present invention. The system includes one or more processors 104, a main memory 106 and a non-volatile storage medium 108. The storage medium includes one or more applications or modules 110, 114, 112, 116, 120, 122, which are configured to perform one or more data processing tasks, such as automatic detection of cells of different cell types (module 110), measurement of intercellular distance (module 112), simulation of cell distribution of a given density (module 116), drawing of various scores and curves (module 122), calculation of proximity scores and combined scores (module 120) and automatic determination or prediction of biomedical states based on combined scores (module 114). Various modules can be arbitrarily combined in one or more applications. For example, the functions provided by the modules can be combined into a single application. Alternatively, some functions can be performed by separate applications. For example, the simulation of a given distribution can be performed by a mathematical program such as R, the calculation of distance can be performed by a spatial DBMS or interacting with a spatial DBMS, and the identification of cells of a specific cell type can be realized by digital pathology image analysis software.

[0207] The storage medium 108, such as an electromagnetic hard drive, may include one or more digital pathology images 118 that respectively depict a tissue sample. For example, the digital image 118 may be a monochrome image or a multi-channel image. For example, the image 118 may be an RGB image. The image 118 may be a bright field microscopic image or a fluorescence image. Typically, the image is a multi-channel image obtained by fluorescence microscopy from a tissue sample that has been stained with one or more biomarker-specific stains having different colors.

[0208] The cell type detection module can be configured to automatically detect tumor cells and immune cells, and to store the location and type of automatically detected cells in a spatial database. Identification of cells and corresponding cell types can perform connected component analysis and edge detection routines to identify pixel spots representing cells. Information encoded in color or other image features can be used to identify cell types. The color information encoded in the digital image 118 is generally indicative of a specific biomarker or a specific set of biomarkers. For example, a specific set of cytokeratins can be stained with a suitable antibody coupled to a fluorescent or colored dye. By applying a color deconvolution algorithm, multiple monochrome images 118 can be derived from a multispectral fluorescence image of a specific tissue sample.

[0209] For example, cells expressing specific tumor markers or proliferation markers that do not express immune cell specific markers can be considered tumor cells. Cells expressing immune cell specific markers such as CD8A can be identified as immune cells.

[0210] According to another example, the anti-pan-keratin primary antibodies "AE1 / AE3 / PCK26" of Ventana Medical Systems can be used to stain poorly differentiated malignant tumors. A set of anti-pan-keratin antibodies "AE1 / AE3 / PCK26" specifically binds to antigens located in the cytoplasm of simple and complex epithelial cells. This is a mouse monoclonal antibody mixture directed against epitopes found on human epidermal keratins reported by Woodcock-Mitchell et al. This antibody mixture reacts with 56.5kD, 50kD, 50'kD, 48kD, and 40kD cytokeratins of the acidic subfamily and 65-67kD, 64kD, 59kD, 58kD, 56kD, and 52kD cytokeratins of the basic subfamily.

[0211] Depending on the embodiment, the one or more digital images 118 are generated by a color deconvolution algorithm or stored directly in a storage medium after being acquired by a camera or other image capturing device.The image analysis system may optionally include an image capturing device, such as a camera (not shown).

[0212] In addition, the system 100 is coupled to or includes a display 102, such as a liquid crystal display. The system uses the display 102 to display digital images 118 of tissue samples of various patients, to display various plots including density, relative distribution, and combined score values, and / or to display cluster analysis results of biomedical tissue states that are automatically identified by combined score values ​​obtained from specific tissue samples.

[0213] Fig. 2 is a flow chart of an image analysis method for determining the biomedical state of a tissue sample. The method will be described below using tumor cells and immune cells as examples of different cell types, and using the immune cell infiltration state of a tumor tissue sample as the biomedical tissue state to be automatically determined. The method can allow for the automatic stratification of patients based on their potential response to cancer immunotherapy using image analysis with spatial information. However, this is merely an example of how a combined score as described herein is calculated based on the relative distribution of two different cell types for rapid, automatic and accurate determination of the biomedical state of a particular tissue in the context of biology and medicine. The difference between the distribution of the two cell types and the expected reference distribution can be used for the determination and subclassification of many other diseases or physiological states and / or for identifying appropriate treatment modalities or drugs.

[0214] The method can be implemented and executed in, for example, the image analysis system 100 shown in FIG1 . In a first step 202, the image analysis system 100 receives a digital image of a tissue sample. For example, the digital image can be received directly from an image capture device (e.g., a microscope), can be received via a network (e.g., the Internet), or can be read from a non-volatile storage medium 108.

[0215] Next, in step 204, the image analysis system analyzes the received image to identify the number and location of type A cells (e.g., tumor cells) and type B cells (e.g., immune cells) observed in the region of the received image. Image analysis may include the application of various image processing and analysis techniques, such as color deconvolution, image segmentation, blob detection, connected component analysis, feature extraction, future clustering, etc.

[0216] Next, in step 206, the image analysis system analyzes the location of the type A and type B cells in the region to obtain an observed relative distribution. The observed relative distribution indicates the distance between the type A cells and type B cells observed in the region. In addition, the image analysis system determines the density of tumor cells and immune cells observed in the image region, respectively.

[0217] Next, at step 208, the image analysis system obtains a reference relative distribution. The reference relative distribution can be obtained by, for example, performing a plurality of simulations for generating expected distributions of tumor cells and immune cells, respectively, and determining distances between different types of simulated cells. The expected distribution can be a random distribution, such as a Poisson distribution assuming that the density of tumor cells and immune cells observed in the image region is the same.

[0218] Next, in step 210, a proximity score is calculated as the difference between the reference relative distribution and the observed relative distribution. As used herein, the expression "calculating the difference between the reference relative distribution and the observed relative distribution" means that the proximity score is calculated as a function of the observed and reference relative distributions, wherein the function includes at least one operation that calculates the difference between one or more "observed" distances encoded in the observed relative distribution and one or more "reference" distances encoded in the reference relative distribution.

[0219] Next in step 212, the image analysis system combines the proximity score with the immune cell density measured in the received digital image 118 to provide a "combined score."

[0220] Next, in step 214, the image analysis system uses the combined score to automatically determine the biomedical state of the tissue sample, such as a specific infiltration state. In addition, or alternatively, the image analysis system displays the combined score obtained for the tissue sample to the user on the display 102 of the image analysis system in step 216. Therefore, the user can intuitively evaluate the current biomedical state of the tissue and make corresponding conclusions about the progression of the disease and appropriate treatment options. For example, as shown in Figures 9, 10-13, the combined score of the tissue sample can be represented as a data point in a 2D score plot.

[0221] According to a preferred embodiment, the observed relative distribution and the reference relative distribution are calculated using Ripley's K function.

[0222] According to alternative embodiments, simpler methods may also be used. For example, the image analysis system 100 may receive 202 a digital image 118 of a tissue sample. The image analysis system analyzes 204 the received image and identifies CD8+ immune cells and Ki-67+ tumor cells in the image or in a specific sub-region of the image. For example, the sub-region may be an image block of a predetermined size, or may be an automatically detected tumor tissue region. For each identified tumor cell, the image analysis system determines 206 the distance between the tumor cell and its nearest immune cell. After determining the distances of all tumor cells to their respective nearest immune cells, the total distance may correspond to and represent the "observed relative distribution" of tumor cells and immune cells in the tissue. In the next step, the identified density of tumor cells and immune cells in the image or in the tumor tissue region is determined and used to simulate the Poisson distribution of simulated tumor cells and simulated immune cells, respectively. Therefore, the density of simulated tumor cells and simulated immune cells is the same as the density of tumor cells and immune cells observed in the tissue. Then, the distances of all simulated tumor cells to their respective nearest simulated immune cells are determined 208 and expressed as the "reference relative distribution". Then, a proximity score is calculated 210 based on the determined distance. For example, for the observed relative distribution and the reference relative distribution, the distance at which each tumor cell contains an immune cell in its neighborhood on average can be determined, and the difference of these two differences can be used as the proximity score. The image analysis system then provides 212 a combined score including the proximity score and the immune cell density observed in the image 118, and uses 214 the combined score for automatically determining the biomedical state of the tissue (e.g., infiltration state) and / or for displaying 216 the combined score on a 2D score plot, for example, as shown in FIGS. 9, 10-13, on the display 102 of the image analysis system, for enabling a user to intuitively assess the current biomedical state of the tissue sample.

[0223] FIG3 depicts images of three tumor tissue samples with different degrees of immune cell infiltration and a schematic diagram of the corresponding cell distribution.

[0224] Image 302 depicts a digital pathology image of a tumor tissue having an "infiltrated" state, which means that the tumor tissue is heavily infiltrated by immune cells. The relative spatial distribution of immune cells (triangles) and tumor cells (dots) in the "infiltrated" tumor tissue is shown in the schematic diagram 308 below.

[0225] Image 304 depicts a digital pathology image of tumor tissue having a "blocking" state, meaning that the tumor has created a "border" that prevents immune cells from migrating into the tumor tissue. The relative spatial distribution of immune cells and tumor cells in the "blocking" tumor tissue is shown in the schematic diagram 310 below.

[0226] Image 306 depicts a digital pathology image of tumor tissue having a "desert" / "abandoned" state, meaning that the tumor is substantially devoid of any immune cells. The relative spatial distribution of immune cells and tumor cells and "desert" tumor tissue is shown in the schematic diagram 312 below.

[0227] The three images and distributions depicted are clear representations of the three different biomedical states described above. However, in most cases, tissue samples are between two different states. Therefore, currently available methods cannot provide a reliable, objective and reproducible way to determine the biomedical state of tissue samples that do not clearly belong to one of the three categories.

[0228] FIG4 depicts the density of immune cells and tumor cells in the three tissue samples of FIG3, respectively. In addition, FIG4 includes plots 402, 404, and 406, which show the density of immune cells (CD8A+) and tumor cells (Ki-67+) in the corresponding states. The figure shows that the density of immune cells is an important prognostic parameter for distinguishing "desert" tissue from the other two tissue types, but is not sufficient to distinguish "infiltrated" and "blocked" tissue samples.

[0229] FIG5 depicts the use of radius to define spatial cell neighborhoods and calculate the relative distribution of cells of different cell types. The image depicts six tumor cells 504 indicated by small circles and multiple immune cells indicated by small triangles. In order to obtain the relative distribution between immune cells and tumor cells observed in the image, each identified tumor cell is used as the center of a circle with a radius of r. The radius r defines a circular neighborhood. By increasing the radius of the circle surrounding the tumor cell in steps and counting the number of immune cells contained in the circles created respectively, a cumulative measure of the number of immune cells around the tumor cell can be obtained. The number of steps and the respective radii can be freely selected as a compromise between computational resource consumption and accuracy. For example, each step can increase the radius by 1 μm, 5 μm or 10 μm, etc. The association between each radius and the number of immune cells counted in the circular neighborhood defined by the radius can be represented by a variety of different mathematical formulas and distributions. Preferably, the Ripley K function is used to describe the relative spatial distribution of immune cells and tumor cells obtained by the step-by-step increase of the radius.

[0230] FIG6 depicts the observed relative distribution obtained from images showing the "infiltration" and "blocking" infiltration states. The upper part of FIG6 depicts the spatial distribution scheme of immune cells and tumor cells, which has been depicted and described in FIG3. The lower part of FIG6 depicts plots 602, 604, 606, which have a y-axis representing the result of the Ripley K function K(r) and an x-axis representing the (step-increasing) radius, respectively. In the depicted example, the radius increases by 1 μm. Therefore, for each micron on the x-axis, a data point is obtained. The curves in the plot are obtained by curve fitting the data points obtained in the plot. As can be seen in the case of infiltration, the result obtained by the Ripley K function is positively correlated with the increase in radius r. In the case of "blocking", K(r) hardly increases with the increase of r. In the case of "desert", K(r) increases slightly with the increase of r. Therefore, the relative spatial distribution may not always be sufficient to distinguish between blocking and desert states, but it can clearly distinguish and infiltration states from blocking or desert states. Therefore, the combination of proximity scores and density information encoding differences in relative spatial distribution can allow the current biomedical state of a tissue to be clearly identified across a complex continuum of states.

[0231] FIG. 7A depicts a 2D plot 702 whose y-axis represents the number of immune cells observed in a circular neighborhood around a tumor cell defined by the circle radius r and whose x-axis represents the radius r.

[0232] The first plot 702 depicts the average number of immune cells observed within a circle of radius r for different radii of tumor cells in a tissue sample having an "infiltrated" state. Curve 706 represents the observed relative distribution obtained by curve fitting the various data points in plot 702.

[0233] The second plot 704 depicts the average number of immune cells observed within a circle of radius r for different radii of tumor cells in a tissue sample with a "blocked" state. Curve 708 represents the observed relative distribution and is obtained by curve fitting the various data points in plot 704.

[0234] FIG. 7B depicts 2D plots 702 , 704 supplemented with additional information.

[0235] Plot 702 is supplemented by an average reference relative distribution 714. The average reference relative distribution is obtained by generating multiple (e.g., 40) initial reference relative distributions. Each initial relative distribution is calculated by simulating a Poisson distribution of simulated tumor cells with the same cell density as the observed tumor cells, by simulating a Poisson distribution of simulated immune cells with the same cell density as the observed immune cells, and determining a distance metric that provides information about the relative spatial distribution of immune cells and tumor cells (e.g., the number of immune cells obtained in a circular neighborhood of a tumor cell using radius r). For each radius examined (r=1 μm, 2 μm, ..., 98 μm, 99 μm, 100 μm), and for each of the 40 simulations, the average number of immune cells in a circle with a radius of r around the tumor cell is determined. Then, for each radius examined, the minimum, average, and maximum number of immune cells within a circle with a radius of r around the tumor cell is determined. The average is plotted as a data point for each r, which is used to generate curve 714 based on curve fitting. Curve 714 represents a reference relative distribution based on simulation. The minimum number is plotted as each r data point for generating the lower boundary of the confidence band 716 by curve fitting of the data points. The maximum number is plotted as each r data point for generating the upper boundary of the confidence band 716 by curve fitting of the data points. The confidence band 716 represents the region in which any observed relative distribution is assumed to be not significantly different from the reference relative distribution 714.

[0236] "Increment = 19" is a proximity score obtained by identifying a first point RP in the reference relative distribution 714 where the average number of immune cells in a circular tumor cell environment is "1", identifying a second point OP in the observed relative distribution 706 where the average number of immune cells in a circular tumor cell environment is "1", and subtracting the x value (radius) of the first point from the second point. The resulting difference ("radius increment" or "distance increment") is used as the proximity score for the tissue sample. The user can easily infer from the fact that the second point OP is outside the confidence band 716 that the observed relative distribution 706 is significantly different from the expected / reference relative distribution 714.

[0237] Plot 704 is supplemented by an average reference relative distribution 710. The average reference relative distribution is obtained by generating a plurality (e.g., 40) of initial reference relative distributions. Each initial relative distribution is calculated by simulating a Poisson distribution of simulated tumor cells having the same cell density as the observed tumor cells of the tissue represented in plot 704, as described above for plot 702. Curve 710 represents the reference relative distribution based on the simulation. The minimum numbers are plotted as data points for each r for curve fitting through the data points to generate the lower boundary of the confidence band 712. The maximum numbers are plotted as data points for each r for curve fitting through the data points to generate the upper boundary of the confidence band 712. The confidence band 712 represents the region where any observed relative distribution is assumed to not be significantly different from the reference relative distribution 710.

[0238] "Increment = 32" is a proximity score obtained by identifying a first point RP where the average number of immune cells in a circular tumor cell environment in the reference relative distribution 710 is "1", identifying a second point OP where the average number of immune cells in a circular tumor cell environment in the observed relative distribution 708 is "1", and subtracting the x-values (radii) of the first and second points. The resulting difference ("radius increment") is used as the proximity score for the tissue sample. The user can easily infer from the fact that the second point OP lies outside the confidence band 714 that the observed relative distribution 708 is significantly different from the expected / reference relative distribution 710.

[0239] FIG. 8 depicts a plot 750, which is a modified version of plot 702 depicted in FIG. 7B. Plot 750 can be used to automatically or manually determine alternative, region-based "increments" to be used as proximity scores.

[0240] Plot 750 shows an average reference relative distribution 714 and an observed relative distribution 706. Additionally, plot 750 shows a first predetermined distance threshold t1, whose value is, for example, 10 - 20 μm, 15 μm in this example, and a second predetermined distance threshold t2, whose value is, for example, 30 - 40 μm, 35 μm in this example. The two thresholds t1, t2 each correspond to a reference line.

[0241] The two reference lines corresponding to t1 and t2, the baseline of "CD8# T cell count < r = 0", and the observed relative distribution curve 706 define a first region graphically represented by " / / / " hatching. For example, the vertical line 754 is the boundary of the first region defined by the second distance threshold t2.

[0242] Two auxiliary lines corresponding to t1 and t2, the baseline of "CD8#T cell count < r = 0", and the average reference relative distribution curve 714 define a second region graphically represented by the "\\" hatched line. For example, the vertical line 752 is the boundary of the second region defined by the second distance threshold t2.

[0243] The region where the first region and the second region overlap is represented by the crossed "XXX" hatched line.

[0244] The "incremental value" can now be calculated as the difference or ratio of the sizes of the first and second regions. This "incremental value", also known as the "increment of the region size", is used as the proximity score of the tissue sample. The user can easily infer that the observed relative distribution 706 is significantly different from the reference relative distribution 714 from the position of the curve segment of the curve 706 that is far from the gray confidence band around the curve 714.

[0245] Figure 9 depicts the fluorescence image 804 of a tumor tissue sample with a "blocked" infiltration state and the fluorescence image 802 of a tumor tissue sample with an "infiltrated" infiltration state. Tumor samples from clinical trials were fixed with formalin, cut into 2.5-μm thick sections, and stained with double staining assays for CD8 and Ki-67. The stained slides were scanned and imported into a digital pathology platform for manual annotation by pathologists to identify tumor regions, normal tissue regions, and necrotic regions. Regions with artifacts were excluded. Automatic whole-slide image analysis was performed on the images using software logic suitable for automatically detecting CD8-Ki-67 double-positive cells, CD8-positive-Ki-67-negative T cells, and Ki-67-positive and CD8-negative tumor cells. The pathologist checked the accuracy of the detection before entering the results into the database. Only the objects detected in the tumor annotation region were used for further analysis. The densities of CD8+ T cells, CD8+ / Ki-67+ cells, CD8+ / Ki-67- cells, and CD8- / Ki-67+ tumor cells were calculated. As described below, this type of fluorescence image can be used to stratify patients.

[0246] Figure 10 depicts a 2D score plot that includes 80 combined scores, which are clustered into three groups. Each combined score represents one of 80 different patients (from whom tissue samples were taken).

[0247] The x-axis of the 2D score plot 902 represents the logarithm of the density of immune cells (CD8A+ cells). The y-axis represents the proximity score. Cluster 906 represents tumor tissue that is "infiltrated" by immune cells. Cluster 908 represents tumor tissue that is determined to have a "desert" infiltration state. Cluster 910 represents tumor tissue that is determined to have a "blocking" infiltration state. It can be inferred from the 2D score plot that immune cell density is an important parameter for distinguishing desert and infiltrating tissue samples. However, density alone may not be sufficient to distinguish "blocking" tissue samples from "infiltrating" tissue samples. However, the additional dimension (y-axis) representing the proximity score distinguishes "blocking" tumor samples from "infiltrating" tumor samples.

[0248] The bold line 904 indicates that the combination of immune cell density and proximity score may allow for better differentiation of biomedical tissue states than density information alone.

[0249] FIG11 depicts a logarithmic 2D score plot, including points representing combined score values ​​obtained from multiple patients. In general, it can be inferred that an increase in immune cell density correlates with an increased binding of immune cells to tumor cells. However, there is a great deal of variability between different patients and even between different tissue samples from the same patient. These differences can be used to determine the current biomedical state of the tissue and identify appropriate treatment options.

[0250] FIG. 12 depicts a logarithmic 2D score plot including paired connected points representing combined score values ​​obtained from multiple patients before and after treatment. The two data points connected by each arrow represent a combined score for a tissue sample obtained from a patient before the patient was treated with a drug and a combined score for another sample of the same patient obtained a few days or weeks after the patient was treated with the drug. For example, the drug may be a drug that enhances the immune system. The drug may be a drug applied during immunotherapy of the patient.

[0251] As can be inferred from the plot, in response to treatment of the patient with a particular drug, almost all patients had a significant increase in immune cell-tumor cell binding (most arrows in the plot point in an upward direction). In addition, almost all patients had an increase in immune cell density in response to drug application (most arrows go from left to right). However, some patients did not respond and even showed a decrease in immune cell numbers and / or immune cell-tumor binding.

[0252] Thus, repeated acquisition of a combined score from one or more patients over a period of time can detect trends in the progression of a disease or other physiological state. Typically, biological driving mechanisms are reflected in tissues as relative shifts in the distribution of different cell types. Applicants have observed that, in addition to the density of immune cells, the relative shift in distribution between tumor cells and immune cells provides meaningful information about a patient's position on the tumor elimination-equilibrium-escape balance, as well as an indication of the direction of the process over time toward more elimination or more escape.

[0253] Figure 13 depicts a logarithmic 2D score plot, which includes a combined score value obtained from multiple patients. According to the position of the patient's respective combined score value in the figure, the patient is divided into three clusters. Therefore, the comprehensive score allows the tumor infiltration state of a specific patient to be mapped to a continuous scale representing different infiltration states of the tumor. Samples and patients can be divided into three different clusters, but a more fine-grained classification is also possible, as shown in Figure 14. Cluster 972 represents the "infiltration" state, cluster 974 represents the "blocking" state, and cluster 978 represents the "desert" state.

[0254] Figure 14 depicts a logarithmic 2D score plot including combined score values ​​for the five clusters obtained from multiple patients. In addition to the three different biomedical states represented by the individual clusters in Figure 13, Figure 14 also includes combined score clusters for the "saturated inflammation" and "moderate inflammation" states.

Claims

1. An image analysis method for determining a biomedical state of a tissue sample, the method being implemented by an image analysis system (100) and comprising: - receiving a digital image (118) of a tissue sample; - analyzing the received image to identify the number and location of type A cells and type B cells observed in the area of ​​the received image, wherein the type A cells and the type B cells are different cell types; - analyzing the locations of the type A and type B cells in the region to obtain an observed relative distribution (708, 706), the observed relative distribution indicating an observed distance between the type A cells and the type B cells in the region; - obtaining a reference relative distribution (710, 714) indicating an expected distance between a reference type A cell and a reference type B cell, wherein the expected distance is a distance expected under the assumption that the type A cells and / or the type B cells are distributed in an image region according to a specific predefined mathematical distribution; - calculating a closeness score as the difference between the reference relative distribution and the observed relative distribution; - calculating a combined score comprising the proximity score and comprising the density of the type A cells and / or the density of the type B cells; - using the combined score to determine the biomedical status of the tissue sample and / or outputting the combined score to enable a user to determine the biomedical status of the tissue sample.

2. The image analysis method according to claim 1, wherein obtaining the observed relative distribution comprises: -For each of said identified type A cells observed in said image region: a) selecting the type A cell as the center of a circle with a radius of 0; b) increasing the radius by one step to produce an enlarged circle; c) determining the number of type B cells contained in the circle generated in step b); d) storing the current radius of the circle associated with the number of type B cells determined in step b) in a storage medium, and repeating steps b), c) and d) until a termination criterion is reached; e) selecting one of the unselected type A cells, and continuing step a) with the newly selected type A cells until all type A cells are selected; - The observed association radius and number of type B cells are provided as the observed relative distribution.

3. The image analysis method according to claim 1 or 2, wherein the observed relative distribution is calculated as K i，j (r) or K i，j (r) derivative function, where K i，j (r) is calculated according to the following formula: - K i，j (r) is the binary Ripley K(t) function, - where i is the occurrence of object type "observed type A cells"; - where j is the occurrence of object type "observed B-type cells"; - where λ j is the density of observed B-type cells per image area; -Where X (i) is the total number of observed type A cells identified within the image area; -Where X j is the total number of observed B-type cells identified within the image area; - where u∈X (i) Indicates that the cells are observed type A cells; -where r is the radius of the step increase centered at the observed A-type cell; - where t is u, r and X j A function of , and counting the number of type B cells observed within a circle with a radius of r surrounding the observed type A cell u in the image region; -where "|u∈X (i) ” means “on all observed A-type cells u”; -where E is the expected value of t obtained over all u.

4. The method according to claim 1 or 2, wherein obtaining the reference relative distribution comprises: - computationally simulating a distribution of simulated reference type A cells in the region, the number of simulated reference type A cells being the same as the identified number of type A cells observed in the image region; - computationally simulating a distribution of simulated reference B-type cells in the region, the number of simulated reference B-type cells being the same as the identified number of B-type cells observed in the image region; - calculating the reference relative distribution as a function of the computationally simulated distribution of reference type A cells and reference type B cells, the reference relative distribution being indicative of the distance between the simulated reference type A cells and the simulated reference type B cells in the region. 5 . The image analysis method according to claim 4 , wherein the distribution of the simulated reference A-type cells is a Poisson distribution, and wherein the distribution of the simulated reference B-type cells is a Poisson distribution.

6. The image analysis method of claim 4, calculating the reference relative distribution as a function of the computationally simulated distribution of reference type A and reference type B cells comprises: -For each randomly distributed simulated reference A-type cell: a) selecting the simulated reference A-type cell as the center of a circle with a radius of 0; b) increasing the radius by one step to produce an enlarged circle; c) determining the number of simulated reference B-type cells contained in the circle generated in step b); d) storing the current radius of the circle associated with the number of simulated reference B-type cells determined in step b) in a storage medium, and repeating steps b), c) and d) until a termination criterion is met; e) selecting one of the simulated reference A-type cells that has not been selected, and continuing step a) using the newly selected simulated reference A-type cell until all simulated reference A-type cells are selected; -Providing the associated radius and number of simulated reference B-type cells as the reference relative distribution.

7. The image analysis method according to claim 4, wherein the reference relative distribution is calculated as K i，j (r) or K i，j (r) derivative function, where K i，j (r) is calculated according to the following formula: - K i，j (r) is the binary Ripley K(t) function, - where i is the occurrence of object type "Simulated Reference A-Type Cell"; - where j is the occurrence of object type "Simulated Reference B-cell"; - where λ j is the density per image area of ​​the simulated reference B-type cells; -Where X (i) is the total number of simulated reference A-type cells in the image region; -Where X j is the total number of simulated reference B-type cells in the image region; - where u∈X (i) Indicates that the cell is a simulated reference type A cell; -where r is the radius of the step increase centered on the simulated reference A-type cell; - where t is u, r and X j A function of , and counting the number of simulated reference B-type cells in a circle with a radius of r surrounding the simulated reference A-type cell u in the image region; -where "|u∈X (i) ” means “on all simulated reference A-type cells u”; -where E is the expected value of t obtained over all u.

8. The image analysis method according to claim 1 or 2, the calculation of the proximity score as a difference between the reference relative distribution and the observed relative distribution comprises: - providing a predetermined number representing a predetermined minimum number of type B cells; - Identify the observation radius r within the observed relative distribution o-min , where the observation radius r o-min is a radius that, if drawn around each of said observed type A cells, would define a circle that, on average, includes a predetermined number of said observed type B cells; - identifying a reference radius r within the reference relative distribution r-min , where the reference radius r r-min is a radius that, if drawn around each of said reference type A cells, would define a circle that, on average, includes a predetermined number of said reference type B cells to provide said reference relative distribution; - Calculated as the observation radius r o-min and the reference radius r r-min The proximity score is a function of the difference between the observation radius and the reference radius.

9. The image analysis method according to claim 1 or 2, further comprising: - graphically representing the observed relative distribution as an observed distribution curve in a 2D plot, a first dimension of which represents the radius r(Ro) and a second dimension of which represents the number of observed B-type cells associated with the radius r(Ro); and displaying the 2D plot on a display device of the image analysis system; and / or - graphically representing the reference relative distribution as a reference distribution curve in a 2D plot, a first dimension of which represents the radius r (Rs) and a second dimension of which represents the number of reference B-type cells associated with the radius r (Rs); and displaying the 2D plot on a display device of the image analysis system. 10 . The image analysis method according to claim 1 or 2 , wherein the type A cells are tumor cells, and the type B cells are immune cells.

11. The image analysis method according to claim 1 or 2, wherein the tissue sample is a tumor tissue sample, and determining the biomedical state of the tissue sample includes determining the infiltration state of the tumor tissue depicted in the area of ​​the received digital image having immune cells.

12. The image analysis method according to claim 1 or 2, further comprising: - graphically representing the combined score as symbols in a 2D score plot, wherein a first dimension of the 2D score plot represents the proximity score, and wherein a second dimension of the 2D score plot represents the density of the type B cells or the type A cells; and outputting the 2D score plot on a display of the image analysis system to enable a person to determine the biomedical status of the tissue sample; and / or -Determination of the biomedical state of the tissue sample is performed by the image analysis system by automatically identifying the biomedical state of the tissue sample within a limited set of predetermined biomedical states or within a predetermined continuous range of biomedical states, and outputting the identified biomedical state as the determined biomedical state.

13. The image analysis method according to claim 12, wherein: - said receiving a digital image of a tissue sample (118) comprises receiving a digital image of a tissue sample from each of a plurality of different patients; - said calculating a combined score comprises calculating a combined score for each of said patients using images received from tissue samples of each of said patients; And, the method further comprises: - graphically representing said biomedical state of each patient by a corresponding biomedical state specific symbol on a 2D score plot, the position of said symbol for each patient in said plot being dependent on said B-cell density and said proximity score calculated for said patient.

14. The image analysis method according to claim 1 or 2, wherein the identified biomedical state is selected from the group consisting of: - Inflammation, wherein said inflammation is indicative of an immune tissue state in which immune cells have a significantly increased cell density, which indicates that tumor tissue is heavily infiltrated with immune cells in all compartments of said tumor; - blocking, wherein the blocking indicates an immune tissue state in which immune cells are present in the tissue sample but are prevented from coming into close contact with tumor cells, whereby the immune cells are concentrated in the invasive margin and / or in the intratumoral stroma but separated from the tumor cells; - A desert, wherein the desert indicates an immune tissue state in which the cell density of immune cells in a tumor tissue region is zero or close to zero.

15. The image analysis method according to claim 1 or 2, further comprising: - in the case where the tumor infiltration type classified by the tissue sample is inflammation, outputting a treatment recommendation through a user interface to prescribe a drug that is a checkpoint inhibitor; - in the case where the tumor infiltration type classified by the tissue sample is blocking, outputting a treatment suggestion through the user interface to prescribe a drug suitable for attracting immune cells closer to tumor cells; - In case the tumor infiltration type of the tissue sample is classified as desert, outputting a treatment recommendation through the user interface to prescribe a drug suitable for generally strengthening the immune system.

16. The image analysis method according to claim 1 or 2, wherein the density of the reference A-type cells is the same as the density of the A-type cells observed in the image area, and wherein the density of the reference B-type cells is the same as the density of the B-type cells observed in the image area.

17. A method of determining the efficacy of a drug for treating a particular type of cancer, the method comprising: - receiving a plurality of first digital images, each first image depicting a tissue sample of a respective organism having a specific type of cancer before treatment of said organism with said drug; - calculating a first combined score from each image region of said first digital image, said calculation comprising: (a): analyzing the received image to identify the number and location of type A cells and type B cells observed in a region of the received image, wherein the type A cells and the type B cells are different cell types; (b): analyzing the positions of the type A and type B cells in the region to obtain an observed relative distribution, the observed relative distribution indicating an observed distance between the type A cells and the type B cells in the region; (c): obtaining a reference relative distribution, the reference relative distribution indicating an expected distance between a reference type A cell and a reference type B cell, wherein the expected distance is a distance expected under the assumption that the type A cells and / or the type B cells are distributed in an image region according to a specific predefined mathematical distribution; (d): calculating a proximity score as the difference between the reference relative distribution and the observed relative distribution; (e): calculating a combined score, the combined score comprising the proximity score and the density of the type A cells and / or the density of the type B cells; wherein the biomedical state of the organism is the immune cell infiltration state of a tumor, the type A cells are tumor cells, and the type B cells are cancer cells, and the first combined score is the combined score of step (e); - receiving a plurality of second digital images, each second image depicting a tissue sample of a respective organism having a specific type of cancer after treatment of said organism with said drug; - calculating a second combined score from the image area of ​​each of the second digital images, the calculating comprising performing steps (a) to (e) for each received second digital image, wherein the biomedical state of the organism is a tumor infiltration state, the type A cells are tumor cells, and the type B cells are cancer cells, and the second combined score is the combined score of step (e); - displaying a 2D score plot on a display device, wherein each first combined score is represented by a first symbol in the 2D score plot, wherein each second combined score is represented by a second symbol in the 2D score plot, wherein the positions of the first and second symbols in the plot depend on the immune cell density observed in the image region of the corresponding image and the proximity score calculated for the image region of the corresponding image, wherein first symbols and second symbols representing the same organism are visually linked in the 2D score plot for visualizing any offset in the x and / or y coordinates from any one of the first symbols to its corresponding linked second symbol.

18. A storage medium comprising computer-interpretable instructions, which, when executed by a processor, cause the processor to perform the image analysis method according to any one of claims 1 to 16.

19. An image analysis system (100) for determining a biomedical state of a tissue sample, the system comprising a processor and computer interpretable instructions, the instructions being configured to cause the processor executing the instructions to perform a method comprising: - receiving a digital image (118) of said tissue sample; - analyzing the received image to identify the number and location of type A cells and type B cells observed in the area of ​​the received image, wherein the type A cells and the type B cells are different cell types; - analyzing the positions of the type A and type B cells in the region to obtain an observed relative distribution indicating an observed distance between the type A cells and the type B cells in the region; - obtaining a reference relative distribution indicating an expected distance between the reference type A cells and the reference type B cells, wherein the expected distance is a distance expected under the assumption that the type A cells and / or the type B cells are distributed in an image region according to a specific predefined mathematical distribution; - calculating a closeness score as the difference between the reference relative distribution and the observed relative distribution; - calculating a combined score comprising the proximity score and comprising the density of the type A cells and / or the density of the type B cells; - using the combined score to determine the biomedical status of the tissue sample and / or outputting the combined score to enable a user to determine the biomedical status of the tissue sample.

20. A data processing system for determining a biomedical status of a tissue sample, the system comprising a spatial database management system and an application program, the spatial database management system comprising: - observation space data, the observation space data comprising locations of type A cells (604) and type B cells observed in a region of the digital image (118) of the tissue sample, wherein the type A cells and the type B cells are different cell types; - reference spatial data, the reference spatial data comprising the positions of reference type A cells and reference type B cells; The application is configured to instruct the data processing system to: - analyzing the observation spatial data to obtain an observation relative distribution indicating an observation distance between the type A cells and the type B cells in the region, wherein the application is configured to calculate the observation distance using spatial operations provided by the spatial database management system; - analyzing the reference spatial data to obtain a reference relative distribution indicating an expected distance between a reference type A cell and a reference type B cell in the region, wherein the expected distance is a distance expected under the assumption that type A cells and / or type B cells are distributed in the image region according to a specific predefined mathematical distribution, and wherein the application is configured to calculate the expected distance using spatial operations provided by the spatial database management system; - calculating a closeness score as the difference between the reference relative distribution and the observed relative distribution; - calculating a combined score comprising the proximity score and comprising the density of the type A cells and / or the density of the type B cells; - using the combined score to determine the biomedical status of the tissue sample, and / or outputting the combined score to enable a user to determine the biomedical status of the tissue sample.

Images