IMAGE ANALYSIS METHOD, IMAGE ANALYSIS APPARATUS, AND COMPUTER PROGRAM FOR IMPLEMENTING IMAGE ANALYSIS METHOD

Abstract

This image analysis method comprises a calculating unit for calculating a colocalization value indicating colocalization between two cell objects of interest among a plurality of cell objects of interest, and an output unit for externally outputting output data obtained using the colocalization value. The calculating unit calculates the colocalization value by utilizing: a first computation including multiplying a first matching rate specified with respect to a first cell object of interest by a second matching rate specified with respect to a second cell object of interest; and a second computation including dividing a computation value obtained by the first computation by the square of a first distance, the first distance being the distance between the first cell object of interest and the second cell object of interest.

Description

[Technical field]

[0001] Disclosed herein is an image analysis method for analyzing a cell image. [Background technology]

[0002] In recent years, histopathological diagnosis or cytological diagnosis using an image analysis method has been performed (for example, JP2011-527056A). In this technology, image data obtained from a histopathological specimen, a cytological specimen, or the like is input into an image analyzer to obtain a cell classification result. Summary of the Invention [Problem to be solved by the invention]

[0003] Disclosed herein is a technique for not only classifying cells but also quantifying and outputting colocalization between cells.

[0004] The image analysis method disclosed in the present specification includes an acquisition step of acquiring cell image data obtained from an organism, and an analysis step of performing an image matching analysis on the cell image data to identify a plurality of target cell objects that match any one of one or more predetermined types of cells from among the cell images represented by the cell image data, in which the image matching analysis identifies, for each of the plurality of target cell objects, the position of the target cell object and the matching rate of the target cell object, and a calculation step of calculating a colocalization value indicating colocalization between two target cell objects of the plurality of target cell objects, and an output value obtained using the colocalization value. The method may also include an output process for outputting force data to the outside, and the calculation process may include calculating the colocalization value using a predetermined formula representing a first operation including multiplying a first matching rate identified for a first target cell object of the two target cell objects by a second matching rate identified for a second target cell object of the two target cell objects, and a second operation including dividing a calculated value obtained from the first operation by the square of a first distance, wherein the first distance is the distance between a first position identified for the first target cell object and a second position identified for the second target cell object.

[0005] According to the above configuration, each of the multiple target cell objects in the cell image is classified as a matching rate of the target cell object by the image matching analysis. Furthermore, a colocalization value indicating the colocalization between two target cell objects is calculated using the matching rate determined by the image matching analysis, and output data obtained using the colocalization value is output. In addition to classifying cells, the colocalization between cells can be quantified and output. In particular, the colocalization value can be calculated taking into account the phenomenon that the degree of colocalization between cells changes inversely proportional to the square of the distance.

[0006] Another image analysis method disclosed in the present specification includes an acquisition step of acquiring cell image data obtained from an organism, and an analysis step of performing an image matching analysis on the cell image data to identify a plurality of target cell objects that match any one of one or more predetermined types of cells from among the cell images represented by the cell image data, in which the image matching analysis identifies, for each of the plurality of target cell objects, the position of the target cell object and a matching rate of the target cell object; a calculation step of calculating a colocalization value indicating colocalization between at least three target cell objects among the plurality of target cell objects; and an output step of outputting output data obtained using the colocalization value to the outside, in which the calculation step includes calculating a first matching value identified for a first target cell object among the at least three target cell objects. and calculating the colocalization value using a predetermined formula representing a first operation including multiplying a matching rate determined for a first target cell object among the at least three target cell objects by a second matching rate determined for a second target cell object among the at least two target cell objects by a third matching rate determined for a third target cell object among the at least two target cell objects; and a second operation including dividing a calculated value obtained by the first operation by a first distance, a second distance, and a third distance, wherein the first distance is a distance between a first position determined for the first target cell object and a second position determined for the second target cell object, the second distance is a distance between the second position and a third position determined for the third target cell object, and the third distance is a distance between the third position and the first position.

[0007] According to such a configuration, it is possible to calculate not only the colocalization value between two cells, but also the colocalization value between three or more cells.

[0008] An image analysis device for implementing the above image analysis method is also novel and useful. Also novel and useful are a computer program for controlling the image analysis device and a computer-readable medium storing the computer program. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 shows a block diagram of an image analysis device. [Diagram 2] 1 shows an example of a cell image. [Diagram 3] 1 shows an example of an image resulting from image matching analysis. [Figure 4] An example of an image of the results of colocalization analysis is shown. [Diagram 5] Illustrates the concept of colocalization between cells. [Figure 6] 1 shows how to calculate colocalization between two types of cells. [Figure 7] The method for calculating colocalization between three types of cells is shown.

[0010] (First embodiment) (Image analysis device configuration; Fig. 1) 1 shows the configuration of an image analyzing device 10. The image analyzing device 10 includes an operation unit 12, a display unit 14, an input unit 16, and a control unit 30. The units 12 to 30 are connected to a bus line (reference numbers omitted). The operation unit 12 includes, for example, a mouse, a keyboard, and the like. A user can give various instructions to the image analyzing device 10 by operating the operation unit 12. The display unit 14 is a display for displaying various information.

[0011] The input unit 16 is a device for inputting cell image data representing a cell image including a plurality of cell objects to the image analyzing device 10. The input unit 16 may be a communication interface for performing wired or wireless communication, or may be a memory interface into which a USB memory or the like is inserted. For example, the input unit 16 may perform wired or wireless communication with a device that stores cell image data captured by a microscope, a Whole Slide Image, a virtual slide, or the like, and receive the cell image data from the device, thereby inputting the cell image data to the image analyzing device 10. Also, for example, the input unit 16 may read the cell image data from a memory that stores the cell image data, thereby inputting the cell image data to the image analyzing device 10.

[0012] The control unit 30 includes a CPU 32 and a memory 34. The CPU 32 executes various processes according to programs 40 and 42 stored in the memory 34. The memory 34 stores an OS program 40 for implementing basic operations of the image analysis device 10, and an analysis program 42 for executing image analysis according to a convolutional neural network (hereinafter referred to as "CNN (abbreviation of Convolutional Neural Network)"). For example, the image analysis device 10 is realized by installing the analysis program 42 in a general-purpose PC, server, etc. Here, the image analysis according to CNN is a concept that also includes image analysis according to a large-scale network having a CNN as a partial structure (for example, GoogLeNet (registered trademark), Residual Network, etc.).

[0013] The memory 34 further stores model parameters 50 of a CNN that performs matching with a plurality of predetermined types of cells (e.g., lymphocytes, macrophages, carcinomas, etc.). In a modified example, the memory 34 may store model parameters 50 of a CNN that performs matching with one predetermined type of cell.

[0014] The model parameters 50 are, for example, data provided by a seller who sells the analysis program 42. The model parameters 50 are stored in the memory 34, for example, when the analysis program 42 is installed. The model parameters 50 may also be data obtained by updating initial model parameters stored in the memory 34 when the analysis program 42 is installed, through machine learning. The machine learning may be performed by the seller or by the user.

[0015] (Image analysis; Figures 2 to 4) With reference to FIG. 2 to FIG. 4, an image analysis method (i.e., image analysis processing) that the CPU 32 realizes according to the analysis program 42 will be described. The image analysis method includes a step of performing an image matching analysis and a step of performing a colocalization analysis. The image matching analysis is a process of executing a CNN using the model parameters 50 to identify M1 target cell objects that match any of a predetermined number of types of cells from a cell image represented by cell image data input to the input unit 16. "M1" indicates an integer of 2 or more. In the image matching analysis, for each of the M1 target cell objects, the position of the target cell object and the matching rate of the target cell object are identified. The matching rate of the target cell object indicates the probability that a cell represented by the target cell object in the cell image matches any of a number of types of cells. For example, for each of a number of types of cells, the probability that a cell represented by the target cell object matches with the cell of that type is calculated, and the maximum probability among the multiple probabilities with the multiple types of cells is determined as the matching rate of the target cell object. For example, if the probability with a first type of cell (e.g., lymphocyte) is 70%, the probability with a second type of cell (e.g., an image of a macrophage) is 20%, and the probability with a third type of cell (e.g., an image of a carcinoma) is 10%, the matching rate of the target cell object is determined to be the maximum probability with the first type of cell, 70%. Note that, for example, if the maximum probability among multiple probabilities with multiple types of cells does not exceed a predetermined threshold (e.g., 50%), the cell represented by the corresponding target cell object may not be identified as any of the multiple types of cells.

[0016] For example, the CPU 32 executes an image matching analysis on the cell image data corresponding to the cell image IM1 in FIG. 2. FIG. 3 shows an example of an image IM2 resulting from the image matching analysis on the area indicated by the dashed line III in FIG. 2. The image IM2 is displayed by the display unit 14. The image IM2 includes three types of matching pins MP1 to MP3. The matching pin MP1 indicates a target cell object identified as, for example, a lymphocyte. Note that in FIG. 3, one of the multiple matching pins MP1 is given a reference symbol, and the reference symbols of the other matching pins MP1 are omitted. Reference symbols are also omitted for the matching pins MP2 and MP3.

[0017] The data corresponding to the matching pin MP1 includes coordinate values ​​indicating the position of the target cell object indicated by the matching pin MP1, and a matching rate of the target cell object. As shown in Fig. 3, the matching pin MP1 is located at a position indicated by coordinate values ​​in the data corresponding to the matching pin MP1. Similarly, the data corresponding to each of the matching pins MP2 and MP3 includes coordinate values ​​and a matching rate. For example, the matching pin MP2 indicates a target cell object identified as a macrophage, and the matching pin MP3 indicates a target cell object identified as a carcinoma.

[0018] Next, FIG. 4 shows an example of an image IM3 as a result of the colocalization analysis using the result of the image matching analysis. The image IM3 is displayed by the display unit 14. The image IM3 is an image in which a plurality of value pins are superimposed on the cell image IM1 of FIG. 2. The value pin indicates a colocalization value I between M2 target cell objects present in a region of interest including the position where the value pin is located. The colocalization value I is a value that quantifies the colocalization between cells. For example, as shown in the lower left of FIG. 5, the colocalization value I is indicated by the color of the value pin. The darker the color of the value pin, the higher the colocalization value I.

[0019] For example, a user of the image analysis device 10 can classify cells in the specimen shown by the cell image IM1 by looking at the image IM2 in FIG. 3. Furthermore, a user can distinguish between a location where colocalization between cells is likely to occur and a location where colocalization between cells is unlikely to occur in the specimen shown by the cell image IM1 by looking at the image IM3 in FIG. 4. For example, a user can determine the presence or absence of interaction between cells from the degree of colocalization. By displaying not only the image IM2 but also the image IM3, not only the classification of cells but also the quantification and output of colocalization between cells can be performed. This can assist the user in histopathological diagnosis or cytological diagnosis.

[0020] (Concept of cell-cell colocalization; Figure 5) Quantification of colocalization between cells is expressed by the following formula:

[0021]

number

[0022] Here, I indicates the value that quantifies colocalization. 1 indicates the matching rate of one of the two cells, and P 2 indicates the matching rate of the other cell out of the two cells. P 1 , P 2 is determined by image matching analysis. r indicates the distance between two cells. r is calculated using the coordinate values ​​of the cells determined by image matching analysis. kc indicates a predetermined coefficient.

[0023] As shown in Fig. 5, the two cells may be two cells C1 of the same type. Alternatively, the two cells may be two cells C1, C2 of different types. The cell C1 corresponds to a target cell object determined in the image matching analysis to match a first type of cell (e.g., lymphocyte) among a plurality of predetermined types of cells. The cell C2 corresponds to a target cell object determined in the image matching analysis to match a second type of cell (e.g., carcinoma) different from the first type of cell among a plurality of types of cells.

[0024] (Calculation method of colocalization value; Figure 6) A method for calculating the colocalization value of a value pin in the colocalization analysis will be described. The CPU 32 scans the entire cell image IM1 in a specific region of interest. Here, the specific region of interest is, for example, the region indicated by the dashed line III in FIG. 2, and is a square of N×N pixels as shown in FIG. 3. The value of N is set by, for example, a user. In addition, the specific region of interest is not limited to a square, and may be, for example, a rectangle.

[0025] For example, assume that in scanning the cell image IM1, a particular region of interest is located at the dashed line III in Fig. 2. In this case, M2 target cell objects are represented by M2 matching pins in Fig. 3. "M2" represents an integer equal to or less than M1.

[0026] 6 is a schematic diagram of M2 target cell objects in a specific region of interest. FIG. 6 shows a case where M2 cells represented by M2 target cell objects include a first type of cell C1 and a second type of cell C2, but do not include other types of cells. In this case, the CPU 32 calculates the colocalization value I between different types of cells in the M2 target cell objects in the specific region of interest using the following formula: 2 Calculate.

[0027]

number

[0028] Pi denotes the matching rate of the i-th first-type cell C1, and P j indicates the matching rate of the jth second type cell C2. i and j are integers equal to or greater than 1. In this case, M2 is the sum of the maximum value of i and the maximum value of j. r ij denotes the vector from the i-th cell C1 to the j-th cell C2, and r ji indicates a vector from the jth cell C2 to the ith cell C1. The coefficient kc is determined in advance, for example, taking into consideration the arrangement of the M2 target cell objects. As shown in the above formula (2), the CPU 32 calculates the value I shown in the above formula (1) for all combinations of the M2 target cell objects (i.e., groups of objects) in a specific region of interest, and determines the colocalization value I in the specific region of interest as the sum of all the calculated values ​​I. 2 Calculate.

[0029] (Calculation method of colocalization value; Figure 7) 7 shows a case where M2 cells represented by M2 target cell objects include a first type of cell C1, a second type of cell C2, and a third type of cell C3, but do not include other types of cells. In this case, the CPU 32 calculates the colocalization value I between different types of cells in the M2 target cell objects in a specific region of interest using the following formula: 3 Calculate.

[0030]

number

[0031] P k indicates the matching rate of the k-th third type cell C3. k is an integer equal to or greater than 1. In this case, M2 is the sum of the maximum value of i, the maximum value of j, and the maximum value of k. r jk denotes the vector from the jth cell C2 to the kth cell C3, and r ki denotes the vector from the kth cell C3 to the ith cell C1. This formula (3) gives the colocalization value I 3 is calculated.

[0032] According to the configuration of the present embodiment, as shown in the above formula (1), the colocalization value can be calculated by considering the phenomenon that the degree of colocalization between cells changes inversely proportional to the square of the distance. Also, as shown in the above formulas (2) to (3), the colocalization value between cells can be calculated by considering all of the M2 target cell objects in a specific region of interest. Also, as shown in the above formulas (2) to (3), not only the colocalization value between two types of cells but also the colocalization value between three types of cells can be calculated. Although not shown, the colocalization value between four or more types of cells can be calculated by expanding the above formula (3). Here, in the above formulas (2) to (3), the colocalization value between M2 cells is calculated by considering the colocalization between different types of cells in a specific region of interest. In a modified example, the colocalization value between M2 cells may be calculated by considering not only the colocalization between different types of cells in a specific region of interest but also the colocalization between the same type of cells (for example, the colocalization between the i-th cell C1 and the i+1-th cell C1).

[0033] (Correspondence) The image analysis device 10, the CPU 32, the memory 34, and the analysis program 42 are examples of an "image analysis device," a "computer," a "memory," and a "computer program," respectively. A plurality of predetermined types of cells, such as lymphocytes, macrophages, and carcinomas, are examples of "one or more predetermined types of cells." The CPU 32 that controls the input unit 16, the CPU 32 that performs the image matching analysis, the CPU 32 that performs the colocalization analysis, and the CPU 32 that controls the display unit 14 are examples of an "acquisition unit," an "analysis unit," a "calculation unit," and an "output unit," respectively. Data representing the cell image IM1 in FIG. 2 is an example of "cell image data." Data representing the image IM3 in FIG. 4 is an example of "output data."

[0034] In FIG. 6, a target cell object representing a cell C1 and a target cell object representing a cell C2 are examples of a "first target cell object" and a "second target cell object", respectively. i , P j, are examples of the "first matching rate" and the "second matching rate", respectively. ij and r ji is an example of the "first distance". 2 is an example of a "colocalization value." Formula (2) and kc are examples of a "predetermined formula" and a "coefficient," respectively.

[0035] In FIG. 7, the target cell object representing cell C1, the target cell object representing cell C2, and the target cell object representing cell C3 are examples of the "first target cell object," the "second target cell object," and the "third target cell object," respectively. i , P j , P k are examples of the "first matching rate", "second matching rate", and "third matching rate", respectively. ij , r jk , r ki are examples of the "first distance", "second distance", and "third distance", respectively. 3 is an example of a "colocalization value." Formula (3) and kc are examples of a "predetermined formula" and a "coefficient," respectively.

[0036] For example, the calculation using formula (2) is P i and P j The multiplication result of r ij andr ji This may be achieved by dividing the result of multiplication by P i r ij The calculated value multiplied by P j r ji Generally speaking, the "calculation unit" may calculate the colocalization value by using a "predetermined formula", and the method and order of calculation are not particularly limited.

[0037] Although the embodiments have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and changes of the specific examples exemplified above.

[0038] (Modification 1) In the above embodiment, the image matching analysis is performed using CNN or the like, and model parameters 50 of the CNN or the like are generated by machine learning. In a modification, the image matching analysis may be realized by image processing using predetermined image data representing predetermined cells, without using CNN or the like. In this modification, the image analyzing device 10 may store, instead of the model parameters 50, the predetermined image data used in the image processing.

[0039] (Variation 2) The "output data" is not limited to the value pins in FIG. 4, but may be character strings indicating co-localized values, contour lines connecting identical co-localized values, or the like.

[0040] (Modification 3) The above formulas (2) and (3) are merely examples, and may not include the coefficient kc, for example.

[0041] In addition, the technical elements described in this specification or drawings exert technical utility alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. In addition, the technologies exemplified in this specification or drawings achieve multiple objectives simultaneously, and achieving one of those objectives is itself technically useful. [Explanation of symbols]

[0042] 10: image analysis device, 12: operation unit, 16: input unit, 30: control unit, 32: CPU, 34: memory, 40: OS program, 42: analysis program, 50: model parameters, C1 to C3: cells, IM1: cell image, IM2, IM3: images, MP1 to MP3: matching pins

Claims

1. A computer-implemented method for image analysis, comprising: An acquisition step of acquiring cell image data obtained from an organism; an analysis step of performing an image matching analysis on the cell image data to identify a plurality of target cell objects that match any one of one or more predetermined types of cells from among the cell images represented by the cell image data, the image matching analysis identifying, for each of the plurality of target cell objects, the position of the target cell object and the matching rate of the target cell object; a calculating step of calculating a colocalization value indicative of colocalization between two target cell objects of the plurality of target cell objects; and an output step of outputting output data obtained using the colocalization value to an outside, The calculation step includes: a first operation including multiplying a first matching rate determined for a first one of the two target cell objects by a second matching rate determined for a second one of the two target cell objects; a second operation including dividing a calculated value obtained by the first operation by the square of a first distance, the first distance being a distance between a first position identified for the first target cell object and a second position identified for the second target cell object; Calculating the colocalization value using a predetermined formula Image analysis methods.

2. A method for image analysis carried out by a computer, comprising: An acquisition step of acquiring cell image data obtained from an organism; an analysis step of performing an image matching analysis on the cell image data to identify a plurality of target cell objects that match any one of one or more predetermined types of cells from among the cell images represented by the cell image data, the image matching analysis identifying, for each of the plurality of target cell objects, the position of the target cell object and the matching rate of the target cell object; calculating a colocalization value indicative of colocalization between at least three of the plurality of target cell objects; and an output step of outputting output data obtained using the colocalization value to an outside, The calculation step includes: a first operation including multiplying a first matching rate determined for a first target cell object of the at least three target cell objects, a second matching rate determined for a second target cell object of the at least three target cell objects, and a third matching rate determined for a third target cell object of the at least three target cell objects; a second calculation including dividing a calculation value obtained by the first calculation by a first distance, a second distance, and a third distance, the first distance being a distance between a first position identified for the first target cell object and a second position identified for the second target cell object, the second distance being a distance between the second position and a third position identified for the third target cell object, and the third distance being a distance between the third position and the first position; Calculating the colocalization value using a predetermined formula Image analysis methods.

3. 3. An image analysis device for implementing the image analysis method according to claim 1.

4. A computer program for implementing the image analysis method according to claim 1 or 2.

Images