Threshold determination method

Abstract

To efficiently optimize a threshold used when determining a dyeing state from an image of a dyed biological specimen while using determination by a human.SOLUTION: A threshold determination method according to the present invention comprises: region specification steps (S201-S203) of specifying a cell region corresponding to individual cells in a processed image obtained by imaging a dyed biological specimen; determination steps (S205-S206) of temporarily setting a threshold for the image density of the processed image and comparing the density and the threshold for each cell region to determine whether the cell is positive or negative to dyeing; reception steps (S207-S211) of displaying a determination result of the dyeing state of each cell region and receiving the operation input of a user for changing the determination result for each cell region; and a re-setting step (S212) of re-setting the threshold according to the determination result after change.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a method for determining a threshold value when evaluating the staining state of a stained biological specimen by image analysis. [Background technology]

[0002] In the fields of medical and biological research, biological specimens such as tissue specimens and cell specimens are analyzed on a single cell basis in order to elucidate disease mechanisms, biological mechanisms, and drug action mechanisms. In these analyses, biological substances such as proteins and RNA (ribonucleic acid) are stained with various staining methods, and quantitative and qualitative analyses are performed by evaluating the staining state of cells, specifically, whether the cells are positive or negative for the staining. The evaluation results may contain artifacts such as noise during measurement and staining variations due to the staining method. Therefore, in order to perform biologically significant analyses, an analysis method that is not affected by such artifacts is required.

[0003] In flow cytometry, which is one method of single cell analysis, for example, isotype controls are used to judge nonspecific antigen-antibody reactions, and a threshold for determining whether a result is positive or negative is manually determined (see, for example, Patent Document 1). On the other hand, in image cytometry using tissue specimens, methods have been proposed for automatically determining a threshold from image information, for manually optimizing the threshold stepwise, and for reducing artifacts from data on control specimens or cells (see, for example, Patent Documents 2 and 3). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6187989 [Patent Document 2] Special Publication No. 2010-500573 [Patent Document 3] Special Publication No. 2019-513255 Summary of the Invention [Problem to be solved by the invention]

[0005] When tissue specimens are used as evaluation targets, unlike flow cytometry, which uses samples that have been previously separated into cell units, there are many causes of artifacts, and therefore optimizing the threshold often requires subjective judgment and adjustment by an expert. However, the number of cells contained in a tissue specimen is enormous, and the optimal threshold is not necessarily unique and may vary depending on the specimen and imaging conditions. Furthermore, in a situation where multiple immunohistochemical stainings (immunohistochemistry) are performed sequentially on the same specimen to analyze multiple biological substances (called multiple immunohistochemical staining), it is necessary to understand that there is a possibility of inconsistency in the results that would not occur with a single staining.

[0006] For these reasons, it cannot be said that the environment is currently sufficiently in place to efficiently carry out adjustment work when optimizing thresholds while incorporating human judgment.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that can efficiently optimize, while incorporating human judgment, a threshold value used when determining the staining state from an image of an immunohistochemically stained specimen. [Means for solving the problem]

[0008] One aspect of the present invention is a threshold determination method for evaluating a stained biological specimen, which, to achieve the above-mentioned object, includes an area identification step of identifying cellular areas corresponding to individual cells in a processed image of the biological specimen, a determination step of setting a predetermined threshold for the image density of the processed image and comparing the density of each of the cellular areas with the threshold to determine whether the staining state is positive or negative for the staining, a reception step of displaying the determination result of the staining state of each of the cellular areas and receiving user input for changing the determination result for each of the cellular areas, and a resetting step of resetting the threshold in accordance with the changed determination result.

[0009] In the invention thus configured, the staining state of each cellular region extracted from the processed image, specifically, whether the cellular region is positive or negative for staining, is automatically determined based on a preset threshold value. However, since the threshold value at this time is not necessarily the optimal value, the threshold value is optimized by reflecting the judgment of an expert.

[0010] Specifically, the system displays the results of the staining state judgment for each cell region based on a provisionally set threshold, and accepts user input to change the results. This allows the user to instruct the correct judgment for a cell region where an inappropriate judgment result was shown. By resetting the threshold according to the results thus instructed, the threshold can be corrected to approach the optimal value.

[0011] As an example of a method for optimizing the threshold based on human judgment, there is a method in which the threshold can be changed and set by the user, and the judgment results based on the set threshold are displayed and the user evaluates the validity of the results. However, with this method, since many cellular regions that are affected by the judgment results due to the change in the threshold appear in various places in the image, the task of checking the presence or absence of such changes and their validity and converging the threshold to the optimal value places a heavy burden on the user.

[0012] In the present invention, the user is not required to directly operate the threshold value, but rather inputs for correction of the displayed judgment result are accepted, and the threshold value is reset according to the result. Therefore, if the user finds a judgment result that requires correction, he or she only needs to point it out, and there is no need to check the results for all cell regions. This makes it possible to greatly improve the efficiency of the work, and reduces the burden on the user. Effect of the Invention

[0013] As described above, the present invention performs a staining state determination based on a provisionally set threshold for an image of a stained biological specimen and displays the result. Then, a change in the determination result due to an operation input from a user is accepted, and the threshold is reset accordingly. This makes it possible to efficiently optimize the threshold for determining the staining state while incorporating human judgment. [Brief description of the drawings]

[0014] [Figure 1] 1 is a flowchart showing a sample analysis method to which a threshold determination method according to the present invention is applied. [Diagram 2] FIG. 10 is a diagram illustrating an example of a specimen image. [Diagram 3] 10 is a flowchart showing a threshold value determination process according to the present embodiment. [Figure 4] FIG. 13 is a diagram showing an example of a cell region extraction result. [Diagram 5] FIG. 10 is a diagram illustrating an example of a determination result. [Figure 6] 13A and 13B are diagrams illustrating an example of an image showing a determination result and an example of modified input. [Figure 7] 1A and 1B are diagrams illustrating the principle of threshold determination processing using a combination of multiple specimen images. [Figure 8] FIG. 13 is a diagram showing an example of an interface screen when a plurality of determination results are combined. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] 1 is a flowchart showing an outline of a specimen analysis method to which the threshold determination method according to the present invention can be applied. This specimen analysis method is a method for performing so-called image cytometry, in which a tissue specimen subjected to multiple immunostaining is imaged and the image is analyzed to obtain various quantitative information.

[0016] First, a specimen image to be processed is collected. Specifically, a tissue specimen to be analyzed is stained with a predetermined staining method (step S101), and then imaged to obtain digital image data (step S102). After imaging, the tissue specimen is decolorized (step S103). This operation is repeated until imaging is completed for all of the multiple staining methods prepared in advance (step S104). In this way, a number of specimen images corresponding to the number of staining types are collected. These specimen images are images of the same tissue specimen stained differently from each other and then imaged.

[0017] Next, a portion of the tissue specimen is set as an analysis target region to be analyzed (step S105). Then, a partial image corresponding to the analysis target region is cut out from each specimen image as an analysis target image, and these analysis target images are aligned (registered) so that corresponding positions match each other (step S106).

[0018] Also, from each image to be analyzed, a cellular region corresponding to cells in tissue is extracted (step S107). Various methods are known for extracting a cellular region from an image, and such known methods can be appropriately selected and applied in this embodiment. For example, the image to be analyzed can be input to a learning model or artificial intelligence that has been trained in advance using various stained images as teacher images using an appropriate learning algorithm, thereby extracting a cellular region from the image. For example, a method of distinguishing between a cellular region and other regions by binarizing the image using an appropriate threshold value, or a method of identifying the outline of a cell by edge extraction can be adopted.

[0019] Depending on the type and condition of staining, the image to be analyzed may not be suitable for extracting the cell region. For example, an image to be analyzed in which a specific structure inside a cell, such as a cell nucleus, is selectively stained may not necessarily clearly show the outer shape of the cell, and may not be suitable for identifying the area occupied by each cell in the image. Even in such a case, it is possible to use the extraction result in an image stained with another staining method that clearly shows the area occupied by the cell. That is, since the registration process (step S106) enables alignment between the stained images, an area corresponding to the cell region extracted in one or more other stained images can be regarded as the cell region in the image to be analyzed.

[0020] In this manner, the identification of cellular regions in each image to be analyzed may be performed by a method in which the cellular regions are extracted directly from the image to be analyzed by image processing, or by an indirect identification method in which areas corresponding to individual cellular regions extracted using one or more images stained with another staining method are regarded as individual cellular regions in the image to be analyzed.

[0021] FIG. 2 is a diagram showing a schematic example of a specimen image. The specimen image Is is an image including, as a subject, at least a part of a tissue specimen S that has been collected from a living body or prepared by culturing and then stained by an appropriate staining method. The image can be captured, for example, by optical microscope imaging. One specimen image Is is acquired for each type of staining.

[0022] A partial image corresponding to at least a part of the region Ra to be analyzed is cut out from the specimen image Is as an image to be analyzed Ia. The image to be analyzed Ia contains images of multiple cells C that compose the tissue. Cell regions Rc corresponding to the cells C are extracted by appropriate image processing for region extraction. Here, the cells C are represented by circles of the same size for convenience, but in actual tissues, the size and shape of cells vary, and multiple types of cells may be included. The purpose of this processing is to calculate quantitative information of these cells C, such as their type, number, size, shape, density, distribution within the tissue, and the amount of a specific substance.

[0023] Returning to the explanation of FIG. 1, at least one quantitative value related to the staining concentration is calculated for each extracted cell region (step S108). Various values ​​that quantitatively represent the degree of staining concentration of the cell or the biological substance in the cell are used as the quantitative value. For example, the average concentration of the entire cell region, the maximum concentration or the average concentration of a region in the cell region where the color corresponding to the staining is particularly strong (e.g., the center or periphery of the cell), etc. can be used. This makes it possible to quantify the degree to which the staining has affected each cell region. Hereinafter, this value will be referred to as the "staining concentration."

[0024] For each specimen image, a threshold value for determining the staining state of the cells in the tissue for the stain is determined (step S109). The "staining state" here is an index that indicates whether or not the cells are stained with one type of stain. Here, when a cell or a biological substance in the cell is stained with the stain and the staining concentration is equal to or higher than a predetermined threshold, the cell is determined to be "positive" for the stain, whereas when the staining concentration is equal to or lower than the threshold, the cell is determined to be "negative" for the stain. Step S109 is a process for determining a threshold value for this purpose, and the threshold determination method of the present invention is applied here. The specific process contents will be described later.

[0025] Using the threshold thus determined, the staining state of each cell region is determined (step S110), and cytometry is then performed to obtain various quantitative information representing the properties of each cell region (step S111). The contents of the cytometry process and the type of quantitative information obtained thereby are arbitrary. Such information, together with an image of the tissue specimen, is provided for various pathological diagnoses and research. Thus, this embodiment belongs to image cytometry technology that performs analysis on a single cell basis without separating the tissue specimen into individual cells.

[0026] When cells in a tissue specimen are individually analyzed in this way, various artifacts that affect the judgment results, such as staining variations, image noise, and the influence of surrounding cells and background density, may be included. Therefore, even if a threshold is automatically determined using a known image processing method used for binarizing an image, the result is not necessarily the optimal value. In other words, erroneous judgments such as "false negatives" in which cells that should be judged as positive are judged as negative due to inappropriate setting of the threshold, and "false positives" in which cells that should be judged as negative are judged as positive may be included.

[0027] Therefore, whether the determined threshold value is appropriate or not still requires the subjective judgment of the user (expert). The threshold value determination process of this embodiment is designed to incorporate the judgment of the user while enabling the task to be performed efficiently.

[0028] FIG. 3 is a flowchart showing the threshold determination process of this embodiment. This process is for reducing the influence of artifacts by optimizing the threshold set for the staining concentration in order to determine the staining state (positive / negative) of a cell. This threshold determination process is performed for each of a plurality of types of staining. In other words, a threshold is set for each specimen image captured with one staining. In actual processing, it is possible to simultaneously perform threshold setting for several stains, but in the following, for the purpose of explaining the principle, first the process for determining a threshold for one staining will be described.

[0029] This processing can be realized by a processing device having an appropriate calculation function, such as a general-purpose computer device, executing a predetermined control program. The processing device for this purpose needs a calculation processing unit having a calculation function, a reception unit for receiving operation input from a user, and a display unit for displaying an image. For image processing other than imaging of a specimen, a processing device having a general hardware configuration, such as a personal computer, can be used, so a detailed description of the device configuration for this purpose will be omitted here.

[0030] In this process, first, a specimen image Is for each stain is acquired (step S201), an analysis target region Ra is set therein (step S202), and an analysis target image Ia is cut out. A cellular region Rc is extracted from the cut-out analysis target image Ia (step S203), and the staining concentration of each cellular region is calculated (step S204). These processes are basically the same as steps S101 to S108 in FIG. 1, and the results of steps S101 to S108 can be used as they are. That is, if the process in FIG. 1 has already started and the results of steps S101 to S108 can be used, steps S201 to S204 can be omitted. Conversely, steps S105 to S108 may be omitted, and steps S110 to S111 may be executed using the results of steps S201 to S204 executed in the threshold determination process in step S109.

[0031] FIG. 4 is a diagram showing a schematic example of the extraction result of a cell region. As shown in FIG. 4, the analysis target image Ia is divided into a cell region Rc corresponding to the cell C and the surrounding background region by appropriate image processing. By creating a mask that divides the cell region Rc from other regions in this way and applying it to the analysis target image Ia, it is possible to extract only the image content within the cell region Rc. The following quantification process is performed for each of these cell regions Rc.

[0032] Returning to FIG. 3, an appropriate threshold value is then provisionally set for the staining concentration (step S205), and the staining concentration, which is a quantitative value relating to the concentration of each cell region Rc previously obtained, is compared with the threshold value to determine the staining state of each cell, i.e., whether the cell is positive or negative for the staining (step S206). Specifically, the staining concentration obtained for each cell region Rc is compared with the set threshold value, and if the staining concentration is equal to or greater than the threshold value, the cell region Rc is determined to be positive for the staining. On the other hand, if the staining concentration is equal to or less than the threshold value, the cell region Rc is determined to be negative for the staining. When the staining concentration is equal to the threshold value, it may be determined to be positive or negative, as long as the criteria for determination are constant.

[0033] The threshold value at this time is a provisionally set initial value, and may be a value determined in advance or may be a value designated by a user's input. In addition, a threshold value calculated by analyzing an image using a known image processing algorithm used for binarizing an image, such as a discriminant analysis method or Otsu's method, may be applied as the initial value.

[0034] The judgment result is displayed on an appropriate display device, for example, a display screen of a computer device, and presented to the user (step S207). As shown in the following example, it is preferable that the judgment result is displayed by superimposing visual information representing the judgment result on the original analysis target image Ia. Also, the original analysis target image Ia that has not been processed and an image representing the judgment result may be displayed side by side.

[0035] FIG. 5 is a diagram showing an example of a determination result. FIG. 5(a) is a diagram showing an example of an analysis target image Ia. In a stained tissue specimen, each cell is stained to various densities, and in the analysis target image Ia obtained by capturing this, each cell region Rc has various brightness according to the staining density of the cell. The cells marked with symbols C1 to C12 in FIG. 5(a) are all cells stained to a certain degree or higher, and FIG. 5(b) is a graph showing these staining densities.

[0036] In addition, in FIG. 5 and the following figures, the circles represented by reference signs C1, etc. are, strictly speaking, regions in an image that have an image corresponding to the cells C in the specimen S, that is, cell regions Rc. However, for the sake of convenience in illustration and explanation, unless there is a particular need to distinguish, the cell regions in the image may also simply be referred to as "cells".

[0037] Here, cells with a concentration higher than the threshold value determined for the staining concentration are defined as "positive", and cells with a lower concentration are defined as "negative". FIGS. 5(c) and 5(d) schematically show examples of images in which image processing indicating cells determined to be positive by hatching is superimposed on the analysis target image Ia shown in FIG. 5(a).

[0038] When the threshold value is set to the value Th1 shown in FIG. 5(b), cells C1 to C3, C6, C8, C10, and C11 having a concentration higher than this are determined to be positive. FIG. 5(c) corresponds to this state, and hatching is applied to each of the cells listed here. Here, the left figure is an image Ia1 in which the determination result is superimposed on the original analysis target image Ia while leaving the concentration information of the cells, and the right figure shows only the cell contours and the determination result.

[0039] On the other hand, when the threshold value is set to another value Th2 (<Th1), in addition to the above, cells C7 and C12 are further determined to be positive. FIG. 5(d) shows this state. Also here, the left figure is an image Ia2 in which the determination result is superimposed on the cell concentration information, and the right figure shows only the cell contours and the determination result.

[0040] In this way, the determination result changes by changing the threshold value, but it is not always possible to say that the change is easily intuitively understandable by the user. That is, for example, when the user manually changes the set value of the threshold value, the determination result changes accordingly, but the change can appear in various forms at various positions within the image. Therefore, it is not easy for the user to check them one by one and determine whether the change in the threshold value is in the direction approaching the optimal value in a short time.

[0041] Therefore, in this embodiment, the judgment result for each cell is presented to the user by a screen display, and input for modification is accepted, and the threshold is recalculated to reflect the result of the modification. That is, for a cell whose judgment result is judged to be inappropriate and in need of modification, the user instructs the correct judgment, and the threshold is corrected. This allows the judgment of a user with specialized knowledge to be reflected in the threshold. In addition, instead of directly manipulating the threshold, the user only needs to look over the image to which the judgment results have been added and change the judgment of those that seem to require modification, allowing the user to make operational inputs based on more intuitive judgment.

[0042] Returning to Fig. 3, the specific process will be described. The image displayed in step S207 shows the result of the staining state (positive / negative) determination based on the current threshold. In this state, operation input from the user is accepted. Operation input can be accepted from an appropriate input device such as a keyboard, mouse, touch panel, etc.

[0043] If an operation input indicating that no correction is required is received (NO in step S208), the threshold at that time is confirmed as the optimal value (step S209), and the threshold determination process ends. On the other hand, if the user provides input for change, it is determined that correction is required (YES in step S208). In this case, the change input is accepted (step S210) until the user indicates his / her intention to complete the input (step S211). When the user completes the change input (YES in step S211), the threshold is reset to reflect the change (step S212).

[0044] The staining state of each cell is then judged again using the reset threshold (step S206), and the result is displayed (step S207). This allows the user to confirm the judgment result based on the newly set threshold. As described above, the judgment result can be changed as necessary, and the process of resetting the threshold accordingly can be repeated, allowing the threshold to approach an optimal value.

[0045] The method of displaying the judgment result, the specific manner in which the user can input changes, and the basic concept of resetting the threshold value will be described in more detail below. For ease of understanding, the example of the analysis target image Ia will be the same as that shown in Figure 5 above.

[0046] FIG. 6 is a diagram showing an example of an image showing the judgment result and a modification input. The image Ia3 shown in the upper part of FIG. 6(a) is an image in which hatching is added to the original analysis target image Ia as information showing the judgment result based on the initially set threshold value Th1. The shading of each cell region Rc in the figure represents the staining concentration of the corresponding cell. FIG. 6(b) is a graph of the staining concentration of each cell C1 to C12, and the staining concentrations are the same as those shown in FIG. 5(b).

[0047] Among the cell regions Rc, the hatched ones indicate that the cells corresponding to the cell regions are determined to be positive. The cell regions corresponding to the cells C1 to C3, C6, C8, C10, and C11, which have a staining density higher than the threshold value Th1, are hatched to indicate a positive determination. On the other hand, the cells not hatched indicate that the cells are determined to be negative. As can be seen from the figure, the cells determined to be negative may include not only cells that are not stained at all or almost not stained, but also cells that are affected by staining to some extent but whose density does not reach the threshold value.

[0048] The image displayed in step S206 may be an image in which information representing the determination result is superimposed on the original analysis target image Ia, as exemplified here, or may be an image representing only the determination result. In this case, it is more preferable to display the unprocessed analysis target image Ia and the image Ia3 representing the determination result side by side to facilitate the user's observation. Furthermore, the image may be an image in which the unprocessed analysis target image Ia and the image Ia3 with the determination result superimposed thereon are displayed side by side.

[0049] In this way, the user can input a change to the judgment result for the image in which a positive or negative judgment result has been assigned to each cell region. Specifically, the user can use the pointer or marker on the screen labeled with the symbol 61 in Fig. 6(a) to specify the cell for which the judgment result is to be changed. In this case, the cell region corresponding to cell C7 is specified. In this state, the user can express an intention to change the judgment by, for example, clicking the mouse.

[0050] If the specified cell was determined to be negative, the determination result is changed to positive. The lower part of Fig. 6(a) shows an example image Ia4 after the change input, in which cell C7 specified by the user is newly hatched. This indicates that the determination result has been changed from negative to positive. Conversely, if the specified cell was determined to be positive, the determination result is changed to negative. In this case, the hatching is removed from the changed image Ia4. In this way, the user can sequentially select cells that he or she wishes to change and change the determination result.

[0051] Such a change operation by the user can be considered as an instruction input by the user regarding the appropriateness of the judgment result. That is, for cells whose judgment has been changed, it can be said to be an instruction by the user indicating that the judgment result is an incorrect judgment. On the other hand, for cells whose judgment has not been changed, it can be said to be an instruction by the user that the automatically judged result is correct. However, it does not indicate that all judgment results after the change input are correct. This is because it may include incorrect judgments overlooked by the user or cases where the user's change itself is incorrect. In principle, it is practically impossible to appropriately judge the staining state of all cells with only a single threshold value. For this reason, it is preferable to reset the threshold value by regarding the results of the user's change operation, including those that have not been changed, as "probably correct".

[0052] In this sense, the reset threshold does not need to return a judgment result that is completely consistent with the judgment result after the user has made the change. In other words, the threshold may be adjusted so that the result of a re-judgment based on the new threshold approaches the judgment result after the user has made the change as close as possible. Specifically, it is sufficient that the deviation between the judgment result obtained when the judgment is performed based on the reset threshold and the judgment result after the user has made the change is smaller than the deviation between the previous judgment result performed with the threshold temporarily set and the judgment result after the user has made the change. This allows the judgment result based on the threshold to be closer to the user's subjectivity.

[0053] For example, the threshold can be changed to a smaller value if the number of cells changed from "negative to positive" is significantly greater than the number of cells changed from "positive to negative," and to a larger value if the opposite is true, thereby reflecting user instructions in the threshold.

[0054] Various methods can be used to calculate the threshold. For example, the staining density of cells whose judgment has been changed by the user can be statistically processed, and the average or median can be used as the new threshold. In this case, the staining density value of cells whose judgment has been changed from "negative to positive" and the staining density value of cells whose judgment has been changed from "positive to negative" can be treated separately, for example, by applying different weights to each of them and using them in the calculation.

[0055] Furthermore, the calculation may be performed taking into account the values ​​of the staining concentration of cells whose judgment results have not been changed. In this case, the cell concentrations of all cells may be used, or only the cell concentrations of some cells whose cell concentrations are close to the threshold may be used. In this way, by reflecting the numerical values ​​of cells whose judgment results have not been changed in the calculation, it is possible to limit the range of influence of the threshold change to only cells whose staining concentration is near the threshold and whose judgment results require fine adjustment, without significantly changing the judgment results that are currently considered to be clearly correct.

[0056] In addition, various automatic calculation methods such as dividing the area into two in the distribution of the staining density of each cell, for example, the above-mentioned Otsu's method, discriminant analysis method, and calculation methods using multivariate analysis or machine learning algorithms such as support vector machines (SVM), can be applied. In the case of machine learning methods, it is also possible to apply fine tuning, which performs additional learning using data changed by the user from an already-learned state.

[0057] By such recalculation, the threshold is changed from the initial value Th1 to a value Th3 as shown in FIG. 6(b). This value Th3 is a threshold for determining cell C7 as positive. Here, cell C7, which was originally determined to be negative, is treated as positive, while cell C12, whose determination as negative has not been changed, is set to a value higher than the staining concentration of cell C12 and lower than the staining concentration of cell C7 in order to maintain the determination result. If the determination of cell C12 has also been changed, the value will be lower than the staining concentration. Conversely, for example, if the positive determination of cell C10 has been changed to negative, the threshold is changed in the direction of determining cell C10 as negative, that is, to a larger value.

[0058] After resetting the threshold, the result of the re-determination based on the new threshold is displayed (steps S207, S208). At this time, it is preferable that visual information different from that of other cell regions is added to the cell region where a result different from the previous determination is obtained, in order to make it easier to find the cell. For example, the outline of the cell region can be emphasized, the cell region can be made to blink, or the previous determination result and the new determination result can be displayed alternately in time, thereby making it easier for the user to find the cell whose determination has changed. An image showing the previous determination result and an image showing the new determination result may be displayed side by side.

[0059] In addition, in the task of confirming and correcting the judgment results for individual cells, it is preferable that the displayed image is a high magnification image. On the other hand, in order to generally confirm the overall judgment results, a lower magnification image is preferable. Therefore, a high-magnification image and a low-magnification image, each of which has the judgment result superimposed thereon, may be displayed selectively or simultaneously. In this case, it is preferable that the field of view of the low-magnification image includes the entire field of view of the high-magnification image, in other words, the high-magnification image is an enlarged portion of the low-magnification image.

[0060] In the above, in order to make it easier to understand the principle of the threshold determination process, a method for determining a threshold corresponding to one type of staining from a specimen image has been described. In this case, the user must judge the appropriateness of the judgment result only from the position and concentration of the stained (or unstained) parts in the specimen image. For this reason, depending on the type and condition of the staining, even an experienced person may find it difficult to make a judgment. As will be explained next, in actual processing, multiple staining results can be presented in combination to make the threshold determination task more efficient.

[0061] Fig. 7 is a diagram showing the principle of threshold determination processing that combines multiple specimen images. Fig. 7(a) and Fig. 7(b) show images obtained by staining the same specimen with two different staining methods and imaging it. In the following explanation, the staining method corresponding to Fig. 7(a) will be referred to as "Staining A" and the staining method corresponding to Fig. 7(b) will be referred to as "Staining B".

[0062] For purposes of determining a threshold, it is preferable that these two staining methods do not stain the same cells, i.e., the same cell will not be positive for both stain A and stain B. Many such combinations are known, for example, proteins CD3, CD4, CD8, etc., known to be expressed in T cells, are not expressed in B cells, and conversely, proteins CD20, CD79a, etc., expressed in B cells, are not expressed in T cells.

[0063] By combining specimen images obtained by staining methods having such a relationship, it becomes possible to judge the appropriateness of the judgment result for one staining method by taking into account the judgment results for the other staining methods. For example, when stains that do not appear in the same cell are combined as described above, it can be said that for cells that are judged positive for both stains, at least one of the judgment results is highly likely to be inappropriate. Conversely, when only one of a combination of stains that should appear in the same cell is judged positive, it can also be considered that at least one of the judgment results is inappropriate. Such combinations have the potential to make it easier to find artifacts.

[0064] The specific content of the threshold determination processing method that embodies this technical idea will be described below. In image Ia5 shown in FIG. 7(a), cells stained with various densities by stain A are distributed mainly on the upper left side of the figure. On the other hand, in image Ia6 shown in FIG. 7(b), cells stained with stain B are distributed mainly on the lower right side of the figure. As described above, it is assumed that stain A and stain B do not appear together in one cell. Image Ia7 shown in the upper part of FIG. 7(c) is an image in which the staining state determination results performed by provisionally setting appropriate thresholds for stain A and stain B are superimposed on each other.

[0065] In this image Ia7, cells stained with a staining density equal to or greater than the threshold in both image Ia5 corresponding to stain A and image Ia6 corresponding to stain B are determined to be positive. However, cells C32 to C34 are determined to be positive in both stain A and stain B. Moreover, cells C31 and C35 are determined to be negative, although they have a certain staining density in image Ia5. Such inconsistencies in results are considered to be artifacts caused by inappropriate threshold settings.

[0066] By instructing such inconsistencies, the threshold can be readjusted. That is, when the user checks the judgment result and judges that correction is necessary, the user can specify the cell and instruct the correct judgment. Specifically, for example, as shown as the corrected image Ia8 in the lower part of FIG. 7(c), the judgment of cells C31 and C35 can be changed from negative to positive, and for cells C32 to C34 that are judged as positive by both staining A and staining B, one of the judgments can be changed based on the staining state and comparison with surrounding cells. In this example, the judgment of cell C32 by staining B is changed from positive to negative, and the judgment of cells C33 and C34 by staining A is changed from negative to positive. In this way, the inconsistencies in the judgment results at the provisionally set threshold can be eliminated.

[0067] It is desirable that the user can comprehensively carry out the confirmation and correction work by looking at the images Ia5 and Ia6 showing the staining results and the image Ia7 combining the individual judgment results. In this sense, it is preferable that the images Ia5 to Ia7 are displayed side by side on the same screen. Alternatively, at least two of these images Ia5 to Ia7 may be displayed superimposed on each other. While looking at the display screen, the user can determine which cells require correction based on the degree of staining of each cell and comparison with surrounding cells, etc.

[0068] For a staining method for which the judgment result has been changed, the threshold for that staining is recalculated and a new threshold is set. The specific method is as described above. If the judgment has not been changed, the current threshold for that staining can be confirmed as the optimal value. Note that, although it is possible to change the judgment for both staining A and staining B here, it is also possible to make it possible to change the judgment for only one type of staining, for example. For example, when the threshold for staining A has already been optimized, it is not preferable to change the judgment result for staining A in order to determine the threshold for staining B, and in this case, it is possible to prohibit the change for staining A.

[0069] Fig. 8 shows an example of an interface screen when multiple judgment results are combined and change input is accepted. As shown in Fig. 8(a), when the user designates a cell for which he / she wishes to check the judgment result with pointer 61 by operating the mouse or the like, the judgment result for that cell is displayed in a pop-up. In this example, cell C33 is designated, and in the pop-up screen P, the check boxes for both staining A and staining B are checked. This means that the judgment results for both staining A and staining B were positive.

[0070] As shown in Figure 8(b), the user can input an operation to change the judgment from positive to negative by unchecking the staining for which the judgment is to be changed. Conversely, to change from negative to positive, simply check the corresponding staining. Even if three or more stainings are combined, the judgment results are displayed in the pop-up screen P and can be changed in the same way.

[0071] Note that the above is merely an example, and the user interface for inputting changes is not limited to this. For example, as described above, when the judgment results are displayed together but change input is accepted only for staining B, the judgment may be changed when the cell region corresponding to the cell is clicked, as in the previous example.

[0072] Next, a modified example of the threshold determination process will be described. In the above embodiment, one threshold is set for a rectangular analysis target area Ra selected by a user from a specimen image Is. However, when a plurality of different structures are included in the tissue within the analysis target area, the staining density may differ even for the same type of cells. For example, this is the case when the tissue is divided into the cortex and the medulla.

[0073] In such a case, even if a single threshold value is set for the entire analysis target region Ra, it may not be possible to analyze the entire region well. Therefore, when multiple structures are included like this, it is desirable to further divide one analysis target region Ra into smaller regions and set a threshold value for each small region.

[0074] The region to be analyzed can be divided, for example, as follows. Various region division methods can be used as a method for automatically dividing the region by image processing. For example, the image in the region to be analyzed can be smoothed by low-pass filter processing, and the image can be divided into a plurality of regions based on the difference in density. As a specific method, for example, a method can be considered in which a region where stained cells exist at a high density is extracted from a specimen image in which a protein expressed by cells known to exist in a specific region is stained, and the image is divided into this region and other regions.

[0075] Alternatively, the system may be configured to receive an operational input from the user regarding region designation and perform division accordingly. For example, an experienced user may be able to distinguish structural differences from the shape and arrangement of cells using an image in which cell nuclei are stained. For example, a region may be designated by drawing a free-form curve on the image of the region to be analyzed displayed on the screen.

[0076] In the above embodiment, the analysis target region Ra is selected by the user, but the analysis target region Ra may be automatically determined from the specimen image Is. For example, the specimen image Is may be automatically divided into several blocks of the same size, and the block images may be prioritized and presented in order as the analysis target region Ra. In this case, for example, giving priority to blocks that may contain many artifacts as the analysis target region is effective in reducing the influence of artifacts.

[0077] For example, in a block in which the distribution of cells determined to be positive between two types of staining is similar, that is, in a block in which the distribution ranges overlap greatly, it is expected that a process of adjusting each threshold value while referring to each other's staining results will function effectively. Therefore, it is preferable to perform threshold optimization preferentially in such a block. Conversely, when there is little overlap in the distribution of cells determined to be positive between two types of staining, the block is likely to include a boundary where different cells or structures meet, and therefore it can be said to be suitable for adjusting the threshold value.

[0078] As described above, in the above embodiment, the tissue specimen S corresponds to the "specimen" of the present invention, and the specimen image Is and the analysis target image Ia captured from the tissue specimen S correspond to the "original image" and the "processed image", respectively, of the present invention. Also, for example, the images shown in Figures 5(c) and 6(a) correspond to the "result image" of the present invention.

[0079] In the threshold determination process (FIG. 3) of the above embodiment, steps S201 to S203 correspond to the "area specifying step" of the present invention, and steps S205 to S206 correspond to the "determination step" of the present invention. Steps S207 to S211 correspond to the "receiving step" of the present invention, and step S212 corresponds to the "resetting step" of the present invention. Step S207, which is executed after execution of step S212, corresponds to the "redisplay step" of the present invention.

[0080] The present invention is not limited to the above-mentioned embodiment, and various modifications can be made without departing from the spirit of the present invention. For example, in the above-mentioned embodiment, two types of staining results that are not expressed in the same cell are combined, so that the optimization of the threshold corresponding to each staining can be efficiently performed. However, the combination of multiple types of staining results is not limited to this and is arbitrary. For example, a specimen image obtained by a staining method that can stain specific cells or biological substances well, such as nuclear staining, can effectively function as a comparison target with specimen images obtained by other staining methods. Therefore, it is reasonable to combine such an image with another image and optimize the threshold for the other image.

[0081] Furthermore, the threshold determination process of the above embodiment is a process for optimizing the thresholds of stain A and stain B using specimen images and the respective judgment results for stain A and stain B. In other words, stain A and stain B are treated equally. However, for example, priorities may be set for multiple types of stains, and the thresholds may be optimized in order according to the priorities. For example, the results can be stabilized by first determining an optimal threshold for a stain that functions as a reference for setting thresholds for other stains, and then optimizing the thresholds for the other stains using the result.

[0082] For example, if stain A is used as a reference, the threshold for stain A can be optimized taking into account the results of stain B, stain C, etc., and then the threshold for other stains can be optimized based on the result. Note that there may be multiple types of stains used as references.

[0083] Furthermore, the user interface in the above embodiment is merely one example, and the method of presenting the determination result to the user and the method of accepting operational input from the user are not limited to the above, and any method can be used.

[0084] As described above by way of example of specific embodiments, in the threshold determination method according to the present invention, for example, in the reception step, a resultant image in which information representing the judgment result is superimposed on the processed image may be displayed. With such a configuration, an image that makes it easy for the user to confirm the result and input operations can be presented, and it becomes possible to efficiently determine the threshold. In this case, the processed image and the resultant image may be displayed on the same screen. By displaying the unprocessed processed image together with the judgment result, the user can always check the original image content.

[0085] Furthermore, for example, the biological specimen may be multiplexed immunohistochemically stained, in which case the reception step may be configured to display the determination results of the multiple immunohistochemical stains. With such a configuration, the user can confirm the results by comparing them with other staining results and determination results, making it possible to work more efficiently than when only the determination results of a single stain are presented.

[0086] In this case, the determining step may be configured to determine the staining state for each of the multiple types of immunohistochemical staining, the receiving step may be configured to receive an operation input for each of the multiple types of immunohistochemical staining, and the resetting step may be configured to reset the threshold for each of the multiple types of immunohistochemical staining. With this configuration, the optimization of the thresholds for the multiple types of staining can proceed in parallel.

[0087] Alternatively, for example, in the case where the biological specimen is subjected to multiple immunohistochemical staining, the determining step, the receiving step, and the resetting step may be executed for each of the multiple immunohistochemical stainings. For example, this corresponds to a process of determining thresholds for the multiple stainings one by one in order. With such a configuration, the thresholds are optimized from the highest priority among the multiple stainings, thereby making the work more efficient.

[0088] In the threshold determination method according to the present invention, it is preferable that in the resetting step, the threshold is reset so that the deviation between the judgment result obtained when the judgment step is executed based on the reset threshold and the judgment result after the change based on the operation input is smaller than the deviation between the judgment result in the previous judgment step executed based on the provisionally set threshold and the judgment result after the change based on the operation input. With this configuration, it is possible to obtain a judgment result closer to the user's subjective judgment by resetting the threshold.

[0089] Also, for example, the processed image may be divided into a plurality of regions, and a threshold value may be set for each of the plurality of regions. There may be cases where the staining state of the same cell differs depending on the state of the surrounding tissue or when a specific structure is formed. By setting a threshold value for each of the plurality of regions into which the processed image is divided, it becomes possible to make an appropriate judgment regardless of such differences in staining state.

[0090] Also, for example, the processed image may be a partial region cut out from an original image of a biological specimen, and a threshold may be set individually for each of a plurality of processed images that are located at different positions in the original image. Even between such processed images, differences in the staining state as described above may occur. Therefore, a similar effect can be obtained by setting a threshold individually for each processed image.

[0091] Also, for example, a result display step may be provided in which a determination step is executed based on the reset threshold value and a new determination result is displayed. With such a configuration, the determination result based on the reset threshold value is presented to the user, so that the user can verify the result of the adjustment.

[0092] Also, for example, the determination step, the acceptance step, and the re-setting step may be executed again based on the reset threshold value. With such a configuration, it is possible to make the threshold value closer to an optimum value by repeating the resetting of the threshold value, the verification thereof, and the readjustment of the threshold value based on the resetting. [Industrial Applicability]

[0093] This invention is applicable in the fields of medical and biological research to determining the staining state based on images of immunohistochemically stained tissue specimens, and is particularly suitable for analyzing multiple immunohistochemically stained specimens on a single-cell basis. [Explanation of symbols]

[0094] C1~C12,C31~C35 cells Ia Image to be analyzed (image to be processed) Is Specimen image (original image) Ra Analysis area Rc cell area S Tissue specimen (specimen) S201~S203 Area identification process S205~S206 Judgment process S207~S211 Reception process, redisplay process S212 Resetting process

Claims

1. 1. A method for determining a threshold for evaluating a stained biological specimen, comprising: a region identifying step of identifying a cell region corresponding to each cell in the processed image obtained by capturing the biological specimen; a determining step of provisionally setting a threshold value for the image density of the processed image, and comparing the density of each of the cellular regions with the threshold value to determine whether the staining state is positive or negative; a receiving step of displaying a determination result of the staining state of each of the cell regions and receiving a user's operation input for changing the determination result for each of the cell regions; a resetting step of resetting the threshold value according to the changed determination result; A method for determining a threshold value comprising:

2. The method according to claim 1 , wherein the receiving step displays a resultant image in which information representing the determination result is superimposed on the processed image.

3. 3. The method according to claim 2, wherein the processed image and the resultant image are displayed on the same screen.

4. 4. The method according to claim 1, wherein the biological specimen is subjected to multiple immunohistochemical staining, and the receiving step displays the determination results in a plurality of types of immunohistochemical staining.

5. In the determination step, the staining state is determined for each of a plurality of types of immunohistochemical staining, In the receiving step, the operation input is received for each of a plurality of types of immunohistochemical staining, The method of claim 4 , wherein the resetting step resets the threshold for each of a plurality of types of immunohistochemical staining.

6. The threshold determination method according to any one of claims 1 to 3, wherein the biological specimen is subjected to multiple immunohistochemical staining, and the determination step, the reception step, and the resetting step are performed for each of a plurality of types of immunohistochemical staining.

7. A threshold determination method as described in any one of claims 1 to 6, wherein in the resetting step, the threshold is reset so that the deviation between the judgment result obtained when the judgment step is executed based on the reset threshold and the judgment result after the change based on the operation input is smaller than the deviation between the judgment result in the previous judgment step executed based on the provisionally set threshold and the judgment result after the change based on the operation input.

8. 8. The method according to claim 1, further comprising the steps of: dividing the image to be processed into a plurality of regions; and setting the thresholds for the plurality of regions individually.

9. A threshold determination method according to any one of claims 1 to 8, wherein the processed image is a cut-out region of an original image of the biological specimen, and the threshold is set individually for each of a plurality of processed images having different positions in the original image.

10. 10. The method according to claim 1, further comprising a step of executing the determination step based on the reset threshold and displaying a new determination result.

11. 11. The method according to claim 1, further comprising the steps of: executing the determining step, the accepting step, and the resetting step again based on the reset threshold.

Images