Classification assistance device, classification assistance method, and classification assistance program

Abstract

A classification assistance device comprises: a division unit that divides an input image, which includes a specimen being a subject that includes a plurality of cells, into a plurality of partial images, each including a plurality of cells as a subject; a patch classification unit that receives input of an image including a plurality of cells as a subject, inputs the partial images to a trained model trained to estimate whether the image includes malignant cells, and classifies whether a partial image includes malignant cells, on the basis of the estimation result of the trained model; a detection unit that detects cells from a partial image, among the plurality of partial images obtained by division by the division unit, that is classified by the patch classification means as including malignant cells; and a cell classification unit that classifies whether the cells detected by the detection unit are malignant cells. According to this classification assistance device, a user, such as a medical worker, can be assisted in making decisions.

Description

Classification Support Device, Classification Support Method, and Classification Support Program

[0001] The present disclosure relates to a classification support device, a classification support method, and a classification support program.

[0002] Techniques for performing benign and malignant classification of cells, tissues, etc. using pathological images are known. For example, in Patent Document 1, a system that outputs classification information indicating whether the tissue in each local region of a pathological image is abnormal based on input information that is a pathological image including a tissue and a non-tissue that is other than the tissue, or processed information obtained by processing the pathological image, is disclosed.

[0003] International Publication No. 2022 / 049663

[0004] In benign and malignant classification, for example, when a false positive occurs where it is classified as malignant even though it is actually benign, a reexamination becomes necessary even though it should not be. Also, when a false negative occurs where it is classified as benign even though it is actually malignant, there is a risk of overlooking cancer. In order to reduce the burden on patients of reexaminations and oversights, higher accuracy is preferable in benign and malignant classification.

[0005] The present disclosure has been made in view of the above problems, and an exemplary object thereof is to provide a technique for improving the accuracy of benign and malignant classification of specimen cells in pathological diagnosis.

[0006] A classification support device according to an exemplary aspect of the present disclosure includes a dividing unit that divides an input image including a specimen containing a plurality of cells as a subject into a plurality of partial images each including a plurality of cells as a subject, a patch classification unit that inputs an image including a plurality of cells as a subject, inputs the partial image to a learned model that is learned to estimate whether the image contains malignant cells, and classifies whether the partial image contains malignant cells based on the estimation result of the learned model, a detection unit that detects cells from the plurality of partial images obtained by the division of the dividing unit and classified as containing malignant cells by the patch classification unit, and a cell classification unit that classifies whether the cells detected by the detection unit are malignant cells.

[0007] A classification support method relating to an exemplary aspect of the present disclosure includes: a division process in which at least one processor divides an input image, which includes a sample containing multiple cells as its subject, into a plurality of partial images, each containing multiple cells as its subject; a patch classification process in which the at least one processor takes an image containing multiple cells as its subject as input, inputs the partial images to a trained model that has been trained to estimate whether the image contains malignant cells, and classifies whether the partial images contain malignant cells based on the estimation results of the trained model; a detection process in which the at least one processor detects cells from the partial images that have been classified as containing malignant cells by the patch classification process among the plurality of partial images obtained by the division process; and a cell classification process in which the at least one processor classifies whether the cells detected in the detection process are malignant cells.

[0008] An exemplary classification support program relating to this disclosure is a classification support program for causing a computer to function as a classification support device, wherein the computer functions as: a division means for dividing an input image, which includes a sample containing multiple cells as its subject, into a plurality of partial images, each containing multiple cells as its subject; a patch classification means for inputting the partial images to a trained model that has been trained to take an image containing multiple cells as its subject as input and estimate whether the image contains malignant cells, and classifying whether the partial images contain malignant cells based on the estimation results of the trained model; a detection means for detecting cells from the plurality of partial images obtained by the division by the division means, specifically from the partial images classified by the patch classification means as containing malignant cells; and a cell classification means for classifying whether the cells detected by the detection means are malignant cells.

[0009] One illustrative aspect of this disclosure demonstrates the potential to provide a technique that improves the accuracy of classifying specimen cells as benign or malignant in pathological diagnosis.

[0010] This is a block diagram showing the configuration of the classification support device related to this disclosure. This is a flowchart showing the flow of the classification support method related to this disclosure. This is a block diagram showing the configuration of the classification support device related to this disclosure. This is a flowchart showing the flow of the classification support method related to this disclosure. This is a diagram showing the contents of each process included in the classification support method related to this disclosure. This is a diagram for explaining the learning method related to this disclosure. This is a table showing an example of the classification of the first benign subclass group and the first malignant subclass group related to this disclosure. This is a table showing an example of the classification of the second benign subclass and the second malignant subclass related to this disclosure. This is a table showing an example of the classification of benign cells and malignant cells related to this disclosure. This is a block diagram showing the configuration of a computer that functions as a classification support device related to this disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining some or all of the technologies (things or methods) employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, the effects mentioned in each of the exemplary embodiments shown below are examples of effects that can be expected in that exemplary embodiment and do not define the scope of the present invention. That is, embodiments that do not produce the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form for each of the exemplary embodiments described later. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur. Furthermore, each technology shown in the drawings referenced to explain this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur.

[0013] (Configuration of the Classification Support Device) The configuration of the classification support device 1 will be explained with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the classification support device 1. As shown in Figure 1, the classification support device 1 comprises a division unit 11, a patch classification unit 12, a detection unit 13, and a cell classification unit 14. The division unit 11 divides an input image, which includes a sample containing multiple cells as its subject, into multiple partial images, each containing multiple cells as its subject. The patch classification unit 12 takes an image containing multiple cells as its subject as input and inputs the partial images to a trained model that has been trained to estimate whether malignant cells are included in the image. Based on the estimation results of the trained model, the patch classification unit 13 classifies whether malignant cells are included in the partial images. The detection unit 13 detects cells from the partial images that have been classified by the patch classification unit 12 as containing malignant cells, among the multiple partial images obtained by the division unit 11. The cell classification unit 14 classifies whether the cells detected by the detection unit 13 are malignant cells.

[0014] (Effects of the Classification Support Device) As described above, the classification support device 1 employs a configuration comprising: a division unit 11 that divides an input image containing a sample with multiple cells as the subject into multiple partial images, each containing multiple cells as the subject; a patch classification unit 12 that takes an image containing multiple cells as the subject as input, inputs the partial images to a trained model that has been trained to estimate whether malignant cells are contained in the image, and classifies whether malignant cells are contained in the partial images based on the estimation results of the trained model; a detection unit 13 that detects cells from the partial images that have been classified by the patch classification unit 12 as containing malignant cells among the multiple partial images obtained by the division unit 11; and a cell classification unit 14 that classifies whether the cells detected by the detection unit 13 are malignant cells. Therefore, the classification support device 1 has the effect of improving the accuracy of classifying sample cells as benign or malignant in pathological diagnosis.

[0015] (Flowchart of Classification Support Method) The flowchart of classification support method S1 will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flowchart of classification support method S1. Classification support method S1 includes a division process S11, a patch classification process S12, a detection process S13, and a cell classification process S14, as shown in Figure 2.

[0016] In the splitting process S11, at least one processor splits an input image, which includes a sample containing multiple cells as its subject, into multiple partial images, each containing multiple cells as its subject. In the patch classification process S12, the at least one processor takes the image containing multiple cells as its subject as input and inputs the partial images to a trained model that has been trained to estimate whether the image contains malignant cells. Based on the estimation results of the trained model, the processor classifies whether the partial images contain malignant cells. In the detection process S13, the at least one processor detects cells from the partial images that were classified as containing malignant cells by the patch classification process S12, among the multiple partial images obtained by the splitting process S11. In the cell classification process S14, the at least one processor classifies whether the cells detected in the detection process S13 are malignant cells.

[0017] (Effects of the classification support method) As described above, the classification support method S1 employs a configuration that includes: a division process S11 in which at least one processor divides an input image containing a sample containing multiple cells into multiple partial images, each containing multiple cells as its subject; a patch classification process S12 in which the at least one processor takes an image containing multiple cells as its subject as input, inputs the partial images to a trained model that has been trained to estimate whether malignant cells are contained in the image, and classifies whether malignant cells are contained in the partial images based on the estimation results of the trained model; a detection process S13 in which the at least one processor detects cells from the partial images that have been classified as containing malignant cells by the patch classification process S12 among the multiple partial images obtained by the division process S11; and a cell classification process S14 in which the at least one processor classifies whether the cells detected by the detection process S13 are malignant cells. Therefore, the classification support method S1 has the effect of improving the accuracy of classifying sample cells as benign or malignant in pathological diagnosis.

[0018] [Second Exemplary Embodiment] A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0019] <Classification Support Device> The classification support device 2 described herein is a device that classifies specimen cells to determine whether they are benign or malignant for pathological diagnosis. Pathological diagnosis refers to the process of observing a sample taken from the human body under a microscope to diagnose the presence or absence of a lesion and the type of lesion. The classification support device 2 is used, for example, in cytology during rapid on-site evaluation (ROSE). Cytology is a type of pathological diagnosis that involves classifying cells to determine whether they are benign or malignant.

[0020] <Configuration of the Classification Support Device> The configuration of the classification support device 2 will be described with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the classification support device 2 according to this exemplary embodiment. As shown in Figure 3, the classification support device 2 comprises a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, and an output unit 25.

[0021] (Memory Unit) The memory unit 22 stores various types of information that the control unit 21 refers to. Examples of such information include image PT, patch classification model LM1, and cell classification model LM2. Patch classification model LM1 and cell classification model LM2 are trained models generated by machine learning. When we say that trained models are stored in the memory unit 22, we mean that the parameters that define the trained models are stored in the memory unit 22.

[0022] Image PT is an image that includes a specimen containing multiple cells as its subject. Image PT is an example of an input image related to this disclosure. An example of an image PT is an image obtained by photographing respiratory cells collected using an endoscope. More specifically, an image PT is, for example, an image taken by a camera attached to a microscope or an image transmitted from a microscope when collecting cells and observing them under a microscope.

[0023] The patch classification model LM1 is a pre-trained model used by the patch classification unit 213 (described later) to classify partial images (patches) of the image PT. The cell classification model LM2 is a pre-trained model used by the cell classification unit 215 (described later) to classify cells. Details of these pre-trained models will be described later.

[0024] (Communication Unit) The communication unit 23 is a communication module that communicates with other devices via a communication line N. The communication line N may be, for example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public network, a mobile data communication network, or a combination thereof. For example, the communication unit 23 may output data supplied from the control unit 21 to other devices via the communication line N, or acquire data output from other devices via the communication line N and supply it to the control unit 21.

[0025] (Input Unit) The input unit 24 is an interface for acquiring data from other connected devices. The input unit 24 supplies the data acquired from the input device to the control unit 21. Examples of input devices include, but are not limited to, a keyboard, mouse, touch panel, camera, and microphone.

[0026] (Output Unit) The output unit 25 is an interface that outputs data to other connected devices. The output unit 25 outputs the data supplied from the control unit 21 to an output device. Examples of output devices include, but are not limited to, displays, printers, touch panels, and speakers.

[0027] (Control Unit) The control unit 21 controls each part of the classification support device 2. For example, the control unit 21 stores data acquired from the communication unit 23 or the input unit 24 in the storage unit 22, or supplies data stored in the storage unit 22 to the communication unit 23 or the output unit 25.

[0028] As shown in Figure 3, the control unit 21 also functions as an acquisition unit 211, a division unit 212, a patch classification unit 213, a detection unit 214, a cell classification unit 215, and a learning unit 216. The division unit 212, the patch classification unit 213, the detection unit 214, and the cell classification unit 215 are configured to implement a division means, a patch classification means, a detection means, and a cell classification means, respectively, in this disclosure. The learning unit 216 is configured to implement a partial image extraction means, a labeling means, and a learning means, in this disclosure.

[0029] (Acquisition Unit) The acquisition unit 211 acquires data supplied from the communication unit 23 or the input unit 24 and stores the acquired data in the storage unit 22. Alternatively, the acquisition unit 211 may acquire the data by reading it from a storage location specified by the user of the classification support device 2 (which may be a storage device within the classification support device 2 or a storage device outside the classification support device 2). An example of data acquired by the acquisition unit 211 is image PT. More specifically, the acquisition unit 211 acquires image PT output by an imaging device that images cells collected using an endoscope. Examples of imaging devices include microscope cameras and virtual slide scanners. The acquisition unit 211 stores the acquired data in the storage unit 22.

[0030] (Dividing section) The dividing section 212 divides the image PT into multiple partial images, each containing multiple cells as subjects. The partial images obtained by the division of the dividing section 212 are also called "patches". As an example, the dividing section 212 generates multiple partial images by dividing the image PT in a grid pattern.

[0031] (Patch Classification Unit) The patch classification unit 213 receives a partial image as input to the patch classification model LM1 and classifies whether malignant cells are included in the partial image based on the estimation results of the patch classification model LM1. The patch classification model LM1 is an example of a trained model related to this disclosure.

[0032] (Patch Classification Model) The patch classification model LM1 is a pre-trained model that takes an image containing multiple cells as input and estimates whether the image contains malignant cells. For example, the patch classification model LM1 can use a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or a combination thereof. Alternatively, the patch classification model LM1 may use a non-neural network type model such as a random forest or a support vector machine.

[0033] The input to the patch classification model LM1 is an image containing multiple cells as its subject. The output of the patch classification model LM1 includes a classification result indicating whether the input image contains malignant cells. The classification result may be binary data indicating whether or not malignant cells are present, or it may be a real number indicating the probability of malignant cells being present.

[0034] (Detection Unit) The detection unit 214 detects cells from among the multiple partial images obtained by the division of the division unit 212, specifically from the partial images classified by the patch classification unit 213 as containing malignant cells. As an example, the detection unit 214 detects cells by inputting the partial images into a detection model generated by machine learning. Here, as an example, the detection model is a model trained to take an image as input and detect the location of cells contained in the image. As an example, the input to the detection model is an image, and as an example, the output of the detection model includes location information indicating the location of cells contained in the input image. As the detection model, for example, CNN, RNN, or a combination thereof can be used. Alternatively, as the detection model, for example, a non-neural network type model such as a random forest or a support vector machine may be used.

[0035] However, the method by which the detection unit 214 detects cells from a partial image is not limited to the example described above, and the detection unit 214 may detect cells from a partial image by other methods. For example, the detection unit 214 may detect cells by performing predetermined image processing on the partial image, such as pattern matching or edge extraction.

[0036] (Cell Classification Unit) The cell classification unit 215 classifies whether the cells detected by the detection unit 214 are benign or malignant. As an example, the cell classification unit 215 inputs a cell image containing the cells detected by the detection unit 214 as the subject into a cell classification model LM2 generated by machine learning, and classifies whether the cells are malignant based on the estimation results of the cell classification model LM2.

[0037] The cell classification model LM2 is a pre-trained model that takes cell images containing cells as input and estimates whether the cells are malignant. For the patch classification model LM1, for example, CNN, RNN, or a combination thereof can be used. Alternatively, for the cell classification model LM2, a non-neural network model such as a random forest or support vector machine may be used.

[0038] The input to the cell classification model LM2 is an image containing cells as the subject. The output of the cell classification model LM2 includes a classification result indicating whether the cells are benign or malignant. The classification result may be binary data indicating whether the cells are benign or malignant, or it may be a real number indicating the certainty that the cells are malignant.

[0039] The cell classification unit 215 outputs the classification results. For example, the cell classification unit 215 may transmit the classification results to another device via the communication unit 23, or it may output the classification results to an output device such as a display. More specifically, the cell classification unit 215 may output the classification results to an output device used for intraoperative rapid diagnosis of a specimen. Here, the output device may include, but is not limited to, a display device such as a display or projector used for intraoperative rapid diagnosis. Alternatively, for example, the cell classification unit 215 may output the classification results by writing them to a storage location specified by the user of the classification support device 2 (which may be a storage device within the classification support device 2 or a storage device outside the classification support device 2).

[0040] (Learning Unit) The learning unit 216 trains the patch classification model LM1 using machine learning with training data. The training data used to train the patch classification model LM1 includes, as an example, a set of partial images extracted from a training image that includes a sample containing multiple cells as the subject, and a label indicating whether the partial image contains malignant cells. In other words, the learning unit 216 trains the patch classification model LM1 using the labeled partial images. The learning unit 216 also trains the cell classification model LM2. The training data used to train the cell classification model LM2 includes, as an example, a set of cell images that include cells as the subject, and a label indicating whether the cells are benign or malignant.

[0041] <Flowchart of Classification Support Method> The flowchart of the classification support method S2 according to this disclosure will be explained with reference to Figures 4 and 5. Figure 4 is a flowchart showing the flowchart of the classification support method S2 according to this exemplary embodiment. Figure 5 is a diagram that schematically shows the contents of each process included in the classification support method S2.

[0042] (Steps S21 and S22) In step S21, the acquisition unit 211 acquires an image PT that includes the sample cells as the subject. The acquisition unit 211 stores the acquired image PT in the storage unit 22. In step S22, the division unit 212 divides the image PT into multiple partial images. In the example in Figure 5, the division unit 212 generates multiple partial images by dividing the image PT into a grid.

[0043] (Step S23) In step S23, the patch classification unit 213 inputs partial images to the patch classification model LM1 and classifies whether the input partial images contain malignant cells based on the estimation results of the patch classification model LM1. As a result, the partial images obtained by the division process of the division unit 212 are classified into partial images containing malignant cells and partial images not containing malignant cells. In the example in Figure 5, among the partial images obtained by dividing the image PT into a grid, the partial images enclosed by solid lines are classified as containing malignant cells, and the partial images enclosed by dotted lines are classified as not containing malignant cells.

[0044] (Step S24) In step S24, the detection unit 214 detects cells from the partial images. At this time, the detection unit 214 does not detect cells from all the partial images obtained by the division process of the division unit 212, but detects cells from the partial images classified by the patch classification unit 213 as including malignant cells. In the example of FIG. 5, cells are detected from the partial image surrounded by the solid line. That is, the detection unit 214 does not detect cells from the partial images classified as not including malignant cells (the partial images surrounded by the dotted line in FIG. 5).

[0045] (Step S25) In step S25, the cell classification unit 215 classifies whether the cells detected by the detection unit 214 are benign cells or malignant cells by inputting the cell image including the cells detected by the detection unit 214 into the cell classification model LM2.

[0046] (Step S26) In step S26, the cell classification unit 215 outputs the classification result. In the example of FIG. 5, the cell classification unit 215 causes the display to display an image PT2 (an image in which a figure indicating the position of the cells is superimposed on the image PT) in which the cells classified as malignant cells in the image PT are marked. However, the method by which the cell classification unit 215 outputs the classification result is not limited to the above-described example. As an example, the cell classification unit 215 may output data indicating the number or ratio of malignant cells in the cells included in the input image.

[0047] <Flow of the learning method of the patch classification model> Next, the learning method of the patch classification model LM1 will be described. FIG. 6 is a diagram schematically showing an example of the learning method of the patch classification model LM1 by the learning unit 216. In the example of FIG. 6, the learning unit 216 generates training data using the training image PT31. The training image PT31 is an image including a specimen including a plurality of cells as a subject, and a label indicating the position of malignant cells is attached to the training image PT31. As an example, the label indicating the position of malignant cells is attached to the training image in advance by medical staff or the like.

[0048] In the example of FIG. 6, the learning unit 216 cuts out a plurality of partial images each including a plurality of cells as subjects from the training image PT31. As an example, the learning unit 216 cuts out a partial image from the training image based on coordinates randomly selected in the training image. In the image PT32 of FIG. 6, the positions of the partial images randomly cut out from the training image PT31 are shown. However, the method by which the learning unit 216 cuts out the partial images is not limited to this, and the learning unit 216 may cut out the partial images by other methods. For example, the learning unit 216 may generate a divided image by dividing the training image into a grid.

[0049] Next, the learning unit 216 attaches a label indicating that malignant cells are included to the partial images containing malignant cells among the plurality of partial images cut out from the training image PT31, and attaches a label indicating that no malignant cells are included to the partial images not containing malignant cells. The learning unit 216 uses the set PT33 of the labeled partial images as training data to train the patch classification model LM1.

[0050] <Specific Example of Cell Classification Processing> A specific example of the cell classification processing performed by the cell classification unit 215 will be described. In the following example, the cell classification unit 215 classifies cells using at least one of the first cell classification model, the second cell classification model, and the third cell classification model.

[0051] The first cell classification model, as an example, is a model that has been trained to take an image containing cells as its subject as input and estimate the subclass to which the cells contained as the subject in the image PT belong, from among the first benign subclass group (which classifies benign cells into multiple subclasses) and the first malignant subclass group (which classifies malignant cells into multiple subclasses). In other words, when an image containing cells as its subject is input to the first cell classification model, it is trained to output an estimation result indicating which of the first benign subclass group or which of the first malignant subclass group the cells belong to. When using the first cell classification model, in other words, the cell classification unit 215 inputs a cell image containing cells detected by the detection unit 214 to the first cell classification model and classifies whether the cells are malignant or not based on the estimation result of the first cell classification model.

[0052] Here, a subclass refers to a classification of both benign and malignant cells into multiple subclasses. The first benign subclass group refers to a classification of benign cells into multiple subclasses, and the first malignant subclass group refers to a classification of malignant cells into multiple subclasses. Estimating the subclass to which a cell belongs means estimating which subclass of the first benign subclass group and the first malignant subclass group the cell belongs to. There is no limit to the number of subclasses into which benign and malignant cells are classified; it is sufficient if the number of subclasses is identifiable by the first cell classification model.

[0053] Figure 7 shows Table t1, an example of the classification of the first benign subclass group and the first malignant subclass group. Table t1 in Figure 7 shows the first benign subclass group, which is classified into five subclasses, and the first malignant subclass group, which is classified into four subclasses. As shown in Figure 7, in Table t1, each of the subclass numbers "0" to "4" is associated with each of the five subclasses included in the first benign subclass group. For example, subclass number "0" is associated with subclass "EC" in the first benign subclass group.

[0054] Similarly, Table t1 associates each of the subclass numbers "5" through "8" with each of the four subclasses included in the first group of malignant subclasses. For example, subclass number "5" is associated with subclass "S" in the first group of malignant subclasses.

[0055] As described above, there is no limit to the number of subclasses into which benign and malignant cells are classified, but they are classified into a number that can be classified by the first cell classification model. For example, any subclass within the first benign subclass group is classified such that it differs from other subclasses within the first benign subclass group in terms of visual findings or histological type. Similarly, any subclass within the first malignant subclass group is classified such that it differs from other subclasses within the first malignant subclass group in terms of visual findings or histological type.

[0056] The second cell classification model is, for example, a model that has been trained to take an image containing cells as an input and estimate the subclass to which the cells included as an input belong, from a second group of benign subclasses (which classifies benign cells into multiple subclasses) and a second group of malignant subclasses (which classifies malignant cells into multiple subclasses). When using the second cell classification model, in other words, the cell classification unit 215 can also input a cell image containing cells detected by the detection unit 214 as an input to the second cell classification model and further classify whether the cells detected by the detection unit 214 are benign or malignant based on the estimation results of the second cell classification model.

[0057] Similar to the first benign and malignant subclasses, the second benign subclass and the second malignant subclass are not limited in the number of subclasses they are classified into. However, as an example, at least one of the first benign subclass and the second benign subclass, and the first malignant subclass and the second malignant subclass, are different.

[0058] Figure 8 is Table t2, which shows an example of the classification of the second benign subclass and the second malignant subclass. Table t2 in Figure 8 shows the second benign subclass group, which is classified into three subclasses, and the second malignant subclass group, which is classified into two subclasses. The subclass numbers in Table t2 refer to the same subclass numbers as in Table t1.

[0059] As shown in Figure 8, in Table t2, each of the numbers of one or more subclasses is associated with each of the three subclasses included in the first benign subclass group. For example, subclass number "0" is associated with subclass "BT" in the second benign subclass group. As another example, subclass numbers "1" and "2" are associated with subclass "BO" in the second benign subclass group.

[0060] Similarly, in Table t2, each of the subclass numbers is associated with each of the two subclasses included in the second group of malignant subclasses. For example, subclass number "5" is associated with subclass "MS" in the second group of malignant subclasses. As another example, subclass numbers "6", "7", and "8" are associated with subclass "MN" in the second group of malignant subclasses.

[0061] The third cell classification model is, for example, a model that has been trained to estimate whether the cells included as subjects in the image PT belong to benign cells or malignant cells. When using the third cell classification model, in other words, the cell classification unit 215 can also input a cell image containing the cells detected by the detection unit 214 as subjects to the third cell classification model, and further classify whether the cells detected by the detection unit 214 are benign or malignant cells based on the estimation results of the third cell classification model.

[0062] Figure 9 is Table t3, which shows an example of the classification of benign and malignant cells in this exemplary embodiment. In Table t3 shown in Figure 9, each subclass is indicated as belonging to either benign or malignant cells. As shown in Figure 9, subclass numbers "0" to "4" are associated with "Benign," indicating benign cells, and subclass numbers "5" to "8" are associated with "Malignant," indicating malignant cells. Similar to Table t2, the subclass numbers in Table t3 refer to the same subclass numbers as in Table t1.

[0063] As an example, the cell classification unit 215 classifies the sample cells as benign cells if at least one of the estimation results output from the first cell classification model, the second cell classification model, and the third cell classification model (multiple learning models) indicates that the sample cells are classified into a benign subclass or benign cells. In other words, the cell classification unit 215 classifies the sample cells as malignant cells if estimation results indicating that the sample cells are classified into a malignant subclass or malignant cells are obtained from all of the multiple learning models. However, the method by which the cell classification unit 215 classifies cells is not limited to the example described above, and the cell classification unit 215 may classify cells by other methods.

[0064] <Learning of the first, second, and third cell classification models> As an example, the learning unit 216 trains at least one of the first and second cell classification models using training data that includes pairs of images containing cells as subjects and labels indicating the subclass to which the cells belong. Also, as an example, the learning unit 216 trains a third cell classification model using training data that includes pairs of images containing cells as subjects and labels indicating whether the cells are benign or malignant.

[0065] <Effects of Classification Support Devices> Incidentally, the main characteristics of benign cells include, for example, being relatively small in size, having a fairly neat circular shape, and having a clean outline. On the other hand, the main characteristics of malignant cells include being large in size (swollen), being deformed, having an irregular outline, and having an enlarged nucleolus. However, since there are malignant cells that have the characteristics of benign cells, and conversely, benign cells that have the characteristics of malignant cells, it is not always possible to properly classify cells as benign or malignant based solely on images of individual cells. In other words, based only on the appearance of individual cells, benign cells may be classified as malignant cells, and malignant cells may be classified as benign cells.

[0066] In contrast, in the classification support device 2 according to this disclosure, the patch classification unit 213 classifies whether malignant cells are included in a partial image, the detection unit 214 detects cells from the partial image classified as containing malignant cells, and the cell classification unit 215 classifies whether the cells detected by the detection unit 214 are malignant cells. In other words, the classification support device 2 does not detect cells from partial images classified as not containing malignant cells.

[0067] Thus, the classification support device 2 can improve the accuracy of classification by considering not only the characteristics of the cell itself but also the characteristics of cells surrounding it, when classifying whether a cell is benign or malignant. In other words, by not detecting cells from partial images classified as benign, the possibility of false positives, where benign cells are classified as malignant, can be reduced. By reducing the possibility of false positives, the classification support device 2 can reduce the burden on patients due to re-examinations.

[0068] Furthermore, in the classification support device 2, the cell classification unit 215 is configured to output the classification results to an output device used for intraoperative rapid diagnosis of the specimen. Therefore, the classification support device 2 has the effect of enabling more accurate cytological diagnosis during intraoperative rapid diagnosis.

[0069] Furthermore, the classification support device 2 employs a configuration that includes a learning unit 216 that trains a patch classification model LM1 using machine learning with training data that includes multiple partial images, each containing multiple cells, extracted from a training image containing a sample containing multiple cells, and sets of the extracted partial images and labels indicating whether malignant cells are included in the partial images. By using the trained patch classification model LM1, the classification support device 2 can improve the accuracy of classifying sample cells as benign or malignant in pathological diagnosis.

[0070] Furthermore, in the classification support device 2, the training images are labeled to indicate the location of malignant cells. The learning unit 216 labels the partial images containing malignant cells from among the multiple extracted partial images, indicating that malignant cells are present, and labels the partial images that do not contain malignant cells, indicating that malignant cells are not present. The patch classification model LM1 is then trained using these labeled partial images. By using the trained patch classification model LM1, the classification support device 2 can improve the accuracy of classifying specimen cells as benign or malignant in pathological diagnosis.

[0071] Furthermore, in the classification support device 2, the image PT is configured to be an image obtained by photographing respiratory cells collected using an endoscope. Therefore, the classification support device 2 has the effect of enabling more accurate cytological examination of respiratory cells collected using an endoscope.

[0072] Furthermore, in the classification support device 2, the cell classification unit 215 takes an image containing cells as input and inputs a cell image containing cells detected by the detection unit 214 to a first cell classification model that has been trained to estimate the subclass to which the cell belongs from a first benign subclass group which classifies benign cells into multiple subclasses and a first malignant subclass group which classifies malignant cells into multiple subclasses. Based on the estimation results of the first cell classification model, the cell is classified as a malignant cell. Therefore, the classification support device 2 can improve the accuracy of classifying both malignant and malignant cells in a sample.

[0073] [Example of implementation by software] Some or all of the functions of classification support devices 1 and 2 (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0074] In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 10. Figure 10 is a block diagram showing the hardware configuration of computer C, which functions as each of the above devices.

[0075] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P for operating Computer C as each of the above-mentioned devices. In Computer C, the processor C1 reads and executes the program P from memory C2, thereby realizing each of the above-mentioned devices.

[0076] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0077] Furthermore, computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Computer C may also be equipped with a communication interface for sending and receiving data with other devices. Furthermore, computer C may also be equipped with an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0078] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such recording medium M can include, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such recording medium M. Program P can also be transmitted via a transmission medium. Such transmission mediums can include, for example, a communication network or broadcast waves. Computer C can also acquire program P via such transmission medium.

[0079] Furthermore, each of the above functions of each of the above devices may be implemented by a single processor in a single computer, by multiple processors in a single computer working together, or by multiple processors in each of multiple computers working together. In addition, the programs for implementing each of the above functions in each of the above devices may be stored in a single memory in a single computer, distributed and stored in multiple memories in a single computer, or distributed and stored in multiple memories in each of multiple computers.

[0080] [Addendum 1] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0081] [Addendum A] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0082] (Note A1) A classification support device comprising: a division means for dividing an input image containing a sample containing multiple cells into multiple partial images, each containing multiple cells as its subject; a patch classification means for inputting the partial images to a trained model that has been trained to take an image containing multiple cells as its subject as input and estimate whether the image contains malignant cells, and classifying whether the partial images contain malignant cells based on the estimation results of the trained model; a detection means for detecting cells from the partial images that have been classified by the patch classification means as containing malignant cells among the multiple partial images obtained by the division means; and a cell classification means for classifying whether the cells detected by the detection means are malignant cells.

[0083] (Note A2) The classification support device according to Note A1, wherein the cell classification means outputs the classification result by the cell classification means to an output device used for rapid on-site evaluation (ROSE) of the specimen.

[0084] (Appendix A3) The classification support device according to Appendix A1 or A2, further comprising: a partial image extraction means for extracting multiple partial images, each containing multiple cells, from a training image containing a sample containing multiple cells as the subject; and a learning means for training the trained model using training data which includes a set of partial images obtained by the partial image extraction means and a label indicating whether malignant cells are contained in the partial image.

[0085] (Appendix A4) The training images are labeled to indicate the location of malignant cells, and the classification support device according to Appendix A3 further comprises labeling means for labeling partial images containing malignant cells among a plurality of partial images obtained by the partial image extraction means, labeling partial images not containing malignant cells to indicate that malignant cells are not included, and the learning means trains the trained model using the partial images labeled by the labeling means.

[0086] (Note A5) The input image is an image obtained by photographing respiratory cells collected using an endoscope, a classification support device as described in any one of Notes A1 to A4.

[0087] (Note A6) The cell classification means inputs an image containing a cell as the subject, and inputs a cell image containing the cell detected by the detection means to a first cell classification model that has been trained to estimate the subclass to which the cell belongs from a first benign subclass group which classifies benign cells into multiple subclasses and a first malignant subclass group which classifies malignant cells into multiple subclasses, and classifies whether the cell is a malignant cell based on the estimation result of the first cell classification model, as described in any one of Notes A1 to A5.

[0088] (Appendix A7) The classification support device as described in Appendix A6, wherein any subclass included in the first benign subclass group differs from other subclasses included in the first benign subclass group in terms of visual findings or histological type, and any subclass included in the first malignant subclass group differs from other subclasses included in the first malignant subclass group in terms of visual findings or histological type.

[0089] (Note A8) The cell classification means further inputs a cell image containing the cell detected by the detection means to a second cell classification model that has been trained to estimate the subclass to which the cell belongs from a second benign subclass group which classifies benign cells into a plurality of subclasses and a second malignant subclass group which classifies malignant cells into a plurality of subclasses, and the cell classification means further classifies whether the cell detected by the detection means is a benign cell or a malignant cell based on the estimation result of the second cell classification model, wherein at least one of the first benign subclass group and the second benign subclass group, and the first malignant subclass group and the second malignant subclass group are different, the classification support device according to Note A6 or A7.

[0090] (Note A9) The cell classification means further inputs a cell image containing the cell detected by the detection means to a third cell classification model that has been trained to take an image containing a cell as input and estimate whether the cell belongs to benign or malignant cells, and the cell classification means further classifies whether the cell detected by the detection means is a benign or malignant cell based on the estimation result of the third cell classification model, the classification support device according to any one of Notes A1 to A8.

[0091] (Note A10) The classification support device according to any one of Notes A6 to A9, wherein the cell classification means classifies the sample cells as benign cells if at least one of the results output from each of the plurality of cell classification models indicates that they are classified as benign subclasses or benign cells.

[0092] (Appendix A11) A classification support device according to any one of Appendix A6 to A8, further comprising: a learning means for training the first cell classification model using training data which includes a pair of an image containing a cell as a subject and a label indicating the subclass to which the cell belongs.

[0093] [Addendum B] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0094] (Note B1) A classification support method comprising: a division process in which at least one processor divides an input image, which includes a sample containing multiple cells as its subject, into multiple partial images, each containing multiple cells as its subject; a patch classification process in which the at least one processor takes an image containing multiple cells as its subject as input, inputs the partial images to a trained model that has been trained to estimate whether the image contains malignant cells, and classifies whether the partial images contain malignant cells based on the estimation results of the trained model; a detection process in which the at least one processor detects cells from among the multiple partial images obtained by the division process, specifically from the partial images that have been classified as containing malignant cells by the patch classification process; and a cell classification process in which the at least one processor classifies whether the cells detected in the detection process are malignant cells.

[0095] (Appendix B2) The classification support method according to Appendix B1, wherein in the cell classification process, at least one processor outputs the classification result obtained by the cell classification process to an output device used for rapid on-site evaluation (ROSE) of the specimen.

[0096] (Appendix B3) The classification support method according to Appendix B1 or B2, further comprising: a partial image extraction process in which at least one processor extracts a plurality of partial images, each containing a plurality of cells, from a training image containing a sample containing a plurality of cells as the subject; and a learning process in which at least one processor trains the trained model using training data that includes a set of partial images obtained by the partial image extraction process and a label indicating whether malignant cells are contained in the partial image.

[0097] (Appendix B4) The training images are labeled to indicate the location of malignant cells, and the classification support method according to Appendix B3 further includes a labeling process in which at least one processor labels partial images containing malignant cells with a label indicating that malignant cells are present, and labels partial images not containing malignant cells with a label indicating that malignant cells are not present, and in the learning process, the at least one processor trains the trained model using the partial images labeled by the labeling process.

[0098] (Note B5) The input image is an image obtained by photographing respiratory cells collected using an endoscope, the classification support method described in any one of Notes B1 to B4.

[0099] (Note B6) The classification support method according to any one of Notes B1 to B5, wherein in the cell classification process, the at least one processor takes an image containing cells as input and inputs a cell image containing the cells detected in the detection process to a first cell classification model that has been trained to estimate the subclass to which the cells belong from a first benign subclass group which classifies benign cells into a plurality of subclasses and a first malignant subclass group which classifies malignant cells into a plurality of subclasses, and classifies whether the cells are malignant cells based on the estimation results of the first cell classification model.

[0100] (Appendix B7) The classification support method described in Appendix B6, wherein any subclass included in the first benign subclass group differs from other subclasses included in the first benign subclass group in terms of visual findings or histological type, and any subclass included in the first malignant subclass group differs from other subclasses included in the first malignant subclass group in terms of visual findings or histological type.

[0101] (Note B8) The classification support method according to Note B6 or B7, wherein in the cell classification process, the at least one processor takes an image containing cells as input and inputs a cell image containing the cells detected in the detection process as input to a second cell classification model that has been trained to estimate the subclass to which the cells belong from a second benign subclass group which classifies benign cells into a plurality of subclasses and a second malignant subclass group which classifies malignant cells into a plurality of subclasses; and in the cell classification process, the at least one processor further classifies whether the cells detected in the detection process are benign or malignant based on the estimation results of the second cell classification model, and at least one of the first benign subclass group and the second benign subclass group, and the first malignant subclass group and the second malignant subclass group are different.

[0102] (Note B9) The classification support method according to any one of Notes B1 to B8, wherein in the cell classification process, the at least one processor takes an image containing cells as input and inputs a cell image containing the cells detected in the detection process to a third cell classification model that has been trained to estimate whether the cells belong to benign or malignant cells, and in the cell classification process, the at least one processor further classifies whether the cells detected in the detection process are benign or malignant cells based on the estimation results of the third cell classification model.

[0103] (Note B10) The classification support method according to any one of Notes B6 to B9, wherein in the cell classification process, if at least one of the results output from each of the plurality of cell classification models indicates that the sample cells are classified as benign cells, the at least one processor classifies the sample cells as benign cells.

[0104] (Appendix B11) The classification support method according to any one of Appendix B6 to B8, further comprising a learning process in which at least one processor trains the first cell classification model using training data which includes a pair of an image containing a cell as a subject and a label indicating the subclass to which the cell belongs.

[0105] [Addendum C] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0106] (Note C1) A program for causing a computer to function as a classification support device, wherein the computer is a division means that divides an input image containing a sample containing multiple cells into multiple partial images, each containing multiple cells as its subject; a patch classification means that takes an image containing multiple cells as its subject as input, inputs the partial images to a trained model that has been trained to estimate whether the image contains malignant cells, and classifies whether the partial images contain malignant cells based on the estimation results of the trained model; a detection means that detects cells from among the multiple partial images obtained by the division means, specifically from partial images that have been classified by the patch classification means as containing malignant cells; and a cell classification means that classifies whether the cells detected by the detection means are malignant cells.

[0107] (Appendix C2) The classification support program described in Appendix C1, wherein the cell classification means outputs the classification result by the cell classification means to an output device used for rapid on-site evaluation (ROSE) of the specimen.

[0108] (Note C3) A classification support program as described in Note C1 or C2, wherein the computer further functions as a partial image extraction means for extracting a plurality of partial images, each containing a plurality of cells, from a training image containing a sample containing a plurality of cells as the subject, and a learning means for training the trained model using training data which includes a set of partial images obtained by the partial image extraction means and a label indicating whether malignant cells are contained in the partial image.

[0109] (Note C4) The training images are labeled to indicate the location of malignant cells, and the computer is further configured as a labeling means that labels indicating the presence of malignant cells to partial images containing malignant cells among the multiple partial images obtained by the partial image extraction means, and labels indicating the absence of malignant cells to partial images that do not contain malignant cells, and the learning means trains the trained model using the partial images labeled by the labeling means, as described in Note C3.

[0110] (Note C5) The input image is an image obtained by photographing respiratory cells collected using an endoscope, a classification support program as described in any one of Notes C1 to C4.

[0111] (Note C6) The cell classification means inputs an image containing cells as the subject, and inputs a cell image containing cells detected by the detection means to a first cell classification model that has been trained to estimate the subclass to which the cell belongs from a first benign subclass group which classifies benign cells into multiple subclasses and a first malignant subclass group which classifies malignant cells into multiple subclasses, and classifies whether the cell is a malignant cell based on the estimation result of the first cell classification model, as described in any one of Notes C1 to C5.

[0112] (Appendix C7) The classification support program as described in Appendix C6, wherein any subclass included in the first benign subclass group differs from other subclasses included in the first benign subclass group in terms of visual findings or histological type, and any subclass included in the first malignant subclass group differs from other subclasses included in the first malignant subclass group in terms of visual findings or histological type.

[0113] (Note C8) The cell classification means further inputs a cell input image containing the cell detected by the detection means to a second cell classification model that has been trained to estimate the subclass to which the cell belongs from a second benign subclass group which classifies benign cells into a plurality of subclasses and a second malignant subclass group which classifies malignant cells into a plurality of subclasses, and the cell classification means further classifies whether the cell detected by the detection means is a benign cell or a malignant cell based on the estimation result of the second cell classification model, wherein at least one of the first benign subclass group and the second benign subclass group, and the first malignant subclass group and the second malignant subclass group are different, the classification support program as described in Note C6 or C7.

[0114] (Note C9) The cell classification means further inputs a cell image containing the cell detected by the detection means to a third cell classification model that has been trained to take an image containing a cell as input and estimate whether the cell belongs to benign or malignant cells, and the cell classification means further classifies whether the cell detected by the detection means is a benign or malignant cell based on the estimation result of the third cell classification model, the classification support program according to any one of Notes C1 to C8.

[0115] (Note C10) The classification support program according to any one of Notes C6 to C9, wherein the cell classification means classifies the sample cells as benign cells if at least one of the results output from each of the plurality of cell classification models indicates that they are classified as benign subclasses or benign cells.

[0116] (Note C11) A classification support program according to any one of Notes C6 to C8, which further causes the computer to function as a learning means for training the first cell classification model using training data that includes a pair of an image containing a cell as the subject and a label indicating the subclass to which the cell belongs.

[0117] [Addendum D] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0118] (Note D1) A classification support device comprising at least one processor, the at least one processor performing: a division process that divides an input image, which includes a sample containing multiple cells as its subject, into multiple partial images, each containing multiple cells as its subject; a patch classification process that takes an image containing multiple cells as its subject as input, inputs the partial images to a trained model that has been trained to estimate whether the image contains malignant cells, and classifies whether the partial images contain malignant cells based on the estimation results of the trained model; a detection process that detects cells from the partial images that have been classified as containing malignant cells by the patch classification process among the multiple partial images obtained by the division process; and a cell classification process that classifies whether the cells detected in the detection process are malignant cells. The classification support device may further include memory. The memory may also store a program for causing the at least one processor to perform each of the above processes.

[0119] (Note D2) The classification support device according to Note D1, wherein in the cell classification process, at least one processor outputs the classification result obtained by the cell classification process to an output device used for rapid on-site evaluation (ROSE) of the specimen.

[0120] (Note D3) The classification support device according to Note D1 or D2, further comprising: a partial image extraction process that extracts multiple partial images, each containing multiple cells, from a training image containing a sample containing multiple cells as the subject; and a learning process in which at least one processor trains the trained model using training data that includes a set of partial images obtained by the partial image extraction process and a label indicating whether malignant cells are contained in the partial image.

[0121] (Note D4) The training images are labeled to indicate the location of malignant cells, and the at least one processor further performs a labeling process in which it labels the partial images containing malignant cells with a label indicating that malignant cells are present, and the partial images not containing malignant cells with a label indicating that malignant cells are not present, and in the learning process, the at least one processor trains the trained model using the partial images labeled by the labeling process, as described in Note D3.

[0122] (Note D5) The input image is an image obtained by photographing respiratory cells collected using an endoscope, the classification support device as described in any one of Notes D1 to D4.

[0123] (Note D6) The classification support device according to any one of Notes D1 to D5, wherein in the cell classification process, the at least one processor takes an image containing cells as input and inputs a cell image containing the cells detected in the detection process to a first cell classification model that has been trained to estimate the subclass to which the cell belongs from a first benign subclass group which classifies benign cells into a plurality of subclasses and a first malignant subclass group which classifies malignant cells into a plurality of subclasses, and classifies whether the cell is a malignant cell based on the estimation result of the first cell classification model.

[0124] (Appendix D7) The classification support device according to Appendix D6, wherein any subclass included in the first benign subclass group differs from other subclasses included in the first benign subclass group in terms of visual findings or histological type, and any subclass included in the first malignant subclass group differs from other subclasses included in the first malignant subclass group in terms of visual findings or histological type.

[0125] (Note D8) The classification support device according to Note D6 or D7, wherein in the cell classification process, the at least one processor takes an image containing cells as input and inputs a cell image containing the cells detected in the detection process as input to a second cell classification model that has been trained to estimate the subclass to which the cells belong from a second benign subclass group which classifies benign cells into a plurality of subclasses and a second malignant subclass group which classifies malignant cells into a plurality of subclasses; and in the cell classification process, the at least one processor further classifies whether the cells detected in the detection process are benign or malignant based on the estimation results of the second cell classification model, and at least one of the first benign subclass group and the second benign subclass group, and the first malignant subclass group and the second malignant subclass group are different.

[0126] (Note D9) The classification support device according to any one of Notes D1 to D8, wherein in the cell classification process, the at least one processor takes an image containing cells as input and inputs a cell image containing the cells detected in the detection process to a third cell classification model that has been trained to estimate whether the cells belong to benign or malignant cells, and in the cell classification process, the at least one processor further classifies whether the cells detected in the detection process are benign or malignant cells based on the estimation results of the third cell classification model.

[0127] (Note D10) The classification support device according to any one of Notes D6 to D9, wherein in the cell classification process, if at least one of the results output from each of the plurality of cell classification models indicates that the sample cells are classified as benign cells, the at least one processor classifies the sample cells as benign cells.

[0128] (Note D11) The classification support device according to any one of Notes D6 to D8, wherein the at least one processor further performs a learning process to train the first cell classification model using training data which includes a pair of an image containing a cell as a subject and a label indicating the subclass to which the cell belongs.

[0129] [Addendum E] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0130] (Appendix E1) A non-temporary recording medium that records a classification support program for causing a computer to function as a classification support device, the program that causes the computer to perform: a division process that divides an input image containing a sample containing multiple cells into multiple partial images, each containing multiple cells as its subject; a patch classification process that takes an image containing multiple cells as its subject as input, inputs the partial images to a trained model that has been trained to estimate whether the image contains malignant cells, and classifies whether the partial images contain malignant cells based on the estimation results of the trained model; a detection process that detects cells from the partial images among the multiple partial images obtained by the division process that have been classified as containing malignant cells by the patch classification process; and a cell classification process that classifies whether the cells detected in the detection process are malignant cells.

[0131] 1, 2 Classification support device 11, 212 Dividing unit 12, 213 Patch classification unit 13, 214 Detection unit 14, 215 Cell classification unit 216 Learning unit S1, S2 Classification support method S11 Dividing process S12 Patch classification process S13 Detection process S14 Cell classification process

Claims

1. A classification support device comprising: a division means for dividing an input image containing a sample containing multiple cells into multiple partial images, each containing multiple cells as its subject; a patch classification means for inputting the partial images to a trained model that has been trained to take an image containing multiple cells as its subject as input and estimate whether the image contains malignant cells, and classifying whether the partial images contain malignant cells based on the estimation results of the trained model; a detection means for detecting cells from the partial images that have been classified by the patch classification means as containing malignant cells among the multiple partial images obtained by the division means; and a cell classification means for classifying whether the cells detected by the detection means are malignant cells.

2. The classification support device according to claim 1, wherein the cell classification means outputs the classification result by the cell classification means to an output device used for rapid on-site evaluation (ROSE) of the specimen.

3. A classification support device according to claim 1 or 2, further comprising: a partial image extraction means for extracting multiple partial images, each containing multiple cells, from a training image containing a sample containing multiple cells as the subject; and a learning means for training the trained model by machine learning using training data which includes a set of partial images obtained by the partial image extraction means and a label indicating whether malignant cells are contained in the partial image.

4. The training images are labeled to indicate the location of malignant cells, and the classification support device according to claim 3 further comprises a labeling means for labeling partial images containing malignant cells from among a plurality of partial images obtained by the partial image extraction means, labeling partial images not containing malignant cells to indicate that malignant cells are not included, and the learning means trains the trained model using the partial images labeled by the labeling means.

5. The classification support device according to any one of claims 1 to 4, wherein the input image is an image obtained by photographing respiratory cells collected using an endoscope.

6. The classification support device according to any one of claims 1 to 5, wherein the cell classification means takes an image containing a cell as an input, and inputs a cell image containing the cell detected by the detection means to a first cell classification model that has been trained to estimate the subclass to which the cell belongs from a first benign subclass group which classifies benign cells into a plurality of subclasses and a first malignant subclass group which classifies malignant cells into a plurality of subclasses, and classifies whether the cell is a malignant cell based on the estimation result of the first cell classification model.

7. A classification support method comprising: a splitting process in which at least one processor splits an input image, which includes a sample containing multiple cells as its subject, into multiple sub-images, each sub-image containing multiple cells as its subject; a patch classification process in which the at least one processor takes an image containing multiple cells as its subject as input, inputs the sub-images to a trained model that has been trained to estimate whether the image contains malignant cells, and classifies whether the sub-images contain malignant cells based on the estimation results of the trained model; a detection process in which the at least one processor detects cells from the sub-images that have been classified as containing malignant cells by the patch classification process among the multiple sub-images obtained by the splitting process; and a cell classification process in which the at least one processor classifies whether the cells detected in the detection process are malignant cells.

8. A classification support program for causing a computer to function as a classification support device, the program comprising: a division means for dividing an input image containing a sample containing multiple cells into multiple partial images, each containing multiple cells as its subject; a patch classification means for inputting the partial images to a trained model that has been trained to take an image containing multiple cells as its subject as input and estimate whether the image contains malignant cells, and classifying whether the partial images contain malignant cells based on the estimation results of the trained model; a detection means for detecting cells from the partial images that have been classified as containing malignant cells by the patch classification means among the multiple partial images obtained by the division means; and a cell classification means for classifying whether the cells detected by the detection means are malignant cells.

Images