Image processing device, image processing device operation method, image processing device operation program, and learning device

The image processing device uses a machine learning model trained with relative rankings to estimate morphological abnormality levels in tissue images, addressing the labor-intensive annotation challenges of conventional methods by focusing on main findings, thus enhancing efficiency and accuracy.

WO2024228362A1PCT designated stage expired Publication Date: 2024-11-07FUJIFILM CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/016298
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-01
Filing Date
2024-04-25
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

The conventional method of creating learning data for machine learning models to detect morphological abnormalities in tissue images is labor-intensive due to the need for extensive annotation by experts, requiring a large workload both qualitatively and quantitatively, and varying criteria for abnormality judgment among individuals.

Method used

An image processing device and method that uses a machine learning model trained with tissue image sets, where relative rankings based on morphological characteristics are provided as teacher information, allowing the model to estimate evaluation values and reduce the annotation workload by focusing on main abnormal findings.

Benefits of technology

This approach reduces the workload of annotation and improves learning efficiency by allowing the model to adapt to relative rankings, enabling accurate estimation of morphological abnormality levels with reduced expert input and increased generalization performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024016298_07112024_PF_FP_ABST
    Figure JP2024016298_07112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided is an image processing device that comprises a processor and estimates an evaluation value according to a morphological feature of a tissue image by using a machine-learning model, wherein: the machine-learning model grants, in a tissue image set including a plurality of training tissue images, a relative order based on the morphological features of the training tissue images; the training causes the machine learning model to estimate the evaluation values corresponding to the morphological features of the training tissue images and fit the estimated evaluation values with the relative order; and the processor uses the trained machine learning model subjected to training to estimate an evaluation value of the acquired evaluation target.
Need to check novelty before this filing date? Find Prior Art

Description

Image processing device, operation method of image processing device, operation program of image processing device, and learning device

[0001] The technology of the present disclosure relates to an image processing device, an operating method for an image processing device, an operating program for an image processing device, and a learning device.

[0002] The specification of U.S. Patent Application Publication No. 2021 / 0342570 describes a technology that uses a machine learning model such as an autoencoder to extract features from patch images, which are tissue images obtained by dividing a specimen image of a tissue specimen such as an animal liver, and determines whether or not the tissue specimen depicted in the patch image has a morphological abnormality (hyperplasia, infiltration, congestion, inflammation, tumor, canceration, proliferation, bleeding, glycogen depletion, etc.) based on the extracted features, and clusters patch images determined to have a morphological abnormality into one of multiple clusters based on the features.

[0003] Drug discovery staff, including toxicology pathologists, are highly interested in morphological abnormalities, which contribute significantly to estimating the mechanism of action of toxicity based on their specialized knowledge. The degree of interest, referred to as the degree of abnormality, varies depending on the type and severity of the morphological abnormality. For example, if three types of morphological abnormalities—necrosis, cellular infiltration, and eosinophilic change—are ranked by degree of abnormality, the order is generally as follows: When using a machine learning model to output multiple types of morphological abnormalities in order of abnormality degree, one possible training method is to rank all of the abnormality degrees of the multiple types of morphological abnormalities on a uniform scale and then train the machine learning model on all of the abnormality degrees ranked on the uniform scale.

[0004] This method requires annotation work to create training data by looking at tissue images, classifying and identifying the degree of morphological abnormality using a unified scale, and annotating the tissue images according to the degree of abnormality. However, the problem is that this annotation work imposes a very large workload.

[0005] One problem is the heavy qualitative workload. Specifically, there are many different types of morphological anomalies, and classifying and identifying these diverse morphological anomalies requires specialized knowledge. Furthermore, the criteria for determining the degree of abnormality vary from person to person. Therefore, classifying and identifying all morphological anomalies using a uniform scale is extremely difficult in terms of accuracy and consistency of judgment. This is especially difficult when annotation is performed by an operator with little knowledge of morphological anomalies. The other problem is the heavy quantitative workload. Because a large number of tissue images are required as training data, the number of tissue images to be annotated is also large. The task of accurately classifying and identifying morphological anomalies while closely examining such a large number of tissue images is also enormous. As such, conventional methods of creating training data have the problem of a heavy workload for annotation, both qualitatively and quantitatively.

[0006] One embodiment of the technology disclosed herein provides an image processing device, an operating method for the image processing device, an operating program for the image processing device, and a learning method that can reduce the workload when creating learning data for a machine learning model compared to conventional methods when using a machine learning model that estimates an evaluation value based on the morphological characteristics of a tissue image.

[0007] The image processing device according to the disclosed technique includes a processor and uses a machine learning model to estimate an evaluation value according to the morphological features of tissue images obtained by segmenting a specimen image of a tissue specimen from a subject. The machine learning model is trained using a plurality of tissue image sets, each of which includes a plurality of training tissue images that are tissue images used for learning. The plurality of tissue image sets are assigned a relative ranking based on the morphological features of the training tissue images, i.e., the relative ranking between the training tissue images within each tissue image set, as teaching information. The learning involves having the machine learning model estimate an evaluation value according to the morphological features of the training tissue images, and adapting the estimated evaluation value to the relative ranking assigned in the tissue image set to which the training tissue image belongs. The processor executes an acquisition process to acquire a tissue image with an unknown evaluation value as an evaluation target, and an estimation process to estimate the evaluation value of the acquired evaluation target using the trained machine learning model.

[0008] Preferably, each of the plurality of tissue image sets includes at least one training tissue image having a different relative order.

[0009] Within one tissue image set, the relative ranking is preferably two-level.

[0010] It is preferable that at least one of the multiple tissue image sets is an image set that contains a mixture of multiple learning tissue images divided from two or more specimen images, each of which represents two or more different tissue specimens derived from one or more subjects.

[0011] The specimen image is preferably an image of a tissue specimen used in a test to evaluate at least one of the efficacy and toxicity of a substance administered to a subject.

[0012] The specimen images preferably include a first specimen image of a tissue specimen from a subject to which a substance has been administered, and a second specimen image of a tissue specimen from a subject to which a substance has not been administered, and each of the multiple tissue image sets preferably includes, as learning tissue images, a first learning tissue image divided from the first specimen image and a second learning tissue image divided from the second specimen image.

[0013] Preferably, each of the plurality of tissue image sets is comprised of only a plurality of training tissue images derived from one or more subjects on which a single test has been performed.

[0014] Within one tissue image set, the relative ranking is preferably two-level, and the two levels are distinguished by whether or not the training tissue image contains a major abnormal finding for each test that evaluates at least one of the efficacy and toxicity of a substance administered to a subject.

[0015] The relative ranking based on the morphological features is preferably based on any one of the abnormality level of the morphological features, the severity of the lesion having the morphological features, and the progression level of the lesion.

[0016] The processor preferably outputs an evaluation result based on the estimated evaluation value of the evaluation target.

[0017] The processor preferably outputs the magnitude of the evaluation value of the evaluation target as the evaluation result in a manner that allows comparison with other evaluation targets.

[0018] When the evaluation objects are multiple images divided from a single specimen image, it is preferable that the processor outputs, as the evaluation result, the magnitude of the evaluation value for each area in the specimen image corresponding to the multiple evaluation objects in an identifiable manner.

[0019] The processor preferably generates a heat map that allows the magnitude of the evaluation value for each region to be identified by the shade of color and that can be superimposed on the specimen image.

[0020] A method for operating an image processing device according to the disclosed technology is an image processing device that includes a processor and uses a machine learning model to estimate an evaluation value corresponding to the morphological features of tissue images obtained by segmenting a specimen image depicting a tissue specimen from a subject. The machine learning model is trained using a plurality of tissue image sets, each of which includes a plurality of training tissue images that are tissue images used for learning. The plurality of tissue image sets are assigned a relative ranking based on the morphological features of the training tissue images, i.e., the relative ranking between the training tissue images within each tissue image set, as teaching information. The learning involves having the machine learning model estimate an evaluation value corresponding to the morphological features of the training tissue images and adapting the estimated evaluation value to the relative ranking assigned in the tissue image set to which the training tissue image belongs. The processor executes an acquisition process that acquires a tissue image with an unknown evaluation value as an evaluation target, and an estimation process that uses the trained machine learning model to estimate the evaluation value of the acquired evaluation target.

[0021] An operating program for an image processing device according to the disclosed technology includes a processor, and causes a computer to function as an image processing device that uses a machine learning model to estimate an evaluation value according to the morphological features of tissue images obtained by segmenting a specimen image depicting a tissue specimen from a subject. The machine learning model is trained using a plurality of tissue image sets, each of which includes a plurality of training tissue images that are tissue images used for learning. The plurality of tissue image sets are assigned a relative ranking based on the morphological features of the training tissue images, i.e., the relative ranking between the training tissue images within each tissue image set, as teacher information. The learning involves having the machine learning model estimate an evaluation value according to the morphological features of the training tissue images, and adapting the estimated evaluation value to the relative ranking assigned in the tissue image set to which the training tissue image belongs. The operating program causes the computer to execute an acquisition process for acquiring a tissue image with an unknown evaluation value as an evaluation target, and an estimation process for estimating the evaluation value of the acquired evaluation target using the trained machine learning model.

[0022] The learning device according to the disclosed technology is equipped with a processor and trains a machine learning model that estimates an evaluation value according to the morphological features of tissue images obtained by subdividing a specimen image of a tissue specimen from a subject. The machine learning model is trained using a plurality of tissue image sets, each of which includes a plurality of training tissue images that are tissue images used for learning. The plurality of tissue image sets are assigned a relative ranking based on the morphological features of the training tissue images, i.e., the relative ranking between the training tissue images within each tissue image set, as training information. The learning is performed by having the machine learning model estimate an evaluation value according to the morphological features of the training tissue images, and adapting the estimated evaluation value to the relative ranking assigned in the tissue image set to which the training tissue image belongs.

[0023] According to the technology disclosed herein, when using a machine learning model that estimates an evaluation value based on the morphological characteristics of a tissue image, the workload required to create training data for the machine learning model can be reduced compared to conventional techniques.

[0024] 1 is a diagram illustrating a method for creating a specimen image, and an image processing device. FIG. 1 is a diagram illustrating an example of evaluation results output by the image processing device. FIG. 2 is a block diagram illustrating a computer constituting the image processing device. FIG. 3 is a block diagram illustrating a processor of the image processing device. FIG. 4 is a diagram illustrating an overview of processing for estimating an evaluation value from a patch image. FIG. 5 is a diagram illustrating a feature space used for estimating an evaluation value. FIG. 6 is a diagram illustrating the relationship between morphological abnormality findings in patch images and the degree of abnormality. FIG. 7 is a diagram illustrating a learning unit that trains an estimation model. FIG. 8 is a diagram illustrating labels assigned to a tissue image set. FIG. 9 is a diagram conceptually illustrating an example in which a label and an evaluation value do not match in learning. FIG. 10 is a diagram conceptually illustrating an example in which a label and an evaluation value match in learning. FIG. 11 is a diagram conceptually illustrating another example in which a label and an evaluation value match in learning. FIG. 12 is a diagram illustrating the relationship between a label indicating a relative ranking within a tissue image set and an evaluation value. FIG. 13 is a flowchart illustrating the processing procedure of the image processing device in the learning phase. FIG. 14 is a flowchart illustrating the processing procedure of the image processing device in the operation phase. FIG. 15 is a diagram illustrating an example of a display screen that displays a heat map as an evaluation result. FIG. 16 is a diagram illustrating another example of a display screen that displays a heat map as an evaluation result. FIG. 17 is a diagram illustrating a display screen that displays another aspect of the evaluation result. FIG. 18 is a diagram illustrating an example of a toxicity test report. FIG. 19 is a diagram illustrating labels with three relative rankings.

[0025] First Embodiment As shown in FIG. 1 as an example, an image processing device 10 of the present disclosure is used to evaluate the efficacy and toxicity of a drug candidate substance 11. The image processing device 10 is, for example, a desktop personal computer, and includes a display 12 that displays various screens, and an input device 13 such as a keyboard, a mouse, a touch panel, and / or a microphone for voice input. The image processing device 10 is installed, for example, in a drug development facility, and is operated by drug development staff DS who are involved in drug development at the drug development facility. The drug development staff DS also includes pathologists. The drug development staff DS is an example of a "user" related to the technology of the present disclosure.

[0026] A specimen image 15 is input to the image processing device 10. The specimen image 15 is an image used to evaluate the efficacy and toxicity of a candidate substance 11 administered to a subject S. The image processing device 10 uses a machine learning model to detect morphological abnormalities in the specimen image 15 caused by the candidate substance 11. The specimen image 15 is generated, for example, by the following procedure. First, a subject S, such as a rat, prepared for evaluation of the candidate substance 11 is autopsied, and multiple tissue specimens (hereinafter referred to as liver specimens) LVS of cross sections of the subject S's organs, in this case the liver LV, are collected. Next, the collected liver specimens LVS are attached one by one to glass slides 16, and then stained, in this case with hematoxylin and eosin dye. Next, the stained liver specimens LVS are covered with a cover glass 17 to complete a slide specimen 18. The slide specimen 18 is then placed in an imaging device 19, such as a digital optical microscope, and the specimen image 15 is captured by the imaging device 19. The specimen image 15 thus obtained is assigned a specimen ID (Identification Data) for uniquely identifying the specimen S, a specimen image ID for uniquely identifying the specimen image 15, and the date and time of capture. A tissue specimen is also called a tissue section, and the specimen image 15 is also called a WSI (Whole Slide Image) because it is an image of an entire tissue section. The specimen may be stained with hematoxylin dye alone or with nuclear fast red dye, for example.

[0027] Here, the administration group and the control group will be described. The administration group is composed of a plurality of subjects S to whom the candidate substance 11 has been administered. The control group, in contrast to the administration group, is composed of a plurality of subjects S to whom the candidate substance 11 has not been administered. When evaluating the efficacy and toxicity of the candidate substance 11, an image of the liver specimen LVS of the subjects S in the administration group is used as the specimen image 15. Of course, an image of the liver specimen LVS of the subjects S in the control group may also be used as the specimen image 15 for reference.

[0028] As will be described later, the specimen images 15 for the training data of the machine learning model used by the image processing device 10 include images of liver specimens LVS from the control group in addition to specimen images 15 of liver specimens LVS from the treatment group. The number of subjects S constituting the treatment group and the number of subjects S constituting the control group are both, for example, approximately 5 to 10. The subjects S constituting the treatment group and the subjects S constituting the control group have the same attributes and are placed in the same rearing environment. The same attributes include, for example, the same age in weeks and / or the same gender. The same attributes also include the same age composition ratio and / or the same gender composition ratio (e.g., five males and five females). The same rearing environment includes, for example, the same food, the same temperature and humidity of the rearing space, and / or the same size of the rearing space. The term "same" in the term "same rearing environment" refers not only to "completely the same" but also to "same" in the sense of including a margin of error generally acceptable in the technical field to which the technology of the present disclosure pertains, and that does not contradict the spirit of the technology of the present disclosure.

[0029] The administration group includes a plurality of groups with different doses of the candidate substance 11. For example, the dose of the candidate substance 11 is varied in three stages, namely, a high dose group, a medium dose group, and a low dose group. This makes it possible to determine the effect of the dose of the candidate substance 11 on the subject S. In FIG. 1, two administration groups are shown as an example: a low dose group and a high dose group.

[0030] As an example, as shown in FIG. 2 , the image processing device 10 recognizes a liver specimen LVS depicted in an input specimen image 15 using well-known image recognition technology and subdivides the recognized liver specimen LVS into multiple patch images 65. In other words, the patch images 65 are images obtained by dividing the area of ​​the liver specimen LVS, which is the entire tissue slice, into small sections. The patch images 65 have a predetermined size that can be handled by the machine learning model of the image processing device 10. The image processing device 10 assigns an image ID to each subdivided patch image 65 and associates the image ID with positional information indicating which portion of the specimen image 15 each patch image 65 is cut from. Note that, although the patch images 65 in FIG. 2 do not have areas that overlap with other patch images 65, the patch images 65 may partially overlap with other patch images 65.

[0031] The image processing device 10 uses a machine learning model to estimate an evaluation value 66 (see FIG. 4 , etc.) representing the degree of morphological abnormality for each of the thus subdivided patch images 65. The machine learning model of the image processing device 10 is trained to estimate an evaluation value 66 that conforms to the judgment criteria of the drug discovery staff DS, including a pathologist, for the morphological abnormality of the patch image 65. The judgment criteria of the drug discovery staff DS are criteria by which the drug discovery staff DS considers the degree of abnormality of the morphological abnormality of the patch image 65 to be high or low.

[0032] Morphological abnormalities refer to lesions not observed in normal liver specimens (LVS). Morphological abnormalities are an example of a "morphological feature" according to the technology of the present disclosure. Examples of morphological abnormality findings include "necrosis," "cellular infiltration," and "eosinophilic change." "Cellular infiltration" refers to, for example, inflammatory cell infiltration. In normal cases, no abnormal findings are observed, resulting in the lowest level of abnormality. Generally, the stronger or more severe the morphological abnormality, the greater the level of interest from the drug discovery staff DS. Because the drug discovery staff DS must observe a large number of patch images 65, the drug discovery staff DS prefers to focus on observing the highly abnormal parts of the patch images 65. Therefore, the image processing device 10 estimates an evaluation value 66 representing the degree of morphological abnormality for each patch image 65, thereby outputting an evaluation result that enables highlighted or preferential display of highly abnormal areas in the liver specimen (LVS).

[0033] The heat map HMP shown in FIG. 2 is an example of an evaluation result output by the image processing device 10. In this example, the patch images 65 to be evaluated are multiple patch images 65 divided from a single specimen image 15. The heat map HMP has an outline of the liver specimen LVS and displays the magnitude of the evaluation value 66 representing the degree of abnormality for each region of the specimen image 15 corresponding to the multiple patch images 65 in a distinguishable manner. The heat map HMP, for example, has an outline of the liver specimen LVS. The heat map HMP distinguishes the magnitude of the evaluation value 66 representing the degree of abnormality for each region of the liver specimen LVS by color shading, with higher degrees of abnormality indicated by darker colors and lower degrees of abnormality indicated by lighter colors. In this example, the color of the necrotic region, which has the highest degree of morphological abnormality, is dark, and the colors gradually become lighter in the order of necrosis, cellular infiltration, acidophilic change, and normality. This heat map HMP is an example of a display mode that highlights regions with high abnormality.

[0034] In this example, three types of morphological abnormalities are shown as examples: necrosis, cell infiltration, and eosinophilic change. Other types include hyperplasia, congestion, inflammation, tumor, canceration, proliferation, hemorrhage, and glycogen depletion.

[0035] 3, the computer constituting the image processing device 10 includes, in addition to the display 12 and input device 13, a storage 30, a memory 31, a CPU (Central Processing Unit) 32, and a communication unit 33. These are interconnected via a bus line 34.

[0036] The storage 30 is a hard disk drive built into the computer constituting the image processing device 10 or connected via a cable or network. Alternatively, the storage 30 is a disk array consisting of multiple hard disk drives. The storage 30 stores control programs such as an operating system, various application programs, and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0037] The memory 31 is a work memory for the CPU 32 to execute processing. The CPU 32 loads programs stored in the storage 30 into the memory 31 and executes processing in accordance with the programs. In this way, the CPU 32 comprehensively controls each part of the computer. The CPU 32 is an example of a "processor" according to the technology of the present disclosure. Note that the memory 31 may be built into the CPU 32. The communication unit 33 controls the transmission of various information to and from external devices such as the imaging device 19.

[0038] As an example, as shown in FIG. 4 , an operating program 40 is stored in the storage 30 of the image processing device 10. The operating program 40 is an application program for causing a computer to function as the image processing device 10. In other words, the operating program 40 is an example of an "operating program of an image processing device" according to the technology of the present disclosure. The storage 30 also stores an estimation model 41. The estimation model 41 is a machine learning model that is part of the operating program 40, and the estimation model 41 is an example of a "machine learning model" according to the technology of the present disclosure.

[0039] When the operating program 40 is started, the CPU 32 of the computer constituting the image processing device 10 cooperates with the memory 31 and the like to function as a processor 36 having a read / write (hereinafter abbreviated as RW (Read Write)) control unit 50, an estimation unit 51, a division unit 52, and an evaluation result output unit 53.

[0040] The RW control unit 50 controls the storage of various data in the storage 30 and the reading of various data from the storage 30. For example, the RW control unit 50 stores a specimen image 15 from the imaging device 19 in the storage 30. Note that multiple specimen images 15 are obtained from one subject S, and therefore multiple specimen images 15 derived from one subject S are stored in the storage 30.

[0041] The RW control unit 50 acquires a specimen image 15 corresponding to a specification by the drug discovery staff DS via the input device 13 by reading it from the storage 30. The specimen image 15 acquired by the RW control unit 50 is a target for determining, for example, whether or not morphological abnormalities have occurred in the liver specimen LVS due to the administration of the candidate substance 11. The RW control unit 50 outputs the specimen image 15 to the division unit 52. The division unit 52 subdivides the specimen image 15 and divides it into a plurality of patch images 65. The division unit 52 outputs the divided plurality of patch images 65 to the estimation unit 51.

[0042] The RW control unit 50 reads the estimation model 41 from the storage 30 and outputs the read estimation model 41 to the estimation unit 51 .

[0043] The estimation model 41 has a feature extractor 411 and an evaluation value estimator 412. The estimation unit 51 extracts a feature 60 from a patch image 65 using the feature extractor 411. The estimation unit 51 estimates an evaluation value 66 based on the feature 60 using the evaluation value estimator 412, and outputs the estimated evaluation value 66 to the evaluation result output unit 53.

[0044] Here, the patch image 65 is an example of a "tissue image with an unknown evaluation value" according to the technique of the present disclosure. The processor 46, which has the RW control unit 50 and the estimation unit 51, executes an acquisition process of acquiring the patch image 65 with an unknown evaluation value as an evaluation target, and an estimation process of estimating an evaluation value 66 of the acquired patch image 65 as an evaluation target using a trained estimation model 41 that has undergone learning, which will be described later.

[0045] The evaluation result output unit 53 generates and outputs, for example, a heat map HMP of the liver specimen LVS based on the evaluation values ​​66 of each of the plurality of patch images 65 .

[0046] As shown in FIG. 5 , the feature extractor 411 is a well-known convolutional neural network having a convolutional layer 411A and a pooling layer 411B, and extracts feature quantities 60 representing morphological features of the patch image 65. Specifically, the convolutional layer 411A uses a filter having a square coefficient matrix, e.g., 3×3, to output the sum of products of pixel values ​​of the patch image 65 and filter coefficients as feature quantities. The convolutional layer 411A then applies the filter to each region of the patch image 65 while shifting it, thereby outputting a feature map 60A as the feature quantity 60. The feature map 60A is two-dimensional data that retains the spatial features of the patch image 65 and represents the shape and texture features of the liver specimen LVS depicted in the patch image 65. The feature extractor 411 is provided with multiple different types of filters to extract various morphological features of the patch image 65, and outputs as many feature maps 60A as the number of filters. The number of feature maps 60A output corresponds to the number of filters.

[0047] The pooling layer 411B gradually reduces the size of the patch image 65 by maximum value pooling or the like according to the pixel values ​​of the patch image 65. The convolutional layer 411A applies a filter to the patch image 65 whose size is gradually reduced, thereby outputting a feature map 60A. In other words, the convolutional layer 411A outputs the feature map 60A for each patch image 65 of different sizes. This extracts various features, from fine features to global features, possessed by the patch image 65. In this way, the feature maps 60A are output in a number corresponding to the type of filter and the number of patch images 65 of different sizes.

[0048] Such a feature extractor 411 is, for example, a repurposed encoder unit of an autoencoder, which is one of the machine learning models used for images. As is well known, an autoencoder has an encoder unit that performs convolution processing and pooling processing on an input image to extract features of the input image, and a decoder unit that restores the input image from the extracted features. The encoder and decoder units are trained so that the input image can be accurately restored in the decoder unit. Accurate restoration of the input image in the decoder unit means that appropriate features of the input image have been extracted in the encoder unit. Therefore, by extracting only the encoder unit from an autoencoder that can appropriately restore the input image, it is possible to use it as the feature extractor 411.

[0049] The evaluation value estimator 412 includes, for example, a flattening layer 412A and a fully connected layer 412B. The flattening layer 412A converts the feature map 60A into an N-dimensional vector. N-dimensionality is, for example, on the order of 512 or 1024. As shown in FIG. 6 as an example, the flattening layer 412A performs a first flattening process (denoted as "flattening process 1" in FIG. 6). The feature map 60A has a size of W (width) × H (height) × C (number of channels). The first flattening process converts the feature map 60A into a vector with a number of dimensions of W × H × C. In other words, the first flattening process converts the feature map 60A into a vector with a number of dimensions equal to the number of data elements (W × H × C) of the feature map 60A.

[0050] Alternatively, the flattening layer 412A may convert the feature map 60A into an N-dimensional vector by a second flattening process (denoted as "flattening process 2" in FIG. 6). The second flattening process is, for example, global average pooling, in which case the feature map 60A is converted into a vector with 1×1×C dimensions. That is, in global average pooling, the feature map 60A with a size of W×H×C is pooled to a size of 1×1×C. The flattening layer 412A outputs the N-dimensional vector flattened by the first flattening process or the second flattening process as the feature vector 60B.

[0051] The fully connected layer 412B is configured, for example, by a neural network that combines multiple perceptrons, each of which has multiple input nodes to which multiple input values ​​are input, and an output node that outputs a single output value that is the sum of the products of the input values ​​input to each input node and a coefficient, and is a layer that connects all of the input values ​​input to all of the input nodes of the multiple perceptrons. The fully connected layer 412B receives the feature vector 60B as input and outputs the evaluation value 66, which is a single scalar value.

[0052] FIG. 7 is a diagram conceptually illustrating the relationship between the magnitude of the evaluation value 66 output by the evaluation value estimator 412 and the degree of abnormality of morphological abnormality. As shown in FIG. 7 as an example, the evaluation value estimator 412 outputs the estimated evaluation value 66 as a normalized value, for example, in the range of 1 to 100. The feature extractor 411 extracts features according to the morphological features of the patch images 65, and therefore extracts similar features for patch images 65 having similar morphological features. That is, the evaluation value estimator 412 outputs evaluation values ​​66 of approximately the same magnitude for patch images 65 having similar morphological features. The evaluation value estimator 412 is trained to output a larger evaluation value 66 for each patch image 65, as the morphological features corresponding to findings with a higher degree of abnormality of morphological abnormality are observed. That is, morphological abnormalities corresponding to findings of "necrosis," "cellular infiltration," and "eosinophilic change" have larger evaluation values ​​66, in that order.

[0053] Hereinafter, learning data and a learning method for learning the estimation model 41 that estimates such an evaluation value 66 will be described with reference to FIGS.

[0054] 8 , the processor 36 also functions as a learning unit 56 that trains the estimation model 41. The learning unit 56 includes an estimation unit 51 and a model update unit 57. The learning unit 56 trains the estimation model 41 using learning data stored in a learning data DB (Data Base) 58, and updates the estimation model 41 by correcting various parameters of the estimation model 41 using the model update unit 57. The image processing device 10 having the learning unit 56 is an example of a learning device according to the technology of the present disclosure.

[0055] The training data DB 58 stores tissue image sets TDS. A tissue image set TDS is created for each type of test, such as a toxicity test, that has been conducted. FIG. 8 illustrates a tissue image set TDS1 including a patch image 65L (see FIG. 9) from test 1, a tissue image set TDS2 including a patch image 65L from test 2, and a tissue image set TDS3 including a patch image 65L from test 3. As such, the multiple tissue image sets TDS1 to TDS3 belong to multiple test groups that differ in the type of test, such as toxicity testing. Hereinafter, when distinguishing between tissue image sets TDS by test type, a subdivision code "1" to "3" is added after the symbol "TDS." However, when no distinction is necessary, the symbol "TDS" is simply added. Because patch image 65L is a patch image of the training data, the subdivision code "L" is added to patch image 65L. In the technique of the present disclosure, the patch image 65L is an example of a "learning tissue image."

[0056] As an example, as shown in FIG. 9 , the tissue image set TDS includes a plurality of patch images 65L for learning. The patch images 65L are tissue images obtained by subdividing specimen images 15, which depict liver specimens LVS used in a test to evaluate at least one of the efficacy and toxicity of a substance administered to a subject S. In other words, the tissue image set TDS is a collection of patch images 65L for which testing has already been performed. As described above, the specimen images 15 include specimen images 15 (an example of a first specimen image according to the technology of the present disclosure) depicting liver specimens LVS from an administration group (subjects S administered with the candidate substance 11) and specimen images 15 (an example of a second specimen image according to the technology of the present disclosure) depicting liver specimens LVS from a control group (subjects S not administered with the candidate substance 11). In this example, the multiple tissue image sets TDS1 to TDS3 include both patch images 65L (an example of a first learning tissue image according to the technology of the present disclosure) divided from specimen images 15 derived from the treatment group, and patch images 65L (an example of a second learning tissue image according to the technology of the present disclosure) divided from specimen images 15 derived from the control group.

[0057] In this way, each of the multiple tissue image sets TDS1 to TDS3 is an image set that includes a mixture of multiple patch images 65L (examples of learning tissue images) that are divided from two or more specimen images 15 that respectively depict two or more different liver specimens LVS (examples of tissue specimens) derived from one or more subjects S. Furthermore, each of the multiple tissue image sets TDS1 to TDS3 is created for each type of test, and therefore is composed only of multiple patch images 65L derived from one or more subjects S on which one test was performed.

[0058] A label LB is assigned to each patch image 65L. The label LB indicates the relative ranking of the patch images 65L within each tissue image set TDS, based on morphological features. The label LB is an example of "teaching information" according to the technology of the present disclosure. In this example, the estimation model 41 outputs the degree of abnormality of the morphological abnormality of the patch image 65 as the evaluation value 66. Therefore, the label LB indicating the relative ranking of the patch image 65L in the tissue image set TDS indicates a ranking corresponding to the degree of abnormality. In other words, the relative ranking is information indicating which patch image 65L within the tissue image set TDS has a high degree of abnormality, and is not an index that can be compared between patch images 65L included in other tissue image sets TDS. In contrast, as shown in Figures 5 to 7, the evaluation value 66 is a scalar value output based on morphological features extracted from each patch image 65L, and similar morphological features have similar magnitudes. Therefore, the evaluation value 66 is an index that can be compared between patch images 65L included in different tissue image sets TDS. In this respect, the evaluation value 66 differs from the label LB, which indicates a relative ranking.

[0059] Specifically, in the tissue image set TDS1 of Test 1, the patch image 65L in which "cellular infiltration" is observed as a finding is assigned a label LB of "2." Meanwhile, the other patch images 65L, i.e., the patch image 65L in which "eosinophilic change" is observed as a finding and the normal patch image 65L in which no abnormal findings are observed, are assigned a label LB of "1." In this example, the ranking of the label LB is prioritized by a larger number. That is, "2" is ranked higher than "1." In Test 1, the morphological abnormality with the highest degree of abnormality is "cellular infiltration," and whether or not "cellular infiltration" is observed is the main abnormal finding that the drug discovery staff DS is most interested in. Therefore, in the tissue image set TDS1 of Test 1, the patch image 65L in which "cellular infiltration" is observed is assigned the highest ranking of "2" within the tissue image set TDS1. Furthermore, in Test 1, "eosinophilic changes" are considered relatively unimportant, and patch images 65L in which "eosinophilic changes" are observed as findings are given the label LB of "1," the same rank as normal patch images 65L.

[0060] Although not illustrated in this example, if there is a patch image 65L in the tissue image set TDS1 that contains findings other than "eosinophilic changes" that are less abnormal than "cellular infiltration," the label LB for that patch image 65L will also be assigned the same rank as "eosinophilic changes," with "1" assigned.

[0061] On the other hand, in Test 2, the morphological abnormality with the highest degree of abnormality is "necrosis," and whether or not "necrosis" is observed is the main abnormal finding of greatest interest to the drug discovery staff DS. Therefore, in the tissue image set TDS2 of Test 2, patch images 65L in which "necrosis" is observed are assigned the highest ranking of "2" in the tissue image set TDS2. Furthermore, "cellular infiltration" is considered relatively unimportant in Test 2, and patch images 65L in which "cellular infiltration" is observed as a finding are assigned the label LB of "1," the same ranking as normal patch images 65L. Although not shown, in the tissue image set TDS2, patch images 65L in which "eosinophilic change" is observed as a finding are also assigned the label LB of "1." In the tissue image set TDS2, patch images 65L in which findings other than "cellular infiltration," such as "eosinophilic change," which are less abnormal than "necrosis," are also assigned the label LB of "1."

[0062] Similarly, in Test 3, the morphological abnormality with the highest degree of abnormality was "eosinophilic change," and whether or not "eosinophilic change" was observed was the main abnormal finding that the drug discovery staff DS was most interested in. Therefore, in the tissue image set TDS3 of Test 3, the label LB of the patch image 65L in which "eosinophilic change" was observed as a finding was assigned "2," and the label LB of the other patch images 65L, i.e., normal patch images 65L in which no abnormal findings were observed, was assigned "1."

[0063] According to the judgment criteria of the drug discovery staff DS, the degree of abnormality of the morphological abnormality of the patch image 65 is ranked in descending order of "necrosis," "cellular infiltration," and "eosinophilic change." Although the label LB is a relative ranking within the tissue image set TDS, the label LB is assigned in a manner that conforms to the judgment criteria of the drug discovery staff DS. In other words, the label LB is not assigned independently of the morphological characteristics of the patch image 65L, but is a relative ranking based on the morphological characteristics. In this example, in all tissue image sets TDS, the ranking of "necrosis" is higher than that of other findings. Therefore, a reversal of ranking, such as a ranking higher than "cellular infiltration" in one tissue image set TDS but a ranking lower than "cellular infiltration" in another tissue image set TDS, does not generally occur. Of course, there may be exceptional cases in which a reversal of ranking occurs in some tissue image sets TDS, but even in such cases, the number of tissue image sets TDS in which a reversal of ranking does not occur should be greater. Therefore, the estimation model 41 can learn the ranking of the majority, that is, the ranking of abnormal findings that meets the judgment criteria of the drug discovery staff DS.

[0064] In this way, each of the multiple tissue image sets TDS1 to TDS3 includes at least one patch image 65L with a different relative order. In this example, within one tissue image set TDS, there are two labels LB, "2" and "1," and the relative order is two levels. Furthermore, the two levels of order are distinguished depending on whether the patch image 65L includes a major abnormal finding for each test.

[0065] 10 to 12 , a learning method by the learning unit 56 using multiple tissue image sets TDS to which such labels LB have been assigned will be described. The learning unit 56 performs ranking learning on the estimation model 41 based on the labels LB of each of the multiple tissue image sets TDS. As is well known, ranking learning is a type of machine learning and is a technique suitable for ranking multiple items, such as determining the appropriate order in which search results should be displayed. The estimation model 41 in this example aims to estimate an evaluation value 66 representing the degree of morphological abnormality for each of multiple patch images 65, and the degree of abnormality can be an index for ranking the patch images 65. Therefore, ranking learning can be applied to the estimation model 41.

[0066] In this example, the learning unit 56 causes the estimation model 41 to perform ranking learning using, for example, pairwise loss. Ranking learning using pairwise loss is a learning method that enables overall ranking of multiple items based on ranking information, i.e., which of two items selected from multiple items is more highly rated. Pairwise loss is one of the loss functions used in ranking learning, and is used to evaluate the relative importance between two items. Pairwise loss results in a larger loss for results that do not match the ranking information.

[0067] In this example, the model update unit 57 performs a loss calculation to calculate a pairwise loss based on two evaluation values ​​66 estimated by the estimation model 41. A label LB indicating a relative order within the tissue image set TDS is assigned to the patch image 65L in advance. In the loss calculation, if the relationship between the two evaluation values ​​66 does not match the relative order indicated by the label LB (in the case of incompatibility), the model update unit 57 increases the pairwise loss, and if it matches, decreases the pairwise loss. If the pairwise loss is large, the model update unit 57 updates the estimation model 41 by adjusting the parameters of the estimation model 41 so that the pairwise loss is reduced. The amount of parameter adjustment varies depending on the magnitude of the pairwise loss.

[0068] In the example shown in Fig. 10, the estimation unit 51 estimates the evaluation value 66 of the patch image 65L of "eosinophilic change," which is assigned a label LB of "1," in the tissue image set TDS1, as "40." On the other hand, the estimation unit 51 estimates the evaluation value 66 of the patch image 65L of "cellular infiltration," which is assigned a label LB of "2," as "20." In this case, the order of the magnitudes of the two evaluation values ​​66 does not match the order of the labels LB, resulting in a large pairwise loss. In this case, the amount of adjustment of the parameters of the estimation model 41 becomes large.

[0069] 11 , the estimation unit 51 estimates the evaluation value 66 of the patch image 65L of "eosinophilic change," which is assigned a label LB of "1," in the tissue image set TDS1, as "30." On the other hand, the estimation unit 51 estimates the evaluation value 66 of the patch image 65L of "cellular infiltration," which is assigned a label LB of "2," as "70." In this case, the order of the magnitudes of the two evaluation values ​​66 matches the order of the labels LB, so the pairwise loss is small. In this case, the amount of adjustment of the parameters of the estimation model 41 is small.

[0070] The learning unit 56 repeatedly performs the learning process for all combinations of two patch images 65L selected from the plurality of patch images 65L included in the tissue image set TDS1, for example, so as to reduce the pairwise loss. As a result, the estimation model 41 is trained to output an evaluation value 66 that matches the label LB for each patch image 65L in the tissue image set TDS1.

[0071] When learning using the learning data of the tissue image set TDS1 of test 1 is completed, the learning unit 56 performs learning using the learning data of the tissue image sets TDS2 and TDS3 other than test 1, for example, in the order of test 2 and test 3.

[0072] As an example, as shown in FIG. 12 , for the tissue image set TDS2 of Test 2, similarly to the tissue image set TDS1 of Test 1, the learning unit 56 estimates two evaluation values ​​66 for two patch images 65L in the tissue image set TDS2 using the estimation unit 51. Then, the model update unit 57 performs loss calculation to calculate the pairwise loss of the two evaluation values ​​66. The learning unit 56 adjusts the parameters of the estimation model 41 according to the magnitude of the calculated pairwise loss. In the example shown in FIG. 12 , the evaluation value 66 of the patch image 65L to which the label LB of "2" is assigned is "90," and the evaluation value 66 of the patch image 65L to which the label LB of "1" is assigned is "70," so that the order of the magnitudes of the evaluation values ​​66 matches the order of the labels LB. Therefore, the pairwise loss is small.

[0073] In this way, the estimation model 41 is trained to match the evaluation value 66 of the patch image 65L to the label LB of each patch image 65L in the multiple tissue image sets TDS. As also shown in FIG. 13 , training is performed for each tissue image set TDS based on the label LB. As described above, the order of the labels LB in each tissue image set TDS is assigned in a manner that matches the judgment criteria of the drug discovery staff DS (the order of abnormality levels, such as "necrosis," "cellular infiltration," and "eosinophilic change"). Therefore, by performing training to match the label LB for each tissue image set TDS, the evaluation value 66 estimated by the estimation model 41 also becomes a value that matches the order of abnormality levels in the judgment criteria of the drug discovery staff DS.

[0074] More specifically, as described above, the evaluation value 66 estimated by the estimation model 41 is a value corresponding to the morphological features of the patch image 65L, and when multiple patch images 65L have similar morphological features, the evaluation values ​​66 have approximately the same magnitude. Based on this estimation model 41, learning is performed to adapt the order of magnitudes of the evaluation values ​​66 estimated by the estimation model 41 to the relative ranking based on the label LB.

[0075] That is, through learning the tissue image sets TDS1 to TDS3 of Tests 1 to 3, the estimation model 41 also learns the correspondence between the feature amounts of the patch images 65L included in each of the tissue image sets TDS1 to TD3 and the evaluation value 66 corresponding to the degree of abnormality. For example, through learning the tissue image set TDS1 of Test 1, the estimation model 41 learns that the feature amount corresponding to the morphological abnormality of "cellular infiltration" has a higher degree of abnormality and a larger evaluation value 66 than the feature amounts corresponding to the morphological abnormality of "eosinophilic change" and "normal." Furthermore, through learning the tissue image set TDS2 of Test 2, the estimation model 41 learns that the feature amount corresponding to the morphological abnormality of "necrosis" has a higher degree of abnormality and a larger evaluation value 66 than the feature amounts corresponding to the morphological abnormality of "cellular infiltration" and "normal." Similarly, through learning the tissue image set TDS3 of Test 3, the estimation model 41 learns that the feature amount corresponding to the morphological abnormality of "eosinophilic change" has a higher degree of abnormality and a larger evaluation value 66 than the feature amounts corresponding to "normal."

[0076] Through this learning, the estimation model 41 directly compares the feature values ​​for each combination of "cellular infiltration" and "eosinophilic change and normal," "necrosis" and "cellular infiltration and normal," and "eosinophilic change" and "normal." The estimation model 41 learns the relative merits and demerits of the evaluation values ​​66 for each feature value for each combination. As a result, even for combinations such as "necrosis" and "eosinophilic change," which are not directly compared because the associated tissue image sets TDS are different, the estimation model 41 can learn the relative merits and demerits of the evaluation values ​​66 for each feature value. That is, as shown in the lower part of FIG. 13 , the estimation model 41 learns such that the evaluation value 66 for the feature value of "necrosis" is the largest, followed by "cellular infiltration," "eosinophilic change," and "normal," in that order. Therefore, the evaluation values ​​66 are comparable among multiple patch images 65L that belong to different tissue image sets TDS.

[0077] Furthermore, as described above, in each tissue image set TDS, the label LB is assigned so as to conform to the judgment criteria of the drug discovery staff DS, and therefore the magnitude of the evaluation value 66 is determined in order of conformance to the judgment criteria of the drug discovery staff DS.

[0078] Note that while Figure 13 shows a range of the evaluation value 66 corresponding to each finding, this is shown for convenience of explanation, and the boundary region of each finding is not clearly defined. This is because the estimation model 41 in this example estimates a scalar value, the evaluation value 66, and does not identify the name of the finding. As described above, the estimation model 41 outputs evaluation values ​​66 of approximately the same magnitude for similar morphological abnormalities. As a result, the range of the evaluation value 66 is determined according to the degree of abnormality of each finding, as shown in Figure 13.

[0079] Next, the operation of the above configuration will be described with reference to the flowcharts shown in FIGS. 14 and 15 as an example. First, in the learning phase, the processor 36 of the image processing device 10 executes a learning process according to the processing procedure shown in the flowchart in FIG. 14. In step ST1000, the processor 36 acquires two patch images 65L from one tissue image set TDS. In step ST1100, the processor 36 estimates the evaluation values ​​66 of each of the acquired two patch images 65L. In step ST1200, the processor 36 performs a pairwise loss calculation based on the evaluation values ​​66 of the two patch images 65L and the label LB. In step ST1300, the processor 36 updates the estimation model 41 so that the evaluation values ​​66 match the label LB. More specifically, the processor 36 adjusts the parameters of the estimation model 41 according to the magnitude of the pairwise loss.

[0080] In step ST1400, if there is an unlearned combination of patch images 65L in the tissue image set TDS (YES in step ST1400), the processor 36 returns to step ST1000. Then, for a new combination of two patch images 65L, the processor 36 repeats the processes from step ST1000 onwards. In step ST1400, if there is no unlearned patch image 65L in the tissue image set TDS (NO in step ST1400), the processor 36 proceeds to step ST1500.

[0081] In step ST1500, if there is an unlearned tissue image set TDS (YES in step ST1500), the processor 36 returns to step ST1000 and repeats the processes from step ST1000 onwards for the unlearned tissue image set TDS. The processor 36 repeats the processes from step ST1000 to step ST1400 until there is no unlearned tissue image set TDS.

[0082] In the example shown in Figure 14, the learning of the estimation model 41 is repeated with the termination condition being that there are no unlearned patch images 65L and tissue image sets TDS. However, the termination condition for learning may be other conditions, such as until the pairwise loss becomes equal to or less than a preset threshold.

[0083] In the operation phase after the learning phase, the processor 36 performs evaluation processing of the specimen image 15 according to the processing procedure shown in the flowchart of FIG. 15 . In the operation phase, for example, the drug discovery staff DS evaluates the toxicity of the specimen image 15 of the subject S administered with the candidate substance 11 using the image processing device 10. In step ST2000, the processor 36 acquires one specimen image 15 selected by the drug discovery staff DS. In step ST2100, the processor 36 subdivides the specimen image 15 into multiple patch images 65 of a preset size. In step ST2200, the processor 36 estimates an evaluation value 66 for each divided patch image 65. In step ST2300, if there are unevaluated patch images 65 among the divided patch images 65 (YES in step ST2300), the processor 36 returns to step ST2000 and repeats the processing from step ST2000 onward. If there are no unevaluated patch images 65 (NO in step ST2300), the processor 36 proceeds to step ST2400.

[0084] In step ST2400, the processor 36 outputs the evaluation result of the specimen image 15. Specifically, the processor 36 generates a heat map HMP based on the evaluation value 66 for the liver specimen LVS of the specimen image 15, which is the source of division of the patch image 65, and outputs the heat map HMP as the evaluation result. Since the evaluation value 66 is an index of the degree of abnormality of morphological abnormality, the heat map HMP displays the degree of abnormality for each region of the liver specimen LVS in a distinguishable manner, as shown in FIG. 2 .

[0085] In step ST2500, if there is another specimen image 15 to be evaluated, the processor 36 returns to step ST2000 and repeats the processes from step ST2000 onwards. If there is no another specimen image 15 to be evaluated, the processor 36 ends the operation phase.

[0086] 16 shows an example of a display screen 71 that displays the evaluation results. The display screen 71 displays a heat map HMP and a legend that indicates the correspondence between the shade of color and the level of abnormality.

[0087] 17, the heat map HMP may be displayed in a manner that allows it to be superimposed on the liver specimen LVS so that the correspondence between the liver specimen LVS of the specimen image 15 and each region of the heat map HMP can be seen at a glance. The liver specimen LVS and the heat map HMP may be switchable between superimposed display and separate display. The example of FIG. 17 shows a separate display of the liver specimen LVS and the heat map HMP while maintaining the correspondence between each region of the liver specimen LVS and the heat map HMP. Separate display makes it easy to confirm the morphological characteristics of the liver specimen LVS.

[0088] Furthermore, the evaluation results do not necessarily have to be displayed as a heat map HMP. For example, as shown in FIG. 18 , evaluated patch images 65 may be displayed in order of abnormality level corresponding to the evaluation value 66. Note that in FIG. 18 , the names of the findings, "necrosis," "cellular infiltration," and "eosinophilic change," are written on the display screen 17 within the dashed rectangular frame. However, this is for convenience of explanation, and the names of the findings are not displayed on the display screen 71. Of course, the names of the findings may be displayed on the display screen 71. In this case, for example, in addition to the estimation model 41 that estimates the evaluation value 66 based on the patch images 65, a finding name estimation model that estimates the names of the findings is required.

[0089] The number of patch images 65 displayed may be all evaluated patch images 65, or may be limited to a preset number, such as the top 100. When the heat map HMP is not displayed, it is preferable to display the specimen image 15 alongside the patch images 65 and display the correspondence between the patch images 65 and the regions of the liver specimen LVS, as shown in FIG. 18 . For example, when a patch image 65 is designated with an input device such as a mouse, the correspondence is displayed by an arrow connecting the designated patch image 65 to the corresponding region in the liver specimen LVS. Even in the configuration shown in FIG. 18 , it is possible to display the magnitude of the evaluation value 66 of the patch image 65 to be evaluated in a manner that allows comparison with other patch images 65 to be evaluated.

[0090] As described above, the image processing device 10 according to the technique of the present disclosure is an image processing device that includes a processor 36 and uses an estimation model 41 (an example of a machine learning model) to estimate evaluation values ​​66 corresponding to morphological features of patch images 65 (an example of a tissue image) obtained by dividing a specimen image 15 depicting a tissue specimen of a subject 2. The estimation model 41 is trained using a plurality of tissue image sets TDS, each of which includes a plurality of patch images 65L (an example of a training tissue image) used for training. The plurality of tissue image sets TDS are assigned, as training information, a relative order based on the morphological features of the patch images 65L, i.e., a relative order among the patch images 65L within each tissue image set TDS. The training involves having the estimation model 41 estimate evaluation values ​​66 corresponding to the morphological features of the patch images 65L, and adapting the estimated evaluation values ​​66 to the relative orders assigned in the tissue image sets TDS to which the patch images 65L belong. The processor 36 executes an acquisition process to acquire a patch image 65 having an unknown evaluation value 66 as an evaluation target, and an estimation process to estimate the evaluation value 66 of the acquired evaluation target using a trained estimation model 41.

[0091] This makes it possible to reduce the workload when creating learning data for a machine learning model compared to conventional methods when using a machine learning model that estimates an evaluation value based on the morphological characteristics of a tissue image.

[0092] As described above, when training a machine learning model such as the estimation model 41 that estimates evaluation values ​​for multiple types of morphological anomalies, a conventional method for creating training data involves ranking the abnormality levels of morphological anomalies using a uniform scale and annotating tissue images with annotations corresponding to the ranked abnormality levels. However, there are many different types of morphological anomalies, and classifying and identifying these diverse morphological anomalies requires specialized knowledge. Furthermore, the criteria for determining the abnormality level vary from person to person. Therefore, classifying and identifying the abnormality levels of all morphological anomalies using a uniform scale is extremely difficult in terms of accuracy and consistency of judgment. This is especially difficult when annotation is performed by an operator with little knowledge of morphological anomalies. Furthermore, because a large number of tissue images are required as training data, the number of tissue images to be annotation-targeted increases. The task of accurately classifying and identifying morphological anomalies while closely examining such a large number of tissue images is also enormous. As such, the conventional method for creating training data suffers from the problem of a heavy annotation workload, both qualitatively and quantitatively.

[0093] In the image processing device 10 according to the technology of the present disclosure, the labels LB, which are teaching information, are assigned based on the relative ranking within the tissue image set TDS, rather than the degree of abnormality itself, which requires classification and identification on a unified scale. Therefore, in the annotation work, it is sufficient to focus only on the main morphological features. This reduces the workload of the annotation work compared to conventional methods.

[0094] This effect can be explained more specifically as follows. Consider the task of an annotation worker looking at patch images 65L and assigning labels LB according to the degree of abnormality. In this case, one of the sources on which the annotation worker determines the degree of abnormality is, for example, a toxicity test report. Reports often include only the main abnormal findings considered by the pathologist for each test, along with the tissue image to be evaluated. For example, in the report 76 shown in FIG. 19 , "necrosis" is the main abnormal finding, so only the main abnormal finding, such as "necrosis observed," is listed. However, the liver specimen LVS listed in the report 76 may contain abnormal findings such as "cellular infiltration" in addition to "necrosis." In this case, the annotation worker assigns an abnormality level label corresponding to "necrosis" to patch images 65L obtained by subdividing the specimen image 15 containing the liver specimen LVS, for patch images 65L in which the main abnormal finding, "necrosis," is observed. On the other hand, since "cellular infiltration" is not listed in the report 76, no label is assigned to patch images 65L in which "cellular infiltration" is observed. In conventional training data, effective training cannot be performed unless all morphological abnormalities are assigned different labels, and therefore, if a morphological abnormality is overlooked, effective training cannot be performed.

[0095] In contrast, in the image processing device 10 according to the technology of the present disclosure, for example, based on the description in the report 76, a label LB of "2" indicating a relatively superior ranking is assigned to a patch image 65L in which "necrosis," which is a major abnormal finding, is observed. A label LB of "1" indicating a relatively inferior ranking is assigned to other patch images 65L. In other words, in the image processing device 10 according to the technology of the present disclosure, it is acceptable for morphological abnormalities other than the major abnormal finding to be overlooked. Therefore, the workload of the annotation work is reduced both qualitatively and quantitatively.

[0096] 10 to 12, learning is performed so that the evaluation value 66 conforms to the order indicated by the label LB. As a result, as shown in FIG. 13, the estimation model 41 can estimate an evaluation value 66 that is comparable between patch images 65L, which are examples of tissue images in the tissue image set TDS, as the evaluation value 66 of the patch image 65, which is an example of the tissue image to be evaluated. In short, the technology disclosed herein performs weakly supervised learning to train a machine learning model using training data having incomplete labels LB compared to conventional training data creation methods, thereby achieving the effect of reducing the workload when creating training data.

[0097] Furthermore, in the above embodiment, each of the multiple tissue image sets TDS includes at least one patch image 65L (an example of a tissue image) having a different relative order. For example, each tissue image set TDS includes at least one patch image 65L with a label LB of "2" and one with a label LB of "1." This improves learning efficiency compared to when a tissue image set TDS containing only patch images 65L (an example of a tissue image) with the same relative order is used for learning.

[0098] In the above embodiment, the relative order within one tissue image set TDS is two levels, which makes it easier to assign a label LB (an example of training information) compared to when the relative order is three or more levels.

[0099] In the above embodiment, an example has been described in which a label LB of "2" is assigned to one of the two-level rankings and a label LB of "1" is assigned to the other. However, in the case of a two-level ranking, it is also possible to assign a label LB of "2" to one and omit the label LB from the other. In this way, it is only necessary to assign the label LB to only one of the patch images 65 with different rankings, which further reduces the workload of the annotation work.

[0100] In the above embodiment, the two-stage ranking is determined based on whether the patch image 65L (an example of a tissue image) includes the major abnormal findings for each test described above. Therefore, in the annotation process, for example, it is easy to assign a label LB (an example of training information) based on a report 76 that includes only the major abnormal findings as shown in FIG. 19 .

[0101] Each of the multiple tissue image sets TDS contains a mixture of multiple patch images 65L (examples of tissue images) divided from two or more specimen images 15, each of which depicts two or more different liver specimens LVS (examples of tissue specimens) derived from one or more subjects S on whom a single test was performed. Since morphological abnormalities are estimated for the multiple tissue specimens, generalization performance for estimating morphological abnormalities in unknown tissue specimens may be improved.

[0102] In the above embodiment, the specimen image 15 is an image of a liver specimen LVS (an example of a tissue specimen) used in a test to evaluate at least one of the efficacy and toxicity of a candidate substance 11 (an example of a substance) to be administered to a subject S. This improves the ability to estimate morphological abnormalities in the evaluation of tests in drug discovery.

[0103] Furthermore, in the above embodiment, the specimen images 15 include specimen images 15 (an example of a first specimen image) derived from an administration group, which are images of a liver specimen LVS (an example of a tissue specimen) from a subject S to whom the candidate substance 11 (an example of a substance) has been administered, and specimen images 15 (an example of a second specimen image) derived from a control group, which are images of a liver specimen LVS (an example of a tissue specimen) from a subject S to whom the candidate substance 11 has not been administered, and each of the multiple tissue image sets TDS includes a patch image 65L (an example of a tissue image) divided from the specimen image 15 derived from the administration group, and a patch image 65L (an example of a tissue image) divided from the specimen image 15 derived from the control group. In this way, the tissue image sets include tissue images derived from the administration group and tissue images derived from the control group, making it possible to efficiently learn about morphological abnormalities caused by the efficacy or toxicity of a substance.

[0104] In the above embodiment, each of the multiple tissue image sets TDS is an image set belonging to one of multiple test groups with different types of tests, which enables evaluation of various morphological abnormalities observed in multiple tests.

[0105] In the above embodiment, the relative ranking based on the morphological characteristics is the degree of abnormality of morphological abnormality, which is an example of a morphological characteristic. The degree of abnormality of morphological abnormality is an index that is generally noted in tests such as toxicity tests. Therefore, estimating the evaluation value 66 that represents such an abnormality degree is highly useful.

[0106] Note that the relative ranking based on morphological characteristics may be, in addition to the degree of abnormality, for example, the severity of a lesion having morphological characteristics, or the progression of the lesion. Since the severity and progression of a lesion are also indicators that receive attention in testing, estimating an evaluation value 66 that represents the severity or progression is highly useful.

[0107] Furthermore, in the above embodiment, the processor 36 outputs an evaluation result based on an evaluation value 66 estimated for a patch image 65 (an example of a tissue image to be evaluated). Because the processor 36 has such an output function, it is not necessary to provide an output device separate from the image processing device 10. Note that output modes include display on a display, output to a file, and, in the case where the image processing device 10 is a server, distribution to a client terminal, etc.

[0108] Furthermore, in the above embodiment, the processor 36 outputs, as the evaluation result, the magnitude of the evaluation value 66 for a patch image 65 (an example of a tissue image to be evaluated) in a format that allows comparison with other patch images 65. In the above embodiment, the processor 36 outputs the evaluation result in the format of a heat map HMP as shown in Fig. 16 , or outputs the patch images 65 in a format in which the evaluation values ​​66 representing the degree of abnormality are arranged in order as shown in Fig. 18 . This makes it easy to distinguish between patch images 65 with different evaluation values ​​66.

[0109] Furthermore, in the above embodiment, when the patch image 65 (an example of an evaluation target) is a plurality of images divided from a single specimen image 15, the processor 36 outputs, as an evaluation result, a heat map HMP as an example, in which the magnitude of the evaluation value 66 for each region in the specimen image 15 corresponding to the plurality of patch images 65 is distinguishable. This makes it easy to distinguish regions in the specimen image 15 that have different evaluation values ​​66. Note that, although the heat map HMP is shown as a mode in which the magnitude of the evaluation value 66 is distinguishable, and the difference in evaluation value 66 is represented by a shade of color, a mode in which, instead of a shade of color, a numerical value of the evaluation value 66 is assigned to each region and displayed may also be used.

[0110] 16 and 17 , the processor 36 generates a heat map HMP that allows the magnitude of the evaluation value 66 for each region to be identified by the color shading and that can be superimposed on the specimen image 15. As shown in FIG. 17 , the heat map HMP can be superimposed on the liver specimen LVS of the specimen image 15. This makes it easy to identify regions with different evaluation values ​​66 in the liver specimen LVS of the specimen image 15. Note that instead of superimposing the heat map HMP on the specimen image 15, it may be displayed in parallel. Even when displayed in parallel, it is possible to identify the magnitude of the evaluation value 66 for each region in the liver specimen LVS of the specimen image 15.

[0111] (Modification 1) In the above embodiment, as shown in Fig. 9, an example of two levels of relative ranking as teacher information has been described, but as shown in Fig. 20, the relative ranking may be three or more levels. In the example shown in Fig. 20, three labels LB, "3," "2," and "1," are assigned according to the degree of abnormality of the morphological abnormality. In Modification 1, the larger the number in the relative ranking, the more important it is.

[0112] Furthermore, the organ is not limited to the liver, etc., but may be the stomach, lung, small intestine, large intestine, etc. Furthermore, the subject S is not limited to a rat, but may be a mouse, guinea pig, gizzard shad, hamster, ferret, rabbit, dog, cat, monkey, etc.

[0113] The image processing device 10 may be a personal computer installed in a pharmaceutical development facility as shown in FIG. 1, or may be a server computer installed in a data center independent of the pharmaceutical development facility.

[0114] When the image processing device 10 is configured as a server computer, the specimen images 15 are transmitted from personal computers installed in each pharmaceutical development facility to the server computer via a network such as the Internet. The server computer distributes various screens, such as the display screen 71, to the personal computer in the form of screen data for web distribution created using a markup language such as XML (Extensible Markup Language). The personal computer reproduces the screen to be displayed on the web browser based on the screen data and displays it on a display. Note that other data description languages, such as JSON (Javascript (registered trademark) Object Notation), may be used instead of XML.

[0115] The image processing device 10 according to the technique of the present disclosure can be widely used across all stages of pharmaceutical development, from the earliest stage of drug discovery target setting to the final stage of clinical trials.

[0116] The hardware configuration of the computer that constitutes the image processing device 10 according to the technology of the present disclosure can be modified in various ways. For example, the image processing device 10 can be configured with multiple computers separated as hardware in order to improve processing power and reliability. For example, the functions of the processor 36 can be distributed to two computers. In this case, the image processing device 10 is configured with two computers.

[0117] In this way, the hardware configuration of the computer of the image processing device 10 can be changed as appropriate depending on the required performance such as processing power, safety, reliability, etc. Furthermore, not only the hardware but also application programs such as the operating program 40 can be duplicated or stored in a distributed manner across multiple storage devices in order to ensure safety and reliability.

[0118] In each of the above embodiments, the following various processors can be used as the hardware structure of a processing unit such as the processor 36 that performs various processes. The various processors include the CPU 32, which is a general-purpose processor that executes software (operating program 40) and functions as various processing units, as described above, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0119] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA).Furthermore, multiple processing units may be configured with a single processor.

[0120] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0121] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0122] From the above description, the technology described in the following supplementary paragraphs can be understood.

[0123] [Supplementary Item 1] An image processing device including a processor and using a machine learning model to estimate an evaluation value according to morphological features of tissue images obtained by segmenting a specimen image of a tissue specimen from a subject, wherein the machine learning model is trained using a plurality of tissue image sets each including a plurality of training tissue images that are tissue images used for training, and the plurality of tissue image sets are assigned a relative order based on the morphological features of the training tissue images, the relative order between the training tissue images within each tissue image set, as teacher information, wherein the training is a learning process in which the machine learning model estimates an evaluation value according to the morphological features of the training tissue images and adapts the estimated evaluation value to the relative order assigned in the tissue image set to which the training tissue image belongs, and the processor executes an acquisition process in which a tissue image having an unknown evaluation value is acquired as an evaluation target, and an estimation process in which the trained machine learning model is used to estimate an evaluation value for the acquired evaluation target. [Supplementary Item 2] The image processing device according to Supplementary Item 1, wherein each of the plurality of tissue image sets includes at least one tissue image having a different relative order. [Supplementary Item 3] The image processing device according to Supplementary Item 2, wherein within one tissue image set, there are two levels of relative ranking. [Supplementary Item 4] The image processing device according to any one of Supplementary Item 1 to Supplementary Item 3, wherein at least one of the plurality of tissue image sets is an image set containing a mixture of a plurality of training tissue images divided from two or more specimen images each of two or more different tissue specimens derived from one or more subjects. [Supplementary Item 5] The image processing device according to Supplementary Item 4, wherein the specimen images are images of tissue specimens used as tissue specimens in a test to evaluate at least one of the efficacy and toxicity of a substance administered to the subject. [Supplementary Item 6] The image processing device described in Supplementary Item 5, wherein the specimen images include a first specimen image of a tissue specimen of a subject to which a substance has been administered and a second specimen image of a tissue specimen of a subject to which a substance has not been administered, and each of the plurality of tissue image sets includes, as learning tissue images, a first learning tissue image divided from the first specimen image and a second learning tissue image divided from the second specimen image.[Supplementary Item 7] The image processing device of Supplementary Item 6, wherein each of the plurality of tissue image sets consists of only a plurality of training tissue images derived from one subject for which one test was performed. [Supplementary Item 8] The image processing device of Supplementary Item 7, wherein the two-stage ranking is distinguished based on whether or not the training tissue images include a major abnormal finding for each test evaluating at least one of the efficacy and toxicity of a substance administered to the subject. [Supplementary Item 9] The image processing device of any one of Supplementary Item 1 to Supplementary Item 9, wherein the relative ranking based on morphological features is given based on any of the degree of abnormality of the morphological features, the severity of a lesion having the morphological features, and the stage of progression of the lesion. [Supplementary Item 10] The image processing device of any one of Supplementary Item 1 to Supplementary Item 9, wherein the processor outputs an evaluation result based on an evaluation value estimated for the evaluation object. [Supplementary Item 11] The image processing device of Supplementary Item 10, wherein the processor outputs, as the evaluation result, the magnitude of the evaluation value of the evaluation object in a manner that allows comparison with other evaluation objects. [Supplementary Item 12] The image processing device according to Supplementary Item 11, wherein when the evaluation targets are multiple images divided from a single specimen image, the processor outputs, as the evaluation result, the magnitude of the evaluation value for each region in the specimen image corresponding to the multiple evaluation targets in a distinguishable manner. [Supplementary Item 13] The image processing device according to Supplementary Item 12, wherein the processor generates a heat map in which the magnitude of the evaluation value for each region is distinguishable by color shading and which can be superimposed on the specimen image.[Supplementary Item 14] A method for operating an image processing device including a processor, the method using a machine learning model to estimate an evaluation value according to morphological features of tissue images obtained by segmenting a specimen image showing a tissue specimen of a subject, wherein the machine learning model is trained using a plurality of tissue image sets each including a plurality of training tissue images that are tissue images used for training, and the plurality of tissue image sets are assigned with teacher information that is a relative order based on the morphological features of the training tissue images, whereby the relative order between the training tissue images within each tissue image set is assigned, and the training is a learning process in which the machine learning model estimates an evaluation value according to the morphological features of the training tissue images and adapts the estimated evaluation value to the relative order assigned in the tissue image set to which the training tissue image belongs, and the processor executes an acquisition process in which a tissue image with an unknown evaluation value is acquired as an evaluation target, and an estimation process in which the trained machine learning model is used to estimate an evaluation value of the acquired evaluation target. [Supplementary Item 15] An operating program for an image processing device that includes a processor and causes a computer to function as an image processing device that uses a machine learning model to estimate an evaluation value according to morphological features of tissue images obtained by segmenting a specimen image of a tissue specimen from a subject, wherein the machine learning model is trained using a plurality of tissue image sets that each include a plurality of training tissue images that are tissue images used for training, and the plurality of tissue image sets are assigned with teacher information that is a relative order based on the morphological features of the training tissue images, whereby the relative order between the training tissue images within each tissue image set is assigned, and the training is a learning that causes the machine learning model to estimate an evaluation value according to the morphological features of the training tissue images and adapts the estimated evaluation value to the relative order assigned in the tissue image set to which the training tissue image belongs, and the operating program for an image processing device that causes a computer to execute an acquisition process that acquires a tissue image with an unknown evaluation value as an evaluation target, and an estimation process that uses the trained machine learning model that has been trained to estimate an evaluation value for the acquired evaluation target.[Supplementary Item 16] A learning device including a processor for training a machine learning model that estimates an evaluation value according to the morphological features of tissue images obtained by segmenting a specimen image showing a tissue specimen of a subject, wherein the machine learning model is trained using a plurality of tissue image sets each including a plurality of training tissue images that are tissue images used for training, and the plurality of tissue image sets are assigned a relative ranking based on the morphological features of the training tissue images, i.e., a relative ranking between the training tissue images within each tissue image set, as teaching information, and the learning is a learning process in which the machine learning model estimates an evaluation value according to the morphological features of the training tissue images, and the estimated evaluation value is adapted to the relative ranking assigned in the tissue image set to which the training tissue image belongs.

[0124] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs but also to storage media that non-temporarily store programs.

[0125] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0126] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0127] The disclosure of Japanese Patent Application No. 2023-075855, filed on May 1, 2023, is incorporated herein by reference in its entirety. In addition, all documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An image processing device having a processor and estimating an evaluation value according to morphological features of tissue images obtained by subdividing a specimen image of a tissue specimen from a subject using a machine learning model, wherein the machine learning model is trained using a plurality of tissue image sets including a plurality of training tissue images which are the tissue images used for training, and a relative order based on the morphological features of the training tissue images, which is a relative order between the training tissue images within each of the tissue image sets, is assigned as teacher information to the plurality of tissue image sets, and the training is a training in which the machine learning model estimates an evaluation value according to the morphological features of the training tissue images and adapts the estimated evaluation value to the relative order assigned to the tissue image set to which the training tissue image belongs, and the processor executes an acquisition process of acquiring the tissue image as an evaluation target, the evaluation value of which is unknown, and an estimation process of estimating the evaluation value of the acquired evaluation target using the trained machine learning model that has been trained.

2. The image processing device according to claim 1, wherein each of the plurality of tissue image sets includes at least one learning tissue image having a different relative order.

3. The image processing apparatus according to claim 2, wherein within one tissue image set, the relative order is two-level.

4. The image processing device of claim 1, wherein at least one of the multiple tissue image sets is an image set that includes a mixture of multiple learning tissue images divided from two or more specimen images each of two or more different tissue specimens derived from one or more of the subjects.

5. The image processing device according to claim 4, wherein the specimen image is an image of a tissue specimen used as the tissue specimen in a test for evaluating at least one of the efficacy and toxicity of a substance administered to the subject.

6. The image processing device of claim 5, wherein the specimen images include a first specimen image of the tissue specimen of the subject to which the substance has been administered and a second specimen image of the tissue specimen of the subject to which the substance has not been administered, and each of the multiple tissue image sets includes, as the learning tissue images, a first learning tissue image divided from the first specimen image and a second learning tissue image divided from the second specimen image.

7. The image processing device according to claim 6, wherein each of the plurality of tissue image sets is composed of only a plurality of training tissue images derived from one or more of the subjects on which one of the tests was performed.

8. The image processing device according to claim 7, wherein within one tissue image set, the relative ranking has two levels, and the two levels of ranking are distinguished depending on whether or not the learning tissue image contains the main abnormal finding for each test.

9. The image processing device according to claim 1, wherein the relative ranking based on the morphological feature is given based on any one of the degree of abnormality of the morphological feature, the severity of a lesion having the morphological feature, and the progression of the lesion.

10. The image processing device according to claim 1, wherein the processor outputs an evaluation result based on the evaluation value estimated for the evaluation object.

11. The image processing device according to claim 10, wherein the processor outputs, as the evaluation result, the magnitude of the evaluation value of the evaluation target in a form that allows comparison with other evaluation targets.

12. The image processing device according to claim 10, wherein when the evaluation objects are multiple images divided from a single specimen image, the processor outputs, as the evaluation result, the magnitude of the evaluation value for each area in the specimen image corresponding to the multiple evaluation objects in an identifiable manner.

13. The image processing device according to claim 12, wherein the processor generates a heat map in which the magnitude of the evaluation value for each of the regions can be identified by shades of color and which can be superimposed on the specimen image.

14. A method for operating an image processing device including a processor and estimating, using a machine learning model, an evaluation value corresponding to morphological features of tissue images obtained by dividing a specimen image showing a tissue specimen of a subject, wherein the machine learning model is trained using a plurality of tissue image sets including a plurality of training tissue images which are the tissue images used for learning, and a relative order based on the morphological features of the training tissue images, which is a relative order between the training tissue images within each of the tissue image sets, is assigned as teacher information to the plurality of tissue image sets, wherein the learning is to have the machine learning model estimate an evaluation value corresponding to the morphological features of the training tissue images, and to adapt the estimated evaluation value to the relative order assigned in the tissue image set to which the training tissue image belongs, and wherein the processor executes an acquisition process for acquiring the tissue image, the evaluation value of which is unknown, as an evaluation target, and an estimation process for estimating the evaluation value of the acquired evaluation target using the trained machine learning model that has been trained.

15. An operating program for an image processing device, comprising a processor, causing a computer to function as an image processing device that estimates an evaluation value according to morphological features of tissue images obtained by dividing a specimen image of a tissue specimen of a subject using a machine learning model, wherein the machine learning model is trained using a plurality of tissue image sets including a plurality of training tissue images which are the tissue images used for training, and a relative order based on the morphological features of the training tissue images, which is a relative order between the training tissue images within each of the tissue image sets, is assigned as teacher information, the training is a training in which the machine learning model estimates an evaluation value according to the morphological features of the training tissue images, and the estimated evaluation value is adapted to the relative order assigned in the tissue image set to which the training tissue images belong, and the operating program for an image processing device causes the computer to execute an acquisition process of acquiring the tissue image with an unknown evaluation value as an evaluation target, and an estimation process of estimating the evaluation value of the acquired evaluation target using the trained machine learning model.

16. A learning device having a processor, which trains a machine learning model that estimates an evaluation value corresponding to morphological features of tissue images obtained by dividing a specimen image of a tissue specimen from a subject, wherein the machine learning model is trained using a plurality of tissue image sets each including a plurality of training tissue images which are the tissue images used for learning, and the plurality of tissue image sets are assigned a relative order based on the morphological features of the training tissue images, the relative order between the training tissue images within each of the tissue image sets, as teaching information, and the learning is performed by having the machine learning model estimate an evaluation value corresponding to the morphological features of the training tissue images, and adapting the estimated evaluation value to the relative order assigned in the tissue image set to which the training tissue images belong.

Citation Information

Patent Citations

  • Door unit

    JP2023075855A

  • Automated clustering of anomalous histopathology tissue samples

    US20210342570A1

  • Assessing heterogeneity of features in digital pathology images using machine learning techniques

    WO2022015819A1

  • Dynamically prioritizing, organizing and displaying cells and tissue in order of their diagnostic significance

    WO2022170235A1