Image processing device, method for operation of image processing device, and program for operation of image processing device

Abstract

An image processing device comprising a processor, wherein the processor: acquires a first sample image in which a tissue sample of a subject appears; uses a machine learning model to extract first feature amounts from first patch images obtained by subdividing the first sample image; determines, on the basis of the first feature amounts, whether there is a malformation in the tissue sample appearing in the first patch images; and performs either a manual clustering process, in which the specification of which cluster each of the first patch images in which there was determined to be a malformation in the tissue sample belongs is received from a user and the first patch images are clustered into a plurality of clusters on the basis of the specifications, or a soft clustering process, in which the degree of attribution to each of the plurality of clusters is calculated for each of the first patch images in which there was determined to be a malformation in the tissue sample.

Description

Image processing device, operation method of image processing device, and operation program of image processing device

[0001] The technology of the present disclosure relates to an image processing device, an operating method for an image processing device, and an operating program for an image processing device.

[0002] The specification of U.S. Patent Application Publication No. 2021 / 0342570 describes a technology that uses a machine learning model such as an autoencoder to extract features from patch images that are created by dividing a specimen image containing a tissue specimen such as an animal liver, and based on the extracted features, determines whether the tissue specimen depicted in the patch image has a morphological abnormality (hyperplasia, infiltration, congestion, inflammation, tumor, canceration, proliferation, bleeding, glycogen depletion, etc.), and clusters (hard clusters) patch images that are determined to have a morphological abnormality into one of multiple clusters based on the features.

[0003] To determine whether a morphological abnormality has occurred, image information of not only the area where the morphological abnormality has occurred but also the surrounding area is required. For example, when determining whether bile duct hyperplasia has occurred as a morphological abnormality, even a specialized pathologist has difficulty determining whether the hyperplasia is present simply by observing the hyperplastic bile duct portion. Therefore, it is necessary to observe the area with a wider field of view, including the area and its surrounding area. For this reason, in the specification of U.S. Patent Application Publication No. 2021 / 0342570, it is desirable to input patch images to a machine learning model that extracts features in a size that covers not only the area of ​​the morphological abnormality but also the surrounding area.

[0004] However, when the patch image size is set as described above, the feature values ​​are derived not only from the morphologically abnormal portion but also from the surrounding area. Therefore, the distribution of the feature values ​​of the patch image determined to have a morphological abnormality is not discrete. If the distribution of the feature values ​​is not discrete, it is difficult to perform hard clustering of the patch image determined to have a morphological abnormality by computer processing, as in U.S. Patent Application Publication No. 2021 / 0342570.

[0005] One embodiment of the technology disclosed herein provides an image processing device, an operating method for the image processing device, and an operating program for the image processing device that are capable of clustering patch images of a sample image whose feature distribution is not discrete.

[0006] The image processing device of the present disclosure includes a processor, which acquires a first specimen image in which a tissue specimen of a subject is captured, extracts a first feature amount from a first patch image obtained by subdividing the first specimen image using a machine learning model, determines whether or not a morphological abnormality has occurred in the tissue specimen captured in the first patch image based on the first feature amount, receives a user specification as to which cluster the first patch image determined to have a morphological abnormality in the tissue specimen should belong, and performs one of a manual clustering process that clusters the first patch image into multiple clusters based on the specification, and a soft clustering process that calculates the degree of belonging of the first patch image determined to have a morphological abnormality in the tissue specimen to each of the multiple clusters.

[0007] The processor preferably controls the display of the results of the manual clustering process or the results of the soft clustering process.

[0008] The results are displayed using a plurality of cluster images generated by processing the first sample image, and it is preferable that the plurality of cluster images are images that allow the plurality of clusters to be distinguished using a display format that is pre-set for each of the plurality of clusters.

[0009] Preferably, the processor superimposes and displays at least one of the plurality of cluster images on the first specimen image.

[0010] The processor preferably receives from the user a specification of the cluster image to be superimposed on the first specimen image.

[0011] The processor preferably displays statistics based on the results.

[0012] In the manual clustering process, it is preferable that the processor reduces the number of dimensions of the first feature to two or three dimensions, displays a graph in which the first feature with reduced number of dimensions is plotted in two or three-dimensional space, and accepts specifications on the graph.

[0013] It is preferable that the machine learning model is a model that has been trained using second patch images, which are obtained by subdividing second specimen images of tissue specimens from multiple subjects constituting a control group to which the candidate substance was not administered in a past evaluation test of a candidate pharmaceutical substance, as training data.

[0014] The training data preferably also includes patch images of tissue specimens in which morphological abnormalities have occurred.

[0015] The machine learning model is preferably a model tasked with identifying types of morphological abnormalities.

[0016] It is preferable that the processor obtains information on the distribution of the second feature extracted using a machine learning model from a second patch image obtained by subdividing the second sample image, calculates the distance between the distribution and the first feature, and makes a judgment based on the distance.

[0017] The method of operating the image processing device disclosed herein includes acquiring a first specimen image in which a tissue specimen of a subject is captured, extracting a first feature amount from a first patch image obtained by subdividing the first specimen image using a machine learning model, determining whether or not a morphological abnormality has occurred in the tissue specimen captured in the first patch image based on the first feature amount, and performing one of a manual clustering process that receives a user's specification as to which cluster the first patch image, for which it has been determined that the tissue specimen has a morphological abnormality, should belong, and clusters the first patch image into a plurality of clusters based on the specification, and a soft clustering process that calculates the degree of belonging of the first patch image, for which it has been determined that the tissue specimen has a morphological abnormality, to each of a plurality of clusters.

[0018] The operating program of the image processing device disclosed herein causes a computer to execute processes including acquiring a first specimen image in which a tissue specimen of a subject is captured, extracting a first feature amount from a first patch image obtained by subdividing the first specimen image using a machine learning model, determining whether or not a morphological abnormality has occurred in the tissue specimen captured in the first patch image based on the first feature amount, and performing one of a manual clustering process that receives a user's specification as to which cluster a first patch image determined to have a morphological abnormality in the tissue specimen should belong, and clusters the first patch image into multiple clusters based on the specification, and a soft clustering process that calculates the degree of belonging of a first patch image determined to have a morphological abnormality in the tissue specimen to each of the multiple clusters.

[0019] According to the technology of the present disclosure, it is possible to provide an image processing device, an operating method for an image processing device, and an operating program for an image processing device that are capable of clustering patch images of a sample image whose feature distribution is not discrete.

[0020] 1 is a diagram illustrating a process of an evaluation test, a first sample image, and an image processing device. FIG. 2 is a block diagram illustrating a computer constituting the image processing device. FIG. 3 is a block diagram illustrating a processing unit of a CPU of the image processing device. FIG. 4 is a diagram illustrating a first patch image obtained by dividing a first sample image. FIG. 5 is a diagram illustrating how a feature extractor extracts a first feature from a first patch image. FIG. 6 is a diagram illustrating the structure of a feature extractor. FIG. 7 is a diagram illustrating processing in the learning phase of an autoencoder. FIG. 8 is a diagram illustrating the structure of a past control group and a second patch image for learning. FIG. 9 is a diagram illustrating how a feature extractor extracts a second feature from a second patch image. FIG. 10 is a diagram illustrating a graph plotting second features in feature space, and second feature distribution information. FIG. 11 is a diagram illustrating the distance between the position of the first feature and the representative position of the second feature. FIG. 12 is a diagram illustrating the processing of a determination unit. FIG. 13 is a diagram illustrating the processing of a determination unit. FIG. 14 is a diagram illustrating a detailed configuration of a clustering processing unit. FIG. 15 is a diagram illustrating a target designation screen. FIG. 16 is a diagram illustrating the processing of a dimension reduction unit. FIG. 17 is a diagram illustrating a cluster designation screen. FIG. 18 is a diagram illustrating the processing of a designation receiving unit when a drug discovery staff member designates a cluster on the cluster designation screen. FIG. 19 is a diagram illustrating a cluster image and a superimposed image. FIG. 19 is a diagram illustrating a clustering result display screen. 10 is a diagram showing a clustering result display screen on which a superimposed image in which only one cluster image is superimposed on a target first specimen image is displayed. FIG. 11 is a flowchart showing the processing procedure of an image processing device. FIG. 12 is a flowchart showing the processing procedure of an image processing device. FIG. 13 is a diagram showing the processing of a dimension reduction unit of a modified example. FIG. 14 is a diagram showing a cluster designation screen of a modified example. FIG. 15 is a diagram showing a detailed configuration of a clustering processing unit of a first_2 embodiment. FIG. 16 is a diagram showing the processing of a membership calculation unit and a clustering information generation unit. FIG. 17 is a flowchart showing the processing procedure of an image processing device of a first_2 embodiment. FIG. 18 is a diagram showing a clustering result display screen on which statistical information is displayed. FIG. 19 is a diagram showing the processing in the learning phase of an autoencoder of a third embodiment. FIG. 19 is a diagram showing the structure of a past administration group and a third patch image for learning. FIG. 19 is a diagram showing the structure of a feature extractor of a fourth embodiment. FIG. 19 is a diagram showing a fifth embodiment in which specimen images obtained by photographing a slide specimen on which tissue specimens of multiple types of organs are placed.FIG. 10 is a diagram showing how a feature extractor for a cardiac specimen extracts a first feature from a first patch image obtained by dividing the cardiac specimen.

[0021] [First Embodiment] As shown in FIG. 1 as an example, an image processing device 10 of the present disclosure is used to evaluate the efficacy and toxicity of a drug candidate substance 11. The image processing device 10 is, for example, a desktop personal computer, and includes a display 12 that displays various screens, and an input device 13 such as a keyboard, a mouse, a touch panel, and / or a microphone for voice input. The image processing device 10 is installed, for example, in a drug development facility, and is operated by drug development staff DS who are involved in drug development at the drug development facility. The drug development staff DS also includes pathologists. The drug development staff DS is an example of a "user" related to the technology of the present disclosure.

[0022] A first specimen image 151 is input to the image processing device 10. The first specimen image 151 is an image for evaluating the efficacy and toxicity of the candidate substance 11. The first specimen image 151 is generated, for example, by the following procedure. First, a subject S, such as a rat, prepared for evaluation of the candidate substance 11 is autopsied, and multiple tissue specimens (hereinafter referred to as liver specimens) LVS of cross sections of the subject S's organs, in this case the liver LV, are collected. Next, the collected liver specimens LVS are attached one by one to glass slides 16, and then stained, in this case with hematoxylin and eosin dye. Next, the stained liver specimens LVS are covered with a cover glass 17 to complete a slide specimen 18. The slide specimen 18 is then placed in an imaging device 19, such as a digital optical microscope, and a first specimen image 151 is captured by the imaging device 19. The first specimen image 151 thus obtained is assigned a specimen ID (Identification Data) for uniquely identifying the specimen S, a specimen image ID for uniquely identifying the first specimen image 151, and the date and time of capture. The tissue specimen is also called a tissue section. The staining may be performed using hematoxylin dye alone, nuclear fast red dye, or the like.

[0023] Here, the administration group and the control group will be described. The administration group is composed of multiple subjects S to which the candidate substance 11 has been administered. The control group, in contrast to the administration group, is composed of multiple subjects S to which the candidate substance 11 has not been administered. The number of subjects S constituting the administration group and the number of subjects S constituting the control group are both, for example, approximately 5 to 10. The subjects S constituting the administration group and the subjects S constituting the control group are subjects S with the same attributes and placed in the same rearing environment. The same attributes include, for example, the same age in weeks and / or the same gender. The same attributes also include the same age composition ratio and / or the same gender composition ratio (e.g., five males and five females). The same rearing environment includes, for example, the same food, the same temperature and humidity of the rearing space, and / or the same size of the rearing space. The "same" in the "same rearing environment" refers not only to the exact sameness, but also to the sameness in the sense of a tolerance generally acceptable in the technical field to which the technology of the present disclosure pertains, which tolerance does not contradict the spirit of the technology of the present disclosure.

[0024] The administration group includes a plurality of groups with different doses of the candidate substance 11. For example, the dose of the candidate substance 11 is differentiated into three levels: a high dose group, a medium dose group, and a low dose group. This allows the influence of the dose of the candidate substance 11 on the subject S to be ascertained.

[0025] The first specimen image 151 may be an image of the liver specimen LVS of the subject S of the administration group, or may be an image of the liver specimen LVS of the subject S of the control group. Fig. 1 illustrates an example in which the first specimen image 151 is an image of the liver specimen LVS of the subject S of the administration group.

[0026] 2, the computer constituting the image processing device 10 includes, in addition to the display 12 and input device 13, a storage 30, a memory 31, a CPU (Central Processing Unit) 32, and a communication unit 33. These are interconnected via a bus line 34.

[0027] The storage 30 is a hard disk drive built into the computer constituting the image processing device 10 or connected via a cable or network. Alternatively, the storage 30 is a disk array consisting of multiple hard disk drives. The storage 30 stores control programs such as an operating system, various application programs, and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0028] The memory 31 is a work memory for the CPU 32 to execute processing. The CPU 32 loads programs stored in the storage 30 into the memory 31 and executes processing in accordance with the programs. In this way, the CPU 32 comprehensively controls each part of the computer. The CPU 32 is an example of a "processor" according to the technology of the present disclosure. Note that the memory 31 may be built into the CPU 32. The communication unit 33 controls the transmission of various information to and from external devices such as the imaging device 19.

[0029] As an example, as shown in FIG. 3 , an operating program 40 is stored in the storage 30 of the image processing device 10. The operating program 40 is an application program for causing a computer to function as the image processing device 10. In other words, the operating program 40 is an example of an "operating program of an image processing device" according to the technology of the present disclosure. The storage 30 also stores a feature extractor 41, second feature distribution information 42, and the like. The feature extractor 41 is an example of a "machine learning model" according to the technology of the present disclosure. Furthermore, the second feature distribution information 42 is an example of "information on the distribution of second features" according to the technology of the present disclosure.

[0030] When the operating program 40 is started, the CPU 32 of the computer constituting the image processing device 10 works in cooperation with the memory 31 and the like to function as a read / write (hereinafter abbreviated as RW (Read Write)) control unit 50, a feature extraction unit 51, a judgment unit 52, and a clustering processing unit 53.

[0031] The RW control unit 50 controls the storage of various data in the storage 30 and the reading of various data from the storage 30. For example, the RW control unit 50 stores a first specimen image 151 from the imaging device 19 in the storage 30. Note that, since a plurality of first specimen images 151 are actually obtained from one subject S, a plurality of first specimen images 151 for one subject S are stored in the storage 30.

[0032] The RW control unit 50 acquires a first specimen image 151 by reading it from the storage 30 in accordance with the specification of the drug discovery staff DS via the input device 13. The RW control unit 50 outputs the read first specimen image 151 to the feature extraction unit 51 and the clustering processing unit 53. The first specimen image 151 output from the RW control unit 50 to the feature extraction unit 51, etc., is a target for determining whether or not a morphological abnormality has occurred in the liver specimen LVS. Hereinafter, the first specimen image 151 that is a target for determining whether or not a morphological abnormality has occurred in the liver specimen LVS will be referred to as a target first specimen image 151T (see FIG. 4, etc.).

[0033] The RW control unit 50 reads the feature extractor 41 from the storage 30 and outputs the read feature extractor 41 to the feature extraction unit 51. The RW control unit 50 also obtains second feature distribution information 42 by reading it from the storage 30. The RW control unit 50 outputs the read second feature distribution information 42 to the determination unit 52.

[0034] The feature extraction unit 51 extracts a first feature 601 from the target first sample image 151T using the feature extractor 41. The feature extraction unit 51 outputs the first feature 601 to the determination unit 52.

[0035] The determination unit 52 determines whether or not a morphological abnormality has occurred in the liver specimen LVS depicted in the target first specimen image 151T, based on the first feature 601 and the second feature distribution information 42. Morphological abnormalities include lesions that are not seen in normal liver specimen LVS, such as hyperplasia, infiltration, congestion, inflammation, tumor, canceration, proliferation, bleeding, glycogen depletion, etc. The determination unit 52 outputs a determination result 61 indicating whether or not a morphological abnormality has occurred in the liver specimen LVS depicted in the target first specimen image 151T to the clustering processing unit 53.

[0036] In this embodiment, the clustering processing unit 53 performs manual clustering processing, which will be described later.

[0037] As an example, as shown in FIG. 4 , the feature extraction unit 51 recognizes the liver specimen LVS depicted in the target first specimen image 151T using a well-known image recognition technique and subdivides the recognized liver specimen LVS into multiple first patch images 651. The first patch images 651 have a predetermined size that can be handled by the feature extractor 41. The first patch images 651 are also sized to cover not only the morphological abnormality but also its surrounding area. The feature extraction unit 51 assigns a patch image ID 85 (see FIG. 12 , etc.) to the first patch image 651. The feature extraction unit 51 also associates information indicating which portion of the first specimen image 151T the first patch image 651 is cut out from, i.e., position information 86 of the first patch image 651 (see FIG. 12 , etc.), with the patch image ID 85. Although the first patch image 651 does not have an area overlapping with other first patch images 651 in FIG. 4, the first patch image 651 may partially overlap with other first patch images 651 .

[0038] 5 , the feature extraction unit 51 extracts a first feature 601 for each of a plurality of first patch images 651 obtained by subdividing the target first sample image 151T, using the feature extractor 41. Therefore, the number of first feature 601 is the same as the number of first patch images 651.

[0039] As an example, as shown in FIG. 6 , the feature extractor 41 is configured by reusing an encoder unit 71 of an autoencoder 70. The autoencoder 70 includes a decoder unit 72 in addition to the encoder unit 71. A patch image 65 such as a first patch image 651 is input to the encoder unit 71. The encoder unit 71 converts the patch image 65 into a feature 60. The encoder unit 71 passes the feature 60 to the decoder unit 72. The decoder unit 72 generates a restored image 73 of the patch image 65 from the feature 60.

[0040] As is well known, the encoder unit 71 includes a convolution layer that performs convolution processing using a filter, a pooling layer that performs pooling processing such as maximum value pooling, and the like. The decoder unit 72 is similar. The encoder unit 71 extracts feature quantities 60 by repeating convolution processing by the convolution layer and pooling processing by the pooling layer multiple times on the input patch image 65. The extracted feature quantities 60 represent the shape and texture characteristics of the liver specimen LVS depicted in the patch image 65.

[0041] The feature 60 is a set of multiple numerical values. In other words, the feature 60 is multidimensional data. The number of dimensions of the feature 60 is, for example, 512, 1024, or 2048. The first feature 601 and a second feature 602 (described later) (see FIG. 9) have the same number of dimensions and can be compared in the same feature space 81 (see FIG. 10, etc.).

[0042] As an example, as shown in FIG. 7 , in a learning phase before the encoder unit 71 is converted into the feature extractor 41, the autoencoder 70 receives a second training patch image 652L as input and trains. The autoencoder 70 outputs a training reconstructed image 73L for the second training patch image 652L. A loss calculation is performed on the autoencoder 70 using a loss function based on the second training patch image 652L and the training reconstructed image 73L. Then, various coefficients of the autoencoder 70 (such as the filter coefficients of the convolution layer) are updated according to the results of the loss calculation, and the autoencoder 70 is updated according to the update setting. The second training patch image 652L is an example of "teacher data" according to the technology disclosed herein.

[0043] During the learning phase of the autoencoder 70, the above-described series of processes, including input of the second training patch image 652L to the autoencoder 70, output of the training restored image 73L from the autoencoder 70, loss calculation, update setting, and updating of the autoencoder 70, are repeatedly performed while the second training patch image 652L is being replaced. The repetition of the above-described series of processes is terminated when the restoration accuracy of the training restored image 73L from the second training patch image 652L reaches a predetermined set level. The encoder unit 71 of the autoencoder 70 whose restoration accuracy has reached the set level is stored in the storage 30 of the image processing device 10 as the feature extractor 41. Note that learning may be terminated when the above-described series of processes has been repeated a set number of times, regardless of the restoration accuracy of the training restored image 73L from the second training patch image 652L.

[0044] The learning of the autoencoder 70 may be performed by the image processing device 10 or by a device separate from the image processing device 10. In the latter case, the feature extractor 41 is transmitted from the separate device to the image processing device 10, and the feature extractor 41 is stored in the storage 30 by the RW control unit 50.

[0045] As an example, as shown in FIG. 8 , the second patch image 652L for learning is supplied from a plurality of second patch images 652 obtained by subdividing the second specimen image 152. The second specimen image 152 is an image of a liver specimen LVS of a subject S in a past control group 75. The past control group 75 is composed of a plurality of subjects S to whom a candidate substance was not administered in a past evaluation test. For this reason, the number of subjects S constituting the past control group 75 is significantly greater than the number of subjects S constituting the administration group and the control group, for example, on the order of several hundred to several thousand. Like the first specimen image 151, multiple second specimen images 152 are obtained from one subject S. Therefore, the number of second specimen images 152 obtained from the past control group 75 is calculated by multiplying the number obtained from one subject S by the number of subjects S.

[0046] Next, we will explain how the second feature distribution information 42 is constructed. First, as an example, as shown in Fig. 9, a feature extractor 41 is used to extract a plurality of second feature amounts 602 from each of a plurality of second patch images 652 based on a plurality of second sample images 152 obtained from all of the subjects S constituting the past control group 75.

[0047] As an example, a graph 80 shown in FIG. 10 is a graph in which multiple second feature quantities 602 extracted in FIG. 9 are plotted in a feature quantity space 81. The second feature quantity distribution information 42 includes coordinates 82 in the feature quantity space 81 of representative positions of the second feature quantities 602 indicated by crosses (hereinafter referred to as representative position coordinates). The representative position is, for example, the center point or average point of a distribution 83 of the second feature quantities 602. Note that, for convenience of explanation, the dimension of the feature quantity space 81 is shown as two-dimensional, having a D1 axis and a D2 axis, in FIG. 10 , but the actual dimension of the feature quantity space 81 is, as described above, 512 dimensions or the like. Similarly, for convenience of explanation, the dimension of the feature quantity space 81 is shown as two-dimensional in FIG. 11 .

[0048] As with the learning of the autoencoder 70, the second feature amount distribution information 42 may be created by the image processing device 10, or may be created by a device other than the image processing device 10. In the latter case, the second feature amount distribution information 42 is transmitted from the other device to the image processing device 10, and the RW control unit 50 stores the second feature amount distribution information 42 in the storage 30. The second feature amount distribution information 42 may also be a plurality of second feature amounts 602 themselves. In this case, the representative position coordinates 82 are derived by the image processing device 10.

[0049] As an example, as shown in FIG. 11 , the determination unit 52 calculates the distance D in the feature space 81 between the representative position of the second feature 602 represented by the representative position coordinates 82 of the second feature distribution information 42 and the position of the first feature 601. The determination unit 52 calculates the distance D between the multiple first feature 601 extracted for each of the multiple first patch images 651 obtained by subdividing a single target first sample image 151T. The distance D is the Mahalanobis distance. Note that the distance D may be calculated as the average, median, or maximum value of the Euclidean distance between the positions of the k-neighbor samples in the distribution 83 of the second feature 602 and the position of the first feature 601. Alternatively, instead of the distance D, a value obtained by subtracting from 1.0 the cosine similarity between the vector representing the representative position of the second feature 602 and the vector representing the position of the first feature 601 may be calculated. The cosine similarity takes a value between −1.0 and 1.0, and the larger the value, the more similar the vectors' orientations are.

[0050] 12 , the determination unit 52 compares the calculated distance D with a preset threshold, and if the distance D is less than the threshold, determines that no morphological abnormality has occurred in the liver specimen LVS depicted in the first patch image 651. The determination unit 52 outputs a determination result 61 indicating that no morphological abnormality has occurred in the liver specimen LVS depicted in the first patch image 651. The determination result 61 in this case includes a patch image ID 85 and position information 86.

[0051] On the other hand, as shown in FIG. 13 as an example, when the distance D is equal to or greater than the threshold, the determination unit 52 determines that a morphological abnormality has occurred in the liver specimen LVS depicted in the first patch image 651. The determination unit 52 outputs a determination result 61 indicating that a morphological abnormality has occurred in the liver specimen LVS depicted in the first patch image 651. In this case, the determination result 61 includes a first feature amount 601 in addition to a patch image ID 85 and position information 86. Note that the threshold may be configured to be configurable by the drug discovery staff DS. Instead of the distance D, it may be determined whether or not a morphological abnormality has occurred in the liver specimen LVS depicted in the first patch image 651 by comparing the cosine similarity described above with the threshold.

[0052] As shown in FIGS. 8 and 9 , the second feature 602 is a feature extracted from a second patch image 652 obtained by subdividing the second specimen image 152 depicting the liver specimen LVS of the subject S in the historical control group 75. Because the subject S in the historical control group 75 was not administered the candidate substance, the liver specimen LVS depicted in the second specimen image 152 does not exhibit morphological abnormalities at least due to the toxicity of the candidate substance. Therefore, the representative position of the second feature 602 is regarded as the representative position of the feature of a specimen image depicting a normal liver specimen LVS. Therefore, the distance D between the position of the first feature 601 and the representative position of the second feature 602 serves as an index indicating the degree to which the liver specimen LVS depicted in the first patch image 651 deviates from the normal liver specimen LVS. Therefore, for the first patch image 651 in which the distance D is less than the threshold, the determination unit 52 determines that the liver specimen LVS does not deviate from the normal liver specimen LVS and that no morphological abnormality has occurred in the liver specimen LVS. On the other hand, for the first patch image 651 in which the distance D is equal to or greater than the threshold, the determination unit 52 determines that the liver specimen LVS deviates from the normal liver specimen LVS and that a morphological abnormality has occurred in the liver specimen LVS.

[0053] As an example, as shown in FIG. 14 , the clustering processing unit 53 includes a dimension reduction unit 90, a display control unit 91, and a designation receiving unit 92. The dimension reduction unit 90, the display control unit 91, and the designation receiving unit 92 perform the manual clustering process described below. The dimension reduction unit 90 receives an input of a determination result 61 from the determination unit 52. The display control unit 91 receives an input of a target first specimen image 151T from the RW control unit 50. The display control unit 91 controls the display of various screens on the display 12. The designation receiving unit 92 receives various designations from the drug discovery staff DS via the input device 13.

[0054] As an example, as shown in FIG. 15 , the display control unit 91 controls the display of a target designation screen 95 on the display 12. The display control unit 91 displays the target designation screen 95 when the drug discovery staff member DS issues an instruction to display a first specimen image 151 via the input device 13 and the instruction is accepted by the designation accepting unit 92. The target designation screen 95 displays a plurality of first specimen images 151 side by side. These first specimen images 151 are first specimen images 151 obtained from one of the plurality of subjects S constituting the administration group. FIG. 15 shows an example in which ten first specimen images 151 with specimen image IDs "SI00001" to "SI00010" obtained from the subject S with subject ID "R001" are displayed side by side. The subject ID is displayed as a pull-down menu 96, making it possible to switch the subject S whose first specimen image 151 is displayed on the target designation screen 95.

[0055] The target designation screen 95 is a screen for designating one target first specimen image 151T from among a plurality of first specimen images 151. The target designation screen 95 is provided with a selection frame 97 that can move between each of the first specimen images 151. In addition, an analysis button 98 is provided at the bottom of the target designation screen 95. The drug discovery staff member DS aligns the selection frame 97 with the desired first specimen image 151 and then selects the analysis button 98. As a result, the first specimen image 151 with the selection frame 97 aligned is designated as the target first specimen image 151T, and the feature extraction unit 51 extracts the first feature 601, and the determination unit 52 determines whether or not a morphological abnormality has occurred.

[0056] 16 , the dimension reduction unit 90 reduces the number of dimensions of the first feature 601 included in the determination result 61 of the first patch image 651 in which it has been determined that the liver specimen LVS has a morphological abnormality to two dimensions. The dimension reduction unit 90 generates dimension-reduced data 100 based on the determination result 61. The dimension-reduced data 100 includes the dimension-reduced first feature 601R, a patch image ID 85, and position information 86. The dimension reduction unit 90 outputs the dimension-reduced data 100 to the display control unit 91. As a method for dimension reduction, principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), or UMAP (Uniform Manifold Approximation and Projection) can be used.

[0057] When the display control unit 91 receives the dimension-reduced data 100 from the dimension reduction unit 90, the display control unit 91 controls the display of a cluster designation screen 105 shown in FIG. 17 as an example, on the display 12. A graph 106 is displayed on the cluster designation screen 105. The graph 106 is a graph in which the two-dimensional first feature 601R after dimension reduction included in the dimension-reduced data 100 is plotted in a two-dimensional feature space 107.

[0058] As described above, the first patch image 651 has a size that covers not only the morphological abnormality but also the surrounding area, so the distribution of the first feature amount 601R is not discrete.

[0059] A cluster designation toolbar 108 is provided to the right of the graph 106. The cluster designation toolbar 108 is a toolbar that aggregates icons of various tools used by the drug discovery staff DS when designating, on the graph 106, to which cluster a first patch image 651 determined to have morphological abnormalities in the liver specimen LVS belongs. The tools include a tool for enclosing a group of first feature values ​​601R plots considered to be a cluster with a rectangle, circle, ellipse, or free curve. The tools also include a tool for erasing the rectangle, circle, ellipse, or free curve enclosing the plot of the first feature values ​​601R, or a tool for changing the designated cluster.

[0060] The drug discovery staff member DS uses various tools on the cluster designation toolbar 108 to designate to which cluster the first patch image 651 belongs on the graph 106, as shown in Fig. 18 as an example. Fig. 18 illustrates an example in which three clusters, clusters 1, 2, and 3, are designated by encircling them with ellipses. Note that, as shown in the example, some of the plots of the first feature amount 601R do not belong to any cluster.

[0061] A designation completion button 109 is provided at the bottom of the cluster designation screen 105. After designating a cluster on the graph 106, the drug discovery staff member DS selects the designation completion button 109. This causes the designation receiving unit 92 to receive designation by the drug discovery staff member DS as to which cluster the first patch image 651, which has been determined to have a morphological abnormality in the liver specimen LVS, belongs.

[0062] The designation receiving unit 92 generates clustering information 112 based on the received designation. The clustering information 112 is information in which the cluster to which each patch image ID 85 of the first patch image 651 belongs is registered. A first patch image 651 in which the plot of the first feature amount 601R does not belong to any cluster is not registered in the clustering information 112. The designation receiving unit 92 outputs the clustering information 112 to the display control unit 91.

[0063] 19 , the display control unit 91 processes the target first specimen image 151T in accordance with the clustering information 112 from the designation receiving unit 92 to generate cluster images 115, 116, and 117. The cluster image 115 is an image corresponding to cluster 1 shown in FIG. 18 . The cluster image 116 is an image corresponding to cluster 2 shown in FIG. 18 . The cluster image 117 is an image corresponding to cluster 3 shown in FIG. 18 .

[0064] The display control unit 91 generates cluster images 115 to 117 according to a display format 118 preset for each of clusters 1 to 3. The display format 118, for example, displays cluster 1 in indigo blue, cluster 2 in yellow-green, and cluster 3 in gray. The display control unit 91 generates cluster image 115 by filling in the position (determined from the position information 86) of the first patch image 651 of patch image ID 85 for which cluster 1 is registered in the clustering information 112 with indigo blue in the target first specimen image 151T. Similarly, the display control unit 91 generates cluster image 116 by filling in the position of the first patch image 651 of patch image ID 85 for which cluster 2 is registered in the clustering information 112 with yellow-green in the target first specimen image 151T. Furthermore, the display control unit 91 generates a cluster image 117 by filling in gray the position of the first patch image 651 of patch image ID 85, for which cluster 3 is registered in the clustering information 112, in the target first specimen image 151T. By changing the display color in this manner, the cluster images 115 to 117 become images in which clusters 1 to 3 can be distinguished. The cluster images 115 to 117 are an example of the "result of manual clustering processing" according to the technology of the present disclosure. Note that the display format 118 may be configured so that it can be freely changed by the drug discovery staff DS.

[0065] The display control unit 91 generates a superimposed image 119 in which the target first specimen image 151T is superimposed on at least one of the cluster images 115 to 117. Fig. 19 illustrates an example of the superimposed image 119 in which all of the cluster images 115 to 117 are superimposed on the target first specimen image 151T.

[0066] As an example, as shown in FIGS. 20 and 21 , the display control unit 91 controls the display of a clustering result display screen 125 including a superimposed image 119 on the display 12. In addition to the superimposed image 119, the clustering result display screen 125 is provided with a legend 126 for clusters 1 to 3 and display switch buttons 127, 128, and 129. The display switch button 127 is a button for selecting whether to superimpose the cluster image 115 on the target first specimen image 151T. The display switch button 128 is a button for selecting whether to superimpose the cluster image 116 on the target first specimen image 151T. The display switch button 129 is a button for selecting whether to superimpose the cluster image 117 on the target first specimen image 151T. The designation receiving unit 92 receives a selection instruction from the display switch buttons 127 to 129 by the drug discovery staff DS.

[0067] When the display switching button 127 is selected, the display control unit 91 superimposes the cluster image 115 on the target first specimen image 151T. On the other hand, when the display switching button 127 is not selected, the display control unit 91 does not superimpose the cluster image 115 on the target first specimen image 151T. Similarly, when the display switching button 128 is selected, the display control unit 91 superimposes the cluster image 116 on the target first specimen image 151T, and when the display switching button 128 is not selected, the display control unit 91 does not superimpose the cluster image 116 on the target first specimen image 151T. Furthermore, when the display switching button 129 is selected, the display control unit 91 superimposes the cluster image 117 on the target first specimen image 151T, and when the display switching button 129 is not selected, the display control unit 91 does not superimpose the cluster image 117 on the target first specimen image 151T.

[0068] Therefore, as shown in FIG. 20 , when all of the display switching buttons 127 to 129 are selected, the display control unit 91 displays a superimposed image 119 in which all of the cluster images 115 to 117 are superimposed on the target first specimen image 151T. In contrast, as shown in FIG. 21 , when the display switching button 127 is selected but the display switching buttons 128 and 129 are not selected, the display control unit 91 displays a superimposed image 119 in which only the cluster image 115 is superimposed on the target first specimen image 151T. In this manner, the display control unit 91 displays at least one of the multiple cluster images 115 to 117 superimposed on the target first specimen image 151T. Furthermore, the designation receiving unit 92 receives, from the drug discovery staff DS, a designation of the cluster images 115 to 117 to be superimposed on the target first specimen image 151T. The clustering result display screen 125 is initially displayed with all of the display switching buttons 127 to 129 selected. Furthermore, the clustering result display screen 125 disappears when a confirmation button 130 at the bottom is selected.

[0069] Next, the operation of the above configuration will be described with reference to the flowcharts shown in Figures 22 and 23 as an example. First, when the operating program 40 is started in the image processing device 10, the CPU 32 of the image processing device 10 functions as an RW control unit 50, a feature extraction unit 51, a determination unit 52, and a clustering processing unit 53, as shown in Figure 3. As shown in Figure 14, the clustering processing unit 53 functions as a dimension reduction unit 90, a display control unit 91, and a designation receiving unit 92.

[0070] The imaging device 19 captures a first specimen image 151 of a liver specimen LVS of the subject S. The first specimen image 151 is output from the imaging device 19 to the image processing device 10. In the image processing device 10, the first specimen image 151 from the imaging device 19 is stored in the storage 30 by the RW control unit 50.

[0071] 22 , when the drug discovery staff member DS issues an instruction to display a first specimen image 151 via the input device 13, the RW control unit 50 reads and acquires the first specimen image 151 specified in the display instruction from the storage 30 (step ST100). The first specimen image 151 is output from the RW control unit 50 to the display control unit 91 of the clustering processing unit 53. As shown in FIG. 15 , the first specimen image 151 is displayed on the display 12 via the target designation screen 95 under the control of the display control unit 91 (step ST105).

[0072] When the drug discovery staff member DS positions the selection frame 97 over the desired first specimen image 151 on the target designation screen 95 and selects the analysis button 98 (YES in step ST110), the RW control unit 50 reads out and acquires the first specimen image 151 on which the selection frame 97 was positioned at that time from the storage 30 as a target first specimen image 151T (step ST115). The target first specimen image 151T is output from the RW control unit 50 to the feature extraction unit 51 and the display control unit 91.

[0073] The RW control unit 50 reads the feature extractor 41 from the storage 30, and outputs the read feature extractor 41 to the feature extraction unit 51. The RW control unit 50 also reads and acquires second feature distribution information 42 from the storage 30, and outputs the read second feature distribution information 42 to the determination unit 52.

[0074] 4, the feature extraction unit 51 subdivides the target first sample image 151T into a plurality of first patch images 651 (step ST120). Subsequently, as shown in Fig. 5, the feature extraction unit 51 extracts first feature amounts 601 from the first patch images 651 using the feature extractor 41 (step ST125). The first feature amounts 601 are output from the feature extraction unit 51 to the determination unit 52.

[0075] 11 , the determination unit 52 calculates a distance D between the representative position of the second feature amount 602 and the position of the first feature amount 601. Then, as shown in FIGS. 12 and 13 , the determination unit 52 compares the distance D with a preset threshold value and determines whether or not a morphological abnormality has occurred in the liver specimen LVS depicted in the first patch image 651 (step ST130). A determination result 61 as to whether or not a morphological abnormality has occurred in the liver specimen LVS depicted in the first patch image 651 is output from the determination unit 52 to the dimension reduction unit 90 of the clustering processing unit 53.

[0076] The processes of steps ST125 and ST130 are performed on all first patch images 651. After the processes of steps ST125 and ST130 have been performed on all first patch images 651 (YES in step ST135), the process proceeds to step ST140 shown in FIG.

[0077] 16 , the dimension reduction unit 90 reduces the number of dimensions of the first feature 601 of the first patch image 651 determined to have a morphological abnormality to two dimensions (step ST140). Next, under the control of the display control unit 91, the cluster designation screen 105 shown in FIG. 17 is displayed on the display 12 (step ST145).

[0078] The cluster designation screen 105 includes a graph 106 in which the two-dimensional first feature 601R after dimension reduction is plotted in a two-dimensional feature space 107. On this graph 106, as shown in FIG. 18 , the drug discovery staff member DS designates which cluster the first patch image 651 belongs to, and selects the designation completion button 109. When the designation completion button 109 is selected, the designation receiving unit 92 accepts the designation by the drug discovery staff member DS of which cluster the first patch image 651 belongs (step ST150). Based on this designation, the designation receiving unit 92 generates clustering information 112. This completes the manual clustering process of clustering the first patch image 651 into multiple clusters (step ST155). The clustering information 112 is output from the designation receiving unit 92 to the display control unit 91.

[0079] 19, the display control unit 91 processes the target first specimen image 151T in accordance with the clustering information 112 to generate cluster images 115 to 117 in which multiple clusters 1 to 3 can be distinguished (step ST160). Then, as shown in FIGS. 20 and 21, under the control of the display control unit 91, a clustering result display screen 125 including a superimposed image 119 in which at least one of the cluster images 115 to 117 is superimposed on the target first specimen image 151T is displayed on the display 12 (step ST165).

[0080] The drug discovery staff member DS views the superimposed image 119 through the clustering result display screen 125 and evaluates the efficacy and toxicity of the candidate substance 11. At this time, the drug discovery staff member DS operates the display switching buttons 127 to 129 as necessary to switch the cluster images 115 to 117 to be superimposed on the target first specimen image 151T.

[0081] As described above, the CPU 32 of the image processing device 10 includes the RW control unit 50, the feature extraction unit 51, the determination unit 52, and the clustering processing unit 53. The RW control unit 50 acquires the target first specimen image 151T by reading it from the storage 30. The target first specimen image 151T is an image of the liver specimen LVS of the subject S.

[0082] The feature extraction unit 51 uses the feature extractor 41 to extract first feature values ​​601 from first patch images 651 obtained by subdividing the target first specimen image 151T. The determination unit 52 determines whether or not a morphological abnormality has occurred in the liver specimen LVS depicted in the first patch image 651 based on the first feature value 601. The clustering processing unit 53 receives a designation from the drug discovery staff DS as to which cluster the first patch image 651 determined to have a morphological abnormality in the liver specimen LVS belongs. Then, a manual clustering process is performed to cluster the first patch image 651 into multiple clusters based on the designation. This makes it possible to cluster the first patch image 651 of the target first specimen image 151T in which the distribution of the first feature value 601 is not discrete.

[0083] The display control unit 91 controls the display of a superimposed image 119 including cluster images 115 to 117, which are the results of the manual clustering process, on the display 12. This allows the drug discovery staff DS to easily know the results of the manual clustering process.

[0084] The results of the manual clustering process are displayed by a plurality of cluster images 115 to 117 generated by processing the target first specimen image 151T. The plurality of cluster images 115 to 117 are images that make it possible to distinguish the plurality of clusters 1 to 3 using a display format 118 that is preset for each of the plurality of clusters 1 to 3. This allows the drug discovery staff DS to easily understand the results of the manual clustering process.

[0085] The display control unit 91 superimposes and displays at least one of the plurality of cluster images 115 to 117 on the target first specimen image 151T, so that the drug discovery staff DS can easily grasp which part of the liver specimen LVS belongs to which cluster, and furthermore, what kind of morphological abnormality has occurred in which part of the liver specimen LVS.

[0086] The type of morphological abnormality can be estimated with a certain degree of certainty based on the part where it occurs. Therefore, the drug discovery staff DS can use the superimposed image 119 to distinguish whether the morphological abnormality is caused by the toxicity of the candidate substance 11, such as hyperplasia, or whether it is a spontaneous abnormality not caused by the toxicity of the candidate substance 11, such as glycogen depletion. Therefore, the drug discovery staff DS can consider the efficacy and toxicity of the candidate substance 11 based on the superimposed image 119.

[0087] The designation receiving unit 92 receives, from the drug discovery staff DS, designation of the cluster images 115 to 117 to be superimposed on the target first specimen image 151T. This allows the drug discovery staff DS to superimpose only one cluster image that they wish to focus on on the target first specimen image 151T, or to superimpose all cluster images 115 to 117 on the target first specimen image 151T. This facilitates the drug discovery staff DS's evaluation of the efficacy and toxicity of the candidate substance 11.

[0088] The dimension reduction unit 90 reduces the number of dimensions of the first feature 601 to two. The display control unit 91 displays on the display 12 a cluster designation screen 105 including a graph 106 in which the first feature 601R, with the number of dimensions reduced to two, is plotted in a two-dimensional feature space 107. The designation receiving unit 92 receives designations on the graph 106. This allows the drug discovery staff DS to easily designate to which cluster the first patch image 651 belongs.

[0089] As shown in FIGS. 7 and 8 , the feature extractor 41 is a model trained using second patch images 652, which are generated by subdividing second specimen images 152 depicting liver specimens LVS of multiple subjects S constituting a historical control group 75 to which the candidate substance was not administered in a past evaluation test of a drug candidate substance, as training data. This allows for easy extraction of first feature values ​​601 that accurately represent the shape and texture characteristics of the liver specimens LVS. Furthermore, the second patch images 652 can be unconditionally adopted as training data without the need for tedious preprocessing, such as a pathologist or other expert identifying whether or not the liver specimens LVS have morphological abnormalities and selectively adopting second patch images 652 in which the liver specimens LVS do not have morphological abnormalities as training data. This allows for easy acquisition of a large amount of training data.

[0090] The RW control unit 50 acquires second feature distribution information 42, which is information on a distribution 83 of second features 602 extracted by the feature extractor 41 from a second patch image 652 obtained by dividing the second sample image 152, by reading it from the storage 30. The determination unit 52 calculates the distance D between the distribution 83 and the first feature 601, more specifically, the distance D between the representative position of the distribution 83 and the position of the first feature, and makes a determination based on the distance D.

[0091] The distance D indicates the degree of deviation between the target first specimen image 151T and the second specimen image 152. The liver specimen LVS shown in the second specimen image 152 does not have any morphological abnormalities at least due to the toxicity of the candidate substance. Therefore, if a determination is made based on the distance D, a valid determination result 61 can be obtained.

[0092] (Modification) The number of dimensions of the first feature 601R after dimension reduction is not limited to two, as illustrated in FIG. 16 . For example, the dimension reduction unit 135 shown in FIG. 24 may output dimension-reduced data 136 in which the number of dimensions of the first feature 601 is reduced to three. In this case, the display control unit 91 controls the display 12 to display a cluster designation screen 142 including a graph 141 plotting the first feature 601R after dimension reduction in a three-dimensional feature space 140, as shown in FIG. 25 . In this case, a tool is provided for enclosing a plot of a group of first feature 601R considered to be a cluster with a cube, sphere, ellipsoid, or free curved body. This modification also allows the drug discovery staff DS to easily designate which cluster the first patch image 651 belongs to.

[0093] 26 as an example, a clustering processing unit 145 of this embodiment includes a degree of belonging calculation unit 146, a clustering information generation unit 147, and a display control unit 91. The degree of belonging calculation unit 146 performs soft clustering processing described below. A determination result 61 from the determination unit 52 is input to the degree of belonging calculation unit 146.

[0094] 27 as an example, the belonging degree calculation unit 146 calculates the degree of belonging of the first patch image 651 to each cluster by applying a well-known soft clustering method to the first feature amount 601 included in the determination result 61. Here, the belonging degree calculation unit 146 calculates the degree of belonging to three clusters, clusters 1 to 3, and the probability of not belonging to any of clusters 1 to 3 (indicated as "not belonging" in FIG. 27). The sum of the degree of belonging to clusters 1 to 3 and the probability of not belonging to any of clusters 1 to 3 is 100%. As a soft clustering method, a Gaussian Mixture Model (GMM), a Probabilistic Latent Semantic Analysis (PLSA), a Non-negative Matrix Factorization (NMF), a Fuzzy c-Means (FCM) method, or the like can be used.

[0095] The belonging degree calculation unit 146 generates belonging degree information 160 in which the belonging degree and the probability of non-belonging are registered for each patch image ID 85 of the first patch image 651. The belonging degree calculation unit 146 outputs the belonging degree information 160 to the clustering information generation unit 147.

[0096] The clustering information generation unit 147 determines the cluster to which each of the multiple first patch images 651 belongs, based on the belonging degree information 160. More specifically, the clustering information generation unit 147 determines the cluster having the maximum value of the belonging degrees registered in the belonging degree information 160 as the cluster to which the first patch image 651 belongs. If the probability of non-belonging is the maximum value, the clustering information generation unit 147 determines that the first patch image 651 does not belong to any cluster.

[0097] The clustering information generation unit 147 generates clustering information 161 that represents the results of the above-mentioned certification. Similar to the clustering information 112 of the above-mentioned 1_1 embodiment, the clustering information 161 is information in which the cluster to which each patch image ID 85 of the first patch image 651 belongs is registered. The clustering information generation unit 147 outputs the clustering information 161 to the display control unit 91. Thereafter, similar to the above-mentioned 1_1 embodiment, the display control unit 91 generates a plurality of cluster images by processing the target first specimen image 151T in accordance with the clustering information 161. Then, similar to the above-mentioned 1_1 embodiment, the display control unit 91 generates a superimposed image in which the target first specimen image 151T and the cluster image are superimposed, and controls the display 12 to display a clustering result display screen including the superimposed image.

[0098] 28 , in this embodiment, after the processes of steps ST125 and ST130 are performed on all first patch images 651 (YES in step ST135), the belongingness calculation unit 146 of the clustering processing unit 145 calculates the belongingness to multiple clusters and the probability of non-belonging of each first patch image 651 determined to have a morphological abnormality in the liver specimen LVS (step ST200). The belongingness calculation unit 146 generates belongingness information 160 that summarizes the calculation results of the belongingness and the probability of non-belonging for each first patch image 651. The belongingness information 160 is output from the belongingness calculation unit 146 to the clustering information generation unit 147. This completes the soft clustering process.

[0099] The clustering information generation unit 147 generates clustering information 161 based on the membership information 160. As a result, the first patch image 651 is clustered into a plurality of clusters (step ST205). The clustering information 161 is output from the clustering information generation unit 147 to the display control unit 91. The subsequent processing is the same as in the first embodiment, and therefore description thereof will be omitted.

[0100] In this way, the clustering processing unit 145 of this embodiment performs soft clustering processing to calculate the degree of belonging to each of multiple clusters of the first patch image 651 determined to have a morphological abnormality in the liver specimen LVS. Therefore, this embodiment also makes it possible to cluster the first patch image 651 of the target first specimen image 151T in which the distribution of the first feature amount 601 is not discrete. Furthermore, the drug discovery staff DS does not need to specify to which cluster the first patch image 651 belongs, thereby reducing the burden on the drug discovery staff DS.

[0101] As in the first_1 embodiment, the dimensions of the first feature amount 601 may be reduced, and then the degree of belonging may be calculated by the degree of belonging calculation unit 146 .

[0102] In this embodiment, when only one cluster image is superimposed on the target first specimen image 151T, a display mode such as the clustering result display screen 165 shown in FIG. 29 may be used. That is, in one cluster image superimposed on the target first specimen image 151T, the color intensity is changed depending on the degree of attribution. FIG. 29 illustrates an example in which only the cluster image 115 of cluster 1 is superimposed on the target first specimen image 151T. An example is also shown in which the color intensity is changed in three stages depending on the degree of attribution. The drug discovery staff DS can easily grasp the degree of attribution based on the color intensity. This further facilitates evaluation of the efficacy and toxicity of the candidate substance 11.

[0103] Second Embodiment In this embodiment, the display control unit 91 displays statistical information 171 based on the results of the clustering process, as shown in a clustering result display screen 170 in Fig. 30 as an example. The statistical information 171 is the number and proportion of first patch images 651 belonging to each of clusters 1 to 3. The proportion is calculated by dividing the number of first patch images 651 belonging to each of clusters 1 to 3 by the total number of first patch images 651.

[0104] By displaying the statistical information 171 in this way, the drug discovery staff DS can easily grasp the extent to which the first patch images 651 belong to each of clusters 1 to 3. Note that instead of or in addition to the number and proportion of the first patch images 651 belonging to each of clusters 1 to 3, the area of ​​the first patch images 651 belonging to each of clusters 1 to 3 may be displayed as statistical information.

[0105] [Third Embodiment] As an example, as shown in Figure 31, in this embodiment, in addition to the second learning patch image 652L, a third learning patch image 653L, which depicts a liver specimen LVS with morphological abnormality, is also used as training data for the autoencoder 70.

[0106] As an example, as shown in FIG. 32 , the third patch image for learning 653L is supplied from a plurality of third patch images 653 obtained by subdividing the third specimen image 153. The third specimen image 153 is an image of a liver specimen LVS of a subject S in a previously administered group 175. The previously administered group 175 is composed of a plurality of subjects S to whom a candidate substance was administered in a previous evaluation test. The number of subjects S constituting the previously administered group 175 is, for example, several hundred to several thousand, similar to the previous control group 75. Like the first specimen image 151 and the like, a plurality of third specimen images 153 are obtained from one subject S. Therefore, the number of third specimen images 153 obtained from the previously administered group 175 is calculated by multiplying the number obtained from one subject S by the number of subjects S.

[0107] The third learning patch image 653L is obtained by selecting a third patch image 653 depicting a liver specimen LVS having a morphological abnormality from among the plurality of third patch images 653. The third learning patch image 653L is selected manually, for example, by a pathologist or the like. Alternatively, the third learning patch image 653L may be selected by applying the method described in the first embodiment above, extracting a third feature amount from the third patch image 653, and determining whether or not a morphological abnormality has occurred in the liver specimen LVS based on the third feature amount.

[0108] In this way, in the third embodiment, not only the second training patch image 652L but also the third training patch image 653L, which depicts a liver specimen LVS with a morphological abnormality, is used as training data. Therefore, the autoencoder 70, and in turn the feature extractor 41, can learn about liver specimens LVS with a wider variety of shape and texture characteristics. As a result, it is possible to extract the first feature 601 that better represents the shape and texture characteristics of the liver specimen LVS.

[0109] Note that the patch image depicting the liver specimen LVS with morphological abnormality is not limited to the third learning patch image 653L obtained from the subject S constituting the exemplary past administration group 175. Morphological abnormalities may also occur in the subject S constituting the past control group 75. Therefore, as long as the patch image depicts a liver specimen LVS with morphological abnormality, it does not matter whether the subject S is in the past control group 75 or the past administration group 175. In other words, the second learning patch image 652L depicting a liver specimen LVS with morphological abnormality may be used as training data. Furthermore, the patch image depicting the liver specimen LVS with morphological abnormality may be an image obtained from a subject S in which various stresses have been applied to intentionally induce morphological abnormality. Alternatively, the patch image depicting a normal liver specimen LVS may be an image in which morphological abnormality is artificially created by processing a patch image depicting a normal liver specimen LVS.

[0110] 33 as an example, in this embodiment, an encoder unit 181 of a convolutional neural network (hereinafter referred to as a CNN (Convolutional Neural Network)) 180 is diverted as a feature extractor 182 that extracts a first feature 601 from a first patch image 651. The feature extractor 182 is an example of a "machine learning model" according to the technology of the present disclosure.

[0111] The CNN 180 has an output unit 183 in addition to an encoder unit 181. A patch image 65 such as a first patch image 651 is input to the encoder unit 181. The encoder unit 181 converts the patch image 65 into a feature 60. The encoder unit 181 passes the feature 60 to the output unit 183. The output unit 183 outputs a prediction result 184 based on the feature 60. The prediction result 184 is a prediction of one type of morphological abnormality occurring in the liver specimen LVS captured in the patch image 65 from multiple types such as hyperplasia, infiltration, congestion, and inflammation.

[0112] In the learning phase of the CNN 180, in addition to the second learning patch image 652L, patch images depicting liver specimens LVS with morphological abnormalities, as shown in the third embodiment, are primarily used as training data. As mentioned above, morphological abnormalities may also occur in the subjects S constituting the historical control group 75. Therefore, it is possible that the second learning patch image 652L also depicts liver specimens LVS with morphological abnormalities. Therefore, it is preferable to perform label smoothing on the correct answer data in the learning phase and add an error rate rather than distinguishing the type of morphological abnormality as 1 or 0. Furthermore, regions of the liver specimen LVS depicted in the third learning patch image 653L that are likely to be erroneously detected as morphological abnormalities may be masked before use in learning. Examples of regions that are likely to be erroneously detected as morphological abnormalities include dust particles adhering during the preparation of the slide specimen 18.

[0113] In this way, in the fourth embodiment, the CNN 180, and therefore the feature extractor 182, are models that are responsible for the task of identifying the type of morphological abnormality, and are therefore able to extract the first feature 601 that better represents the shape and texture characteristics of the liver specimen LVS.

[0114] Note that the machine learning model to be used as the feature extractor is not limited to the exemplified autoencoder 70 and CNN 180. A generator of a Generative Adversarial Network (GAN) may be used as the feature extractor. A machine learning model without a convolutional layer, such as a Vision Transformer (ViT), may also be used as the feature extractor.

[0115] Contrastive learning may be performed to reduce the distance between features derived from the same image in feature space and increase the distance between features derived from different images in feature space. Known examples of contrastive learning include learning methods such as SimCLR (A Simple Framework for Contrastive Learning of Visual Representations). Furthermore, learning methods such as BYOL (Bootstrap Your Own Latent) that do not use the above-described pairs of different images (also called negative samples) may also be used. Furthermore, constraints may be imposed on the distribution of extracted features, such as distribution on a unit sphere or distribution following a standard normal distribution.

[0116] [Fifth Embodiment] As an example, as shown in Fig. 34, the fifth embodiment deals with a slide specimen 190 in which tissue specimens of multiple organs are mounted on a single slide glass 16. Fig. 34 illustrates a case in which a heart specimen HS, a brain specimen BS, and a bone marrow specimen BMS are mounted in addition to a liver specimen LVS. In this case, the first specimen image 151 shows the heart specimen HS, the brain specimen BS, and the bone marrow specimen BMS in addition to the liver specimen LVS.

[0117] The CPU 32 of the image processing device 10 of the fifth embodiment functions as an identification unit 191 in addition to the respective processing units 50 to 53 of the first embodiment. The identification unit 191 identifies the tissue specimen of each organ from the first specimen image 151, for example, using a template for identifying the tissue specimen of each organ or a machine learning model. The identification unit 191 outputs, as an identification result, coordinate information of boxes 192 to 195 surrounding the tissue specimen of each organ. Box 192 is a box surrounding the heart specimen HS, and box 193 is a box surrounding the liver specimen LVS. Furthermore, box 194 is a box surrounding the brain specimen BS, and box 195 is a box surrounding the bone marrow specimen BMS.

[0118] 35 shows how the feature extraction unit 51 subdivides the cardiac specimen HS in the frame 192 into a plurality of first patch images 651, and how the feature extractor 200 for the cardiac specimen HS extracts first feature quantities 601 from the first patch images 651. The feature extraction unit 51 also extracts first feature quantities 601 for the liver specimen LVS in the frame 193, the brain specimen BS in the frame 194, and the bone marrow specimen BMS in the frame 195. That is, the feature extraction unit 51 extracts the first feature quantities 601 for each of the tissue specimens of each organ identified by the identification unit 191. The subsequent processing by the determination unit 52 is the same as the processing shown in the first embodiment and the like, except that determination is made for each of the tissue specimens of each organ depicted in the target first specimen image 151T, and therefore will not be illustrated or described again.

[0119] As described above, in the fifth embodiment, the first specimen image 151 is an image obtained by capturing a specimen slide 190 on which tissue specimens from multiple organs are placed. The identification unit 191 identifies the tissue specimens from each organ from the first specimen image 151. The feature extraction unit 51 extracts a first feature 601 for each identified tissue specimen from each organ. The determination unit 52 performs determination for each tissue specimen from each organ appearing in the target first specimen image 151T based on the first feature 601. This allows for support of the specimen slide 190 on which tissue specimens from multiple organs are placed. The specimen slide 190 on which tissue specimens from multiple organs are placed, as in this embodiment, is more common than the specimen slide 18 on which a tissue specimen from a single organ is placed, as in the first embodiment. This allows for processing that is more suited to general operations.

[0120] The drug discovery staff DS may manually define a frame showing the tissue specimen of each organ in the first specimen image 151 .

[0121] The feature 60 is not limited to that extracted by the feature extractor 41. It may be the average value, maximum value, minimum value, mode, or variance of the pixel values ​​of the patch image 65.

[0122] The organ is not limited to the liver LV, but may be the stomach, lung, small intestine, or large intestine. The subject S is not limited to a rat, but may be a mouse, guinea pig, gizzard shad, hamster, ferret, rabbit, dog, cat, monkey, or the like.

[0123] The image processing device 10 may be a personal computer installed in a pharmaceutical development facility as shown in FIG. 1, or may be a server computer installed in a data center independent of the pharmaceutical development facility.

[0124] When the image processing device 10 is configured as a server computer, the first specimen image 151 is transmitted from a personal computer installed in each pharmaceutical development facility to the server computer via a network such as the Internet. The server computer distributes various screens, such as the target selection screen 95, to the personal computer in the form of screen data for web distribution created using a markup language such as XML (Extensible Markup Language). The personal computer reproduces the screen to be displayed on the web browser based on the screen data and displays it on a display. Note that other data description languages, such as JSON (Javascript (registered trademark) Object Notation), may be used instead of XML.

[0125] The image processing device 10 according to the technique of the present disclosure can be widely used across all stages of pharmaceutical development, from the earliest stage of drug discovery target setting to the final stage of clinical trials.

[0126] The hardware configuration of the computer constituting the image processing device 10 according to the technology of the present disclosure can be modified in various ways. For example, the image processing device 10 can be configured with multiple computers separated as hardware in order to improve processing power and reliability. For example, the functions of the feature extraction unit 51 and the determination unit 52 and the functions of the clustering processing unit 53 or 145 can be distributed and performed by two computers. In this case, the image processing device 10 is configured with two computers.

[0127] In this way, the hardware configuration of the computer of the image processing device 10 can be changed as appropriate depending on the required performance such as processing power, safety, reliability, etc. Furthermore, not only the hardware but also application programs such as the operating program 40 can be duplicated or stored in a distributed manner across multiple storage devices in order to ensure safety and reliability.

[0128] In each of the above embodiments, the hardware structure of the processing units that perform various processes, such as the RW control unit 50, the feature extraction unit 51, the determination unit 52, the clustering processing units 53 and 145, the dimension reduction unit 90, the display control unit 91, the specification reception unit 92, the membership calculation unit 146, the clustering information generation unit 147, and the identification unit 191, can be any of the various processors listed below. As described above, the various processors include the CPU 32, which is a general-purpose processor that executes software (operating program 40) and functions as various processing units, as well as programmable logic devices (PLDs) that are processors whose circuit configuration can be changed after manufacture, such as FPGAs (Field Programmable Gate Arrays), and dedicated electrical circuits that are processors having a circuit configuration designed specifically for executing specific processing, such as ASICs (Application Specific Integrated Circuits).

[0129] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA).Furthermore, multiple processing units may be configured with a single processor.

[0130] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0131] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0132] From the above description, the technology described in the following supplementary paragraphs can be understood.

[0133] [Supplementary Item 1] An image processing device comprising a processor that acquires a first specimen image of a tissue specimen of a subject, extracts first feature amounts from first patch images obtained by dividing the first specimen image using a machine learning model, determines whether or not the tissue specimen depicted in the first patch image has a morphological abnormality based on the first feature amount, receives from a user a designation of to which cluster the first patch image, for which the tissue specimen has been determined to have the morphological abnormality, belongs, and clusters the first patch image into a plurality of clusters based on the designation, and calculates a degree of belonging of the first patch image, for which the tissue specimen has been determined to have the morphological abnormality, to each of a plurality of clusters. [Supplementary Item 2] The image processing device according to Supplementary Item 1, wherein the processor controls displaying a result of the manual clustering process or a result of the soft clustering process. [Supplementary Item 3] The image processing device according to Supplementary Item 2, wherein the result is displayed using a plurality of cluster images generated by processing the first specimen image, and the plurality of cluster images are images that make the plurality of clusters identifiable using a display format that is preset for each of the plurality of clusters. [Supplementary Item 4] The image processing device according to Supplementary Item 3, wherein the processor superimposes and displays at least one of the plurality of cluster images on the first specimen image. [Supplementary Item 5] The image processing device according to Supplementary Item 4, wherein the processor accepts, from the user, a designation of the cluster image to be superimposed and displayed on the first specimen image. [Supplementary Item 6] The image processing device according to any one of Supplementary Items 2 to 5, wherein the processor displays statistical information based on the result. [Supplementary Item 7] The image processing device according to any one of Supplementary Items 1 to 6, wherein, in the manual clustering process, the processor reduces the number of dimensions of the first feature amounts to two or three dimensions, displays a graph in which the dimensionally reduced first feature amounts are plotted in a two-dimensional or three-dimensional space, and accepts the designation on the graph.[Supplementary Item 8] The image processing device of any one of Supplementary Items 1 to 7, wherein the machine learning model is a model trained using, as training data, second patch images obtained by subdividing second specimen images, which depict tissue specimens from a plurality of subjects constituting a control group to which the candidate substance was not administered in a past evaluation test of a drug candidate substance. [Supplementary Item 9] The image processing device of Supplementary Item 8, wherein the training data also includes patch images of the tissue specimens in which the morphological abnormality has occurred. [Supplementary Item 10] The image processing device of Supplementary Item 9, wherein the machine learning model is a model responsible for identifying the type of the morphological abnormality. [Supplementary Item 11] The image processing device of any one of Supplementary Items 8 to 10, wherein the processor acquires information on the distribution of second feature amounts extracted using the machine learning model from second patch images obtained by subdividing the second specimen image, calculates a distance between the distribution and the first feature amount, and makes the determination based on the distance.

[0134] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs but also to storage media that non-temporarily store programs.

[0135] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0136] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0137] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A processor is provided. The processor, acquiring a first specimen image of a tissue specimen of a subject; Extracting a first feature amount from a first patch image obtained by dividing the first sample image using a machine learning model; determining whether or not a morphological abnormality occurs in the tissue specimen depicted in the first patch image based on the first feature amount; a manual clustering process for receiving, from a user, a designation to which cluster the first patch image, which has been determined to have the morphological abnormality in the tissue specimen, belongs, and clustering the first patch image into one of a plurality of clusters based on the designation; and A soft clustering process for calculating the degree of belonging to each of a plurality of clusters of the first patch image determined to have the morphological abnormality in the tissue specimen. Do one of the following: Image processing device.

2. The processor, The image processing device according to claim 1 , further comprising control for displaying a result of the manual clustering process or a result of the soft clustering process.

3. the result is represented by a plurality of cluster images generated by processing the first sample image; The image processing apparatus according to claim 2 , wherein the plurality of cluster images are images in which the plurality of clusters can be identified by a display format preset for each of the plurality of clusters.

4. The processor, The image processing apparatus according to claim 3 , wherein at least one of the plurality of cluster images is displayed in a superimposed manner on the first specimen image.

5. The processor, The image processing apparatus according to claim 4 , further comprising: receiving, from the user, designation of at least one of the cluster images to be superimposed and displayed on the first specimen image.

6. The processor, The image processing device according to claim 2 , wherein statistical information based on the results is displayed.

7. The processor, 2. The image processing device according to claim 1, wherein, in the manual clustering process, the number of dimensions of the first features is reduced to two or three dimensions, a graph is displayed in which the first features with the reduced number of dimensions are plotted in a two or three dimensional space, and the specification is accepted on the graph.

8. 2. The image processing device according to claim 1, wherein the machine learning model is a model trained using as training data second patch images that are subdivisions of second specimen images depicting tissue specimens of multiple subjects constituting a control group to which the candidate substance was not administered in a past evaluation test of a candidate pharmaceutical substance.

9. The image processing apparatus according to claim 8 , wherein the training data also includes a patch image in which the tissue specimen in which the morphological abnormality occurs is captured.

10. The image processing device according to claim 9 , wherein the machine learning model is a model tasked with identifying the type of the morphological abnormality.

11. The processor, acquiring information on a distribution of a second feature amount extracted using the machine learning model from a second patch image obtained by subdividing the second sample image; Calculating a distance between the distribution and the first feature amount; The image processing device according to claim 8 , wherein the determination is made based on the distance.

12. acquiring a first specimen image of a tissue specimen of a subject; extracting a first feature amount from a first patch image obtained by dividing the first sample image using a machine learning model; determining whether or not a morphological abnormality occurs in the tissue specimen depicted in the first patch image based on the first feature amount; and a manual clustering process for receiving, from a user, a designation to which cluster the first patch image, which has been determined to have the morphological abnormality in the tissue specimen, belongs, and clustering the first patch image into one of a plurality of clusters based on the designation; and A soft clustering process for calculating the degree of belonging to each of a plurality of clusters of the first patch image determined to have the morphological abnormality in the tissue specimen. To do one of the following: A method of operating an image processing apparatus comprising:

13. acquiring a first specimen image of a tissue specimen of a subject; extracting a first feature amount from a first patch image obtained by dividing the first sample image using a machine learning model; determining whether or not a morphological abnormality occurs in the tissue specimen depicted in the first patch image based on the first feature amount; and a manual clustering process for receiving, from a user, a designation to which cluster the first patch image, which has been determined to have the morphological abnormality in the tissue specimen, belongs, and clustering the first patch image into one of a plurality of clusters based on the designation; and A soft clustering process for calculating the degree of belonging to each of a plurality of clusters of the first patch image determined to have the morphological abnormality in the tissue specimen. To do one of the following: An operating program for an image processing device that causes a computer to execute a process including the steps of:

Images