Image processing device, method for operating image processing device, and program for operating image processing device

Abstract

The image processing apparatus includes a processor that acquires a first specimen image in which a tissue specimen of a subject is captured, extracts a first feature amount from a first small block image obtained by subdividing the first specimen image using a machine learning model, and calculates a first feature amount from a second small block image obtained by subdividing the first specimen image on the basis of the first feature amount. The present invention determines whether or not a morphological abnormality has occurred in a tissue specimen captured on a first tile image, and performs either a manual clustering process in which a designation of which cluster the first tile image determined to have occurred a morphological abnormality in the tissue specimen belongs to is accepted from a user, or a soft clustering process in which the first tile image determined to have occurred a morphological abnormality in the tissue specimen belongs to is accepted from the user. And a soft clustering process in which the first tile image is clustered into a plurality of clusters on the basis of the designation, and in which the degree of attribution of the first tile image, which has been determined to have a morphological abnormality in the tissue specimen, to each of the plurality of clusters is calculated.

Description

Technical Field

[0001] The technology disclosed herein relates to an image processing device, an operating method of the image processing device, and an operating program of the image processing device. Background Art

[0002] The specification of U.S. patent application publication No. 2021 / 0342570 describes a technology that uses machine learning models such as autoencoders to extract feature quantities from small blocks of images obtained by subdividing specimen images of tissue specimens such as animal livers, and based on the extracted feature quantities, determines whether morphological abnormalities (hyperplasia, infiltration, congestion, inflammation, tumors, canceration, proliferation, hemorrhage, glycogen reduction, etc.) occur in the tissue specimens photographed on the small blocks of images, and clusters (hard clustering) the small blocks of images that are determined to have morphological abnormalities into one of multiple clusters based on the feature quantities. Summary of the invention

[0003] Technical issues to be solved by the invention

[0004] In order to determine whether morphological abnormalities occur, not only image information of the part where the morphological abnormalities occur is required, but also image information of the surrounding area. For example, as a morphological abnormality, when determining whether bile duct hyperplasia occurs, even a specialized pathologist can hardly determine whether the part is hyperplasia by only observing the hyperplastic bile duct part, and it is necessary to observe with a larger field of view including the part and its surrounding area. Therefore, in the specification of U.S. Patent Application Publication No. 2021 / 0342570, it is also expected that the small block image input into the machine learning model for extracting feature quantities will be set to a size that covers not only the part with morphological abnormalities but also the surrounding area.

[0005] However, if the size of the small image is set as described above, the feature quantity becomes a feature quantity derived not only from the morphologically abnormal part but also from the surrounding area. Therefore, the distribution of the feature quantity of the small image determined to have morphologically abnormalities does not become discrete. If the distribution of the feature quantity is not discrete, it is difficult to perform hard clustering of the small image determined to have morphologically abnormalities through computer processing as in the specification of U.S. Patent Application Publication No. 2021 / 0342570.

[0006] One embodiment of the technology disclosed herein provides an image processing device capable of clustering small image blocks of a specimen image whose feature quantity distribution is not discrete, an operating method of the image processing device, and an operating program of the image processing device.

[0007] Means for solving technical problems

[0008] The image processing device disclosed in the present invention includes a processor, which obtains a first specimen image of a tissue specimen of a subject, uses a machine learning model to extract a first feature value from a first small block image obtained by subdividing the first specimen image, determines based on the first feature value whether morphological abnormalities occur in the tissue specimen photographed on the first small block image, and performs either a manual clustering process or a soft clustering process. In the manual clustering process, the processor receives a designation from a user as to which cluster the first small block image determined to have morphological abnormalities in the tissue specimen belongs to, and clusters the first small block image into a plurality of clusters based on the designation. In the soft clustering process, the degree of belonging of the first small block image determined to have morphological abnormalities in the tissue specimen to each of the plurality of clusters is calculated.

[0009] Preferably, the processor performs control to display a result of the manual clustering process or a result of the soft clustering process.

[0010] Preferably, the result is displayed using a plurality of cluster images generated by processing the first specimen image, and the plurality of cluster images are images that allow the plurality of clusters to be identified using a display format that is preset for each of the plurality of clusters.

[0011] Preferably, the processor displays at least one of the plurality of cluster images in an overlapping manner on the first specimen image.

[0012] Preferably, the processor accepts a designation of a cluster image to be displayed superimposed on the first specimen image from a user.

[0013] Preferably, the processor displays statistical information based on the results.

[0014] Preferably, the processor reduces the dimension of the first feature value to two or three dimensions in the manual clustering process, displays a graph in which the first feature value with reduced dimension is plotted in a two-dimensional space or a three-dimensional space, and accepts a designation on the graph.

[0015] Preferably, the machine learning model is a model obtained by learning using a second small block image obtained by subdividing a second specimen image as training data, wherein the second specimen image is captured with tissue specimens of multiple subjects constituting a control group that was not fed with the candidate substance in past evaluation tests of the candidate substance of the drug.

[0016] Preferably, the training data also includes small images of tissue samples with morphological abnormalities.

[0017] Preferably, the machine learning model is a model responsible for the task of identifying the type of morphological abnormality.

[0018] Preferably, the processor obtains information on the distribution of the second feature quantity, calculates the distance between the distribution and the first feature quantity, and makes a determination based on the distance, wherein the second feature quantity is a feature quantity extracted from a second small block image obtained by subdividing the second specimen image using a machine learning model.

[0019] The working method of the image processing device disclosed in the present invention includes: a step of acquiring a first specimen image of a tissue specimen of a subject; a step of extracting a first feature quantity from a first small block image obtained by subdividing the first specimen image using a machine learning model; a step of determining whether morphological abnormalities occur in the tissue specimen photographed on the first small block image based on the first feature quantity; and a step of performing either a manual clustering process or a soft clustering process, wherein in the manual clustering process, a designation is received from a user as to which cluster a first small block image determined to have morphological abnormalities in the tissue specimen belongs to, and the first small block image is clustered into a plurality of clusters based on the designation, and in the soft clustering process, a degree of belonging of the first small block image determined to have morphological abnormalities in the tissue specimen to each of the plurality of clusters is calculated.

[0020] The working program of the image processing device disclosed in the present invention causes a computer to execute processing including the following steps: a step of acquiring a first specimen image of a tissue specimen of a subject; a step of extracting a first feature quantity from a first small block image obtained by subdividing the first specimen image using a machine learning model; a step of determining whether morphological abnormalities occur in the tissue specimen photographed on the first small block image based on the first feature quantity; and a step of performing either a manual clustering process or a soft clustering process, wherein in the manual clustering process, a designation of which cluster a first small block image determined to have morphological abnormalities in the tissue specimen belongs to is accepted from a user, and the first small block image is clustered into a plurality of clusters based on the designation, and in the soft clustering process, a degree of belonging of the first small block image determined to have morphological abnormalities in the tissue specimen to each of the plurality of clusters is calculated.

[0021] Effects of the Invention

[0022] According to the technology disclosed in the present disclosure, it is possible to provide an image processing device capable of clustering small image blocks of a specimen image in which the distribution of feature quantities is not discrete, an operating method of the image processing device, and an operating program of the image processing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a diagram showing the process of an evaluation test, a first specimen image, and an image processing device.

[0024] Figure 2 This is a block diagram showing a computer constituting the image processing apparatus.

[0025] Figure 3This is a block diagram showing a processing unit of a CPU of an image processing device.

[0026] Figure 4 A diagram showing a first small block image obtained by segmenting the first specimen image.

[0027] Figure 5 This is a diagram showing how the first feature quantity is extracted from the first small block image using the feature quantity extractor.

[0028] Figure 6 It is a diagram showing the formation process of the feature quantity extractor.

[0029] Figure 7 is a diagram showing the processing in the learning phase of the autoencoder.

[0030] Figure 8 It is a diagram showing the formation process of the control group and the second small block image for learning in the past.

[0031] Fig. 9 This is a diagram showing how the second feature quantity is extracted from the second small block image using the feature quantity extractor.

[0032] Fig.10 It is a graph in which the second feature quantity is plotted in the feature quantity space, and a diagram showing the second feature quantity distribution information.

[0033] Fig.11 This is a diagram showing the distance between the position of the first feature value and the representative position of the second feature value.

[0034] Fig.12 It is a diagram showing the processing of the determination unit.

[0035] Fig.13 It is a diagram showing the processing of the determination unit.

[0036] Fig.14 It is a diagram showing the detailed structure of the clustering processing unit.

[0037] Fig.15 This is a diagram showing an object specifying screen.

[0038] Fig.16 It is a diagram showing the processing of the dimensionality reduction unit.

[0039] Fig.17 This figure shows a cluster specification screen.

[0040] Fig.18 This is a diagram showing the processing of the designation receiving unit when a drug developer designates a cluster on the cluster designation screen.

[0041] Fig.19 A diagram showing a cluster image and an overlapped image.

[0042] Fig. 20 This figure shows the clustering result display screen.

[0043] Fig.21 This is a diagram showing a clustering result display screen displaying a superimposed image obtained by superimposing only one cluster image on a target first specimen image.

[0044] Fig. 22 This is a flowchart showing the processing steps of the image processing device.

[0045] Fig.23 This is a flowchart showing the processing steps of the image processing device.

[0046] Fig.24 It is a diagram showing the processing of the dimensionality reduction unit of the modified example.

[0047] Fig.25 It is a diagram showing a cluster specifying screen according to a modified example.

[0048] Fig.26 This is a diagram showing the detailed structure of the clustering processing unit of the first and second embodiments.

[0049] Fig. 27 This is a diagram showing the processing of the attribution degree calculation unit and the cluster information generation unit.

[0050] Fig.28 It is a flowchart showing the processing steps of the image processing device of the 1st_2nd embodiment.

[0051] Fig.29 This is a diagram showing a clustering result display screen of the first and second implementation modes.

[0052] Fig.30 This figure shows a clustering result display screen showing statistical information.

[0053] Fig.31 It is a diagram showing the processing in the learning phase of the autoencoder according to the third embodiment.

[0054] Fig.32 It is a figure which shows the formation process of the past feeding group and the third small image block for learning.

[0055] Fig.33 It is a diagram showing a process of forming a feature quantity extractor according to the fourth embodiment.

[0056] Fig.34 This is a diagram showing a fifth embodiment of processing a specimen image obtained by imaging a glass slide specimen on which tissue specimens of various organs are mounted.

[0057] Fig.35This is a diagram showing how a first feature quantity is extracted from a first small block image obtained by segmenting a heart sample using a feature quantity extractor for a heart sample. DETAILED DESCRIPTION

[0058] [Implementation Method 1_1]

[0059] As an example, Figure 1 As shown, the image processing device 10 of the present disclosure is used to evaluate the efficacy and toxicity of a candidate substance 11 of a drug. The image processing device 10 is, for example, a desktop personal computer, and is equipped with a display 12 for displaying various screens and an input device 13 such as a keyboard, a mouse, a touch screen, and / or a microphone for sound input. The image processing device 10 is, for example, set in a drug development facility and operated by a drug developer DS who participates in drug development in the drug development facility. The drug developer DS also includes a pathologist. The drug developer DS is an example of a "user" involved in the technology of the present disclosure.

[0060] The first specimen image 151 is input to the image processing device 10. The first specimen image 151 is an image used to evaluate the efficacy and toxicity of the candidate substance 11. The first specimen image 151 is generated, for example, by the following steps. First, a subject S such as a rat prepared for the evaluation of the candidate substance 11 is dissected, and the organs of the subject S are collected. Here, a plurality of tissue specimens (hereinafter referred to as liver specimens) LVS of the cross section of the liver LV are collected. Next, after the collected liver specimens LVS are pasted one by one on the slide 16, the liver specimen LVS is stained, and here, hematoxylin-eosin staining is used. Next, the stained liver specimen LVS is covered with a cover glass 17 to complete the slide specimen 18. Then, the slide specimen 18 is placed on a photographic device 19 such as a digital optical microscope, and the first specimen image 151 is photographed by the photographic device 19. The first specimen image 151 thus obtained is labeled with a specimen ID (Identification Data) for uniquely identifying the specimen S, a specimen image ID for uniquely identifying the first specimen image 151, and a photographing date, etc. In addition, a tissue specimen is also referred to as a tissue section. In addition, staining may be staining using hematoxylin alone, staining using nuclear fast red pigment, etc.

[0061] Here, the feeding group and the control group are explained. The feeding group is composed of a plurality of subjects S fed with the candidate substance 11. The control group is the opposite of the feeding group and is composed of a plurality of subjects S not fed with the candidate substance 11. The number of subjects S constituting the feeding group and the number of subjects S constituting the control group are both about 5 to 10, for example. The subjects S constituting the feeding group and the subjects S constituting the control group are subjects S having the same attributes and placed in the same breeding environment. The same attributes refer to, for example, the same age and / or the same sex. In addition, the same attributes also include the same age composition ratio and / or the same sex composition ratio (5 males and 5 females each, etc.). The same breeding environment refers to, for example, the same feed, the same temperature and humidity of the breeding space, and / or the same area of ​​the breeding space. The "same" of the same breeding environment means that, in addition to being completely the same, it also includes the same meaning of errors that are generally allowed in the technical field to which the technology of the present disclosure belongs and within the scope of the purpose of the technology of the present disclosure.

[0062] In addition, among the feeding groups, there are multiple groups with different feeding amounts of the candidate substance 11. For example, the feeding amount of the candidate substance 11 is different in three stages, namely, a high feeding amount group, a medium feeding amount group, and a small feeding amount group. In this way, the effect of the feeding amount of the candidate substance 11 on the subject S can be clearly seen.

[0063] The first specimen image 151 may be an image of a liver specimen LVS of a subject S in a feeding group or an image of a liver specimen LVS of a subject S in a control group. Figure 1 , a case where the first sample image 151 is an image of a liver sample LVS of a subject S in a feeding group is illustrated.

[0064] As an example, Figure 2 As shown, the computer constituting the image processing apparatus 10 includes a storage device 30, a memory 31, a CPU (Central Processing Unit) 32, and a communication unit 33 in addition to the display 12 and the input device 13 described above. These are connected to each other via a bus line 34.

[0065] The storage device 30 is a hard disk drive built into the computer constituting the image processing device 10 or connected via a cable or a network. Alternatively, the storage device 30 is a disk array formed by connecting a plurality of hard disk drives. The storage device 30 stores control programs such as an operating system, various application programs, and various data attached to these programs. In addition, a solid-state drive may be used instead of a hard disk drive.

[0066] The memory 31 is a working memory for the CPU 32 to perform processing. The CPU 32 loads the program stored in the storage device 30 into the memory 31 and performs processing according to the program. Thus, the CPU 32 controls the various parts of the computer in a unified manner. The CPU 32 is an example of a "processor" involved in the technology disclosed in the present invention. In addition, the memory 31 can also be built into the CPU 32. The communication unit 33 controls the transmission of various information with external devices such as the photographic device 19.

[0067] As an example, Figure 3 As shown, in the storage device 30 of the image processing device 10, a working program 40 is stored. The working program 40 is an application program for making the computer function as the image processing device 10. That is, the working program 40 is an example of a "working program of the image processing device" involved in the technology disclosed in the present invention. In the storage device 30, a feature value extractor 41 and second feature value distribution information 42 are also stored. The feature value extractor 41 is an example of a "machine learning model" involved in the technology disclosed in the present invention. In addition, the second feature value distribution information 42 is an example of "information on the distribution of the second feature value" involved in the technology disclosed in the present invention.

[0068] When the work program 40 is started, the CPU 32 of the computer constituting the image processing device 10 cooperates with the memory 31 and the like to function as a read / write (hereinafter referred to as RW (Read Write)) control unit 50 , a feature extraction unit 51 , a determination unit 52 , and a clustering processing unit 53 .

[0069] The RW control unit 50 controls storage of various data in the storage device 30 and reading of various data in the storage device 30. For example, the RW control unit 50 stores the first specimen image 151 from the imaging device 19 in the storage device 30. In addition, in practice, a plurality of first specimen images 151 are obtained from one subject S, so a plurality of first specimen images 151 are stored in the storage device 30 for one subject S.

[0070] The RW control unit 50 obtains the first specimen image 151 corresponding to the designation made by the drug developer DS through the input device 13 by reading from the storage device 30. The RW control unit 50 outputs the read first specimen image 151 to the feature extraction unit 51 and the clustering processing unit 53. The first specimen image 151 output from the RW control unit 50 to the feature extraction unit 51 and the like is a target for determining whether or not morphological abnormalities occur in the liver specimen LVS. Hereinafter, the first specimen image 151 for determining whether or not morphological abnormalities occur in the liver specimen LVS is referred to as the target first specimen image 151T (refer to Figure 4 wait).

[0071] The RW control unit 50 reads the feature quantity extractor 41 from the storage device 30, and outputs the read feature quantity extractor 41 to the feature quantity extractor 51. In addition, the RW control unit 50 obtains the second feature quantity distribution information 42 by reading it from the storage device 30. The RW control unit 50 outputs the read second feature quantity distribution information 42 to the determination unit 52.

[0072] The feature amount extraction section 51 extracts the first feature amount 601 from the target first specimen image 151T using the feature amount extractor 41. The feature amount extraction section 51 outputs the first feature amount 601 to the determination section 52.

[0073] The determination unit 52 determines whether morphological abnormality occurs in the liver sample LVS captured on the target first sample image 151T based on the first feature quantity 601 and the second feature quantity distribution information 42. Morphological abnormality refers to lesions that cannot be seen in normal liver sample LVS, such as hyperplasia, infiltration, congestion, inflammation, tumor, canceration, proliferation, hemorrhage, glycogen reduction, etc. The determination unit 52 outputs the determination result 61 of whether morphological abnormality occurs in the liver sample LVS captured on the target first sample image 151T to the clustering processing unit 53.

[0074] In this embodiment, the clustering processing unit 53 performs manual clustering processing. The manual clustering processing will be described later.

[0075] As an example, Figure 4 As shown, the feature extraction unit 51 uses a known image recognition technology to identify the liver sample LVS captured on the target first sample image 151T, and divides the identified liver sample LVS into a plurality of first small block images 651. The first small block images 651 have a predetermined size that can be processed by the feature extractor 41. In addition, the first small block images 651 have a size that covers not only the morphologically abnormal portion but also the surrounding area. The feature extraction unit 51 labels the first small block images 651 with small block image IDs 85 (see Fig.12 In addition, the feature amount extraction unit 51 extracts information indicating the position of the first specimen image 151T where the first small block image 651 is cut out, that is, the position information 86 (see Fig.12 etc.) are associated with the small block image ID 85. Figure 4 In the embodiment, the first small block image 651 does not have an area that overlaps with other first small block images 651 , but the first small block image 651 may also partially overlap with other first small block images 651 .

[0076] As an example, Figure 5As shown, the feature amount extracting unit 51 extracts the first feature amount 601 for each of the plurality of first small block images 651 obtained by segmenting the target first specimen image 151T using the feature amount extractor 41. Therefore, the number of the first feature amounts 601 is the same as the number of the first small block images 651.

[0077] As an example, Figure 6 As shown, the encoding unit 71 of the automatic encoder 70 is transferred to the feature quantity extractor 41. The automatic encoder 70 has a decoding unit 72 in addition to the encoding unit 71. The small block image 65 such as the first small block image 651 is input to the encoding unit 71. The encoding unit 71 converts the small block image 65 into the feature quantity 60. The encoding unit 71 transfers the feature quantity 60 to the decoding unit 72. The decoding unit 72 generates a restored image 73 of the small block image 65 based on the feature quantity 60.

[0078] As is well known, the encoding unit 71 includes a convolution layer that performs convolution processing using a filter, a pooling layer that performs pooling processing such as maximum pooling processing, and the like. The same is true for the decoding unit 72. The encoding unit 71 extracts the feature amount 60 by repeating the convolution processing by the convolution layer and the pooling processing by the pooling layer for a plurality of times on the input small block image 65. The extracted feature amount 60 represents the features of the shape and texture of the liver specimen LVS captured on the small block image 65.

[0079] The feature quantity 60 is a set of multiple numerical values. That is, the feature quantity 60 is multidimensional data. The dimension of the feature quantity 60 is, for example, 512, 1024, or 2048. The first feature quantity 601 and the second feature quantity 602 described later (see Fig. 9 ) have the same dimension and can be in the same feature space 81 (refer to Fig.10 etc.) for comparison.

[0080] As an example, Figure 7 As shown, in the learning stage before the encoding unit 71 is transferred to the feature extractor 41, the second small block image 652L for learning is input to the autoencoder 70 and learned. The autoencoder 70 outputs a restored image 73L for learning with respect to the second small block image 652L for learning. Based on these second small block images 652L for learning and the restored image 73L for learning, a loss operation of the autoencoder 70 using a loss function is performed. Then, based on the result of the loss operation, update settings of various coefficients of the autoencoder 70 (coefficients of filters of the convolutional layer, etc.) are performed, and the autoencoder 70 is updated according to the update settings. The second small block image 652L for learning is an example of "training data" involved in the technology disclosed in the present invention.

[0081] In the learning phase of the auto encoder 70, the above series of processes of inputting the second small block image 652L for learning to the auto encoder 70, outputting the restored image 73L for learning from the auto encoder 70, loss calculation, update setting, and updating of the auto encoder 70 are repeated while exchanging the second small block image 652L for learning. The repetition of the above series of processes ends when the restoration accuracy of the restored image 73L for learning from the second small block image 652L for learning reaches a preset level. In this way, the encoding unit 71 of the auto encoder 70 whose restoration accuracy reaches the set level is stored in the storage device 30 of the image processing device 10 as the feature quantity extractor 41. In addition, the learning may be ended when the above series of processes are repeated a set number of times, regardless of the restoration accuracy of the restored image 73L for learning from the second small block image 652L for learning.

[0082] Such learning of the autoencoder 70 may be performed by the image processing device 10 or by a device different from the image processing device 10. In the latter case, the feature extractor 41 is sent from another device to the image processing device 10, and the feature extractor 41 is stored in the storage device 30 through the RW control unit 50.

[0083] As an example, Figure 8 As shown, the second small block image 652L for learning is provided from a plurality of second small block images 652 obtained by subdividing the second specimen image 152. The second specimen image 152 is an image of a liver specimen LVS of a subject S of a past control group 75. The past control group 75 is composed of a plurality of subjects S to which the candidate substance was not administered in past evaluation tests. Therefore, the number of subjects S constituting the past control group 75 is much greater than the number of subjects S constituting the administration group and the control group, for example, about several hundred to several thousand. The second specimen image 152 is also obtained from a plurality of subjects S as in the first specimen image 151, so the number of second specimen images 152 obtained by multiplying the number obtained from one subject S by the number of subjects S is obtained from the past control group 75.

[0084] Next, the formation process of the second feature quantity distribution information 42 is described. First, as an example, Fig. 9 As shown, the feature amount extractor 41 is used to extract a plurality of second feature amounts 602 from each of a plurality of second small block images 652 obtained based on a plurality of second specimen images 152 obtained from all the subjects S constituting the past control group 75 .

[0085] As an example, Fig.10 The diagram 80 shown is the Fig. 9The second feature quantity distribution information 42 includes coordinates 82 of the representative position of the second feature quantity 602 in the feature quantity space 81, which are indicated by the mark x (hereinafter referred to as representative position coordinates). The representative position is, for example, the center point or average point of the distribution 83 of the second feature quantity 602. Fig.10 In the description, for convenience, the dimension of the feature space 81 is set to two dimensions with a D1 axis and a D2 axis, but in fact, the dimension of the feature space 81 is 512 dimensions as described above. Fig.11 Likewise, for convenience of explanation, the dimension of the feature space 81 is expressed in two dimensions.

[0086] Similar to the learning of the auto encoder 70, the second feature quantity distribution information 42 may be created by the image processing device 10 or by a device different from the image processing device 10. In the latter case, the second feature quantity distribution information 42 is sent from another device to the image processing device 10, and the second feature quantity distribution information 42 is stored in the storage device 30 through the RW control unit 50. In addition, the second feature quantity distribution information 42 may also be a plurality of second feature quantities 602 themselves. In this case, the representative position coordinates 82 are derived by the image processing device 10.

[0087] As an example, Fig.11 As shown, the determination unit 52 calculates the distance D between the representative position of the second feature quantity 602 represented by the representative position coordinate 82 of the second feature quantity distribution information 42 and the position of the first feature quantity 601 in the feature quantity space 81. The determination unit 52 calculates the distance D between the plurality of first feature quantities 601 extracted from each of the plurality of first small block images 651 obtained by subdividing one target first specimen image 151T. The distance D is the Mahalanobis distance. In addition, as the distance D, any one of the average value, the median value, and the maximum value of the Euclidean distance between the position of the k nearest neighbor samples of the distribution 83 of the second feature quantity 602 and the position of the first feature quantity 601 may be calculated. Alternatively, instead of the distance D, a value obtained by subtracting the cosine similarity between the vector representing the representative position of the second feature quantity 602 and the vector representing the position of the first feature quantity 601 from 1.0 may be calculated. The cosine similarity takes a value between -1.0 and 1.0, and it can be said that the larger the value, the more similar the directions of the vectors are.

[0088] As an example, Fig.12As shown, the determination unit 52 compares the calculated distance D with a preset threshold value, and when the distance D is lower than the threshold value, determines that no morphological abnormality occurs in the liver sample LVS imaged on the first small block image 651. The determination unit 52 outputs a determination result 61 that no morphological abnormality occurs in the liver sample LVS imaged on the first small block image 651. The determination result 61 in this case includes a small block image ID 85 and position information 86.

[0089] On the other hand, as an example, Fig.13 As shown, when the distance D is greater than the threshold, the determination unit 52 determines that a morphological abnormality has occurred in the liver sample LVS captured on the first small block image 651. The determination unit 52 outputs a determination result 61 that a morphological abnormality has occurred in the liver sample LVS captured on the first small block image 651. The determination result 61 in this case includes the first feature quantity 601 in addition to the small block image ID 85 and the position information 86. In addition, the threshold value may be configured so that the drug developer DS can set and change it. Instead of the distance D, it is also possible to determine whether a morphological abnormality has occurred in the liver sample LVS captured on the first small block image 651 by comparing the above cosine similarity with the threshold value.

[0090] like Figure 8 and Fig. 9 As shown, the second feature quantity 602 is a feature quantity extracted from a second small image 652 obtained by segmenting the second sample image 152 of the liver sample LVS of the subject S of the past control group 75. Since the subject S of the past control group 75 is a subject to which the candidate substance is not administered, at least no morphological abnormality due to the toxicity of the candidate substance occurs in the liver sample LVS photographed on the second sample image 152. Therefore, the representative position of the second feature quantity 602 is regarded as the representative position of the feature quantity of the sample image in which the normal liver sample LVS is photographed. Therefore, the distance D between the position of the first feature quantity 601 and the representative position of the second feature quantity 602 is an indicator indicating how much the liver sample LVS photographed on the first small image 651 deviates from the normal liver sample LVS. Therefore, for the first small image 651 whose distance D is less than the threshold, the determination unit 52 regards that it does not deviate from the normal liver sample LVS and determines that no morphological abnormality occurs in the liver sample LVS. On the other hand, the determination unit 52 regards the first small block image 651 for which the distance D is equal to or greater than the threshold as being different from the normal liver sample LVS, and determines that a morphological abnormality has occurred in the liver sample LVS.

[0091] As an example, Fig.14As shown, the clustering processing unit 53 includes a dimensionality reduction unit 90, a display control unit 91, and a designation receiving unit 92. These dimensionality reduction unit 90, display control unit 91, and designation receiving unit 92 perform manual clustering processing described below. The determination result 61 from the determination unit 52 is input to the dimensionality reduction unit 90. The target first specimen image 151T from the RW control unit 50 is input to the display control unit 91. The display control unit 91 controls the display of various screens on the display 12. The designation receiving unit 92 receives various designations made by the drug developer DS through the input device 13.

[0092] As an example, Fig.15 As shown, the display control unit 91 controls the display of the object designation screen 95 on the display 12. When the drug developer DS issues a display instruction of the first specimen image 151 through the input device 13 and the designation receiving unit 92 receives the display instruction, the display control unit 91 displays the object designation screen 95. A plurality of first specimen images 151 are displayed in parallel on the object designation screen 95. These plurality of first specimen images 151 are first specimen images 151 obtained from one of the plurality of specimens S constituting the feeding group. Fig.15 , an example is shown in which 10 first specimen images 151 of specimen image IDs “SI00001” to “SI00010” obtained from a specimen S having a specimen ID “R001” are displayed in parallel. In addition, the specimen ID is a pull-down menu 96 that can switch the specimen S whose first specimen image 151 is displayed in the object designation screen 95 .

[0093] The target designation screen 95 is a screen for designating one target first specimen image 151T from among the plurality of first specimen images 151. The target designation screen 95 is provided with a selection frame 97 that can be moved between the first specimen images 151. In addition, an analysis button 98 is provided at the lower portion of the target designation screen 95. The drug developer DS selects the analysis button 98 after aligning the selection frame 97 with the desired first specimen image 151. Thus, the first specimen image 151 aligned with the selection frame 97 is used as the target first specimen image 151T, and the first feature amount 601 is extracted by the feature amount extraction unit 51, and the determination of whether morphological abnormality occurs is performed by the determination unit 52.

[0094] As an example, Fig.16As shown, the dimension reduction unit 90 reduces the dimension of the first feature quantity 601 contained in the judgment result 61 of the first small block image 651 judged to have morphological abnormality in the liver specimen LVS to two dimensions. The dimension reduction unit 90 generates the dimension reduced data 100 based on the judgment result 61. The dimension reduced data 100 includes the first feature quantity 601R after dimension reduction, the small block image ID 85, and the position information 86. The dimension reduction unit 90 outputs the dimension reduced data 100 to the display control unit 91. In addition, as a dimension reduction method, principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), or UMAP (Uniform Manifold Approximation and Projection) can be used.

[0095] When the display control unit 91 receives the dimension-reduced data 100 from the dimension-reducing unit 90, as an example, Fig.17 The cluster designation screen 105 shown is displayed on the display 12. A graph 106 is displayed on the cluster designation screen 105. The graph 106 is a graph in which the first two-dimensional feature quantity 601R after dimension reduction included in the dimension reduced data 100 is plotted in a two-dimensional feature quantity space 107.

[0096] As described above, the first small block image 651 has a size that covers not only the morphological abnormality portion but also the area around it. Therefore, the distribution of the first feature quantity 601R does not become discrete.

[0097] On the right side of the chart 106, a cluster designation toolbar 108 is provided. The cluster designation toolbar 108 is a toolbar that contains icons of various tools that are used by the drug developer DS to designate which cluster the first small block image 651 determined to have morphological abnormality in the liver specimen LVS belongs to on the chart 106. The tool includes a tool for enclosing a plot of the first feature quantity 601R considered to be a cluster with a rectangle, circle, ellipse, or arbitrary curve. In addition, the tool also includes a tool for eliminating the rectangle, circle, ellipse, or arbitrary curve that encloses the first feature quantity 601R, or a tool for changing the designated cluster.

[0098] The drug developer DS uses various tools of the cluster-specific toolbar 108, such as Fig.18 As shown, it is specified on the chart 106 to which cluster the first small piece of image 651 belongs. Fig.18, a case where three clusters, clusters 1, 2, and 3, are specified by enclosing them with ellipses. In addition, as illustrated, among the points of the first feature amount 601R, there are also points that do not belong to any cluster.

[0099] A designation completion button 109 is provided at the lower part of the cluster designation screen 105. After the drug developer DS designates a cluster on the graph 106, he selects the designation completion button 109. Thus, the designation acceptance unit 92 accepts the designation by the drug developer DS as to which cluster the first small block image 651 determined to have morphological abnormality in the liver sample LVS belongs.

[0100] The designation accepting unit 92 generates clustering information 112 based on the accepted designation. The clustering information 112 is information that registers the cluster to which the small block image ID 85 of each first small block image 651 belongs. The first small block image 651 whose points of the first feature amount 601R do not belong to any cluster is not registered in the clustering information 112. The designation accepting unit 92 outputs the clustering information 112 to the display control unit 91.

[0101] As an example, Fig.19 As shown in FIG. 1 , the display control unit 91 generates cluster images 115, 116, and 117 by processing the target first specimen image 151T according to the cluster information 112 from the designation receiving unit 92. The cluster image 115 is Fig.18 Cluster image 116 is an image corresponding to cluster 1 shown in FIG. Fig.18 Cluster image 117 is an image corresponding to cluster 2 shown in FIG. Fig.18 The image corresponding to cluster 3 is shown in .

[0102] The display control unit 91 generates cluster images 115 to 117 according to the display format 118 preset for each of the clusters 1 to 3. For example, the display format 118 displays cluster 1 in blue, cluster 2 in yellow-green, and cluster 3 in gray. The display control unit 91 fills the position of the first small block image 651 of the small block image ID 85 of cluster 1 registered in the cluster information 112 with blue (as known from the position information 86) in the target first specimen image 151T, thereby generating the cluster image 115. Similarly, the display control unit 91 fills the position of the first small block image 651 of the small block image ID 85 of cluster 2 registered in the cluster information 112 with yellow-green in the target first specimen image 151T, thereby generating the cluster image 116. Furthermore, the display control unit 91 fills the position of the first small block image 651 of the small block image ID 85 of cluster 3 registered in the cluster information 112 with gray in the target first specimen image 151T, thereby generating the cluster image 117. Thus, by changing the displayed color, cluster images 115 to 117 become images that can identify clusters 1 to 3. Cluster images 115 to 117 are examples of "manual clustering processing results" involved in the technology disclosed herein. In addition, the display format 118 can also be configured so that the drug developer DS can freely change the settings.

[0103] The display control unit 91 generates a superimposed image 119 by superimposing the target first specimen image 151T on at least one of the cluster images 115 to 117. Fig.19 , a superimposed image 119 obtained by superimposing all the cluster images 115 to 117 on the target first specimen image 151T is illustrated.

[0104] As an example, Fig. 20 and Fig.21 As shown, the display control unit 91 controls the display 12 to display a clustering result display screen 125 including the overlapped image 119. In the clustering result display screen 125, in addition to the overlapped image 119, there are provided a legend 126 of clusters 1 to 3 and display switching buttons 127, 128, and 129. The display switching button 127 is a button for selecting whether to display the cluster image 115 overlapped on the target first specimen image 151T. The display switching button 128 is a button for selecting whether to display the cluster image 116 overlapped on the target first specimen image 151T. The display switching button 129 is a button for selecting whether to display the cluster image 117 overlapped on the target first specimen image 151T. The designation receiving unit 92 receives a selection instruction of the display switching buttons 127 to 129 from the drug developer DS.

[0105] When the display switching button 127 is selected, the display control unit 91 displays the cluster image 115 in a superimposed manner on the target first specimen image 151T. On the other hand, when the display switching button 127 is not selected, the display control unit 91 does not display the cluster image 115 in a superimposed manner on the target first specimen image 151T. Similarly, when the display switching button 128 is selected, the display control unit 91 displays the cluster image 116 in a superimposed manner on the target first specimen image 151T, and when the display switching button 128 is not selected, the display control unit 91 does not display the cluster image 116 in a superimposed manner on the target first specimen image 151T. Furthermore, when the display switching button 129 is selected, the display control unit 91 displays the cluster image 117 in a superimposed manner on the target first specimen image 151T, and when the display switching button 129 is not selected, the display control unit 91 does not display the cluster image 117 in a superimposed manner on the target first specimen image 151T.

[0106] Therefore, if Fig. 20 As shown in FIG. 1 , when all display switching buttons 127 to 129 are selected, the display control unit 91 displays a superimposed image 119 in which all cluster images 115 to 117 are superimposed on the target first specimen image 151T. Fig.21 As shown, when the display switching button 127 is selected and the display switching buttons 128 and 129 are not selected, the display control unit 91 displays the superimposed image 119 in which only the cluster image 115 is superimposed on the target first specimen image 151T. In this way, the display control unit 91 displays at least one of the plurality of cluster images 115 to 117 superimposed on the target first specimen image 151T. In addition, the designation accepting unit 92 accepts the designation of the cluster images 115 to 117 to be superimposed and displayed on the target first specimen image 151T from the drug developer DS. In addition, the clustering result display screen 125 is initially displayed in a state in which all the display switching buttons 127 to 129 are selected. In addition, the clustering result display screen 125 disappears when the confirmation button 130 at the lower portion is selected.

[0107] Next, as an example, refer to Fig. 22 and Fig.23 First, in the image processing device 10, when the working program 40 is started, as shown in the flowchart, the function of the above structure is explained. Figure 3 As shown in FIG. 1 , the CPU 32 of the image processing device 10 functions as an RW control unit 50, a feature extraction unit 51, a determination unit 52, and a clustering processing unit 53. Fig.14 As shown, the clustering processing unit 53 functions as a dimension reduction unit 90 , a display control unit 91 , and a designation acceptance unit 92 .

[0108] The first specimen image 151 of the liver specimen LVS of the subject S is captured by the imaging device 19. The first specimen image 151 is output from the imaging device 19 to the image processing device 10. In the image processing device 10, the first specimen image 151 from the imaging device 19 is stored in the storage device 30 via the RW control unit 50.

[0109] exist Fig. 22 In the case where the drug developer DS issues a display instruction for the first specimen image 151 via the input device 13, the first specimen image 151 designated in the display instruction is read out from the storage device 30 by the RW control unit 50 (step ST100). The first specimen image 151 is output from the RW control unit 50 to the display control unit 91 of the clustering processing unit 53. Fig.15 As shown, under the control of the display control section 91 , the first specimen image 151 is displayed on the display 12 via the object designation screen 95 (step ST105 ).

[0110] When the drug developer DS places the selection frame 97 on the desired first specimen image 151 in the target designation screen 95 and selects the analysis button 98 ("Yes" in step ST110), the RW control unit 50 reads out the first specimen image 151 currently placed on the selection frame 97 from the storage device 30 as the target first specimen image 151T (step ST115). The target first specimen image 151T is output from the RW control unit 50 to the feature amount extraction unit 51 and the display control unit 91.

[0111] The feature quantity extractor 41 is read out from the storage device 30 by the RW control unit 50, and the read feature quantity extractor 41 is output to the feature quantity extractor 51. In addition, the second feature quantity distribution information 42 is obtained by reading out from the storage device 30 by the RW control unit 50, and the read second feature quantity distribution information 42 is output to the determination unit 52.

[0112] like Figure 4 As shown in FIG. 1 , in the feature quantity extraction unit 51, the target first specimen image 151T is subdivided into a plurality of first small block images 651 (step ST120). Figure 5 As shown, in the feature amount extraction section 51, the first feature amount 601 is extracted from the first small block image 651 using the feature amount extractor 41 (step ST125). The first feature amount 601 is output from the feature amount extraction section 51 to the determination section 52.

[0113] like Fig.11 As shown, in the determination unit 52, the distance D between the display position of the second feature quantity 602 and the position of the first feature quantity 601 is calculated. Then, as Fig.12 and Fig.13As shown, in the determination unit 52, the distance D is compared with a preset threshold value to determine whether morphological abnormality occurs in the liver sample LVS captured on the first small block image 651 (step ST130). The determination result 61 of whether morphological abnormality occurs in the liver sample LVS captured on the first small block image 651 is output from the determination unit 52 to the dimension reduction unit 90 of the clustering processing unit 53.

[0114] The processing of steps ST125 and ST130 is performed on all first small blocks of images 651. After the processing of steps ST125 and ST130 is performed on all first small blocks of images 651 ("Yes" in step ST135), the process moves to Fig.23 The processing of step ST140 is shown.

[0115] like Fig.16 As shown, in the dimensionality reduction unit 90, the dimension of the first feature quantity 601 of the first small block image 651 determined to have morphological abnormality is reduced to two dimensions (step ST140). Then, under the control of the display control unit 91, Fig.17 The cluster specifying screen 105 shown is displayed on the display 12 (step ST145 ).

[0116] The cluster designation screen 105 includes a graph 106 in which the two-dimensional first feature quantity 601R after dimensionality reduction is plotted in a two-dimensional feature quantity space 107. On the graph 106, as shown in FIG. Fig.18 As shown, the drug developer DS specifies which cluster the first small block image 651 belongs to, and selects the designation completion button 109. When the designation completion button 109 is selected, the designation acceptance unit 92 accepts the designation of which cluster the first small block image 651 belongs to by the drug developer DS (step ST150). Based on the designation, the designation acceptance unit 92 generates clustering information 112. Thus, the manual clustering process of clustering the first small block image 651 into a plurality of clusters is completed (step ST155). The clustering information 112 is output from the designation acceptance unit 92 to the display control unit 91.

[0117] like Fig.19 As shown in FIG. 1 , the display control unit 91 processes the target first specimen image 151T according to the cluster information 112 to generate cluster images 115 to 117 that can identify a plurality of clusters 1 to 3 (step ST160). Fig. 20 and Fig.21 As shown, under the control of the display control unit 91 , the clustering result display screen 125 including the superimposed image 119 obtained by superimposing at least one of the cluster images 115 to 117 on the target first specimen image 151T is displayed on the display 12 (step ST165 ).

[0118] The drug developer DS views the superimposed image 119 on the clustering result display screen 125 to evaluate the efficacy and toxicity of the candidate substance 11. At this time, the drug developer DS operates the display switching buttons 127 to 129 as needed to switch the cluster images 115 to 117 displayed superimposed on the target first specimen image 151T.

[0119] As described above, the CPU 32 of the image processing device 10 includes the RW control unit 50, the feature extraction unit 51, the determination unit 52, and the clustering processing unit 53. The RW control unit 50 acquires the target first specimen image 151T by reading it from the storage device 30. The target first specimen image 151T is an image of the liver specimen LVS of the subject S.

[0120] The feature extraction unit 51 extracts the first feature 601 from the first small block image 651 obtained by segmenting the target first specimen image 151T using the feature extractor 41. The determination unit 52 determines whether morphological abnormality occurs in the liver specimen LVS captured on the first small block image 651 based on the first feature 601. The clustering processing unit 53 receives the designation of which cluster the first small block image 651 determined to have morphological abnormality in the liver specimen LVS belongs to from the drug developer DS. Then, based on the designation, the manual clustering processing is performed to cluster the first small block image 651 into a plurality of clusters. Therefore, the first small block image 651 of the target first specimen image 151T in which the distribution of the first feature 601 is not discrete can be clustered.

[0121] The display control unit 91 performs control to display the result of the manual clustering process, that is, the superimposed image 119 including the cluster images 115 to 117, on the display 12. Therefore, the drug developer DS can easily know the result of the manual clustering process.

[0122] The result of the manual clustering process is displayed by a plurality of cluster images 115 to 117 generated by processing the target first specimen image 151T. The plurality of cluster images 115 to 117 are images that can identify the plurality of clusters 1 to 3 by using a display format 118 that is preset for each of the plurality of clusters 1 to 3. Therefore, the drug developer DS can easily grasp the result of the manual clustering process.

[0123] The display control unit 91 displays at least one of the plurality of cluster images 115 to 117 superimposed on the target first specimen image 151T. Therefore, the drug developer DS can easily understand which part of the liver specimen LVS belongs to which cluster, and further, can easily understand which part of the liver specimen LVS has what kind of morphological abnormality.

[0124] Regarding morphological abnormalities, the type can be inferred with a certain degree of certainty from the part where they occur. Therefore, the drug developer DS can identify whether the morphological abnormality is caused by the toxicity of the candidate substance 11, such as hyperplasia, or whether it is caused by the toxicity of the candidate substance 11, such as glycogen reduction, which occurs naturally and is not caused by the toxicity of the candidate substance 11, by using the superimposed image 119. Therefore, the drug developer DS can examine the mechanism of action of the efficacy and toxicity of the candidate substance 11 based on the superimposed image 119.

[0125] The designation accepting unit 92 accepts the designation of the cluster images 115 to 117 to be displayed superimposed on the target first specimen image 151T from the drug developer DS. Therefore, the drug developer DS can display only one cluster image of interest superimposed on the target first specimen image 151T, or can display all the cluster images 115 to 117 superimposed on the target first specimen image 151T. The drug developer DS can smoothly evaluate the efficacy and toxicity of the candidate substance 11.

[0126] The dimension reduction unit 90 reduces the dimension of the first feature quantity 601 to two dimensions. The display control unit 91 displays a cluster designation screen 105 on the display 12. The cluster designation screen 105 includes a graph 106 obtained by plotting the first feature quantity 601R whose dimension is reduced to two dimensions in a two-dimensional feature quantity space 107. The designation acceptance unit 92 accepts the designation on the graph 106. Therefore, the drug developer DS can easily designate to which cluster the first small block image 651 belongs.

[0127] like Figure 7 and Figure 8 As shown, the feature quantity extractor 41 is a model obtained by learning using the second small block image 652 obtained by segmenting the second specimen image 152 as training data, the second specimen image 152 capturing the liver specimens LVS of a plurality of subjects S constituting a past control group 75 that was not fed with the candidate substance in a past evaluation test of the candidate substance of the drug. Therefore, the first feature quantity 601 that well expresses the characteristics of the shape and texture of the liver specimen LVS can be easily extracted. In addition, the second small block image 652 can be unconditionally used as training data without the need for complicated pre-processing in which a pathologist or the like identifies whether morphological abnormalities occur in the liver specimen LVS and selectively uses the second small block image 652 in which morphological abnormalities do not occur in the liver specimen LVS as training data. Therefore, a large amount of training data can be easily obtained.

[0128] The RW control unit 50 obtains the second feature quantity distribution information 42 by reading it out from the storage device 30. The second feature quantity distribution information 42 is information on the distribution 83 of the second feature quantity 602 extracted from the second small block image 652 obtained by segmenting the second specimen image 152 using the feature quantity extractor 41. The determination unit 52 calculates the distance D between the distribution 83 and the first feature quantity 601, and more specifically, calculates the distance D between the representative position of the distribution 83 and the position of the first feature quantity, and performs determination based on the distance D.

[0129] The distance D indicates the degree of deviation between the first specimen image 151T and the second specimen image 152. In addition, at least no morphological abnormality due to the toxicity of the candidate substance occurs in the liver specimen LVS captured in the second specimen image 152. Therefore, as long as the determination is made based on the distance D, an appropriate determination result 61 can be obtained.

[0130] (Variation Example)

[0131] The dimension of the first feature quantity 601R after dimensionality reduction is not limited to Fig.16 As an example, it is also possible to Fig.24 135, the dimension reduction unit 135 outputs the dimension-reduced data 136 obtained by reducing the dimension of the first feature quantity 601 to three dimensions. In this case, the display control unit 91, as an example, Fig.25 As shown, a control is performed to display a cluster designation screen 142 on the display 12, and the cluster designation screen 142 includes a graph 141 obtained by plotting the first feature quantity 601R after dimensionality reduction in the three-dimensional feature quantity space 140. In this case, a tool is prepared to surround a piece of the first feature quantity 601R considered to be a cluster with a cube, a sphere, an ellipsoid, or an arbitrary curved body. According to this modification, the drug developer DS can easily specify to which cluster the first small block image 651 belongs.

[0132] [1st_2nd Implementation Method]

[0133] As an example, Fig.26 As shown, the cluster processing unit 145 of this embodiment includes an attribution degree calculation unit 146, a cluster information generation unit 147, and a display control unit 91. The attribution degree calculation unit 146 performs soft clustering processing described below. The attribution degree calculation unit 146 receives the determination result 61 from the determination unit 52.

[0134] As an example, Fig. 27As shown, the attribution degree calculation unit 146 calculates the attribution degree of the first small block image 651 to each cluster by applying a known soft clustering method to the first feature quantity 601 included in the determination result 61. Here, the attribution degree calculation unit 146 calculates the attribution degree to the three clusters 1 to 3 and the probability of not belonging to any of the clusters 1 to 3 (in Fig. 27 , recorded as "not belonging"). The total of the degree of belonging to clusters 1 to 3 and the probability of not belonging to any of clusters 1 to 3 is 100%. In addition, as a soft clustering method, a Gaussian mixture model (GMM: Gaussian Mixture Model), a probabilistic latent semantic analysis method (PLSA: Probabilistic Latent Semantic Analysis), a non-negative matrix factorization (NMF: Non-negative Matrix Factorization), or a FCM (Fuz zy c-Means: Fuzzy C-means clustering) method can be used.

[0135] The belonging degree calculation unit 146 generates belonging degree information 160 in which the belonging degree and the probability of non-belonging are registered for each small block image ID85 of the first small block image 651 . The belonging degree calculation unit 146 outputs the belonging degree information 160 to the cluster information generation unit 147 .

[0136] The clustering information generating unit 147 identifies the cluster to which each of the plurality of first small image blocks 651 belongs based on the belonging degree information 160. More specifically, the clustering information generating unit 147 identifies the cluster having the maximum value among the belonging degrees registered in the belonging degree information 160 as the cluster to which the first small image block 651 belongs. When the probability of not belonging is the maximum value, the clustering information generating unit 147 identifies that the first small image block 651 does not belong to any cluster.

[0137] The clustering information generating unit 147 generates clustering information 161 indicating the result of the above-mentioned identification. The clustering information 161 is information that registers the cluster to which each small block image ID 85 of each first small block image 651 belongs, similarly to the clustering information 112 of the above-mentioned first embodiment. The clustering information generating unit 147 outputs the clustering information 161 to the display control unit 91. Thereafter, the display control unit 91 generates a plurality of cluster images by processing the target first specimen image 151T according to the clustering information 161, similarly to the above-mentioned first embodiment. Then, the display control unit 91 generates a superimposed image obtained by superimposing the target first specimen image 151T and the cluster image, and controls the display 12 to display a clustering result display screen including the superimposed image.

[0138] As an example, Fig.28As shown in the flowchart, in this embodiment, after the processing of steps ST125 and ST130 is performed on all the first small block images 651 ("Yes" in step ST135), the belonging degree calculation unit 146 of the clustering processing unit 145 calculates the belonging degree and the probability of non-belonging to multiple clusters of the first small block image 651 determined to have morphological abnormalities in the liver sample LVS (step ST200). In the belonging degree calculation unit 146, belonging degree information 160 is generated in which the calculation results of the belonging degree and the probability of non-belonging are summarized for each first small block image 651. The belonging degree information 160 is output from the belonging degree calculation unit 146 to the clustering information generation unit 147. In this way, the soft clustering process is completed.

[0139] The clustering information generating unit 147 generates clustering information 161 based on the belonging degree information 160. Thus, the first small block image 651 is clustered into a plurality of clusters (step ST205). The clustering information 161 is output from the clustering information generating unit 147 to the display control unit 91. Since the subsequent processing is the same as that of the above-mentioned 1-1 embodiment, the description thereof is omitted.

[0140] In this way, the clustering processing unit 145 of the present embodiment performs a soft clustering process for calculating the degree of belonging of the first small block image 651 determined to have morphological abnormality in the liver sample LVS to each of the plurality of clusters. Therefore, according to the present embodiment, the first small block image 651 of the target first specimen image 151T whose distribution of the first feature quantity 601 is not discrete can also be clustered. In addition, the drug developer DS does not need to specify to which cluster the first small block image 651 belongs, so the burden on the drug developer DS can be reduced.

[0141] As described in the first embodiment, the belonging degree calculation unit 146 may calculate the belonging degree after reducing the dimension of the first feature quantity 601 .

[0142] In addition, in the present embodiment, when only one cluster image is superimposed and displayed on the target first specimen image 151T, as an example, it may be set as follows: Fig.29 In other words, in one cluster image superimposed on the target first specimen image 151T, the color depth is changed according to the degree of belonging. Fig.29 In the example, the case where only the cluster image 115 in cluster 1 is overlapped and displayed on the first specimen image 151T of the object is illustrated. In addition, the case where the color depth is changed to three levels according to the degree of attribution is illustrated. The drug developer DS can easily grasp the degree of attribution according to the color depth. In this way, the efficacy and toxicity of the candidate substance 11 can be evaluated more smoothly.

[0143] [Second Embodiment]

[0144] In this embodiment, as an example, Fig.30 The display control unit 91 displays statistical information 171 based on the result of the clustering process, as shown in the clustering result display screen 170. The statistical information 171 is the number and ratio of the first small block images 651 belonging to each cluster 1 to 3. The ratio is calculated by dividing the number of the first small block images 651 belonging to each cluster 1 to 3 by the total number of the first small block images 651.

[0145] In this way, by displaying the statistical information 171, the drug developer DS can easily understand how many first small block images 651 belonging to each cluster 1 to 3 exist. In addition, instead of or in addition to the number and ratio of the first small block images 651 belonging to each cluster 1 to 3, the area of ​​the first small block images 651 belonging to each cluster 1 to 3 may be displayed as the statistical information.

[0146] [Third Embodiment]

[0147] As an example, Fig.31 As shown, in this embodiment, in addition to the second small image block 652L for learning, a third small image block 653L for learning in which a liver specimen LVS with morphological abnormality is imaged is also used as training data for the autoencoder 70 .

[0148] As an example, Fig.32 As shown, the third small block image 653L for learning is provided by a plurality of third small block images 653 obtained by subdividing the third specimen image 153. The third specimen image 153 is an image of a liver specimen LVS of a subject S of a past feeding group 175. The past feeding group 175 is composed of a plurality of subjects S to whom the candidate substance was fed in a past evaluation test. The number of subjects S constituting the past feeding group 175 is similar to the past control group 75, for example, several hundreds to several thousands. The third specimen image 153 is also obtained from one subject S in the same manner as the first specimen image 151, etc., so the number of third specimen images 153 obtained from the past feeding group 175 is obtained by multiplying the number obtained from one subject S by the number of subjects S.

[0149] The third small block image 653L for learning is obtained by selecting a third small block image 653 in which a liver sample LVS having morphological abnormality is photographed from a plurality of third small block images 653. The selection of the third small block image 653L for learning is performed manually, for example, by a pathologist or the like. Alternatively, the selection of the third small block image 653L for learning may also be performed by applying the method described in the first embodiment to extract a third feature value from the third small block image 653, and determining whether a morphological abnormality occurs in the liver sample LVS based on the third feature value.

[0150] In this way, in the third embodiment, not only the second small block image 652L for learning but also the third small block image 653L for learning in which a liver specimen LVS with morphological abnormality is photographed is used as training data. Therefore, the autoencoder 70 and the feature quantity extractor 41 can learn liver specimens LVS having more diverse shape and texture features. As a result, the first feature quantity 601 that better expresses the shape and texture features of the liver specimen LVS can be extracted.

[0151] In addition, the small block image of the liver specimen LVS with morphological abnormalities is not limited to the third small block image 653L for learning obtained from the subject S constituting the exemplified past feeding group 175. Morphological abnormalities may sometimes occur in the subject S constituting the past control group 75. Therefore, as long as it is a small block image of the liver specimen LVS with morphological abnormalities, the subject S can be either the past control group 75 or the past feeding group 175. That is, the second small block image 652L for learning in which the liver specimen LVS with morphological abnormalities is captured can also be used as training data. In addition, the small block image of the liver specimen LVS with morphological abnormalities can also be an image obtained from the subject S that is intentionally caused to have morphological abnormalities by applying various pressures. In addition, it can also be an image in which morphological abnormalities are artificially caused by processing the small block image of the normal liver specimen LVS.

[0152] [Fourth Embodiment]

[0153] As an example, Fig.33 As shown, in this embodiment, the encoding unit 181 of the convolutional neural network (hereinafter, described as CNN (Convolutional Neural Network)) 180 is used as a feature extractor 182 for extracting the first feature 601 from the first small block image 651. The feature extractor 182 is an example of a "machine learning model" involved in the technology disclosed in this disclosure.

[0154] The CNN 180 includes an output unit 183 in addition to the encoding unit 181. The encoding unit 181 inputs a small block image 65 such as the first small block image 651. The encoding unit 181 converts the small block image 65 into a feature amount 60. The encoding unit 181 transfers the feature amount 60 to the output unit 183. The output unit 183 outputs a prediction result 184 based on the feature amount 60. The prediction result 184 is a result of predicting a type of morphological abnormality occurring in the liver specimen LVS photographed on the small block image 65 from a plurality of types such as hyperplasia, infiltration, congestion, and inflammation.

[0155] In the learning stage of CNN180, in addition to the second small image 652L for learning, the small image of the liver specimen LVS with morphological abnormalities shown in the third embodiment is mainly used as training data. Here, as described above, morphological abnormalities may sometimes occur in the subject S constituting the past control group 75. Therefore, there is a possibility that the liver specimen LVS with morphological abnormalities is also photographed in the second small image 652L for learning. Therefore, it is preferable to perform label smoothing on the correct data in the learning stage, not to strictly distinguish the types of morphological abnormalities as 1 and 0, but to add an error rate. In addition, in the liver specimen LVS photographed on the third small image 653L for learning, the area that is easily misdetected as morphological abnormalities can be shielded and used for learning. As an area that is easily misdetected as morphological abnormalities, dust attached when the slide specimen 18 is made can be cited.

[0156] Thus, in the fourth embodiment, the CNN 180 and the feature extractor 182 are models that are responsible for the task of identifying the type of morphological abnormality. Therefore, the first feature 601 that better expresses the characteristics of the shape and texture of the liver sample LVS can be extracted.

[0157] In addition, the machine learning model used for the feature extractor is not limited to the illustrated autoencoder 70 and CNN 180. The generator of the generative adversarial network (GAN) can be used for the feature extractor. The machine learning model without a convolutional layer such as the Vision Transformer (ViT) can also be used for the feature extractor.

[0158] The following comparative learning can also be performed: the distance between feature quantities from the same image in the feature quantity space is made closer, and the distance between feature quantities from different images in the feature quantity space is made farther. As comparative learning, for example, learning methods such as SimCLR (A Simple Framework for Contrastive Learning of Visual Representations) are known. In addition, learning methods such as BYOL (Bootstrap Your Own Latent) that do not use the dual (also called negative samples) of the above-mentioned different images can also be used. In addition, the following constraints can be added to the distribution of the extracted feature quantities: distribution on the unit sphere, distribution following the standard normal distribution, etc.

[0159] [Fifth Embodiment]

[0160] As an example, Fig.34 As shown in FIG. 1 , in the fifth embodiment, a slide specimen 190 in which tissue specimens of various organs are placed on one slide 16 is processed. Fig.34 , a case where a heart sample HS, a brain sample BS, and a bone marrow sample BMS are placed in addition to a liver sample LVS is illustrated. In this case, in the first sample image 151, in addition to the liver sample LVS, the heart sample HS, the brain sample BS, and the bone marrow sample BMS are also captured.

[0161] The CPU 32 of the image processing device 10 of the fifth embodiment functions as a recognition unit 191 in addition to the processing units 50 to 53 of the first embodiment. The recognition unit 191 recognizes the tissue specimens of each organ from the first specimen image 151, for example, using a template for recognizing the tissue specimens of each organ or a machine learning model. The recognition unit 191 outputs coordinate information of frames 192 to 195 surrounding the tissue specimens of each organ as recognition results. Frame 192 is a frame surrounding the heart specimen HS, and frame 193 is a frame surrounding the liver specimen LVS. In addition, frame 194 is a frame surrounding the brain specimen BS, and frame 195 is a frame surrounding the bone marrow specimen BMS.

[0162] Fig.35 The following is shown: the feature quantity extraction unit 51 divides the heart sample HS in the frame 192 into a plurality of first small block images 651, and extracts the first feature quantity 601 from the first small block image 651 by the feature quantity extractor 200 for the heart sample HS. The feature quantity extraction unit 51 also extracts the first feature quantity 601 for the liver sample LVS in the frame 193, the brain sample BS in the frame 194, and the bone marrow sample BMS in the frame 195. That is, the feature quantity extraction unit 51 extracts the first feature quantity 601 for each of the tissue samples of each organ identified by the identification unit 191. The subsequent processing of the determination unit 52 is the same as the processing shown in the above-mentioned first embodiment, etc., except that the determination is performed for each of the tissue samples of each organ captured on the target first sample image 151T, and therefore the illustration and description are omitted.

[0163] Thus, in the fifth embodiment, the first specimen image 151 is an image obtained by photographing the specimen slide 190 on which tissue specimens of various organs are mounted. The identification unit 191 identifies the tissue specimens of various organs from the first specimen image 151 thus obtained. The feature extraction unit 51 extracts the first feature 601 of each of the identified tissue specimens of various organs. The determination unit 52 determines each of the tissue specimens of various organs photographed on the target first specimen image 151T based on the first feature 601. Therefore, it is also possible to deal with the specimen slide 190 on which tissue specimens of various organs are mounted. As for the specimen slide, it is more common to use the specimen slide 190 on which tissue specimens of various organs are mounted as in the present embodiment, rather than the specimen slide 18 on which tissue specimen of one organ is mounted as in the first embodiment described above. Therefore, it is possible to perform processing more suitable for general operation.

[0164] The drug developer DS may manually define a frame representing the tissue sample of each organ in the first sample image 151 .

[0165] The feature amount 60 is not limited to the feature amount extracted by the feature amount extractor 41. It may be an average value, a maximum value, a minimum value, a mode value, or a discrete value of the pixel values ​​of the small block image 65.

[0166] The organ is not limited to the liver LV, etc., as exemplified, but may be the stomach, lung, small intestine, large intestine, etc. In addition, the subject S is not limited to rats, but may be mice, guinea pigs, gerbils, hamsters, ferrets, rabbits, dogs, cats, monkeys, etc.

[0167] The image processing device 10 may be Figure 1 The personal computer installed in the drug development facility shown may be a server computer installed in a data center separate from the drug development facility.

[0168] In the case where the image processing device 10 is constituted by a server computer, the first specimen image 151 is sent from a personal computer installed in each pharmaceutical development facility to the server computer via a network such as the Internet. The server computer transmits various screens such as the object designation screen 95 to the personal computer in the form of screen data for network transmission created by a markup language such as XML (Extensible Markup Language). The personal computer reproduces the screen displayed on the web browser based on the screen data and displays it on the display. In addition, other data markup languages ​​such as JSON (JavaScript (registered trademark) Object Notation) may be used instead of XML.

[0169] The image processing device 10 according to the technology disclosed herein can be widely used in all stages of drug development, from the initial stage, that is, setting of drug development targets, to the final stage, that is, clinical trials.

[0170] The hardware structure of the computer constituting the image processing device 10 involved in the technology of the present disclosure can be modified in many ways. For example, in order to improve processing capabilities and reliability, the image processing device 10 can also be composed of multiple computers separated as hardware. For example, the functions of the feature extraction unit 51 and the determination unit 52 and the functions of the clustering processing unit 53 or 145 are distributed to two computers. In this case, the image processing device 10 is composed of two computers.

[0171] In this way, the hardware structure of the computer of the image processing device 10 can be appropriately changed according to the required performance such as processing power, safety, and reliability. Furthermore, in addition to the hardware, the application programs such as the work program 40 can also be duplicated or distributed and stored in multiple storage devices in order to ensure safety and reliability.

[0172] In each of the above-mentioned embodiments, for example, as the hardware structure of the processing unit (Processing Unit) that performs various processes such as the RW control unit 50, the feature extraction unit 51, the determination unit 52, the clustering processing unit 53 and 145, the dimensionality reduction unit 90, the display control unit 91, the designated acceptance unit 92, the attribution calculation unit 146, the clustering information generation unit 147, and the identification unit 191, various processors (Processor) as shown below can be used. Among the various processors, as described above, in addition to the general-purpose processor CPU32 that executes the software (work program 40) and functions as various processing units, there are also processors such as FPGA (Field Programmable Gate Array) that can change the circuit structure after production, that is, programmable logic devices (PLD), ASIC (Application Specific Integrated Circuit) and other processors with circuit structures specially designed to perform specific processing, that is, dedicated circuits.

[0173] A processing unit may be composed of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, and / or a combination of a CPU and an FPGA). In addition, multiple processing units may be composed of one processor.

[0174] As an example of using one processor to form multiple processing units, the first method is as follows: as represented by computers such as clients and servers, a processor is formed by combining one or more CPUs and software, and the processor functions as multiple processing units. The second method is as follows: as represented by a system on chip (System On Chip: SoC), a processor that uses an IC (Integrated Circuit) chip to implement the overall function of the system including multiple processing units is used. In this way, various processing units are formed using one or more of the above-mentioned processors as a hardware structure.

[0175] Furthermore, as the hardware structure of these various processors, more specifically, a circuit (circuitry) formed by combining circuit elements such as semiconductor elements can be used.

[0176] Based on the above description, the technology described in the following supplementary items can be understood.

[0177] [Supplementary Item 1]

[0178] An image processing device, wherein:

[0179] With processor,

[0180] The processor

[0181] acquiring a first specimen image in which a tissue specimen of a subject is photographed,

[0182] Using a machine learning model, extracting a first feature quantity from a first small block image obtained by segmenting the first specimen image,

[0183] Based on the first feature value, determining whether morphological abnormality occurs in the tissue sample photographed on the first small piece of image,

[0184] Perform either manual clustering or soft clustering.

[0185] In the manual clustering process, a designation is received from a user as to which cluster the first small image block determined to have the morphological abnormality in the tissue sample belongs to, and the first small image block is clustered into a plurality of clusters based on the designation.

[0186] In the soft clustering process, the degree of belonging of the first small block of images determined to have the morphological abnormality in the tissue sample to each of a plurality of clusters is calculated.

[0187] [Supplementary Item 2]

[0188] The image processing device according to supplementary item 1, wherein:

[0189] The processor

[0190] A control is performed to display the result of the manual clustering process or the result of the soft clustering process.

[0191] [Supplementary item 3]

[0192] The image processing device according to supplementary item 2, wherein:

[0193] The result is displayed using a plurality of cluster images generated by processing the first specimen image,

[0194] The plurality of cluster images are images by which the plurality of clusters can be identified by a display form preset for each of the plurality of clusters.

[0195] [Supplementary Item 4]

[0196] The image processing device according to supplementary item 3, wherein:

[0197] The processor

[0198] At least one of the plurality of cluster images is displayed superimposed on the first specimen image.

[0199] [Supplementary Note 5]

[0200] The image processing device according to supplementary item 4, wherein:

[0201] The processor

[0202] A designation of the cluster image to be displayed superimposed on the first specimen image is accepted from the user.

[0203] [Supplementary Item 6]

[0204] An image processing device according to any one of supplementary items 2 to 5, wherein:

[0205] The processor

[0206] Displays statistics based on the results.

[0207] [Supplementary Note 7]

[0208] An image processing device according to any one of supplementary items 1 to 6, wherein:

[0209] The processor

[0210] In the manual clustering process, the dimension of the first feature value is reduced to two or three dimensions, a graph in which the first feature value with reduced dimension is plotted in a two-dimensional space or a three-dimensional space is displayed, and the designation on the graph is accepted.

[0211] [Supplementary Item 8]

[0212] An image processing device according to any one of supplementary items 1 to 7, wherein:

[0213] The machine learning model is a model obtained by learning using a second small block image obtained by subdividing a second specimen image as training data, wherein the second specimen image is captured with tissue specimens of multiple subjects constituting a control group that was not fed with the candidate substance in a past evaluation test of the candidate substance of a drug.

[0214] [Supplementary Item 9]

[0215] The image processing device according to supplementary note 8, wherein:

[0216] The training data also includes a small image of the tissue sample in which the morphological abnormality occurs.

[0217] [Supplementary Item 10]

[0218] The image processing device according to supplementary note 9, wherein:

[0219] The machine learning model is a model responsible for the task of identifying the type of morphological abnormality.

[0220] [Supplementary Item 11]

[0221] An image processing device according to any one of supplementary items 8 to 10, wherein:

[0222] The processor

[0223] obtaining information on the distribution of a second feature quantity, wherein the second feature quantity is a feature quantity extracted from a second small block image obtained by subdividing the second specimen image using the machine learning model,

[0224] calculating the distance between the distribution and the first feature quantity,

[0225] The determination is made based on the distance.

[0226] The technology disclosed in the present invention can also appropriately combine the above-mentioned various embodiments and / or various modifications. In addition, it is of course not limited to the above-mentioned embodiments, and a variety of structures can be adopted within the scope of the purpose. In addition, the technology disclosed in the present invention also relates to a storage medium that stores the program non-temporarily in addition to the program.

[0227] The above-mentioned records and illustrated contents are detailed descriptions of the parts involved in the technology of the present disclosure, but are only examples of the technology of the present disclosure. For example, the descriptions related to the above-mentioned structures, functions, actions, and effects are descriptions of an example of the structures, functions, actions, and effects of the parts involved in the technology of the present disclosure. Therefore, without departing from the scope of the purpose of the technology of the present disclosure, it is natural to delete unnecessary parts, add new elements, or replace the above-mentioned records and illustrated contents. In addition, in order to avoid complexity and make the parts involved in the technology of the present disclosure easy to understand, the descriptions of technical common sense that does not need to be specifically explained when the technology of the present disclosure can be implemented are omitted in the above-mentioned records and illustrated contents.

[0228] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. In addition, in this specification, when three or more items are expressed by "and / or", the same concept as "A and / or B" is also applied.

[0229] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, or technical standard was specifically and individually described as being incorporated by reference.

Claims

1. An image processing device, wherein: With processor, The processor acquiring a first specimen image in which a tissue specimen of a subject is photographed, Using a machine learning model, extracting a first feature quantity from a first small block image obtained by segmenting the first specimen image, Based on the first feature value, determining whether morphological abnormality occurs in the tissue sample photographed on the first small piece of image, and Perform either manual clustering or soft clustering. In the manual clustering process, a designation of which cluster the first small image block determined to have the morphological abnormality in the tissue sample belongs to is received from a user, and the first small image block is clustered into a plurality of clusters based on the designation. In the soft clustering process, the degree of belonging of the first small block image determined to have the morphological abnormality in the tissue sample to each of a plurality of clusters is calculated.

2. The image processing device according to claim 1, wherein: The processor A control is performed to display the result of the manual clustering process or the result of the soft clustering process.

3. The image processing device according to claim 2, wherein: The result is displayed using a plurality of cluster images generated by processing the first specimen image, The plurality of cluster images are images by which the plurality of clusters can be identified by a display form preset for each of the plurality of clusters.

4. The image processing device according to claim 3, wherein: The processor At least one of the plurality of cluster images is displayed superimposed on the first specimen image.

5. The image processing device according to claim 4, wherein: The processor A designation of the cluster image to be displayed superimposed on the first specimen image is accepted from the user.

6. The image processing device according to claim 2, wherein: The processor Displays statistics based on the results.

7. The image processing device according to claim 1, wherein: The processor In the manual clustering process, the dimension of the first feature value is reduced to two or three dimensions, a graph in which the first feature value with reduced dimension is plotted in a two-dimensional space or a three-dimensional space is displayed, and the designation on the graph is accepted.

8. The image processing device according to claim 1, wherein: The machine learning model is a model obtained by learning using a second small block image obtained by subdividing a second specimen image as training data, wherein the second specimen image is captured with tissue specimens of multiple subjects constituting a control group that was not fed with the candidate substance in a past evaluation test of the candidate substance of a drug.

9. The image processing apparatus according to claim 8, wherein: The training data also includes a small block of images of the tissue sample in which the morphological abnormality occurs.

10. The image processing device according to claim 9, wherein: The machine learning model is a model responsible for the task of identifying the type of morphological abnormality.

11. The image processing apparatus according to claim 8, wherein: The processor obtaining information on the distribution of a second feature quantity, wherein the second feature quantity is a feature quantity extracted from a second small block image obtained by segmenting the second specimen image using the machine learning model, calculating the distance between the distribution and the first feature quantity, The determination is made based on the distance.

12. A method for operating an image processing device, wherein: include: Acquiring a first specimen image in which a tissue specimen of a subject is photographed; Using a machine learning model, extracting a first feature quantity from a first small block of images obtained by segmenting the first specimen image; Based on the first feature value, determining whether morphological abnormality occurs in the tissue sample photographed on the first small image block; as well as Perform either manual clustering or soft clustering. In the manual clustering process, a designation of which cluster the first small image block determined to have the morphological abnormality in the tissue sample belongs to is received from a user, and the first small image block is clustered into a plurality of clusters based on the designation. In the soft clustering process, the degree of belonging of the first small block image determined to have the morphological abnormality in the tissue sample to each of a plurality of clusters is calculated.

13. A working program of an image processing device, wherein: Causes a computer to execute a process comprising the following steps: Acquiring a first specimen image in which a tissue specimen of a subject is photographed; Using a machine learning model, extracting a first feature quantity from a first small block of images obtained by segmenting the first specimen image; Based on the first feature value, determining whether morphological abnormality occurs in the tissue sample photographed on the first small image block; as well as Perform either manual clustering or soft clustering. In the manual clustering process, a designation of which cluster the first small image block determined to have the morphological abnormality in the tissue sample belongs to is received from a user, and the first small image block is clustered into a plurality of clusters based on the designation. In the soft clustering process, the degree of belonging of the first small block image determined to have the morphological abnormality in the tissue sample to each of a plurality of clusters is calculated.

Images