Image processing device, operation method for image processing device, and operation program for image processing device

Abstract

This image processing device comprises a processor. The processor acquires a sample image capturing a tissue sample of a subject subjected to an evaluation test of a candidate substance for a drug, and applies image processing to the sample image according to the type of the original organ of the tissue sample and / or the type of a morphological abnormality estimated to have occurred in the tissue sample.

Description

Image processing device, operation method of image processing device, and operation program of image processing device

[0001] The technology of the present disclosure relates to an image processing device, an operating method for an image processing device, and an operating program for an image processing device.

[0002] In the pharmaceutical manufacturing process, drug candidate substances are administered to test subjects such as rats to evaluate the efficacy and toxicity of the candidate substances. In these evaluation tests, specimen images of organ tissue specimens (brain specimens, liver specimens, heart specimens, etc.) collected by autopsy are used. The specimen images are digitized into WSIs (Whole Slide Images). These images are then displayed on a computer screen for viewing by users such as pathologists who are responsible for evaluating the candidate substances.

[0003] Various techniques have been proposed to assist users in evaluating candidate substances by performing image processing on specimen images. For example, Japanese Patent No. 6,780,045 describes a technique in which the quality of specimen images is evaluated by a computer and low-quality specimen images are subjected to image processing such as contrast correction, color correction, and sharpening. Furthermore, Japanese Patent No. 5,996,494 describes a technique in which the resolution of a user's region of interest in a specimen image is gradually increased as the display time of the region of interest increases, and when the maximum resolution is reached, specific staining in the region of interest is highlighted.

[0004] There are many types of organs from which tissue specimens can be taken, including the brain, esophagus, stomach, large intestine, small intestine, liver, kidney, spleen, pancreas, heart, testes or ovaries, lymph nodes, bone marrow, etc. When evaluating candidate substances, users focus their observations on areas of the tissue specimen where morphological abnormalities (lesions not seen in normal tissue specimens) are suspected to have occurred, and there are many types of morphological abnormalities, including hyperplasia, infiltration, congestion, cysts, inflammation, tumors, canceration, proliferation, hemorrhage, glycogen depletion, inclusion bodies, granular cytoplasm, foamy cytoplasm, etc.

[0005] In order to accurately evaluate candidate substances using specimen images of tissue specimens from multiple types of organs and tissue specimens containing multiple types of morphological abnormalities, it is necessary to perform appropriate image processing on the specimen images. However, the image processing methods described in Japanese Patent Nos. 6,780,045 and 5,996,494 were not necessarily suitable in some cases. This could result in a low accuracy in the evaluation of candidate substances.

[0006] One embodiment of the technique of the present disclosure provides an image processing device, an operating method for the image processing device, and an operating program for the image processing device that can suppress a decrease in the accuracy of evaluation of candidate substances.

[0007] The image processing device of the present disclosure includes a processor that acquires a specimen image of a tissue specimen from a subject subjected to an evaluation test of a drug candidate substance, and performs image processing on the specimen image according to the type of organ from which the tissue specimen originated and / or the type of morphological abnormality that is presumed to have occurred in the tissue specimen.

[0008] The image processing is preferably image processing that simulates a change in image quality due to a change in the settings of the optical system in the photographing device for the specimen image.

[0009] The image processing preferably includes at least one of brightness adjustment processing, contrast adjustment processing, and enlargement processing.

[0010] It is preferable that the processor performs image processing only on the region where a morphological abnormality is suspected to occur.

[0011] Preferably, the processor determines the type of organ from which the tissue specimen originates and / or the type of morphological abnormality.

[0012] The processor preferably receives instructions from a user to perform image processing.

[0013] Preferably, the processor extracts a region where a morphological abnormality is suspected to occur from the specimen image.

[0014] Preferably, the processor performs the extraction using a machine learning model.

[0015] Preferably, the processor performs extraction by comparing the features obtained by inputting the specimen image into the machine learning model with reference features obtained by inputting a reference specimen image of a tissue specimen considered to be normal into the machine learning model.

[0016] Preferably, the processor classifies regions where morphological abnormalities are suspected to occur into a plurality of groups based on the similarity of the morphological abnormalities, and performs the same image processing on each group.

[0017] The method of operating the image processing device of the present disclosure includes obtaining a specimen image of a tissue specimen of a subject subjected to an evaluation test of a candidate drug substance, and applying image processing to the specimen image according to the type of organ from which the tissue specimen originated and / or the type of morphological abnormality presumed to have occurred in the tissue specimen.

[0018] The operating program of the image processing device of the present disclosure causes a computer to perform processing including acquiring a specimen image of a tissue specimen of a subject subjected to an evaluation test of a drug candidate substance, and performing image processing on the specimen image according to the type of organ from which the tissue specimen originated and / or the type of morphological abnormality presumed to have occurred in the tissue specimen.

[0019] According to the technology of the present disclosure, it is possible to provide an image processing device, an operating method for an image processing device, and an operating program for an image processing device that are capable of suppressing a decrease in the accuracy of evaluation of candidate substances.

[0020] 1 is a diagram showing a procedure for generating a specimen image, a specimen image, and an evaluation support device. FIG. 1 is a diagram showing an administration group and a control group. FIG. 2 is a block diagram showing a computer constituting the evaluation support device. FIG. 3 is a block diagram showing a processing unit of a CPU of the evaluation support device. FIG. 2 is a diagram showing processing by a discrimination unit. FIG. 3 is a diagram showing processing by a first judgment unit. FIG. 4 is a diagram showing processing by the first judgment unit. FIG. 4 is a diagram showing an extraction model group. FIG. 5 is a diagram showing patch images obtained by dividing a region image. FIG. 6 is a diagram showing how features are extracted from a patch image using an extraction model. FIG. 5 is a diagram showing the structure of an extraction model. FIG. 6 is a diagram showing processing in the learning phase of an autoencoder. FIG. 7 is a diagram showing the structure of a past control group and a reference patch image for learning. FIG. 7 is a diagram showing how reference features are extracted from a reference patch image using an extraction model. FIG. 8 is a diagram showing a graph in which reference features are plotted in feature space, and extracted reference information. FIG. 9 is a diagram showing the distance between the position of a feature and the representative position of the reference feature. FIG. 10 is a diagram showing processing by an extraction unit and judgment results. FIG. 11 is a diagram showing processing by an extraction unit and judgment results. FIG. 12 is a diagram showing processing by a second judgment unit. FIG. 13 is a diagram showing image processing information. FIG. 14 is a diagram showing image processing. FIG. 15 is a diagram showing an image list display screen. FIG. 16 is a diagram showing a first image display screen. FIG. 17 is a diagram showing a state in which a frame indicating an estimated region where morphological abnormality has occurred is displayed on the first image display screen. FIG. 1 is a diagram showing a second image display screen. FIG. 2 is a flowchart showing the processing procedure of the evaluation support device. FIG. 3 is a flowchart showing the processing procedure of the evaluation support device. FIG. 4 is a flowchart showing the processing procedure of the evaluation support device. FIG. 5 is a flowchart showing the processing procedure of the evaluation support device. FIG. 6 is a diagram showing a first image display screen of the second embodiment. FIG. 7 is a diagram showing a second image display screen of the second embodiment. FIG. 8 is a diagram showing clustering processing. FIG. 9 is a diagram showing the display format of each cluster, cluster images, and superimposed images. FIG. 10 is a diagram showing a first image display screen of the third embodiment. FIG. 11 is a diagram showing image processing of the third embodiment.

[0021] First Embodiment As shown in FIG. 1 as an example, an evaluation support device 10 is used to evaluate the efficacy and toxicity of a drug candidate substance 27 (see FIG. 2). The evaluation support device 10 is an example of an "image processing device" according to the technology of the present disclosure. The drug may be, for example, a biopharmaceutical such as an antibody drug that uses an antibody as an active ingredient, or a peptide drug or nucleic acid drug that uses a peptide or nucleic acid as an active ingredient.

[0022] The evaluation support device 10 is, for example, a desktop personal computer, and includes a display 11 for displaying various screens and an input device 12 such as a keyboard, a mouse, a touch panel, and / or a microphone for voice input. The evaluation support device 10 is installed, for example, in a pharmaceutical company developing drugs or an organization contracted by a pharmaceutical company to carry out drug development work, i.e., a contract research organization (CRO). The evaluation support device 10 is operated by a user U involved in drug development at the pharmaceutical company or contract research organization (hereinafter collectively referred to as a pharmaceutical facility). The user U is, for example, a pathologist responsible for evaluating a candidate substance 27.

[0023] A plurality of specimen images 15 are input to the evaluation support device 10. The specimen images 15 are images used to evaluate the efficacy and toxicity of a candidate substance 27. The specimen images 15 are generated, for example, by the following procedure. First, a subject S, such as a rat, prepared for evaluation of the candidate substance 27 is autopsied, and tissue specimens are obtained by slicing organs from the subject S. The tissue specimens include a brain specimen BS, a heart specimen HS, a lung specimen LS, a liver specimen LVS, a kidney specimen KDS, a spleen specimen SPS, an adrenal specimen AGS, and a pituitary specimen PGS. Although not shown in FIG. 1 , other tissue specimens include specimens of various organs, such as the esophagus, stomach, large intestine, small intestine, pancreas, gallbladder, aorta, vena cava, lymph nodes, trachea, bronchi, diaphragm, pineal gland, testes or ovaries, and spinal cord.

[0024] After the tissue specimens are collected, they are attached to glass slides 16 in accordance with standard operating procedures (SOPs) established in advance for each pharmaceutical facility. The SOPs specify detailed layout specifications for the tissue specimens, such as attaching a heart specimen HS and a lung specimen LS side by side on the same glass slide 16. In this way, multiple tissue specimens are attached to one glass slide 16.

[0025] The tissue specimen is then stained, in this case with hematoxylin-eosin dye. The stained tissue specimen is then covered with a cover glass 17 to complete a slide specimen 18. The slide specimen 18 is then placed in an imaging device 19, such as a digital optical microscope, and a specimen image 15 is captured by the imaging device 19. The resulting specimen image 15 captures the entire tissue specimen attached to the slide glass 16. In other words, a single specimen image 15 captures multiple tissue specimens. The specimen image 15 is called a WSI (Whole Slide Image). The specimen image 15 is assigned a specimen ID (Identification Data) for uniquely identifying the specimen S, a specimen image ID for uniquely identifying the specimen image 15, and the date and time of capture. The tissue specimen is also called a tissue section. The staining may be performed using hematoxylin dye alone, nuclear fast red dye, or the like.

[0026] As an example, as shown in FIG. 2 , subjects S are divided into a treatment group 25 and a control group 26. The treatment group 25 is composed of multiple subjects S administered with a candidate substance 27. The treatment group 25 is further divided into a high treatment group 25H, a medium treatment group 25M, and a low treatment group 25L according to the dose of the candidate substance 27. By dividing the treatment group 25 into the high treatment group 25H, the medium treatment group 25M, and the low treatment group 25L in this manner, the effect of the dose of the candidate substance 27 on the subjects S can be determined. The treatment group 25 is not limited to the exemplary three-group division of the treatment group 25H, the medium treatment group 25M, and the low treatment group 25L. The treatment group 25 may also be divided into two groups, the high treatment group 25H and the low treatment group 25L, or into four or more groups. The treatment group 25 may also be divided according to the length of the administration period of the candidate substance 27. Alternatively, the administration group 25 may be divided according to the frequency of administration of the candidate substance 27 .

[0027] Contrary to the administration group 25, the control group 26 is composed of a plurality of subjects S who were not administered the candidate substance 27. The number of subjects S constituting each of the high-administration group 25H, the medium-administration group 25M, and the low-administration group 25L is the same as the number of subjects S constituting the control group 26, e.g., about 5 to 10. The subjects S constituting each of the high-administration group 25H, the medium-administration group 25M, and the low-administration group 25L and the subjects S constituting the control group 26 have the same attributes and are reared in the same rearing environment. "Same attributes" refers to, for example, the same age in weeks and / or the same gender. "Same attributes" also includes the same age composition ratio and / or the same gender composition ratio (e.g., five males and five females). "Same rearing environment" refers to, for example, the same food, the same temperature and humidity of the rearing space, and / or the same size of the rearing space. The "same" in "same rearing environment" refers not only to the exact sameness, but also to the sameness in the sense of an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs, and that does not contradict the spirit of the technology of the present disclosure.

[0028] Since multiple specimen images 15 are obtained from one subject S, the number of specimen images 15 obtained from each group is calculated by multiplying the number of specimens S by the number of specimens S. For example, if the number of specimen images 15 obtained from one subject S is 100 and the number of subjects S constituting each group is 10, then 100 x 10 = 1000 specimen images 15 are obtained from each group.

[0029] 3, the computer constituting the evaluation support device 10 includes, in addition to the display 11 and input device 12, a storage 30, a memory 31, a CPU (Central Processing Unit) 32, and a communication unit 33. These components are interconnected via a bus line 34.

[0030] The storage 30 is a hard disk drive built into the computer constituting the evaluation support device 10 or connected via a cable or network. Alternatively, the storage 30 is a disk array consisting of multiple hard disk drives. The storage 30 stores control programs such as an operating system, various application programs, and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0031] The memory 31 is a work memory for the CPU 32 to execute processing. The CPU 32 loads programs stored in the storage 30 into the memory 31 and executes processing in accordance with the programs. In this way, the CPU 32 comprehensively controls each part of the computer. The CPU 32 is an example of a "processor" according to the technology of the present disclosure. Note that the memory 31 may be built into the CPU 32. The communication unit 33 controls the transmission of various information to and from external devices such as the imaging device 19.

[0032] 4 , an operating program 40 is stored in the storage 30 of the evaluation support device 10. The operating program 40 is an application program for causing a computer to function as the evaluation support device 10. In other words, the operating program 40 is an example of an “operating program for an image processing device” according to the technology of the present disclosure. The storage 30 also stores an identification model 41, a first determination model 42, an extraction model group 43, extraction reference information 44, a second determination model 45, image processing information 46, etc.

[0033] When the operating program 40 is started, the CPU 32 of the computer constituting the evaluation support device 10 cooperates with the memory 31 and the like to function as a read / write (hereinafter abbreviated as RW (Read Write)) control unit 50, an identification unit 51, a first determination unit 52, an extraction unit 53, a second determination unit 54, an image processing unit 55, and a display control unit 56. In addition to these processing units 50 to 56, the CPU 32 also includes an instruction receiving unit that receives various operation instructions from the input device 12, etc.

[0034] The RW control unit 50 controls the storage of various data in the storage 30 and the reading of various data from the storage 30. For example, the RW control unit 50 acquires a specimen image group 60 from the imaging device 19 and stores the specimen image group 60 in the storage 30. The specimen image group 60 is a collection of multiple specimen images 15 generated for evaluation of the candidate substance 27.

[0035] When a user U issues an instruction to display the image list display screen 120 (see FIG. 23 ) via the input device 12, the RW control unit 50 reads out the specimen image group 60 from the storage 30. The RW control unit 50 outputs the read-out specimen image group 60 to the identification unit 51.

[0036] The RW control unit 50 reads the discrimination model 41 from the storage 30 and outputs the read discrimination model 41 to the discrimination unit 51. The RW control unit 50 also reads the first judgment model 42 from the storage 30 and outputs the read first judgment model 42 to the first judgment unit 52. The RW control unit 50 also reads the extraction model group 43 and the extraction reference information 44 from the storage 30 and outputs the read extraction model group 43 and the extraction reference information 44 to the extraction unit 53.

[0037] The RW control unit 50 reads the second determination model 45 from the storage 30 and outputs the read second determination model 45 to the second determination unit 54. The RW control unit 50 also reads the image processing information 46 from the storage 30 and outputs the read image processing information 46 to the image processing unit 55.

[0038] The identification unit 51 uses the identification model 41 to identify multiple tissue specimens appearing in one specimen image 15. Then, it generates region images 70 (see FIG. 5 ) of the identified tissue specimens from the specimen image 15. The identification unit 51 outputs a region image group 61, which is a collection of the multiple generated region images 70, to the first determination unit 52 and the display control unit 56.

[0039] The first determination unit 52 uses the first determination model 42 to determine the type of organ in the tissue specimen shown in the region image 70. Then, the first determination unit 52 outputs a first determination result group 62, which is a collection of pairs of the region image 70 and the first determination results 75 of the organ type (see FIGS. 6 and 7 ), to the extraction unit 53, the image processing unit 55, and the display control unit 56.

[0040] The extraction unit 53 uses the extraction model group 43 and the extraction reference information 44 to extract a region in the tissue specimen captured in the region image 70 that is estimated to have a morphological abnormality (hereinafter referred to as an estimated morphological abnormality region). Then, the extraction result 63 of the estimated morphological abnormality region is output to the second determination unit 54 and the display control unit 56. Note that morphological abnormalities include lesions that are not seen in normal tissue specimens, such as hyperplasia, infiltration, congestion, cysts, inflammation, tumors, canceration, proliferation, hemorrhage, glycogen depletion, inclusion bodies, granular cytoplasm, and foamy cytoplasm.

[0041] The second determination unit 54 uses the second determination model 45 to determine the type of morphological abnormality that is presumed to have occurred in the tissue specimen shown in the region image 70. Then, the second determination unit 54 outputs a second determination result 64 of the type of morphological abnormality to the image processing unit 55 and the display control unit 56.

[0042] The image processing unit 55 performs image processing on the region image 70 in accordance with the image processing information 46, the first determination result 75, and the second determination result 64. The image processing unit 55 outputs the region image 70 after the image processing to the display control unit 56.

[0043] The display control unit 56 controls the display of various screens on the display 11. The various screens include an image list display screen 120 that displays a list of area images 70, and a first image display screen 125 (see FIGS. 24 and 25 ) that displays each area image 70.

[0044] As an example, as shown in FIG. 5 , the identification unit 51 inputs the specimen image 15 to the identification model 41. Then, the identification model 41 is made to identify the multiple tissue specimens appearing in the specimen image 15. The identification model 41 is a machine learning model such as a convolutional neural network. The identification model 41 identifies each of the multiple tissue specimens appearing in the specimen image 15 and outputs the position coordinates of a rectangular frame (called a bounding box) surrounding the tissue specimen as the identification result. The identification unit 51 cuts out a rectangular frame from the specimen image 15 in accordance with the identification result output by the identification model 41, thereby generating a region image 70 of each identified tissue specimen. Like the specimen image 15, the region image 70 is assigned a region image ID for uniquely identifying the region image 70.

[0045] 5 shows an example of a specimen image 15 including two kidney specimens KDS, a spleen specimen SPS, a sublingual gland specimen SLGS, and a submandibular gland specimen SMGS. As can be seen from the examples of the sublingual gland specimen SLGS and the submandibular gland specimen SMGS, the region image 70 may include a mixture of tissue specimens from multiple organs.

[0046] The user U may be able to modify the rectangular frame surrounding the tissue specimen, which is the identification result of the identification model 41. Alternatively, the tissue specimen appearing in the specimen image 15 may be identified by template matching instead of the identification model 41. Alternatively, the tissue specimen appearing in the specimen image 15 may be identified by inputting a rectangular frame surrounding the tissue specimen with the user U's hand, without using the identification model 41 or template matching.

[0047] 6 and 7 , the first determination unit 52 inputs a region image 70 to the first determination model 42. Then, the first determination model 42 determines the type of organ in the tissue specimen shown in the region image 70. Like the discrimination model 41, the first determination model 42 is a machine learning model such as a convolutional neural network. The first determination model 42 outputs a first determination result 75 of the type of organ in the tissue specimen shown in the region image 70.

[0048] 6 illustrates an example in which a region image 70 showing a kidney specimen KDS is input to the first determination model 42, and a first determination result 75 of "kidney" is output from the first determination model 42. Also, Fig. 7 illustrates an example in which a region image 70 showing a sublingual gland specimen SLGS and a submandibular gland specimen SMGS is input to the first determination model 42, and a first determination result 75 of "sublingual gland submandibular gland" is output from the first determination model 42.

[0049] The first determination result 75 may be configured to be modifiable by the user U, or the determination of the type of organ in the tissue specimen depicted in the region image 70 may be left to the user U. Furthermore, instead of the first determination model 42, the type of organ in the tissue specimen depicted in the region image 70 may be determined by template matching.

[0050] 8 , the extraction model group 43 is a collection of extraction models 80 prepared for each organ. That is, the extraction model group 43 includes a brain specimen extraction model 80A used for the area image 70 showing a brain specimen BS, a heart specimen extraction model 80B used for the area image 70 showing a heart specimen HS, a liver specimen extraction model 80C used for the area image 70 showing a liver specimen LVS, and a pituitary specimen extraction model 80D used for the area image 70 showing a pituitary specimen PGS. The extraction unit 53 uses different extraction models 80 depending on the first determination result 75. The extraction models 80 are an example of a "machine learning model" according to the technology of the present disclosure.

[0051] As an example, as shown in FIG. 9 , the extraction unit 53 recognizes a tissue specimen (a liver specimen LVS is illustrated in FIG. 9 ) depicted in the region image 70 using well-known image recognition technology and subdivides the recognized tissue specimen into multiple patch images 85. The patch images 85 have a predetermined size that can be handled by the extraction model 80 (in this case, the liver specimen extraction model 80C). The extraction unit 53 assigns a patch image ID to the patch image 85. The extraction unit 53 also associates information indicating which position in the region image 70 the patch image 85 is cut out from, i.e., position information of the patch image 85, with the patch image ID. Note that, although the patch image 85 does not have an area that overlaps with other patch images 85 in FIG. 9 , the patch image 85 may partially overlap with other patch images 85.

[0052] 10 , as an example, the extraction unit 53 uses an extraction model 80 (an extraction model 80C for a liver specimen in FIG. 10 ) to extract a feature 87 for each of a plurality of patch images 85 obtained by subdividing the regional image 70. Therefore, the number of feature amounts 87 is the same as the number of patch images 85.

[0053] As an example, as shown in FIG. 11 , an encoder unit 91 of an autoencoder 90 is diverted to an extraction model 80. The autoencoder 90 has a decoder unit 92 in addition to the encoder unit 91. A patch image 85 is input to the encoder unit 91. The encoder unit 91 converts the patch image 85 into a feature 87. The encoder unit 91 passes the feature 87 to the decoder unit 92. The decoder unit 92 generates a restored image 93 of the patch image 85 from the feature 87.

[0054] As is well known, the encoder unit 91 includes a convolution layer that performs convolution processing using a filter, a pooling layer that performs pooling processing such as maximum value pooling, and the like. The decoder unit 92 is similar. The encoder unit 91 extracts features 87 by repeating convolution processing by the convolution layer and pooling processing by the pooling layer multiple times on the input patch image 85. The extracted features 87 represent the shape and texture features of the tissue sample depicted in the patch image 85.

[0055] The feature 87 is a set of multiple numerical values. In other words, the feature 87 is multidimensional data. The number of dimensions of the feature 87 is, for example, 512, 1024, or 2048. The feature 87 and a reference feature 87R (see FIG. 14 ), which will be described later, have the same number of dimensions and can be compared in the same feature space 101 (see FIG. 15 , etc.).

[0056] 12 , in a learning phase before the encoder unit 91 is converted into the extraction model 80, the autoencoder 90 receives a training reference patch image 85RL as input and trains on it. The autoencoder 90 outputs a training reconstructed image 93L in response to the training reference patch image 85RL. A loss calculation is performed on the autoencoder 90 using a loss function based on the training reference patch image 85RL and the training reconstructed image 93L. Then, update settings are made for various coefficients of the autoencoder 90 (such as the filter coefficients of the convolution layer) according to the results of the loss calculation, and the autoencoder 90 is updated according to the update settings.

[0057] During the learning phase of the autoencoder 90, the above-described series of processes, including input of the learning reference patch image 85RL to the autoencoder 90, output of the learning reconstructed image 93L from the autoencoder 90, loss calculation, update setting, and updating of the autoencoder 90, are repeatedly performed while the learning reference patch image 85RL is replaced. The repetition of the above-described series of processes is terminated when the accuracy of restoration from the learning reference patch image 85RL to the learning reconstructed image 93L reaches a predetermined set level. The encoder unit 91 of the autoencoder 90 whose restoration accuracy has thus reached the set level is stored as the extraction model 80 in the storage 30 of the evaluation support device 10. Note that learning may be terminated when the above-described series of processes has been repeated a set number of times, regardless of the accuracy of restoration from the learning reference patch image 85RL to the learning reconstructed image 93L.

[0058] The learning of the autoencoder 90 may be performed by the evaluation support device 10 or by a device separate from the evaluation support device 10. In the latter case, the extraction model 80 is transmitted from the separate device to the evaluation support device 10, and the RW control unit 50 stores the extraction model 80 in the storage 30.

[0059] As an example, as shown in FIG. 13 , the learning reference patch image 85RL is supplied from multiple reference patch images 85R obtained by subdividing the reference region image 70R. The reference region image 70R is an image of a tissue specimen of a subject S in the past control group 26P. The past control group 26P is composed of multiple subjects S who were not administered the candidate substance 27 in past evaluation tests. Therefore, the number of subjects S constituting the past control group 26P is significantly greater than the number of subjects S constituting the administration group 25 and the control group 26, for example, on the order of several hundred to several thousand. Like the region image 70, multiple reference region images 70R are obtained from one subject S. Therefore, the number of reference region images 70R obtained from the past control group 26P is calculated by multiplying the number obtained from one subject S by the number of subjects S. The tissue specimens of the subjects S in the past control group 26P are an example of a "tissue specimen considered normal" according to the technology of the present disclosure. In addition to the area image 70 of the tissue specimen of the subject S in the past control group 26P, the area image 70 of the tissue specimen that a pathologist or other expert has determined to be normal in the past administration group consisting of multiple subjects S to whom the candidate substance was administered in a past evaluation test may also be used as the reference area image 70R.

[0060] 13 illustrates a reference region image 70R depicting a liver specimen LVS. The encoder unit 91 of the autoencoder 90 trained using a learning reference patch image 85RL based on the reference region image 70R depicting the liver specimen LVS is diverted to the liver specimen extraction model 80C. Similarly, for example, the encoder unit 91 of the autoencoder 90 trained using a learning reference patch image 85RL based on the reference region image 70R depicting a brain specimen BS is diverted to the brain specimen extraction model 80A.

[0061] Next, we will explain how the extracted reference information 44 is formed. First, as an example, as shown in Fig. 14, an extraction model 80 is used to extract multiple reference feature amounts 87R from multiple reference patch images 85R based on all of the multiple reference area images 70R.

[0062] As an example, a graph 100 shown in FIG. 15 plots multiple reference features 87R extracted in FIG. 14 in a feature space 101. The extracted reference information 44 includes coordinates 102 in the feature space 101 of representative positions of the reference features 87R, indicated by crosses (hereinafter referred to as representative position coordinates). The representative position is, for example, the center point or average point of a distribution 103 of the reference features 87R. The extracted reference information 44 also includes a judgment threshold 104. Note that, for convenience of explanation, the dimension of the feature space 101 is shown as two-dimensional, having axes D1 and D2, in FIG. 15; however, the actual dimension of the feature space 101 is, as described above, 512 or the like. Similarly, for convenience of explanation, the dimension of the feature space 101 is shown as two-dimensional in the following figures such as FIG. 16.

[0063] Similar to the learning of the autoencoder 90, the representative position coordinates 102 of the extracted reference information 44 may be derived by the evaluation support device 10, or may be derived by a device other than the evaluation support device 10. In the latter case, the representative position coordinates 102 are transmitted from the other device to the evaluation support device 10, and the RW control unit 50 stores the representative position coordinates 102 in the storage 30.

[0064] As an example, as shown in FIG. 16 , the extraction unit 53 calculates the distance D in the feature space 101 between the representative position of the reference feature 87R represented by the representative position coordinates 102 of the extracted reference information 44 and the position of the feature 87. The extraction unit 53 calculates the distance D between the multiple feature values ​​87 extracted for each of the multiple patch images 85 obtained by subdividing one region image 70. The distance D is the Mahalanobis distance. The distance D indicates the degree of deviation of the feature value 87 from the reference feature 87R, or more specifically, the degree of deviation of the tissue specimen depicted in the patch image 85 from a tissue specimen deemed normal. In other words, the larger the distance D, the more the tissue specimen depicted in the patch image 85 deviates from a tissue specimen deemed normal. Therefore, the larger the distance D, the more likely it is that a morphological abnormality has occurred in the tissue specimen depicted in the patch image 85.

[0065] Note that the distance D may be calculated as the average, median, or maximum value of the Euclidean distance between the position of the k-neighbor sample in the distribution 103 of the reference feature 87R and the position of the feature 87. Alternatively, instead of the distance D, a value may be calculated by subtracting from 1.0 the cosine similarity between a vector representing the representative position of the reference feature 87R and a vector representing the position of the feature 87. The cosine similarity takes a value between -1.0 and 1.0, and it can be said that the larger the value, the more similar the orientations of the vectors are.

[0066] 17 and 18 , the extraction unit 53 compares the calculated distance D with the judgment threshold 104. As shown in Fig. 17 , if the distance D is less than the judgment threshold 104, the extraction unit 53 determines that no morphological abnormality has occurred in the tissue specimen captured in the patch image 85. The extraction unit 53 outputs a judgment result 110 indicating that no morphological abnormality has occurred in the tissue specimen captured in the patch image 85.

[0067] On the other hand, as shown in FIG. 18 , if the distance D is equal to or greater than the judgment threshold 104, the extraction unit 53 determines that a morphological abnormality has occurred in the tissue specimen captured in the patch image 85. The extraction unit 53 outputs a judgment result 110 indicating that a morphological abnormality has occurred in the tissue specimen captured in the patch image 85. The region of the patch image 85 determined to have a morphological abnormality corresponds to the estimated morphological abnormality region, i.e., the "region estimated to have a morphological abnormality" according to the technology of the present disclosure. The extraction result 63 output from the extraction unit 53 is a collection of patch images 85 for which the judgment result 110 indicating that a morphological abnormality has occurred in the tissue specimen has been output. Note that the judgment threshold 104 may be common to the administration group 25 and the control group 26, or may be different between these groups. Alternatively, instead of the distance D, the determination of whether a morphological abnormality has occurred in the tissue specimen captured in the patch image 85 may be performed by comparing the cosine similarity with the judgment threshold 104.

[0068] As shown in FIGS. 13 and 14 , the reference feature 87R is a feature extracted from a reference patch image 85R obtained by subdividing a reference region image 70R depicting a tissue specimen of a subject S in the past control group 26P. The subject S in the past control group 26P is a subject S to whom the candidate substance 27 was not administered. Therefore, the tissue specimen depicted in the reference region image 70R does not exhibit morphological abnormalities at least due to the toxicity of the candidate substance 27. Therefore, the representative position of the reference feature 87R is regarded as the representative position of the feature of the region image 70 depicting a normal tissue specimen. Therefore, as described above, the distance D between the representative position of the reference feature 87R and the position of the feature 87 serves as an index indicating the degree to which the tissue specimen depicted in the patch image 85 deviates from a normal tissue specimen. 17, for a patch image 85 in which the distance D is less than the determination threshold 104, the extraction unit 53 determines that the tissue specimen depicted in the patch image 85 does not deviate from a normal tissue specimen and therefore determines that no morphological abnormality has occurred. On the other hand, for a patch image 85 in which the distance D is equal to or greater than the determination threshold 104, the extraction unit 53 determines that the tissue specimen depicted in the patch image 85 deviates from a normal tissue specimen and therefore determines that a morphological abnormality has occurred.

[0069] As an example, as shown in Figure 19, the second judgment unit 54 inputs a patch image 85 to the second judgment model 45. Then, the second judgment model 45 judges the type of morphological abnormality that is presumed to have occurred in the tissue specimen captured in the patch image 85. Like the discrimination model 41, the second judgment model 45 is also a machine learning model such as a convolutional neural network. The second judgment model 45 outputs a second judgment result 64 of the type of morphological abnormality that is presumed to have occurred in the tissue specimen captured in the patch image 85. In Figure 19, inclusion bodies are exemplified as a type of morphological abnormality.

[0070] As an example, as shown in FIG. 20 , the image processing information 46 is information summarizing image processing to be performed on the region image 70 or patch image 85 by the image processing unit 55. The image processing information 46 registers image processing appropriate for each organ type or each type of morphological abnormality. The image processing is image processing that simulates changes in image quality due to changes in the optical system settings of the imaging device 19. Specifically, the image processing includes brightness adjustment when the organ type is lymph node or bone marrow. The image processing also includes contrast adjustment and enlargement when the morphological abnormality type is inclusion body, granular cytoplasm, or foamy cytoplasm.

[0071] The brightness adjustment process is a process that simulates the increase in image brightness when the aperture diaphragm of the image capture device 19 is opened. Therefore, the brightness adjustment process may also be referred to as a brightness increase process. The contrast adjustment process is a process that simulates the increase in the light-dark contrast and / or color contrast of the image when the aperture diaphragm and field diaphragm of the image capture device 19 are narrowed. Therefore, the contrast adjustment process may also be referred to as a contrast increase process. The enlargement process is a process that simulates the increase in the image magnification when the magnification of the objective lens of the image capture device 19 is increased, and is a so-called electronic zoom process. The brightness adjusted in the brightness adjustment process and the contrast adjusted in the contrast adjustment process are set to predetermined levels. The magnification by which the image is enlarged in the enlargement process is also set to a predetermined magnification (e.g., 1.25x to 1.5x). The aperture diaphragm, field diaphragm, and objective lens are examples of an "optical system" according to the technology of the present disclosure.

[0072] If the organ type is lymph node or bone marrow, it is recommended to increase the brightness of the imaging device 19 for observation. For this reason, the brightness adjustment process is registered as described above. Inclusion bodies, granular cytoplasm, and foamy cytoplasm are very small and discrete because they occur within cells. For this reason, if the type of morphological abnormality is inclusion bodies, granular cytoplasm, or foamy cytoplasm, the contrast adjustment process and magnification process are registered as described above.

[0073] 21 , if the organ type of the first determination result 75 is an organ type for which image processing suitable for the first determination result 75 is registered in the image processing information 46, the image processing unit 55 applies the image processing registered in the image processing information 46 to the region image 70. Hereinafter, the region image 70 after image processing will be referred to as a processed region image 70A. Fig. 21 illustrates an example in which the tissue specimen shown in the region image 70 is a bone marrow specimen BMS, the organ type of the first determination result 75 is bone marrow, and brightness adjustment processing has been applied to the region image 70.

[0074] 22 , when the type of morphological abnormality in the second determination result 64 is a type of morphological abnormality for which image processing suitable for the type of morphological abnormality is registered in the image processing information 46, the image processing unit 55 applies the image processing registered in the image processing information 46 to the patch image 85. Hereinafter, the patch image 85 after the image processing will be referred to as a post-processing patch image 85A. Fig. 22 illustrates a case in which the type of morphological abnormality in the second determination result 64 is an inclusion, and a contrast adjustment process and an enlargement process (the processes are performed in the order of the enlargement process and the contrast adjustment process) have been applied to the patch image 85.

[0075] In response to a display instruction from the user U via the input device 12, the display control unit 56 controls the display 11 to display an image list display screen 120, as shown in FIG. 23 , for example. The image list display screen 120 has a display area 121. The display area 121 displays a list of all region images 70 generated by the identification unit 51. In the display area 121, the region images 70 are arranged in order of region image ID, from top to bottom and from left to right. The region image 70 displays the organ name based on the first determination result 75 along with the region image ID. An OK button 122 is provided below the display area 121. When the OK button 122 is selected, the display control unit 56 removes the display of the image list display screen 120. Note that the region images 70 displayed on the image list display screen 120 may be limited to only one subject S, or only the high-dose group 25H, for example.

[0076] Each area image 70 in the display area 121 can be selected. When the user U selects an area image 70 and issues an instruction to enlarge and display the area image 70, the display control unit 56 controls the display 11 to display a first image display screen 125, as shown in FIG. 24 , as an example. The first image display screen 125 is a screen that enlarges and displays the area image 70 selected by the user U on the image list display screen 120. When the image processing unit 55 applies image processing to the area image 70, the first image display screen 125 displays an enlarged version of the processed area image 70A, and displays a message indicating that image processing has been applied to the area image 70. Furthermore, when the image processing unit 55 applies image processing to the area image 70, the first image display screen 125 is provided with a back button 126 for returning the processed area image 70A to the area image 70 before image processing. When the image processing unit 55 does not apply image processing to the area image 70, the first image display screen 125 displays an enlarged version of the area image 70, and the message and back button 126 are not displayed. FIG. 24 illustrates an example in which the tissue specimen shown in the region image 70 is a bone marrow specimen BMS, and the region image 70 has undergone brightness adjustment as image processing.

[0077] An analysis button 127 and an OK button 128 are provided at the bottom of the first image display screen 125. When the analysis button 127 is selected to issue an instruction to analyze the region image 70, the extraction unit 53 extracts an estimated region where morphological abnormality has occurred from the region image 70 or the processed region image 70A displayed on the first image display screen 125. On the other hand, when the OK button 128 is selected, the display control unit 56 erases the display on the first image display screen 125.

[0078] When the extraction result 63 is input from the extraction unit 53, the display control unit 56 superimposes a frame 130 indicating a patch image 85 extracted as a region where morphological abnormality is estimated to occur on the tissue specimen of the region image 70 or the processed region image 70A, as shown in Fig. 25 as an example. Fig. 25 illustrates a case where the tissue specimen shown in the region image 70 is a liver specimen LVS.

[0079] The frame 130 is selectable. When the user U selects the frame 130 and issues an instruction to enlarge and display the patch image 85 indicated by the frame 130, the display control unit 56 controls the display 11 to display a second image display screen 135 shown in FIG. 26 . The second image display screen 135 is a screen that enlarges and displays the patch image 85 indicated by the frame 130 selected by the user U. When the image processing unit 55 applies image processing to the patch image 85, the second image display screen 135 displays an enlarged version of the processed patch image 85A, and displays a message indicating that image processing has been applied to the patch image 85. Furthermore, when the image processing unit 55 applies image processing to the patch image 85, the second image display screen 135 is provided with a revert button 136 for reverting the processed patch image 85A to the patch image 85 before image processing. When the image processing unit 55 does not apply image processing to the patch image 85, the second image display screen 135 displays an enlarged version of the patch image 85, and the message and the revert button 136 are not displayed. Figure 26 shows an example in which the type of morphological abnormality estimated to have occurred in the tissue specimen shown in patch image 85 is an inclusion body, and image processing in which contrast adjustment processing and enlargement processing have been performed on patch image 85.

[0080] Below the patch image 85 or 85A, a reduced-size area image 70 is displayed, with a frame 130 superimposed on the tissue specimen. In this reduced-size area image 70, the display control unit 56 distinguishes the frame 130 corresponding to the currently displayed patch image 85 or 85A from the other frames 130, for example, by changing the color of the frame 130. In addition, an OK button 137 is provided below the second image display screen 135. When the OK button 137 is selected, the display control unit 56 erases the display of the second image display screen 135.

[0081] When the enlarged regional image 70 displayed on the first image display screen 125 is obtained from a subject S belonging to the administration group 25, a regional image 70 of the same organ obtained from a subject S belonging to the control group 26 may be displayed next to the enlarged regional image 70 for comparison. Alternatively, on the image list display screen 120, a regional image 70 obtained from a subject S belonging to the administration group 25 and a regional image 70 obtained from a subject S belonging to the control group 26 may be displayed next to each other.

[0082] Next, the operation of the above configuration will be described with reference to the flowcharts shown in Figures 27 to 31. First, when the operating program 40 is started in the evaluation support device 10, the CPU 32 of the evaluation support device 10 functions as an RW control unit 50, a recognition unit 51, a first determination unit 52, an extraction unit 53, a second determination unit 54, an image processing unit 55, and a display control unit 56, as shown in Figure 4.

[0083] The photographing device 19 photographs specimen images 15 of a tissue specimen of the subject S. The specimen images 15 are transmitted from the photographing device 19 to the evaluation support device 10. In the evaluation support device 10, as shown in Fig. 27 , a specimen image group 60, which is a collection of specimen images 15 from the photographing device 19, is acquired by the RW control unit 50 (step ST100). The specimen image group 60 is stored in the storage 30 under the control of the RW control unit 50 (step ST110).

[0084] 28 , when the user U issues an instruction to display the image list display screen 120 via the input device 12 and the display instruction is accepted by the CPU 32 (YES in step ST200), the RW control unit 50 reads out the specimen image group 60 specified in the display instruction from the storage 30 (step ST210). The specimen image group 60 is output from the RW control unit 50 to the identification unit 51 and the display control unit 56.

[0085] 5 , in the identification unit 51, the specimen image 15 is input to the identification model 41, which identifies the multiple tissue specimens appearing in the specimen image 15, and the identification results are output from the identification model 41. Then, based on the identification results, a region image 70 of each tissue specimen is generated (step ST220). A region image group 61, which is a collection of the region images 70, is output from the identification unit 51 to the first determination unit 52.

[0086] In the first determination unit 52, as shown in Fig. 6 and Fig. 7 , the region image 70 is input to the first determination model 42. As a result, the type of organ of the tissue specimen shown in the region image 70 is determined (step ST230), and the first determination result 75 is output from the first determination model 42. A first determination result group 62, which is a collection of the first determination results 75, is output to the image processing unit 55 and the display control unit 56. As shown in Fig. 23 , under the control of the display control unit 56, an image list display screen 120 in which the region images 70 are arranged in a display area 121 is displayed on the display 11 (step ST240).

[0087] 29 , when the user U selects one of the area images 70 displayed on the image list display screen 120 and the CPU 32 receives an instruction to enlarge the area image 70 (YES in step ST300), the image processing unit 55 refers to the image processing information 46 and the first determination result 75. Then, it is searched whether or not image processing corresponding to the type of organ of the tissue specimen shown in the area image 70 for which the instruction to enlarge the area image 70 has been received is registered in the image processing information 46 (step ST310).

[0088] If image processing corresponding to the organ of the tissue specimen shown in the region image 70 for which the enlargement instruction has been received is registered in the image processing information 46 (YES in step ST310), the image processing unit 55 applies the image processing registered in the image processing information 46 to the region image 70 (step ST320), as shown in Fig. 21. On the other hand, if image processing corresponding to the organ of the tissue specimen shown in the region image 70 for which the enlargement instruction has been received is not registered in the image processing information 46 (NO in step ST310), no image processing is applied to the region image 70. As shown in Fig. 24, under the control of the display control unit 56, the first image display screen 125 in which the region image 70 or the processed region image 70A is enlarged is displayed on the display 11 (step ST330).

[0089] In FIG. 30, when the user U selects the analysis button 127 and an instruction to analyze the region image 70 is received by the CPU 32 (YES in step ST400), the extraction unit 53 extracts an estimated region where morphological abnormality has occurred (step ST410), as shown in FIGS. 9, 10, and 16 to 18.

[0090] Specifically, first, the region image 70 is subdivided into patch images 85 as shown in Fig. 9. Next, the patch images 85 are input to the extraction model 80 as shown in Fig. 10, which outputs a feature 87. Then, as shown in Fig. 16, a distance D in the feature space 101 between the feature 87 and the representative position of the reference feature 87R is calculated. Finally, the distance D is compared with a determination threshold 104 as shown in Figs. 17 and 18.

[0091] If the distance D is less than the determination threshold 104, a determination result 110 is output indicating that no morphological abnormality has occurred in the tissue specimen captured in the patch image 85. On the other hand, if the distance D is equal to or greater than the determination threshold 104, a determination result 110 is output indicating that a morphological abnormality has occurred in the tissue specimen captured in the patch image 85. A set of patch images 85 for which the determination results 110 indicating that a morphological abnormality has occurred have been output is output from the extraction unit 53 to the second determination unit 54 and the display control unit 56 as an extraction result 63 of an estimated morphological abnormality occurring region. As shown in FIG. 25 , under the control of the display control unit 56, a frame 130 indicating the patch image 85 extracted as an estimated morphological abnormality occurring region is superimposed on the tissue specimen in the region image 70 or the processed region image 70A (step ST420).

[0092] 31 , when the user U selects frame 130 and the CPU 32 receives an instruction to enlarge and display the patch image 85 indicated by frame 130 (YES in step ST500), the second determination unit 54 determines the type of morphological abnormality that is presumed to have occurred in the tissue specimen depicted in the patch image 85 for which the enlargement instruction was received (step ST510). Specifically, as shown in FIG. 19 , the patch image 85 for which the enlargement instruction was received is input to the second determination model 45, and a second determination result 64 of the type of morphological abnormality is output from the second determination model 45. The second determination result 64 is output from the second determination unit 54 to the image processing unit 55 and the display control unit 56.

[0093] The image processing unit 55 refers to the image processing information 46 and the second judgment result 64, and searches whether image processing corresponding to the type of morphological abnormality in the second judgment result 64 is registered in the image processing information 46 (step ST520). If image processing corresponding to the type of morphological abnormality in the second judgment result 64 is registered in the image processing information 46 (YES in step ST520), the image processing unit 55 applies the image processing registered in the image processing information 46 to the patch image 85 (step ST530), as shown in FIG. 22. On the other hand, if image processing corresponding to the type of morphological abnormality in the second judgment result 64 is not registered in the image processing information 46 (NO in step ST520), no image processing is applied to the patch image 85. As shown in FIG. 26, under the control of the display control unit 56, the second image display screen 135, in which the patch image 85 or the processed patch image 85A is enlarged, is displayed on the display 11 (step ST540).

[0094] The user U evaluates the candidate substance 27 by closely observing the region image 70 or the processed region image 70A on the first image display screen 125, or by closely observing the patch image 85 or the processed patch image 85A on the second image display screen 135.

[0095] As described above, the CPU 32 of the evaluation support device 10 includes the RW control unit 50 and the image processing unit 55. The RW control unit 50 acquires the specimen image 15, which is a photograph of a tissue specimen of a subject S subjected to an evaluation test for a drug candidate substance 27. The image processing unit 55 performs image processing on the region image 70 or the patch image 85 according to the type of organ from which the tissue specimen originated and / or the type of morphological abnormality estimated to have occurred in the tissue specimen. Therefore, the candidate substance 27 can be evaluated using the processed region image 70A that has been subjected to image processing appropriate for the type of organ, or the processed patch image 85A that has been subjected to image processing appropriate for the type of morphological abnormality. This makes it possible to suppress a decrease in the accuracy of the evaluation of the candidate substance 27.

[0096] 20, the image processing is image processing that simulates a change in image quality due to a change in the optical system settings in the photographing device 19 of the specimen image 15. Therefore, when photographing the specimen image 15 with the photographing device 19, it is not necessary to strictly change the optical system settings each time depending on the type of organ or the type of morphological abnormality. This can significantly reduce the effort required of the user U when photographing the specimen image 15.

[0097] The image processing includes at least one of brightness adjustment, contrast adjustment, and enlargement. Therefore, the candidate substance 27 can be evaluated using the processed region image 70A or the processed patch image 85A with appropriate brightness, contrast, or enlargement ratio. This makes it possible to further suppress a decrease in the accuracy of the evaluation of the candidate substance 27.

[0098] 22, the image processing unit 55 performs image processing only on the patch image 85 extracted as the estimated morphological abnormality occurrence region. Therefore, the load and time required for image processing can be reduced compared to when image processing is performed on the entire region image 70.

[0099] The first determination unit 52 determines the type of organ from which the tissue specimen originated, and the second determination unit 54 determines the type of morphological abnormality. This significantly reduces the effort required by the user U compared to when the user U visually determines the type of organ from which the tissue specimen originated and / or the type of morphological abnormality and inputs the determination result into the evaluation support device 10 via the input device 12.

[0100] The extraction unit 53 extracts the estimated morphological abnormality occurrence region from the region image 70. This can significantly reduce the effort of the user U compared to when the user U extracts the estimated morphological abnormality occurrence region visually.

[0101] The extraction unit 53 extracts an estimated region where morphological abnormality is occurring using an extraction model 80, which is a machine learning model. Recently, machine learning models have made remarkable progress, and relatively accurate models can be easily prepared. Therefore, an estimated region where morphological abnormality is occurring can be easily extracted with high accuracy.

[0102] The extraction unit 53 extracts the region where the morphological abnormality is estimated to occur by comparing the feature amount 87 obtained by inputting the patch image 85 to the extraction model 80 with the reference feature amount 87R obtained by inputting the reference patch image 85R of the reference region image 70R, which is an image of a tissue specimen considered to be normal, to the extraction model 80. This makes it possible to extract the region where the morphological abnormality is estimated to occur more easily and accurately.

[0103] Second Embodiment In the first embodiment, the image processing unit 55 automatically performs image processing, but this is not limiting. The image processing unit 55 may perform image processing when the CPU 32 receives an instruction from the user U to perform image processing.

[0104] As an example, as shown in FIG. 32 , if the organ type in the first determination result 75 is an organ type for which suitable image processing is registered in the image processing information 46, a button for instructing the implementation of image processing is provided on the first image display screen 150 of the second embodiment. FIG. 32 illustrates an example in which the tissue specimen shown in the area image 70 is a bone marrow specimen BMS. In this case, a brightness adjustment button 151 is provided on the first image display screen 150 as a button for instructing the implementation of image processing. When the brightness adjustment button 151 is selected by the user U, an instruction to implement brightness adjustment processing is received by the CPU 32. In response to this, the image processing unit 55 performs brightness adjustment processing on the area image 70.

[0105] As an example, as shown in FIG. 33 , if the type of morphological abnormality in the second determination result 64 is a type of morphological abnormality for which image processing suitable for the image processing information 46 is registered, the second image display screen 155 of the second embodiment is provided with a button for instructing the implementation of image processing. FIG. 33 illustrates an example in which the type of morphological abnormality estimated to have occurred in the tissue specimen shown in the patch image 85 is an inclusion body. In this case, the second image display screen 155 is provided with a contrast adjustment button 156 as a button for instructing the implementation of image processing. When the contrast adjustment button 156 is selected by the user U, the CPU 32 receives an instruction to implement a contrast adjustment process. In response to this, the image processing unit 55 performs a contrast adjustment process on the patch image 85. The image processing unit 55 automatically performs the enlargement process on the patch image 85.

[0106] In this way, in the second embodiment, an instruction to perform image processing is accepted from the user U. Therefore, whether or not to perform image processing can be determined at the discretion of the user U. If the user U actually observes the region image 70 or the patch image 85 and determines that image processing is unnecessary, unnecessary image processing can be avoided.

[0107] The first image display screen and the second image display screen may be provided with a plurality of buttons for instructing the implementation of various image processing, such as a brightness adjustment button, a contrast adjustment button, and an enlargement display button, to allow the user U to determine the type of organ or the type of morphological abnormality and to instruct the implementation of image processing. For example, if the tissue specimen shown in the region image 70 is a bone marrow specimen BMS, the user U observes the region image 70 and determines that the organ type is bone marrow. Then, the user selects the brightness adjustment button from the plurality of buttons to cause the image processing unit 55 to perform a brightness adjustment process.

[0108] 34 as an example, in the third embodiment, the extraction unit 53 performs a clustering process on feature quantities 87 of a patch image 85 extracted as a region estimated to have a morphological abnormality, to define the cluster to which each of the feature quantities 87 belongs. As the clustering process, k-means, HDBSCAN (Hierarchical Density-Based Spatial Clustering), Gaussian Mixture Model (GMM), Probabilistic Latent Semantic Analysis (PLSA), Non-negative Matrix Factorization (NMF), Fuzzy c-Means (FCM), or the like can be used. 34 shows an example in which feature quantities 87 are clustered into three clusters: cluster 1, cluster 2, and cluster 3. As shown in the example, some feature quantities 87 do not belong to any cluster. Clusters 1 to 3 are examples of "groups" according to the technology of the present disclosure.

[0109] The extraction unit 53 generates clustering information 160. The clustering information 160 is information in which the cluster to which each patch image 85 belongs is registered for each patch image ID of the patch image 85. A patch image 85 whose feature amount 87 does not belong to any cluster is not registered in the clustering information 160. The extraction unit 53 outputs the clustering information 160 to the display control unit 56.

[0110] 35 , the display control unit 56 processes the region image 70 in accordance with the clustering information 160 to generate cluster images 165, 166, and 167. Cluster image 165 is an image corresponding to cluster 1. Cluster image 166 is an image corresponding to cluster 2. Cluster image 167 is an image corresponding to cluster 3.

[0111] The display control unit 56 generates cluster images 165 to 167 according to a display format 168 preset for each cluster. The display format 168 displays, for example, cluster 1 in indigo blue, cluster 2 in yellow-green, and cluster 3 in gray. The display control unit 56 generates cluster image 165 by filling in the positions (determined from the position information) of patch images 85 corresponding to patch image IDs for which cluster 1 is registered in the clustering information 160 with indigo blue in the region image 70. Similarly, the display control unit 56 generates cluster image 166 by filling in the positions of patch images 85 corresponding to patch image IDs for which cluster 2 is registered in the clustering information 160 with yellow-green in the region image 70. Furthermore, the display control unit 56 generates cluster image 167 by filling in the positions of patch images 85 corresponding to patch image IDs for which cluster 3 is registered in the clustering information 160 with gray in the region image 70. By changing the display color in this way, the cluster images 165 to 167 become images that enable identification of clusters 1 to 3. Note that the display format 168 may be configured so that the user U can freely change the settings.

[0112] The display control unit 56 generates a superimposed image 169 by superimposing the region image 70 and at least one of the cluster images 165 to 167. Fig. 35 illustrates an example of the superimposed image 169 in which all of the cluster images 165 to 167 are superimposed on the region image 70.

[0113] When the analysis button 127 is selected to issue an instruction to analyze the region image 70 and the extraction unit 53 extracts an estimated region where morphological abnormality has occurred, the display control unit 56 controls the display 11 to display a first image display screen 175, as shown in FIG. 36 , as an example. The first image display screen 175 displays a superimposed image 169 and a legend 176. Display switch buttons 177, 178, and 179 are provided below the legend 176. The display switch button 177 is a button for selecting whether to superimpose the cluster image 165 on the region image 70. The display switch button 178 is a button for selecting whether to superimpose the cluster image 166 on the region image 70. The display switch button 179 is a button for selecting whether to superimpose the cluster image 167 on the region image 70. For this reason, for example, when all of the display switching buttons 177 to 179 are selected as shown in the figure, the display control unit 56 displays a superimposed image 169 in which all of the cluster images 165 to 167 are superimposed on the area image 70. In this way, the display control unit 56 displays at least one of the plurality of cluster images 165 to 167 superimposed on the area image 70. Note that the first image display screen 175 is initially displayed in a state in which all of the display switching buttons 177 to 179 are selected.

[0114] A button for instructing the implementation of image processing is provided for each cluster on the first image display screen 175. Specifically, the buttons for instructing the implementation of image processing are a brightness adjustment button 180, a contrast adjustment button 181, and an enlargement button 182. The user U determines the type of morphological abnormality that is presumed to have occurred in the tissue specimen shown in the patch image 85 belonging to each cluster, and selects a button from the buttons 180 to 182 that corresponds to the determined type of morphological abnormality.

[0115] When the brightness adjustment button 180 is selected by the user U, an instruction to perform brightness adjustment processing is accepted by the CPU 32. In response to this, the image processing unit 55 performs brightness adjustment processing on all patch images 85 belonging to the cluster in a batch. Similarly, when the contrast adjustment button 181 is selected by the user U, an instruction to perform contrast adjustment processing is accepted by the CPU 32. In response to this, the image processing unit 55 performs contrast adjustment processing on all patch images 85 belonging to the cluster in a batch. Furthermore, when the enlargement display button 182 is selected by the user U, an instruction to perform enlargement processing is accepted by the CPU 32. In response to this, the image processing unit 55 performs enlargement processing on all patch images 85 belonging to the cluster in a batch.

[0116] As described above, in the third embodiment, the extraction unit 53 classifies estimated morphological abnormality occurrence regions into multiple groups (clusters) based on the similarity of the morphological abnormalities. The image processing unit 55 performs the same image processing on each group. Therefore, compared to the second embodiment, in which image processing must be instructed for each patch image 85, the user U can significantly reduce the effort required to instruct image processing.

[0117] The first embodiment and the third embodiment may be combined. That is, as shown in FIG. 37 as an example, the type of morphological abnormality estimated to have occurred in the tissue specimen depicted in the patch image 85 belonging to each cluster is determined by the second determination unit 54, not by the user U. In this case, for example, the type of morphological abnormality is determined for all patch images 85 belonging to the cluster. Then, a majority vote is taken for the determined types of morphological abnormality, and the type of morphological abnormality with the largest number of determinations is determined as the final type of morphological abnormality. The image processing unit 55 performs image processing according to the finally determined type of morphological abnormality on all patch images 85 belonging to the cluster. FIG. 37 illustrates a case in which the type of morphological abnormality estimated to have occurred in the tissue specimen depicted in the patch image 85 belonging to cluster 1 is determined to be an inclusion body, and contrast adjustment processing and enlargement processing are performed as image processing. This further reduces the user U's workload.

[0118] Before the clustering process, a dimension reduction process may be performed on the feature 87. The dimension reduction process is a process of converting, for example, 512-dimensional feature 87 into a two-dimensional feature 87. As the dimension reduction process, principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), uniform manifold approximation and projection (UMAP), or the like can be used.

[0119] A graph 100 of the feature amount 87 shown in FIG. 34 may be displayed on the display 11, and the user U may manually divide the clusters.

[0120] Image processing that simulates changes in image quality due to changes in the settings of the optical system in the photographing device 19 for the specimen image 15 is not limited to the exemplified processes of adjusting brightness, adjusting contrast, and enlargement. It may also be processing that adjusts the resolution of the region image 70 or the patch image 85. Specifically, the processing of adjusting the resolution is processing that increases the resolution, and is realized, for example, by applying a well-known super-resolution technique to the region image 70 or the patch image 85.

[0121] When the organ type of the first determination result 75 is the organ type for which image processing suitable for the image processing information 46 is registered, the image processing registered in the image processing information 46 is applied to the region image 70, but this is not limitative. The image processing suitable for the organ type of the first determination result 75, which is registered in the image processing information 46, may be applied only to the patch image 85 extracted as the region estimated to have developed morphological abnormality, rather than to the region image 70.

[0122] The method for determining the type of the original organ of the tissue specimen captured in the region image 70 is not limited to the method using the first determination model 42. The type of organ may also be determined using the method described below. That is, a machine learning model is used to acquire the representative feature of each organ in advance. Furthermore, the representative feature of the region image 70, the type of which is to be determined, is derived using the machine learning model that derived the representative feature of each organ. Next, the distance in feature space between the representative feature of each organ and the representative feature of the region image 70 is calculated. Then, the organ with the representative feature with the shortest distance is determined to be the original organ of the tissue specimen captured in the region image 70.

[0123] Similarly, the method for determining the type of morphological abnormality estimated to have occurred in the tissue specimen captured in the region image 70 is not limited to the method using the second determination model 45. The type of morphological abnormality may be determined using the method described below. That is, a representative feature of each morphological abnormality is acquired in advance using a machine learning model. Furthermore, a representative feature of a patch image 85, the type of which is to be determined, is derived using the machine learning model that derived the representative feature of each morphological abnormality. Next, the distance in feature space between the representative feature of each morphological abnormality and the representative feature of the patch image 85 is calculated. Then, the morphological abnormality with the representative feature having the shortest distance is determined to be the morphological abnormality estimated to have occurred in the tissue specimen captured in the patch image 85.

[0124] In addition to the reference patch image 85R of a tissue specimen deemed normal, a patch image 85 of a tissue specimen with a morphological abnormality may also be used as the training reference patch image 85RL. The patch image 85 of a tissue specimen with a morphological abnormality is obtained, for example, from a past administration group consisting of multiple subjects S to whom the candidate substance 27 was administered in a previous evaluation test. This allows the autoencoder 90, and therefore the extraction model 80, to learn about tissue specimens with a wider variety of shape and texture characteristics. As a result, the extraction model 80 can extract feature quantities 87 that better represent the shape and texture characteristics of the tissue specimen.

[0125] Note that the patch image 85 of a tissue specimen in which a morphological abnormality has occurred is not limited to one acquired from a subject S constituting the exemplified past administration group. Morphological abnormalities may also occur in a subject S constituting the past control group 26P. Therefore, as long as the patch image 85 is of a tissue specimen in which a morphological abnormality has occurred, it does not matter whether the subject S is in the past control group 26P or the past administration group. Furthermore, the patch image 85 of a tissue specimen in which a morphological abnormality has occurred may be an image acquired from a subject S in which various stresses have been applied to intentionally induce the occurrence of a morphological abnormality. Furthermore, the patch image 85 of a tissue specimen in which a morphological abnormality has occurred may be an image artificially created by processing a patch image 85 of a normal tissue specimen.

[0126] Instead of the encoder unit 91 of the autoencoder 90, an encoder unit of a convolutional neural network that outputs a class discrimination result in response to an input of a patch image 85 may be used in the extraction model 80. The class discrimination result is the result of discriminating one type of morphological abnormality occurring in the tissue specimen shown in the patch image 85 from multiple types such as hyperplasia, infiltration, congestion, and inflammation.

[0127] Furthermore, the machine learning model that can be used as the extraction model 80 is not limited to the autoencoder 90 and the convolutional neural network shown as examples. A generator of a generative adversarial network (GAN) may be used as the extraction model 80. A machine learning model that does not have a convolutional layer, such as a Vision Transformer (ViT), may also be used as the extraction model 80.

[0128] Contrastive learning may be performed to reduce the distance between features derived from the same image in feature space and increase the distance between features derived from different images in feature space. Known examples of contrastive learning include learning methods such as SimCLR (A Simple Framework for Contrastive Learning of Visual Representations). Furthermore, learning methods such as BYOL (Bootstrap Your Own Latent) that do not use the above-described pairs of different images (also called negative samples) may also be used. Furthermore, constraints may be imposed on the distribution of extracted features, such as distribution on a unit sphere or distribution following a standard normal distribution.

[0129] The feature amount 87 is not limited to that extracted by the extraction model 80. It may be the average value, maximum value, minimum value, mode, or variance of the pixel values ​​of the patch image 85.

[0130] In the above-described embodiments, the case where one slide specimen 18 includes multiple tissue specimens has been described as an example, but the present disclosure is not limited to this. The technology of the present disclosure can also be applied to the case where one slide specimen 18 includes one tissue specimen.

[0131] The subject S is not limited to a rat, but may be a mouse, guinea pig, gizzard shad, hamster, ferret, rabbit, dog, cat, monkey, etc. The subject S may also be a human.

[0132] The evaluation support device 10 may be a personal computer installed in a pharmaceutical facility as shown in FIG. 1, or may be a server computer installed in a data center independent of the pharmaceutical facility.

[0133] When the evaluation support device 10 is configured as a server computer, the specimen images 15 are transmitted from personal computers installed in each pharmaceutical facility to the server computer via a network such as the Internet. The server computer distributes various screens, such as the image list display screen 120, to the personal computer in the form of screen data for web distribution created using a markup language such as XML (Extensible Markup Language). The personal computer reproduces the screen to be displayed on the web browser based on the screen data and displays it on a display. Note that other data description languages, such as JSON (Javascript (registered trademark) Object Notation), may be used instead of XML.

[0134] The evaluation support device 10 according to the technique of the present disclosure can be widely used across all stages of pharmaceutical manufacturing, from the earliest stage of setting drug discovery targets to the final stage of clinical trials.

[0135] The hardware configuration of the computer constituting the evaluation support device 10 according to the technology of the present disclosure can be modified in various ways. For example, the evaluation support device 10 can be configured with multiple computers separated as hardware in order to improve processing power and reliability. For example, the functions of the identification unit 51 and the first determination unit 52 and the functions of the extraction unit 53, the second determination unit 54, and the image processing unit 55 can be distributed and performed by two computers. In this case, the evaluation support device 10 is configured with two computers.

[0136] In this way, the hardware configuration of the computer of the evaluation support device 10 can be changed as appropriate depending on the required performance such as processing power, safety, reliability, etc. Furthermore, not only the hardware but also application programs such as the operating program 40 can be duplicated or stored in multiple storage devices in order to ensure safety and reliability.

[0137] In each of the above embodiments, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the RW control unit 50, the identification unit 51, the first determination unit 52, the extraction unit 53, the second determination unit 54, the image processing unit 55, and the display control unit 56. As described above, the various processors include the CPU 32, which is a general-purpose processor that executes software (operation program 40) to function as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0138] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA).Furthermore, multiple processing units may be configured with a single processor.

[0139] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0140] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0141] From the above description, the technology described in the following supplementary paragraphs can be understood.

[0142] [Supplementary Item 1] An image processing device including a processor, wherein the processor acquires a specimen image of a tissue specimen from a subject subjected to an evaluation test of a drug candidate substance, and performs image processing on the specimen image according to the type of organ from which the tissue specimen originated and / or the type of morphological abnormality estimated to have occurred in the tissue specimen. [Supplementary Item 2] The image processing device according to Supplementary Item 1, wherein the image processing simulates a change in image quality due to a change in settings of an optical system in an imaging device for the specimen image. [Supplementary Item 3] The image processing device according to Supplementary Item 2, wherein the image processing includes at least one of a brightness adjustment process, a contrast adjustment process, and an enlargement process. [Supplementary Item 4] The image processing device according to any one of Supplementary Items 1 to 3, wherein the processor performs the image processing only on a region estimated to have the morphological abnormality. [Supplementary Item 5] The image processing device according to any one of Supplementary Items 1 to 4, wherein the processor determines the type of organ from which the tissue specimen originated and / or the type of morphological abnormality. [Supplementary Item 6] The image processing device according to any one of Supplementary Items 1 to 5, wherein the processor accepts a user's instruction to perform the image processing. [Supplementary Item 7] The image processing device according to any one of Supplementary Items 1 to 6, wherein the processor extracts the region where the morphological abnormality is estimated to have occurred from the specimen image. [Supplementary Item 8] The image processing device according to Supplementary Item 7, wherein the processor performs the extraction using a machine learning model. [Supplementary Item 9] The image processing device according to Supplementary Item 8, wherein the processor performs the extraction by comparing feature amounts obtained by inputting the specimen image into the machine learning model with reference feature amounts obtained by inputting a reference specimen image of a tissue specimen considered to be normal into the machine learning model. [Supplementary Item 10] The image processing device according to any one of Supplementary Items 7 to 9, wherein the processor classifies the region where the morphological abnormality is estimated to have occurred into a plurality of groups based on the similarity of the morphological abnormality, and performs the same image processing on each group.

[0143] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs, but also to storage media that non-temporarily store programs, and computer program products that include programs.

[0144] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0145] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0146] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An image processing device comprising a processor that acquires a specimen image of a tissue specimen from a subject subjected to an evaluation test of a drug candidate substance, and applies image processing to the specimen image according to the type of organ from which the tissue specimen originated and / or the type of morphological abnormality presumed to have occurred in the tissue specimen.

2. The image processing device according to claim 1, wherein the image processing is image processing that simulates a change in image quality caused by a change in the settings of an optical system in a photographing device for the specimen image.

3. The image processing device according to claim 2, wherein the image processing includes at least one of a brightness adjustment process, a contrast adjustment process, and an enlargement process.

4. The image processing device according to claim 1, wherein the processor applies the image processing only to an area in which the morphological abnormality is suspected to occur.

5. The image processing device according to claim 1, wherein the processor determines the type of organ from which the tissue specimen originates and / or the type of morphological abnormality.

6. The image processing device according to claim 1, wherein the processor receives an instruction from a user to perform the image processing.

7. The image processing device according to claim 1, wherein the processor extracts an area in which the morphological abnormality is presumed to occur from the specimen image.

8. The image processing device according to claim 7, wherein the processor performs the extraction using a machine learning model.

9. The image processing device described in claim 8, wherein the processor performs the extraction by comparing features obtained by inputting the specimen image into the machine learning model with reference features obtained by inputting a reference specimen image of a tissue specimen considered to be normal into the machine learning model.

10. The image processing device according to claim 7, wherein the processor classifies the area in which the morphological abnormality is suspected to occur into a plurality of groups based on the similarity of the morphological abnormality, and applies the same image processing to each group.

11. A method for operating an image processing device, comprising: acquiring a specimen image of a tissue specimen from a subject subjected to an evaluation test of a drug candidate substance; and applying image processing to the specimen image according to the type of organ from which the tissue specimen originated and / or the type of morphological abnormality presumed to have occurred in the tissue specimen.

12. An operating program for an image processing device that causes a computer to execute processes including: obtaining a specimen image of a tissue specimen from a subject subjected to an evaluation test of a drug candidate substance; and applying image processing to the specimen image according to the type of organ from which the tissue specimen originated and / or the type of morphological abnormality presumed to have occurred in the tissue specimen.

Images