Image analysis device, operation method of image analysis device, and operation program of image analysis device

Abstract

Provided is an image analysis device comprising a processor, wherein the processor: acquires a target sample image in which a tissue sample of a target subject subjected to an evaluation test of a target candidate substance is captured; inputs the target sample image into a feature quantity extractor common to a plurality of types of organs and / or a plurality of types of tissues; causes the feature quantity extractor to output the target feature quantity of the target sample image; and calculates a degree of divergence of the target feature quantity with respect to the distribution of a reference feature quantity obtained by inputting, to the feature quantity extractor, a plurality of reference sample images in which tissue samples of the organ and / or tissue of reference subjects belonging to a control group are captured, such organ and / or tissue being the organ / or tissue corresponding to the organ and / or tissue of the tissue sample captured in the target sample image.

Description

Image analysis device, operation method for image analysis device, and operation program for image analysis device

[0001] The technology of the present disclosure relates to an image analysis device, an operating method for an image analysis device, and an operating program for an image analysis device.

[0002] In the drug discovery process, drug candidate substances are administered to test subjects such as rats to evaluate the toxicity of the candidate substances. These evaluation tests use images of tissue specimens (brain, liver, heart, etc.) collected from the test subjects after autopsy. Traditionally, pathologists performed evaluations by observing the specimen images. However, recent advances in image analysis technology have led to the development of automated evaluation techniques that can perform evaluations without the need for pathologists.

[0003] For example, Japanese Patent No. 7220017 describes a technique for evaluating tissue samples free of morphological abnormalities such as tumors or inflammation (referred to in Japanese Patent No. 7220017 as "tissue samples substantially free of abnormalities"). Specifically, in Japanese Patent No. 7220017, specimen images of a treatment group, which are subjects administered with a candidate substance, are first input into the machine learning model. Next, the difference between the extracted feature values ​​and those extracted from specimen images of tissue samples free of morphological abnormalities (referred to in Japanese Patent No. 7220017 as "deviation from normal score") is calculated. Based on the calculated difference, it is determined whether the tissue samples in the treatment group have morphological abnormalities.

[0004] As mentioned above, there are multiple types of organs that are used as tissue specimens. Furthermore, organs are composed of multiple types of tissues (vascular tissue, muscle tissue, glandular tissue, adipose tissue, etc.). Furthermore, when evaluating toxicity in the field of drug discovery, the morphological abnormalities to be detected differ for each organ and are of multiple types. Specifically, the total number of morphological abnormalities for all organs in the body exceeds 2,000. Thus, there is a need for a general-purpose automatic evaluation technology that can detect a wide range of morphological abnormalities from specimen images of tissue specimens spanning multiple types of organs and / or multiple types of tissues.

[0005] However, the technology described in Patent No. 7220017 could not meet the above demands due to various problems, such as differences in characteristics between organs, such as an image of one organ in a normal state resembling a morphological abnormality in another organ, an imbalance in the amount of training data, such as a large amount of training data being available for some organs due to their large size, but insufficient training data being available for other organs due to their small size, and an inability to detect morphological abnormalities in multiple types of organs.

[0006] One embodiment of the technology disclosed herein provides an image analysis device, an operating method for an image analysis device, and an operating program for an image analysis device that are capable of performing general-purpose automatic evaluation of specimen images of tissue specimens spanning multiple types of organs and / or multiple types of tissues.

[0007] The image analysis device of the present disclosure includes a processor, which acquires a target specimen image depicting a tissue specimen of a target subject subjected to an evaluation test of a target candidate substance, inputs the target specimen image to a feature extractor common to multiple types of organs and / or multiple types of tissues, causes the feature extractor to output target features of the target specimen image, and calculates the degree of deviation of the target feature from the distribution of reference features obtained by inputting multiple reference specimen images depicting tissue specimens of organs and / or tissues of a reference subject belonging to a control group, which are organs and / or tissues corresponding to the organs and / or tissues of the tissue specimen depicted in the target specimen image, into the feature extractor.

[0008] The processor preferably determines whether or not a morphological abnormality has occurred in the tissue specimen shown in the target specimen image based on the degree of discrepancy.

[0009] It is preferable that at least a part of a machine learning model that outputs an identification result of the organ and / or tissue of the tissue specimen shown in the input specimen image is diverted to the feature extractor.

[0010] The control group preferably includes at least one of a first control group and a second control group used in an evaluation test different from that of the first control group.

[0011] The control group preferably consists of only reference subjects who have not received the candidate substance of interest in the evaluation test.

[0012] The control group preferably consists of only reference subjects who have not received the reference candidate substance in a previous evaluation study.

[0013] It is preferable that the processor acquires identification information of the organs and / or tissues of the tissue specimen shown in the target specimen image, and selects a distribution corresponding to the identification information from multiple distributions prepared for multiple types of organs and / or multiple types of tissues.

[0014] Preferably, the processor analyzes the target specimen image to identify the organs and / or tissues of the tissue specimen shown in the target specimen image, and obtains the identification result as identification information.

[0015] It is preferable that the processor integrates a first distribution, which is a distribution relating to the first control group, and a second distribution, which is a distribution relating to the second control group, to generate an integrated distribution, and calculates the deviation of the target feature from the integrated distribution.

[0016] Preferably, the first control group comprises reference subjects who have not been administered the candidate substance of interest, and the second control group comprises reference subjects who have not been administered the candidate substance of interest in a previous evaluation test.

[0017] Preferably, the device supports evaluation of the toxicity of a target candidate substance based on the degree of deviation.

[0018] The method of operating the image analysis device of the present disclosure includes acquiring a target specimen image that depicts a tissue specimen of a target subject that has been subjected to an evaluation test of a target candidate substance, inputting the target specimen image into a feature extractor that is common to multiple types of organs and / or multiple types of tissues and outputting target features of the target specimen image from the feature extractor, and inputting multiple reference specimen images that depict tissue specimens of organs and / or tissues of reference subjects belonging to a control group, which correspond to the organs and / or tissues of the tissue specimen depicted in the target specimen image, into the feature extractor and calculating the degree of deviation of the target feature from the distribution of reference features obtained.

[0019] The operating program for the image analysis device of the present disclosure causes a computer to execute processes including acquiring a target specimen image that depicts a tissue specimen of a target subject that has been subjected to an evaluation test of a target candidate substance, inputting the target specimen image into a feature extractor that is common to multiple types of organs and / or multiple types of tissues and outputting target features of the target specimen image from the feature extractor, and inputting multiple reference specimen images that depict tissue specimens of organs and / or tissues of a reference subject belonging to a control group that correspond to the organs and / or tissues of the tissue specimen depicted in the target specimen image into the feature extractor and calculating the degree of deviation of the target feature from the distribution of reference features obtained.

[0020] According to the technology disclosed herein, it is possible to provide an image analysis device, an operating method for an image analysis device, and an operating program for an image analysis device that are capable of performing general-purpose automatic evaluation of specimen images of tissue specimens spanning multiple types of organs and / or multiple types of tissues.

[0021] 1 is a diagram illustrating a procedure for generating a specimen image, a specimen image, and an evaluation support device.

[0023] FIG. 1 is a diagram illustrating an administration group and a control group.

[0024] FIG. 1 is a block diagram illustrating a computer constituting the evaluation support device.

[0025] FIG. 1 is a block diagram illustrating a processing unit of a CPU of the evaluation support device.

[0026] FIG. 1 is a diagram illustrating processing by a generation unit.

[0027] FIG. 1 is a diagram illustrating processing by an identification unit.

[0028] FIG. 1 is a diagram illustrating processing by an identification unit.

[0029] FIG. 2 is a diagram illustrating the processing of a target patch image obtained by subdividing a target region image.

[0029] FIG. 2 is a diagram illustrating the extraction of target features from a target patch image by a feature extractor.

[0029] FIG. 3 is a diagram illustrating the structure of a feature extractor.

[0030] FIG. 3 is a diagram illustrating the processing in the learning phase of a discriminative model.

[0031] FIG. 4 is a diagram illustrating the structure of a past control group and a learning reference patch image.

[0032] FIG. 4 is a diagram illustrating the extraction of reference features from a reference patch image by a feature extractor.

[0033] FIG. 5 is a graph plotting reference features in feature space.

[0034] FIG. 6 is a diagram illustrating the positions in feature space of reference features of reference patch images depicting tissue specimens with different shape and texture features.

[0035] FIG. 7 is a diagram illustrating the distribution of reference features and judgment reference information.

[0036] FIG. 8 is a diagram illustrating the distance between the position of a target feature and a representative position of a reference feature.

[0037] FIG. 9 is a diagram illustrating a group of judgment reference information.

[0038] FIG. 10 is a diagram illustrating the selection of judgment reference information corresponding to the identification result. [0 1 is a diagram showing a group of judgment results. 2 is a diagram showing an image list display screen. 3 is a diagram showing an analysis result display screen. 4 is a flowchart showing the processing procedure of the evaluation support device. 5 is a flowchart showing the processing procedure of the evaluation support device. 6 is a flowchart showing the processing procedure of the evaluation support device. 7 is a diagram showing a second embodiment in which a distribution of extracted reference features is generated by inputting a reference patch image obtained from a reference subject of a control group into a feature extractor, and judgment reference information is further generated from the distribution. 8 is a diagram showing a third embodiment in which a distribution of extracted reference features is generated by inputting a reference patch image obtained from a reference subject of a past control group into a feature extractor, and a distribution of extracted reference features is generated by inputting a reference patch image obtained from a reference subject of a control group into the feature extractor, the two generated distributions are treated as an integrated distribution, and judgment reference information is further generated from the integrated distribution. 9 is a diagram showing how distributions based on a past control group and a distribution based on a control group, which are distributions of one organ and / or tissue, are integrated.FIG. 10 is a diagram showing how distributions based on a historical control group and a control group are integrated together, with distributions of multiple types of organs and / or multiple types of tissues.

[0022] First Embodiment As shown in FIG. 1 as an example, an evaluation support device 10 is a device that supports the evaluation of the toxicity of a drug target candidate substance 27T (see FIG. 2). The evaluation support device 10 is an example of an "image analysis device" according to the technology of the present disclosure. The drug may be, for example, a biopharmaceutical such as an antibody drug that uses an antibody as an active ingredient, or a peptide drug or nucleic acid drug that uses a peptide or nucleic acid as an active ingredient.

[0023] The evaluation support device 10 is, for example, a desktop personal computer, and includes a display 11 for displaying various screens and an input device 12 such as a keyboard, a mouse, a touch panel, and / or a microphone for voice input. The evaluation support device 10 is installed, for example, in a pharmaceutical company developing drugs, or in an organization contracted by a pharmaceutical company to develop drugs, i.e., a contract research organization (CRO). The evaluation support device 10 is operated by a user U involved in drug development at the pharmaceutical company or contract research organization (hereinafter collectively referred to as a pharmaceutical facility). The user U is, for example, a pathologist responsible for evaluating the target candidate substance 27T.

[0024] A plurality of specimen images 15 are input to the evaluation support device 10. The specimen images 15 are images used to evaluate the toxicity of the target candidate substance 27T. The specimen images 15 are generated, for example, by the following procedure. First, a subject S, such as a rat, prepared for evaluation of the target candidate substance 27T is autopsied, and tissue specimens are obtained by slicing the organs and tissues of the subject S. The tissue specimens include a brain specimen BS, a heart specimen HS, a lung specimen LS, a liver specimen LVS, a kidney specimen KDS, a spleen specimen SPS, an adrenal specimen AGS, and a pituitary specimen PGS. Although not shown in FIG. 1 , the tissue specimens also include specimens of various organs and tissues, such as the esophagus, stomach, large intestine, small intestine, pancreas, gallbladder, aorta, vena cava, lymph nodes, trachea, bronchi, diaphragm, pineal gland, testes or ovaries, and spinal cord.

[0025] After the tissue specimens are collected, they are attached to glass slides 16 in accordance with standard operating procedures (SOPs) established in advance for each pharmaceutical facility. The SOPs include specifications for the slice thickness of the tissue specimen, the dye to be used for staining, and detailed layout specifications for the tissue specimens, such as attaching a heart specimen HS and a lung specimen LS side by side on the same glass slide 16. In this manner, multiple tissue specimens are attached to a single glass slide 16.

[0026] The tissue specimen is then stained, in this case with hematoxylin-eosin dye. The stained tissue specimen is then covered with a cover glass 17 to complete a slide specimen 18. The slide specimen 18 is then placed in an imaging device 19, such as a digital optical microscope, and a specimen image 15 is captured by the imaging device 19. The resulting specimen image 15 captures the entire tissue specimen attached to the slide glass 16. In other words, a single specimen image 15 captures multiple tissue specimens. The specimen image 15 is called a WSI (Whole Slide Image). The specimen image 15 is assigned a specimen ID (Identification Data) for uniquely identifying the specimen S, a specimen image ID for uniquely identifying the specimen image 15, and the date and time of capture. The tissue specimen is also called a tissue section. The staining may be performed using hematoxylin dye alone, nuclear fast red dye, or the like.

[0027] As an example, as shown in FIG. 2 , the administration group 25 is composed of multiple subjects S (hereinafter referred to as target subjects ST) administered with a target candidate substance 27T. The administration group 25 may be further divided into a high-dose group, a medium-dose group, and a low-dose group according to the dose of the target candidate substance 27T. By dividing the administration group 25 into a high-dose group, a medium-dose group, and a low-dose group in this manner, the effect of the dose of the target candidate substance 27T on the target subjects ST can be determined. The administration group 25 is not limited to being divided into three groups: a high-dose group, a medium-dose group, and a low-dose group. The administration group 25 may also be divided into two groups: a high-dose group and a low-dose group, or into four or more groups. The administration group 25 may also be divided according to the duration of administration of the target candidate substance 27T. Alternatively, the administration group 25 may be divided according to the frequency of administration of the target candidate substance 27T.

[0028] Contrary to the administration group 25, the control group 26 is composed of a plurality of subjects S (hereinafter referred to as reference subjects SR) to which the target candidate substance 27T has not been administered. The number of target subjects ST constituting the administration group 25 and the number of reference subjects SR constituting the control group 26 are the same, for example, about 5 to 10. In the following description, the specimen image 15 obtained from the target subjects ST will be referred to as the subject specimen image 15T, and the specimen image 15 obtained from the reference subjects SR will be referred to as the reference specimen image 15R.

[0029] The target specimens ST constituting the administration group 25 and the reference specimens SR constituting the control group 26 are specimens S having the same attributes and reared under the same rearing environment. Examples of the same attributes include the same age in weeks, the same gender, and / or the same genetic lineage. The same attributes also include the same age composition ratio, the same gender composition ratio (e.g., five males and five females), and / or the same genetic lineage composition ratio. The same rearing environment includes, for example, the same food, the same temperature and humidity of the rearing space, and / or the same size of the rearing space. The same genetic lineage includes, for example, the same ancestor five generations ago and / or the same gene sequence in a specific region. The "same" in the "same rearing environment" refers not only to the exact sameness, but also to the sameness in the sense of a margin of error generally acceptable in the technical field to which the technology of the present disclosure pertains, which does not contradict the spirit of the technology of the present disclosure.

[0030] Since multiple target specimen images 15T are obtained from one target specimen ST, the number of target specimen images 15T obtained from the administration group 25 is equal to the number obtained from one target specimen ST multiplied by the number of target specimens ST. For example, if the number of target specimen images 15T obtained from one target specimen ST is 100 and the number of target specimens ST constituting the administration group 25 is 10, 100 x 10 = 1000 target specimen images 15T are obtained from the administration group 25. Similarly, the number of reference specimen images 15R obtained from the control group 26 is equal to the number obtained from one reference specimen SR multiplied by the number of reference specimens SR.

[0031] 3, the computer constituting the evaluation support device 10 includes, in addition to the display 11 and input device 12, a storage 30, a memory 31, a CPU (Central Processing Unit) 32, and a communication unit 33. These components are interconnected via a bus line 34.

[0032] The storage 30 is a hard disk drive built into the computer constituting the evaluation support device 10 or connected via a cable or network. Alternatively, the storage 30 is a disk array consisting of multiple hard disk drives. The storage 30 stores control programs such as an operating system, various application programs, and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0033] The memory 31 is a work memory for the CPU 32 to execute processing. The CPU 32 loads programs stored in the storage 30 into the memory 31 and executes processing in accordance with the programs. In this way, the CPU 32 comprehensively controls each part of the computer. The CPU 32 is an example of a "processor" according to the technology of the present disclosure. Note that the memory 31 may be built into the CPU 32. The communication unit 33 controls the transmission of various information to and from external devices such as the imaging device 19.

[0034] 4, an operating program 40 is stored in the storage 30 of the evaluation support device 10. The operating program 40 is an application program for causing a computer to function as the evaluation support device 10. In other words, the operating program 40 is an example of an "operating program for an image analysis device" according to the technology of the present disclosure. The storage 30 also stores a defined model 41, a specific model 42, a feature extractor 43, a judgment reference information group 44, and the like.

[0035] When the operating program 40 is started, the CPU 32 of the computer constituting the evaluation support device 10 cooperates with the memory 31 and the like to function as a read / write (hereinafter abbreviated as RW (Read Write)) control unit 50, a generation unit 51, an identification unit 52, an extraction unit 53, a determination unit 54, and a display control unit 55. In addition to these processing units 50 to 55, the CPU 32 also includes an instruction receiving unit that receives various operation instructions from the input device 12, etc.

[0036] The RW control unit 50 controls the storage of various data in the storage 30 and the reading of various data from the storage 30. For example, the RW control unit 50 acquires a specimen image group 60 from the imaging device 19 and stores the specimen image group 60 in the storage 30. The specimen image group 60 is a collection of a plurality of target specimen images 15T and a plurality of reference specimen images 15R.

[0037] When a user U issues an instruction to display the image list display screen 120 (see FIG. 23 ) via the input device 12, the RW control unit 50 reads out the specimen image group 60 from the storage 30. The RW control unit 50 outputs the read out specimen image group 60 to the generation unit 51.

[0038] The RW control unit 50 reads the definite model 41 from the storage 30 and outputs the read definite model 41 to the generation unit 51. The RW control unit 50 also reads the specific model 42 from the storage 30 and outputs the read specific model 42 to the identification unit 52. The RW control unit 50 also reads the feature extractor 43 from the storage 30 and outputs the read feature extractor 43 to the extraction unit 53.

[0039] The generation unit 51 uses the demarcation model 41 to demarcate multiple tissue specimen regions shown in each specimen image 15. Then, an image of the demarcated tissue specimen region (hereinafter referred to as region image) 71 (see FIG. 5 ) is generated from the specimen image 15. The generation unit 51 outputs a region image group 61, which is a collection of the multiple generated region images 71, to the identification unit 52, extraction unit 53, and display control unit 55. In the following description, the region image 71 generated from the target specimen image 15T will be referred to as the target region image 71T (see FIG. 8 , etc.), and the region image 71 generated from the reference specimen image 15R will be referred to as the reference region image 71R (see FIG. 28 , etc.).

[0040] The identification unit 52 uses the identification model 42 to identify the organs and / or tissues of the tissue specimen shown in the region image 71. That is, the identification unit 52 analyzes the region image 71 to identify the organs and / or tissues of the tissue specimen shown in the region image 71. The identification model 42 is capable of identifying several tens of types of organs and / or tissues, for example, approximately 40 types of organs and / or tissues. Because the region image 71 is a part of the specimen image 15, analyzing the region image 71 is equivalent to analyzing the specimen image 15. The identification unit 52 outputs an identification result group 62, which is a collection of identification results 75 (see FIGS. 6 and 7 ) of the organs and / or tissues of the tissue specimen shown in the region image 71, to the extraction unit 53, the determination unit 54, and the display control unit 55.

[0041] The extraction unit 53 uses the feature extractor 43 to extract target features 87T (see FIG. 9 ) of the target area image 71T for which an analysis instruction has been given by the user U via the input device 12 on the image list display screen 120. The extraction unit 53 outputs a target feature group 63, which is a collection of the target features 87T, to the determination unit 54.

[0042] The determination unit 54 determines whether or not a morphological abnormality has occurred in the tissue specimen depicted in the target region image 71T, based on the target feature set 63 and the determination reference information 64 selected from the determination reference information set 44. The determination unit 54 outputs a determination result set 65, which is a collection of determination results 110 (see FIGS. 20 and 21 ) indicating whether or not a morphological abnormality has occurred, to the display control unit 55. Here, morphological abnormalities refer to lesions that are not found in normal tissue specimens, such as hyperplasia, infiltration, congestion, cysts, inflammation, tumors, canceration, proliferation, hemorrhage, glycogen depletion, inclusion bodies, granular cytoplasm, foamy cytoplasm, etc.

[0043] The display control unit 55 controls the display of various screens on the display 11. The various screens include an image list display screen 120 that displays a list of the target area images 71T, an analysis result display screen 130 (see FIG. 24 ) that displays the analysis results of the target area images 71T, and the like.

[0044] As shown in FIG. 5 as an example, the generation unit 51 inputs the specimen image 15 to the demarcation model 41. The demarcation model 41 then identifies the multiple tissue specimens appearing in the specimen image 15. The demarcation model 41 is a machine learning model such as a convolutional neural network. The demarcation model 41 identifies each of the multiple tissue specimens appearing in the specimen image 15 and outputs the position coordinates of a rectangular frame (called a bounding box) surrounding the tissue specimen as a demarcation result 70. The generation unit 51 generates a region image 71 for each tissue specimen by cutting out a rectangular frame from the specimen image 15 according to the demarcation result 70. The generation unit 51 performs the above-described process of generating region images 71 for all specimen images 15 constituting the specimen image group 60. Like the specimen images 15, the region images 71 are assigned a region image ID for uniquely identifying the region image 71.

[0045] 5 illustrates an example of specimen images 15 that include two kidney specimens KDS, a spleen specimen SPS, and a sublingual gland specimen and a submandibular gland specimen SLGSMGS. As can be seen from the example of the sublingual gland specimen and the submandibular gland specimen SLGSMGS, the region image 71 may include a mixture of tissue specimens of multiple organs and / or tissues.

[0046] The rectangular frame surrounding the tissue specimen, which is the demarcation result 70 of the demarcation model 41, may be configured to be modifiable by the user U. Also, instead of using the demarcation model 41, the area of ​​the tissue specimen appearing in the specimen image 15 may be demarcated by template matching. Alternatively, the area of ​​the tissue specimen appearing in the specimen image 15 may be demarcated by the user U's hand by inputting a rectangular frame surrounding the tissue specimen without using the demarcation model 41 or template matching.

[0047] As an example, as shown in Figures 6 and 7, the identification unit 52 inputs a region image 71 to the identification model 42. Then, the identification model 42 identifies the organs and / or tissues of the tissue specimen shown in the region image 71. Like the definition model 41, the identification model 42 is a machine learning model such as a convolutional neural network. The identification model 42 outputs an identification result 75 of the organs and / or tissues of the tissue specimen shown in the region image 71. The identification result 75 is an example of "identification information" according to the technology of the present disclosure.

[0048] FIG. 6 illustrates an example in which a region image 71 showing a kidney specimen KDS is input to the identification model 42, and an identification result 75 of "kidney" is output from the identification model 42. FIG. 7 illustrates an example in which a region image 71 showing a sublingual gland specimen and a submandibular gland specimen SLGSMGS is input to the identification model 42, and an identification result 75 of "sublingual gland and submandibular gland" is output from the identification model 42. Note that while FIG. 6 illustrates an example in which an organ identification result 75 is output, and FIG. 7 illustrates an example in which a tissue identification result 75 is output, this is not limiting. The identification model 42 can also output an identification result 75 of a combination of organs and tissues, such as "lung vascular tissue" or "liver adipose tissue."

[0049] The identification result 75 may be configured to be modifiable by the user U. Furthermore, instead of using the identification model 42, the organs and / or tissues of the tissue specimen shown in the region image 71 may be identified by template matching.

[0050] The identification of the organs and / or tissues of the tissue specimen shown in the region image 71 may be left to the user U. Specifically, the user U may input text data of the organs and / or tissues shown in the region image 71, or may operate a GUI (Graphical User Interface) such as a pull-down menu or radio buttons to select the organs and / or tissues shown in the region image 71. In these cases, the text data or input information of the pull-down menu or radio buttons is an example of "identification information" according to the technology of the present disclosure. Furthermore, the organ and / or tissue name attached by the creator to the label of the slide specimen 18 from which the region image 71 is derived may be read by character recognition. In this case, the organ and / or tissue name read by character recognition is an example of "identification information" according to the technology of the present disclosure.

[0051] As an example, as shown in FIG. 8 , the extraction unit 53 uses well-known image recognition technology to recognize a tissue specimen (a liver specimen LVS is exemplified in FIG. 8 ) depicted in a target region image 71T for which an analysis instruction has been given by the user U, and subdivides the recognized tissue specimen into multiple target patch images 85T. The target patch image 85T has a predetermined size that can be handled by the feature extractor 43. The extraction unit 53 assigns a patch image ID to the target patch image 85T. The extraction unit 53 also associates information indicating which position in the target region image 71T the target patch image 85T is cut out from, i.e., position information of the target patch image 85T, with the patch image ID. Note that, although adjacent target patch images 85T do not have overlapping regions in FIG. 8 , adjacent target patch images 85T may partially overlap.

[0052] As an example, as shown in FIG. 9 , the extraction unit 53 uses the feature extractor 43 to extract target features 87T from a target patch image 85T. The extraction unit 53 performs extraction of target features 87T using this feature extractor 43 for all target patch images 85T. Therefore, the number of target features 87T is the same as the number of target patch images 85T. The target feature group 63 is a collection of target features 87T from all target patch images 85T. Note that the target patch image 85T is part of the target region image 71T, and the target region image 71T is part of the target specimen image 15T. Therefore, extracting the target feature 87T from the target patch image 85T is synonymous with extracting the target feature 87T from the target specimen image 15T.

[0053] As an example, as shown in FIG. 10 , the feature extractor 43 utilizes the encoder unit 91 of the discriminative model 90. Like the definition model 41, the discriminative model 90 is a machine learning model, such as a convolutional neural network, and includes a decoder unit 92 in addition to the encoder unit 91. A patch image 85, such as a target patch image 85T, is input to the encoder unit 91. The encoder unit 91 converts the patch image 85 into a feature 87. The encoder unit 91 passes the feature 87 to the decoder unit 92. The decoder unit 92 identifies the organs and / or tissues of the tissue specimen depicted in the patch image 85 based on the feature 87 and outputs the resulting identification result 93. In other words, the discriminative model 90 is similar to the identification model 42 in that it performs the task of identifying the organs and / or tissues of the tissue specimen depicted in the input image, although the input image is the patch image 85 rather than the region image 71. Like the identification model 42, the discriminative model 90 is capable of identifying several tens of types of organs and / or tissues, for example, approximately 40 types of organs and / or tissues. The discrimination model 90 is an example of a "machine learning model that outputs a discrimination result of an organ and / or tissue of a tissue specimen shown in an input specimen image" according to the technology of the present disclosure. The encoder unit 91 is an example of "at least a part of a machine learning model" according to the technology of the present disclosure.

[0054] As is well known, the encoder unit 91 includes a convolution layer that performs convolution processing using a filter, a pooling layer that performs pooling processing such as maximum value pooling, and the like. The decoder unit 92 is similar. The encoder unit 91 extracts features 87 by repeating convolution processing by the convolution layer and pooling processing by the pooling layer multiple times on the input patch image 85. The extracted features 87 represent the shape and texture characteristics of the tissue sample captured in the patch image 85. The features 87 are a set of multiple numerical values. In other words, the features 87 are multidimensional data. The number of dimensions of the features 87 is, for example, 512, 1024, or 2048.

[0055] 11 , in the learning phase before the encoder unit 91 is converted into the feature extractor 43, training data 95 is provided to the discrimination model 90. The training data 95 is composed of a set of a training reference patch image 85RL and supervised answer data 93CA. The supervised answer data 93CA is data in which the organs and / or tissues of the tissue specimen shown in the training reference patch image 85RL are registered, and is, so to speak, data for checking the answer.

[0056] A training reference patch image 85RL is input to the discriminative model 90. The discriminative model 90 outputs a training classification result 93L in response to the input of the training reference patch image 85RL. A loss calculation is performed for the discriminative model 90 using a loss function based on this training classification result 93L and supervised data 93CA. Then, update settings are made for various coefficients of the discriminative model 90 (such as coefficients of the filters in the convolutional layer) in response to the result of the loss calculation, and the discriminative model 90 is updated in accordance with the update settings.

[0057] In the learning phase of the discriminative model 90, the above-described series of processes, including input of the training reference patch image 85RL to the discriminative model 90, output of the training classification result 93L from the discriminative model 90, loss calculation, update setting, and update of the discriminative model 90, are repeatedly performed while exchanging training data 95. The repetition of the above-described series of processes is terminated when the classification accuracy of the training classification result 93L reaches a predetermined set level. The encoder unit 91 of the discriminative model 90 whose classification accuracy has thus reached the set level is stored in the storage 30 of the evaluation support device 10 as the feature extractor 43. Note that learning may be terminated when the above-described series of processes has been repeated a set number of times, regardless of the classification accuracy of the training classification result 93L.

[0058] The training of the discrimination model 90 may be performed by the evaluation support device 10 or by a device separate from the evaluation support device 10. In the latter case, the feature extractor 43 is transmitted from the separate device to the evaluation support device 10, and the RW control unit 50 stores the feature extractor 43 in the storage 30.

[0059] 12, the learning reference patch image 85RL is supplied from a plurality of reference patch images 85RP obtained by subdividing a reference region image 71RP generated from a reference specimen image 15RP. The reference specimen image 15RP is an image of a tissue specimen of a reference subject SRP in a past control group 26P. The past control group 26P is composed of a plurality of reference subjects SRP to which a reference candidate substance 27R was not administered in a past evaluation test. Therefore, the number of reference subjects SRP constituting the past control group 26P is significantly greater than the number of reference subjects SR constituting the control group 26, for example, on the order of several hundred to several thousand.

[0060] The reference specimens SRP constituting the past control group 26P differ from the reference specimens SR constituting the control group 26 in at least one of the following: attributes, rearing environment, and tissue specimen preparation method. Examples of different attributes include different ages in weeks, different genders, and / or different genetic lineages. Different attributes also include different age composition ratios, different gender composition ratios, and / or different genetic lineage composition ratios. Examples of different rearing environments include different diets, different temperature and humidity of the rearing space, and / or different size of the rearing space. Examples of different tissue specimen preparation methods include different tissue specimen slice thicknesses, different dyes used for staining, and different organs and / or tissues attached side by side to the same slide glass 16. The control group 26 is an example of a "first control group" according to the technology disclosed herein. The past control group 26P is also an example of a "control group" and a "second control group" according to the technology disclosed herein. The reference specimens SRP constituting the past control group 26P may include reference specimens SR that have the same attributes, rearing environment, and tissue specimen preparation method as the reference specimens SR constituting the control group 26.

[0061] Since multiple reference region images 71RP are obtained from one reference subject SRP, the number of reference region images 71RP obtained from the past control group 26P is calculated by multiplying the number of reference region images 71RP obtained from one reference subject SRP by the number of reference subject SRPs. The tissue specimens captured in the reference region images 71RP span multiple types of organs and / or multiple types of tissues. Therefore, the training reference patch images 85RL based on the reference region images 71RP also span multiple types of organs and / or multiple types of tissues, and the feature extractor 43, which utilizes the encoder unit 91 of the discrimination model 90 trained using these training reference patch images 85RL, can be commonly applied to multiple types of organs and / or multiple types of tissues.

[0062] In this way, by using training reference patch images 85RL covering multiple types of organs and / or multiple types of tissues in training the discriminant model 90, even if the discriminant model 90 is trained by labeling only relatively rough features such as organs and / or tissues, training proceeds while comparing differences in features between different organs or tissues. This allows for the acquisition of features focusing on the finer structure of tissues, i.e., latent features not labeled in the training reference patch images 85RL, which cannot be acquired when training is performed using training reference patch images 85RL of a single type of organ or tissue. Obtaining these latent features enables the detailed capture of normal tissue structure, enabling accurate discrimination between normal tissue and tissue with morphological abnormalities. Therefore, the feature extractor 43, which utilizes the encoder unit 91 of the discriminant model 90, can also identify finer features (latent features) not labeled in the training reference patch images 85RL. In this way, because the feature extractor 43 is capable of identifying latent features, it is possible to identify differences in features between organs, such as when an image of one organ in a normal state resembles a morphological abnormality of another organ. Furthermore, by learning multiple types of organs and / or multiple types of tissues in the identification model 90 and comparing the differences in features between different organs or tissues, the feature extractor 43 can obtain features that can identify different organs or tissues even when the amount of training data between the organs is uneven.

[0063] In addition to the reference area image 71RP that depicts a tissue specimen of the reference subject SRP in the past control group 26P, a target area image that depicts a tissue specimen that an expert such as a pathologist has determined to be normal in a past administration group consisting of multiple target subjects to whom the reference candidate substance 27R was administered in a past evaluation test may also be used as the reference area image 71RP.

[0064] 12 illustrates a reference region image 71RP depicting a liver specimen LVS. The correct answer data 93CA paired with a learning reference patch image 85RL based on the reference region image 71RP depicting this liver specimen LVS has "liver" registered as the organ type. Similarly, the correct answer data 93CA paired with a learning reference patch image 85RL based on a reference region image 71RP depicting, for example, a sublingual gland specimen and a submandibular gland specimen SLGSMGS has "sublingual gland and submandibular gland" registered as the tissue type.

[0065] Next, the structure of the determination reference information 64 will be described. First, as shown in FIG. 13 as an example, a feature extractor 43 is used to extract reference features 87RP of a reference patch image 85RP obtained by subdividing a reference area image 71RP. Then, extraction of the reference features 87RP using this feature extractor 43 is performed on the reference patch images 85RP of all reference area images 71RP obtained from the reference subject SRP of the past control group 26P. The reference features 87RP are associated with the organs and / or tissues of the tissue specimen depicted in the original reference area image 71RP. Note that the reference patch image 85RP is a part of the reference area image 71RP, and the reference area image 71RP is a part of the reference specimen image 15RP. Therefore, extracting the reference features 87RP of the reference patch image 85RP is synonymous with extracting the reference features 87RP of the reference specimen image 15RP.

[0066] As an example, the graph 100 shown in FIG. 14 plots the reference features 87RP extracted in FIG. 13 in a feature space 101. As described above, the reference features 87RP are extracted for the reference patch images 85RP of all reference region images 71RP obtained from the reference subject SRP of the past control group 26P. The reference features 87RP are associated with the organs and / or tissues of the tissue specimen depicted in the original reference region images 71RP. Therefore, a distribution 103P of the reference features 87RP is generated for each of multiple types of organs and / or multiple types of tissues. The distribution 103P can be expressed mathematically by fitting a parametric probability distribution, such as a multivariate normal distribution or a multivariate mixed normal distribution, to the multiple reference features 87RP for each organ and / or tissue.

[0067] FIG. 14 illustrates distributions 103P of reference features 87RP for multiple types of organs and / or multiple types of tissues, such as "liver," "liver and adipose tissue," "liver and vascular tissue," and "sublingual and submandibular glands." In reality, distributions 103P of reference features 87RP for a wide variety of organs and / or tissues are generated in addition to the illustrated organs and / or tissues. Note that in FIG. 14, for convenience of explanation, the dimension of the feature space 101 is shown as two-dimensional having a D1 axis and a D2 axis, but the actual dimension of the feature space 101 is 512 dimensions, as described above. Similarly, in the following FIG. 15 and the like, for convenience of explanation, the dimension of the feature space 101 is shown as two-dimensional.

[0068] As an example, as shown in FIG. 15 , in the distribution 103P of the reference feature 87RP, the reference feature 87RP of the reference patch image 85RP depicting the tissue specimens TSA, TSB, TSC, and TSD, each with different shape and texture characteristics, is located in approximately the same region of the feature space 101. Furthermore, the reference feature 87RP of the reference patch image 85RP depicting the tissue specimens with the same latent feature is located close to each other, while the reference feature 87RP of the reference patch image 85RP depicting the tissue specimens with different latent features is located far from each other. In this way, by using a feature extractor 43 capable of identifying finer features (latent features) that are not labeled in the training reference patch image 85RL, it is possible to extract a reference feature 87RP that accurately reflects the finer features (latent features) that are not labeled. The same can be said for the target feature 87T. Therefore, it is considered possible to distinguish between tissue specimens in which no morphological abnormality has occurred and tissue specimens in which morphological abnormality has occurred using the target feature 87T and the reference feature 87RP.

[0069] As an example, as shown in FIG. 16 , the determination reference information 64 includes coordinates 102 in a feature space 101 of a representative position of a reference feature 87RP, indicated by a diamond (hereinafter referred to as a representative position coordinate). The representative position can be derived from a mathematical formula obtained by fitting a probability distribution. The representative position is, for example, the center point, center of gravity, or average point of a distribution 103P of the reference feature 87RP. FIG. 16 illustrates an example of the distribution 103P of the reference feature 87RP of the liver and the determination reference information 64. The determination reference information 64 also includes a determination threshold 104.

[0070] 17 , the determination unit 54 calculates the distance D in the feature space 101 between the representative position of the reference feature 87RP represented by the representative position coordinates 102 of the determination reference information 64 and the position of the target feature 87T. The distance D is, for example, the Euclidean distance. The determination unit 54 calculates the distance D between the multiple target features 87T extracted for each of the multiple target patch images 85T. The distance D indicates the degree of deviation of the target feature 87T from the reference feature 87RP, or more specifically, the degree of deviation of the tissue specimen depicted in the target patch image 85T from the tissue specimen of the reference subject SRP that constitutes the past control group 26P. In other words, the distance D is an example of the "degree of deviation" according to the technology of the present disclosure.

[0071] The tissue specimens of the reference subject SRP that constitute the past control group 26P do not have morphological abnormalities at least due to the toxicity of the reference candidate substance 27R, and in that sense can be considered normal. Therefore, the larger the distance D, the more the tissue specimen depicted in the target patch image 85T deviates from a tissue specimen considered normal. Therefore, the larger the distance D, the more likely it is that the tissue specimen depicted in the target patch image 85T has morphological abnormalities.

[0072] Note that instead of Euclidean distance, Mahalanobis distance may be calculated as distance D. Alternatively, the average, median, or maximum value of the Euclidean distance between the position of the k-neighbor sample in the distribution 103P of the reference feature 87RP and the position of the target feature 87T may be calculated. Alternatively, instead of distance D, the value obtained by subtracting the cosine similarity between a vector representing the representative position of the reference feature 87RP and a vector representing the position of the target feature 87T from 1.0 may be calculated as the deviation. The cosine similarity takes a value between -1.0 and 1.0, and the larger the value, the more similar the orientations of the vectors are. Furthermore, instead of distance D, a likelihood function such as negative logarithmic likelihood may be calculated as the deviation.

[0073] Considering the possibility that spontaneous morphological abnormalities may also occur in the tissue specimen of the reference subject SRP, the determination threshold 104 is set to, for example, the 95th percentile value of the distance D between the representative position of the reference feature 87RP and the position of the target feature 87T. Note that spontaneous morphological abnormalities refer to morphological abnormalities that occur during the growth stage of the reference subject SRP and are not caused by the toxicity of the reference candidate substance 27R.

[0074] As with the learning of the discrimination model 90, the judgment reference information 64 may be derived by the evaluation support device 10, or may be derived by a device other than the evaluation support device 10. In the latter case, the judgment reference information 64 is transmitted from the other device to the evaluation support device 10, and the RW control unit 50 stores the judgment reference information 64 in the storage 30.

[0075] As an example, as shown in Figure 18, the judgment reference information group 44 is one in which the organ and / or tissue name, representative position coordinates 102, and judgment threshold 104 are registered as judgment reference information 64 for each of multiple types of organs and / or multiple types of tissues.

[0076] 19 , the determination unit 54 selects determination reference information 64 corresponding to the identification result 75 from the identification unit 52 from among the plurality of determination reference information 64 in the determination reference information group 44 prepared for each of the plurality of types of organs and / or the plurality of types of tissues. The determination reference information 64 is generated based on the distribution 103P of the reference feature 87RP. Therefore, in other words, by selecting the determination reference information 64 as described above, the determination unit 54 selects the distribution 103P corresponding to the identification result 75 from the identification unit 52 from among the plurality of distributions 103P prepared for each of the plurality of types of organs and / or the plurality of types of tissues.

[0077] More specifically, the determination unit 54 instructs the RW control unit 50 to read out determination reference information 64 corresponding to the type of organ and / or tissue obtained as the identification result 75. The RW control unit 50 reads out the determination reference information 64 corresponding to the instruction of the determination unit 54 from the determination reference information group 44, and outputs the read determination reference information 64 to the determination unit 54. Fig. 19 illustrates a case where the identification result 75 is "liver" and the determination reference information 64 for "liver" is selected.

[0078] The determination unit 54 compares the calculated distance D with the determination threshold 104. As shown in FIG. 20 as an example, if the distance D is less than the determination threshold 104, the determination unit 54 determines that no morphological abnormality has occurred in the tissue specimen captured in the target patch image 85T. The determination unit 54 outputs a determination result 110 indicating that no morphological abnormality has occurred in the tissue specimen captured in the target patch image 85T. On the other hand, as shown in FIG. 21 , if the distance D is equal to or greater than the determination threshold 104, the determination unit 54 determines that a morphological abnormality has occurred in the tissue specimen captured in the target patch image 85T. The determination unit 54 outputs a determination result 110 indicating that a morphological abnormality has occurred in the tissue specimen captured in the target patch image 85T.

[0079] As an example, as shown in FIG. 22, the determination result group 65 is a group in which a determination result 110 is registered for each patch image ID and position information of a target patch image 85T.

[0080] As shown in FIGS. 12 and 13 , the reference feature 87RP is a feature extracted from the reference patch image 85RP, which is a subdivision of the reference region image 71RP depicting the tissue specimen of the reference subject SRP of the past control group 26P. The reference subject SRP of the past control group 26P is a subject to which the reference candidate substance 27R was not administered. Therefore, as described above, the tissue specimen depicted in the reference region image 71RP does not exhibit morphological abnormalities at least due to the toxicity of the reference candidate substance 27R. Therefore, the representative position of the reference feature 87RP is regarded as the representative position of the feature of the specimen image depicting a normal tissue specimen. Therefore, as described above, the distance D between the representative position of the reference feature 87RP and the position of the target feature 87T serves as an indicator of the degree to which the tissue specimen depicted in the target patch image 85T deviates from the normal tissue specimen. 20, the determination unit 54 determines that the tissue specimen depicted in the target patch image 85T in which the distance D is less than the determination threshold 104 does not deviate from a normal tissue specimen and therefore does not have a morphological abnormality. On the other hand, as shown in FIG. 21, the determination unit 54 determines that the tissue specimen depicted in the target patch image 85T in which the distance D is equal to or greater than the determination threshold 104 does deviate from a normal tissue specimen and therefore does have a morphological abnormality.

[0081] The display control unit 55 controls the display 11 to display an image list display screen 120 shown in FIG. 23 as an example, in response to a display instruction from the user U via the input device 12. The image list display screen 120 has a display area 121. The display area 121 displays a list of area images 71 generated by the generation unit 51. In the display area 121, the area images 71 are arranged in order of area image ID, from top to bottom and from left to right. The area image 71 displays the organ and / or tissue name based on the identification result 75 together with the area image ID.

[0082] The display area 121 displays a region image 71 obtained from a group selected from the administration group 25 and the control group 26 using a pull-down menu 122. Fig. 23 shows an example in which ten target region images 71T with region image IDs "SI00001" to "SI00010" obtained from a target subject ST in the administration group 25 with a subject ID of "R001" are displayed side by side. The subject ID is displayed as a pull-down menu 123, making it possible to switch the subject S whose region image 71 is displayed in the display area 121.

[0083] The image list display screen 120 is a screen that displays a list of area images 71, and is also a screen for issuing an instruction to analyze one of the multiple target area images 71T. The image list display screen 120 has a selection frame 124 that can be moved between the target area images 71T. An analysis button 125 is also provided at the bottom of the image list display screen 120. The user U aligns the selection frame 124 with the desired target area image 71T and then selects the analysis button 125. As a result, the extraction unit 53 extracts the target feature amount 87T for the target area image 71T aligned with the selection frame 124, and the determination unit 54 determines whether or not a morphological abnormality has occurred.

[0084] When the extraction of the target feature 87T by the extraction unit 53 and the determination of whether or not a morphological abnormality has occurred by the determination unit 54 are completed, the display control unit 55 controls the display 11 to display an analysis result display screen 130, as shown in FIG. 24 , as an example. The analysis result display screen 130 displays the target region image 71T for which an analysis instruction was given on the image list display screen 120. The display control unit 55 displays the portion of the target patch image 85T determined by the determination unit 54 to have a morphological abnormality in a manner that makes it distinguishable from the portion of the target patch image 85T determined by the determination unit 54 to have no morphological abnormality, for example, by filling it in red as shown by hatching. In this case, the greater the difference between the distance D and the determination threshold 104, the darker the color may be displayed. Furthermore, the number and / or percentage of the target patch images 85T determined by the determination unit 54 to have a morphological abnormality may be displayed.

[0085] A save button 131 and an OK button 132 are provided at the bottom of the analysis result display screen 130. When the save button 131 is selected, the RW control unit 50 associates the target area image 71T, the target feature amount group 63, and the determination result group 65 and stores them in the storage 30. When the OK button 132 is selected, the display control unit 55 erases the display of the analysis result display screen 130.

[0086] Next, the operation of the above configuration will be described with reference to the flowcharts shown in Figures 25 to 27. First, when the operating program 40 is started in the evaluation support device 10, the CPU 32 of the evaluation support device 10 functions as a RW control unit 50, a generation unit 51, an identification unit 52, an extraction unit 53, a determination unit 54, and a display control unit 55, as shown in Figure 4.

[0087] The photographing device 19 photographs specimen images 15 of a tissue specimen of the subject S. The specimen images 15 are transmitted from the photographing device 19 to the evaluation support device 10. In the evaluation support device 10, as shown in Fig. 25, a specimen image group 60, which is a collection of specimen images 15 from the photographing device 19, is acquired by the RW control unit 50 (step ST100). The specimen image group 60 is stored in the storage 30 under the control of the RW control unit 50 (step ST110).

[0088] 26 , when the user U issues an instruction to display the image list display screen 120 via the input device 12 and the display instruction is accepted by the CPU 32 (YES in step ST200), the RW control unit 50 reads out the specimen image group 60 specified in the display instruction from the storage 30 (step ST210). The specimen image group 60 is output from the RW control unit 50 to the generation unit 51.

[0089] 5 , in the generation unit 51, the specimen image 15 is input to the demarcation model 41, which demarcates the regions of the organs and / or tissues of the multiple tissue specimens depicted in the specimen image 15, and the demarcation results 70 are output from the demarcation model 41. Then, based on the demarcation results 70, a region image 71 of each tissue specimen is generated (step ST220). A region image group 61, which is a collection of the region images 71, is output from the generation unit 51 to the identification unit 52, extraction unit 53, and display control unit 55.

[0090] In the identification unit 52, as shown in Figures 6 and 7, the region image 71 is input to the identification model 42. As a result, the organs and / or tissues of the tissue specimen shown in the region image 71 are identified (step ST230), and the identification results 75 are output from the identification model 42. An identification result group 62, which is a collection of the identification results 75, is output to the extraction unit 53, the determination unit 54, and the display control unit 55. Thereafter, as shown in Figure 23, under the control of the display control unit 55, an image list display screen 120 in which the region images 71 are arranged in the display area 121 is displayed on the display 11 (step ST240).

[0091] 27 , when the user U positions the selection frame 124 on one of the target area images 71T displayed on the image list display screen 120, selects the analysis button 125, and an analysis instruction is accepted by the CPU 32 (YES in step ST300), the target area image 71T is subdivided into target patch images 85T (step ST310), as shown in FIG. 8 . Next, as shown in FIG. 9 , the target patch image 85T is input to the feature extractor 43, which then outputs a target feature 87T (step ST320). Then, as shown in FIG. 19 , the determination unit 54 selects, from the plurality of determination reference information 64 in the determination reference information group 44, the determination reference information 64 corresponding to the identification result 75 from the identification unit 52 (step ST330).

[0092] 17 , the determination unit 54 calculates the distance D in the feature space 101 between the representative position of the reference feature 87RP and the position of the target feature 87T (step ST340). Then, the distance D is compared with the judgment threshold 104 to determine whether or not a morphological abnormality has occurred in the tissue specimen captured in the target patch image 85T (step ST350). As shown in FIG. 20 , if the distance D is less than the judgment threshold 104, a determination result 110 is output indicating that no morphological abnormality has occurred in the tissue specimen captured in the target patch image 85T. On the other hand, as shown in FIG. 21 , if the distance D is equal to or greater than the judgment threshold 104, a determination result 110 is output indicating that a morphological abnormality has occurred in the tissue specimen captured in the target patch image 85T. The calculation of the distance D in step ST340 and the determination of whether or not a morphological abnormality has occurred in the tissue specimen captured in the target patch image 85T in step ST350 are performed for each of the multiple target patch images 85T. A determination result group 65 , which is a collection of the determination results 110 obtained in this way, is output from the determination unit 54 to the display control unit 55 .

[0093] 24 is displayed on the display 11 under the control of the display control unit 55 (step ST360). On the analysis result display screen 130, the portion of the target patch image 85T determined by the determination unit 54 to have a morphological abnormality is displayed so as to be distinguishable from the portion of the target patch image 85T determined by the determination unit 54 to have no morphological abnormality. The user U closely observes the target patch image 85T on the analysis result display screen 130 and evaluates the target candidate substance 27T.

[0094] As described above, the CPU 32 of the evaluation support device 10 includes a RW control unit 50, an extraction unit 53, and a determination unit 54. The RW control unit 50 acquires a target specimen image 15T depicting a tissue specimen of a target specimen ST subjected to an evaluation test for a target candidate substance 27T. The extraction unit 53 inputs a target patch image 85T to a feature extractor 43 common to multiple types of organs and / or multiple types of tissues, and causes the feature extractor 43 to output a target feature 87T of the target patch image 85T. The determination unit 54 calculates a distance D, which is the degree of deviation of the target feature 87T from the distribution 103P of the reference feature 87RP. The distribution 103P is an organ and / or tissue corresponding to the organ and / or tissue of the tissue specimen depicted in the target specimen image 15T, and is obtained by inputting multiple reference patch images 85RP depicting tissue specimens of organs and / or tissues of a reference subject SRP belonging to the past control group 26P to the feature extractor 43.

[0095] In the technology disclosed herein, a feature extractor 43 common to training data 95 covering multiple types of organs and / or multiple types of tissues is used to extract target features 87T. Therefore, as shown in Fig. 15, target features 87T that can distinguish between tissue specimens without morphological abnormalities and tissue specimens with morphological abnormalities can be extracted. This makes it possible to solve a variety of problems at once, such as differences in features between organs, such as an image of one organ in a normal state resembling a morphological abnormality in another organ, an imbalance in the amount of training data 95 between organs, and an inability to detect morphological abnormalities in multiple types of organs.

[0096] Furthermore, in the technology disclosed herein, to calculate the distance D, a distribution 103P of reference features 87RP of a reference patch image 85RP of a reference subject SRP of a past control group 26P, which is a distribution 103P specialized for the organ and / or tissue for which the distance D is to be calculated, is used. Therefore, compared to the technology described in Japanese Patent No. 7220017, the range of tissue specimens considered normal is not excessively large, and the range of tissue specimens considered normal can be set to the minimum necessary range corresponding to the organ and / or tissue. Furthermore, by preparing multiple distributions 103P of reference features 87RP for each organ and / or tissue for which morphological abnormalities are to be detected and using distributions 103P specialized for the organ and / or tissue for which morphological abnormalities are to be detected, regions where morphological abnormalities occur in a wide variety of organs and / or tissues can be accurately detected. From the above, it can be said that the technology disclosed herein is capable of general-purpose automatic evaluation of target specimen images 15T of tissue specimens spanning multiple types of organs and / or multiple types of tissues.

[0097] 20 and 21 , the determination unit 54 determines whether or not a morphological abnormality has occurred in the tissue specimen depicted in the target patch image 85T based on the distance D. Therefore, as shown in the analysis result display screen 130 shown in FIG. 24 , it is possible to provide the user U with information useful for evaluating the target candidate substance 27T, such as by displaying to the user U the portion where the morphological abnormality has occurred. This can facilitate the user U's evaluation of the target candidate substance 27T. Note that instead of making a determination based on the distance D, it is also possible to leave the determination of whether or not a morphological abnormality has occurred to the user U by, for example, displaying the target patch images 85T in descending order of the difference between the distance D and the determination threshold 104.

[0098] 10, the feature extractor 43 utilizes an encoder unit 91 of a discrimination model 90 that outputs a discrimination result 93 of the organ and / or tissue of the tissue specimen depicted in the input patch image 85. Therefore, as shown in Fig. 15, it is possible to extract a target feature 87T that can distinguish between a tissue specimen with no morphological abnormality and a tissue specimen with a morphological abnormality. The same applies to the reference feature 87RP.

[0099] As shown in Figure 12, the control group is a past control group 26P consisting only of reference analytes SRP that were not administered with the reference candidate substance 27R in a past evaluation test. Because the number of reference analytes SRP that make up the past control group 26P is very large, it is possible to prepare a huge number of reference features 87RP obtained from the reference analytes SRP. This makes it possible to generate clear and plausible distributions 103P for each organ and / or tissue.

[0100] 19 , the determination unit 54 acquires identification results 75 of the organs and / or tissues of the tissue specimen shown in the target area image 71T from the identification unit 52. Then, the determination unit 54 selects determination reference information 64 corresponding to the identification results 75 from among the plurality of pieces of determination reference information 64 generated from the plurality of distributions 103P prepared for the plurality of types of organs and / or the plurality of types of tissues. Therefore, the distance D can be calculated without error using the determination reference information 64 corresponding to the organs and / or tissues of the tissue specimen shown in the target area image 71T.

[0101] 6 and 7 , the identification unit 52 analyzes the target area image 71T to identify the organs and / or tissues of the tissue specimen shown in the target area image 71T. The determination unit 54 acquires the identification result 75 by the identification unit 52 as identification information. This makes it possible to acquire the identification result 75 of the organs and / or tissues of the tissue specimen shown in the target area image 71T without bothering the user U.

[0102] The evaluation support device 10 as an image analysis device is a device that supports the evaluation of the toxicity of the target candidate substance 27T based on the distance D as the degree of deviation. This facilitates the user U's evaluation of the toxicity of the target candidate substance 27T.

[0103] The distribution 103P and the determination reference information 64 may be generated each time an analysis instruction for the target region image 71T is received from the user U. Furthermore, since the reference subject SRP constituting the past control group 26P increases over time, the distribution 103P and the determination reference information 64 may be updated periodically.

[0104] Identification of organs and / or tissues of the tissue specimen captured in the target area image 71T is not limited to the method using the identification model 42. The organs and / or tissues may also be identified using the method described below. That is, representative features of each organ and / or tissue are acquired in advance using a machine learning model. Furthermore, representative features of the target area image 71T from which the organs and / or tissues are to be identified are derived using the machine learning model that derived the representative features of each organ and / or tissue. Next, the distance in feature space between the representative features of each organ and / or tissue and the representative features of the target area image 71T is calculated. The organ and / or tissue whose representative feature has the shortest distance is then identified as the organ and / or tissue of the tissue specimen captured in the target area image 71T.

[0105] The discriminative model 90 is not limited to the illustrated convolutional neural network. The discriminative model 90 may be a generative adversarial network (GAN), and its generator may be diverted to the feature extractor 43. A machine learning model without a convolutional layer, such as a Vision Transformer (ViT), with an attention mechanism may also be diverted to the feature extractor 43.

[0106] The target feature 87T is not limited to that extracted by the feature extractor 43 that utilizes at least a part of a machine learning model. The target feature 87T may be the average value, maximum value, minimum value, mode, variance, or the like of the pixel values ​​of the target patch image 85T. The same applies to the reference features 87RP and 87R (see FIG. 28).

[0107] In addition, the judgment threshold 104 may be configured so that the user can change its setting, such as by resetting the judgment threshold 104 to a stricter value if the judgment result 110 indicating that a morphological abnormality has occurred is excessively high, or by resetting the judgment threshold 104 to a looser value if the judgment result 110 indicating that a morphological abnormality has occurred is too low.

[0108] Second Embodiment In the first embodiment, a reference patch image 85RP obtained from a reference subject SRP of a past control group 26P is input to the feature extractor 43 to generate a distribution 103P of extracted reference features 87RP, and further, determination reference information 64 is generated from the distribution 103P, and the distance D between the representative position coordinates 102 of the determination reference information 64 and the position of the target feature 87T is calculated. However, this is not limited to this. As an example, as shown in FIG. 28 , a reference patch image 85R obtained from a reference subject SR of a control group 26 may be input to the feature extractor 43 to generate a distribution 103 of extracted reference features 87R, and further, determination reference information 64 may be generated from the distribution 103, and the distance D between the representative position coordinates 102 of the determination reference information 64 and the position of the target feature 87T may be calculated. As described above, the control group 26 is composed only of reference subjects SR to whom the target candidate substance 27T was not administered in the evaluation test. That is, in the second embodiment, the control group 26 is an example of the “control group” according to the technique of the present disclosure. Note that the reference patch image 85R is an image obtained by dividing the reference area image 71R obtained from the reference specimen image 15R.

[0109] The reference specimen SR constituting the control group 26 has the same attributes, is kept in the same rearing environment as the target specimen ST constituting the administration group 25, and uses the same tissue specimen preparation method. Therefore, it is possible to eliminate differences in the appearance and color of the tissue specimens between the reference specimen image 15R and the target specimen image 15T, which are caused by differences in at least one of the attributes, rearing environment, and tissue specimen preparation method, as occurs when the reference specimen SRP constituting the past control group 26P is used. This can improve the reliability of the calculation result of the distance D and the determination result 110 based on the distance D.

[0110] [Third Embodiment] In the first embodiment, the past control group 26P is given as an example of a "control group" according to the technology of the present disclosure, and in the second embodiment, the control group 26 is given as an example of a "control group" according to the technology of the present disclosure, but this is not limiting. In the third embodiment, both the past control group 26P and the control group 26 are considered to be "control groups" according to the technology of the present disclosure.

[0111] As an example, as shown in FIG. 29 , in the third embodiment, a reference patch image 85RP obtained from a reference subject SRP of a past control group 26P is input to the feature extractor 43 to generate a distribution 103P of extracted reference features 87RP. A reference patch image 85R obtained from a reference subject SR of the control group 26 is also input to the feature extractor 43 to generate a distribution 103 of extracted reference features 87R. Next, these distributions 103P and 103 are integrated to generate an integrated distribution 103IN. For example, the distribution 103P is moved so that the distribution 103 coincides with the representative position coordinates 102, thereby generating an integrated distribution 103IN. Then, determination reference information 64 is generated from the integrated distribution 103IN, and a distance D between the representative position coordinates 102 of the determination reference information 64 and the position of the target feature 87T is calculated. The distribution 103 is an example of a “first distribution” according to the technology disclosed herein. Moreover, distribution 103P is an example of a "second distribution" according to the technology of the present disclosure.

[0112] As described above, the reference specimens SRP constituting the past control group 26P are significantly more numerous than the reference specimens SR constituting the control group 26. Therefore, integrating the distribution 103P and the distribution 103 may improve the reliability of the calculation result of the distance D and the determination result 110 based on the distance D. However, the distributions 103P and 103 may have differences in shape and / or position (called domain shift) due to differences in at least one of the attributes of the reference specimens SRP and the reference specimens SR, the rearing environment, and the preparation method of the tissue specimens. Therefore, if the distributions 103P and 103 are integrated as they are, the range of tissue specimens considered normal will be excessively widened, which will in turn undermine the reliability of the calculation result of the distance D and the determination result 110 based on the distance D.

[0113] Therefore, as an example, as shown in Figure 30, when integrating distribution 103P of one type of organ and / or tissue with distribution 103, distribution 103P is moved so that the representative position of distribution 103P approaches the representative position of distribution 103, and the shape of distribution 103P is changed to match distribution 103, and then an integrated distribution 103IN is generated.

[0114] Alternatively, as an example, as shown in FIG. 31, when integrating distribution 103P and distribution 103 of multiple types of organs and / or tissues, in the figure distributions 103P_1 to 103P_3 and distributions 103_1 to 103_3 of three types of organs and / or tissues together, the representative positions of distribution 103P_1 and distribution 103_1, distribution 103P_2 and distribution 103_2, and distribution 103P_3 and distribution 103_3, which share common organs and / or tissues, are associated with each other, and integrated distribution 103IN is generated by making full use of projective transformation such as affine transformation.

[0115] As described above, in the third embodiment, the distribution 103, which is the distribution related to the control group 26, and the distribution 103P, which is the distribution related to the past control group 26P, are integrated to generate the integrated distribution 103IN. Then, the distance D, which is the degree of deviation of the target feature 87T from the integrated distribution 103IN, is calculated. This makes it possible to further improve the reliability of the calculation result of the distance D and the determination result 110 based on the distance D.

[0116] When integrating distribution 103, which is a distribution related to control group 26, and distribution 103P, which is a distribution related to past control group 26P, a large weight may be assigned to distribution 103, which has a relatively small number of samples. By assigning a large weight to distribution 103 in this way, it is possible to prevent distribution 103P, which has a relatively large number of samples, from dominating integrated distribution 103IN.

[0117] In the above-described embodiments, a single slide 18 includes multiple tissue specimens, but the present disclosure is not limited to this. The technology of the present disclosure can also be applied to a single slide 18 that includes one tissue specimen. In this case, the generation unit 51 does not need to generate the region image 71.

[0118] The subject S is not limited to a rat, but may be a mouse, guinea pig, gizzard shad, hamster, ferret, rabbit, dog, cat, monkey, etc. The subject S may also be a human.

[0119] Candidate substances such as target candidate substance 27T are not limited to drugs, but may be other chemical substances such as pesticides or radioactive materials.

[0120] The evaluation support device 10 may be a personal computer installed in a pharmaceutical facility as shown in FIG. 1, or may be a server computer installed in a data center independent of the pharmaceutical facility.

[0121] When the evaluation support device 10 is configured as a server computer, the specimen image group 60 is transmitted from a personal computer installed in each pharmaceutical facility to the server computer via a network such as the Internet. The server computer distributes various screens, such as the image list display screen 120, to the personal computer in the form of screen data for web distribution created using a markup language such as XML (Extensible Markup Language). The personal computer reproduces the screen to be displayed on the web browser based on the screen data and displays it on a display. Note that other data description languages, such as JSON (Javascript (registered trademark) Object Notation), may be used instead of XML.

[0122] The evaluation support device 10 according to the technique of the present disclosure can be widely used across all stages of pharmaceutical manufacturing, from the earliest stage of setting drug discovery targets to the final stage of clinical trials.

[0123] The hardware configuration of the computer constituting the evaluation support device 10 according to the technology of the present disclosure can be modified in various ways. For example, the evaluation support device 10 can be configured with multiple computers separated as hardware in order to improve processing power and reliability. For example, the functions of the RW control unit 50, generation unit 51, and identification unit 52 and the functions of the extraction unit 53, determination unit 54, and display control unit 55 can be distributed and performed by two computers. In this case, the evaluation support device 10 is configured with two computers.

[0124] In this way, the hardware configuration of the computer of the evaluation support device 10 can be changed as appropriate depending on the required performance such as processing power, safety, reliability, etc. Furthermore, not only the hardware but also application programs such as the operating program 40 can be duplicated or stored in multiple storage devices in order to ensure safety and reliability.

[0125] In each of the above embodiments, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the RW control unit 50, the generation unit 51, the identification unit 52, the extraction unit 53, the determination unit 54, and the display control unit 55. The various processors include the CPU 32, which is a general-purpose processor that executes software (operation program 40) and functions as various processing units, as described above, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0126] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA).Furthermore, multiple processing units may be configured with a single processor.

[0127] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0128] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0129] From the above description, the technology described in the following supplementary paragraphs can be understood.

[0130] [Supplementary Item 1] An image analysis device comprising a processor, which acquires a target specimen image depicting a tissue specimen of a target subject subjected to an evaluation test of a target candidate substance, inputs the target specimen image to a feature extractor common to multiple types of organs and / or multiple types of tissues, and causes the feature extractor to output target features of the target specimen image, and calculates a degree of deviation of the target feature from a distribution of reference features obtained by inputting to the feature extractor multiple reference specimen images depicting organs and / or tissues of a reference subject belonging to a control group, the organs and / or tissues corresponding to the organs and / or tissues of the tissue specimen depicted in the target specimen image. [Supplementary Item 2] The image analysis device of Supplementary Item 1, in which the processor determines whether a morphological abnormality has occurred in the tissue specimen depicted in the target specimen image based on the degree of deviation. [Supplementary Item 3] The image analysis device of Supplementary Item 1 or Supplementary Item 2, in which the feature extractor utilizes at least a portion of a machine learning model that outputs an identification result of the organ and / or tissue of the tissue specimen depicted in the input specimen image. [Supplementary Item 4] The image analyzing device of any one of Supplementary Items 1 to 3, wherein the control group includes at least one of a first control group and a second control group used in an evaluation test different from the first control group. [Supplementary Item 5] The image analyzing device of Supplementary Item 4, wherein the control group consists only of reference subjects to which the target candidate substance was not administered in the evaluation test. [Supplementary Item 6] The image analyzing device of Supplementary Item 4, wherein the control group consists only of reference subjects to which a reference candidate substance corresponding to the target candidate substance was not administered in a previous evaluation test. [Supplementary Item 7] The image analyzing device of Supplementary Item 6, wherein the processor acquires identification information of organs and / or tissues of a tissue specimen shown in the target specimen image, and selects a distribution corresponding to the identification information from a plurality of distributions prepared for a plurality of types of organs and / or a plurality of types of tissue. [Supplementary Item 8] The image analysis device according to Supplementary Item 7, wherein the processor analyzes the target specimen image to identify organs and / or tissues of the tissue specimen shown in the target specimen image, and obtains the identification result as the identification information.[Supplementary Item 9] The image analysis device according to Supplementary Item 4, wherein the processor generates an integrated distribution by integrating a first distribution, which is the distribution related to the first control group, and a second distribution, which is the distribution related to the second control group, and calculates a deviation of the target feature from the integrated distribution. [Supplementary Item 10] The image analysis device according to Supplementary Item 9, wherein the first control group is made up of reference subjects to which the target candidate substance was not administered, and the second control group is made up of reference subjects to which a reference candidate substance corresponding to the target candidate substance was not administered in a past evaluation test. [Supplementary Item 11] The image analysis device according to any one of Supplementary Item 1 to Supplementary Item 10, wherein the image analysis device supports evaluation of toxicity of the target candidate substance based on the deviation.

[0131] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs, but also to storage media that non-temporarily store programs, and computer program products that include programs.

[0132] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0133] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0134] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An image analysis device comprising a processor, which acquires a target specimen image depicting a tissue specimen of a target subject subjected to an evaluation test of a target candidate substance, inputs the target specimen image to a feature extractor common to multiple types of organs and / or multiple types of tissues, causes the feature extractor to output target features of the target specimen image, and calculates the degree of deviation of the target feature from a distribution of reference features obtained by inputting into the feature extractor multiple reference specimen images depicting tissue specimens of organs and / or tissues of reference subjects belonging to a control group, the organs and / or tissues corresponding to the organs and / or tissues of the tissue specimen depicted in the target specimen image.

2. The image analysis device according to claim 1, wherein the processor determines whether or not a morphological abnormality has occurred in the tissue specimen depicted in the target specimen image based on the degree of deviation.

3. The image analysis device according to claim 1, wherein the feature extractor utilizes at least a portion of a machine learning model that outputs an identification result of the organs and / or tissues of a tissue specimen shown in an input specimen image.

4. The image analysis device according to claim 1, wherein the control group includes at least one of a first control group and a second control group used in an evaluation test different from the first control group.

5. An image analysis device according to claim 4, wherein the control group is composed of only the reference subjects to whom the candidate substance was not administered in the evaluation test.

6. An image analysis device according to claim 4, wherein the control group is composed of only the reference subjects who were not administered a reference candidate substance corresponding to the target candidate substance in a previous evaluation test.

7. The image analysis device of claim 6, wherein the processor acquires identification information of the organs and / or tissues of the tissue specimen shown in the target specimen image, and selects a distribution corresponding to the identification information from among a plurality of distributions prepared for each of a plurality of types of organs and / or a plurality of types of tissues.

8. The image analysis device according to claim 7, wherein the processor analyzes the target specimen image to identify the organs and / or tissues of the tissue specimen shown in the target specimen image, and obtains the identification results as the identification information.

9. The image analysis device described in claim 4, wherein the processor generates an integrated distribution by integrating a first distribution, which is the distribution related to the first control group, and a second distribution, which is the distribution related to the second control group, and calculates the degree of deviation of the target feature from the integrated distribution.

10. The image analysis device described in claim 9, wherein the first control group is composed of reference subjects to whom the target candidate substance was not administered, and the second control group is composed of reference subjects to whom a reference candidate substance corresponding to the target candidate substance was not administered in a past evaluation test.

11. The image analysis device according to claim 1, which is a device that supports evaluation of the toxicity of the target candidate substance based on the degree of deviation.

12. A method for operating an image analysis device, comprising: acquiring a target specimen image that depicts a tissue specimen of a target subject subjected to an evaluation test of a target candidate substance; inputting the target specimen image into a feature extractor common to multiple types of organs and / or multiple types of tissues and outputting target features of the target specimen image from the feature extractor; and inputting multiple reference specimen images that depict tissue specimens of organs and / or tissues of reference subjects belonging to a control group, which correspond to the organs and / or tissues of the tissue specimen depicted in the target specimen image, into the feature extractor, and calculating the degree of deviation of the target feature from the distribution of reference features obtained.

13. An operating program for an image analysis device that causes a computer to execute processes including: acquiring a target specimen image that depicts a tissue specimen of a target subject subjected to an evaluation test of a target candidate substance; inputting the target specimen image into a feature extractor common to multiple types of organs and / or multiple types of tissues and outputting target features of the target specimen image from the feature extractor; and inputting multiple reference specimen images that depict tissue specimens of organs and / or tissues of a reference subject belonging to a control group, which correspond to the organs and / or tissues of the tissue specimen depicted in the target specimen image, into the feature extractor, and calculating the degree of deviation of the target feature from the distribution of reference features obtained.

Images