Image processing device, method for operating image processing device, and program for operating image processing device

Abstract

This image processing device is provided with a processor that performs: a process for acquiring a specimen image obtained by capturing an image of a tissue specimen of a subject to be subjected to an evaluation test of a candidate substance of a drug; and performing image processing on the specimen image in accordance with the type of the origin organ of the tissue specimen and / or the type of the morphological abnormality estimated to occur in the tissue specimen.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing apparatus operating method, and an image processing apparatus operating procedure. Background Technology

[0002] In the pharmaceutical manufacturing process, the following experiments are conducted: candidate substances are administered to subjects such as rats, and their efficacy and toxicity are evaluated. In such evaluation experiments, images of tissue specimens (brain, liver, heart, etc.) collected from the subjects through necropsy are used. These specimen images are digitized into WSI (Whole Slide Image). They are then displayed on a computer monitor for viewing by pathologists and other users responsible for evaluating the candidate substances.

[0003] Previously, various techniques have been proposed to support user evaluation of candidate substances by performing image processing on specimen images. For example, Japanese Patent No. 6780045 discloses a technique that evaluates the quality of specimen images by computer and performs image processing such as contrast correction, color correction, and sharpening on specimen images with low quality. Furthermore, Japanese Patent No. 5996494 discloses a technique that, as the display time of the user's area of ​​interest in the specimen image increases, the resolution energy of the area of ​​interest is progressively increased for display, and when the highest resolution energy is reached, image processing is performed on the area of ​​interest to highlight specific staining. Summary of the Invention

[0004] The technical problem to be solved by the invention Organs that can be sources of tissue specimens include the brain, esophagus, stomach, large intestine, small intestine, liver, kidneys, spleen, pancreas, heart, testes or ovaries, lymph nodes, bone marrow, and many others. Furthermore, when evaluating candidate substances, users focus on areas in the tissue specimen inferred to have morphological abnormalities (lesions not observed in normal tissue specimens). However, morphological abnormalities can include many types such as hyperplasia, infiltration, stasis, cysts, inflammation, tumors, carcinogenesis, proliferation, hemorrhage, glycogen depletion, inclusion bodies, granular cytoplasm, and foamy cytoplasm.

[0005] In order to evaluate candidate substances with high precision using images of tissue specimens from various organs and tissue specimens containing various morphological abnormalities, appropriate image processing of the specimen images is required. However, the image processing described in Japanese Patent Nos. 6780045 and 5996494 is not always suitable. Therefore, the accuracy of candidate substance evaluation may be reduced.

[0006] One embodiment of the present invention provides an image processing apparatus, an image processing method, and an image processing procedure capable of suppressing the reduction in the evaluation accuracy of candidate substances.

[0007] means for solving technical problems The image processing apparatus of the present invention includes a processor that performs the following processing: acquiring a specimen image obtained by photographing a tissue specimen of a subject in an evaluation test for a candidate substance for a pharmaceutical agent; and performing image processing on the specimen image corresponding to the type of organ from which the tissue specimen originates and / or the type of morphological abnormality inferred to occur in the tissue specimen.

[0008] The preferred image processing is image processing that simulates the image quality changes caused by changes in the settings of the optical system in the photographic device.

[0009] Preferred image processing includes at least one of brightness adjustment processing, contrast adjustment processing, and magnification processing.

[0010] The preferred processor performs the following processing: image processing is only applied to regions inferred to have morphological abnormalities.

[0011] The preferred processor performs the following processing: determining the type of the source organ and / or the type of morphological abnormality of the tissue specimen.

[0012] The preferred processor performs the following processing: accepts instructions from the user to implement image processing.

[0013] The preferred processor performs the following processing: extracting regions from the specimen image that are inferred to have morphological abnormalities.

[0014] The preferred processor performs the following processing: extraction using a machine learning model.

[0015] The preferred processor performs the following processing: extracting features by comparing the feature quantities obtained by inputting the specimen image into the machine learning model with the reference feature quantities obtained by inputting the reference specimen image, which is taken as a normal tissue specimen, into the machine learning model.

[0016] The preferred processor performs the following processing: based on the similarity of morphological anomalies, it classifies regions inferred to have morphological anomalies into multiple groups; and performs the same image processing on a group-by-group basis.

[0017] The image processing apparatus of the present invention includes the following operating method: acquiring a specimen image obtained by photographing a tissue specimen of a subject in an evaluation test for a candidate substance for a drug; and performing image processing on the specimen image corresponding to the type of organ from which the tissue specimen originates and / or the type of morphological abnormality inferred to occur in the tissue specimen.

[0018] The operating procedure of the image processing apparatus of the present invention causes a computer to perform the following processing: acquiring a specimen image obtained by photographing a tissue specimen of a subject in an evaluation test for a candidate substance for a pharmaceutical agent; and performing image processing on the specimen image corresponding to the type of organ from which the tissue specimen originates and / or the type of morphological abnormality inferred to occur in the tissue specimen.

[0019] Invention Effects According to the technology of the present invention, an image processing apparatus, an image processing method, and an image processing procedure capable of suppressing the reduction in the evaluation accuracy of candidate substances can be provided. Attached Figure Description

[0020] Figure 1 It is a diagram showing the generation order of specimen images, specimen images, and evaluation support devices.

[0021] Figure 2 This is a graph showing the treatment group and the control group.

[0022] Figure 3 This is a block diagram representing the computer that constitutes the evaluation support device.

[0023] Figure 4 This is a block diagram representing the CPU processing unit of the evaluation support device.

[0024] Figure 5 This is a diagram illustrating the processing of the recognition unit.

[0025] Figure 6 This is a diagram showing the processing of the first decision unit.

[0026] Figure 7 This is a diagram showing the processing of the first decision unit.

[0027] Figure 8 This is a diagram representing the extracted model group.

[0028] Figure 9 It is a graph that represents a block image obtained by subdividing a region image.

[0029] Figure 10 It is a graph representing the state of extracting features from a block image through an extraction model.

[0030] Figure 11 This is a diagram representing the structure of the extraction model.

[0031] Figure 12 This is a diagram representing the processing during the learning phase of an autoencoder.

[0032] Figure 13 This is a diagram showing the composition of past control group and learning reference block images.

[0033] Figure 14 It is a graph representing the state of extracting reference features from a reference block image through an extraction model.

[0034] Figure 15 It represents a graph that plots reference features in the feature space and a graph from which reference information is extracted.

[0035] Figure 16 It is a graph representing the distance between the location of a feature quantity and the representative location of a reference feature quantity.

[0036] Figure 17 This is a diagram showing the processing and judgment results of the extraction unit.

[0037] Figure 18 This is a diagram showing the processing and judgment results of the extraction unit.

[0038] Figure 19 This is a diagram illustrating the processing of the second decision unit.

[0039] Figure 20 It is a graph that represents image processing information.

[0040] Figure 21 It is a diagram representing image processing.

[0041] Figure 22 It is a diagram representing image processing.

[0042] Figure 23 It is a diagram that represents a view of the images.

[0043] Figure 24 This is a diagram representing the first image displayed.

[0044] Figure 25 This is a diagram showing the state of a frame in the first image display that indicates the region where the morphological anomaly is inferred to have occurred.

[0045] Figure 26 This is a diagram representing the second image display.

[0046] Figure 27 This is a flowchart showing the processing sequence of the evaluation support device.

[0047] Figure 28 This is a flowchart showing the processing sequence of the evaluation support device.

[0048] Figure 29 This is a flowchart showing the processing sequence of the evaluation support device.

[0049] Figure 30 This is a flowchart showing the processing sequence of the evaluation support device.

[0050] Figure 31 This is a flowchart showing the processing sequence of the evaluation support device.

[0051] Figure 32 This is a diagram showing the first image display screen of the second embodiment.

[0052] Figure 33 This is a diagram showing the second image display screen of the second embodiment.

[0053] Figure 34 This is a graph representing the clustering process.

[0054] Figure 35 It is a diagram that represents the display format of each cluster, the cluster image, and the overlapping image.

[0055] Figure 36 This is a diagram showing the first image display screen of the third embodiment.

[0056] Figure 37 This is a diagram illustrating the image processing of the third embodiment. Detailed Implementation

[0057] [First Implementation] As an example, such as Figure 1 As shown, the evaluation support device 10 is used to evaluate candidate substances 27 of the drug (reference). Figure 2 The efficacy and toxicity of the drug are evaluated. The evaluation support device 10 is an example of an "image processing device" according to the technology of the present invention. Furthermore, the drug may be, for example, a biological drug such as an antibody drug with an antibody as the active ingredient, or a peptide drug or nucleic acid drug with peptides or nucleic acids as the active ingredients.

[0058] The evaluation support device 10 is, for example, a desktop personal computer equipped with a monitor 11 for displaying various screens and input devices 12 such as a keyboard, mouse, touch panel, and / or microphone for voice input. The evaluation support device 10 is, for example, installed in a pharmaceutical company developing a drug or an organization that undertakes drug development work from a pharmaceutical company, i.e., a contract research organization (CRO). The evaluation support device 10 is operated by a user U who participates in the development of the drug at the pharmaceutical company or contract research organization (hereinafter collectively referred to as a pharmaceutical facility). The user U is, for example, a pathologist responsible for evaluating candidate substance 27.

[0059] Multiple specimen images 15 are input into the evaluation support device 10. Specimen images 15 are used to evaluate the efficacy and toxicity of candidate substance 27. Specimen images 15 are generated, for example, in the following order: First, a subject S, such as a rat, prepared for the evaluation of candidate substance 27, is dissected, and tissue specimens are collected by sectioning the organs of the subject S. Tissue specimens include brain specimen BS, heart specimen HS, lung specimen LS, liver specimen LVS, kidney specimen KDS, spleen specimen SPS, adrenal gland specimen AGS, and pituitary gland specimen PGS, etc. Although in Figure 1 Not shown in the image, but in addition to these, the tissue specimens also include specimens of various organs such as the esophagus, stomach, large intestine, small intestine, pancreas, gallbladder, aorta, venous vein, lymph nodes, trachea, bronchi, diaphragm, pineal gland, testes or ovaries, and spinal cord.

[0060] After collecting tissue specimens, each tissue specimen is affixed to a glass slide 16 according to the standard operating procedures (SOPs) pre-defined for each pharmaceutical facility. The SOPs, for example, specify detailed arrangements of tissue specimens, such as affixing a heart specimen (HS) and a lung specimen (LS) side-by-side to the same glass slide 16. In this way, multiple tissue specimens are affixed to a single glass slide 16.

[0061] Next, the tissue specimen is stained, here using hematoxylin-eosin. Then, the stained tissue specimen is covered with a coverslip 17, completing the slide specimen 18. The slide specimen 18 is then placed on a photographic device 19 such as a digital optical microscope, and an image 15 is captured by the photographic device 19. In the specimen image 15 thus obtained, the entirety of each tissue specimen attached to the slide 16 is reflected. In other words, multiple tissue specimens are reflected in one specimen image 15. Specimen image 15 is called WSI (Whole Slide Image). The specimen image 15 is labeled with the subject ID (Identification Data) used to uniquely identify the subject S, the specimen image ID used to uniquely identify the specimen image 15, and the date and time of capture, etc. Furthermore, the tissue specimen is also called a tissue section. Staining can be based solely on hematoxylin, or on nucleoresin, etc.

[0062] As an example, such as Figure 2As shown, the subjects S were divided into a treatment group 25 and a control group 26. The treatment group 25 consisted of multiple subjects S to which candidate substance 27 was administered. The treatment group 25 was further divided into a high-dose group 25H, a medium-dose group 25M, and a low-dose group 25L based on the dosage of candidate substance 27. Thus, by dividing the treatment group 25 into these three groups, the effect of the dosage of candidate substance 27 on the subjects S can be confirmed. Furthermore, the division of the treatment group 25 is not limited to the three groups shown (high-dose group 25H, medium-dose group 25M, and low-dose group 25L). It could also be divided into two groups (high-dose group 25H and low-dose group 25L), or even four or more groups. Additionally, the treatment group 25 could be divided based on the duration of candidate substance 27 administration, or based on the frequency of candidate substance 27 administration.

[0063] In contrast to the treatment group 25, the control group 26 consisted of multiple subjects S that were not given candidate substance 27. The number of subjects S constituting the high-dose group 25H, medium-dose group 25M, and low-dose group 25L was the same as the number of subjects S constituting the control group 26, for example, about 5 to 10. The subjects S constituting the high-dose group 25H, medium-dose group 25M, and low-dose group 25L and the subjects S constituting the control group 26 were subjects S with the same attributes and placed in the same rearing environment. The same attributes refer to, for example, the same age and / or the same sex. Furthermore, the same attributes also include the same age composition ratio and / or the same sex composition ratio (e.g., 5 males and 5 females). The same rearing environment refers to, for example, the same feed, the same temperature and humidity of the rearing space, and / or the same width of the rearing space. The "same" in "same rearing environment" means not only completely identical, but also the same error in the sense of error to the extent that is generally permissible in the technical field to which the present invention pertains and does not depart from the technical spirit of the present invention.

[0064] Multiple specimen images 15 are obtained from one subject S, and thus, from each group, the number of specimen images 15 obtained is the product of the number obtained from one subject S and the number of subjects S. For example, if the number of specimen images 15 obtained from one subject S is 100 and the number of subjects S constituting each group is 10, then 100 × 10 = 1000 specimen images 15 are obtained from each group.

[0065] As an example, such as Figure 3As shown, in addition to the aforementioned display 11 and input device 12, the computer constituting the evaluation support device 10 also includes a storage device 30, a memory 31, a CPU (Central Processing Unit) 32, and a communication unit 33. These are interconnected via a bus 34.

[0066] Storage device 30 is either a hard disk drive built into the computer constituting the evaluation support device 10 or connected via cable or network. Alternatively, storage device 30 may be a disk array consisting of multiple hard disk drives connected together. Storage device 30 stores control programs such as operating systems, various application programs, and various data associated with these programs. Alternatively, solid-state drives (SSDs) may be used instead of hard disk drives.

[0067] Memory 31 is the working memory used by CPU 32 to perform processing. CPU 32 loads the program stored in storage device 30 into memory 31 and executes the processing according to the program. Thus, CPU 32 centrally controls the various parts of the computer. CPU 32 is an example of a "processor" according to the technology of this invention. In addition, memory 31 may be built into CPU 32. Communication unit 33 performs various information transmission control with external devices such as camera device 19.

[0068] As an example, such as Figure 4 As shown, the evaluation support device 10 stores a working program 40 in its storage device 30. The working program 40 is an application program used to enable the computer to function as the evaluation support device 10. That is, the working program 40 is an example of the "working program of an image processing apparatus" according to the technology of this invention. The storage device 30 also stores a recognition model 41, a first determination model 42, an extraction model group 43, extraction reference information 44, a second determination model 45, and image processing information 46, etc.

[0069] If the operating program 40 is started, the CPU 32 and memory 31 of the computer constituting the evaluation support device 10 cooperate to function as a read / write (RW) control unit 50, an identification unit 51, a first determination unit 52, an extraction unit 53, a second determination unit 54, an image processing unit 55, and a display control unit 56. In addition to these processing units 50 to 56, the CPU 32 also includes an instruction receiving unit that receives various operation instructions from the input device 12.

[0070] The RW control unit 50 controls the storage of various data in the storage device 30 and the reading of various data from the storage device 30. For example, the RW control unit 50 acquires a set of specimen images 60 from the imaging device 19 and stores the set of specimen images 60 in the storage device 30. The set of specimen images 60 is a collection of multiple specimen images 15 generated for evaluating candidate substance 27.

[0071] The image overview display screen 120 (reference) was accessed by user U via input device 12. Figure 23 When indicated by the display, the RW control unit 50 reads the specimen image group 60 from the storage device 30. The RW control unit 50 then outputs the read specimen image group 60 to the recognition unit 51.

[0072] The RW control unit 50 reads the recognition model 41 from the storage device 30 and outputs the read recognition model 41 to the recognition unit 51. Furthermore, the RW control unit 50 reads the first determination model 42 from the storage device 30 and outputs the read first determination model 42 to the first determination unit 52. Additionally, the RW control unit 50 reads the extraction model group 43 and extraction reference information 44 from the storage device 30 and outputs the read extraction model group 43 and extraction reference information 44 to the extraction unit 53.

[0073] The RW control unit 50 reads the second determination model 45 from the storage device 30 and outputs the read second determination model 45 to the second determination unit 54. Furthermore, the RW control unit 50 reads image processing information 46 from the storage device 30 and outputs the read image processing information 46 to the image processing unit 55.

[0074] The recognition unit 51 uses the recognition model 41 to identify multiple tissue specimens mapped onto a single specimen image 15. Then, it generates a region image 70 of the identified tissue specimen based on the specimen image 15 (see reference). Figure 5 The recognition unit 51 outputs the collection of multiple generated region images 70, i.e., the region image group 61, to the first determination unit 52 and the display control unit 56.

[0075] The first determination unit 52 uses the first determination model 42 to determine the type of organ in the tissue specimen reflected in the region image 70. Then, the region image 70 and the first determination result 75 of the organ type (see reference) are used to determine the type of organ. Figure 6 and Figure 7 The set of groups, namely the first determination result group 62, is output to the extraction unit 53, the image processing unit 55, and the display control unit 56.

[0076] The extraction unit 53 uses the extraction model group 43 and extraction reference information 44 to extract regions inferred to have morphological abnormalities (hereinafter referred to as morphological abnormality inference regions) from the tissue specimen projected onto the region image 70. Then, the extraction result 63 of the morphological abnormality inference regions is output to the second determination unit 54 and the display control unit 56. Furthermore, morphological abnormalities refer to lesions not observed in normal tissue specimens, such as hyperplasia, infiltration, stasis, cysts, inflammation, tumors, carcinogenesis, proliferation, hemorrhage, glycogen depletion, inclusion bodies, granular cytoplasm, foamy cytoplasm, etc.

[0077] The second determination unit 54 uses the second determination model 45 to determine the type of morphological abnormality that is reflected in the tissue specimen of the region image 70. Then, the second determination result 64 of the type of morphological abnormality is output to the image processing unit 55 and the display control unit 56.

[0078] The image processing unit 55 performs image processing on the region image 70 in accordance with the image processing information 46, the first determination result 75, and the second determination result 64. The image processing unit 55 outputs the image-processed region image 70 to the display control unit 56.

[0079] The display control unit 56 controls the display of various images on the monitor 11. Among the various images are an image overview display screen 120 that shows an overview of the display area image 70 and a first image display screen 125 that displays the images 70 of each area (see reference). Figure 24 and Figure 25 )wait.

[0080] As an example, such as Figure 5 As shown, the recognition unit 51 inputs the specimen image 15 into the recognition model 41. Then, the recognition model 41 identifies multiple tissue specimens mapped onto the specimen image 15. The recognition model 41 is, for example, a machine learning model such as a convolutional neural network. The recognition model 41 identifies each of the multiple tissue specimens mapped onto the specimen image 15 and outputs the position coordinates of the rectangular boxes (called bounding boxes) surrounding the tissue specimens as the recognition result. Based on the recognition result output by the recognition model 41, the recognition unit 51 cuts out the rectangular boxes from the specimen image 15, thereby generating a region image 70 for each identified tissue specimen. Similar to the specimen image 15, a region image ID is marked in the region image 70 for unique identification.

[0081] exist Figure 5 The example shown is a specimen image 15 obtained by photographing two kidney specimens (KDS), a spleen specimen (SPS), a sublingual gland specimen (SLGS), and a submandibular gland specimen (SMGS). Furthermore, as can be seen from the example of the sublingual gland specimen (SLGS) and the submandibular gland specimen (SMGS), tissue specimens from multiple organs can be mixed in the region image 70.

[0082] The bounding box surrounding the tissue specimen, which can be modified by user U as the recognition result of recognition model 41, can be configured to be used instead of recognition model 41. Alternatively, the tissue specimen reflected in specimen image 15 can be identified by template matching instead of recognition model 41. Or, the tissue specimen reflected in specimen image 15 can be identified by user U manually inputting the bounding box surrounding the tissue specimen, without using recognition model 41 or template matching.

[0083] As an example, such as Figure 6 and Figure 7 As shown, the first determination unit 52 inputs the region image 70 into the first determination model 42. Then, the first determination model 42 determines the type of organ of the tissue specimen reflected in the region image 70. The first determination model 42 is also a machine learning model, such as a convolutional neural network, similar to the recognition model 41. The first determination model 42 outputs a first determination result 75 of the type of organ of the tissue specimen reflected in the region image 70.

[0084] exist Figure 6 The example illustrates a scenario where a regional image 70 reflecting a kidney specimen's KDS is input into a first determination model 42, and a first determination result 75 for "kidney" is output from the first determination model 42. Furthermore, in... Figure 7 The example illustrates a case where a region image 70, which reflects the sublingual gland specimen SLGS and the submandibular gland specimen SMGS, is input into a first determination model 42, and a first determination result 75 for "sublingual gland and submandibular gland" is output from the first determination model 42.

[0085] It can be configured such that user U can correct the first determination result 75, or delegate the determination of the type of organ of the tissue specimen reflected in the region image 70 to user U. Furthermore, the first determination model 42 can also be replaced by template matching to determine the type of organ of the tissue specimen reflected in the region image 70.

[0086] As an example, such as Figure 8 As shown, the extraction model group 43 is a collection of extraction models 80 prepared for each organ. Specifically, the extraction model group 43 includes a brain specimen extraction model 80A used in the regional image 70 of the present brain specimen BS, a heart specimen extraction model 80B used in the regional image 70 of the present heart specimen HS, a liver specimen extraction model 80C used in the regional image 70 of the present liver specimen LVS, and a pituitary specimen extraction model 80D used in the regional image 70 of the present pituitary specimen PGS, etc. The extraction unit 53 distinguishes which extraction model 80 to use based on the first determination result 75. The extraction model 80 is an example of a "machine learning model" according to the technology of this invention.

[0087] As an example, such as Figure 9As shown, the extraction unit 53 uses known image recognition technology to identify the tissue specimen reflected in the region image 70 (in Figure 9 In the example, a liver specimen (LVS) is shown, and the identified tissue specimen is subdivided into multiple block images 85. Each block image 85 has a pre-set size that can be processed by an extraction model 80 (in this case, an extraction model 80C for the liver specimen). The extraction unit 53 labels the block images 85 with block image IDs. Furthermore, the extraction unit 53 associates information indicating which position of the cut-out region image 70 the block image 85 belongs to with the block image ID. Figure 9 In this context, block image 85 does not have an area that overlaps with other block images 85, but block image 85 may overlap with a portion of other block images 85.

[0088] As an example, such as Figure 10 As shown, the extraction unit 53 uses the extraction model 80 (in... Figure 10 In the liver specimen extraction model 80C, feature quantity 87 is extracted for each of the multiple block images 85 obtained by subdividing the region image 70. Therefore, the number of feature quantities 87 is the same as the number of block images 85.

[0089] As an example, such as Figure 11 As shown, the encoder section 91 of the autoencoder 90 is used to extract the model 80. In addition to the encoder section 91, the autoencoder 90 also has a decoder section 92. A block image 85 is input to the encoder section 91. The encoder section 91 converts the block image 85 into feature values ​​87. The encoder section 91 transmits the feature values ​​87 to the decoder section 92. The decoder section 92 generates a restored image 93 of the block image 85 based on the feature values ​​87.

[0090] As is well known, the encoder unit 91 has convolutional layers that perform convolutional processing using filters and pooling layers that perform pooling processing such as max pooling. The decoder unit 92 is the same. The encoder unit 91 extracts feature quantities 87 by repeatedly performing convolutional processing based on convolutional layers and pooling processing based on pooling layers on the input block image 85. The extracted feature quantities 87 represent the shape and texture features of the tissue specimen reflected in the block image 85.

[0091] Feature 87 is a set of multiple numerical values. That is, feature 87 is multidimensional data. The dimension of feature 87 may be, for example, 512, 1024, or 2048. Feature 87 and the reference feature 87R (described later) Figure 14 They have the same dimension and can be in the same feature space 101 (reference). Figure 15 Compare with (etc.).

[0092] As an example, such as Figure 12As shown, in the learning phase before the encoder unit 91 is used to extract the model 80, the autoencoder 90 learns from the learning reference block image 85RL. The autoencoder 90 outputs a learning restored image 93L as input to the learning reference block image 85RL. Based on these learning reference block images 85RL and the learning restored image 93L, a loss calculation using a loss function is performed on the autoencoder 90. Then, based on the result of the loss calculation, update settings are made for various coefficients of the autoencoder 90 (e.g., coefficients of filters in convolutional layers), and the autoencoder 90 is updated according to these update settings.

[0093] During the learning phase of the autoencoder 90, while changing the learning reference block image 85RL, the following series of processes are repeatedly performed: inputting the learning reference block image 85RL to the autoencoder 90, outputting the learning restored image 93L from the autoencoder 90, loss calculation, setting update, and updating the autoencoder 90. This series of processes is repeated until the restoration accuracy from the learning reference block image 85RL to the learning restored image 93L reaches a preset level. Thus, the encoder section 91 of the autoencoder 90, having achieved the preset restoration accuracy, is stored as the extraction model 80 in the storage device 30 of the evaluation support device 10. Alternatively, the learning can end after repeating the above series of processes a preset number of times, regardless of the restoration accuracy from the learning reference block image 85RL to the learning restored image 93L.

[0094] The learning of such an autoencoder 90 can be performed by the evaluation support device 10 or by a device different from the evaluation support device 10. In the latter case, the extracted model 80 is sent from another device to the evaluation support device 10, and the extracted model 80 is stored in the storage device 30 by the RW control unit 50.

[0095] As an example, such as Figure 13As shown, a learning reference block image 85RL is supplied from multiple reference block images 85R obtained by subdividing a reference region image 70R. The reference region image 70R is an image obtained by photographing a tissue specimen of a subject S from a past control group 26P. The past control group 26P consists of multiple subjects S from which candidate substance 27 was not administered in past evaluation trials. Therefore, the number of subjects S constituting the past control group 26P is much greater than the number of subjects S constituting the administration group 25 and the control group 26, for example, around several hundred to several thousand. The reference region images 70R are also obtained from multiple subjects S, similar to region images 70, thus obtaining a number of reference region images 70R from the past control group 26P equal to the number obtained from a single subject S multiplied by the number of subjects S. The tissue specimen of the subject S from the past control group 26P is an example of a "tissue specimen considered normal" according to the technology of this invention. In addition, not only the regional image 70 obtained from the tissue specimens of the subject S in the past control group 26P, but also the regional image 70 obtained from the tissue specimens judged to be normal by pathologists and other experts in the past drug administration group, which consists of multiple subjects S in the past evaluation test in which the candidate substance was introduced, can be used as a reference regional image 70R.

[0096] exist Figure 13 In the example shown, a reference region image 70R mapping a liver specimen LVS is illustrated. The encoder section 91 of an autoencoder 90, which learns using a learning reference block image 85RL based on the reference region image 70R mapping the liver specimen LVS, is applied to an extraction model 80C for liver specimens. Similarly, for example, the encoder section 91 of an autoencoder 90, which learns using a learning reference block image 85RL based on the reference region image 70R mapping a brain specimen BS, is applied to an extraction model 80A for brain specimens.

[0097] Next, the structure of the extracted reference information 44 will be explained. First, as an example, such as Figure 14 As shown, using extraction model 80, multiple reference feature quantities 87R are extracted from each of multiple reference block images 85R based on all multiple reference region images 70R.

[0098] As an example, Figure 15 The diagram 100 shown is plotted in the feature space 101. Figure 14 A graph of multiple reference feature quantities 87R extracted from the data. The extracted reference information 44 includes the coordinates (hereinafter referred to as representative position coordinates) 102 of the representative position of the reference feature quantity 87R, indicated by an × symbol, in the feature quantity space 101. The representative position is, for example, the center point or average point of the distribution 103 of the reference feature quantity 87R. Furthermore, the extracted reference information 44 also includes a decision threshold 104. Additionally, in... Figure 15For ease of explanation, the feature space 101 is assumed to be 2-dimensional with axes D1 and D2, but the actual feature space 101 is 512-dimensional as previously mentioned. (The rest of the text is incomplete and requires further context.) Figure 16 Similarly, for ease of explanation, the dimension of the feature space 101 is represented by 2 dimensions.

[0099] Similar to the learning process of the self-encoder 90, the representative position coordinates 102 for extracting reference information 44 can be derived by the evaluation support device 10 or by a device different from the evaluation support device 10. In the latter case, the representative position coordinates 102 are sent from another device to the evaluation support device 10, and the RW control unit 50 stores the representative position coordinates 102 in the storage device 30.

[0100] As an example, such as Figure 16 As shown, the extraction unit 53 calculates the distance D between the representative position of the reference feature quantity 87R, represented by the representative position coordinates 102 of the extraction reference information 44, and the position of the feature quantity 87 in the feature quantity space 101. The extraction unit 53 calculates the distance D for each of the multiple feature quantities 87 extracted from each of the multiple block images 85 obtained by subdividing a region image 70. The distance D is the Mahalanobis distance. The distance D represents the degree of deviation of the feature quantity 87 from the reference feature quantity 87R, and in other words, the degree of deviation of the tissue specimen reflected in the block image 85 from the tissue specimen considered normal. That is, it can be said that the larger the distance D, the more the tissue specimen reflected in the block image 85 deviates from the tissue specimen considered normal. Therefore, the larger the distance D, the higher the probability of morphological abnormalities occurring in the tissue specimen reflected in the block image 85.

[0101] Alternatively, the distance D can be calculated as the mean, median, or maximum of the Euclidean distances between the positions of the k nearest neighbors of the distribution 103 of the reference feature quantity 87R and the position of feature quantity 87. Alternatively, it can be calculated as a value obtained by subtracting the cosine similarity between the vector representing the position of the reference feature quantity 87R and the vector representing the position of feature quantity 87 from 1.0, and used instead of the distance D. The cosine similarity takes values ​​between -1.0 and 1.0; a larger value indicates that the vectors are more similar in direction.

[0102] As an example, such as Figure 17 and Figure 18 As shown, the extraction unit 53 compares the calculated distance D with the magnitude of the judgment threshold 104. Figure 17 As shown, when the distance D is less than the determination threshold 104, the extraction unit 53 determines that no morphological abnormality has occurred in the tissue specimen projected onto the block image 85. The extraction unit 53 outputs a determination result 110 indicating that no morphological abnormality has occurred in the tissue specimen projected onto the block image 85.

[0103] On the other hand, such as Figure 18 As shown, when the distance D is greater than or equal to the determination threshold 104, the extraction unit 53 determines that a morphological abnormality has occurred in the tissue specimen projected onto the block image 85. The extraction unit 53 outputs a determination result 110 indicating that a morphological abnormality has occurred in the tissue specimen projected onto the block image 85. The region of the block image 85 projected as having a morphological abnormality in the tissue specimen is equivalent to the morphological abnormality inference region, i.e., the "region inferred to have a morphological abnormality" involved in the technology of the present invention. The extraction result 63 output from the extraction unit 53 is a set of block images 85 that output the determination result 110 indicating that a morphological abnormality has occurred in the tissue specimen. In addition, the determination threshold 104 can be universal in the drug administration group 25 and the control group 26, or it can be different in these groups. Furthermore, instead of the distance D, it is also possible to determine whether a morphological abnormality has occurred in the tissue specimen projected onto the block image 85 by comparing the aforementioned cosine similarity with the determination threshold 104.

[0104] like Figure 13 and Figure 14 As shown, the reference feature quantity 87R is a feature quantity extracted from the reference region image 70R obtained by subdividing the tissue specimen of the subject S from the past control group 26P. The subject S of the past control group 26P is the subject S without the introduction of candidate substance 27. Therefore, in the tissue specimen reflected in the reference region image 70R, at least no morphological abnormalities caused by the toxicity of candidate substance 27 will occur. Therefore, the representative position of the reference feature quantity 87R is regarded as the representative position of the feature quantity of the region image 70 obtained by photographing a normal tissue specimen. Therefore, as mentioned above, the distance D between the representative position of the reference feature quantity 87R and the position of the feature quantity 87 becomes an indicator of the degree to which the tissue specimen reflected in the block image 85 deviates from the normal tissue specimen. Therefore, as Figure 17 As shown, the extraction unit 53 considers block images 85 with a distance D less than the determination threshold 104 as showing tissue specimens that do not deviate from normal tissue specimens, and thus determines that no morphological abnormalities have occurred. On the other hand, as Figure 18 As shown, the extraction unit 53 considers the block image 85 with a distance D of more than the judgment threshold of 104 as a deviation between the reflected tissue specimen and the normal tissue specimen, and thus judges it as a morphological abnormality.

[0105] As an example, such as Figure 19As shown, the second determination unit 54 inputs the block image 85 into the second determination model 45. Then, the second determination model 45 determines the type of morphological abnormality inferred to occur in the tissue specimen mapped to the block image 85. The second determination model 45, like the recognition model 41, is, for example, a machine learning model such as a convolutional neural network. The second determination model 45 outputs a second determination result 64, inferred as the type of morphological abnormality in the tissue specimen mapped to the block image 85. Figure 19 In this context, inclusion bodies are exemplified as a type of morphological abnormality.

[0106] As an example, such as Figure 20 As shown, image processing information 46 summarizes the image processing information performed by image processing unit 55 on region image 70 or block image 85. In image processing information 46, image processing suitable for each organ type or morphological abnormality is registered according to the type of the organ or morphological abnormality. The image processing simulates image quality changes caused by settings changes in the optical system of the imaging device 19. Specifically, image processing includes brightness adjustment when the organ type is lymph node or bone marrow. Furthermore, image processing includes contrast adjustment and magnification processing when the morphological abnormality type is inclusion body, granular cytoplasm, or foamy cytoplasm.

[0107] The brightness adjustment process simulates the increase in image brightness when the aperture stop of the imaging device 19 is opened. Therefore, the brightness adjustment process can also be described as a brightness enhancement process. The contrast adjustment process simulates the increase in contrast of light and / or color of the image when the aperture stop and field stop of the imaging device 19 are reduced. Therefore, the contrast adjustment process can also be described as a contrast enhancement process. The magnification process simulates the change in image magnification when the magnification of the objective lens of the imaging device 19 is increased, which is called electronic zoom processing. The brightness adjusted in the brightness adjustment process and the contrast adjusted in the contrast adjustment process are preset to a predetermined level. Furthermore, the magnification in the magnification process is also preset to a predetermined magnification (e.g., 1.25x to 1.5x). In addition, the aperture stop, field stop, and objective lens are examples of the "optical system" involved in the technology of this invention.

[0108] When the organ type is a lymph node or bone marrow, it is recommended to increase the brightness in the imaging device 19 for observation. Therefore, a brightness adjustment process as described above is registered. Inclusion bodies, granular cytoplasm, and foamy cytoplasm are produced within cells and are therefore very small and discrete. Therefore, when the morphological abnormalities are inclusion bodies, granular cytoplasm, or foamy cytoplasm, a contrast adjustment process and a magnification process as described above are registered.

[0109] As an example, such as Figure 21 As shown, if the organ type in the first determination result 75 is an organ type suitable for image processing registered in the image processing information 46, the image processing unit 55 performs image processing on the region image 70 registered in the image processing information 46. Hereinafter, the image-processed region image 70 will be referred to as the processed region image 70A. Figure 21 In the example shown, the tissue specimen reflected in the region image 70 is a bone marrow specimen (BMS), and the type of organ in the first determination result 75 is bone marrow, and the brightness of the region image 70 is adjusted.

[0110] Furthermore, as an example, such as Figure 22 As shown, if the type of morphological anomaly in the second determination result 64 is a type of morphological anomaly suitable for image processing registered in the image processing information 46, the image processing unit 55 performs image processing on the block image 85 registered in the image processing information 46. Hereinafter, the block image 85 after image processing will be referred to as the processed block image 85A. Figure 22 In the example, the type of morphological abnormality in the second determination result 64 is inclusion body, and the block image 85 is subjected to contrast adjustment and magnification processing (the order of implementation is magnification processing and contrast adjustment processing).

[0111] The display control unit 56, for example, performs the following based on the display instruction given by the user U via the input device 12: Figure 23 The image overview display screen 120 shown is displayed under the control of the display 11. The image overview display screen 120 has a display area 121. In the display area 121, all region images 70 generated by the recognition unit 51 are displayed in one view. In the display area 121, the region images 70 are arranged sequentially from top to bottom and from left to right according to the region image ID. In the region images 70, the organ name based on the first determination result 75 is displayed along with the region image ID. A confirmation button 122 is provided at the bottom of the display area 121. When the confirmation button 122 is selected, the display control unit 56 turns off the display of the image overview display screen 120. In addition, the objects displayed on the region images 70 of the image overview display screen 120 can be limited, such as only one subject S or only the high-dose group 25H.

[0112] The images 70 within the display area 121 can be selected. For example, if the user U selects an image 70 and instructs the user to zoom in on the image 70, the display control unit 56 will, as an example,... Figure 24The first image display screen 125 shown is displayed under the control of the display 11. The first image display screen 125 is a magnified display of the region image 70 selected by the user U in the image overview display screen 120. When image processing has been performed on the region image 70 by the image processing unit 55, the processed region image 70A is magnified in the first image display screen 125, and information indicating that image processing has been performed on the region image 70 is displayed. Furthermore, when image processing has been performed on the region image 70 by the image processing unit 55, a return button 126 is provided in the first image display screen 125 to return the processed region image 70A to the region image 70 before image processing. When image processing has not been performed on the region image 70 by the image processing unit 55, the region image 70 is magnified in the first image display screen 125, and no information or return button 126 is displayed. Figure 24 In the example shown, the tissue specimen reflected in the region image 70 is a bone marrow specimen (BMS), and the brightness of the region image 70 is adjusted as part of the image processing.

[0113] An analysis button 127 and a confirmation button 128 are provided at the bottom of the first image display screen 125. When the analysis button 127 is selected and the analysis instruction for the region image 70 is given, the extraction unit 53 extracts the region where the morphological anomaly occurs from the region image 70 or the processed region image 70A displayed on the first image display screen 125. On the other hand, when the confirmation button 128 is selected, the display control unit 56 turns off the display of the first image display screen 125.

[0114] As an example, if the extraction result 63 from the extraction unit 53 is input, such as... Figure 25 As shown, the display control unit 56 overlays a box 130 representing the block image 85 extracted as the region where morphological abnormalities are inferred onto the tissue specimen of the region image 70 or the processed region image 70A. Figure 25 In the example shown, the tissue specimen reflected in region image 70 is a liver specimen LVS.

[0115] Box 130 can be selected. When user U selects box 130 and gives an instruction to zoom in on the block image 85 represented by box 130, the display control unit 56 performs... Figure 26The second image display screen 135 shown is displayed under the control of the display 11. The second image display screen 135 is a magnified display of the block image 85 represented by the box 130 selected by the user U. When image processing is performed on the block image 85 by the image processing unit 55, the processed block image 85A is magnified on the second image display screen 135, and information indicating that image processing was performed on the block image 85 is displayed. Furthermore, when image processing is performed on the block image 85 by the image processing unit 55, a return button 136 is provided on the second image display screen 135 to return the processed block image 85A to the block image 85 before image processing. When image processing is not performed on the block image 85 by the image processing unit 55, the block image 85 is magnified on the second image display screen 135, and no information or return button 136 is displayed. Figure 26 In the example shown, the type of morphological abnormality in the tissue specimen reflected in block image 85 is inferred to be inclusion bodies, and the block image 85 is subjected to contrast adjustment and magnification processing as image processing.

[0116] Below block image 85 or 85A, a reduced-size area image 70 with frames 130 superimposed on the tissue specimen is displayed. In this reduced-size area image 70, the display control unit 56 changes the color, etc., of the frame 130 corresponding to the currently displayed block image 85 or 85A, thereby distinguishing it from other frames 130. Furthermore, a confirmation button 137 is provided at the bottom of the second image display screen 135. When the confirmation button 137 is selected, the display control unit 56 turns off the display of the second image display screen 135.

[0117] Furthermore, if the magnified region image 70 displayed in the first image display screen 125 is an image obtained from the subject S belonging to the drug administration group 25, then for comparison, region images 70 of the same organ obtained from the subject S of the control group 26 can be displayed side by side. Alternatively, in the image overview display screen 120, region images 70 obtained from the subject S belonging to the drug administration group 25 and region images 70 obtained from the subject S belonging to the control group 26 can be displayed side by side.

[0118] Next, as an example, refer to Figures 27-31 The flowchart shown illustrates the function based on the above structure. First, if the working procedure 40 is initiated in the evaluation support device 10, then as follows... Figure 4 As shown, the CPU 32 of the evaluation support device 10 functions as the RW control unit 50, the recognition unit 51, the first determination unit 52, the extraction unit 53, the second determination unit 54, the image processing unit 55, and the display control unit 56.

[0119] A specimen image 15 of a tissue specimen of the subject S is captured by the imaging device 19. The specimen image 15 is transmitted from the imaging device 19 to the evaluation support device 10. (Example...) Figure 27 As shown, in the evaluation support device 10, the RW control unit 50 acquires a set of specimen images 15 from the imaging device 19, namely the specimen image group 60 (step ST100). Under the control of the RW control unit 50, the specimen image group 60 is stored in the storage device 30 (step ST110).

[0120] exist Figure 28 In the case where the user U sends a display instruction to the image overview display screen 120 via the input device 12 and the CPU 32 receives the display instruction ("Yes" in step ST200), the RW control unit 50 reads the specimen image group 60 specified in the display instruction from the storage device 30 (step ST210). The specimen image group 60 is then output from the RW control unit 50 to the recognition unit 51 and the display control unit 56.

[0121] In the identification unit 51, such as Figure 5 As shown, specimen image 15 is input into recognition model 41, thereby recognizing multiple tissue specimens reflected in specimen image 15, and outputting the recognition result from recognition model 41. Then, based on the recognition result, region images 70 of each tissue specimen are generated (step ST220). The set of region images 70, i.e., region image group 61, is output from recognition unit 51 to first determination unit 52.

[0122] In the first determination section 52, such as Figure 6 and Figure 7 As shown, the region image 70 is input to the first determination model 42. Thereby, the type of organ of the tissue specimen reflected in the region image 70 is determined (step ST230), and the first determination result 75 is output from the first determination model 42. The set of first determination results 75, i.e., the first determination result group 62, is output to the image processing unit 55 and the display control unit 56. Figure 23 As shown, under the control of the display control unit 56, the image overview display screen 120 of the area images 70 arranged on the display area 121 is displayed on the monitor 11 (step ST240).

[0123] exist Figure 29 In the case where the user U selects one of the region images 70 displayed on the image overview screen 120 and the CPU 32 receives a magnification display instruction for the region image 70 ("Yes" in step ST300), the image processing unit 55 refers to the image processing information 46 and the first determination result 75. Then, it searches whether the image processing corresponding to the type of organ of the tissue specimen reflected in the region image 70 that received the magnification display instruction is registered in the image processing information 46 (step ST310).

[0124] If the image processing corresponding to the organ of the tissue specimen reflected in the magnified display area image 70 is registered in the image processing information 46 ("Yes" in step ST310), then... Figure 21 As shown, in the image processing unit 55, image processing registered in the image processing information 46 is performed on the region image 70 (step ST320). On the other hand, if image processing corresponding to the organ of the tissue specimen reflected in the region image 70 that has received the magnification display instruction is not registered in the image processing information 46 ("No" in step ST310), image processing is not performed on the region image 70. Figure 24 As shown, under the control of the display control unit 56, the first image display screen 125, which enlarges the display of the area image 70 or the processed area image 70A, is displayed on the monitor 11 (step ST330).

[0125] exist Figure 30 In the case where user U selects analysis button 127 and CPU 32 receives analysis instruction for region image 70 ("Yes" in step ST400), as Figure 9 , Figure 10 as well as Figures 16-18 As shown, the region inferred from the occurrence of morphological abnormalities is extracted in the extraction unit 53 (step ST410).

[0126] Specifically, firstly, such as Figure 9 As shown, region image 70 is subdivided into block images 85. Next, as... Figure 10 As shown, the block image 85 is input into the extraction model 80, thereby outputting feature quantity 87 from the extraction model 80. Then, as... Figure 16 As shown, the distance D between the representative positions of feature quantity 87 and reference feature quantity 87R in feature quantity space 101 is calculated. Finally, as... Figure 17 and Figure 18 As shown, the distance D is compared with the decision threshold 104.

[0127] If the distance D is less than the determination threshold 104, a determination result 110 indicating that no morphological abnormality has occurred in the tissue specimen projected onto the block image 85 is output. Conversely, if the distance D is greater than or equal to the determination threshold 104, a determination result 110 indicating that morphological abnormality has occurred in the tissue specimen projected onto the block image 85 is output. The set of block images 85 for which this determination result 110 indicating morphological abnormality has occurred is output from the extraction unit 53 as an extraction result 63 of the morphological abnormality occurrence inference region to the second determination unit 54 and the display control unit 56. Figure 25As shown, under the control of the display control unit 56, the box 130 representing the block image 85 extracted as the region where the morphological abnormality occurred is superimposed on the tissue specimen of the region image 70 or the processed region image 70A (step ST420).

[0128] exist Figure 31 In the case where the user selects box 130 and the CPU 32 receives a magnified display instruction for the block image 85 represented by box 130 ("Yes" in step ST500), the second determination unit 54 determines the type of morphological abnormality that occurs in the tissue specimen reflected in the block image 85 that received the magnified display instruction (step ST510). Specifically, as follows Figure 19 As shown, the block image 85, which has received a magnification display instruction, is input into the second determination model 45, thereby outputting a second determination result 64 of the type of morphological abnormality from the second determination model 45. The second determination result 64 is output from the second determination unit 54 to the image processing unit 55 and the display control unit 56.

[0129] In the image processing unit 55, referring to the image processing information 46 and the second determination result 64, it searches whether the image processing corresponding to the type of morphological abnormality in the second determination result 64 is registered in the image processing information 46 (step ST520). If the image processing corresponding to the type of morphological abnormality in the second determination result 64 is registered in the image processing information 46 ("Yes" in step ST520), then... Figure 22 As shown, in the image processing unit 55, image processing registered in the image processing information 46 is performed on the block image 85 (step ST530). On the other hand, if image processing corresponding to the type of morphological abnormality in the second determination result 64 is not registered in the image processing information 46 ("No" in step ST520), image processing is not performed on the block image 85. Figure 26 As shown, under the control of the display control unit 56, the second image display screen 135, which enlarges the block image 85 or the processed block image 85A, is displayed on the monitor 11 (step ST540).

[0130] User U evaluates candidate substance 27 by observing region image 70 or processed region image 70A in detail in the first image display screen 125, or by observing block image 85 or processed block image 85A in detail in the second image display screen 135.

[0131] As explained above, the CPU 32 of the evaluation support device 10 includes an RW control unit 50 and an image processing unit 55. The RW control unit 50 acquires a specimen image 15 obtained by photographing a tissue specimen of the subject S for an evaluation test of the candidate substance 27 for the drug. The image processing unit 55 performs image processing on the region image 70 or the block image 85 corresponding to the type of source organ of the tissue specimen and / or the type of morphological abnormality inferred to occur in the tissue specimen. Therefore, the candidate substance 27 can be evaluated using the processed region image 70A after image processing appropriate to the type of organ or the processed block image 85A after image processing appropriate to the type of morphological abnormality. Therefore, the reduction in the evaluation accuracy of the candidate substance 27 can be suppressed.

[0132] like Figure 20 As shown, the image processing simulates the image quality changes caused by changes in the settings of the optical system in the photographic device 19 for specimen image 15. Therefore, when photographing specimen image 15 with photographic device 19, it is not necessary to strictly change the settings of the optical system for each organ type or type of morphological abnormality. This can significantly reduce the user U's working time when photographing specimen image 15.

[0133] Image processing includes at least one of brightness adjustment, contrast adjustment, and magnification. Therefore, candidate material 27 can be evaluated using a processed region image 70A or a processed block image 85A with appropriate brightness, contrast, or magnification. This further suppresses the reduction in evaluation accuracy of candidate material 27.

[0134] like Figure 22 As shown, the image processing unit 55 performs image processing only on the block image 85 extracted as the region where morphological anomalies are inferred. Therefore, compared to performing image processing on the entire region image 70, the load and time required for image processing can be reduced.

[0135] The first determination unit 52 determines the type of organ from which the tissue specimen originates. Furthermore, the second determination unit 54 determines the type of morphological abnormality. Therefore, compared to the user U visually determining the type of organ from which the tissue specimen originates and / or the type of morphological abnormality, and then inputting the determination result into the evaluation support device 10 via the input device 12, the user U's working time can be significantly reduced.

[0136] The extraction unit 53 extracts the region where the morphological anomaly occurs from the region image 70. Therefore, compared with the user U extracting the region where the morphological anomaly occurs by eye, the user U's working time can be greatly reduced.

[0137] The extraction unit 53 uses an extraction model 80, which is a machine learning model, to extract the region in which morphological anomalies are inferred. Recently, machine learning models have made remarkable progress, making it easy to prepare models with relatively high accuracy. Therefore, it is possible to extract the region inferred from which morphological anomalies occur simply and with high accuracy.

[0138] The extraction unit 53 extracts the morphological abnormality inference region by comparing the feature quantity 87 obtained by inputting the block image 85 into the extraction model 80 with the reference feature quantity 87R obtained by inputting the reference block image 85R of the reference region image 70R (which is taken as a normal tissue specimen) into the extraction model 80. Therefore, the morphological abnormality inference region can be extracted more simply and with higher accuracy.

[0139] [Second Implementation] In the first embodiment described above, the image processing unit 55 automatically performs image processing, but is not limited thereto. The image processing unit 55 may also perform image processing when the CPU 32 receives an instruction from the user U to perform image processing.

[0140] As an example, such as Figure 32 As shown, when the type of organ in the first determination result 75 is an organ type suitable for image processing registered in the image processing information 46, a button for instructing the implementation of image processing is provided on the first image display screen 150 of the second embodiment. Figure 32 In this example, the tissue specimen displayed in the region image 70 is a bone marrow specimen (BMS). In this case, a brightness adjustment button 151 is provided on the first image display screen 150 as a button to instruct the implementation of image processing. When the user U selects the brightness adjustment button 151, the CPU 32 receives an instruction to adjust the brightness. Upon receiving this instruction, the image processing unit 55 performs brightness adjustment processing on the region image 70.

[0141] Furthermore, as an example, such as Figure 33 As shown, when the type of morphological anomaly in the second determination result 64 is a type of morphological anomaly suitable for image processing registered in the image processing information 46, a button for instructing the implementation of image processing is provided on the second image display screen 155 in the second embodiment. Figure 33In this example, an inclusion body is inferred to be the type of morphological abnormality occurring in the tissue specimen displayed in block image 85. In this case, a contrast adjustment button 156 is provided on the second image display screen 155 as a button to instruct the implementation of image processing. When the user U selects the contrast adjustment button 156, the CPU 32 receives an instruction to adjust the contrast. Upon receiving this instruction, the image processing unit 55 performs contrast adjustment processing on the block image 85. Additionally, the image processing unit 55 automatically performs magnification processing on the block image 85.

[0142] Thus, in the second embodiment, instructions to perform image processing by user U are received. Therefore, it is possible to decide whether to perform image processing based on user U's judgment. If user U determines that image processing is not required based on actual observation of the area image 70 or the block image 85, unnecessary image processing is not required.

[0143] Multiple buttons, such as brightness adjustment buttons, contrast adjustment buttons, and zoom-in buttons, can be provided on the first and second image display screens to indicate the implementation of various image processing steps. These buttons allow the user U to determine the type of organ or the type of morphological abnormality, and to instruct the user U to perform image processing. For example, if the tissue specimen displayed in region image 70 is a bone marrow specimen (BMS), the user U observes region image 70 to determine that the organ is bone marrow. Then, by selecting the brightness adjustment button from among the multiple buttons, the image processing unit 55 performs brightness adjustment processing.

[0144] [Third Implementation] As an example, such as Figure 34 As shown, in the third embodiment, the extraction unit 53 performs clustering processing on the feature quantities 87 of the block image 85 extracted as the region to which morphological anomalies are inferred, defining the clusters to which each feature belongs. As the clustering processing, k-means, HDBSCAN (Hierarchical Density-Based Spatial Clustering), Gaussian Mixture Model (GMM), Probabilistic Latent Semantic Analysis (PLSA), Non-negative Matrix Factorization (NMF), or Fuzzy c-Means (FCM) methods can be used. Figure 34The example shown illustrates how feature quantity 87 is clustered into three clusters: cluster 1, cluster 2, and cluster 3. Additionally, as illustrated, feature quantity 87 may also contain features that do not belong to any cluster. Clusters 1 to 3 represent an example of a "group" as understood in this invention.

[0145] The extraction unit 53 generates clustering information 160. Clustering information 160 registers the cluster to which each block image 85 belongs, based on its block image ID. Block images 85 that do not belong to any cluster regarding feature value 87 are not registered in clustering information 160. The extraction unit 53 outputs clustering information 160 to the display control unit 56.

[0146] As an example, such as Figure 35 As shown, the display control unit 56 generates cluster images 165, 166, and 167 by processing the region image 70 based on clustering information 160. Cluster image 165 is the image corresponding to cluster 1. Cluster image 166 is the image corresponding to cluster 2. Cluster image 167 is the image corresponding to cluster 3.

[0147] The display control unit 56 generates cluster images 165-167 according to a pre-set display format 168 for each cluster. Display format 168 includes, for example, displaying cluster 1 in blue, cluster 2 in yellow-green, and cluster 3 in gray. The display control unit 56 generates cluster image 165 by filling the area of ​​block image 85 in the region image 70 with blue (as can be seen from the position information) containing the block image ID of cluster 1 registered in the clustering information 160. Similarly, the display control unit 56 generates cluster image 166 by filling the area of ​​block image 85 in the region image 70 with yellow-green, containing the block image ID of cluster 2 registered in the clustering information 160. Furthermore, the display control unit 56 generates cluster image 167 by filling the area of ​​block image 85 in the region image 70 with gray, containing the block image ID of cluster 3 registered in the clustering information 160. By changing the displayed colors in this way, cluster images 165-167 become images that can identify clusters 1-3. Alternatively, the display format 168 can be configured so that the user U can freely change the settings.

[0148] The display control unit 56 generates an overlapping image 169 formed by at least one of the overlapping region image 70 and cluster images 165-167. Figure 35 The example shown is an overlay image 169 formed by overlaying all cluster images 165-167 on the region image 70.

[0149] When the analysis button 127 is selected to indicate the analysis of the region image 70, and the extraction unit 53 extracts the region where the morphological abnormality is inferred, as an example, the display control unit 56 performs the following: Figure 36The first image display screen 175 shown is displayed on the control panel of the display 11. Overlapping image 169 and illustration 176 are displayed on the first image display screen 175. Display switching buttons 177, 178, and 179 are provided at the bottom of illustration 176. Display switching button 177 is used to select whether to overlay cluster image 165 onto area image 70. Display switching button 178 is used to select whether to overlay cluster image 166 onto area image 70. Display switching button 179 is used to select whether to overlay cluster image 167 onto area image 70. Therefore, for example, as shown, when all display switching buttons 177 to 179 are selected, the display control unit 56 displays overlapping image 169, which is formed by overlaying all cluster images 165 to 167 onto area image 70. Thus, the display control unit 56 overlays at least one of the multiple cluster images 165 to 167 onto area image 70. In addition, the first image display screen 175 initially displays a state where all the toggle buttons 177 to 179 are selected.

[0150] In the first image display screen 175, each cluster has a button for indicating the implementation of image processing. Specifically, the buttons for indicating the implementation of image processing are a brightness adjustment button 180, a contrast adjustment button 181, and a zoom-in button 182. The user U determines the type of morphological abnormality in the tissue specimen reflected in the block image 85 belonging to each cluster, and selects the button corresponding to the determined type of morphological abnormality from each of the buttons 180 to 182.

[0151] When user U selects brightness adjustment button 180, CPU 32 receives an instruction to adjust brightness. Upon receiving this instruction, image processing unit 55 uniformly adjusts the brightness of all block images 85 belonging to that cluster. Similarly, when user U selects contrast adjustment button 181, CPU 32 receives an instruction to adjust contrast. Upon receiving this instruction, image processing unit 55 uniformly adjusts the contrast of all block images 85 belonging to that cluster. Furthermore, when user U selects zoom in button 182, CPU 32 receives an instruction to zoom in. Upon receiving this instruction, image processing unit 55 uniformly zooms in on all block images 85 belonging to that cluster.

[0152] Thus, in the third embodiment, the extraction unit 53 classifies the regions where morphological anomalies occur into multiple groups (clusters) based on the similarity of the morphological anomalies. The image processing unit 55 performs the same image processing on a group-by-group basis. Therefore, compared to the second embodiment described above, where the implementation of image processing must be indicated for each block image 85, the user U's time spent indicating the implementation of image processing can be significantly reduced.

[0153] The first embodiment and the third embodiment described above can also be implemented in combination. That is, as an example, such as Figure 37 As shown, the second determination unit 54 determines the type of morphological abnormality inferred to occur in the tissue specimens reflected in the block images 85 belonging to each cluster, rather than the user U making the determination. At this time, for example, the type of morphological abnormality is determined for all block images 85 belonging to that cluster. Then, a majority vote of the determined types of morphological abnormalities is taken, and the type of morphological abnormality with the largest number of determinations is determined as the final type of morphological abnormality. The image processing unit 55 uniformly performs image processing corresponding to the finally determined type of morphological abnormality on all block images 85 belonging to that cluster. Figure 37 The example illustrates a case where a morphological abnormality inferred to occur in a tissue specimen belonging to block image 85 of cluster 1 is identified as an inclusion body, and contrast adjustment and magnification are performed as part of the image processing. This further reduces the user U's workload.

[0154] In addition, dimensionality reduction can be performed on the feature quantity 87 before clustering. Dimensionality reduction, for example, is the process of converting the 512-dimensional feature quantity 87 into a 2-dimensional feature quantity 87. As a dimensionality reduction process, principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), or UMAP (Uniform Manifold Approximation and Projection) can be used.

[0155] You can also Figure 34 The graph 100 showing the characteristic quantity 87 is displayed on the monitor 11, allowing the user U to manually divide the clusters.

[0156] Image processing that simulates the image quality change caused by a change in the settings of the optical system in the photographic apparatus 19, simulating the specimen image 15, is not limited to the illustrated brightness adjustment, contrast adjustment, and magnification processing. It can also be the processing of adjusting the resolution (resolution power) of the region image 70 or the block image 85. Specifically, the resolution adjustment processing is the processing of increasing resolution, for example, by implementing known super-resolution techniques on the region image 70 or the block image 85.

[0157] If the type of organ in the first determination result 75 is an organ type suitable for image processing registered in the image processing information 46, the image processing registered in the image processing information 46 is performed on the region image 70, but it is not limited to this. It is also possible to perform image processing on only the block image 85 extracted as the region where the morphological abnormality occurred, which is suitable for the type of organ registered in the image processing information 46, instead of performing it on the region image 70.

[0158] The determination of the source organ type of the tissue specimen reflected in region image 70 is not limited to the method using the first determination model 42. The organ type can also be determined using the method described below. That is, a machine learning model is used to pre-obtain representative feature quantities for each organ. Furthermore, the machine learning model from which the representative feature quantities of each organ are derived is used to derive representative feature quantities of the region image 70 for determining the organ type. Next, the distance between the representative feature quantities of each organ and the representative feature quantities of the region image 70 in the feature quantity space is calculated. Then, the organ with the shortest distance is determined as the source organ of the tissue specimen reflected in region image 70.

[0159] Similarly, the determination of the type of morphological abnormality inferred to occur in the tissue specimen projected in region image 70 is not limited to the method using the second determination model 45. The type of morphological abnormality can also be determined by the method described below. That is, a machine learning model is used to pre-obtain representative feature quantities for each morphological abnormality. Furthermore, the machine learning model from which the representative feature quantities of each morphological abnormality are derived is used to derive representative feature quantities for the block image 85 of the object from which the type of morphological abnormality is determined. Next, the distance between the representative feature quantities of each morphological abnormality and the representative feature quantities of the block image 85 in the feature quantity space is calculated. Then, the morphological abnormality with the shortest distance is determined as the morphological abnormality inferred to have occurred in the tissue specimen projected in block image 85.

[0160] In addition to the reference block image 85R obtained from photographing tissue specimens considered normal, the block image 85 obtained from photographing tissue specimens with morphological abnormalities can also be used as a learning reference block image 85RL. The block image 85 obtained from photographing tissue specimens with morphological abnormalities is, for example, obtained from a past drug administration group consisting of multiple subjects S who were injected with candidate substance 27 in past evaluation trials. Thus, the autoencoder 90 and the extraction model 80 can learn from tissue specimens with more diverse shape and texture features. As a result, the extraction model 80 can extract feature quantities 87 that further and better represent the shape and texture features of the tissue specimen.

[0161] Furthermore, the block image 85 obtained from photographing a tissue specimen with morphological abnormalities is not limited to images acquired from the subject S constituting the previously administered group as illustrated. Morphological abnormalities sometimes also occur in the subject S constituting the previously controlled group 26P. Therefore, as long as the block image 85 is obtained from photographing a tissue specimen with morphological abnormalities, it does not matter whether the subject S is the previously controlled group 26P or the previously administered group. Moreover, the block image 85 obtained from photographing a tissue specimen with morphological abnormalities can be an image acquired from a subject S whose morphological abnormalities were intentionally induced by applying various pressures. Furthermore, the block image 85 obtained from photographing a tissue specimen with morphological abnormalities can also be an artificially created image by processing a block image 85 obtained from photographing a normal tissue specimen.

[0162] The encoder section of the convolutional neural network that outputs a classification result based on the input of the block image 85 can be used in the extraction model 80 instead of the encoder section 91 of the autoencoder 90. The classification result is, for example, the result of identifying one type of morphological abnormality occurring in the tissue specimen reflected in the block image 85 from multiple types such as hyperplasia, infiltration, stasis, and inflammation.

[0163] Furthermore, the machine learning model used for extracting model 80 is not limited to the illustrated autoencoder 90 and convolutional neural networks. Generative Adversarial Networks (GANs) can also be used to extract model 80. Machine learning models without convolutional layers, such as Vision Transformers (ViTs), can also be used to extract model 80.

[0164] Contrastive learning can be performed, which involves learning in a way that brings features from the same image closer together in the feature space and widens the distance between features from different images in the feature space. Examples of contrastive learning methods include SimCLR (A Simple Framework for Contrastive Learning of Visual Representations). Furthermore, learning methods such as BYOL (Bootstrap Your Own Latent) can also be used without using the aforementioned different image pairs (also known as negative samples). Moreover, restrictions can be imposed on the distribution of the extracted features, such as a distribution on a unit sphere or a distribution mimicking a standard normal distribution.

[0165] Feature 87 is not limited to the features extracted by extraction model 80. It can also be the average, maximum, minimum, mode, or variance of the pixel values ​​in the block image 85.

[0166] In the above embodiments, the case where one slide specimen 18 contains multiple tissue specimens is illustrated, but the invention is not limited thereto. The technology of the present invention can also be applied to the case where one slide specimen 18 contains one tissue specimen.

[0167] The subject S is not limited to rats. It can also be mice, guinea pigs, gerbils, hamsters, ferrets, rabbits, dogs, cats, or monkeys. Furthermore, the subject S can also be a human.

[0168] like Figure 1 As shown, the evaluation support device 10 can be a personal computer installed in a pharmaceutical facility or a server computer installed in a data center independent of the pharmaceutical facility.

[0169] In the case where the evaluation support device 10 is composed of a server computer, specimen images 15 are sent from personal computers installed in various pharmaceutical facilities to the server computer via a network such as the Internet. The server computer transmits various screens, such as an image overview screen 120, to the personal computer in a format of web page transmission screen data generated by a markup language such as XML (Extensible Markup Language). The personal computer reproduces the screen displayed on the web browser based on the screen data and displays it on the monitor. Alternatively, other data description languages ​​such as JSON (Javascript Object Notation) can be used instead of XML.

[0170] The evaluation support device 10 involved in the technology of the present invention can be widely used in all stages of pharmaceutical manufacturing, from the initial stage of drug development target setting to the final stage of clinical trials.

[0171] The hardware structure of the computer constituting the evaluation support device 10 according to the technology of the present invention can be modified in various ways. For example, to improve processing power and reliability, the evaluation support device 10 can be composed of multiple computers that are separate as hardware. For example, two computers can be used to distribute the functions of the identification unit 51 and the first determination unit 52 and the extraction unit 53, the second determination unit 54 and the image processing unit 55. In this case, the evaluation support device 10 is composed of two computers.

[0172] In this way, the hardware structure of the computer used for evaluation support device 10 can be appropriately modified according to the performance requirements such as processing power, security, and reliability. Moreover, not limited to hardware, applications such as working program 40 can also be dual-stored or distributed across multiple storage devices for the purpose of ensuring security and reliability.

[0173] In the above embodiments, for example, as the hardware structure of the processing units that perform various processes, such as the RW control unit 50, the recognition unit 51, the first determination unit 52, the extraction unit 53, the second determination unit 54, the image processing unit 55, and the display control unit 56, various processors as shown below can be used. As described above, among the various processors, in addition to the general-purpose processor, i.e., CPU 32, which executes software (working program 40) to function as various processing units, there are also processors such as FPGA (Field Programmable Gate Array) whose circuit structure can be changed after manufacturing, i.e., Programmable Logic Device (PLD), and processors such as ASIC (Application Specific Integrated Circuit) with circuit structures specifically designed for performing specific processes, i.e., dedicated circuits.

[0174] A processing unit can consist of one of these various processors, or it can consist of a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Furthermore, multiple processing units can also be composed of a single processor.

[0175] As examples of a single processor comprising multiple processing units, firstly, there is the following approach: As exemplified by client and server computers, a single processor is constructed from a combination of one or more CPUs and software, functioning as multiple processing units. Secondly, there is the following approach: As exemplified by System-on-Chip (SoC), a processor that implements the overall system functionality including multiple processing units is implemented using a single integrated circuit (IC) chip. Thus, for each processing unit, the hardware structure utilizes one or more of the aforementioned processors.

[0176] Furthermore, more specifically, the hardware structure of these various processors can utilize circuits composed of combined semiconductor elements and other circuitry.

[0177] Based on the above records, one can master the techniques described in the following notes.

[0178] [Note 1] An image processing apparatus comprising a processor, The processor performs the following processing: Images of specimens obtained by photographing tissue specimens from subjects in evaluation tests for candidate substances used in pharmaceutical preparations; and The specimen image is subjected to image processing corresponding to the type of organ from which the tissue specimen originates and / or the inferred type of morphological abnormality occurring in the tissue specimen.

[0179] [Note 2] According to the image processing apparatus described in Appendix 1, wherein, The image processing simulates the image quality changes caused by changes in the settings of the optical system in the photographic device.

[0180] [Note 3] According to the image processing apparatus described in Appendix 2, wherein, The image processing includes at least one of brightness adjustment, contrast adjustment, and magnification.

[0181] [Note 4] The image processing apparatus according to any one of appendices 1 to 3, wherein, The processor performs the following processing: The image processing is performed only on regions inferred to have experienced the morphological anomaly.

[0182] [Note 5] The image processing apparatus according to any one of appendices 1 to 4, wherein, The processor performs the following processing: Determine the type of organ from which the tissue specimen originated and / or the type of morphological abnormality.

[0183] [Note 6] The image processing apparatus according to any one of appendices 1 to 5, wherein, The processor performs the following processing: Accept instructions from the user to perform the image processing.

[0184] [Note 7] The image processing apparatus according to any one of appendices 1 to 6, wherein, The processor performs the following processing: Extract the regions from the specimen images that are inferred to have experienced the morphological abnormality.

[0185] [Note 8] According to the image processing apparatus described in Appendix 7, wherein, The processor performs the following processing: The extraction is performed using a machine learning model.

[0186] [Note 9] According to the image processing apparatus described in Appendix 8, wherein, The processor performs the following processing: The extraction is performed by comparing the feature values ​​obtained by inputting the specimen image into the machine learning model with the reference feature values ​​obtained by inputting a reference specimen image (which is considered a normal tissue specimen) into the machine learning model.

[0187] [Note 10] The image processing apparatus according to any one of appendices 7 to 9, wherein, The processor performs the following processing: Based on the similarity of the morphological anomalies, regions inferred to have experienced the morphological anomalies are classified into multiple groups; and The same image processing is performed on the group as a unit.

[0188] The technology of the present invention can also be appropriately combined with the various embodiments and / or variations described above. Furthermore, it is not limited to the embodiments described above; various structures can be adopted as long as they do not depart from the spirit of the invention. Moreover, the technology of the present invention relates not only to programs, but also to storage media for non-transitory storage of programs and computer program articles containing programs.

[0189] The descriptions and illustrations above are detailed explanations of the parts related to the technology of this invention, and are merely one example of the technology of this invention. For example, the descriptions related to the above-described structure, function, effect, and effect are examples of the structure, function, effect, and effect of the parts related to the technology of this invention. Therefore, without departing from the technical spirit of this invention, unnecessary parts may be deleted from the descriptions and illustrations above, or new elements may be added or replaced. Furthermore, to avoid complexity and to facilitate understanding of the parts related to the technology of this invention, descriptions related to technical common sense that do not require special explanation in terms of implementing the technology of this invention have been omitted from the descriptions and illustrations above.

[0190] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, the same approach applies to situations where more than three items are connected by "and / or".

[0191] All documents, patent applications and technical standards described herein are incorporated herein by reference to the same extent that each document, patent application and technical standard is specifically and separately described and incorporated herein by reference.

Claims

1. An image processing apparatus comprising a processor, The processor performs the following processing: Images of specimens obtained by photographing tissue specimens from subjects in evaluation tests for candidate substances used in pharmaceutical preparations; and The specimen image is subjected to image processing corresponding to the type of organ from which the tissue specimen originates and / or the inferred type of morphological abnormality occurring in the tissue specimen.

2. The image processing apparatus according to claim 1, wherein, The image processing simulates the image quality changes caused by changes in the settings of the optical system in the photographic device.

3. The image processing apparatus according to claim 2, wherein, The image processing includes at least one of brightness adjustment, contrast adjustment, and magnification.

4. The image processing apparatus according to claim 1, wherein, The processor performs the following processing: The image processing is performed only on regions inferred to have the morphological abnormality.

5. The image processing apparatus according to claim 1, wherein, The processor performs the following processing: Determine the type of organ from which the tissue specimen originated and / or the type of morphological abnormality.

6. The image processing apparatus according to claim 1, wherein, The processor performs the following processing: Accept instructions from the user to perform the image processing.

7. The image processing apparatus according to claim 1, wherein, The processor performs the following processing: Extract the regions from the specimen images that are inferred to have the morphological abnormality.

8. The image processing apparatus according to claim 7, wherein, The processor performs the following processing: The extraction is performed using a machine learning model.

9. The image processing apparatus according to claim 8, wherein, The processor performs the following processing: The extraction is performed by comparing the feature values ​​obtained by inputting the specimen image into the machine learning model with the reference feature values ​​obtained by inputting a reference specimen image (which is considered a normal tissue specimen) into the machine learning model.

10. The image processing apparatus according to claim 7, wherein, The processor performs the following processing: Based on the similarity of the morphological anomalies, regions inferred to have experienced the morphological anomalies are classified into multiple groups; and The same image processing is performed on the group as a unit.

11. A method of operating an image processing apparatus, comprising: Images of specimens obtained by photographing tissue specimens of subjects in evaluation tests for candidate substances used in pharmaceutical preparations; as well as The specimen image is subjected to image processing corresponding to the type of organ from which the tissue specimen originates and / or the inferred type of morphological abnormality occurring in the tissue specimen.

12. An operating procedure for an image processing apparatus, said operating procedure causing a computer to perform a process comprising the following steps: Images of specimens obtained by photographing tissue specimens from subjects in evaluation tests for candidate substances used in pharmaceutical preparations; and The specimen image is subjected to image processing corresponding to the type of organ from which the tissue specimen originates and / or the inferred type of morphological abnormality occurring in the tissue specimen.

Images