DIAGNOSIS SUPPORT DEVICE, OPERATION METHOD OF DIAGNOSIS SUPPORT DEVICE, OPERATION PROGRAM OF DIAGNOSIS SUPPORT DEVICE, AND DEMENTIA DIAGNOSIS SUPPORT METHOD

Abstract

This diagnosis assistance device comprises: a processor; and a memory connect to, or built into, the processor. The processor acquires a target image, which is a medical image for diagnosing a disease; extracts a plurality of anatomical areas of an organ from the target image; inputs images of the plurality of anatomical areas into a plurality of feature quantity derivation models prepared for each of the plurality of anatomical areas; outputs a plurality of feature quantities for each of the plurality of anatomical areas from the feature quantity derivation models; searches for a similar image to the target image using the plurality of feature quantities; and outputs a similar image.

Description

[Technical field]

[0001] The technology disclosed herein relates to a diagnostic support device, an operating method for the diagnostic support device, an operating program for the diagnostic support device, and a dementia diagnostic support method. [Background technology]

[0002] In diagnosing diseases, such as dementia typified by Alzheimer's disease, doctors refer to medical images such as head MRI (Magnetic Resonance Imaging) images. Doctors obtain findings of dementia by observing, for example, the degree of atrophy of the hippocampus, parahippocampal gyrus, amygdala, etc., the degree of vascular disorder in white matter, and the presence or absence of decreased blood flow metabolism in the frontal lobe, temporal lobe, and occipital lobe. Doctors also search for similar images similar to the head MRI image of the diagnosis target, and derive findings of dementia taking the searched similar images into consideration.

[0003] In the search for similar images, a process is performed to calculate a similarity evaluation value between a target image for which similar images are to be searched and a candidate image for similar images. The similarity evaluation value is, for example, the distance between a multidimensional vector whose elements are multiple feature quantities of the target image and a multidimensional vector whose elements are multiple feature quantities of the candidate image. In this case, if the similarity between the target image and the candidate image is high, the distance between the two multidimensional vectors will be short, and the similarity evaluation value will be small.

[0004] Conventionally, features such as the average, maximum, and minimum pixel values, or the circularity of objects in an image have been used. However, these statistically obtained features alone were not always sufficient to search for similar images in a convincing way. <Muhammad Owais, etc., Effective Diagnosis and Treatment through Content-Based Medical Image Retrieval (CBMIR) by Using Artificial Intelligence, Journal of Clinical Medicine, 2019.> As described in the above, a target image is input to a machine learning model, and a similarity evaluation value is calculated using the feature amount output from the machine learning model. Summary of the Invention [Problem to be solved by the invention]

[0005] As mentioned above, an organ such as the brain is a collection of many anatomical regions, such as the hippocampus, amygdala, etc. In order to obtain findings of disease, it is necessary to thoroughly examine the anatomical regions from various viewpoints. <Muhammad Owais, etc., Effective Diagnosis and Treatment through Content-Based Medical Image Retrieval (CBMIR) by Using Artificial Intelligence, Journal of Clinical Medicine, 2019.> In this case, the similarity evaluation value is calculated without taking into account the anatomical region. <Muhammad Owais, etc., Effective Diagnosis and Treatment through Content-Based Medical Image Retrieval (CBMIR) by Using Artificial Intelligence, Journal of Clinical Medicine, 2019.> However, there was a limit to the accuracy of similar images.

[0006] One embodiment of the technique of the present disclosure provides a diagnostic support device capable of searching for more accurate similar images, an operating method of the diagnostic support device, an operating program of the diagnostic support device, and a dementia diagnostic support method. [Means for solving the problem]

[0007] The diagnostic support device disclosed herein includes a processor and a memory connected to or built into the processor, and the processor acquires a target image which is a medical image of a subject for diagnosing a disease, extracts multiple anatomical regions of an organ from the target image, inputs the images of the multiple anatomical regions to multiple feature derivation models prepared for each of the multiple anatomical regions, outputs multiple features for each of the multiple anatomical regions from the feature derivation model, searches for images similar to the target image using the multiple features, and outputs the similar images.

[0008] It is preferable that the processor also extracts multiple anatomical regions from the candidate images of similar images, inputs the images of the multiple anatomical regions of the candidate images into multiple feature derivation models, outputs multiple features for each of the multiple anatomical regions from the feature derivation models, calculates a similarity evaluation value between the target image and the candidate images from the multiple features of the target image and the multiple features of the candidate images, and selects a similar image from the candidate images based on the similarity evaluation value.

[0009] When calculating the similarity evaluation value, the processor preferably weights the plurality of feature amounts of the target image and the plurality of feature amounts of the candidate images according to the anatomical area.

[0010] The feature derivation model preferably includes at least one of an autoencoder, a single-task convolutional neural network for class discrimination, and a multi-task convolutional neural network for class discrimination.

[0011] The processor preferably inputs an image of one anatomical region to a plurality of different feature derivation models and causes each of the plurality of feature derivation models to output a feature.

[0012] When calculating the similarity evaluation value, the processor preferably weights the plurality of feature amounts of the target image and the plurality of feature amounts of the candidate images according to a plurality of different feature amount derivation models.

[0013] The processor preferably performs a normalization process to align the acquired target image with a standard medical image prior to extraction of the anatomical region.

[0014] Preferably, the organ is the brain and the disease is dementia, in which case the plurality of anatomical regions preferably includes at least one of the hippocampus and the anterior temporal lobe.

[0015] A method of operating the diagnostic support device disclosed herein includes obtaining a target image, which is a medical image of a target for diagnosing a disease, extracting multiple anatomical regions of an organ from the target image, inputting the images of the multiple anatomical regions into multiple feature derivation models prepared for each of the multiple anatomical regions and outputting multiple features for each of the multiple anatomical regions from the feature derivation models, searching for similar images to the target image using the multiple features, and outputting the similar images.

[0016] The operating program of the diagnostic support device disclosed herein causes a computer to execute processes including obtaining a target image, which is a medical image of a target for diagnosing a disease, extracting multiple anatomical regions of an organ from the target image, inputting the images of the multiple anatomical regions into multiple feature derivation models prepared for each of the multiple anatomical regions and outputting multiple features for each of the multiple anatomical regions from the feature derivation models, searching for images similar to the target image using the multiple features, and outputting the similar images.

[0017] The dementia diagnosis assistance method disclosed herein involves a computer having a processor and a memory connected to or built into the processor, acquiring a target image which is a medical image showing the brain of a subject to be diagnosed with dementia, extracting multiple anatomical regions of the brain from the target image, inputting the images of the multiple anatomical regions to multiple feature derivation models prepared for each of the multiple anatomical regions, and outputting multiple features for each of the multiple anatomical regions from the feature derivation model, searching for images similar to the target image using the multiple features, and outputting the similar images. Effect of the Invention

[0018] According to the technology of the present disclosure, it is possible to provide a diagnostic support device capable of searching for more accurate similar images, an operating method of a diagnostic support device, an operating program of a diagnostic support device, and a dementia diagnostic support method. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 illustrates a medical system. [Diagram 2] FIG. 2 is a diagram showing information exchanged between a client terminal and a PACS server. [Diagram 3] FIG. 2 is a diagram showing information exchanged between a client terminal and a PACS server. [Figure 4] FIG. 2 is a block diagram showing a computer that constitutes a PACS server. [Diagram 5] FIG. 2 is a block diagram showing a processing unit of a CPU of a PACS server. [Figure 6] FIG. 2 is a block diagram showing details of a preprocessing unit. [Figure 7] FIG. 13 is a diagram illustrating the process of a normalization unit. [Figure 8] FIG. 13 is a diagram illustrating the process of an extraction unit. [Figure 9] FIG. 13 is a diagram illustrating a process performed by a feature amount derivation unit. [Figure 10] FIG. 4 is a block diagram showing details of a search processing unit. [Figure 11] FIG. 13 is a diagram illustrating a process performed by a similarity evaluation value calculation unit. [Figure 12] FIG. 2 is a diagram showing a first display screen. [Figure 13] FIG. 13 is a diagram showing a second display screen. [Figure 14] FIG. 1 is a diagram illustrating the configuration of an autoencoder and the establishment of a feature derivation model. [Figure 15] FIG. 11 is an explanatory diagram of a convolution process. [Figure 16] FIG. 4 is a diagram showing a configuration of calculation data. [Figure 17] FIG. 11 is an explanatory diagram of a pooling process. [Figure 18] FIG. 1 is a diagram showing an overview of the processing in the learning phase of an autoencoder. [Figure 19] 13 is a flowchart showing a processing procedure of a PACS server. [Figure 20] 13 is a flowchart showing a processing procedure of a PACS server. [Figure 21] 13 is a flowchart showing a processing procedure of a PACS server. [Figure 22] FIG. 13 is a diagram showing an example of weighting coefficients set for each anatomical region. [Diagram 23] FIG. 11 is a diagram showing a process of a similarity evaluation value calculation unit in the second embodiment. [Figure 24] FIG. 1 is a diagram showing the configuration of a single-task convolutional neural network for classifying data and the establishment of a feature extraction model. [Diagram 25] FIG. 1 is a diagram showing an overview of processing in the learning phase of a single-task convolutional neural network for classifying classes. [Figure 26] FIG. 1 is a diagram showing the configuration of a convolutional neural network for multitask class discrimination and the establishment of a feature derivation model. [Figure 27] FIG. 1 is a diagram showing an overview of the processing in the learning phase of a multitask convolutional neural network for classification. [Figure 28] FIG. 13 is a diagram illustrating a process performed by a feature amount derivation unit according to the fifth embodiment. [Figure 29]FIG. 13 is a diagram illustrating an example of weighting coefficients set for each type of feature amount derivation model. [Diagram 30] This figure shows the configuration of an autoencoder and a single-task convolutional neural network for classifying, and the formation of a feature extraction model. [Diagram 31] FIG. 2 is a diagram illustrating a detailed configuration of an output unit. [Diagram 32] FIG. 1 is a diagram showing an overview of the processing in the learning phase of an autoencoder and a single-task convolutional neural network for classification. [Diagram 33] 13 is a graph showing the evolution of weights given to autoencoder losses. [Diagram 34] FIG. 23 is a schematic diagram of a process of a search processing unit according to the seventh embodiment. [Diagram 35] FIG. 13 is a diagram showing an aspect of deriving findings of dementia of a patient in a target image based on a group of aggregate features of the target image. [Diagram 36] 36 is a table showing a performance comparison between a method for predicting the progression of dementia described in a conventional document and the method for predicting the progression of dementia illustrated in FIG. 35 . DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] [First embodiment] 1, the medical system 2 includes an MRI apparatus 10, a PACS (Picture Archiving and Communication System) server 11, and a client terminal 12. The MRI apparatus 10, the PACS server 11, and the client terminal 12 are connected to a LAN (Local Area Network) 13 installed in a medical facility, and are capable of communicating with each other via the LAN 13. The PACS server 11 is an example of a "diagnosis support device" according to the technology of the present disclosure.

[0021] The MRI device 10 captures an image of the head of patient P and outputs a head MRI image 15. The head MRI image 15 is voxel data representing the three-dimensional shape of the head of patient P. FIG. 1 shows a head MRI image 15S of a sagittal section. The MRI device 10 transmits the head MRI image 15 to a PACS server 11. The PACS server 11 stores and manages the head MRI image 15 from the MRI device 10. The head MRI image 15 is an example of a "medical image" according to the technology of the present disclosure.

[0022] The client terminal 12 is, for example, a desktop personal computer, and includes a display 17 and an input device 18. The input device 18 is a keyboard, a mouse, a touch panel, a microphone, etc. Although only one MRI apparatus 10 and one client terminal 12 are illustrated in Fig. 1, there may be a plurality of MRI apparatuses 10 and a plurality of client terminals 12.

[0023] As an example, as shown in FIG. 2, a doctor operates an input device 18 to send a delivery request 20 for a head MRI image 15 of a patient P to be diagnosed (hereinafter referred to as a target image 15TI) to the PACS server 11. The delivery request 20 includes an image ID (Identification Data) of the target image 15TI, a patient ID of the patient P, a shooting date and time of the target image 15TI, etc. In response to the delivery request 20, the PACS server 11 searches for the target image 15TI and delivers it to the client terminal 12. The client terminal 12 receives the target image 15TI delivered from the PACS server 11, and displays the received target image 15TI on the display 17.

[0024] As an example, as shown in FIG. 3, the doctor operates the input device 18 to send a search request 23 for a similar image 15SI of the target image 15TI to the PACS server 11. The search request 23 includes the image ID of the target image 15TI, as in the distribution request 20. The PACS server 11 searches for the similar image 15SI and distributes it to the client terminal 12. The client terminal 12 receives the similar image 15SI distributed from the PACS server 11, and displays the received similar image 15SI on the display 17 so as to be comparable with the target image 15TI. The doctor observes the brain of the patient P shown in the target image 15TI, and diagnoses dementia for the patient P while also referring to the similar image 15SI as the case may be. The brain is an example of an "organ" according to the technology of the present disclosure, and dementia is an example of a "disease" according to the technology of the present disclosure.

[0025] 4, the computer constituting the PACS server 11 includes a storage 25, a memory 26, a CPU (Central Processing Unit) 27, and a communication unit 28. These are interconnected via a bus line 29. The CPU 27 is an example of a "processor" according to the technology of the present disclosure.

[0026] The storage 25 is a hard disk drive built into the computer constituting the PACS server 11 or connected via a cable or network. Alternatively, the storage 25 is a disk array in which multiple hard disk drives are connected in series. The storage 25 stores control programs such as an operating system, various application programs, and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0027] The memory 26 is a work memory for the CPU 27 to execute processing. The CPU 27 loads a program stored in the storage 25 into the memory 26 and executes processing according to the program. In this way, the CPU 27 comprehensively controls each part of the computer. The communication unit 28 controls the transmission of various information to and from external devices such as the client terminal 12. The memory 26 may be built into the CPU 27.

[0028] 5, an operating program 35 is stored in the storage 25 of the PACS server 11. The operating program 35 is an application program for causing a computer constituting the PACS server 11 to function as a diagnostic support device. In other words, the operating program 35 is an example of an "operating program for a diagnostic support device" according to the technology of the present disclosure.

[0029] When the operating program 35 is started, the CPU 27 of the computer constituting the PACS server 11, in cooperation with the memory 26, etc., functions as a read / write (hereinafter abbreviated as RW (Read Write)) control unit 40, a pre-processing unit 41, a request receiving unit 42, a search processing unit 43, and a distribution control unit 44.

[0030] The RW control unit 40 controls the storage of various data in the storage 25 and the reading of various data in the storage 25. For example, the RW control unit 40 receives a head MRI image 15 from the MRI device 10, and stores the received head MRI image 15 in the storage 25. Here, the target image 15TI is one of the multiple head MRI images 15 captured by the MRI device 10. Therefore, the RW control unit 40 acquires the target image 15TI by receiving the head MRI image 15 from the MRI device 10.

[0031] The pre-processing unit 41 performs various pre-processing operations, which will be described later, on the head MRI image 15 from the MRI apparatus 10. The pre-processing unit 41 outputs a feature set group 45 derived as a result of the pre-processing to the RW control unit 40. The RW control unit 40 stores the feature set group 45 in the storage 25 in association with the head MRI image 15. As a result, a plurality of pairs of the head MRI image 15 and the feature set group 45 are stored in the storage 25.

[0032] The request receiving unit 42 receives the distribution request 20 and the search request 23 transmitted from the client terminal 12. The request receiving unit 42 outputs the received distribution request 20 and search request 23 to the search processing unit 43.

[0033] The search processing unit 43 is responsible for searching for the target image 15TI and similar images 15SI. When a distribution request 20 is input from the request receiving unit 42, the search processing unit 43 outputs a command to the RW control unit 40 to search for and read out the target image 15TI corresponding to the distribution request 20 from the storage 25. In response to the command, the RW control unit 40 searches for and reads out the target image 15TI corresponding to the distribution request 20 from the storage 25, and outputs the read target image 15TI to the search processing unit 43. The search processing unit 43 outputs the target image 15TI to the distribution control unit 44.

[0034] When a search request 23 is input from the request receiving unit 42, the search processing unit 43 outputs a command to the RW control unit 40 to read all of the feature set groups 45 from the storage 25. In response to the command, the RW control unit 40 reads all of the feature set groups 45 from the storage 25 and outputs the read feature set groups 45 to the search processing unit 43. The search processing unit 43 searches for similar images 15SI using the feature set groups 45. The search processing unit 43 outputs the searched similar images 15SI to the delivery control unit 44.

[0035] The delivery control unit 44 delivers the target image 15TI to the client terminal 12 that issued the delivery request 20. The delivery control unit 44 also delivers the similar image 15SI to the client terminal 12 that issued the search request .

[0036] As an example, as shown in FIG. 6, the preprocessing unit 41 includes a normalization unit 50, an extraction unit 51, and a feature amount derivation unit 52.

[0037] In addition to head MRI image 15 and feature set group 45, storage 25 stores feature derivation model group 58 consisting of standard head MRI image 55, segmentation model 56, and multiple feature derivation models 57. RW control unit 40 reads standard head MRI image 55 from storage 25 and outputs read standard head MRI image 55 to normalization unit 50. RW control unit 40 also reads segmentation model 56 from storage 25 and outputs read segmentation model 56 to extraction unit 51. Furthermore, RW control unit 40 reads feature derivation model group 58 from storage 25 and outputs read feature derivation model group 58 to feature derivation unit 52.

[0038] The normalization unit 50 performs normalization processing to match the head MRI image 15 from the MRI apparatus 10 with the standard head MRI image 55, and converts the head MRI image 15 into a normalized head MRI image 60. The normalization unit 50 outputs the normalized head MRI image 60 to the extraction unit 51.

[0039] The standard head MRI image 55 is a head MRI image that shows a brain with a standard shape, size, and density (pixel value). The standard head MRI image 55 is, for example, an image generated by averaging head MRI images 15 of multiple healthy subjects, or an image generated by computer graphics. The standard head MRI image 55 is an example of a "standard medical image" according to the technology of the present disclosure.

[0040] The extraction unit 51 inputs the normalized head MRI image 60 to the segmentation model 56. The segmentation model 56 is a machine learning model that performs so-called semantic segmentation, in which a label representing each anatomical region of the brain, such as the hippocampus, amygdala, and frontal lobe, is assigned to each pixel of the brain captured in the normalized head MRI image 60. The extraction unit 51 extracts images of multiple anatomical regions of the brain (hereinafter referred to as anatomical region images) 61 from the normalized head MRI image 60 based on the labels assigned by the segmentation model 56. The extraction unit 51 outputs an anatomical region image group 62 consisting of multiple anatomical region images 61 for each of the multiple anatomical regions to the feature derivation unit 52.

[0041] One feature derivation model 57 is prepared for each of the multiple anatomical regions of the brain (see FIG. 9). The feature derivation unit 52 inputs an anatomical region image 61 to the corresponding feature derivation model 57. Then, the feature derivation model 57 outputs a feature set 63 consisting of multiple types of feature amounts Z (see FIG. 9). The feature set group 45 is composed of multiple feature sets 63 corresponding to the multiple anatomical regions. The feature derivation unit 52 outputs the feature set group 45 to the RW control unit 40. The RW control unit 40 stores the feature set group 45 in the storage 25.

[0042] 7 as an example, the normalization unit 50 performs a shape normalization process 65 and a density normalization process 66 on the head MRI image 15 as normalization processes. The shape normalization process 65 is, for example, a process of extracting landmarks serving as references for alignment from the head MRI image 15 and the standard head MRI image 55, and translating, rotating, and / or scaling the head MRI image 15 to match the standard head MRI image 55 so as to maximize the correlation between the landmarks of the head MRI image 15 and the landmarks of the standard head MRI image 55. The density normalization process 66 is, for example, a process of correcting the density histogram of the head MRI image 15 to match the density histogram of the standard head MRI image 55.

[0043] As an example, as shown in FIG. 8, the extraction unit 51 extracts, as the anatomical area images 61, an anatomical area image 61_1 of the hippocampus, an anatomical area image 61_2 of the parahippocampal gyrus, an anatomical area image 61_3 of the frontal lobe, an anatomical area image 61_4 of the anterior temporal lobe, an anatomical area image 61_5 of the occipital lobe, an anatomical area image 61_6 of the thalamus, an anatomical area image 61_7 of the hypothalamus, an anatomical area image 61_8 of the amygdala, an anatomical area image 61_9 of the pituitary gland, etc. In addition to these, the extraction unit 51 extracts anatomical area images 61 of each anatomical area such as the mammillary body, the corpus callosum, the fornix, and the lateral ventricle. The anatomical areas such as the hippocampus, the frontal lobe, the anterior temporal lobe, and the amygdala are a pair on the left and right. Although not shown in the figure, for such a pair of anatomical areas on the left and right, anatomical area images 61 of the left and right anatomical areas are extracted. For example, for the hippocampus, an anatomical area image 61_1 of the left hippocampus and an anatomical area image 61_1 of the right hippocampus are extracted. Among these anatomical areas, it is preferable to include at least one of the hippocampus and the anterior temporal lobe, and it is more preferable to include both the hippocampus and the anterior temporal lobe. The anterior temporal lobe means the anterior part of the temporal lobe. The extraction of the anatomical area by the extraction unit 51 using the segmentation model 56 is performed, for example, using the method described in the following document. <Patrick McClure, etc., Knowing What You Know in Brain Segmentation Using Bayesian Deep Neural Networks, Front. Neuroinform., 17 October 2019.>

[0044] 9, the feature derivation unit 52 inputs a hippocampal anatomical region image 61_1 to a hippocampal feature derivation model 57_1 and causes the hippocampal feature derivation model 57_1 to output a hippocampal feature set 63_1. The hippocampal feature set 63_1 is composed of a plurality of features Z1_1, Z2_1, . . . , ZN_1. N is the number of features, which may range from several tens to several hundreds of thousands, for example.

[0045] Similarly, the feature derivation unit 52 inputs the anatomical area image 61_2 of the parahippocampal gyrus to the feature derivation model 57_2 of the parahippocampal gyrus, inputs the anatomical area image 61_3 of the frontal lobe to the feature derivation model 57_3 of the frontal lobe, and inputs the anatomical area image 61_4 of the anterior temporal lobe to the feature derivation model 57_4 of the anterior temporal lobe. Then, the feature derivation model 57_2 of the parahippocampal gyrus outputs a feature set 63_2 of the parahippocampal gyrus, the feature derivation model 57_3 of the frontal lobe outputs a feature set 63_3 of the frontal lobe, and the feature derivation model 57_4 of the anterior temporal lobe outputs a feature set 63_4 of the anterior temporal lobe. The parahippocampal gyrus feature set 63_2 is composed of multiple features Z1_2, Z2_2, ..., ZN_2, the frontal lobe feature set 63_3 is composed of multiple features Z1_3, Z2_3, ..., ZN_3, and the anterior temporal lobe feature set 63_4 is composed of multiple features Z1_4, Z2_4, ..., ZN_4.

[0046] The feature derivation unit 52 also inputs the occipital lobe anatomical region image 61_5 to the occipital lobe feature derivation model 57_5, and inputs the thalamus anatomical region image 61_6 to the thalamus feature derivation model 57_6. Then, the occipital lobe feature derivation model 57_5 outputs the occipital lobe feature set 63_5, and the thalamus feature derivation model 57_6 outputs the thalamus feature set 63_6. The occipital lobe feature set 63_5 is composed of a plurality of features Z1_5, Z2_5, . . . , ZN_5, and the thalamus feature set 63_6 is composed of a plurality of features Z1_6, Z2_6, . . . , ZN_6. In this way, the plurality of anatomical region images 61 are input to the corresponding feature derivation models 57, and the feature set 63 for each anatomical region image 61 is output from each feature derivation model 57. The number of feature amounts Z may be the same for each anatomical region, such as N in the example, or may be different for each anatomical region.

[0047] The pre-processing unit 41 performs the above-described series of processes, i.e., the normalization process by the normalization unit 50, the extraction process of the anatomical region image 61 by the extraction unit 51, and the derivation process of the feature set group 45 by the feature derivation unit 52, on all of the head MRI images 15 from the MRI device 10. The head MRI images 15 from the MRI device 10 naturally include the target image 15TI and candidate images 15CI (see FIG. 10) of the similar image 15SI. Therefore, the pre-processing unit 41 performs normalization process to match the target image 15TI to the standard head MRI image 55, extracts the anatomical region image 61 from the target image 15TI, and derives the feature set group 45 for the target image 15TI (hereinafter referred to as the feature set group 45TI, see FIG. 10). In addition, the pre-processing unit 41 performs normalization processing to align the candidate image 15CI with the standard head MRI image 55, extracts an anatomical area image 61 from the candidate image 15CI, and derives a group of feature sets 45 of the candidate image 15CI (hereinafter referred to as the group of feature sets 45CI, see Figure 10).

[0048] As an example, as shown in FIG. 10, the search processing unit 43 includes a similarity evaluation value calculation unit 70 and a similar image selection unit 71.

[0049] The storage 25 stores the selection conditions 75. The RW control unit 40 reads the selection conditions 75 from the storage 25, and outputs the selection conditions 75 that have been read out to the similar image selection unit 71.

[0050] The similarity evaluation value calculation unit 70 receives all the feature set groups 45 from the RW control unit 40. The feature set group 45 includes a feature set group 45TI of the target image 15TI and a feature set group 45CI of the head MRI images 15 other than the target image 15TI, i.e., the candidate images 15CI of the similar images 15SI. The similarity evaluation value calculation unit 70 calculates a similarity evaluation value 76 between the target image 15TI and the candidate images 15CI from the feature set group 45TI of the target image 15TI and the feature set group 45CI of the candidate images 15CI. The similarity evaluation value calculation unit 70 calculates the similarity evaluation value 76 for each of the multiple candidate images 15CI. The similarity evaluation value calculation unit 70 outputs the calculated similarity evaluation value 76 to the similar image selection unit 71.

[0051] The similar image selection unit 71 selects a similar image 15SI from the candidate images 15CI based on the selection condition 75 and the similarity evaluation value 76. In this example, the selection condition 75 is "candidate images with the first to third similarities." Therefore, the similar image selection unit 71 selects the candidate images 15CI with the first to third similarities to the target image 15TI represented by the similarity evaluation value 76 as the similar images 15SI. In this embodiment, the candidate images 15CI with the first to third similarities to the target image 15TI are the candidate image 15CI with the smallest similarity evaluation value 76, the candidate image 15CI with the second smallest similarity evaluation value 76, and the candidate image 15CI with the third smallest similarity evaluation value 76.

[0052] As an example, as shown in FIG. 11, the similarity evaluation value calculation unit 70 calculates the distance between a multidimensional vector having each feature amount ZTIi of the feature amount set group 45TI of the target image 15TI as an element and a multidimensional vector having each feature amount ZCIi of the feature amount set group 45CI of the candidate image 15CI as an element as a similarity evaluation value 76. More specifically, as shown in the following formula (1) in which the similarity evaluation value 76 is represented as S, the similarity evaluation value calculation unit 70 calculates the square root of the sum of the squares of the absolute values ​​of the differences (ZTI1_1-ZCI1_1, ZTI2_2-ZCI2_2, etc.) between the feature amount ZTIi and the corresponding feature amount ZCIi, that is, the Euclidean distance, as the similarity evaluation value 76. In this case, the smaller the similarity evaluation value 76, the higher the similarity between the target image 15TI and the candidate image 15CI. Here, i is an integer from 1 to M. Moreover, M is the product of the number N of features Z and the number of anatomical areas from which the anatomical area images 61 are extracted, that is, the total number of features Z included in the feature set group 45TI or the feature set group 45CI.

[0053]

number

[0054] 12 shows an example of a first display screen 80 including a target image 15TI distributed from the PACS server 11 in response to a distribution request 20. The first display screen 80 is displayed on the display 17 of the client terminal 12. The first display screen 80 displays the target image 15TI. The target images 15TI are a target image 15TIS of a sagittal cross section, a target image 15TIA of an axial cross section, and a target image 15TIC of a coronal cross section. Below each of these target images 15TIS, 15TIA, and 15TIC, a button group 81 for switching the display is provided.

[0055] The first display screen 80 is provided with a similar image search button 82. When the doctor wishes to search for a similar image 15SI to the target image 15TI, he or she selects the similar image search button 82. As a result, a search request 23 is transmitted to the PACS server 11.

[0056] FIG. 13 shows an example of the second display screen 85 displayed on the display 17 of the client terminal 12 when the similar image 15SI is distributed from the PACS server 11 in response to the search request 23. On the second display screen 85, the similar image 15SI is displayed in the same manner as the target image 15TI below the target image 15TI. The similar images 15SI are displayed at the top in accordance with the order of similarity. The similar images 15SI are the similar image 15SIS of the sagittal section, the similar image 15SIA of the axial section, and the similar image 15SIC of the coronal section. The similar image 15SI displays the age and sex of the patient P of the similar image 15SI, as well as the findings of dementia. The findings of dementia are quoted from an electronic medical record system (not shown). When the confirmation button 86 is selected, the display of the similar image 15SI is erased, and the second display screen 85 is returned to the first display screen 80.

[0057] As an example, as shown in FIG. 14, a compression unit 91 of an autoencoder (hereinafter abbreviated as AE (Auto Encoder)) 90 is used for the feature derivation model 57. The AE 90 has a compression unit 91 and a restoration unit 92. An anatomical area image 61 is input to the compression unit 91. The compression unit 91 converts the anatomical area image 61 into a feature set 63. The compression unit 91 passes the feature set 63 to the restoration unit 92. The restoration unit 92 generates a restored image 93 of the anatomical area image 61 from the feature set 63.

[0058] The compression unit 91 converts the anatomical region image 61 into a feature set 63 by performing a convolution operation as shown in FIG. 15 as an example. Specifically, the compression unit 91 has a convolution layer 200 represented by "conv (short for convolution)". The convolution layer 200 applies, for example, a 3×3 filter 203 to target data 202 having a plurality of elements 201 arranged two-dimensionally. Then, the convolution layer 200 convolves an element value e of one target element 201I of the elements 201 with element values ​​a, b, c, d, f, g, h, and i of eight elements 201S adjacent to the target element 201I. The convolution layer 200 sequentially performs a convolution operation on each element 201 of the target data 202 while shifting the target element 201I by one element, and outputs an element value of an element 204 of the operation data 205. As a result, operation data 205 having a plurality of elements 204 arranged two-dimensionally like the target data 202 is obtained. The target data 202 inputted first to the convolution layer 200 is the anatomical area image 61, and thereafter, reduced operation data 205S (see FIG. 17) described later is inputted to the convolution layer 200 as the target data 202.

[0059] Assuming that the coefficients of filter 203 are r, s, t, u, v, w, x, y, and z, the element value k of element 204I corresponding to the element of interest 201I in operation data 205, which is the result of the convolution operation on element of interest 201I, can be obtained, for example, by calculating the following formula (A). k=az+by+cx+dw+ev+fu+gt+hs+ir...(A)

[0060] One piece of calculation data 205 is output for one filter 203. When multiple types of filters 203 are applied to one piece of target data 202, calculation data 205 is output for each filter 203. That is, as shown in FIG. 16 as an example, calculation data 205 is generated for the number of filters 203 applied to the target data 202. Furthermore, since the calculation data 205 has multiple elements 204 arranged two-dimensionally, it has a width and a height. The number of calculation data 205 is called the number of channels. FIG. 16 illustrates an example of four-channel calculation data 205 output by applying four filters 203 to the target data 202.

[0061] As an example, as shown in FIG. 17, the compression unit 91 has a pooling layer 210 represented by "pool" (short for pooling) in addition to the convolution layer 200. The pooling layer 210 obtains local statistics of element values ​​of elements 204 of the operation data 205, and generates reduced operation data 205S with the obtained statistics as element values. Here, the pooling layer 210 performs a maximum pooling process to obtain the maximum value of element values ​​in a block 211 of 2×2 elements as the local statistics. If the process is performed while shifting the block 211 by one element in the width direction and height direction, the reduced operation data 205S is reduced to 1 / 2 the size of the original operation data 205. FIG. 17 illustrates a case where b of the element values ​​a, b, e, and f in the block 211A, b of the element values ​​b, c, f, and g in the block 211B, and h of the element values ​​c, d, g, and h in the block 211C are maximum values. Alternatively, mean value pooling processing may be performed in which mean values, rather than maximum values, are obtained as local statistics.

[0062] The compression unit 91 outputs the final calculation data 205 by repeating the convolution process by the convolution layer 200 and the pooling process by the pooling layer 210 multiple times. This final calculation data 205 is the feature set 63, and the element value of each element 204 of the final calculation data 205 is the feature Z. The feature Z thus obtained represents the shape and texture characteristics of the anatomical region, such as the degree of atrophy of the hippocampus, the degree of vascular disorder in the white matter, and the presence or absence of decreased blood flow metabolism in the frontal lobe, anterior temporal lobe, and occipital lobe. Note that, although the description is given in two dimensions for the sake of simplicity, each process is actually performed in three dimensions.

[0063] 18, in a learning phase before the compression unit 91 is converted into the feature derivation model 57, the AE90 receives a learning anatomical area image 61L and learns it. The AE90 outputs a learning reconstructed image 93L for the learning anatomical area image 61L. A loss calculation is performed for the AE90 using a loss function based on the learning anatomical area image 61L and the learning reconstructed image 93L. Then, update settings for various coefficients of the AE90 (such as the coefficients of the filter 203) are performed according to the result of the loss calculation, and the AE90 is updated according to the update settings.

[0064] In the learning phase of the AE90, the above-mentioned series of processes, including input of the learning anatomical area image 61L to the AE90, output of the learning reconstructed image 93L from the AE90, loss calculation, update setting, and update of the AE90, are repeated while the learning anatomical area image 61L is exchanged. The repetition of the above-mentioned series of processes is terminated when the restoration accuracy of the learning reconstructed image 93L from the learning anatomical area image 61L reaches a predetermined set level. The compression unit 91 of the AE90 whose restoration accuracy has thus reached the set level is stored in the storage 25 and used as the trained feature derivation model 57.

[0065] Next, the operation of the above configuration will be described with reference to the flowcharts of Figures 19 to 21. When the operating program 35 is started in the PACS server 11, the CPU 27 of the PACS server 11 functions as a RW control unit 40, a preprocessing unit 41, a request receiving unit 42, a search processing unit 43, and a distribution control unit 44, as shown in Figure 5. As shown in Figure 6, the preprocessing unit 41 includes a normalization unit 50, an extraction unit 51, and a feature derivation unit 52. Also, as shown in Figure 10, the search processing unit 43 includes a similarity evaluation value calculation unit 70 and a similar image selection unit 71.

[0066] First, a head MRI image 15 of the patient P is taken by the MRI apparatus 10 , and the head MRI image 15 is transmitted from the MRI apparatus 10 to the PACS server 11 .

[0067] 19, in the PACS server 11, the head MRI image 15 is input to the RW control unit 40 and the preprocessing unit 41. The head MRI image 15 is stored in the storage 25 by the RW control unit 40 (step ST100).

[0068] 7, the normalization unit 50 performs normalization processing (shape normalization processing 65 and intensity normalization processing 66) to match the head MRI image 15 to the standard head MRI image 55 (step ST110). As a result, the head MRI image 15 is converted into a normalized head MRI image 60. The normalized head MRI image 60 is output from the normalization unit 50 to the extraction unit 51.

[0069] 8, in the extraction unit 51, a plurality of anatomical region images 61 are extracted from a normalized head MRI image 60 using the segmentation model 56 (step ST120). An anatomical region image group 62 consisting of the plurality of anatomical region images 61 is output from the extraction unit 51 to the feature derivation unit 52.

[0070] 9, in the feature derivation unit 52, an anatomical region image 61 is input to a corresponding feature derivation model 57. As a result, a feature set 63 is output from the feature derivation model 57 (step ST130). A feature set group 45 consisting of the multiple feature sets 63 is output from the feature derivation unit 52 to the RW control unit 40. The feature set group 45 is stored in the storage 25 by the RW control unit 40 (step ST140).

[0071] As an example, as shown in FIG. 20, when a delivery request 20 is transmitted from the client terminal 12 to the PACS server 11 and the delivery request 20 is received by the request receiving unit 42 (YES in step ST200), the RW control unit 40, which receives an instruction from the search processing unit 43, searches for and reads the target image 15TI corresponding to the delivery request 20 from the storage 25 (step ST210). The target image 15TI is output from the RW control unit 40 to the search processing unit 43, and is further output from the search processing unit 43 to the delivery control unit 44. The target image 15TI is delivered by the delivery control unit 44 to the client terminal 12 that is the request source of the delivery request 20 (step ST220). In the client terminal 12 that receives the target image 15TI, the first display screen 80 shown in FIG. 12 is displayed on the display 17. The doctor observes the brain of the patient P shown in the target image 15TI through the first display screen 80.

[0072] 21, when the similar image search button 82 on the first display screen 80 is selected and a search request 23 is transmitted from the client terminal 12 to the PACS server 11, and the search request 23 is accepted by the request accepting unit 42 (YES in step ST300), the RW control unit 40 receives an instruction from the search processing unit 43 and reads out all of the feature set groups 45 from the storage 25 (step ST310). The feature set groups 45 are output from the RW control unit 40 to the search processing unit 43.

[0073] 11, the similarity evaluation value calculation unit 70 calculates a similarity evaluation value 76 between the target image 15TI and the candidate image 15CI from the feature amount set group 45TI of the target image 15TI and the feature amount set group 45CI of the candidate image 15CI (step ST320). The similarity evaluation value 76 is output from the similarity evaluation value calculation unit 70 to the similar image selection unit 71. The calculation of the similarity evaluation value 76 is repeated until the calculation is completed for all the candidate images 15CI (NO in step ST330).

[0074] When the calculation of the similarity evaluation value 76 for all the candidate images 15CI is completed (YES in step ST330), the similar image selection unit 71 selects a similar image 15SI from the candidate images 15CI according to the selection condition 75 (step ST340). The similar image 15SI is output from the RW control unit 40 to the search processing unit 43, and is further output from the search processing unit 43 to the delivery control unit 44. The similar image 15SI is delivered by the delivery control unit 44 to the client terminal 12 that is the request source of the search request 23 (step ST350). In the client terminal 12 that receives the similar image 15SI, the second display screen 85 shown in FIG. 13 is displayed on the display 17. The doctor diagnoses dementia for the patient P while referring to the similar image 15SI through the second display screen 85.

[0075] As described above, the CPU 27 of the PACS server 11 includes the RW control unit 40, the extraction unit 51, the feature derivation unit 52, the search processing unit 43, and the distribution control unit 44. The RW control unit 40 receives the head MRI image 15 from the MRI device 10 to obtain the target image 15TI, which is the head MRI image 15 of the subject to be diagnosed with dementia. The extraction unit 51 extracts anatomical region images 61 of a plurality of anatomical regions of the brain from the normalized head MRI image 60 obtained by normalizing the head MRI image 15. The feature derivation unit 52 inputs the plurality of anatomical region images 61 to a plurality of feature derivation models 57 prepared for each of the plurality of anatomical regions, and causes the feature derivation model 57 to output a plurality of feature sets 63 for each of the plurality of anatomical regions. The search processing unit 43 searches for a similar image 15SI of the target image 15TI using a feature set group 45 composed of the plurality of feature sets 63. The distribution control unit 44 distributes the similar image 15SI to the client terminal 12 that has made the search request 23.

[0076] The number of feature values ​​Z is very large, ranging from tens to hundreds of thousands, for example. Therefore, feature values ​​Z do not represent limited features of the brain, but represent comprehensive features of the anatomical region. Furthermore, feature values ​​Z are not values ​​obtained statistically, such as the average pixel values, but are obtained by inputting anatomical region image 61 into feature value derivation model 57. Furthermore, feature values ​​Z are derived for each anatomical region, and similar images 15SI are searched for using feature values ​​Z derived for each anatomical region. Therefore, similar images 15SI are searched for without any particular consideration of the anatomical region. <Muhammad Owais, etc., Effective Diagnosis and Treatment through Content-Based Medical Image Retrieval (CBMIR) by Using Artificial Intelligence, Journal of Clinical Medicine, 2019.> Compared to the conventional method, it is possible to search for more accurate similar images 15SI.

[0077] 10 etc., the search processing unit 43 calculates a similarity evaluation value 76 between the target image 15TI and the candidate image CI from the feature amount set group 45TI of the target image 15TI and the feature amount set group 45CI of the candidate image 15CI, and selects a similar image 15SI from the candidate image 15CI based on the similarity evaluation value 76. This makes it possible to smoothly select a similar image 15SI.

[0078] 14, the feature derivation model 57 is a repurposed version of the compression unit 91 of the AE90. The AE90 is one of the neural network models frequently used in the field of machine learning and is generally very well known. For this reason, it can be relatively easily repurposed into the feature derivation model 57.

[0079] 7, prior to extraction of the anatomical region, the normalization unit 50 performs a normalization process to align the target image 15TI with the standard head MRI image 55. This allows subsequent processes to be performed after almost eliminating the individual differences of the patient P and the device differences of the MRI device 10, which can contribute to a more accurate search for similar images 15SI.

[0080] Dementia has become a social problem with the advent of the recent aging society. For this reason, this embodiment, in which the organ is the brain and the disease is dementia, and similar images 15SI are searched for, can be said to be a form that matches the current social problem.

[0081] The hippocampus and the anterior temporal lobe are anatomical regions that are particularly highly correlated with dementia, including Alzheimer's disease. Therefore, if at least one of the hippocampus and the anterior temporal lobe is included in a plurality of anatomical regions, as in this example, it becomes possible to search for more accurate similar images 15SI.

[0082] The candidate images 15CI are not limited to all head MRI images 15 other than the target image 15TI. For example, the candidate images 15CI may be head MRI images 15 of a patient P having the same attributes as the patient P in the target image 15TI, such as a head MRI image 15 of a patient P having the same sex and age as the patient P in the target image 15TI.

[0083] The selection condition 75 may be "candidate images whose similarity evaluation value is less than a threshold value." In this case, the similar image selection unit 71 selects a candidate image 15CI whose similarity evaluation value 76 is less than a preset threshold value as a similar image 15SI. If there is no candidate image 15CI whose similarity evaluation value 76 is less than the threshold value, the similar image selection unit 71 outputs a message indicating that there is no similar image 15SI. In this case, a message indicating that there is no similar image 15SI is displayed on the second display screen 85.

[0084] The output mode of the similar image 15SI is not limited to delivery to the client terminal 12. The similar image 15SI may be printed out on a paper medium, or may be sent to a mobile terminal of a doctor as an attachment to an e-mail.

[0085] [Second embodiment] In the second embodiment shown in FIG. 22 and FIG. 23, the similarity evaluation value 106 is calculated using the following formula (2).

[0086]

number

[0087] Equation (2) differs from equation (1) of the first embodiment in that the square of the absolute value of the difference between feature amount ZTIi and feature amount ZCIi is multiplied by the weighting coefficient Wi. In other words, according to equation (2), by setting an appropriate weighting coefficient Wi, it is possible to make the similarity evaluation value 106 represented as S a value that contributes to a more accurate search for similar images 15SI.

[0088] As an example, as shown in Table 100 in FIG. 22, the weighting coefficient Wi in this embodiment is set for each anatomical region. For example, weighting coefficients Wi lower than 1.0, such as "0.5", "0.6", and "0.75", are set for the anatomical regions such as the hippocampus, parahippocampal gyrus, frontal lobe, anterior temporal lobe, hypothalamus, and amygdala, which are considered to be particularly highly related to dementia. In contrast, weighting coefficient Wi of "1.0" is set for the anatomical regions such as the occipital lobe, thalamus, and pituitary gland, which are considered to be relatively less related to dementia. A weighting coefficient Wi of "1.0" means that no weighting is applied.

[0089] 23 as an example, the similarity evaluation value calculation unit 105 of this embodiment applies the weighting coefficient Wi set as shown in Table 100 to the feature Z of the feature set 63 of each anatomical region. Specifically, the weighting coefficient Wi is multiplied by "0.5" to the square of the absolute value of the difference between the feature ZTI1_1 of the hippocampus feature set 63TI_1 of the target image 15TI and the feature ZCI1_1 of the hippocampus feature set 63CI_1 of the candidate image 15CI. In addition, the weighting coefficient Wi is multiplied by "0.5" to the square of the absolute value of the difference between the feature ZTI1_3 of the frontal lobe feature set 63TI_3 of the target image 15TI and the feature ZCI1_3 of the frontal lobe feature set 63CI_3 of the candidate image 15CI. Furthermore, the square of the absolute value of the difference between the feature amount ZTI1_4 of the anterior temporal lobe feature amount set 63TI_4 of the target image 15TI and the feature amount ZCI1_4 of the anterior temporal lobe feature amount set 63CI_4 of the candidate image 15CI is multiplied by "0.6" as a weighting coefficient Wi.

[0090] In this way, the similarity evaluation value calculation unit 105 of the second embodiment weights the feature amount set group 45TI of the target image 15TI and the feature amount set group 45CI of the candidate image 15CI according to the anatomical area when calculating the similarity evaluation value 106. This makes it possible to make the similarity evaluation value 106 a value more suitable for diagnosing dementia, and as a result, the accuracy of the similar image 15SI can be further improved.

[0091] It should be noted that the weighting coefficients Wi shown in Fig. 22 are merely examples. For example, a value greater than 1.0 may be set as the weighting coefficients Wi, or a negative value may be set as the weighting coefficients Wi.

[0092] The weighting coefficient Wi may be set, for example, by any of the following first, second, and third methods. In the first method, first, several patterns with different weighting coefficients Wi set for each anatomical region are prepared. A similarity evaluation value 106 is calculated for each pattern, and a similar image 15SI is selected for each pattern. Next, the similar images 15SI selected for each pattern are displayed on the display 17 of the client terminal 12 in a selectable manner, and the doctor is allowed to select the similar image 15SI that is considered to be the most appropriate. Then, the pattern when the similar image 15SI selected by the doctor is selected is set as the weighting coefficient Wi to be used in the future.

[0093] In the second method, first, for any one of the multiple head MRI images 15 stored in the PACS server 11, a doctor ranks the similarity of the other head MRI images 15. Then, as in the first method, several patterns with different weighting coefficients Wi set for each anatomical region are prepared, and a similarity evaluation value 106 is calculated for each pattern. Then, the pattern that has calculated a similarity evaluation value that matches or is most similar to the ranking given by the doctor is set as the weighting coefficient Wi to be used hereafter.

[0094] The third method is the same as the second method in that a doctor ranks the similarities and the similarity evaluation value 106 is calculated for several patterns with different weighting coefficients Wi set for each anatomical region. The third method uses a statistical method such as the least squares method to find an optimal solution and finds a pattern of weighting coefficients Wi that best matches the doctor's ranking of similarities.

[0095] [Third embodiment] In the third embodiment shown in FIGS. 24 and 25, a compression unit 111 of a single-task class discrimination convolutional neural network (hereinafter abbreviated as single-task CNN (Convolutional Neural Network)) 110 is used as a feature derivation model 115 instead of the compression unit 91 of the AE 90.

[0096] As an example, as shown in FIG. 24, a single-task CNN 110 has a compression unit 111 and an output unit 112. An anatomical area image 61 is input to the compression unit 111. The compression unit 111 converts the anatomical area image 61 into a feature set 113, similar to the compression unit 91. The compression unit 111 passes the feature set 113 to the output unit 112. The output unit 112 outputs one class 114 based on the feature set 113. In FIG. 24, the output unit 112 outputs a discrimination result of whether or not the subject has dementia as the class 114. The compression unit 111 of this single-task CNN 110 is used as a feature derivation model 115.

[0097] 25, in a learning phase before the compression unit 111 is converted into a feature derivation model 115, the single-task CNN 110 is given learning data 118 and trained. The learning data 118 is a set of the training anatomical area image 61L and a correct class 114CA corresponding to the training anatomical area image 61L. The correct class 114CA is a result of a doctor's actual determination of whether or not dementia has developed for the head MRI image 15 from which the training anatomical area image 61L was obtained.

[0098] In the learning phase, a training anatomical area image 61L is input to the single-task CNN 110. The single-task CNN 110 outputs a training class 114L for the training anatomical area image 61L. A loss calculation is performed for the single-task CNN 110 based on the training class 114L and a correct answer class 114CA. Then, various coefficients of the single-task CNN 110 are updated according to the result of the loss calculation, and the single-task CNN 110 is updated according to the update setting.

[0099] In the learning phase of the single-task CNN 110, the above series of processes, including input of the learning anatomical area image 61L to the single-task CNN 110, output of the learning class 114L from the single-task CNN 110, loss calculation, update setting, and update of the single-task CNN 110, are repeated while exchanging the learning data 118. The repetition of the above series of processes is terminated when the prediction accuracy of the learning class 114L for the correct class 114CA reaches a predetermined set level. The compression unit 111 of the single-task CNN 110 whose prediction accuracy has thus reached the set level is stored in the storage 25 as a learned feature derivation model 115 and is used by the feature derivation unit 52.

[0100] In this way, in the third embodiment, the compression unit 111 of the single-task CNN 110 is used as the feature derivation model 115. The single-task CNN 110 is also one of the neural network models frequently used in the field of machine learning and is generally very well known. For this reason, it can be relatively easily diverted to the feature derivation model 115.

[0101] The class 114 may be, for example, the age of the patient P being under 75 years old or 75 years old or older, or may be the age group of the patient P, such as in their 60s or 70s.

[0102] [Fourth embodiment] In the fourth embodiment shown in Figures 26 and 27, a compression unit 121 of a multi-task class discrimination CNN (hereinafter abbreviated as multi-task CNN) 120 is used as a feature derivation model 126 instead of the compression unit 91 of the AE 90 and the compression unit 111 of the single-task CNN 110.

[0103] As an example, as shown in FIG. 26, the multitask CNN 120 has a compression unit 121 and an output unit 122. The compression unit 121 receives an anatomical region image 61. The compression unit 121 converts the anatomical region image 61 into a feature set 123, similar to the compression units 91 and 111. The compression unit 121 passes the feature set 123 to the output unit 122. The output unit 122 outputs two classes, a first class 124 and a second class 125, based on the feature set 123. In FIG. 26, the output unit 122 outputs a discrimination result, that is, whether the patient has dementia or does not have dementia, as the first class 124. Also, in FIG. 26, the output unit 122 outputs the age of the patient P as the second class 125. The compression unit 121 of the multitask CNN 120 is used as a feature derivation model 126.

[0104] As an example, as shown in FIG. 27, in a learning phase before converting the compression unit 121 to a feature derivation model 126, the multitask CNN 120 is given learning data 128 and trained. The learning data 128 is a set of the training anatomical area image 61L and a first correct class 124CA and a second correct class 125CA corresponding to the training anatomical area image 61L. The first correct class 124CA is a result of a doctor's actual determination of whether or not dementia has developed for the head MRI image 15 from which the training anatomical area image 61L was obtained. The second correct class 125CA is the actual age of the patient P who is the subject of the head MRI image 15 from which the training anatomical area image 61L was obtained.

[0105] In the learning phase, a training anatomical area image 61L is input to the multi-task CNN 120. The multi-task CNN 120 outputs a first training class 124L and a second training class 125L for the training anatomical area image 61L. A loss calculation is performed for the multi-task CNN 120 based on the first training class 124L and the second training class 125L, and a first correct class 124CA and a second correct class 125CA. Then, update settings of various coefficients of the multi-task CNN 120 are performed according to the result of the loss calculation, and the multi-task CNN 120 is updated according to the update settings.

[0106] In the learning phase of the multitasking CNN 120, a series of processes including input of the learning anatomical area image 61L to the multitasking CNN 120, output of the first learning class 124L and the second learning class 125L from the multitasking CNN 120, loss calculation, update setting, and update of the multitasking CNN 120 are repeatedly performed while exchanging the learning data 128. The repetition of the series of processes is terminated when the prediction accuracy of the first learning class 124L and the second learning class 125L for the first correct class 124CA and the second correct class 125CA reaches a predetermined set level. The compression unit 121 of the multitasking CNN 120 whose prediction accuracy has thus reached the set level is stored in the storage 25 as a learned feature derivation model 126 and is used by the feature derivation unit 52.

[0107] As described above, in the fourth embodiment, the compression unit 121 of the multi-task CNN 120 is used as the feature derivation model 126. Compared with the AE90 and the single-task CNN 110, the multi-task CNN 120 performs a more complicated process of outputting a plurality of classes (a first class 124 and a second class 125). Therefore, the feature set 123 output from the compression unit 121 is likely to more comprehensively represent the features of the anatomical area image 61. Therefore, as a result, the accuracy of the similar image 15SI can be further improved.

[0108] The first class 124 may be, for example, a five-level level of progression of dementia. The second class 125 may be a discrimination result of the age of the patient P. The multitask CNN 120 may output three or more classes.

[0109] [Fifth embodiment] In the fifth embodiment shown in FIG. 28, an anatomical region image 61 of one anatomical region is input to a plurality of different feature amount derivation models.

[0110] In Fig. 28, the feature derivation unit 130 of this embodiment inputs an anatomical area image 61 of one anatomical area to a first feature derivation model 131, a second feature derivation model 132, and a third feature derivation model 133. As a result, the feature derivation unit 130 outputs a first feature set 134 from the first feature derivation model 131, outputs a second feature set 135 from the second feature derivation model 132, and outputs a third feature set 136 from the third feature derivation model 133. The first feature derivation model 131 is a repurposed version of the compression unit 91 of the AE 90 of the first embodiment. The second feature derivation model 132 is a repurposed version of the compression unit 111 of the single task CNN 110 of the third embodiment. The third feature derivation model 133 is a repurposed version of the compression unit 121 of the multitask CNN 120 in the fourth embodiment.

[0111] As described above, in the fifth embodiment, the feature derivation unit 130 inputs the anatomical region image 61 of one anatomical region to the first feature derivation model 131, the second feature derivation model 132, and the third feature derivation model 133. Then, the first feature set 134, the second feature set 135, and the third feature set 136 are output from each of the models 131 to 133. Therefore, a wider variety of feature amounts Z can be obtained compared to the case where one type of feature derivation model 57 is used. As a result, the accuracy of the similar image 15SI can be further improved.

[0112] The different feature derivation models may be, for example, a combination of a first feature derivation model 131 obtained by reusing the compression unit 91 of the AE 90 and a second feature derivation model 132 obtained by reusing the compression unit 111 of the single-task CNN 110. Alternatively, the different feature derivation models may be a combination of a second feature derivation model 132 obtained by reusing the compression unit 111 of the single-task CNN 110 and a third feature derivation model 133 obtained by reusing the compression unit 121 of the multi-task CNN 120. Furthermore, the different feature derivation models may be a combination of a second feature derivation model 132 obtained by reusing the compression unit 111 of the single-task CNN 110 that outputs whether or not dementia has developed as the class 114, and a second feature derivation model 132 obtained by reusing the compression unit 111 of the single-task CNN 110 that outputs the age of the patient P as the class 114.

[0113] [Sixth embodiment] The second embodiment in which the similarity evaluation value 106 is calculated using equation (2) including the weighting coefficient Wi and the fifth embodiment in which a plurality of different feature derivation models 131 to 133 are used may be combined and implemented.

[0114] In this case, as an example, as shown in Table 140 in FIG. 29, a weighting coefficient Wi is set for each type of feature derivation model. Specifically, "1.0" is applied as a weighting coefficient Wi to the feature Z of the first feature set 134 derived by the first feature derivation model 131 repurposed from the compression unit 91 of the AE 90 of the first embodiment. "0.75" is applied as a weighting coefficient Wi to the feature Z of the second feature set 135 derived by the second feature derivation model 132 repurposed from the compression unit 111 of the single-task CNN 110 of the third embodiment. "0.5" is applied as a weighting coefficient Wi to the feature Z of the third feature set 136 derived by the third feature derivation model 133 repurposed from the compression unit 121 of the multi-task CNN 120 of the fourth embodiment.

[0115] In this way, in the sixth embodiment, when calculating the similarity evaluation value 106, weighting is performed on the feature set group 45TI of the target image 15TI and the feature set group 45CI of the candidate image 15CI according to the multiple different feature derivation models 131 to 133. Therefore, the similarity evaluation value 106 can be made a value more suitable for diagnosing dementia, and as a result, the accuracy of the similar image 15SI can be further improved.

[0116] As in the case of Fig. 22, the weighting coefficients Wi shown in Fig. 29 are merely examples. For example, a value greater than 1.0 may be set as the weighting coefficient Wi, or a negative value may be set as the weighting coefficient Wi. Also in this case, as in the second embodiment, the weighting coefficients Wi may be set by the first to third methods.

[0117] Instead of formula (1), the similarity evaluation value may be calculated using formula (3) below. Formula (3) is a formula for calculating the sum of absolute values ​​of differences between feature amounts ZTIi and ZCIi, that is, the Manhattan distance, as the similarity evaluation value.

[0118]

number

[0119] Moreover, instead of formula (2), the similarity evaluation value may be calculated using the following formula (4): Formula (4) is obtained by multiplying the absolute value of the difference between the feature amount ZTIi and the feature amount ZCIi by a weighting coefficient Wi.

[0120]

number

[0121] In each of the above embodiments, the distance between a multidimensional vector having the feature amount ZTIi as an element and a multidimensional vector having the feature amount ZCIi as an element is calculated as the similarity evaluation value, but this is not limited to this. The correlation coefficient between the feature amount ZTIi and the feature amount ZCIi may be calculated as the similarity evaluation value. In this case, contrary to each of the above embodiments, the higher the similarity between the target image 15TI and the candidate image 15CI (there is a correlation between the feature amount ZTIi and the feature amount ZCIi), the larger the similarity evaluation value becomes.

[0122] [Seventh embodiment] In the seventh embodiment shown in FIGS. 30 to 36, a model combining an AE 250 and a single-task CNN 251 is used as a feature derivation model 252.

[0123] 30 as an example, the AE250 has a compression unit 253 and a restoration unit 254, similar to the AE90 of the first embodiment. The compression unit 253 receives an anatomical area image 61. The compression unit 253 converts the anatomical area image 61 into a feature set 255. The compression unit 253 passes the feature set 255 to the restoration unit 254. The restoration unit 254 generates a restored image 256 of the anatomical area image 61 from the feature set 255.

[0124] The single-task CNN 251 has a compression unit 253 and an output unit 257, similar to the single-task CNN 110 of the third embodiment. That is, the compression unit 253 is shared by the AE 250 and the single-task CNN 251. The compression unit 253 transfers the feature set 255 to the output unit 257. The output unit 257 outputs one class 258 based on the feature set 255. In FIG. 30, the output unit 257 outputs, as the class 258, a discrimination result that a patient P currently suffering from mild cognitive impairment will remain suffering from mild cognitive impairment after two years, or will progress to Alzheimer's disease after two years. In addition, the output unit 257 outputs an aggregate feature ZA that aggregates a plurality of feature values ​​Z constituting the feature set 255. The aggregate feature ZA is output for each anatomical region. In this embodiment, the aggregate feature ZA is input to the search processing unit 278 (see FIG. 34) instead of the feature set 255.

[0125] As an example, as shown in FIG. 31, the output unit 257 has a self-attention (hereinafter abbreviated as SA (Self-Attention)) mechanism layer 265, a global average pooling (hereinafter abbreviated as GAP (Global Average Pooling)) layer 266, a fully connected (hereinafter abbreviated as FC (Fully Connected)) layer 267, a softmax function (hereinafter abbreviated as SMF (SoftMax Function)) layer 268, and a principal component analysis (hereinafter abbreviated as PCA (Principal Component Analysis)) layer 269.

[0126] 15 on the feature set 255 while changing the coefficients of the filter 203 according to the element value of the element of interest 201I. Hereinafter, the convolution process performed by the SA mechanism layer 265 is referred to as SA convolution process. The SA mechanism layer 265 outputs the feature set 255 after the SA convolution process to the GAP layer 266.

[0127] The GAP layer 266 performs global average pooling on the feature set 255 after the SA convolution process. The global average pooling process is a process of calculating the average value of the feature Z for each channel of the feature set 255 (see FIG. 16 ). For example, if the number of channels of the feature set 255 is 512, the average value of the 512 features Z is calculated by the global average pooling process. The GAP layer 266 outputs the calculated average value of the feature Z to the FC layer 267 and the PCA layer 269.

[0128] The FC layer 267 converts the average value of the feature Z into a variable to be handled by the SMF of the SMF layer 268. The FC layer 267 has an input layer having units equal to the number of average values ​​of the feature Z (i.e., the number of channels of the feature set 255), and an output layer having units equal to the number of variables to be handled by the SMF. Each unit in the input layer and each unit in the output layer are fully connected to each other, and a weight is set for each. The average value of the feature Z is input to each unit in the input layer. The sum of the products of the average value of the feature Z and the weight set between each unit becomes the output value of each unit in the output layer. This output value is the variable to be handled by the SMF. The FC layer 267 outputs the variables to be handled by the SMF to the SMF layer 268. The SMF layer 268 outputs the class 258 by applying the variables to the SMF.

[0129] The PCA layer 269 performs PCA on the average values ​​of the feature amounts Z, and converts the average values ​​of the multiple feature amounts Z into a smaller number of aggregated feature amounts ZA. For example, the PCA layer 269 aggregates the average values ​​of 512 feature amounts Z into one aggregated feature amount ZA.

[0130] As an example, as shown in Fig. 32, in a learning phase, the AE250 is input with a learning anatomical area image 61L and learns. The AE250 outputs a learning reconstructed image 256L for the learning anatomical area image 61L. Based on the learning anatomical area image 61L and the learning reconstructed image 256L, a loss calculation is performed for the AE250 using a loss function. Then, update settings of various coefficients of the AE250 are performed according to the result of the loss calculation (hereinafter, referred to as loss L1), and the AE250 is updated according to the update settings.

[0131] In the learning phase of AE250, the above series of processes, including input of the training anatomical area image 61L to AE250, output of the training reconstructed image 256L from AE250, loss calculation, update setting, and updating of AE250, are repeated while the training anatomical area image 61L is replaced.

[0132] In the learning phase, the single-task CNN 251 is trained by being given training data 275. The training data 275 is a set of the training anatomical area image 61L and the correct answer class 258CA corresponding to the training anatomical area image 61L. The correct answer class 258CA indicates whether the patient P of the subject of the head MRI image 15 from which the training anatomical area image 61L was obtained actually remains suffering from mild cognitive impairment after two years or has progressed to Alzheimer's disease after two years.

[0133] In the learning phase, a training anatomical area image 61L is input to the single-task CNN 251. The single-task CNN 251 outputs a training class 258L for the training anatomical area image 61L. Based on the training class 258L and the correct answer class 258CA, a loss calculation is performed for the single-task CNN 251 using a cross-entropy function or the like. Then, various coefficients of the single-task CNN 251 are updated according to the result of the loss calculation (hereinafter, referred to as loss L2), and the single-task CNN 251 is updated according to the update setting.

[0134] In the training phase of the single-task CNN 251, the above-mentioned series of processes, including input of training anatomical area images 61L to the single-task CNN 251, output of training classes 258L from the single-task CNN 251, loss calculation, update setting, and updating of the single-task CNN 251, are repeated while the training data 275 is exchanged.

[0135] The update setting of the AE 250 and the update setting of the single-task CNN 251 are performed based on the total loss L shown in the following equation (B), where α is a weight. L = L1 × α + L2 × (1-α) (B) That is, the total loss L is the weighted sum of the loss L1 of the AE 250 and the loss L2 of the single-task CNN 251.

[0136] As an example, as shown in Fig. 33, in the initial stage of the learning phase, the weight α is set to 1. When the weight α is set to 1, the total loss L = L1. Therefore, in this case, only the AE250 is learned, and the single-task CNN251 is not learned.

[0137] The weight α is gradually decreased from 1 as the learning progresses, and eventually becomes a fixed value (0.8 in FIG. 33). In this case, the learning of the AE250 and the learning of the single-task CNN251 are both performed with an intensity according to the weight α. In this way, the weight given to the loss L1 is greater than the weight given to the loss L2. In addition, the weight given to the loss L1 is gradually decreased from the maximum value of 1, and the weight given to the loss L2 is gradually increased from the minimum value of 0, and both are fixed values.

[0138] The learning of the AE250 and the single-task CNN251 is terminated when the restoration accuracy of the learning anatomical area image 61L to the learning restored image 256L by the AE250 reaches a predetermined set level, and the prediction accuracy of the learning class 258L for the correct class 258CA by the single-task CNN251 reaches a predetermined set level. The AE250 and the single-task CNN251 whose restoration accuracy and prediction accuracy have thus reached the set levels are stored in the storage 20 and used as the learned feature derivation model 252.

[0139] As an example, as shown in FIG. 34, the search processing unit 278 of this embodiment receives the aggregate feature group ZAGTI of the target image 15TI and the aggregate feature group ZAGCI of the candidate image 15CI instead of the feature set group 45TI of the target image 15TI and the feature set group 45CI of the candidate image 15CI. The aggregate feature group ZAGTI of the target image 15TI is a collection of a plurality of aggregate feature amounts ZA derived from a plurality of anatomical area images 61 of the target image 15TI. Similarly, the aggregate feature group ZAGCI of the candidate image 15CI is a collection of a plurality of aggregate feature amounts ZA derived from a plurality of anatomical area images 61 of the candidate image 15CI. The search processing unit 278 calculates a similarity evaluation value 76 from the aggregate feature groups ZAGTI and ZAGCI instead of the feature set groups 45TI and 45CI, and selects a similar image 15SI based on the selection condition 75 and the calculated similarity evaluation value 76.

[0140] As an example, as shown in FIG. 35, it is also possible to obtain dementia findings of a patient P of a target image 15TI from an aggregate feature group ZAGTI of the target image 15TI. In this case, a dementia findings derivation unit 280 is constructed in the CPU 27 of the PACS server 11. The dementia findings derivation unit 280 inputs the aggregate feature group ZAGTI of the target image 15TI and dementia-related information 281 to a dementia findings derivation model 282. The dementia-related information 281 includes, for example, the sex and age of the patient P diagnosed with dementia, the volume of an anatomical region such as the volume of the hippocampus, a dementia test score such as the score of the Revised Hasegawa's Dementia Scale (HDS-R), a genetic test result such as the genotype of the ApoE gene, a cerebrospinal fluid test result such as an amyloid β measurement value, and a blood test result such as an apolipoprotein measurement value.

[0141] The dementia finding derivation model 282 has a quantile normalization unit 283 and a linear discriminant analysis unit 284. The quantile normalization unit 283 receives the aggregate feature group ZAGTI and the dementia-related information 281. The quantile normalization unit 283 performs quantile normalization to convert the aggregate feature ZA constituting the aggregate feature group ZAGTI and each parameter of the dementia-related information 281 into data that follows a normal distribution in order to treat them on the same level. The linear discriminant analysis unit 284 performs linear discriminant analysis on the aggregate feature ZA after the quantile normalization process and each parameter of the dementia-related information 281, and outputs dementia finding information 285 as a result. The dementia finding information 285 is either that a patient P who currently has mild cognitive impairment will remain with mild cognitive impairment in two years, or that the patient will progress to Alzheimer's disease in two years.

[0142] In this way, in the seventh embodiment, the single-task CNN 251 that performs a main task such as outputting the class 258 and the AE 250 that is partially common to the single-task CNN 251 and performs a subtask that is more general than the main task such as generating the restored image 256 are used as the feature derivation model 252. Then, the AE 250 and the single-task CNN 251 are trained simultaneously. For this reason, it is possible to output a more appropriate feature set 255 and aggregated features ZA compared to the case where the AE 250 and the single-task CNN 251 are separately used, and as a result, it is possible to search for a more accurate similar image 15SI.

[0143] In the learning phase, the update setting is based on the total loss L, which is the weighted sum of the loss L1 of AE250 and the loss L2 of single-task CNN251. Therefore, by setting the weight α to an appropriate value, it is possible to focus on learning AE250, focus on learning single-task CNN251, or balance the learning of AE250 and single-task CNN251.

[0144] The weight given to the loss L1 is greater than the weight given to the loss L2. This allows the AE250 to always be trained in a focused manner. By always training the AE250 in a focused manner, the compression unit 253 can output a feature set 255 that better represents the shape and texture characteristics of the anatomical area, and as a result, the output unit 257 can output a more plausible aggregate feature ZA.

[0145] Also, the weight given to the loss L1 is gradually decreased from the maximum value, and the weight given to the loss L2 is gradually increased from the minimum value, and both are fixed after a predetermined number of learnings. This allows AE250 to be trained more intensively in the early learning stage. AE250 is responsible for the relatively simple subtask of generating the restored image 256. Therefore, by training AE250 more intensively in the early learning stage, it is possible to cause the compression unit 253 to output a feature set 255 that better represents the shape and texture features of the anatomical area in the early learning stage.

[0146] As an example, table 300 shown in Fig. 36 shows a performance comparison between Nos. 1 to 7 relating to the dementia progression prediction methods described in the following documents A, B, C, D, E, F, and G, and Nos. 8 and 9 relating to the dementia progression prediction method exemplified in Fig. 35. Of the dementia progression prediction methods exemplified in Fig. 35, No. 8 shows a case where only the aggregated feature group ZAGTI is input to the dementia findings derivation model 282, and the dementia-related information 281 is not input. In contrast, No. 9 shows a case where the aggregated feature group ZAGTI and the dementia-related information 281 are input to the dementia findings derivation model 282.

[0147] Document A <Tam, A., Dansereau, C., Iturria-Medina, Y., Urchs, S., Orban, P., Sharmarke, H., Breitner, J., & Alzheimer’s Disease Neuroimaging Initiative., “A highly predictive signature of cognition and brain atrophy for progression to Alzheimer’s dementia.”, GigaScience, 8 (5), giz055 (2019).>

[0148] Document B <Ledig, C., Schuh, A., Guerrero, R., Heckemann, R. A., & Rueckert, D., “Structural brain imaging in Alzheimer’s disease and mild cognitive impairment: biomarker analysis and shared morphometry database.”, Scientific reports, 8 (1), 11258 (2018).>

[0149] Document C <Lu, D., Popuri, K., Ding, G. W., Balachandar, R., & Beg, M. F., “Multimodal and multiscale deep neural networks for the early diagnosis of Alzheimer’s disease using structural MR and FDG-PET images.”, Scientific reports, 8 (1), 5697 (2018).>

[0150] Literature D <Basaia, S., Agosta, F., Wagner, L., Canu, E., Magnani, G., Santangelo, R., Filippi, M., Automated classification of Alzheimer’s disease and mild cognitive impairment using a single MRI and deep neural networks, NeuroImage: Clinical 21, 101645 (2019).>

[0151] Literature E <Nakagawa, T., Ishida, M., Naito, J., Nagai, A., Yamaguchi, S., Onoda, K., “Prediction of conversion to Alzheimer’s disease using deep survival analysis of MRI images”, Brain Communications, Vol. 2(1) (2020).>

[0152] Literature F <Lee, G., Nho, K., Kang, B., Sohn, K. A., & Kim, D., “Predicting Alzheimer’s disease progression using multi-modal deep learning approach.”, Scientific reports, 9 (1), 1952 (2019).>

[0153] Literature G <Goto, T., Wang, C., Li, Y., Tsuboshita, Y., Multi-modal deep learning for predicting progression of Alzheimer’s disease using bi-linear shake fusion, Proc. SPIE 11314, Medical Imaging (2020).>

[0154] The accuracy rates (Accuracy) of No. 8 and 9 are 0.84 and 0.90. In particular, the accuracy rate of No. 9, 0.90, is higher than the accuracy rates of No. 1 to 7. The AUC (Area Under the Curve) of No. 8 and 9 are 0.93 and 0.97. These values ​​are higher than No. 5, which relates to the dementia progression prediction method described in Document E. Therefore, it can be said that the dementia progression prediction method illustrated in FIG. 35 can predict the dementia progression with higher accuracy than the conventional dementia progression prediction methods described in Documents A to G.

[0155] The sensitivity of No. 8 and 9 is 0.85 and 0.91. These values ​​are higher than the sensitivity of any of No. 1 to 7. In particular, the sensitivity of No. 9, 0.91, is the highest among them. Therefore, it can be said that the dementia progression prediction method illustrated in FIG. 35 can predict without missing that patient P, who currently has mild cognitive impairment, will progress to Alzheimer's disease after the prediction period, compared to the conventional dementia progression prediction methods described in Literature A to G.

[0156] The specificity of Nos. 8 and 9 is 0.84 and 0.90. Although these values ​​are smaller than 0.97 of No. 1 related to the dementia progression prediction method described in document A, they are larger than those of other documents B, C, D, and F. Therefore, it can be said that the dementia progression prediction method illustrated in FIG. 35 can more accurately predict that patient P, who currently has mild cognitive impairment, will remain mild cognitive impairment after the prediction period compared to many other conventional dementia progression prediction methods.

[0157] From the above, it can be said that the aggregate feature groups ZAGTI and ZAGCI accurately represent the features of the target image 15TI and the candidate image 15CI. Therefore, it is possible to obtain an effect that it is possible to search for a more accurate similar image 15SI.

[0158] In Table 300, ADNI in the learning image section is an abbreviation for "Alzheimer's Disease Neuroimaging Initiative." AIBL is an abbreviation for "Australian Imaging Biomarkers and Lifestyle Study of Ageing." J-ADNI is an abbreviation for "Japanese Alzheimer's Disease Neuroimaging Intiative." All of these refer to databases in which head MRI images 15 of patient P with Alzheimer's disease and the like are accumulated.

[0159] The multi-task CNN 120 of the fourth embodiment may be used instead of the single-task CNN 251. Furthermore, findings of dementia of the patient P may be derived based on the feature set group 45TI of the target image 15TI, instead of the aggregate feature group ZAGTI of the target image 15TI.

[0160] The learning of the AE90 shown in Fig. 18, the learning of the single-task CNN110 shown in Fig. 25, the learning of the multi-task CNN120 shown in Fig. 27, and the learning of the AE250 and the single-task CNN251 shown in Fig. 32 may be performed in the PACS server 11 or in a device other than the PACS server 11. In addition, these learnings may be performed continuously after the feature derivation model is stored in the storage 25 of the PACS server 11.

[0161] A device other than the PACS server 11 may function as the diagnostic support device. Also, the client terminal 12 may function as the diagnostic support device.

[0162] The medical image is not limited to the illustrative head MRI image 15. It may be a PET (Positron Emission Tomography) image, a SPECT (Single Photon Emission Computed Tomography) image, a CT (Computed Tomography) image, an endoscopic image, an ultrasound image, or the like.

[0163] The organs are not limited to the brain, but may be the heart, lungs, liver, etc. In the case of the lungs, the right lungs S1 and S2 and the left lungs S1 and S2 are extracted as the anatomical regions. In the case of the liver, the right lobe, the left lobe, the gallbladder, etc. are extracted as the anatomical regions. In addition, the diseases are not limited to the dementia, but may be heart disease, pneumonia, liver dysfunction, etc.

[0164] In each of the above embodiments, for example, the following various processors can be used as the hardware structure of the processing units that execute various processes, such as the RW control unit 40, the preprocessing unit 41, the request receiving unit 42, the search processing units 43 and 278, the delivery control unit 44, the normalization unit 50, the extraction unit 51, the feature amount derivation units 52 and 130, the similarity evaluation value calculation units 70 and 105, the similar image selection unit 71, and the dementia findings derivation unit 280. As described above, the various processors include the CPU 27, which is a general-purpose processor that executes software (operation program 35) and functions as various processing units, as well as dedicated electric circuits that are processors having a circuit configuration designed specifically to execute specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0165] A single processing unit may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Also, multiple processing units may be composed of a single processor.

[0166] As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as typified by System On Chip (SoC), etc. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0167] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0168] The technology of the present disclosure can be appropriately combined with the above-mentioned various embodiments and / or various modified examples. In addition, it is not limited to the above-mentioned embodiments, and various configurations can be adopted without departing from the gist of the technology. Furthermore, the technology of the present disclosure extends to a storage medium that non-temporarily stores a program, in addition to a program.

[0169] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replaced with respect to the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the description and illustrations shown above omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0170] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0171] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A processor; A memory connected to or embedded in the processor, The processor, Obtaining a target image, which is a medical image of a target for diagnosing a disease; Extracting a plurality of anatomical regions of an organ from the target image; inputting the images of the plurality of anatomical regions into a plurality of feature derivation models prepared for each of the plurality of anatomical regions, and outputting a plurality of feature amounts for each of the plurality of anatomical regions from the feature derivation models; searching for an image similar to the target image using the plurality of feature amounts; outputting the similar image; Diagnostic aids.

2. The processor, Extracting the plurality of anatomical regions from the candidate images of the similar image; inputting images of the plurality of anatomical regions of the candidate image into the plurality of feature amount derivation models, and outputting a plurality of feature amounts for each of the plurality of anatomical regions from the feature amount derivation models; calculating a similarity evaluation value between the target image and the candidate image from the plurality of feature amounts of the target image and the plurality of feature amounts of the candidate image; The diagnosis support device according to claim 1 , wherein the similar image is selected from the candidate images based on the similarity evaluation value.

3. The processor, The diagnosis support device according to claim 2 , wherein when the similarity evaluation value is calculated, weights are assigned to the plurality of feature amounts of the target image and the plurality of feature amounts of the candidate images in accordance with the anatomical region.

4. 4. The diagnostic support device according to claim 1, wherein the feature derivation model includes at least one of an autoencoder, a single-task convolutional neural network for classifying, and a multi-task convolutional neural network for classifying.

5. The processor, 5. The diagnosis support device according to claim 1, wherein an image of one of the anatomical regions is input to a plurality of different feature derivation models, and the feature is output from each of the plurality of feature derivation models.

6. In the diagnosis support device according to claim 5, which cites claim 2 or claim 3, The processor, a diagnostic support device that, when calculating the similarity evaluation value, weights the plurality of feature amounts of the target image and the plurality of feature amounts of the candidate images according to a plurality of different feature amount derivation models;

7. The processor, 7. The diagnostic support device according to claim 1, further comprising: a normalization process for aligning the acquired target image with a standard medical image prior to the extraction of the anatomical region.

8. The diagnostic support device according to claim 1 , wherein the organ is a brain, and the disease is dementia.

9. The diagnostic support device according to claim 8 , wherein the plurality of anatomical regions includes at least one of the hippocampus and the anterior temporal lobe.

10. obtaining a target image, the target image being a medical image of a target for diagnosing a disease; extracting a plurality of anatomical regions of an organ from said target image; inputting the images of the plurality of anatomical regions into a plurality of feature amount derivation models prepared for each of the plurality of anatomical regions, and outputting a plurality of feature amounts for each of the plurality of anatomical regions from the feature amount derivation models; searching for an image similar to the target image using the plurality of feature amounts; and outputting the similar image; A method for operating a diagnostic support device comprising:

11. obtaining a target image, the target image being a medical image of a target for diagnosing a disease; extracting a plurality of anatomical regions of an organ from said target image; inputting the images of the plurality of anatomical regions into a plurality of feature amount derivation models prepared for each of the plurality of anatomical regions, and outputting a plurality of feature amounts for each of the plurality of anatomical regions from the feature amount derivation models; searching for an image similar to the target image using the plurality of feature amounts; and outputting the similar image; An operating program for a diagnostic support device for causing a computer to execute a process including the steps of:

12. A computer including a processor and a memory connected to or embedded in the processor, obtaining a target image, the target image being a medical image of the brain of a subject to be diagnosed with dementia; Extracting a plurality of anatomical regions of the brain from the subject image; inputting images of a plurality of anatomical regions into a plurality of feature amount derivation models prepared for each of the plurality of anatomical regions, and outputting a plurality of feature amounts for each of the plurality of anatomical regions from the feature amount derivation models; Searching for images similar to a target image using multiple feature amounts; and outputting a similar image.

Images