Information processing device, method of operating the information processing device, operating program for the information processing device, predictive model, learning device, and learning method

Abstract

A prediction model equipped with a processor is used, in which the processor includes a feature amount extraction unit that acquires a medical image obtained by capturing an image of an organ in a subject and data associated with a disease in the subject, divides the medical image into a plurality of patch images and extracts a feature amount from the patch images and the data associated with the disease, and a correlation information extraction unit that extracts at least correlation information between the plurality of patch images and correlation information between the plurality of patch images and the data associated with the disease. The patch images and the data associated with the disease are input into the prediction model, and prediction results associated with the disease are output from the prediction model.

Description

Technical Field

[0001] The technology of the present disclosure relates to an information processing device, a method of operating the information processing device, an operating program of the information processing device, a prediction model, a learning device, and a learning method.

Background Art

[0002] In response to the advent of a full-fledged aging society, the development of prediction models for assisting in the diagnosis of diseases such as dementia represented by Alzheimer's disease and predicting the progression of dementia has been actively promoted. For example, <Goto, T., Wang, C., Li, Y. & Tsuboshita, Y. Multi-modal deep learning for predicting progression of Alzheimer’s disease using bi-linear shake fusion, Proc. SPIE (Medical Imaging) 11314, 452-457 (2020).> (hereinafter referred to as Document 1) describes a so-called multi-modal type prediction model in which a tomographic image (hereinafter referred to as an MRI image) obtained by magnetic resonance imaging (MRI) of the brain of a subject for predicting the progression of dementia, and dementia-related data such as the age, sex, genetic test data, and cognitive function test data (score of a cognitive ability test) of the subject are input, and a prediction result of the progression of dementia is output.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The brain has various anatomical regions such as the hippocampus, parahippocampal gyrus, amygdala, frontal lobe, temporal lobe, and occipital lobe. And the relationship between each anatomical region and cognitive ability is different. However, the prediction model described in Document 1 deals with an MRI image of the whole brain and does not consider anatomical regions.

[0004] Therefore, one possible approach is to subdivide MRI images into multiple patch images, input them into a prediction model, and then extract the features of each of the multiple patch images using the prediction model. However, even with this method, the prediction model described in Reference 1 is structurally unable to utilize correlation information between multiple patch images, and correlation information between multiple patch images and dementia-related data (and, if there are multiple dementia-related data as described above, correlation information between multiple dementia-related data) for prediction, and thus could not significantly improve the accuracy of predicting the progression of dementia.

[0005] One embodiment of the technology of this disclosure provides an information processing device, a method for operating the information processing device, an operating program for the information processing device, a predictive model, a learning device, and a learning method that can improve the prediction accuracy of disease prediction results by a predictive model. [Means for solving the problem]

[0006] The information processing device of the present disclosure includes a processor, which acquires medical images of the subject's organs and disease-related data of the subject, subdivides the medical images into multiple patch images, and uses a prediction model that includes a feature extraction unit that extracts features from the patch images and disease-related data, and a correlation information extraction unit that extracts at least correlation information between the multiple patch images and correlation information between the multiple patch images and disease-related data, inputs the patch images and disease-related data into the prediction model, and outputs a prediction result regarding the disease from the prediction model.

[0007] The predictive model preferably has a transformer encoder that takes in input data containing a mixture of patch images and disease-related data and extracts features.

[0008] Preferably, the feature extraction unit includes a self-attention mechanism layer of the transformer encoder, and the correlation information extraction unit includes a linear transformation layer that linearly transforms the input data to the self-attention mechanism layer to obtain first transformed data, an activation function application layer that applies an activation function to the first transformed data to obtain second transformed data, and an calculation unit that calculates the element-wise product of the output data from the self-attention mechanism layer and the second transformed data as correlation information.

[0009] The disease is dementia, the medical image is an image of the subject's brain, and it is preferable for the processor to extract a first-segment image from the medical image, including the hippocampus, amygdala, and entorhinal cortex, and a second-segment image including the temporal lobe and frontal lobe, and then subdivide the first-segment image and the second-segment image into multiple patch images.

[0010] The disease is dementia, the medical images are morphological imaging data, and the disease-related data preferably includes at least one of the following: the subject's age, sex, blood and cerebrospinal fluid test data, genetic test data, and cognitive function test data.

[0011] Morphological imaging data is preferably tomographic images obtained by nuclear magnetic resonance imaging.

[0012] The method of operating the information processing device of the present disclosure includes: acquiring medical images of the subject's organs and disease-related data of the subject; subdividing the medical images into multiple patch images; using a prediction model that includes a feature extraction unit for extracting features from the patch images and disease-related data, and a correlation information extraction unit for extracting at least correlation information between the multiple patch images and correlation information between the multiple patch images and disease-related data; and inputting the patch images and disease-related data into the prediction model and outputting a prediction result regarding the disease from the prediction model.

[0013] The operating program for the information processing device of this disclosure causes a computer to perform the following processes: acquiring medical images of the subject's organs and disease-related data of the subject; subdividing the medical images into multiple patch images; using a prediction model that includes a feature extraction unit for extracting features from the patch images and disease-related data, and a correlation information extraction unit for extracting at least correlation information between the multiple patch images and correlation information between the multiple patch images and disease-related data; and inputting the patch images and disease-related data into the prediction model and outputting a prediction result regarding the disease from the prediction model.

[0014] The predictive model of this disclosure includes a feature extraction unit that extracts features from multiple patch images obtained by subdividing medical images of the subject's organs, and from the subject's disease-related data, and a correlation information extraction unit that extracts at least correlation information between the multiple patch images and correlation information between the multiple patch images and the disease-related data, and causes the computer to function to output a prediction result regarding the disease in response to the input of patch images and disease-related data.

[0015] The learning device of this disclosure is a learning device that provides a predictive model with training medical images and training disease-related data as training data, and learns the predictive model to output a disease prediction result in response to input of patch images obtained by subdividing medical images showing the organs of a subject and disease-related data of the subject. The predictive model includes a feature extraction unit that extracts features from the patch images and disease-related data, and a correlation information extraction unit that extracts at least correlation information between multiple patch images and correlation information between multiple patch images and disease-related data.

[0016] The learning method of this disclosure is a learning method that provides a predictive model with training medical images and training disease-related data as training data, and trains the predictive model to output a disease prediction result in response to input of patch images obtained by subdividing medical images showing the organs of a subject and disease-related data of the subject, the predictive model includes a feature extraction unit that extracts features from patch images and disease-related data, and a correlation information extraction unit that extracts at least correlation information between multiple patch images and correlation information between multiple patch images and disease-related data. [Effects of the Invention]

[0017] The technology disclosed herein provides an information processing device, a method for operating the information processing device, an operating program for the information processing device, a predictive model, a learning device, and a learning method that can improve the prediction accuracy of disease prediction results by a predictive model. [Brief explanation of the drawing]

[0018] [Figure 1] This diagram shows an information processing server and user terminals. [Figure 2] This figure shows data related to dementia. [Figure 3] This figure shows the prediction results. [Figure 4] This is a block diagram showing the computers that make up an information processing server. [Figure 5] This is a block diagram showing the CPU processing unit of an information processing server. [Figure 6] This diagram conceptually illustrates the processing of the patch image generation unit. [Figure 7] This block shows the detailed configuration of the prediction model. [Figure 8] This diagram shows the detailed configuration of a transformer encoder. [Figure 9] This is a diagram showing the detailed configuration of the first structural section. [Figure 10] This diagram shows an overview of the processing during the training phase of a predictive model. [Figure 11]It is a flowchart showing the processing procedure of an information processing server. [Figure 12] It is a block diagram showing the processing unit of the CPU of the information processing server of the second embodiment and an outline of the processing.

Mode for Carrying Out the Invention

[0019] [First Embodiment] As an example, as shown in FIG. 1, the information processing server 10 is connected to the user terminal 11 via the network 12. The information processing server 10 is an example of an “information processing apparatus” according to the technology of the present disclosure. The user terminal 11 is installed in, for example, a medical facility and is operated by a doctor who diagnoses dementia, particularly Alzheimer's disease, in the medical facility.

[0020] Dementia is an example of a “disease” according to the technology of the present disclosure. Examples of dementia include Alzheimer's disease, Lewy body dementia, and vascular dementia. The content of the diagnosis may be used for Alzheimer's disease other than Alzheimer's disease. Specifically, from the preclinical stage of Alzheimer's disease (PAD: Preclinical Alzheimer's disease), mild cognitive impairment due to Alzheimer's disease (MCI (Mild Cognitive Impairment) due to Alzheimer's disease) can be mentioned. As the disease, a neurodegenerative disease such as the exemplified dementia is preferable.

[0021] Furthermore, diagnostic criteria for dementia include those described in the "Dementia Disease Treatment Guidelines 2017" supervised by the Japanese Society of Neurology, the "International Statistical Classification of Diseases and Related Health Problems (ICD)-11" (11th edition), the "Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5)" by the American Psychiatric Association, and the "National Institute on Aging-Alzheimer's Association workgroup (NIA-AA) criteria." These diagnostic criteria can be referenced, and their contents are incorporated into this specification.

[0022] Data related to the diagnostic criteria for dementia include cognitive function test data, morphological imaging test data, brain function imaging test data, blood and cerebrospinal fluid test data, and genetic test data. Cognitive function test data includes scores from the Clinical Dementia Rating-Sum of Boxes (CDR-SOB), Mini-Mental State Examination (MMSE), and Alzheimer's Disease Assessment Scale (ADAS-Cog - cognitive subscale). Morphological imaging test data includes MRI images16 or tomographic images of the brain obtained by computed tomography (CT) (hereinafter referred to as CT images).

[0023] Brain function imaging data includes cross-sectional images of the brain obtained by positron emission tomography (PET) (hereinafter referred to as PET images) and cross-sectional images of the brain obtained by single-photon emission computed tomography (SPECT) (hereinafter referred to as SPECT images). Blood and cerebrospinal fluid (CSF) test data includes the amount of p-tau (phosphorylated tau protein) 181 in the cerebrospinal fluid (hereinafter abbreviated as CSF). Genetic test data includes the results of ApoE gene genotype testing.

[0024] The user terminal 11 has a display 13 and input devices 14 such as a keyboard and mouse. The network 12 is a WAN (Wide Area Network), such as the Internet or a public communication network. In Figure 1, only one user terminal 11 is connected to the information processing server 10, but in reality, multiple user terminals 11 from multiple medical facilities are connected to the information processing server 10.

[0025] The user terminal 11 sends a prediction request 15 to the information processing server 10. The prediction request 15 is a request for the information processing server 10 to predict the progression of dementia using the prediction model 41 (see Figure 5). The prediction request 15 includes an MRI image 16 and dementia-related data 17. The MRI image 16 and dementia-related data 17 are data from the day the prediction request 15 is sent. However, the MRI image 16 and dementia-related data 17 may also be data from the most recent date of the prediction request 15, for example, data from 3 days to 1 week prior to the date of the prediction request 15.

[0026] MRI image 16 is an image of the brain of a subject whose dementia progression is predicted. MRI image 16 is voxel data representing the three-dimensional shape of the subject's brain (see Figure 6). MRI image 16 is an example of "medical image" and "morphological imaging data" related to the technology disclosed herein. The brain is also an example of an "organ" related to the technology disclosed herein.

[0027] Dementia-related data 17 is data concerning the dementia of the subject. MRI images 16 are obtained, for example, from a PACS (Picture Archiving and Communication System) server. Dementia-related data 17 are obtained, for example, from an electronic medical record server. Alternatively, dementia-related data 17 are entered by a physician operating an input device 14. Dementia-related data 17 is an example of "disease-related data" related to the technology of this disclosure. Although not shown in the diagram, the prediction request 15 also includes terminal ID (Identification Data) to uniquely identify the user terminal 11 that sent the prediction request 15.

[0028] When the information processing server 10 receives a prediction request 15, it uses the prediction model 41 to predict the progression of dementia in the subject and derives a prediction result 18. The information processing server 10 delivers the prediction result 18 to the user terminal 11 that sent the prediction request 15. Upon receiving the prediction result 18, the user terminal 11 displays the prediction result 18 on the display 13 and makes the prediction result 18 available for viewing by the doctor.

[0029] As an example, as shown in Figure 2, dementia-related data 17 includes the subject's age, sex, genetic test data, cognitive function test data, and CSF test data. Genetic test data is, for example, the result of testing the genotype of the ApoE gene. The genotype of the ApoE gene is a combination of two of the three types of ApoE genes (ε2 and ε3, ε3 and ε4, etc.). Compared to subjects with a genotype that does not have any ε4 (ε2 and ε3, ε3 and ε3, etc.), subjects with a genotype that has one or two ε4 genes (ε2 and ε4, ε4 and ε4, etc.) have approximately 3 to 12 times the risk of developing Alzheimer's disease. Cognitive function test data is, for example, the CDR-SOB score. CSF test data is, for example, the amount of p-tau (phosphorylated tau protein) 181 in CSF. CSF test data is an example of "blood and cerebrospinal fluid test data" related to the technology of this disclosure.

[0030] As an example, as shown in Figure 3, prediction result 18 indicates whether the subject will develop Alzheimer's disease within two years or not.

[0031] As an example, as shown in Figure 4, the computer comprising the information processing server 10 includes storage 30, memory 31, CPU (Central Processing Unit) 32, communication unit 33, display 34, and input device 35. These are interconnected via a bus line 36.

[0032] Storage 30 is a hard disk drive built into the computer constituting the information processing server 10, or connected via cable or network. Alternatively, storage 30 is a disk array consisting of multiple hard disk drives installed in series. Storage 30 stores control programs such as the operating system, various application programs, and various data associated with these programs. A solid-state drive may be used instead of a hard disk drive.

[0033] Memory 31 is work memory for the CPU 32 to execute processing. The CPU 32 loads the program stored in storage 30 into memory 31 and executes processing according to the program. In this way, the CPU 32 comprehensively controls each part of the computer. CPU 32 is an example of a "processor" related to the technology of this disclosure. Note that memory 31 may be built into the CPU 32.

[0034] The communication unit 33 controls the transmission of various types of information to external devices such as the user terminal 11. The display 34 displays various screens. These screens are equipped with GUI (Graphical User Interface) operation functions. The computers constituting the information processing server 10 receive operation instructions from input devices 35 through the various screens. The input devices 35 include keyboards, mice, touch panels, and microphones for voice input.

[0035] As an example, as shown in Figure 5, the storage 30 of the information processing server 10 stores an operating program 40. The operating program 40 is an application program that causes the computer to function as an information processing server 10. In other words, the operating program 40 is an example of an "operating program for an information processing device" related to the technology of this disclosure. The storage 30 also stores a prediction model 41.

[0036] When the operating program 40 is started, the CPU 32 of the computer constituting the information processing server 10 works in cooperation with the memory 31 and the like to function as a reception unit 45, a read / write (hereinafter abbreviated as RW (Read Write)) control unit 46, a patch image generation unit 47, a prediction unit 48, and a distribution control unit 49.

[0037] The reception unit 45 receives a prediction request 15 from the user terminal 11. As mentioned above, the prediction request 15 includes an MRI image 16 and dementia-related data 17. Therefore, by receiving the prediction request 15, the reception unit 45 acquires the MRI image 16 and dementia-related data 17. The reception unit 45 outputs the acquired MRI image 16 and dementia-related data 17 to the RW control unit 46. The reception unit 45 also outputs the terminal ID of the user terminal 11 (not shown) to the distribution control unit 49.

[0038] The RW control unit 46 controls the storage of various data to the storage 30 and the reading of various data from the storage 30. For example, the RW control unit 46 stores the MRI image 16 and dementia-related data 17 from the reception unit 45 in the storage 30. The RW control unit 46 also reads the MRI image 16 and dementia-related data 17 from the storage 30, outputs the MRI image 16 to the patch image generation unit 47, and outputs the dementia-related data 17 to the prediction unit 48. Furthermore, the RW control unit 46 reads the prediction model 41 from the storage 30 and outputs the prediction model 41 to the prediction unit 48.

[0039] As an example, as shown in Figure 6, the patch image generation unit 47 subdivides the MRI image 16 into multiple patch images 55. Each patch image 55 has a size of, for example, 8 pixels × 8 pixels × 8 pixels. The patch image generation unit 47 outputs a patch image group 55G, which is a collection of multiple patch images 55, to the prediction unit 48.

[0040] The prediction unit 48 inputs the patch image group 55G and dementia-related data 17 into the prediction model 41 and outputs a prediction result 18 from the prediction model 41. The prediction unit 48 outputs the prediction result 18 to the distribution control unit 49.

[0041] The distribution control unit 49 controls the distribution of the prediction result 18 to the user terminal 11 that sent the prediction request 15. At this time, the distribution control unit 49 identifies the user terminal 11 that sent the prediction request 15 based on the terminal ID from the reception unit 45.

[0042] As an example, as shown in Figure 7, the prediction model 41 includes a patch image linear projection unit 60, a dementia-related data linear projection unit 61, a transformer encoder 62, a sequence pooling unit 63, and a multi-layer perceptron (MLP) head 64. The patch image linear projection unit 60 converts each of the multiple patch images 55 constituting the patch image group 55G into sequence data and then performs linear projection. Specifically, the patch image linear projection unit 60 first converts each patch image 55 into a one-dimensional vector. Then, it linearly projects each one-dimensional patch image 55 onto a multi-dimensional, for example, 64-dimensional tensor through a filter. The filter used for linear projection is learned during the learning phase of the prediction model 41 (see Figure 10). The patch image linear projection unit 60 outputs the multiple tensor data (called patch embeddings) 70 obtained by linearly projecting each patch image 55 in this way to the transformer encoder 62. In this process, positional information 71 is added to the tensor data 70 (this is called positional embedding). The positional information 71 is used to identify the location of the patch image 55 within the MRI image 16.

[0043] The dementia-related data linear projection unit 61 converts each of the subject's age, sex, genetic test data, cognitive function test data, and CSF test data, which constitute the dementia-related data 17, into sequence data and then performs linear projection. Specifically, the dementia-related data linear projection unit 61 first converts each of the dementia-related data 17 into a one-dimensional vector. Then, it linearly projects each of the one-dimensional dementia-related data 17 onto a multi-dimensional, for example, 64-dimensional tensor through a filter. Similar to the patch image linear projection unit 60, the filter used for linear projection is learned during the learning phase of the prediction model 41. The dementia-related data linear projection unit 61 then outputs the tensor data 72 obtained by linearly projecting each of the dementia-related data 17 to the transformer encoder 62. In other words, the transformer encoder 62 receives both the tensor data 70 based on the patch image 55 and the tensor data 72 based on the dementia-related data 17 as input simultaneously. Hereinafter, the set of tensor data 70, location information 71, and tensor data 72 will be referred to as the first input data 73_1. The first input data 73_1 is an example of "input data containing a mixture of patch images and dementia-related data" related to the technology of this disclosure.

[0044] The transformer encoder 62 extracts feature quantities 74 from the first input data 73_1. Feature quantities 74 are a collection of multiple values, for example, several thousand to several hundred thousand. The transformer encoder 62 outputs feature quantities 74 to the sequence pooling unit 63. The transformer encoder 62 is trained during the training phase of the prediction model 41.

[0045] The sequence pooling unit 63 calculates a statistical value of the feature vectors 74, in this case the mean value, and outputs the calculated mean value as the aggregated feature vector 74G to the multilayer perceptron head 64. Note that the statistical value is not limited to the mean value; it may also be the maximum value, etc.

[0046] The multilayer perceptron head 64 converts the aggregated features 74G into prediction results 18. The multilayer perceptron head 64 is trained during the training phase of the prediction model 41.

[0047] As an example, as shown in Figure 8, the transformer encoder 62 includes a plurality of structural parts 80, such as a first structural part 80_1, a second structural part 80_2, ..., and an Nth structural part 80_N (where N is a natural number greater than or equal to 2). These plurality of structural parts 80 have the same structure.

[0048] The first structural unit 80_1 receives the first input data 73_1. Based on the first input data 73_1, the first structural unit 80_1 outputs the first output data 81_1. _1 This is input to the second structural unit 80_2. That is, the first output data 81_1 is also the second input data 73_2 of the second structural unit 80_2. The second structural unit 80_2 outputs the second output data 81_2 based on the second input data 73_2. The second output data 81_2 is input to the third structural unit (not shown). That is, the second output data 81_2 is also the third input data 73_3 of the third structural unit. In this way, the output data 81 of the preceding structural unit 80 is repeatedly input to the subsequent structural unit 80 as input data 73. Finally, the Nth output data 81_N is output from the Nth structural unit 80_N. This Nth output data 81_N is none other than the feature quantity 74, which is the final output of the transformer encoder 62.

[0049] As an example, as shown in Figure 9, the first structural unit 80_1 includes a feature extraction unit 85, a correlation information extraction unit 86, a multilayer perceptron 87, and an additive unit 88. The feature extraction unit 85 includes a self-attention mechanism layer 90. The correlation information extraction unit 86 includes a linear transformation layer 91, an activation function application layer 92, and an arithmetic unit 93. As mentioned above, the other structural units 80 have the same structure as the first structural unit 80_1, so the first structural unit 80_1 will be described as a representative example below.

[0050] The self-attention mechanism layer 90 receives the first input data 73_1. As is well known, the self-attention mechanism layer 90 obtains the queries, keys, and values ​​of each tensor data 70 and 72 of the first input data 73_1 and calculates the similarity of the queries and keys. As a result, the self-attention mechanism layer 90 generates an attention weight map that shows the correspondence between each patch image 55 and the dementia-related data 17. The attention weight map is a set of numbers between 0 and 1 that indicate which of the first input data 73_1 should be paid attention to. The self-attention mechanism layer 90 treats the numbers in the attention weight map as probabilities and calculates the correspondence between queries and values ​​to obtain the first input data 73_1 as intermediate output data 95. The self-attention mechanism layer 90 outputs the intermediate output data 95 to the arithmetic unit 93. The intermediate output data 95 is an example of "output data from the self-attention mechanism layer" relating to the technology of this disclosure.

[0051] The first input data 73_1 is also input to the linear transformation layer 91. The linear transformation layer 91 performs a linear transformation on the first input data 73_1 to obtain the first transformed data 96. The linear transformation layer 91 outputs the first transformed data 96 to the activation function application layer 92.

[0052] The activation function application layer 92 applies an activation function, such as a sigmoid function, to the first transformed data 96 to obtain the second transformed data 97. The activation function application layer 92 outputs the second transformed data 97 to the calculation unit 93.

[0053] The calculation unit 93 calculates the element-wise product of the intermediate output data 95 from the self-attention mechanism layer 90 and the second transformation data 97 from the activation function application layer 92. The result 98 of this element-wise product of the intermediate output data 95 and the second transformation data 97 is correlation information between multiple patch images 55, correlation information between each of the multiple patch images 55 and the dementia-related data 17, and correlation information between each of the dementia-related data 17. The calculation unit 93 outputs the calculation result 98 to the multilayer perceptron 87.

[0054] The multilayer perceptron 87 performs a linear transformation on the calculation result 98 and outputs it to the adder 88. The adder 88 adds the first input data 73_1 and the linearly transformed calculation result 98 to obtain the first output data 81_1. As described above, the first output data 81_1 is input to the second structure unit 80_2 as the second input data 73_2.

[0055] Thus, the prediction model 41 causes the computer to perform the following: a feature extraction process by a feature extraction unit 85, which extracts features 74 from multiple patch images 55 obtained by subdividing an MRI image 16 of the subject's brain, and from the subject's dementia-related data 17; a correlation information extraction process by a correlation information extraction unit 86, which extracts correlation information between the multiple patch images 55 and calculation results 98 as correlation information between the multiple patch images 55 and the dementia-related data 17; and a prediction result output process by a multilayer perceptron head 64, which outputs a prediction result 18 regarding dementia in response to the input of the patch images 55 and the dementia-related data 17.

[0056] As an example, as shown in Figure 10, the predictive model 41 is trained in the learning phase by being given training data (also called teacher data or training data) 100. The training data 100 is a set of training MRI images 16L, training dementia-related data 17L, and ground truth data 18CA. The training MRI images 16L and training dementia-related data 17L are, for example, MRI images 16 and dementia-related data 17 of a sample subject (including patients) stored in a database such as ADNI (Alzheimer's Disease Neuroimaging Initiative). The ground truth data 18CA is the diagnosis of Alzheimer's disease actually made by a physician to the sample subject.

[0057] During the learning phase, the prediction model 41 receives learning MRI images 16L and learning dementia-related data 17L as input. The prediction model 41 outputs learning prediction results 18L based on the learning MRI images 16L and learning dementia-related data 17L. Based on the learning prediction results 18L and the ground truth data 18CA, the prediction model 41 performs a loss calculation. Then, various coefficients of the prediction model 41 are updated according to the results of the loss calculation, and the prediction model 41 is updated according to the update settings.

[0058] In the learning phase, the above series of processes—inputting the learning MRI images 16L and learning dementia-related data 17L into the prediction model 41, outputting the learning prediction results 18L from the prediction model 41, calculating the loss, setting the update, and updating the prediction model 41—are repeated, with the learning data 100 being exchanged at least twice. The repetition of the above series of processes ends when the prediction accuracy of the learning prediction results 18L for the ground truth data 18CA reaches a predetermined set level. The prediction model 41, whose prediction accuracy has reached the set level, is stored in the storage 30 and used by the prediction unit 48. Alternatively, learning may be terminated after the above series of processes has been repeated a set number of times, regardless of the prediction accuracy of the learning prediction results 18L for the ground truth data 18CA.

[0059] Next, the operation of the above configuration will be explained with reference to the flowchart in Figure 11. First, when the operating program 40 is started in the information processing server 10, the CPU 32 of the information processing server 10 functions as the reception unit 45, the RW control unit 46, the patch image generation unit 47, the prediction unit 48, and the distribution control unit 49, as shown in Figure 5.

[0060] First, the reception unit 45 receives a prediction request 15 from the user terminal 11, and as a result, the MRI image 16 and dementia-related data 17 are acquired (step ST100). The MRI image 16 and dementia-related data 17 are output from the reception unit 45 to the RW control unit 46 and stored in the storage 30 under the control of the RW control unit 46.

[0061] The RW control unit 46 reads the MRI image 16 and dementia-related data 17 from the storage 30. The MRI image 16 is output from the RW control unit 46 to the patch image generation unit 47. The dementia-related data 17 is output from the RW control unit 46 to the prediction unit 48.

[0062] As shown in Figure 6, in the patch image generation unit 47, the MRI image 16 is subdivided into multiple patch images 55 (step ST110). The patch image group 55G, which is a collection of multiple patch images 55, is output from the patch image generation unit 47 to the prediction unit 48.

[0063] As shown in Figure 7, in the prediction unit 48, the patch image group 55G and dementia-related data 17 are input to the prediction model 41, and the prediction model 41 outputs the prediction result 18 (step ST120). The prediction result 18 is output from the prediction unit 48 to the distribution control unit 49, and under the control of the distribution control unit 49, it is distributed to the user terminal 11 that sent the prediction request 15 (step ST130). In the user terminal 11, the prediction result 18 is displayed on the display 13 and made available for the physician to view.

[0064] As described above, the CPU 32 of the information processing server 10 includes a reception unit 45, a patch image generation unit 47, and a prediction unit 48. The reception unit 45 receives a prediction request 15 and obtains an MRI image 16 of the brain of a subject whose dementia progression is to be predicted, and dementia-related data 17 regarding the subject's dementia. The patch image generation unit 47 subdivides the MRI image 16 into multiple patch images 55. The prediction unit 48 uses a prediction model 41 that includes a feature extraction unit 85 and a correlation information extraction unit 86. The feature extraction unit 85 extracts features 74 from the patch images 55 and the dementia-related data 17. The correlation information extraction unit 86 extracts calculation results 98 as correlation information between the multiple patch images 55 and between each of the multiple patch images 55 and the dementia-related data 17. The prediction unit 48 inputs the patch images 55 and the dementia-related data 17 into the prediction model 41 and outputs a prediction result 18 of the progression of dementia from the prediction model 41. The correlation information between multiple patch images 55, and the correlation information between each of the multiple patch images 55 and dementia-related data 17, can be effectively used to predict the progression of dementia. Therefore, it is possible to improve the prediction accuracy of the dementia prediction results 18 by the prediction model 41.

[0065] Transformer encoders are models that have achieved state-of-the-art performance (SOA) in various fields of natural language processing, and recently have been applied not only to natural language processing but also to image processing. Transformer encoders applied to image processing are called vision transformer (ViT) encoders. Vision transformer encoders treat patch images, which are subdivided images, in the same way as words in natural language processing. Vision transformer encoders can significantly reduce the computational cost of training compared to conventional models, such as those using convolutional neural networks, and have higher prediction accuracy than conventional models. In the technology disclosed herein, a first input data 73_1, which contains a mixture of patch images 55 and dementia-related data 17, is input to a transformer encoder 62 that has this vision transformer encoder mechanism, and the transformer encoder 62 is made to extract features 74. As a result, it is possible to train with a larger amount of training data 100 in a shorter time, and the prediction accuracy of the dementia-related prediction results 18 by the prediction model 41 can be further improved.

[0066] The feature extraction unit 85 includes a self-attention mechanism layer 90 of the transformer encoder 62. The correlation information extraction unit 86 includes a linear transformation layer 91, an activation function application layer 92, and a calculation unit 93. The linear transformation layer 91 linearly transforms the input data 73 to the self-attention mechanism layer 90 to obtain first transformed data 96. The activation function application layer 92 applies an activation function to the first transformed data 96 to obtain second transformed data 97. The calculation unit 93 calculates the element-wise product of the intermediate output data 95 from the self-attention mechanism layer 90 and the second transformed data 97. As a result, it is possible to easily obtain the calculation result 98, which is correlation information between multiple patch images 55, correlation information between each of the multiple patch images 55 and the dementia-related data 17, and correlation information between each of the dementia-related data 17.

[0067] Morphological imaging data, such as MRI images 16, are taken by almost all individuals with dementia. Therefore, if medical images are used as morphological imaging data such as MRI images 16, there will be no shortage of training data 100 for the predictive model 41, and the training of the predictive model 41 will proceed smoothly.

[0068] The progression of dementia varies depending on age, sex, blood and cerebrospinal fluid (CSF) test data (CSF test data in this example), and genetic test data. Furthermore, cognitive function test data serves as a good indicator for predicting the progression of dementia. Therefore, including the subject's age, sex, blood and cerebrospinal fluid (CSF) test data, genetic test data, and cognitive function test data in the dementia-related data 17 can further improve the prediction accuracy of the dementia prediction results 18 by the prediction model 41. Note that the dementia-related data 17 only needs to include at least one of the subject's age, sex, blood and cerebrospinal fluid (CSF) test data, genetic test data, and cognitive function test data.

[0069] [Second Embodiment] As an example, as shown in Figure 12, the CPU of the information processing server in the second embodiment functions as a region image extraction unit 110 in addition to the processing units 45-49 of the first embodiment (only the patch image generation unit 47 is shown in Figure 12). The region image extraction unit 110 is located before the patch image generation unit 47. The region image extraction unit 110 receives the MRI image 16 from the RW control unit 46. The region image extraction unit 110 extracts a first region image 111 and a second region image 112 from the MRI image 16, for example, using a semantic segmentation model that class-labels each anatomical region of the brain. The first region image 111 is an image of a brain region mainly centered on the hippocampus, and includes the hippocampus, amygdala, and entorhinal cortex. The second region image 112 is an image of a brain region mainly centered on the temporal lobe, and includes the temporal lobe and frontal lobe. The area image extraction unit 110 outputs the first area image 111 and the second area image 112 to the patch image generation unit 47.

[0070] The patch image generation unit 47 subdivides the first region image 111 into a plurality of first patch images 113. The patch image generation unit 47 also subdivides the second region image 112 into a plurality of second patch images 114. Therefore, in this case, the patch image group 115G consists of a first patch image group 113G, which is a collection of a plurality of first patch images 113, and a second patch image group 114G, which is a collection of a plurality of second patch images 114. The patch image generation unit 47 outputs the patch image group 115G to the prediction unit 48. The subsequent processing is the same as in the first embodiment described above, so the explanation is omitted.

[0071] Here, the hippocampus is involved in memory and spatial learning abilities. The amygdala plays a major role in the formation and storage of memories associated with emotional events. The entorhinal cortex is a region necessary for episodic memory to function properly.

[0072] The temporal lobe is an essential area for auditory perception, language reception, visual memory, verbal memory, and emotion. For example, a lesion in the right temporal lobe generally impairs the ability to interpret nonverbal auditory stimuli (e.g., music). Similarly, a lesion in the left temporal lobe significantly impairs language recognition, memory, and construction. The frontal lobe is responsible for initiating or inhibiting actions. It also plays a role in organizing, planning, processing, and judging information necessary for daily life. Furthermore, the ability to perceive oneself objectively, to experience emotions, and to speak are all functions of the frontal lobe.

[0073] In the second embodiment, the region image extraction unit 110 extracts a first region image 111, including the hippocampus, amygdala, and entorhinal cortex, and a second region image 112, including the temporal lobe and frontal lobe, from the MRI image 16. Then, the patch image generation unit 47 subdivides the first region image 111 into a plurality of first patch images 113 and subdivides the second region image 112 into a plurality of second patch images 114. The first patch images 113 and the second patch images 114 include anatomical regions that are important for predicting the progression of dementia, such as the hippocampus, amygdala, entorhinal cortex, temporal lobe, and frontal lobe. Therefore, the prediction accuracy of the dementia prediction result 18 by the prediction model 41 can be further improved.

[0074] Medical images are not limited to MRI images 16. In place of, or in addition to, MRI images 16, other morphological imaging data such as CT images, PET images, or brain function imaging data such as SPECT images may also be used.

[0075] Cognitive function test data may include scores from the Rivermead Behavioral Memory Test (RBMT), Activities of Daily Living (ADL), etc. Alternatively, cognitive function test data may include ADAS-Cog scores, MMSE scores, etc. Multiple types of cognitive function test data may be included in the dementia-related data.17

[0076] CSF test data is not limited to the amount of p-tau181 as exemplified. The amount of t-tau (total tau protein) or Aβ42 (amyloid-beta protein) can also be used.

[0077] Prediction Result 18 is not limited to whether the example subject will develop or not develop Alzheimer's disease within two years. For example, it could also indicate whether the progression of Alzheimer's disease in the subject after three years will be fast or slow. It could also be the probability of normal, mild cognitive impairment, or Alzheimer's disease. It could also be the change in cognitive function test data.

[0078] Prediction Result 18 may be limited to Alzheimer's disease, or more generally, it may state whether the subject is normal, in the pre-symptomatic stage, has mild cognitive impairment, or has dementia. Subjective cognitive impairment (SCI) and / or subjective cognitive decline (SCD) may also be included as predictors. Alternatively, it may state whether the subject will progress from normal or pre-symptomatic to MCI, or whether the subject will progress from normal, pre-symptomatic, or MCI to Alzheimer's disease.

[0079] The predictions include forecasts of cognitive function, such as how much a subject's cognitive function will decline in two years, and forecasts of the risk of developing dementia, such as the subject's risk of developing dementia.

[0080] Instead of distributing the prediction result 18 itself from the information processing server 10 to the user terminal 11, screen data including the prediction result 18 may be distributed from the information processing server 10 to the user terminal 11. Furthermore, the manner in which the prediction result 18 is made available for viewing by physicians is not limited to the method of distributing the prediction result 18 to the user terminal 11. Printed copies of the prediction result 18 may be provided to physicians, or an email with the prediction result 18 attached may be sent to the physician's mobile terminal.

[0081] The training of the predictive model 41 shown in Figure 10 may be performed on the information processing server 10 or on a device other than the information processing server 10. Furthermore, the training of the predictive model 41 may be continued even after deployment. When the training of the predictive model 41 is performed on the information processing server 10, the information processing server 10 is an example of a "training device" related to the technology of this disclosure. When the training of the predictive model 41 is performed on a device other than the information processing server 10, the device other than the information processing server 10 is an example of a "training device" related to the technology of this disclosure.

[0082] The information processing server 10 may be installed in each medical facility, or it may be installed in a data center independent of the medical facilities. Furthermore, the user terminal 11 may perform some or all of the functions of the processing units 45-49 of the information processing server 10.

[0083] While dementia is used as an example of a disease, it is not limited to this. The disease could be, for example, cerebral infarction. In this case, CT or MRI images of the subject's brain, along with disease-related data such as the subject's age and sex, are input into the predictive model, and the model outputs the change in the score of the National Institutes of Health Stroke Scale (NIHSS) or the Japan Stroke Scale (JSS) as the predictive result. Preferably, the disease is a neurological disease including dementia and cerebral infarction as exemplified, or neurodegenerative diseases such as Parkinson's disease and cerebrovascular diseases. Thus, the prediction includes prediction of disease progression and / or prediction for supporting the diagnosis of the disease.

[0084] However, dementia has become a social problem with the arrival of the aging society in recent years. Therefore, this example, which identifies dementia as the disease, can be said to be a form that matches the current social problem.

[0085] Diseases are not limited to neurological disorders, and therefore the organs involved are not limited to the brain.

[0086] In each of the above embodiments, the hardware structure of the Processing Unit that performs various processes, such as the reception unit 45, RW control unit 46, patch image generation unit 47, prediction unit 48, distribution control unit 49, and area image extraction unit 110, can be the following types of processors. As mentioned above, the types of processors include a CPU 32, which is a general-purpose processor that executes software (operation program 40) and functions as various processing units, as well as programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to perform specific processes.

[0087] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, and / or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0088] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, which then functions as multiple processing units, as exemplified by client and server computers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System-on-a-Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0089] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits, which are combinations of circuit elements such as semiconductor devices.

[0090] From the above description, the technology described in the following supplementary information can be understood.

[0091] [Additional note 1] Equipped with a processor, The aforementioned processor, Medical images of the subject's organs and disease-related data of the subject are obtained. The aforementioned medical image is subdivided into multiple patch images, A prediction model is used that includes a feature extraction unit for extracting features from the patch images and disease-related data, and a correlation information extraction unit for extracting at least correlation information between a plurality of patch images and correlation information between a plurality of patch images and disease-related data. The patch image and disease-related data are input into the prediction model, and the prediction model outputs a prediction result regarding the disease. Information processing device. [Additional note 2] The information processing device according to Appendix 1, which has a transformer encoder that takes in input data containing a mixture of the patch image and the disease-related data and extracts the feature quantities. [Additional note 3] The feature extraction unit includes the self-attention mechanism layer of the transformer encoder. The correlation information extraction unit, A linear transformation layer that linearly transforms the input data to the self-attention mechanism layer to obtain first transformed data, An activation function application layer that applies an activation function to the first transformed data to obtain second transformed data, The information processing apparatus according to Appendix 2, which includes a calculation unit that calculates the element-wise product of the output data from the self-attention mechanism layer and the second conversion data as the correlation information. [Additional note 4] The aforementioned disease is dementia. The aforementioned medical image is an image of the subject's brain. The aforementioned processor, From the aforementioned medical images, a first-segment image including the hippocampus, amygdala, and entorhinal cortex, and a second-segment image including the temporal lobe and frontal lobe are extracted. An information processing device according to any one of the appendix 1 to 3, wherein the first area image and the second area image are subdivided into the plurality of patch images. [Additional note 5] The aforementioned disease is dementia. The aforementioned medical images are morphological imaging examination data. The information processing device according to any one of Appendix 1 to Appendix 4, wherein the disease-related data includes at least one of the following: the age, sex, blood / cerebrospinal fluid test data, genetic test data, and cognitive function test data of the subject. [Additional note 6] The morphological image inspection data is a tomographic image obtained by nuclear magnetic resonance imaging, as described in Appendix 5 of the information processing device.

[0092] The technology of this disclosure can be appropriately combined with the various embodiments and / or variations described above. Furthermore, it is understood that various configurations can be adopted without departing from the spirit of the invention, and the invention is not limited to the embodiments described above. Moreover, the technology of this disclosure extends not only to programs but also to storage media for storing programs non-temporarily.

[0093] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0094] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0095] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. Equipped with a processor, The aforementioned processor, Medical images of the subject's organs and disease-related data of the subject are obtained. The aforementioned medical image is subdivided into multiple patch images, A prediction model is used that includes a feature extraction unit for extracting features from the patch images and disease-related data, and a correlation information extraction unit for extracting at least correlation information between a plurality of patch images and correlation information between a plurality of patch images and disease-related data. The patch image and disease-related data are input into the prediction model, and the prediction model outputs a prediction result regarding the disease. Information processing device.

2. The information processing apparatus according to claim 1, comprising a transformer encoder that takes in input data containing a mixture of the patch image and the disease-related data and extracts the feature quantities.

3. The feature extraction unit includes the self-attention mechanism layer of the transformer encoder. The correlation information extraction unit, A linear transformation layer that linearly transforms the input data to the self-attention mechanism layer to obtain first transformed data, An activation function application layer that applies an activation function to the first transformed data to obtain second transformed data, The information processing apparatus according to claim 2, which includes a calculation unit that calculates the element-wise product of the output data from the self-attention mechanism layer and the second conversion data as the correlation information.

4. The aforementioned disease is dementia. The aforementioned medical image is an image of the subject's brain. The aforementioned processor, From the aforementioned medical image, a first-segment image including the hippocampus, amygdala, and entorhinal cortex, and a second-segment image including the temporal lobe and frontal lobe are extracted. The information processing apparatus according to claim 1, wherein the first area image and the second area image are subdivided into a plurality of patch images.

5. The aforementioned disease is dementia. The aforementioned medical images are morphological imaging examination data. The information processing apparatus according to claim 1, wherein the disease-related data includes at least one of the subject's age, sex, blood / cerebrospinal fluid test data, genetic test data, and cognitive function test data.

6. The information processing apparatus according to claim 5, wherein the morphological image inspection data is a tomographic image obtained by nuclear magnetic resonance imaging.

7. To obtain medical images of the subject's organs and disease-related data of the subject. The aforementioned medical image is subdivided into multiple patch images. A predictive model is used that includes a feature extraction unit that extracts features from the patch images and disease-related data, and a correlation information extraction unit that extracts at least correlation information between a plurality of the patch images and correlation information between the plurality of the patch images and the disease-related data, The patch image and disease-related data are input into the prediction model, and the prediction model outputs a prediction result regarding the disease. A method for operating an information processing device, including the device itself.

8. To obtain medical images of the subject's organs and disease-related data of the subject. The aforementioned medical image is subdivided into multiple patch images. A predictive model is used that includes a feature extraction unit that extracts features from the patch images and disease-related data, and a correlation information extraction unit that extracts at least correlation information between a plurality of the patch images and correlation information between the plurality of the patch images and the disease-related data, The patch image and disease-related data are input into the prediction model, and the prediction model outputs a prediction result regarding the disease. An operating program for an information processing device that causes a computer to perform a process that includes [specific details].

9. A feature extraction unit that extracts features from multiple patch images obtained by subdividing medical images of the subject's organs, and from the subject's disease-related data, A correlation information extraction unit that extracts at least correlation information between multiple patch images and correlation information between multiple patch images and disease-related data, Includes, The computer is configured to output predictive results regarding the disease in response to the input of the patch image and the disease-related data. Predictive model.

10. By providing training medical images and training disease-related data to the predictive model as training data, This learning device trains a predictive model to output predictive results regarding a disease, based on input of patch images obtained by subdividing medical images of a subject's organs and disease-related data of the subject. The aforementioned prediction model, A feature extraction unit that extracts features from the patch image and the disease-related data, Includes a correlation information extraction unit that extracts at least correlation information between multiple patch images and correlation information between multiple patch images and disease-related data, Learning device.

11. By providing training medical images and training disease-related data to the predictive model as training data, This learning method involves training a predictive model to output disease-related predictions based on the input of patch images obtained by subdividing medical images of a subject's organs and disease-related data of the subject. The aforementioned prediction model, A feature extraction unit that extracts features from the patch image and the disease-related data, Includes a correlation information extraction unit that extracts at least correlation information between multiple patch images and correlation information between multiple patch images and disease-related data, Learning methods.

Images