Prediction device, prediction method, and program

Abstract

Predict risks with high accuracy. [Solution] In the prediction device, the acquisition means acquires data from multiple different modalities for a single subject. The conversion means converts the data from each modality into probability distribution data that shows the probability distribution in the latent space. The completion means, if data from at least one of the multiple modalities is missing, generates probability distribution data for the missing modality using a trained completion model. The integration means integrates the probability distribution data from each modality to generate integrated probability distribution data. The prediction means outputs a prediction result based on the integrated probability distribution data. By using the risk prediction device to predict disease risk, it is possible to support decision-making regarding the lifestyle habits of the subject.

Description

[Technical Field]

[0001] This disclosure relates to the imputation of missing data. [Background technology]

[0002] A technique for predicting disease risk using machine learning models is known. For example, Patent Document 1 describes a multimodal machine learning model that predicts the progression of dementia using multiple types of input data. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. WO2023 / 276976 [Overview of the project] [Problems that the invention aims to solve]

[0004] A challenge with multimodal machine learning models is that predictions are impaired if input data for some of the modalities is missing.

[0005] One objective of this disclosure is to provide a prediction device that can appropriately fill in missing data even when input data for some of the modalities is missing. [Means for solving the problem]

[0006] From one perspective of this disclosure, the prediction device is A means for acquiring data from multiple different modalities for a single subject, A transformation means that converts the data of each modality into probability distribution data that shows the probability distribution in the latent space, A means for generating probability distribution data for the missing modality when data for at least one of the aforementioned multiple modalities is missing, using a trained interpolation model; An integration means for integrating probability distribution data from each modality and generating unified probability distribution data, A prediction means that outputs prediction results based on the aforementioned integrated probability distribution data, It is equipped with.

[0007] In other respects of this disclosure, the computer-based prediction method is: Obtain data from multiple different modalities for a single subject, The data for each modality is converted into probability distribution data that shows the probability distribution in the latent space. If data for at least one of the aforementioned modalities is missing, the trained interpolation model is used to generate the probability distribution data for the missing modality. The probability distribution data for each modality is integrated to generate a unified probability distribution data. Output the prediction results based on the aforementioned integrated probability distribution data.

[0008] In yet another aspect of this disclosure, the program is Obtain data from multiple different modalities for a single subject, The data for each modality is converted into probability distribution data that shows the probability distribution in the latent space. If data for at least one of the aforementioned modalities is missing, the trained interpolation model is used to generate the probability distribution data for the missing modality. The probability distribution data for each modality is integrated to generate a unified probability distribution data. The computer is instructed to perform a process that outputs prediction results based on the aforementioned integrated probability distribution data. [Effects of the Invention]

[0009] According to the present disclosure, even when input data of some modalities is missing among a plurality of modalities, it is possible to appropriately complement the missing data.

Brief Description of the Drawings

[0010] [Figure 1] Shows the overall configuration of the risk prediction device according to the present disclosure. [Figure 2] It is a block diagram showing the hardware configuration of the risk prediction device. [Figure 3] It is a block diagram showing the functional configuration of the learning device of the risk prediction model. [Figure 4] [[ID=!]]It is a flowchart of the learning process of the risk prediction model. [Figure 5] Schematically shows the learning method of the complementation model. [Figure 6] It is a flowchart of the learning process of the complementation model. [Figure 7] It is a block diagram showing the functional configuration of the risk prediction device. [Figure 8] It is a flowchart of the risk prediction process. <00..097> [Figure 9] It is a block diagram showing the functional configuration of the prediction device according to the present disclosure. [Figure 10] It is a flowchart of the prediction process according to the present disclosure.

Mode for Carrying Out the Invention

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. <First Embodiment> [Overall Configuration] It should be noted that there is an error in the original text. In item

[17] , the content in Chinese is "リスク予測モデルの学習処理のフローチャートである.", but the English translation in the provided content is incorrect. The correct English translation should be "It is a flowchart of the learning process of the risk prediction model." I have corrected this error in the translation.Figure 1 shows the overall configuration of the risk prediction device relating to this disclosure. The risk prediction device 100 is an example of the prediction device relating to this disclosure and predicts the disease risk of a subject based on data related to the subject's health. Specifically, the risk prediction device 100 receives multimodal data, that is, data from multiple different modalities. A modality refers to a method or means for representing information. Multimodal data refers to data in different data formats, such as text, images, audio, and sensor data. In this embodiment, the multimodal data includes various data obtained from health checkups, such as the subject's height, weight, gender, blood pressure, BMI (Body Mass Index), body fat percentage, triglyceride levels, cholesterol levels, smoking status and amount, and alcohol consumption status and amount.

[0012] As shown in Figure 1, the risk prediction device 100 receives multiple data from different modalities (in this example, data D1 to D4). The risk prediction device 100 converts the data from each input modality into a probability distribution in the latent space and generates a combined probability distribution (also called the "combined probability distribution" or "latent representation z") by integrating the probability distributions of each modality. The risk prediction device 100 then predicts and outputs the disease risk based on the combined probability distribution.

[0013] During training, the risk prediction device 100 learns to minimize the error between the predicted disease risk value obtained based on the integrated probability distribution and the true disease risk value prepared in advance as training data. At the same time, the risk prediction device 100 learns to make the integrated probability distribution approach a predetermined reference distribution (e.g., a normal distribution).

[0014] Furthermore, during training, a machine learning model that complements the modality data (hereinafter also referred to as the "complementary model") is trained. If data is missing from some of the multiple modalities, it is necessary to compensate for the missing modalities (hereinafter also referred to as the "missing modalities") in order to predict disease risk. Therefore, a complementary model is generated in advance, using data from modalities other than the missing modality as input, to complement the data of the missing modality.

[0015] On the other hand, when predicting risk, the risk prediction device 100 predicts the disease risk of the subject based on multimodal data related to the subject's health. If data is missing from some of the modalities, the risk prediction device 100 generates data for the missing modalities using a pre-trained interpolation model. Then, the risk prediction device 100 predicts the disease risk of the subject using the data from multiple modalities, including the interpolated modality data. As a result, the risk prediction device 100 can predict disease risk with high accuracy even if data for some modalities is missing.

[0016] The risk prediction device 100 can be suitably applied to the medical or healthcare field. For example, the risk prediction device 100 can be used to predict the risk of lifestyle-related diseases based on data obtained from regular health checkups.

[0017] [Hardware configuration] Figure 2 is a block diagram showing the hardware configuration of the risk prediction device 100. As shown in the figure, the risk prediction device 100 comprises a processor 11, an interface (IF) 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a database (DB) 15, and a storage medium 16. Each component is connected to the others, for example, via a bus 18.

[0018] The processor 11 is a computer such as a CPU (Central Processing Unit) and controls the entire risk prediction device 100 by executing a pre-prepared program. Specifically, the processor 11 can be a CPU, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof.

[0019] Furthermore, the processor 11 loads the programs stored in the ROM 13 and storage medium 16 into the RAM 14 and executes each process coded in the programs. The processor 11 functions as part or all of the risk prediction device 100. Specifically, the processor 11 performs the learning process and risk prediction process described later.

[0020] IF12 transmits and receives data to and from external devices. Specifically, during the learning phase, the risk prediction device 100 receives multimodal data from multiple individuals as learning data through IF12. During the prediction phase, i.e., when predicting risk, the risk prediction device 100 receives multimodal data from the target individuals through IF12 and outputs the disease risk prediction results to a display device or other external devices.

[0021] ROM 13 stores various programs executed by processor 11. RAM 14 is used as working memory while processor 11 is executing various processes.

[0022] DB15 stores various algorithms, data, machine learning models, etc., that the risk prediction device 100 uses when it performs the learning process and risk prediction process described later.

[0023] The storage medium 16 is a non-volatile, non-temporary storage medium such as a disk-shaped recording medium or semiconductor memory. The storage medium 16 may be configured to be detachable from the risk prediction device 100. The storage medium 16 records various programs executed by the processor 11.

[0024] In addition to the above, the risk prediction device 100 may also be equipped with a display device such as a liquid crystal display, and an input device such as a keyboard or mouse. These display devices and input devices are used, for example, by the operator of the risk prediction device 100.

[0025] [Learning Phase] Next, the learning phase of the risk prediction device 100 will be described. In this embodiment, the risk prediction device 100 trains two machine learning models. The first model is a risk prediction model that predicts disease risk from multimodal data. The second model is a complementary model that fills in data for missing modalities.

[0026] (Training of risk prediction models) First, we will explain the training of the risk prediction model. The risk prediction device 100 predicts disease risk using the trained risk prediction model. In the following, the risk prediction model will predict disease risk from data D1 to D4 of four different modalities as an example, but the number of data types that make up the multimodal data is not limited to this.

[0027] Figure 3 is a block diagram showing the functional configuration of the learning device 20 for the risk prediction model. The learning device 20 comprises an encoder unit 21, an integration unit 22, a predictor 23, loss calculation units 24 and 25, a loss integration unit 26, an optimization unit 27, and an interpolation unit 30. The encoder unit 21 includes encoders 21a to 21d corresponding to modalities 1 to 4.

[0028] The risk prediction model consists of an encoder unit 21, an integration unit 22, and a predictor 23. Specifically, the encoder unit 21, the integration unit 22, and the predictor 23 are composed of neural networks. In the learning phase, the learning device 20 optimizes this neural network using training data.

[0029] The training data consists of multimodal disease risk data for multiple individuals. Specifically, the training data is collected for multiple individuals, including their attribute data and disease risk values. Attribute data such as height, weight, gender, blood pressure, BMI, triglyceride levels, cholesterol levels, blood glucose levels, smoking habits (both whether or not they smoke and how much), and alcohol consumption (both whether or not they drink) are used, with those most relevant to the disease risk being predicted being utilized. Each individual's disease risk value corresponds to the ground truth data in supervised learning, and this will also be referred to as the "true value" below. For example, suppose we use blood pressure, BMI, and triglyceride levels as data D1-D4 and predict the risk of heart disease. In this case, the training data would include blood pressure, BMI, and triglyceride levels as input data for multiple individuals, and the presence or absence of heart disease as the true value.

[0030] In Figure 3, data D1 to D4 from each modality 1 to 4 are input to the encoder unit 21. Data D1 is input to encoder 21a, data D2 to encoder 21b, data D3 to encoder 21c, and data D4 to encoder 21d. Each encoder 21a to 21d projects the input data into the latent space. The "latent space" is an abstract space for representing the information contained in the original data in fewer dimensions, and in the latent space, the essential features and patterns of the data are represented in fewer dimensions. "Projecting into the latent space" means converting the original data into points on the latent space, and this is also called "mapping to the latent space."

[0031] Next, each encoder 21a to 21d calculates the probability distribution in the latent space of the input modality data D1 to D4 and outputs probability distribution data representing that probability distribution to the interpolation unit 30. Specifically, the probability distribution data includes the mean μ and the standard deviation σ. The probability distribution data (mean μ and standard deviation σ) for each modality is also called the "expert".

[0032] If there are missing data points in the input modality data, the interpolation unit 30 generates missing modality data using the data of the modalities that do not have missing data. Hereinafter, modalities with missing data will be referred to as "missing modalities," and modalities that do not have missing data will be referred to as "non-missing modalities." If there are no missing data points in the input modality data, the interpolation unit 30 outputs the input data directly to the integration unit 22. In the learning phase of the risk prediction model, learning data for multiple modalities without missing data is input, so the interpolation unit 30 outputs the probability distribution data for each input modality directly to the integration unit 22.

[0033] The integration unit 22 integrates the probability distribution data of each modality and generates a latent representation z as an integrated probability distribution. The latent representation z is given by equation (1) below and is also called an intermediate representation, hidden representation, or latent variable.

number

[0034] For the integration unit 22, for example, the configuration of the PoE (Product of Experts) layer described in the following document can be used. (Literature) Microbiome-based disease prediction with multimodal variational information bottlenecks, https: / / doi.org / 10.1371 / journal.pcbi.1010050

[0035] The predictor 23 calculates a disease risk score (hereinafter referred to as the "risk score") S based on the input latent representation z and outputs it to the loss calculation unit 24.

[0036] The loss calculation unit 24 calculates the cross-entropy loss L between the risk score S and the true values ​​corresponding to its input data D1 to D4. cross-entropy This is output to the loss integration unit 26.

[0037] The loss calculation unit 25 calculates the similarity between the probability distribution shown by the latent representation z input from the integration unit 22 and the reference distribution. When the input data D is real data, a normal distribution is used as the reference distribution. Therefore, the loss calculation unit 25 uses the mean μ and standard deviation σ shown by the latent representation z to calculate the KL (Kullback-Leibler) divergence between the probability distribution of each modality and the normal distribution N(0,1) as the loss L using the following equation (2). KL Calculate as follows.

[0038]

number

[0039] If the input data is not real data, the loss calculation unit 25 can use a log-normal distribution, Poisson distribution, multinomial logit, or ordered logit, etc., as the reference distribution, depending on the format of the input data D.

[0040] The loss integration unit 26 calculates the loss L according to the following equation (3): KL and loss L cross-entropy Calculate the weighted sum with respect to the total loss L total This is output to the optimization unit 27.

[0041]

number

[0042] The optimization unit 27 controls the total loss Ltotal Based on this, the encoder unit 21, the integration unit 22, and the predictor 23 are optimized. Specifically, the optimization unit 27 optimizes the parameters of the neural network that constitutes the encoder unit 21, the integration unit 22, and the predictor 23 so that the total loss L total becomes small. Here, since the total loss L total is the weighted sum of the loss L KL and the loss L cross-entropy the optimization unit 27 performs optimization so that the KL divergence between the probability distribution indicated by the latent representation generated by the integration unit 22 and the reference distribution becomes small, that is, so that the similarity between the probability distribution and the reference distribution is high. At the same time, the optimization unit 27 performs optimization so that the error between the risk score S output by the predictor 23 and the true value becomes small.

[0043] Next, the learning process executed by the learning device 20 will be described. FIG. 4 is a flowchart of the learning process of the risk prediction model. This process is realized by the processor 11 shown in FIG. 2 executing a program prepared in advance and operating as the components shown in FIG. 3.

[0044] First, the encoder unit 21 acquires the data of each modality included in the learning data (step S11). Next, the encoder unit 21 projects each data into the latent space by each encoder 21a to 21d to generate an expert (pair of mean μ and standard deviation σ) for each modality (step S12). During learning, since there is no missing data in the input data of each modality, the expert of each modality is input to the integration unit 22 via the complement unit 30. Next, the integration unit 22 integrates the experts of each modality and generates a latent representation z in the latent space (step S13). Next, the predictor 23 calculates a risk score S based on the latent representation z (step S14).

[0045] Next, the loss calculation unit 24 calculates the loss L cross-entropy based on the risk score S and the true value (step S15). Also, the loss calculation unit 25 calculates the loss L KLThe loss L is calculated (step S16). Next, the loss integration unit 26 calculates the loss L cross-entropy and loss L KL Total loss L total The calculation is performed (step S17). Next, the optimization unit 27 calculates the total loss L total Based on this, the parameters of the encoder unit 21, the integration unit 22, and the predictor 23 are optimized (step S18).

[0046] Next, the learning device 20 determines whether predetermined learning termination conditions have been met (step S19). Examples of learning termination conditions include using a predetermined number of attribute data prepared as learning data, the total loss falling below a predetermined value, and the total loss converging. If the learning termination conditions are not met (step S19: No), the process returns to step S11.

[0047] On the other hand, if the learning termination condition is met (step S19: Yes), the learning process terminates. In this way, a risk prediction model is generated.

[0048] (Learning the complementary model) Next, we will explain how to train a complementary model. When there are missing values ​​in the data of multiple input modalities, the complementary model generates missing data for the missing modalities from the data of the non-missing modalities.

[0049] In this embodiment, a completion model is created using a language model. A language model is a mathematical model used in the field of natural language processing to generate sentences or parts of sentences, or to evaluate the correctness of sentences. Specifically, it is a model that predicts the next word or sentence in a given context. For this reason, in this embodiment, the experts of each modality output by the encoder unit 21 are considered as vectorized tokens. That is, the experts of each modality are input to the completion model as tokens. If some of the multiple input modalities have missing data, the completion model predicts and outputs the tokens of the missing modalities based on the tokens of the non-missing modalities. In this way, the completion model can complete the missing modalities.

[0050] Figure 5 schematically illustrates the learning method of the complementary model. First, let's explain the experts. If we denote one of several subjects as "j", then the j-th training data corresponding to that subject contains data from four modalities. In Figure 3, each encoder 21a to 21d outputs experts (pairs of mean μ and standard deviation σ) E1 to E4 corresponding to each modality. Now, let's assume that the latent representation z in the latent space is 3-dimensional. The expert for a subject j includes the mean μj of subject j and the standard deviation σj of subject j. Here, as shown in Figure 5, the mean μj can be represented by a 4x3 matrix with dimensions z1 to z3 in the row direction and experts E1 to E4 in the column direction. Similarly, the standard deviation σj of subject j can be represented by a 4x3 matrix with dimensions z1 to z3 in the row direction and experts E1 to E4 in the column direction.

[0051] Next, we will explain the training of the complementary model. During training, the learning device 31, which functions as a learning means, inputs the j-th expert data corresponding to subject j into the complementary model. At this time, the learning device 31 randomly masks some of the modalities of the input expert data. Masking here is equivalent to intentionally creating missing modalities. In the example in Figure 5, as indicated by the symbol 91, the learning device 31 masks expert E2 of the second modality in the j-th expert data and inputs the missing mean μj and standard deviation σj of expert E2 into the complementary model M.

[0052] The complementary model M predicts data for masked experts in the input based on data from other experts. Constructed using a language model, the complementary model M treats each input expert as a token and predicts the masked expert. In Figure 5, the complementary model M predicts the data for the masked expert E2, as indicated by the symbol 92. The learning device 31 then trains the complementary model M so that the difference between the expert E2 data predicted by the complementary model M and the masked expert E2 data, as indicated by the symbol 91, becomes smaller. In this way, by randomly masking some of the four modalities in multiple expert datasets and performing training, a complementary model that generates data for the masked modality (i.e., missing modality) is obtained.

[0053] As a language model for constructing the completion model, for example, the Masked Language Model (hereinafter referred to as "MLM") of BERT (Bidirectional Encoder Representations from Transformers) can be used. However, the completion model is not limited to the above MLM; any language model can be used as long as it can predict the location of missing parts when some of the input tokens are missing.

[0054] Figure 6 is a flowchart of the learning process for the complementary model. This process is achieved when the processor 11 shown in Figure 2 executes a pre-prepared program and operates as the learning device 31 shown in Figure 5.

[0055] First, the learning device 31 acquires training data containing multiple modalities (step S21). Next, the learning device 31 randomly masks experts of some modalities in the training data (step S22). Next, the learning device 31 inputs the partially masked training data into a complementary model and predicts the masked experts (step S23). Next, the learning device 31 trains the complementary model so that the predicted experts and the masked experts are close together (step S24).

[0056] Next, the learning device 31 determines whether predetermined learning termination conditions have been met (step S25). Examples of learning termination conditions include using a predetermined number of data prepared as learning data, and the error between the predicted expert and the masked expert falling below a predetermined value. If the learning termination conditions are not met (step S25: No), the process returns to step S21, and learning continues using the next learning data. On the other hand, if the learning termination conditions are met (step S25: Yes), the process ends.

[0057] [Prediction Phase] Next, the prediction phase by the risk prediction device will be described. In the prediction phase, the risk prediction device 100 predicts the disease risk of a given subject based on their multimodal data. In this process, the risk prediction device 100 uses the risk prediction model trained in the learning phase, specifically the encoder unit 21, the integration unit 22, and the predictor 23. Furthermore, if some modalities in the multimodal data of a given subject are missing during the prediction phase, the risk prediction device 100 uses the trained completion model to complete the data for the missing modalities and then predicts the risk.

[0058] (Risk prediction device) Figure 7 is a block diagram showing the functional configuration of the risk prediction device. The risk prediction device 100 includes an encoder unit 21, an integration unit 22, and a predictor 23 that are optimized during the learning phase. The risk prediction device 100 also includes a completion unit 30 for completing data for missing modalities. The completion unit 30 uses a trained completion model.

[0059] In the above configuration, the encoder unit 21 is an example of an acquisition means and a conversion means, the integration unit 22 is an example of an integration means, the predictor 23 is an example of a prediction means, and the interpolation unit 30 is an example of an interpolation means.

[0060] (I) When there is no modality deficiency First, let's explain the case where there are no missing modalities in the input data. In this case, the encoder unit 21 receives data D1 to D4 from four different modalities for a given subject. Each encoder 21a to 21d projects the input data D1 to D4 into the latent space, generates probability distribution data (expert) including the mean μ and standard deviation σ, and outputs it to the interpolation unit 30. If there are no missing modalities, the interpolation unit 30 outputs the probability distribution data for each input modality directly to the integration unit 22.

[0061] The integration unit 22 integrates the probability distribution data of each modality and generates a latent representation z as a unified probability distribution obtained by integrating the probability distributions of each modality. The integration unit 22 outputs the latent representation z to the predictor 23.

[0062] The predictor 23 calculates and outputs a risk score S indicating disease risk based on the input latent representation z. In this way, it is possible to predict the disease risk of a subject based on multimodal data.

[0063] (II) When there is a modality deficiency Next, we will explain the case where there are missing modalities in the input data. In this case, the completion unit 30 generates data for the missing modality using a trained completion model. Once the completion unit 30 completes the missing modality expert, the risk prediction device 100 uses the completed expert to predict the disease risk of the subject. The processing of the risk prediction device 100 after the missing expert has been completed is the same as the processing when there are no missing values ​​in the input data.

[0064] (Risk prediction processing) Next, the risk prediction process performed by the risk prediction device 100 described above will be explained. Figure 8 is a flowchart of the risk prediction process. This process is realized when the processor 11 shown in Figure 2 executes a pre-prepared program and operates as the components shown in Figure 7.

[0065] First, the risk prediction device 100 acquires input data for each modality for the subject (step S31). Next, the encoder unit 21 generates experts for each modality from the input data for each modality (step S32). Note that the encoder unit 21 does not generate experts for missing modalities. That is, in step S32, experts for non-missing modalities are generated.

[0066] Next, the risk prediction device 100 determines whether or not there are missing modalities in the input data (step S33). If there are no missing modalities (step S33: No), the process proceeds to step S35. On the other hand, if there are missing modalities (step S33: Yes), the risk prediction device 100 generates an expert for the missing modality using the completion unit 30 (step S34).

[0067] Next, the integration unit 22 integrates the experts of each modality and generates a latent representation z in the latent space (step S35). If there are missing data in the input data, the integration unit 22 will integrate the experts of the non-missing modalities generated in step S32 with the experts of the missing modalities generated in step S34. Next, the predictor 23 calculates and outputs a risk score S based on the latent representation z (step S36). Then the risk prediction process is completed.

[0068] [Differentiation] (Variation 1) In the first embodiment described above, the predictor 23 predicts the subject's risk of disease based on the integrated latent representation z. In this case, the predictor 23 predicts and outputs the risk of heart disease from, for example, height, weight, blood pressure, and cholesterol levels.

[0069] Alternatively, the predictor 23 may be configured to predict other health data of the subject. Specifically, the predictor 23 may be configured to predict health data of a modality different from the modality of the input data D1 to D4. In this case, the predictor 23 may predict and output triglyceride levels from, for example, height, weight, blood pressure, and cholesterol levels.

[0070] Alternatively, the predictor 23 may be configured to predict the missing modality health data of the subject. Specifically, the predictor 23 may be configured to output the missing modality data from the input data D1 to D4. In this case, the predictor 23 may, for example, predict and output the input data D3 (blood pressure) if it is missing from the input data D1 (height), D2 (weight), D3 (blood pressure), and D4 (cholesterol level).

[0071] (Modification 2) In the first embodiment described above, the risk prediction device is applied to the generation of attribute data related to human health, but the application of this disclosure is not limited to this. For example, this disclosure may be applied to the inspection and diagnosis of machines and devices. That is, the method of this disclosure may be applied to estimate the state of a machine or device based on data from multiple modalities detected and collected in an inspection or diagnosis.

[0072] <Second Embodiment> Figure 9 is a block diagram showing the functional configuration of the prediction device according to the second embodiment. The risk prediction device 70 comprises an acquisition means 71, a conversion means 72, a supplementary means 73, an integration means 74, and a prediction means 75.

[0073] Figure 10 is a flowchart of the processing performed by the prediction device of the second embodiment. The acquisition means 71 acquires data from multiple different modalities for a single subject (step S71). The conversion means 72 converts the data from each modality into probability distribution data that shows the probability distribution in the latent space (step S72). The completion means 73 generates probability distribution data for the missing modality using a trained completion model if data from at least one of the multiple modalities is missing (step S73). The integration means 74 integrates the probability distribution data from each modality to generate integrated probability distribution data (step S74). The prediction means 75 outputs a prediction result based on the integrated probability distribution data (step S75).

[0074] According to the risk prediction device 70 of the second embodiment, it is possible to predict risk with high accuracy even when there are missing parts in the input data.

[0075] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0076] (Note 1) A means for acquiring data from multiple different modalities for a single subject, A transformation means that converts the data of each modality into probability distribution data that shows the probability distribution in the latent space, A means for generating probability distribution data for the missing modality when data for at least one of the aforementioned multiple modalities is missing, using a trained interpolation model; An integration means for integrating probability distribution data from each modality and generating unified probability distribution data, A prediction means that outputs prediction results based on the aforementioned integrated probability distribution data, A prediction device equipped with the following features.

[0077] (Note 2) The complementary model is a prediction device according to Appendix 1 that generates probability distribution data of the missing modalities based on the data of the non-missing modalities among the plurality of modalities.

[0078] (Note 3) The aforementioned complementary model is a predictive device described in Appendix 2, which is a language model that takes probability distribution data for each modality as input and outputs probability distribution data for the missing modality.

[0079] (Note 4) The prediction device according to Appendix 1, comprising a learning means for learning the complementary model by machine learning using the learning data of the aforementioned multiple modalities.

[0080] (Note 5) The prediction device according to Appendix 4, wherein the learning means masks the data of some of the modalities among the learning data of the plurality of modalities and inputs it into the complementary model, and trains the complementary model based on the predicted data output by the complementary model and the masked data.

[0081] (Note 6) The prediction means is a prediction device according to Appendix 1 that predicts the disease risk of a subject based on data from multiple modalities related to the subject's health.

[0082] (Note 7) The prediction means is a prediction device according to Appendix 1, which predicts data of other modalities related to the health of a subject based on data of multiple modalities related to the health of the subject.

[0083] (Note 8) The prediction means is a prediction device according to Appendix 1, which predicts the data of missing modalities from among the data of multiple modalities related to the subject's health, based on the data of modalities that are not missing.

[0084] (Note 9) A prediction method performed by a computer, Obtain data from multiple different modalities for a single subject, The data for each modality is converted into probability distribution data that shows the probability distribution in the latent space. If data for at least one of the aforementioned modalities is missing, the trained interpolation model is used to generate the probability distribution data for the missing modality. The probability distribution data for each modality is integrated to generate a unified probability distribution data. A prediction method that outputs prediction results based on the aforementioned integrated probability distribution data.

[0085] (Note 10) Obtain data from multiple different modalities for a single subject, The data for each modality is converted into probability distribution data that shows the probability distribution in the latent space. If data for at least one of the aforementioned modalities is missing, the trained interpolation model is used to generate the probability distribution data for the missing modality. The probability distribution data for each modality is integrated to generate a unified probability distribution data. A program that causes a computer to perform a process to output prediction results based on the aforementioned integrated probability distribution data.

[0086] Furthermore, some or all of the configurations described in Appendices 2 to 8, which are subordinate to Appendice 1 above, may also be subordinate to Appendices 9 and 10 in the same way as those described in Appendices 2 to 8. Moreover, not limited to Appendices 1, 9, and 10, some or all of the configurations described as appendices may also be subordinate to various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.

[0087] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure. [Explanation of Symbols]

[0088] 11 processors 20 Learning device 21 Encoder section 21a~21d Encoders 22 Integration Department 23 Predictors 24, 25 Loss calculation section 26 Loss integration section 27 Optimization Section 30 Supplementary section 31 Learning device 100 Risk Prediction Devices

Claims

1. A means for acquiring data from multiple different modalities for a single subject, A transformation means that converts the data of each modality into probability distribution data that shows the probability distribution in the latent space, A means for generating probability distribution data for the missing modality when data for at least one of the aforementioned multiple modalities is missing, using a trained interpolation model; An integration means for integrating probability distribution data from each modality and generating unified probability distribution data, A prediction means that outputs prediction results based on the aforementioned integrated probability distribution data, A prediction device equipped with the following features.

2. The prediction device according to claim 1, wherein the complementary model generates probability distribution data of the missing modalities based on the data of the non-missing modalities among the plurality of modalities.

3. The prediction device according to claim 2, wherein the complementary model is a language model that takes probability distribution data of each modality as input and outputs probability distribution data of the missing modality.

4. The prediction device according to claim 1, further comprising a learning means for learning the complementary model by machine learning using the learning data of the plurality of modalities.

5. The prediction device according to claim 4, wherein the learning means masks the data of some of the modalities among the learning data of the plurality of modalities and inputs it into the complementary model, and trains the complementary model based on the predicted data output by the complementary model and the masked data.

6. The prediction device according to claim 1, wherein the prediction means predicts the disease risk of a subject based on data from multiple modalities related to the subject's health.

7. The prediction device according to claim 1, wherein the prediction means predicts data of other modalities related to the health of a subject based on data of multiple modalities related to the health of the subject.

8. The prediction device according to claim 1, wherein the prediction means predicts the data of missing modalities from among the data of multiple modalities related to the health of the subject, based on the data of modalities that are not missing.

9. A prediction method performed by a computer, Obtain data from multiple different modalities for a single subject, The data for each modality is converted into probability distribution data that shows the probability distribution in the latent space. If data for at least one of the aforementioned modalities is missing, the trained interpolation model is used to generate the probability distribution data for the missing modality. The probability distribution data for each modality is integrated to generate a unified probability distribution data. A prediction method that outputs prediction results based on the aforementioned integrated probability distribution data.

10. Obtain data from multiple different modalities for a single subject, The data for each modality is converted into probability distribution data that shows the probability distribution in the latent space. If data for at least one of the aforementioned modalities is missing, the trained interpolation model is used to generate the probability distribution data for the missing modality. The probability distribution data for each modality is integrated to generate a unified probability distribution data. A program that causes a computer to perform a process to output prediction results based on the aforementioned integrated probability distribution data.

Images