Programs, methods, and devices

Abstract

This provides a technique for training multimodal models using batches of limited modality combinations. [Solution] The program causes the controller to perform operations including: grouping training data samples into multiple batches of training data samples; grouping data types into multiple modalities of data types; generating multiple modality lists; obtaining feature codings corresponding to the modalities in order to create a multimodal model; integrating the feature codings of each modality in the modality list to create latent variables; estimating the results; determining the cost by comparing the estimated results with ground truth; determining the divergence by comparing the latent variables with a multidimensional Gaussian distribution; and tuning the parameters of a stochastic encoder and a logistic regression decoder.

Description

Technical Field

[0001] The present disclosure relates to training a multimodal model using batches of limited modality combinations.

Background Art

[0002] Predicting the risk of lifestyle diseases can contribute to disease prevention. In recent years, a large amount of diverse (multimodal) body measurement data has become available. Based on general body measurement data, individuals who recognize a significant risk of a particular lifestyle disease may have a better chance of preventing the disease.

Summary of the Invention

Problems to be Solved by the Invention

[0003] There is a need for a technology that can train a multimodal model using batches of limited modality combinations.

Means for Solving the Problems

[0004] Training a multimodal model using a limited set of modality combinations involves grouping training data samples into batches of training data samples, where each training data sample contains data values ​​corresponding to each of several data types; grouping data types into several modalities of the data type; generating several modality lists, where each modality list contains one or more modalities; and generating a multimodal model for estimating results from incomplete data samples for each of the several modality lists. To create the features, the following steps are performed: for each modality in the modality list, apply a stochastic encoder to the data values ​​corresponding to the modality in a batch of training samples from multiple batches to obtain the feature encoding corresponding to the modality; integrate the feature encoding of each modality in the modality list to create a latent variable; apply a logistic regression decoder to the latent variable to estimate the result, determine the cost by comparing the estimated result with ground truth, determine the divergence by comparing the latent variable with a multidimensional Gaussian distribution; and adjust the parameters of the stochastic encoder and logistic regression decoder based on the cost and divergence.

[0005] The aspects of this disclosure will be best understood from the following detailed description when read in conjunction with the attached figures. Note that, in accordance with standard industry practice, various features are not depicted to scale. In fact, for clarity in the description, the dimensions of various features may be arbitrarily enlarged or reduced. [Brief explanation of the drawing]

[0006] [Figure 1] This is a schematic diagram of a multimodal model according to at least some embodiments of the subject disclosure.

[0007] [Figure 2]This is a schematic diagram of a sample of general anthropometric training data according to at least some embodiments of the present disclosure.

[0008] [Figure 3] This is an operational flow for training a multimodal model using a batch of limited modality combinations, according to at least some embodiments of the present disclosure.

[0009] [Figure 4] This is a schematic diagram of data type modalities according to at least some embodiments of the present disclosure.

[0010] [Figure 5] This is a schematic diagram of modality combinations according to at least some embodiments of the subject matter disclosure.

[0011] [Figure 6] This is a schematic diagram of a modality list according to at least some embodiments of the present disclosure.

[0012] [Figure 7] This is an operational flow for creating a multimodal model according to at least some embodiments of the subject disclosure.

[0013] [Figure 8] This is an operational flow for applying a model to a training data sample, according to at least some embodiments of the subject disclosure.

[0014] [Figure 9] This is an operational flow for tuning model parameters according to at least some embodiments of the present disclosure.

[0015] [Figure 10]A block diagram of a hardware configuration for training a multimodal model using a batch of limited modality combinations, according to at least some embodiments of the present disclosure. **DETAILED DESCRIPTION**

[0016] The following disclosure provides many different embodiments or examples for implementing different features of the provided subject matter. To simplify the present disclosure, specific examples of components, values, operations, materials, or arrangements are described below. Of course, these are merely examples and are not intended to be limiting. Other components, values, operations, materials, or arrangements, etc. are conceivable. In addition, the present disclosure may repeat reference numbers and / or letters in various examples. This repetition is for the purpose of simplification and clarity and does not in itself determine the relationship between the various embodiments and / or configurations described.

[0017] In the applications known to the inventors, not all modalities are always available. One technique known to the inventors for training a multimodal model to be effective even when there are missing modalities is to incorporate all possible modality combinations into the loss function during training. However, as the number of modalities increases, the number of combinations increases exponentially according to the following (Equation 1).

[0018] **[Equation]**

[0019] In (Equation 1), n is the number of combinations and k is the number of modalities, leading to an increase in computational resource requirements.

[0020] In at least some embodiments of the present disclosure, only a certain number of combinations are incorporated into the loss function during batch training of the multimodal model.

[0021] In accordance with at least some embodiments of this subject disclosure, training a multimodal model using a limited combination of modalities reduces computational resource requirements, while the accuracy of the trained multimodal model is approximately the same as that of a multimodal model trained using all possible combinations of modalities.

[0022] Figure 1 is a schematic diagram of a multimodal model according to at least some embodiments of the present subject disclosure. The multimodal model includes a modality grouper 110, encoders 112A, 112B, and 112N, a feature coding integrator 114, a decoder 116, data samples 100, modalities 102A, 102B, and 102N, feature codings 104A, 104B, and 104N, latent variables 106, and estimated results 108.

[0023] Data sample 100 is the input to the multimodal model. In at least some embodiments, data sample 100 represents individual data points containing various anthropometric measurements and their values. In at least some embodiments, data sample 100 is as described with respect to Figure 2.

[0024] The modality grouper 110 is a component of a multimodal model. In at least some embodiments, the modality grouper 110 is of a type implemented in data management systems and preprocessing tools that handle diverse datasets. In at least some embodiments, the modality grouper 110 is configured to organize health data and prepare datasets for analysis. In at least some embodiments, the modality grouper 110 is configured to categorize data types into distinct modalities. In at least some embodiments, the modality grouper 110 is configured to group data types into modalities.

[0025] Modalities 102A, 102B, and 102N are outputs of modality grouper 110 and input to encoders 112A, 112B, and 112N. In at least some embodiments, modalities 102A, 102B, and 102N are groupings of data values ​​from data sample 100. In at least some embodiments, modalities 102A, 102B, and 102N are as described with respect to Figure 4.

[0026] Encoders 112A, 112B, and 112N are components of a multimodal model. In at least some embodiments, encoders 112A, 112B, and 112N are of a type typically used in machine learning and are designed for feature extraction. In at least some embodiments, encoders 112A, 112B, and 112N are configured to transform raw data into usable features. In at least some embodiments, encoders 112A, 112B, and 112N are configured to encode data values ​​into feature representations that capture the essential characteristics of the input data. In at least some embodiments, encoders 112A, 112B, and 112N are configured to handle different data types, such as numerical and categorical data, and to perform dimensionality reduction. In at least some embodiments, encoders 112A, 112B, and 112N are trained to provide feature encodings representing their respective modalities to an integrator, such as a feature encoding integrator 114.

[0027] The feature codes 104A, 104B, and 104N are the outputs of the encoders 112A, 112B, and 112N, and are input to the feature coding integrator 114. In at least some embodiments, the feature codes 104A, 104B, and 104N contain essential characteristics of the input data. In at least some embodiments, the feature codes 104A, 104B, and 104N represent the mean and variance.

[0028] The feature coding integrator 114 is a component of a multimodal model. In at least some embodiments, the feature coding integrator 114 is of a type implemented in data fusion frameworks and statistical analysis tools that combine multiple data sources. In at least some embodiments, the feature coding integrator 114 is configured to combine feature codes from various modalities to create latent variables. In at least some embodiments, the feature coding integrator 114 is configured to receive feature codes such as feature codes 104A, 104B, and 104N, and to transmit the resulting latent variables such as latent variables 106.

[0029] The latent variable 106 is the output of the feature coding integrator 114 and is input to the decoder 116. In at least some embodiments, the latent variable 106 is of a type that can be represented by latent variable models and statistical modeling tools that analyze complex data relationships. In at least some embodiments, the latent variable 106 represents an underlying pattern in the data to facilitate predictive modeling.

[0030] The decoder 116 is a component of the multimodal model. In at least some embodiments, the decoder 116 is of a type used in predictive modeling software and risk assessment tools that generate outcomes based on input data. In at least some embodiments, the decoder 116 is configured to estimate disease risk and interpret model predictions. In at least some embodiments, the decoder 116 is configured to decode latent variables to produce estimated outcomes, often using logistic regression techniques. In at least some embodiments, the decoder 116 is configured to receive latent variables, such as latent variable 106, and transmit estimated outcomes, such as estimated outcome 108.

[0031] The estimated result 10⁸ is the output of the multimodal model. In at least some embodiments, the estimated result 10⁸ represents the likelihood of the outcome. In at least some embodiments, the estimated result 10⁸ represents the disease risk of an individual. In at least some embodiments, once the multimodal model is trained, the operator can apply the multimodal model to a live data sample containing data values ​​corresponding to fewer data types than all of the multiple data types.

[0032] Figure 2 is a schematic diagram of a general anthropometric training data sample according to at least some embodiments of the present disclosure. The general anthropometric training data sample includes data values ​​such as data value 220 and target data value 222. Each data value represents a data type. For example, data value 220 has the value AGE_GROUP and represents the data type "Age (Degree of Separation: 5 years)". In at least some embodiments, all data values ​​except one or more data values ​​that directly indicate a lifestyle-related disease are used as input to train a multimodal model for predicting lifestyle-related diseases. Target data value 222 represents the data type "Pre-meal blood glucose (fasting blood glucose)", which directly indicates diabetes. In at least some embodiments, all data values ​​of the training data sample except target data value 222 are used as input to train a multimodal model to estimate the risk of diabetes. In at least some embodiments, target data value 222 is used as ground truth to determine the loss in order to train a multimodal model to estimate the risk of diabetes.

[0033] Figure 3 shows an operational flow for training a multimodal model using a batch of limited modality combinations, according to at least some embodiments of the present disclosure. In at least some embodiments, the operational flow provides a method for training a multimodal model using a batch of limited modality combinations. In at least some embodiments, the method is performed by a controller of an apparatus, such as controller 1082 of apparatus 1080 in Figure 10, which will be described later.

[0034] In S330, the controller or a section thereof groups the training data samples into batches. In at least some embodiments, the controller shuffles the training data samples, defines a batch size, and assigns the samples to batches. In at least some embodiments, the controller utilizes the batch size parameter to effectively organize the data. In at least some embodiments, the resulting output consists of batches of training samples ready for processing in subsequent operations. In at least some embodiments, the controller performs grouping according to the variable properties of the batch size and the shuffling method. In at least some embodiments, changes in these properties lead to faster training times as the batch size increases, but also affect the model's ability to generalize to batches that do not represent the entire dataset. In at least some embodiments, the controller groups the training data samples from a plurality of training data samples into a plurality of batches of training data samples, each training data sample containing data values ​​corresponding to each of a plurality of data types. In at least some embodiments, each training data sample from a plurality of training data samples corresponds to a person, and each data type from a plurality of data types is an anthropometric measurement.

[0035] In S332, the controller or a section thereof groups data types into modalities. In at least some embodiments, the controller identifies data types. In at least some embodiments, the controller categorizes data types into modalities. In at least some embodiments, the controller creates mappings of these modalities. In at least some embodiments, the controller relies on a list of data types and a definition of a given modality. In at least some embodiments, the controller groups data types from among multiple data types into multiple modalities of the data type.

[0036] In S334, the controller or a section thereof generates a modality list. In at least some embodiments, the controller determines combinations of modalities. In at least some embodiments, generating multiple modality lists includes determining possible combinations of modalities. In at least some embodiments, the controller creates each list based on a predetermined number of combinations. In at least some embodiments, the enumerated modalities of each of the multiple modality lists are included in a predetermined number of modality combinations. In at least some embodiments, the controller stores the modality lists in memory. In at least some embodiments, generating multiple modality lists includes storing multiple modality lists in memory. In at least some embodiments, the controller selects combinations for each list and distributes the modalities evenly among the lists. In at least some embodiments, the controller generates multiple modality lists, each modality list containing one or more modalities. In at least some embodiments, the multiple modality lists include an even distribution of modalities among the multiple modalities.

[0037] In S336, the controller or its section creates a multimodal model. In at least some embodiments, the controller applies the multimodal model to the training data samples, calculates a loss based on a comparison of the estimated results output from the multimodal model with the ground truth of the training data samples, and adjusts the parameters of the multimodal model according to the calculated loss. In at least some embodiments, the controller performs operations for each modality list of a plurality of modality lists to create a multimodal model for estimating results from incomplete data samples. In at least some embodiments, the controller performs the operation flow shown in Figure 7.

[0038] Figure 4 is a schematic diagram of data type modalities according to at least some embodiments of the subject disclosure. The data type modalities include modalities 440, 442, 444, 446, 448, and 449. In at least some embodiments, modalities 440, 442, 444, 446, 448, and 449 include all data types of the training data samples except for one or more target data types.

[0039] Figure 5 is a schematic diagram of modality combinations according to at least some embodiments of the present disclosure. A modality combination includes all possible modality combinations, such as modality combination 524. Modality combination 524 includes modality 440 and modality 442. In at least some embodiments, the number of modality combinations is calculated according to (Equation 1) above. In at least some embodiments, a combination may include a single modality, two modalities, three modalities, and so on, and one combination with all modalities.

[0040] Figure 6 is a schematic diagram of a modality list according to at least some embodiments of the present disclosure. In at least some embodiments, each modality list, such as modality list 626, includes a list number and a combination. In at least some embodiments, each modality list includes a predetermined number of combinations. Modality list 626 includes three combinations: C1, C13, and C33. Combination C1 may include modality 440, combination C13 may include modalities 442 and 446, and combination C33 may include modalities 444, 448, and 449.

[0041] Figure 7 shows an operation flow for creating a multimodal model according to at least some embodiments of the present disclosure. In at least some embodiments, the operation flow provides a method for creating a multimodal model. In at least some embodiments, the method is performed by a controller of the device, such as the controller 1082 of the device 1080 in Figure 10, which will be described later.

[0042] In S750, the controller or its section proceeds to the next batch. In at least some embodiments, the controller proceeds to the next batch of training data samples by checking any remaining batches and loading the next set of training data samples. In at least some embodiments, the controller utilizes a batch data structure to organize the training data samples for processing.

[0043] In S752, the controller or its section applies the model to the training data samples. In at least some embodiments, the controller applies the multimodal model to the training data samples by inputting data values ​​from the training data samples into the multimodal model and performing a forward pass to output the estimated results. In at least some embodiments, the controller performs the operation flow shown in Figure 8 below.

[0044] In S754, the controller or its section adjusts the model parameters. In at least some embodiments, the controller adjusts the model parameters by calculating the gradient and updating the weights based on the loss function. In at least some embodiments, the controller uses the following loss function for one modality:

[0045]

number

[0046] In (Equation 2), J IBAssume that is the loss, N is the number of training data samples in the batch, ε~N(0,I) is the auxiliary Gaussian noise variable, KL is the Kullback-Leibler divergence, f is the vector-valued parametric deterministic coding function, and q(y|z) and r(z) are variational approximations of the true p(y|z) and p(z), respectively. In at least some embodiments, the controller determines the loss for each modality as follows:

[0047]

number

[0048] In (Equation 3), x A and x B is a modality in the combination of modality lists. In at least some embodiments, the controller has a cost J AI and divergence J BI Based on this, the loss is determined as follows:

[0049]

number

[0050] In (Equation 4), J I is the total loss, and a and b are hyperparameter coefficients. In at least some embodiments, the controller performs the operation flow shown in Figure 9 below.

[0051] In S756, the controller or its section determines whether there are remaining batches. In at least some embodiments, the controller checks for remaining batches by evaluating the batch count and deciding whether to continue or terminate the training process. In at least some embodiments, the controller utilizes a batch counter. In response to the controller determining that there are remaining batches, the operation flow returns to S750 and proceeds to the next batch. In response to the controller determining that there are no remaining batches, the operation flow terminates.

[0052] Figure 8 shows an operational flow for applying a model to a training data sample according to at least some embodiments of this subject disclosure. In at least some embodiments, the operational flow provides a method for applying a model to a training data sample. In at least some embodiments, the method is performed by a controller of an apparatus, such as controller 1082 of apparatus 1080 in Figure 10, which will be described later.

[0053] In S860, the controller or its section proceeds to the next list. In at least some embodiments, the controller proceeds by identifying the next modality list to be processed and loading the corresponding modality.

[0054] In S861, the controller or its section proceeds to the next modality. In at least some embodiments, the controller selects the next modality from the current modality list and prepares associated data values ​​from the training data samples for processing.

[0055] In S862, the controller or a section thereof applies a probabilistic encoder to a modality. In at least some embodiments, the controller applies the corresponding probabilistic encoder to the selected modality. In at least some embodiments, the controller determines which probabilistic encoder corresponds to the selected modality. In at least some embodiments, for each modality in the modality list, the controller applies the probabilistic encoder to the data values ​​corresponding to the modality in a batch of training samples from a batch of batches to obtain the feature encoding corresponding to the modality.

[0056] In S863, the controller or its section determines whether there are any remaining modalities. In at least some embodiments, the controller checks if there are any remaining modalities to process. In at least some embodiments, this involves evaluating a list of modalities to determine whether any additional modalities are available for processing. In response that the controller determines there are remaining modalities, the operation flow returns to S861 and proceeds to the next modality. In response that the controller determines there are no remaining modalities, the operation flow proceeds to feature coding integral in S865.

[0057] In S865, the controller or its section integrates the feature coding. In at least some embodiments, the controller integrates the feature coding obtained from each modality in the modality list to generate a latent variable. In at least some embodiments, the latent variable encapsulates the integrated information from the modalities. In at least some embodiments, the controller creates a latent variable by integrating the feature coding of each modality in the modality list.

[0058] In S876, the controller or its section applies a logistic regression decoder to estimate the results. In at least some embodiments, the controller applies the logistic regression decoder to latent variables to estimate the results. In at least some embodiments, the controller applies the logistic regression decoder to latent variables to estimate the risk of lifestyle-related diseases. In at least some embodiments, the logistic regression decoder estimates the risk score for lifestyle-related diseases.

[0059] In S868, the controller or its section determines whether there are any remaining lists. In at least some embodiments, the controller checks if there are any remaining modality lists to process. In response to the controller determining that there are remaining lists, the operation flow returns to S860 and proceeds to the next modality list. In response to the controller determining that there are no remaining lists, the operation flow terminates.

[0060] Figure 9 shows an operational flow for tuning model parameters according to at least some embodiments of the present disclosure. In at least some embodiments, the operational flow provides a method for tuning model parameters. In at least some embodiments, the method is performed by a controller of an apparatus, such as controller 1082 of apparatus 1080 in Figure 10, which will be described later.

[0061] In S970, the controller or its section determines the cost. In at least some embodiments, the controller determines the cost J according to the loss functions (Equation 2) and (Equation 3). AI The cost is calculated. In at least some embodiments, the controller determines the cost by comparing the estimated result with ground truth.

[0062] In S974, the controller or its section determines the divergence. In at least some embodiments, the controller determines the cost J according to the loss functions (Equation 2) and (Equation 3).BI The latent variable is calculated. In at least some embodiments, the controller determines the divergence by comparing the latent variable to a multidimensional Gaussian distribution.

[0063] In S978, the controller or its section tunes the parameters of the encoder and decoder. In at least some embodiments, the controller updates the parameters of both the encoder and decoder components of the model. In at least some embodiments, the controller applies an optimization algorithm to improve the parameters based on the calculated cost and divergence. In at least some embodiments, the controller tunes the parameters using backpropagation and gradient descent. In at least some embodiments, the controller tunes the parameters of the stochastic encoder and logistic regression decoder based on the cost and divergence.

[0064] Figure 10 is a block diagram of a hardware configuration for training a multimodal model using a batch of limited modality combinations, according to at least some embodiments of the present subject disclosure. The hardware configuration includes a display 1088 and a device 1080 that interacts with it directly or via a network 1089. In at least some embodiments, the display 1088 is a touchscreen or any other device configured for input and output. In at least some embodiments, the network 1089 is an Ethernet network or any other wired or wireless network or a combination thereof. In at least some embodiments, the device 1080 is a computer or other computing device that receives input or commands from the display 1088. In at least some embodiments, the device 1080 is integrated with the display 1088. In at least some embodiments, the device 1080 is a computer system that executes computer-readable instructions to perform operations for training a multimodal model using a batch of limited modality combinations.

[0065] The device 1080 includes a controller 1082, storage 1084, an input / output interface 1086, and a communication interface 1087. In at least some embodiments, the controller 1082 includes a processor or programmable circuit configuration that executes instructions, which cause the processor or programmable circuit configuration to perform the operation specified by the instructions. In at least some embodiments, the controller 1082 includes an analog or digital programmable circuit configuration, or any combination thereof. In at least some embodiments, the controller 1082 includes physically isolated storage or circuit configuration that interacts via communication. In at least some embodiments, the storage 1084 includes a non-volatile computer-readable medium that can store executable and non-executable data for the controller 1082 to access during instruction execution. In at least some embodiments, the communication interface 1087 transmits data and receives data from a network 1089. In at least some embodiments, the input / output interface 1086 connects to various input and output units, such as a display 1088, via parallel ports, serial ports, keyboard ports, mouse ports, and monitor ports, to accept commands and present information. In some embodiments, storage 1084 is located outside the device 1080.

[0066] The controller 1082 includes a grouping section 1090, a generation section 1091, and a creation section 1092. The storage 1084 includes training data samples 1094, modalities 1095, a modality list 1096, and model parameters 1097.

[0067] The grouping section 1090 is a circuit configuration or instruction of the controller 1082 configured to group training data samples into batches and data types into modalities. In at least some embodiments, the grouping section 1090 is configured to group training data samples from a plurality of training data samples into a plurality of batches of training data samples, each training data sample containing data values ​​corresponding to each of a plurality of data types. In at least some embodiments, the grouping section 1090 is configured to group data types from a plurality of data types into a plurality of modalities of data types. In at least some embodiments, the grouping section 1090 utilizes storage 1084 to read or record information such as training data samples 1094 and modalities 1095. In at least some embodiments, the grouping section 1090 includes subsections for performing additional functions, as described in the flowchart above. In at least some embodiments, such subsections are referred to by names associated with the corresponding functions.

[0068] The generation section 1091 is a circuit configuration or instruction of the controller 1082 configured to generate a modality list. In at least some embodiments, the generation section 1091 is configured to generate multiple modality lists, each modality list containing one or more modalities. In at least some embodiments, the generation section 1091 utilizes the storage 1084 to read or record information such as modality 1095 and modality list 1096. In at least some embodiments, the generation section 1091 includes subsections for performing additional functions, as described in the flowchart above. In at least some embodiments, such subsections are referred to by names associated with the corresponding functions.

[0069] The creation section 1092 is a circuit configuration or instruction for the controller 1082 configured to create a multimodal model. In at least some embodiments, the creation section 1092 is configured to create a multimodal model for estimating results from incomplete data samples. In at least some embodiments, the creation section 1092 utilizes storage 1084 to read or record information such as model parameters 1097. In at least some embodiments, the creation section 1092 includes subsections for performing additional functions, as described in the flowchart above. In at least some embodiments, such subsections are referred to by names associated with the corresponding functions.

[0070] In at least some embodiments, the apparatus is another device capable of processing logical functions to perform the operations described herein. In at least some embodiments, the controller and storage do not need to be entirely separate devices and share a circuit configuration or one or more computer-readable media. In at least some embodiments, the storage includes a hard drive that stores both computer-executable instructions and data accessed by the controller, and the controller includes a combination of a central processing unit (CPU) and RAM, and the computer-executable instructions can be copied whole or in part for execution by the CPU during the performance of the operations described herein.

[0071] In at least some embodiments where the device is a computer, a program installed on the computer may cause the computer to function as the device of the embodiments described herein, or to cause the computer to perform operations associated with the device of the embodiments described herein. In at least some embodiments, such a program may be executable by a processor to cause the computer to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0072] At least some embodiments are described with reference to flowcharts and block diagrams, where the blocks represent (1) a step in a process in which an operation is performed, or (2) a section of hardware responsible for performing the operation. In at least some embodiments, specific steps and sections are implemented by a dedicated circuit configuration, a programmable circuit configuration supplied with computer-readable instructions stored on a computer-readable medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable medium. In at least some embodiments, the dedicated circuit configuration includes digital and / or analog hardware circuits and includes integrated circuits (ICs) and / or discrete circuits. In at least some embodiments, the programmable circuit configuration includes reconfigurable hardware circuits, such as field-programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including logical AND, OR, XOR, NAND, NOR, and other logical operations, as well as flip-flops, registers, memory elements, and the like.

[0073] In at least some embodiments, the computer-readable medium includes a tangible device capable of holding and storing instructions for use by an instruction execution device. In some embodiments, the computer-readable medium includes, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination thereof. A non-exhaustive list of more specific examples of computer-readable mediums includes, below, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures having stored instructions thereon, and any preferred combination thereof. The computer-readable media used in the embodiments of this specification should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0074] Although embodiments of the present invention have been described above, the technical scope of any subject matter claimed is not limited to the embodiments described above. Those skilled in the art will understand that various modifications and improvements are possible to the embodiments described above. Furthermore, those skilled in the art will understand from the scope of the claims that such modified or improved embodiments are also included in the technical scope of the present invention.

[0075] The operations, procedures, steps, and stages of each process performed by the apparatus, system, program, and method shown in the claims, embodiments, or figures can be performed in any order, unless indicated in order such as “before” or “before,” and unless the output from a previous process is used in a later process. Even if the flow of a process is described in the claims, embodiments, or figures using phrases such as “first” or “next,” such description does not necessarily mean that the processes must be performed in the order described.

[0076] Training a multimodal model using a limited set of modality combinations involves grouping training data samples from multiple training data samples into multiple batches of training data samples, where each training data sample contains data values ​​corresponding to each of multiple data types, grouping data types from multiple data types into multiple modalities of data types, generating multiple modality lists, where each modality list contains one or more modalities, and for each of the multiple modality lists, developing a multimodal model to estimate results from incomplete data samples. To create the model, the following steps are performed: for each modality in the modality list, apply a stochastic encoder to the data values ​​corresponding to the modality in a batch of training samples from multiple batches to obtain the feature encoding corresponding to the modality; integrate the feature encoding of each modality in the modality list to create a latent variable; apply a logistic regression decoder to the latent variable to estimate the result; determine the cost by comparing the estimated result with ground truth; determine the divergence by comparing the latent variable with a multidimensional Gaussian distribution; and adjust the parameters of the stochastic encoder and logistic regression decoder based on the cost and divergence.

[0077] In at least some embodiments, training a multimodal model using a batch of limited modality combinations is further carried out by applying the model to live data samples containing data values ​​corresponding to fewer data types than all of the multiple data types. In at least some embodiments, the multiple modality lists include an even distribution of modalities among the multiple modalities. In at least some embodiments, generating the multiple modality lists includes determining possible combinations of modalities. In at least some embodiments, the enumerated modalities in each of the multiple modality lists are included in a predetermined number of modality combinations. In at least some embodiments, generating the multiple modality lists includes storing the multiple modality lists in memory. In at least some embodiments, feature coding represents the mean and variance. In at least some embodiments, each of the multiple training data samples corresponds to a person, and each of the multiple data types is an anthropometric measurement. In at least some embodiments, applying a logistic regression decoder to a latent variable is to estimate the risk of lifestyle-related diseases.

[0078] Training a multimodal model using a limited set of modality combinations involves grouping training data samples from multiple training data samples into multiple batches of training data samples, where each training data sample contains data values ​​corresponding to each of multiple data types, grouping data types from multiple data types into multiple modalities of data types, generating multiple modality lists, where each modality list contains one or more modalities, and for each of the multiple modality lists, developing a multimodal model to estimate results from incomplete data samples. To create the model, the following steps are performed: for each modality in the modality list, apply a stochastic encoder to the data values ​​corresponding to the modality in a batch of training samples from multiple batches to obtain the feature encoding corresponding to the modality; integrate the feature encoding of each modality in the modality list to create a latent variable; apply a logistic regression decoder to the latent variable to estimate the result; determine the cost by comparing the estimated result with ground truth; determine the divergence by comparing the latent variable with a multidimensional Gaussian distribution; and adjust the parameters of the stochastic encoder and logistic regression decoder based on the cost and divergence.

[0079] In at least some embodiments, training a multimodal model with a batch of limited modality combinations further includes applying the model to a live data sample containing data values ​​corresponding to fewer data types than all of the multiple data types. In at least some embodiments, the multiple modality lists include an even distribution of modalities among the multiple modalities. In at least some embodiments, generating the multiple modality lists includes determining possible combinations of modalities. In at least some embodiments, the enumerated modalities in each of the multiple modality lists are included in a predetermined number of modality combinations. In at least some embodiments, generating the multiple modality lists includes storing the multiple modality lists in memory.

[0080] Training a multimodal model using batches of limited modality combinations is performed by a controller including a circuit configuration configured to perform an operation, which is to group training data samples from a plurality of training data samples into a plurality of batches of training data samples, each training data sample containing data values ​​corresponding to each of a plurality of data types, to group the data types from the plurality of data types into a plurality of modalities of the data type, to generate a plurality of modality lists, each modality list containing one or more modalities, and for each of the plurality of modality lists, incomplete data samples To construct a multimodal model for estimating results from, the following steps are performed: for each modality in the modality list, apply a stochastic encoder to the data values ​​corresponding to the modality in a batch of training samples from multiple batches to obtain a feature encoding corresponding to the modality; integrate the feature encodings of each modality in the modality list to create a latent variable; apply a logistic regression decoder to the latent variable to estimate the result; determine the cost by comparing the estimated result with ground truth; determine the divergence by comparing the latent variable with a multidimensional Gaussian distribution; and adjust the parameters of the stochastic encoder and logistic regression decoder based on the cost and divergence.

[0081] In at least some embodiments, training a multimodal model with a batch of limited modality combinations further includes applying the model to a live data sample containing data values ​​corresponding to fewer data types than all of the multiple data types. In at least some embodiments, the multiple modality lists include an even distribution of modalities among the multiple modalities. In at least some embodiments, generating the multiple modality lists includes determining possible combinations of modalities. In at least some embodiments, the enumerated modalities in each of the multiple modality lists are included in a predetermined number of modality combinations.

[0082] The above outlines some features of embodiments so that those skilled in the art may better understand aspects of this disclosure. Those skilled in the art will understand that this disclosure can be readily used as a basis for designing or modifying other processes and structures to perform the same purposes and / or achieve the same advantages as the embodiments described herein. Those skilled in the art should also recognize that such equivalent configurations would not depart from the spirit and scope of this disclosure, and that various changes, substitutions, and modifications described herein are possible without departing from the spirit and scope of this disclosure.

[0083] This application claims priority based on U.S. Patent Application No. 18 / 983,355, filed on 17 December 2024, and incorporates all of its disclosures herein. [Explanation of Symbols]

[0084] 100 data samples 102A Modality 102B Modality 102N Modality 104A Feature Coding 104B Feature Coding 104N Feature Coding 106 Latent Variables 108 Estimated results 110 Modality Group 112A encoder 112B encoder 112N encoder 114 Feature Encoding Integrator 116 decoders 220 data values 222 Target data values 440 Modalities 442 Modalities 444 Modalities 446 Modalities 448 Modalities 449 Modalities 524 Modality Combinations 626 Modality List 1080 equipment 1082 Controller 1084 storage 1086 Input / Output Interfaces 1087 Communication Interface 1088 displays 1089 Network 1090 Grouping Section 1091 Generation Section 1092 Creation Section 1094 Training Data Samples 1095 Modalities 1096 Modality List 1097 Model Parameters

Claims

1. This involves grouping training data samples into multiple batches, where each training data sample contains data values ​​corresponding to each of multiple data types, Grouping the data types among the aforementioned multiple data types into multiple modalities of the data type, Each modality list generates multiple modality lists containing one or more modalities, For each of the aforementioned modality lists, in order to create a multimodal model for estimating results from incomplete data samples, For each modality in the modality list, a probabilistic encoder is applied to the data value corresponding to the modality in the training sample batch of the multiple batches to obtain the feature encoding corresponding to the modality. Integrating the feature encoding of each modality in the modality list to create latent variables, Applying a logistic regression decoder to the latent variable to estimate the result, determining the cost by comparing the estimated result with ground truth, determining the divergence by comparing the latent variable with a multidimensional Gaussian distribution, and Adjusting the parameters of the probabilistic encoder and the logistic regression decoder based on the aforementioned cost and divergence. To perform, To have the controller perform an action that includes the following: program.

2. Applying the multimodal model to a live data sample containing data values ​​corresponding to fewer data types than all of the aforementioned data types, The controller is instructed to perform the aforementioned operations, which further include the following: The program according to claim 1.

3. The plurality of modality lists include an even distribution of modalities among the plurality of modalities, The program according to claim 1.

4. The generation of the plurality of modality lists includes determining possible combinations of modalities. The program according to claim 1.

5. The modalities listed in each of the aforementioned modality lists are included in a predetermined number of modality combinations. The program according to claim 4.

6. The generation of the plurality of modality lists includes storing the plurality of modality lists in memory. The program according to claim 1.

7. The feature coding described above represents the mean and variance. The program according to claim 1.

8. Each of the aforementioned training data samples corresponds to a person, and each of the aforementioned data types is a physical measurement. The program according to claim 1.

9. This involves grouping training data samples into multiple batches, where each training data sample contains data values ​​corresponding to each of multiple data types, Grouping the data types among the aforementioned multiple data types into multiple modalities of the data type, Each modality list generates multiple modality lists containing one or more modalities, For each of the aforementioned modality lists, in order to create a multimodal model for estimating results from incomplete data samples, For each modality in the modality list, a probabilistic encoder is applied to the data value corresponding to the modality in the training sample batch of the multiple batches to obtain the feature encoding corresponding to the modality. Integrating the feature encoding of each modality in the modality list to create latent variables, Applying a logistic regression decoder to the latent variable to estimate the result, determining the cost by comparing the estimated result with ground truth, determining the divergence by comparing the latent variable with a multidimensional Gaussian distribution, and Adjusting the parameters of the probabilistic encoder and the logistic regression decoder based on the aforementioned cost and divergence. To perform, A method that includes this.

10. A device comprising a controller including a circuit configuration configured to perform an operation, wherein the operation is This involves grouping training data samples into multiple batches, where each training data sample contains data values ​​corresponding to each of multiple data types, Grouping the data types among the aforementioned multiple data types into multiple modalities of the data type, Each modality list generates multiple modality lists containing one or more modalities, For each of the aforementioned modality lists, in order to create a multimodal model for estimating results from incomplete data samples, For each modality in the modality list, a probabilistic encoder is applied to the data value corresponding to the modality in the training sample batch of the multiple batches to obtain the feature encoding corresponding to the modality. Integrating the feature encoding of each modality in the modality list to create latent variables, Applying a logistic regression decoder to the latent variable to estimate the result, determining the cost by comparing the estimated result with ground truth, determining the divergence by comparing the latent variable with a multidimensional Gaussian distribution, and Adjusting the parameters of the probabilistic encoder and the logistic regression decoder based on the aforementioned cost and divergence. To perform, including, device.

Images