Transitive and commutative multimodal models and their use

Abstract

This document describes computer implementations for analyzing multimodal data. These implementations involve generating an output dataset by processing an input dataset containing data from one or more sets of at least three modalities using machine learning models, including transitive and commutative machine learning models. Related methods, products, and systems are described.

Description

[Technical Field]

[0001] The present invention relates to a computer implementation method for analyzing input data using a multimodal machine learning model trained using commutative and transitive modeling, and to a computer implementation method for training a multimodal machine learning model using commutative and transitive modeling. Related systems and computer program products are also described. [Background technology]

[0002] Integrating multiple modalities, such as images, text, or audio, promises to provide a more comprehensive and holistic understanding of problems that are difficult to address in other ways. By leveraging the inherent complementary strengths of each data type, this approach yields a more synergistic representation of the underlying phenomena, enabling deep learning systems to adapt better to real-world scenarios. Consequently, there has been remarkable progress in the field of multimodal learning in recent years. Much of this research has focused on multimodal datasets where all data points have all modalities (see Figure 3a). Techniques addressing the use of data with missing modalities still heavily rely on at least a subset of samples where all modality combinations exist (see Figure 3b). In practice, obtaining such datasets when considering three or more modalities, for example, three modalities, is not straightforward. In the worst case, one might end up with a dataset containing only modalities A and B, and another separate dataset containing modalities B and C. Thus, combinations A and C are unconditionally missing (see Figure 3c).

[0003] This is particularly important for safety-critical domains such as healthcare, infrastructure, and transportation. For example, in medicine, it is already difficult to obtain large datasets with a single modality to train powerful base models such as those used in general computer vision and natural language processing. It is even more difficult to find datasets with two or more modalities. However, combining them is important because it is common practice to examine different types of data such as radiographic images, gene sequences, and microscope slides to determine the optimal treatment. Similarly, infrastructure monitoring models or autonomous driving systems require inputs from various sensors. However, not all combinations of modalities exist in the same training set.

[0004] Therefore, there is a need for new methods to obtain multimodal models that are not plagued by the drawbacks of the prior art. SUMMARY OF THE INVENTION

[0005] The inventors have attempted to find a solution to the problem of pre-training a multimodal neural network that can process any combination of modalities at inference time when there are modalities missing in the pre-training data. The inventors were particularly interested in developing a technique that can handle situations where combinations of modalities are never seen (e.g., (A, C) and (A, B, C) in the example of FIG. 1c). The inventors were able to do this and further developed a technique that enables obtaining a model that achieves better downstream performance as more combinations of modalities are given, including the previously unseen combinations (A, C) and (A, B, C).

[0006] To the inventors' knowledge, this scenario had not been considered in the literature. The inventors designed a novel transitive pre-trained machine learning model called LoReTTa, which exemplifies a transformer-based model capable of addressing such situations. LoReTTa (Linking modalities with a tRansitive and commutativE pre-Training sTrAtegy) links modalities and allows them to transit through each other using generative pre-training (predicting the next token), commutativity (A,B)=(B,A), and transitive relations (A→B)∧(B→C)⇒(A→C). The inventors further demonstrated theoretically and experimentally that LoReTTa is well-suited for training multimodal models by learning intra-to-token and inter-token relationships from different data distributions. This has broad implications for safety-critical domains such as healthcare, infrastructure, and transportation, where, as mentioned earlier, it is common for not all modality combinations to exist in the same training set. LoReTTa demonstrated a unique approach to address this challenging and under-researched problem.

[0007] Thus, in a first aspect, a computer-implemented method is provided, the computer-implemented method comprising obtaining an input data set comprising data from one or more modalities selected from a set of at least three modalities, and generating an output data set by processing the input data set using a machine learning model comprising a transitive and commutative machine learning model, wherein the transitive and commutative machine learning model is a machine learning model trained to convert any one of the modalities of a set of at least three modalities and the corresponding input data set to any one of the other modalities of the set of at least three modalities and the respective output data set, the machine learning model being trained to convert data between two modalities by using training data comprising a data pair corresponding to a first modality of two modalities and another modality, and another data pair corresponding to another modality and a second modality of two modalities, and the machine learning model being trained to perform a corresponding reverse type of modality conversion for any forward type of modality conversion for which the model is trained to perform.

[0008] Furthermore, this embodiment also describes a computer implementation method which includes accessing an input dataset, determining a specific input modality of the input dataset, receiving a representation of a specific target output modality, and generating an output dataset by processing the input dataset and the representation of the specific target output modality using a transitive and commutative machine learning model, wherein the transitive and commutative machine learning model is configured to transform input datasets of any one of at least three modalities and corresponding to each to any other modality of the at least three modalities and corresponding to each output dataset, and is transitive in that it is configured to learn to transform data between two modalities by using training data which includes data pairs corresponding to a first modality of two modalities and another modality, and other data pairs corresponding to another modality and a second modality of two modalities, and is commutative in that, with respect to any forward type of modality transformation that the transitive and commutative machine learning model is configured to perform, the transitive and commutative machine learning model is also configured to perform a corresponding backward type of modality transformation.

[0009] The method according to this embodiment may have one or more of the following features.

[0010] A machine learning model may be a classification model, a regression model, or a generative model. A classification model may comprise a transitive and commutative machine learning model configured to take an input dataset as input, and a trained classification model trained to take features extracted from the transitive and commutative machine learning model as input and produce classifications on the input dataset as output. A generative model may take an input dataset as input and be trained to produce a generated output dataset corresponding to the input dataset, the output dataset corresponding to one of at least three modalities not present in the input dataset. A regression model may comprise a transitive and commutative machine learning model configured to take an input dataset as input, and a trained regression model trained to take features extracted from the transitive and commutative machine learning model as input and produce regression values ​​on the input dataset as output. The regression model may be a Cox proportional hazards model, and the regression values ​​may be survival predictions. Obtaining an input dataset may include accessing the input dataset or receiving the input dataset. Therefore, a transitive and commutative machine learning model can be described as a machine learning model that (i) is configured to transform any input dataset of any modality from a set of at least three modalities and each corresponding to it into any output dataset of any other modality from a set of at least three modalities and each corresponding to it; (ii) is configured to learn to transform data between two modalities by using training data that includes data pairs corresponding to a first modality of two modalities and another modality, and other data pairs corresponding to another modality and a second modality of two modalities; and (ii) is commutative in that, with respect to any forward type of modality transformation that the transitive and commutative machine learning model is configured to perform, the transitive and commutative machine learning model is also configured to perform a corresponding backward type of modality transformation.

[0011] The method may include determining a specific input modality for the input dataset and receiving a representation of a specific target output modality, and generating the output dataset is performed by processing the input dataset and the identification representation of the specific target output modality using transitive and commutative machine learning models.

[0012] Transitive and commutative machine learning models may be trained using causal generative modeling, masked modeling, and / or variations thereof. The variations may be causal masked modeling. Causal generative modeling, masked modeling, and / or variations thereof may use ground truth data that includes concatenated input data for a first modality and a second modality of a data pair for each of several data pairs in the training data, where the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair. Alternatively, or in addition to this, the ground truth data may include a first concatenated input data and a second concatenated input data for each of several data pairs in the training data, where the modality of the data pair used as the first modality differs between the first and second concatenated input data.

[0013] Transitive and commutative machine learning models may include sequence models and / or deep learning models. Transitive and commutative machine learning models may include sequence models selected from recurrent neural networks, models combining recurrent and attention mechanisms, convolutional neural networks or convolution-based models, optionally models combining element-wise multiplication (gating) and long convolution, or models combining convolution and attention mechanisms, state-space models, or models combining state-space model mechanisms and attention mechanisms, or transformer-based models. Recurrent neural networks may be long-term short-term memory (LSTM) models. Models combining recurrent and attention mechanisms may be Receptance Weighted Key Value (RWKV) models or Griffin models. Models combining element-wise multiplication (gating) and long convolution may be Hyena models. Models combining convolution and attention mechanisms may be Striped Hyena models. Attention-based models may also be called transformer-based models if they are attention-only models. State-space models may be Mamba models or Hawk models. A model that combines a state-space model mechanism with an attention mechanism can be called a hybrid transformer state-space model. An example is the Jamba model.

[0014] A transitive and commutative machine learning model may be trained using causally masked modeling to identify a subset of tokens in a training dataset, wherein the tokens in the subset of tokens are located at multiple locations in the training dataset; to append the subset of tokens to the training dataset; and to add a mask to the multiple locations in the training dataset. The method may further include using a masked representation of the multiple locations during training.

[0015] Transitive and commutative machine learning models may include transformer models that use self-attention.

[0016] The method may further include receiving an initial dataset and generating an input dataset by tokenizing the initial dataset.

[0017] Generating an input dataset may include tokenizing an initial dataset and projecting the tokenized initial dataset onto a given feature space using modality-parameterized embedding functions. Generating an input dataset may also include adding and / or appending modality-specific tokens to the tokenized and arbitrarily projected data corresponding to each single modality in the input dataset, and generating an input dataset may also include learning location embeddings.

[0018] The method may further include accessing another input dataset related to the input dataset and determining another specific input modality for the other input dataset, and the process performed to generate the output dataset further includes processing the other dataset and the other specific input modality using transitive and commutative machine learning models.

[0019] A transitive and commutative machine learning model accesses a training dataset containing first data for a specific input modality and corresponding second data for another modality; generates a first predicted output dataset by processing the first data with a discriminative representation of another modality using the transitive and commutative machine learning model, wherein the first predicted output dataset corresponds to the other modality; and generates a second predicted output dataset by processing the first predicted output data with a discriminative representation of a specific target output modality using the transitive and commutative machine learning model. The machine learning model may be trained using a method that generates a second predicted output dataset in which the second predicted output dataset corresponds to a specific target output modality, generates a third predicted output dataset by processing the second predicted output data and a discriminative representation of a specific input modality using a transitive and commutative machine learning model, generates a third predicted output dataset in which the third predicted output dataset corresponds to a specific input modality, and calculates a loss by comparing at least a portion of the third predicted output dataset with the first data.

[0020] A transitive and commutative machine learning model may be trained using training data that lacks data relating a first data of a particular input modality to a second data of a particular output modality. A transitive and commutative machine learning model may also be trained using training data that relates data of modality pairs in a set of at least three modalities, wherein for each pair of first and second modalities in the set of at least three modalities, the training data includes data relating data of modality pairs that together form a path between the first and second modalities. In an embodiment, for each pair of first and second modalities in the set of at least three modalities, the training data includes data relating data of the first modality to a third modality and data relating data of the third modality to a second modality. Therefore, the training data does not have to include training data relating any pair of modalities, but it may include data relating missing modalities within a pair using one or more bridging (also referred to herein as “links”) modalities.

[0021] Transitive and commutative machine learning models are further trained using transitive modeling, with a first step using commutative modeling and self-supervised learning objectives, and a second step in which the weights of the machine learning model are initialized to the weights learned in the first step. The first step may include taking partially masked concatenated input data for a first modality and a second modality of a data pair in a training dataset, each containing ground truth data for multiple data pairs, each containing corresponding data for a modality pair, and training the model to produce predictions for the masked input data as an output, wherein the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair, and / or the ground truth data contains a first concatenated input data and a second concatenated input data for each of the multiple data pairs in the training dataset, wherein the modality of the data pair used as the first modality differs between the first concatenated input data and the second concatenated input data.

[0022] The second step is to train the model, which involves predicting a third data for a third modality in a set of at least three modalities corresponding to the second data, using a machine learning model that takes the second data as input for each of several data pairs in the training dataset, each of which includes a first data for a first modality and a corresponding second data for a second modality, and optionally, iteratively predicting a subsequent data for a further modality in a set of at least three modalities corresponding to the predicted third data or predicted subsequent data, using a transitive and commutative machine learning model that takes the predicted third data or predicted subsequent data as input, and inputting the third data or the most recent iteratively predicted subsequent data. The process may include training a model to predict final data for a first modality corresponding to a predicted third data or the most recent iteratively predicted subsequent data using a transitive and commutative machine learning model incorporated therein, the training data including (i) pairs containing data for a second modality and corresponding data for a third modality, and (ii) pairs containing data for a third modality and corresponding data for a first modality, or pairs containing data for the most recent further modality and corresponding data for a first modality, and each pair containing data for each of the two modalities used in the step of iteratively predicting subsequent data for further modalities.

[0023] Training a model using commutative modeling and self-supervised learning objectives involves taking partially masked concatenated input data for a first modality and a second modality of a data pair in a training dataset that includes ground truth data for multiple data pairs, each containing corresponding data for a modality pair, and training the model to produce predictions for the masked input data as output, wherein the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair, and / or the ground truth data includes the first concatenated input data and the second concatenated input data for each of the multiple data pairs in the training dataset, wherein the modality of the data pair used as the first modality differs between the first concatenated input data and the second concatenated input data.

[0024] Training a model using transitive modeling may include, or may include, for each of several pairs of data in the training dataset, each containing first data for a first modality and corresponding second data for a second modality, using a machine learning model that takes the second data as input to predict third data for a third modality in a set of at least three modalities corresponding to the second data; using a transitive and commutative machine learning model that takes the third data as input to predict final data for the first modality corresponding to the predicted third data; and calculating a loss comparing the first data with the predicted final data, wherein the training dataset includes (i) pairs containing data for a second modality and corresponding data for a third modality, and (ii) pairs containing data for a third modality and corresponding data for a first modality.

[0025] Training a model using transitive modeling involves, for each of several data pairs in the training dataset, each containing first data for a first modality and corresponding second data for a second modality, using a machine learning model that takes the second data as input to predict third data for at least three sets of modalities corresponding to the second data; and iteratively predicting subsequent data for further modalities of at least three sets of modalities corresponding to the predicted third data or predicted subsequent data using a transitive and commutative machine learning model that takes the predicted third data or predicted subsequent data as input, and the latest The process may include, and may include, predicting final data for a first modality corresponding to the most recently predicted iteratively predicted subsequent data using a transitive and commutative machine learning model that takes iteratively predicted subsequent data as input, and calculating a loss comparing the first data with the predicted final data, wherein the training data includes (i) pairs containing data for a second modality and corresponding data for a third modality, and (ii) pairs containing data for the most recent further modality and corresponding data for the first modality, and each pair containing data for each of the two modalities used in the step of iteratively predicting subsequent data for further modalities.

[0026] A transitive and commutative machine learning model may be trained to learn a first conditional distribution (P(C|A), P(A|C)) for a modality pair (A,C), where the training data does not contain enough training data pairs to relate the data of the modality pair for parameterizing the conditional distribution, and the transitive and commutative machine learning model is trained to learn the first conditional distribution (P(C|A), P(A|C)) by predicting data for one or more additional modalities (B) that form a link between the modality pair using a further learned conditional distribution (P(C|B)) for the linking modality pair (C,B), where the training data contains enough training data pairs to relate the data of the modality pair for parameterizing the further conditional distribution.

[0027] Transitive and commutative machine learning models may be trained using training data that includes tuples (e.g., pairs) containing corresponding data for multiple modalities in a set of at least three modalities, wherein tuples containing data for a particular pair of modalities in a set of at least three modalities are present in the training data in less than 50%, less than 40%, less than 30%, less than 20%, less than 10%, less than 5%, or less than 1% of the number of tuples containing data for all other pairs of modalities in a set of at least three modalities. In embodiments, the tuples did not include tuples containing data for a particular pair of modalities in a set of at least three modalities.

[0028] The method may further include training a machine learning model.

[0029] One or more modalities may be selected from medical or biological sample data related to the subject. One or more modalities may be selected from medical imaging data, electronic medical record data, gene sequence data, and digital pathology imaging data. Medical imaging data may be selected from MRI, CT, or PET imaging data. Therefore, a particular input modality or target output modality may include MRI, CT, or PET imaging. A particular input modality or target output modality may include electronic medical record representations. A particular input modality or target output modality may include gene sequence representations. A particular input modality or target output modality may include digital pathology imaging.

[0030] A machine learning model may be configured to take medical or biological sample data related to a subject as input and generate a display of recommended treatments for the subject as output. The display may also be a classification. The medical or biological sample data may include one or more of the following: radiographic images, gene sequences, and microscope slides.

[0031] A machine learning model may be configured to take data collected by a vehicle, or any one or more of several types of sensors associated with the vehicle, as input, and to produce predictive sensor data for another type of sensor associated with the vehicle as output. The machine learning model may be trained using synchronous data from pairs of several types of sensors. According to a second aspect, a computer implementation method is provided which includes inputting medical or biological sample data related to a subject using the method of any embodiment of the first aspect, and providing a diagnosis and / or prognosis and / or treatment recommendation relating to the subject using the output of a machine learning model, wherein the output of the machine learning model indicates a diagnosis, prognosis, or treatment recommendation. According to a third aspect, a computer implementation method for training a multimodal machine learning model is provided, the method comprising: accessing a training dataset containing data relating to each of at least three sets of modalities; and training a machine learning model to transform input datasets corresponding to any of the mobilitys of the at least three sets of modalities and each of them to output datasets of any of the other modalities of the at least three sets of modalities, the machine learning model being trained to transform data between two modalities by using training data containing data pairs corresponding to a first modality of two modalities and another modality, and other data pairs corresponding to the other modality and a second modality of two modalities, the machine learning model being trained to perform corresponding inverse modality transformations with respect to any forward modality transformations that the model is trained to perform.

[0032] Furthermore, according to this embodiment, a computer implementation method for obtaining a trained transitive and commutative machine learning model, wherein the trained transitive and commutative machine learning model is configured to transform input datasets corresponding to any one of at least three modalities and each of them to output datasets corresponding to any other modality of at least three modalities, the trained transitive and commutative machine learning model is transitive in that it is configured to learn to transform data between two modalities by using training data including data pairs corresponding to a first modality of two modalities and another modality, and other data pairs corresponding to another modality and a second modality of two modalities, the trained transitive and commutative machine learning model is commutative in that it is also configured to perform corresponding inverse type modality transformations with respect to any forward type modality transformations configured to be performed by the transitive and commutative machine learning model, and

[0033] The method involves (i) identifying a subset of tokens in a training dataset, wherein the tokens in the subset of tokens are located at multiple locations in the training dataset; training a machine learning model using causally masked modeling to add the subset of tokens to the training dataset; and adding masks to the multiple locations in the training dataset; and (ii) accessing a training dataset containing first data for a particular input modality of at least three modalities and corresponding second data for another modality of at least three modalities; and generating a first predicted output dataset by processing the first data and the identifying representation of another modality using a transitive and commutative machine learning model, wherein the first predicted output dataset corresponds to the other modalities. The method includes one or both of the following: generating a first predicted output dataset; generating a second predicted output dataset by processing the first predicted output data with an identified representation of a specific target output modality using a transitive and commutative machine learning model, wherein the second predicted output dataset corresponds to a specific target output modality; generating a third predicted output dataset by processing the second predicted output data with an identified representation of a specific input modality using a transitive and commutative machine learning model, wherein the third predicted output dataset corresponds to a specific input modality; and calculating a loss by comparing at least a portion of the third predicted output dataset with the first data.

[0034] The method according to this embodiment may have any of the features described with respect to the first embodiment. The method according to this embodiment may have any one or more of the following features:

[0035] Training a machine learning model may involve using causal generative modeling, masked modeling, and / or causal masked modeling, wherein the causal generative modeling, masked modeling, and / or causal masked modeling uses ground truth data from the training dataset, which includes concatenated input data for a first modality and a second modality of a data pair, for each of a plurality of data pairs in the training data, wherein the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair, and / or the ground truth data includes a first concatenated input data and a second concatenated input data for each of a plurality of data pairs in the training data, wherein the modality of the data pair used as the first modality differs between the first concatenated input data and the second concatenated input data.

[0036] Training is performed using ground truth data from the training dataset, which includes concatenated input data for the first and second modalities of each of several data pairs in the training dataset, wherein the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair with a predetermined probability, optionally 50%.

[0037] The machine learning model includes learning a first conditional distribution (P(C|A), P(A|C)) for a modality pair (A,C), wherein the training data does not include enough training data pairs to relate the data of the modality pair for parameterizing the conditional distribution, and the machine learning model is used to learn the first conditional distribution (P(C|A), P(A|C)) by predicting data for one or more additional modalities (B) that form a link between the modality pairs, wherein the prediction uses the learned further conditional distribution (P(C|B)) for the modality pair (C,B) that form the link, wherein the training data includes enough training data pairs to relate the data of the modality pair for parameterizing the further conditional distribution.

[0038] Training a machine learning model may include, for each pair of training data containing pairs of data containing first data for a first modality and corresponding second data for a second modality, using a transitive and common machine learning model that takes the second data as input to predict third data for a third modality from the set of at least three modalities corresponding to the second data; using a transitive and commutative machine learning model that takes the third data or the most recent iteratively predicted subsequent data as input to predict final data for the first modality corresponding to the predicted third data; and calculating a loss comparing the first data with the predicted final data, wherein the training data includes (i) pairs containing data for a second modality and corresponding data for a third modality, and (ii) pairs containing data for a third modality and corresponding data for a first modality.

[0039] Training a machine learning model involves, for each pair of training data containing a data pair containing a first data for a first modality and a corresponding second data for a second modality, predicting the third data for a third modality from the set of at least three modalities corresponding to the second data using a transitive and common machine learning model that takes the second data as input, and then iteratively training the subsequent data for further modalities from the set of at least three modalities corresponding to the predicted third data or predicted subsequent data using a transitive and commutative machine learning model that takes the predicted third data or predicted subsequent data as input. The process may include predicting, predicting final data for a first modality corresponding to the latest iteratively predicted subsequent data using a transitive and commutative machine learning model that takes a third data or the most recent iteratively predicted subsequent data as input, and calculating a loss comparing the first data with the predicted final data, wherein the training data includes (i) pairs containing data for a second modality and corresponding data for a third modality, and (ii) pairs containing data for the most recent further modality and corresponding data for the first modality, and each pair containing data for each of the two modalities used in the step of iteratively predicting subsequent data for further modalities.

[0040] Training a machine learning model may involve identifying a subset of tokens in a training dataset, where the tokens in that subset are located at multiple locations within the training dataset; using causally masked modeling to append the subset of tokens to the training dataset; and adding a mask to the multiple locations within the training dataset.

[0041] Training a machine learning model involves accessing a training dataset containing first data for a specific input modality among at least three modalities and corresponding second data for another modality among at least three modalities, and generating a first predicted output dataset by processing the first data and discriminative representations of the other modalities using a transitive and commutative machine learning model, wherein the first predicted output dataset corresponds to the other modalities, and processing the first predicted output data and discriminative representations of a specific target output modality using a transitive and commutative machine learning model. The method may include generating a second predicted output dataset, wherein the second predicted output dataset corresponds to a specific target output modality; generating a third predicted output dataset by processing the second predicted output data and the discriminative representation of a specific input modality using a transitive and commutative machine learning model, wherein the third predicted output dataset corresponds to a specific input modality; and calculating a loss by comparing at least a portion of the third predicted output dataset with the first data.

[0042] The method may further include training further machine learning models, including the trained machine learning model, which are trained for a specific task and are classification, regression, or generative models. A classification model may include the trained machine learning model and a classification model that takes features extracted from transitive and commutative machine learning models as input and produces classifications on input datasets provided to the trained machine learning model as output, optionally the classification model is trained using training data that includes data on at least three sets of modalities and one or more associated ground truth classification labels. A generative model may take an input dataset as input and produce a generated output dataset corresponding to the input dataset as output, the output dataset corresponding to one of at least three sets of modalities that are not present in the input dataset, optionally the trained machine learning model is used as the generative model, or the trained machine learning model is fine-tuned using further training data to obtain the generative model. The generative model may be trained to take an input dataset containing missing data as input and to produce a generated output dataset corresponding to the missing data in the input dataset as output, optionally a trained machine learning model is used as the generative model, or a trained machine learning model is fine-tuned with further training data to obtain a generative model. The regression model may comprise a trained machine learning model and a regression model trained to take features extracted from the machine learning model trained on the input dataset as input and to produce regression values ​​for the input dataset as output. In embodiments, the regression model is a Cox proportional hazards model, the regression values ​​are survival predictions, and / or the regression model is trained with training data including data from at least three sets of modalities and associated ground truth regression values.

[0043] According to any embodiment of any aspect, acquiring or accessing input data (e.g., medical data or biological sample data) may involve a database, data acquisition means (e.g., medical data image acquisition means, biological sample data acquisition means such as a microscope, and associated sequencers or computing devices), a computer-readable medium, a user interface, or a processor that receives input data from a computing device.

[0044] Any aspect of the method may include providing the user with one or more results of the method, for example, through a user interface. For example, the results may include classification predictions, regression predictions, predicted output data related to one or more of several modalities (e.g., outputs of a generative model), trained parameters of one or more trained machine learning models, the trained machine learning models, and / or any information derived therefrom. The data store may be a public or private database. The information derived from classification predictions may include one or more of the following: prognostic representations derived from classifications obtained using a machine learning model, therapeutic representations derived from classifications obtained using a machine learning model, and representations of suitability for participation in clinical trials derived from classifications obtained using a machine learning model.

[0045] A fourth aspect provides a system comprising one or more processors and one or more computer-readable memories for storing instructions causing one or more processors to perform any method described herein, such as a method of any embodiment of any of the aforementioned embodiments, particularly including any embodiment of the first, second, and / or third aspects. In some embodiments, the system may include, for example, one or more computers, servers, or cloud-based devices.

[0046] According to the fifth aspect, a non-temporary computer-readable storage medium is provided which, when executed on a processor, causes the processor to perform any method described herein, such as the method of any embodiment of the first, second, and / or third aspect, which includes any one of any features described by reference thereto, or any combination thereof, as long as they are compatible.

[0047] According to the sixth aspect, a computer program is provided which, when executed on a computer, causes a computer to perform any embodiment of any method described herein, such as the method of any embodiment of the first, second, and / or third aspect, including any one of any features described by reference thereto, or any combination thereof, as long as they are compatible.

[0048] The present invention includes combinations of the described embodiments and preferred features, unless such combinations are clearly unacceptable or explicitly avoided. [Brief explanation of the drawing]

[0049] Next, embodiments and experiments illustrating the principle of the present invention will be described with reference to the attached figures.

[0050] [Figure 1] This diagram shows a flowchart illustrating how to use and / or obtain the multimodal model relating to this disclosure. [Figure 2] This disclosure shows an embodiment of the system relating to this disclosure. [Figure 3]Figures 3a, b, and c show Venn diagrams illustrating the relationships between datasets with different modalities A, B, and C. Overlapping datasets indicate that the datasets contain samples with aligned modalities (i.e., audio, image, and text files belonging to the same concept). While recent research has primarily focused on datasets where at least some samples have all available modality combinations (a, b), embodiments of this disclosure consider several modality combinations, such as (A, C) and (A, B, C) where (c) is completely missing. [Figure 4] Figures 4a, b, c, and d schematically illustrate training strategies for a multimodal model (LoReTTa) according to embodiments of the present disclosure. These figures illustrate two novel self-supervised strategies, namely commutative modeling and transitive modeling. (a) Commutative modeling, as illustrated, applies causal modeling in a commutative manner to generate modality A from modality B and modality B from modality A, given aligned input samples (A, B). The same technique is applied to aligned but discontinuous data points (B', C). (b) To ensure the model learns bidirectional relationships between input tokens, the illustrated embodiment uses a modified variant of generative pre-training called causal masked modeling. (c, d) Transitive pre-training is used to learn a missing conditional connection distribution. This involves randomly selecting a sample and using link modality B to predict the missing modality C, which is then used to reconstruct an existing modality A. The final step is to ensure that all modalities are properly aligned. [Figure 5] The characteristics of the training and test data (SVL-MNIST dataset) used in an example of this disclosure are shown. [Figure 6] The characteristics of the training and test data (TCGA-OMICS dataset) used in an example of this disclosure are shown. [Figure 7]This diagram shows pseudocode illustrating the pre-training of a model relating to an example of this disclosure. This diagram outlines the key ideas and algorithmic steps for training a model using commutative and transitive modeling, and how both are incorporated into a causal modeling framework. [Modes for carrying out the invention]

[0051] Hereinafter, aspects and embodiments of the present invention will be described with reference to the attached figures. Further aspects and embodiments will be apparent to those skilled in the art. All documents referenced in this text are incorporated herein by reference.

[0052] The methods described herein are computer-implemented unless otherwise indicated by the context. Indeed, the processes of image analysis using deep learning models and training deep learning models are complex, particularly requiring the analysis of large amounts of data with complex mathematics, and the methods described herein far exceed the capabilities of mental investigation. The systems and methods of the embodiments described above may be implemented in computer systems (in particular, computer hardware or computer software), in addition to the structural components and user interactions described.

[0053] The term "computer system" includes hardware, software, and data storage devices for the embodiment of the system or the execution of the method according to the embodiments described above. For example, a computer system may comprise one or more processing units, such as a central processing unit (CPU) and / or graphics processing unit (GPU), input means, output means, and data storage devices. Preferably, the computer system has a monitor for providing a visual output display. The data storage devices may comprise RAM, disk drives, or other computer-readable media. The computer system may comprise a plurality of computing devices connected by a network and capable of communicating with each other through that network. It is explicitly assumed that the computer system may consist of or include cloud computers.

[0054] The methods of the embodiments described above may be provided as computer programs, or as computer program products or computer-readable media carrying computer programs configured to perform the methods described above when executed on a computer. The term "computer-readable media" includes, but is not limited to, any non-temporary media that can be directly read and accessed by a computer or computer system. Examples of media include, but are not limited to, magnetic storage media such as floppy disks, hard disk storage media, and magnetic tapes, optical storage media such as optical disks or CD-ROMs, electrical storage media such as RAM, ROM, and flash memory, and hybrids and combinations of the above, such as magnetic / optical storage media.

[0055] Features disclosed in the foregoing description, or in the following claims, or in the accompanying drawings, and expressed in particular forms therein, or with respect to means for performing the disclosed functions, or methods or processes for obtaining the disclosed results, may be used, as necessary, individually or in any combination of such features, to realize the present invention in its various forms.

[0056] Although the present invention is described in conjunction with the exemplary embodiments described herein, many equivalent modifications and variations will become apparent to those skilled in the art upon this disclosure. Therefore, the typical embodiments of the present invention described herein are considered illustrative and not limiting. Various modifications to the described embodiments may be made without departing from the spirit and scope of the invention.

[0057] To avoid any misunderstanding, any theoretical explanations provided herein are provided solely for the purpose of improving the reader's understanding. The inventors do not wish to be bound by any of these theoretical explanations. Any headings used herein are for organizational purposes only and should not be construed as limiting the subject matter described.

[0058] Throughout this Spec., including the following claims, unless the context requires otherwise, the words “comprise” and “include,” as well as variations such as “comprises,” “includes,” and “include,” will be understood to mean the inclusion of the integer or step or group of integers or steps described, but not the exclusion of any other integer or step or group of integers or steps. As used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context otherwise explicitly indicates. Ranges may be expressed herein as “about” one particular value to and / or “about” another particular value. When such ranges are expressed, another embodiment includes one particular value to and / or another particular value. Similarly, the use of the antecedent “about” will be understood to mean that a particular value forms another embodiment when the value is expressed as an approximation. The term “about” with respect to numbers is arbitrary and means, for example, + / - 10%. Where used herein, the expression “and / or” should be interpreted as a specific disclosure of each of the two specified features or components, the other of which may or may not be present. For example, “A and / or B” should be interpreted as a specific disclosure of (i) A, (ii) B, and (iii) A and B, as if each were described separately herein.

[0059] This disclosure generally relates to multimodal machine learning models trained using commutative and transitive modeling. Since the transitive and commutative properties are inherent properties of the models described herein that arise from the way they are trained, such models may be referred to herein as “transitive and commutative models” or simply “machine learning models” or “models.”

[0060] A multimodal machine learning model is a model trained using data from multiple modalities. For example, a multimodal model can be trained using image and text data. It is assumed that a multimodal model can learn a more robust representation of the underlying phenomena represented in the data. For example, a multimodal model can be used to classify images using both image content and associated caption text, or to predict the caption text of an input image. A multimodal model is typically trained using data containing corresponding observations for each of the multiple modalities being modeled. These can also be called tuples or aligned data. For example, the text and image multimodal model above is trained using paired image and caption data points. This is inflexible (because it is necessary to know what data will be available at model deployment at the start of the training process) and is often impractical when using more than three modalities (because datasets containing aligned data for more than three modalities are rare in real-world scenarios). Collecting multimodal datasets with two pairs of modalities A and B or B and C is practically difficult. Obtaining a dataset with three aligned modalities A, B, and C is even more difficult and, importantly, potentially less reliable. The number of potential relationships between dependent and independent variables scales exponentially depending on the number of independent variables and the intermediate transformations between them. Therefore, the extent to which a given dataset of modalities can be used to predict data from another modality depends on the complexity and relationships of the data in the two modalities. These complexities become more complex when data from three or more datasets is considered. For example, Figure 3a represents a scenario where data from three modalities is available. Figure 3b represents a scenario where some entries from various modalities have corresponding data from one other modality, some other entries have corresponding data from two other modalities, and other entries have no corresponding data from other modalities.Figure 3c illustrates a scenario where some entries in the first modality have corresponding data in the second modality, but none of the entries in the first modality have corresponding data in the third modality. It is certainly not uncommon for data from a given modality of interest to be unavailable for training a model, even if other potentially appropriate data is available. For example, in some cases only modality A and modality B are paired, and in other cases only modality B and C are paired. Combinations A and C may be unconditionally missing, but it may be important to predict data from modality C based on data from modality A (or data from modalities A and B). Furthermore, identifying datasets across modalities that are reliably associated is particularly difficult when timing may be critical. For example, the extent to which the results of multiple medical tests can be significantly associated with each other may depend on the relative timing of the tests, the accuracy of the tests, the reliability of the tests, and / or the state of the subject. Therefore, situations frequently arise where several pairs of data modalities are little to no present in the training dataset used to train a multimodal model, and existing multimodal learning methods cannot adequately address these.

[0061] The introduction of transformers has led to significant changes in performance and capabilities in the fields of natural language processing (NLP) and computer vision (CV). These models excel at a wide range of complex tasks, including machine translation, text summarization, question answering, sentiment analysis, natural language generation, image classification, image captioning, semantic segmentation, object detection, and facial recognition, achieving state-of-the-art performance in each domain. Transformers have also opened up new possibilities for multimodal learning due to their absence of induction bias. These transformers can process and fuse different types of data, such as audio, images, video, and text, using self- and cross-attention mechanisms. As a result, transformers are considered general-purpose, modality-independent learners.

[0062] However, the lack of induced bias comes at a cost. Unlike other neural network architectures, transformers make few assumptions about the structure of the data. This makes them very expressive, but also requires a large amount of data to learn effectively and generalize well [Hoffmann et al., 2022]. As a result, self-supervised learning (SSL) on large amounts of unlabeled data has been proposed for transformers. In NLP, this strategy has been very successful, using methods such as masked modeling and causal modeling to train large language models (LLMs) such as BERT [Devlin et al., 2019] and GPT [Radford et al., 2018] variants. Recently, attempts have been made to unify masked modeling and causal modeling through frameworks such as prefix modeling, permutation modeling, causal masked modeling [Aghajanyan et al., 2022], or unified language learning. When applied to image classification, token-based pre-training methods have demonstrated superior capabilities compared to established computer vision techniques such as SimCLR [Hayes et al. 2022] and DINO [Caron et al. 2021]. As a result, these sequence modeling techniques have been successfully adapted and extended to a wide range of domains, encompassing programming, biochemistry, and reinforcement learning.

[0063] Adding or combining different modalities to the training process can open up three possibilities: positive transfer, cross-modal understanding, and cross-modal generation. Positive transfer occurs when learning in one modality helps improve performance in another. For example, aligning images with text has been shown to result in better classification performance on ImageNet [Radford et al., 2021b]. Cross-modal understanding refers to explaining one modality from the perspective of another. Models with this ability can describe images using text [Alayrac et al., 2022] or infer protein structures from sequences [Jumper et al., 2021]. In cross-modal generation, one modality can even be used to generate something using the other. Examples include text-to-image generation and text-to-music generation. Many modern multimodal foundational models are based on either comparative learning [Girdhar et al., 2023], masked modeling [Wang et al., 2022], or causal modeling [Yu et al., 2022].

[0064] This disclosure presents a novel pre-training method that combines advances in self-supervised learning with an integrated framework. In embodiments, LoReTTa uses a technique that combines causal and masked modeling for self-supervised learning of multimodal models. The proposed method extends both techniques by using commutative and transitive modeling to seamlessly integrate and attribute modalities. This allows the model to learn representational features and relationships within a multimodal dataset. The pre-trained model can be used for a variety of downstream tasks such as filling, classification, survival prediction, or cross-modal generation.

[0065] Multimodal models have been proposed previously. For example, the RNN-based model MCTN (Multimodal Cyclical Translation Network Model, Pham et al., 2019) is a neural model that learns a combined representation of multimodal data using modality transformations. The model is based on the insight that transformations from source modalities to target modalities result in an intermediate representation that captures joint information between modalities, using only the source modality at test time. This method differs from the present method in several respects, as it can only be used as a classifier, does not model all combinations of modalities, and most importantly, requires all modalities to be aligned and present at training time. The method described herein, on the other hand, does not require such strong assumptions. Models such as GPT, BERT, and derivatives (e.g., the transformer-based models CM3 and C2M3 used in the examples below) can perform multimodal predictions using cross-attention, for example. However, they do not learn joint representations by generating missing modalities at training time, and therefore their performance is usually insufficient if missing modality pairs exist in the training data. While performance can be improved with much larger amounts of training data, this does not compare to the methods described herein in that they leverage the additional information present in multimodal data that is not perfectly aligned. In other words, given any multimodal training dataset containing multimodal data that is not perfectly aligned, the methods described herein are expected to outperform models such as BERT and GPT during training time (i.e., using transitive and commutative modeling as described herein) because they are specifically designed to leverage the information present in multimodal data even when there is not perfectly aligned data, by generating missing modalities and bidirectionally connecting pairs of observations. Furthermore, if pairs of modalities are not always present in the training data, these models cannot learn the relationships between these modalities.Models specifically designed for multimodal learning, such as CLIP (Khosla et al. 2020), can handle training datasets with missing modalities. CLIP trains image encoders and text encoders together to predict which of all possible pairs in a set of N texts and N images were actual pairs, using contrast loss that maximizes the cosine similarity of images while minimizing the cosine similarity of false pair embeddings, and text embeddings of N real pairs in the set. While this allows for learning superficial features, it does not explicitly model missing modalities, and its performance degrades rapidly if the training data contains pairs of modalities that are rarely or always absent. Furthermore, CLIP requires a large amount of data for training and is computationally expensive. In contrast, by explicitly modeling missing modalities using transitive and commutative modeling as described herein, the present method can provide a model with improved performance compared to all the above models, as demonstrated in the following examples. As demonstrated in the following examples, the performance of the comparison model can be improved by increasing the computational power used for training. For example, training CLIP three times longer with a larger batch size (5 times) improved performance, but still did not match the performance of the new model described herein. Therefore, this disclosure provides a model that can achieve a level of performance that may only be achievable with the comparison model by using a much larger amount of training data and / or a more computationally intensive training scheme, both of which would be effective over the currently described model, providing a more computationally efficient method for training multimodal models when there are missing modalities in the training data.

[0066] The terms “analyze” and “process” may be used interchangeably in the context of applying the models described herein to input data, since the described models are trained to analyze the input data by processing it.

[0067] A machine learning model may be a deep learning model. A machine learning model may also be a sequence model. A sequence model is a machine learning model that takes a series of data as input. A sequence of data is any string of input data that has a sequence (order). This is in contrast to a set of input data that represents a set of independent data points. A sequence model is typically a deep learning model. A deep learning model can be a deep neural network, such as the following: (i) Recurrent neural networks (RNNs) or RNN-based models such as the Receptance Weighted Key Value (RWKV) model (Peng et al. 2023) and Griffin (De et al., 2024), or the Long-Term Short-Term Memory (LSTM) model; (ii) Convolutional neural networks (CNNs) such as Hyena (Poli et al. 2023), or models that combine element-wise multiplication (gating) and long convolution (convolution with the same filter size as the input), or models that combine convolution and attention mechanisms such as Striped Hyena (Poli et al., 2023b); (iii) State-space models such as Mamba (Gu et al. 2023) and Hawk (De et al., 2024), or hybrid transformer-state-space models such as Jamba (Lieber et al., 2024); or (iv) Fully attention-based models, also called transformer-based models. In one embodiment, the machine learning model is a transformer-based model. In other words, any machine learning model described herein may include a transformer model. The machine learning model may include a transformer model that uses self-attention. Transformer models can solve complex and diverse tasks such as machine translation, text summarization, question answering, sentiment analysis, natural language generation, image classification, object detection, face recognition, and image captioning, delivering state-of-the-art performance in each domain. Because they are free from induction bias, transformers may be well-suited for multimodal learning. Different types of data, such as audio, images, text, and video, can be processed and fused using self- and mutual attention mechanisms.Therefore, the transformer is considered to be general-purpose and the learner is considered to be modality-independent.

[0068] Transformer-based models are a class of machine learning models, particularly deep learning models, that include attention mechanisms (self and / or cross-attention mechanisms). Transformer-based models mainly or exclusively rely on attention mechanisms that learn from tokens within a context and typically do not rely on a recursive structure for this. The attention mechanism in a Transformer uses multiple scaled dot-product attention units. The attention units generate embeddings of all tokens that include information about the token itself and the weighted combination of other relevant tokens (weighted by attention weights). In particular, each attention unit is associated with three learned weight matrices, namely query weight W Q 、key weight W K 、and value weight W V . The input vector x i of each token i is multiplied by each of the three weight matrices to obtain a query vector q i , a key vector k i , and a value vector v i , respectively. In the original Transformer and ViT, the attention weights from token i to token j are calculated as the dot product between qi and kj. These are scaled by, where dk is the dimension of the key vector. The scaled attention weights pass through a softmax function, and the value vectors are multiplied by this, i.e., the output of the attention unit for token i is the weighted sum of the value vectors of all tokens weighted by the attention weights from token i to each token. Any machine learning model that includes an attention mechanism as described above or a derivative thereof can be referred to as a Transformer-based model in the context of this disclosure. For example, the derivative of the original Transformer described above includes a relative position bias B for each head. A set of learned matrices W Q ,W K and W VThe attention heads constitute the attention heads. Transformer models typically have multiple attention heads in each of multiple layers. In fact, to extract more information, the attention mechanism can be repeated h times in parallel. Each attention head focuses on the tokens associated with each of the input tokens, and the use of multiple attention heads means that this can be repeated for different definitions of associated. The outputs of the multiple attention heads in a layer then form the final projection matrix W learned for each multi-head attention unit. O They are concatenated and projected using . Additional layers may exist. For example, a fully connected layer typically follows the attention layer. This may include one or more further transformations (e.g., two linear transformations in a typical transformer) and / or one or more activation functions (e.g., ReLU activation in the original transformer, but other activation functions such as GELU have been proposed). Around each of the two sublayers of the fully connected layer, the original transformer uses residual connections. All these components (multi-head attention and subsequent fully connected layers with residual connections) together constitute a single encoder or decoder block, which is repeated n times to produce a transformer-based encoder or decoder. The output of the encoder can be further processed to generate predictions for a particular task. For example, the output of the encoder can be processed by a decoder having a similar architecture for a generation task. The decoder block may include a self-attention mechanism, an attention mechanism over coding, and a feedforward neural network. As another example, the output of the encoder can be processed for classification using a classification head such as a multilayer perceptron (MLP). For example, a class token can be added to the beginning of the input embedding, and a log of the class token can be calculated from the encoder output using an MLP.

[0069] As described above, the multimodal models trained by this disclosure can be further trained or used for any downstream task in which a pre-trained multimodal sequence model can be used. In particular, the pre-trained models can be applied to (and in that context improve upon) methods of prior art, including positive transfer, cross-modal understanding, and cross-modal generation. Positive transfer refers to situations in which training multimodal learning helps improve the performance of the downstream task in which the model is used, compared to performing the same task with an equivalent model trained using a single modality. Multimodal models can be used in downstream tasks using one or more modalities. For example, a multimodal model can be trained using aligned image and text data and then used for image classification using only images as input. Cross-modal understanding refers to situations in which one modality explains another modality. For example, a model can be trained to describe images using text and answer text-based questions about the images. Cross-modal generation refers to the generation of one or more modalities using one or more other modalities. Positive transfer and cross-modal understanding can be utilized in many downstream tasks, including classification and regression. Furthermore, cross-modal generation can be used to transform data between modalities.

[0070] Specifically, this disclosure relates to a computer implementation method for training a multimodal machine learning model using commutative and transitive modeling, wherein the resulting model may be included as part of a machine learning model that includes additional machine learning modules (e.g., classification models, regression models, or generative layers) that are further trained (also known as fine-tuning or transfer learning) and / or trained for any downstream task of interest. Such a model may be called the “foundation model,” and the derived model trained for a downstream task may be called the “task-specific model.” Training and use of both types of models are expressly included in this disclosure.

[0071] Accordingly, some embodiments of this disclosure involve training and / or using a machine learning model to transform data between modalities. The machine learning model may be configured and / or trained to operate commutatively and / or transitively. Thus, the machine learning model may be trained to receive inputs of any combination of multiple modalities (e.g., three or more modalities) as inputs and transform those inputs into outputs of any of the other multiple modalities. The model can link modalities and transition between them using generative pre-training, commutativity (A,B)=(B,A), and transitive relations (A→B)∧(B→C)⇒(A→C). The machine learning model may include a novel LoReTTa (Linking modalities with a tRansitive and commutative pre-Training sTrAtegy) technique, as further described below. The machine learning model may be trained using a self-supervised framework that uses rules of commutativity and / or transitivity for transitions between different modalities and causal generative pre-training. Therefore, relational modality A → modality C can be learned using modality A → modality B → modality C. Given a dataset containing only combinations (A,B) and (B,C), a transformer (or any general neural sequence model) pre-trained according to the techniques disclosed herein can handle any combination of modalities, including invisible pairs (A,C) and triplets (A,B,C), at fine-tuned inference on downstream tasks, while matching the performance of the Oracle model (a model trained on combinations (A,B,C) for all entries).

[0072] The methods of this disclosure may include training a model using transfer learning. Transfer learning is a machine learning technique in which a model is pre-trained to perform a first task and then used (on its own or as part of a machine learning model including additional machine learning modules such as a classification head, a regression head, etc.) to perform a second task different from the first task. The model may be used as is for the second task, or it may be retrained (also called fine-tuning) at least partially for the second task. In such embodiments, the parameters of the model learned for the first task are used as a starting point for optimization for the second task, and some or all of the parameters may be further optimized. Further optimization may involve freezing some of the parameters learned for the first task (i.e., they are not changed during the optimization for the second task) and / or applying a lower learning rate so that the optimization explores a more constrained space around the weights optimized for the first task. For example, a machine learning model may be trained using training data that includes data from multiple modalities and combination techniques such as causal language modeling, masked language modeling, or causally masked modeling, and may be trained using commutative models as described herein and transitive models as described herein. The resulting base model can then be used directly for any generative task of interest, for example, to generate data for one or more modalities using input data from one or more of the other modalities. In general, any generative task is possible, for example, to generate completed data for one or more modalities using input data that includes incomplete data from any one or more of the modalities. The resulting base model can be fine-tuned for any generative task, for example, by fine-tuning arbitrary weights of the base model using task-specific training data (e.g., training data that includes data related to one or more cohorts of subjects of interest).The resulting foundational models can be used as part of a classification or regression model that further includes classification or regression modules. A classification module refers to any machine learning model that can take features extracted from a foundational model (a trained commutative and transitive machine learning model) trained on an input dataset as input and provide an output indicating the class to which the input dataset belongs. For example, in the context of binary classification, the output may be a single probability of belonging to one of two classes. In the context of multi-class classification, the output may include multiple probabilities of belonging to each of the multiple classes. A classification module is sometimes called a “classification head” or “classification layer,” particularly in the context of deep neural networks. A classification layer is typically a single layer in a deep neural network. A classification head can include multiple layers. A regression module refers to any machine learning model that can take features extracted from a foundational model (a trained commutative and transitive machine learning model) trained on an input dataset as input and provide a continuous output of interest. For example, a regression module can predict continuous values ​​such as the probability of survival over a given period, odds ratios, or a patient’s predicted response to a treatment. For example, a regression module may be a survival model such as the Cox proportional hazards model. Regression modules are sometimes called "regression heads" or "regression layers," particularly in the context of deep neural networks. A regression layer is typically a single layer in a deep neural network. A regression head can contain multiple layers.

[0073] Classification refers to the use of machine learning models to classify input data between two or more categories. For example, audio and / or image and / or text data can be classified between multiple categories. A classification method or model may be associated with a level of accuracy evaluated using one or more metrics on one or more test datasets. For example, any classification accuracy metric known in the art may be used, including top-k accuracy (where k can be, for example, 1, 2, or 3), classification accuracy (% correct prediction), or F1 score. Top-k accuracy is the number or proportion of times a correct classification label is within the top k predicted labels, ranked by the probability of the class. The F1 score is the harmonic mean of binary classification accuracy (number of true positive predictions divided by the total number of positive predictions, including true and false positives) and recall (number of true positive predictions divided by the total number of true positive instances, i.e., the sum of true positive predictions and false negative predictions). In a multi-class classification problem, the F1 score is obtained as the average of the F1 scores for each class (optionally weighted by class proportions to account for class imbalances). Classification accuracy is the ratio of the number of correct predictions made for a dataset to the number of predictions made. A regression method or model may be associated with a level of accuracy assessed using one or more metrics on one or more test datasets. Any regression accuracy metric known in the art may be used, including, for example, the c-exponent (especially in survival models), mean squared error, root mean squared error, and mean absolute error. The c-exponent (coincident exponent) is a generalization of the area under the ROC curve (AUC) that can account for censored data. In survival models, the c-exponent represents the model's ability to accurately provide a reliable ranking of survival times based on individual risk scores. It is a measure of the probability that the predicted event times of two observations (subjects) have the same relative order as their true event times. The c-exponent can be calculated as follows: TIFF2026516833000002.tif1276 Here, α i This is the risk score for the observed value i, and T i and T j These are the event times for observed values ​​i and j, and 1Tj<Ti is T j <T i If it is true, it is equal to 1; otherwise, it is 0. αj>αi is α j >α i It is equal to 1 if the value is 0, and 0 otherwise. A value of 1 corresponds to the best possible prediction, and a value of 0.5 corresponds to a random prediction. The mean squared error (MSE) is the average of the squared differences between predicted values ​​and predicted values ​​(ground truth) in the dataset. The root mean squared error (RMSE) is the square root of the mean squared error. The mean absolute error (MAE) is the average of the absolute differences between predicted values ​​and predicted values ​​(ground truth) in the dataset. RMSE, MSE, and MAE are such that a better-performing model is associated with a lower value of these metrics. A generative sequence model may be associated with a level of accuracy evaluated using one or more metrics on one or more test datasets. Any metric known in the art, e.g., perplexity, may be used to evaluate a sequence model. Perplexity is the power-mean negative log-likelihood of a sequence. Tokenized sequence X = (x0, x1, ..., x t In the case of ), the perplexity of X is The filename is TIFF2026516833000003.tif975. Higher perplexity is associated with higher-performing models.

[0074] The method of this disclosure is expected to perform better than a comparison model trained on the same multimodal training data and / or the same computational load. Performance can be evaluated using any metric known in the art, depending on the task to which the model is configured to perform, as described above, for example. The method of this disclosure is expected to perform better than a comparison model for a prediction task, where the input data for the prediction task includes observations of modality pairs that are not represented, or are not well represented, in the training data used to train the model. A modality pair may be considered poorly represented in the training dataset if the number of training data pairs containing data for a modality pair is significantly less than the number of training data pairs containing data for any other modality pair. For example, the number of training data pairs containing data for a modality pair may be less than 50%, less than 40%, less than 30%, less than 20%, less than 10%, less than 5%, or less than 1% of the number of training data pairs containing data for any other modality pair. The advantages associated with the method disclosed herein are expected to be present in any such situation and increase as the amount of training data for any pair of modalities decreases.

[0075] The methods of embodiments of this disclosure include tokenizing data before providing it to a machine learning model as input data. Tokenization refers to the process of obtaining a representation of data in a given numerical space. A tokenized representation may also be called an embedding. For example, if you use a transformer-based model, you must first tokenize the data because transformer models take a set of tokens as input. There are several methods known in the art for converting data into tokens, and any such known method can be used. For example, data can be converted into tokens directly using the underlying raw byte stream (e.g., Reed et al., 2022). For example, text, nominal, and other discrete data can all be encoded using lookup tables (also called dictionaries). Specifically, examples include tokenization of text data using Byte Pair Encoding (e.g., BERT), SentencePiece (e.g., as in the case of GPT), or WordPiece. In another specific example, raw numerical data can be binned into a predetermined number of bins, each associated with a value, and together form a dictionary of a size corresponding to the predetermined number of bins. As another example, low-dimensional audio, images, and other continuous data can all be encoded by their raw byte values ​​(e.g., ImageGPT, Perceiver, etc.). For example, each pixel of an image or the frequency of an audio file can be used as a single token. High-dimensional data can be tokenized by reducing the first dimension. Dimensionality reduction can be performed by projection onto a lower-dimensional space and / or data partitioning (e.g., patching). For example, high-dimensional continuous data such as a large image or audio can be tokenized by first patching via a codebook or dictionary. A codebook is a vector representation in discrete latent space. Codebooks can be obtained from vector quantized variational autoencoders (e.g., DALLE and MusicLM, etc.). Note that codebooks can also be used without prior dimensionality reduction.A dictionary is a lookup table in which original values ​​in data are mapped to a predetermined number of values ​​in the dictionary. In embodiments, a codebook is obtained using a learned function that projects the input data into a latent space of a predetermined dimension d. This may use multiple trainable weights that are trained concurrently with (or before) the rest of the machine learning model. For example, a flattened image or image patch can be projected linearly into the latent space using weights trained concurrently with the rest of the model. A patch (also called a “tile”) is a subsection of a larger data object, such as an image. Patches are sometimes referred to as “tiles”. A patch may have a predetermined size, typically represented by the number of pixels in each of the two dimensions, in the case of an image. Patches may be non-overlapping patches. A flattened image or patch is a data object corresponding to an image in which a two-dimensional matrix containing the values ​​of image pixels is transformed into a one-dimensional vector by concatenating rows or columns. Dimensionality reduction by projection into a lower-dimensional space can be achieved by applying principal component analysis to the training dataset to identify multiple principal components and then projecting arbitrary input data along a predetermined set of principal components (e.g., the first 10, 20, 100, 200, 1000 principal components, depending on the desired size of the resulting data).

[0076] The methods of this disclosure will find applications in many fields, particularly in fields where it is difficult or impossible to obtain or collect fully aligned multimodal data. Fully aligned multimodal data refers to multimodal data in which each observation is associated with data for each of the multiple modalities considered (strongly aligned data), and / or for all possible pairs of data modalities, there is a significant amount of aligned paired data (weakly aligned data). Multimodal data refers to a dataset that includes data associated with different types of information and / or data acquired using different sensors / acquisition means. For example, multimodal data may include data obtained from multiple sensors measuring different physical, physiological, physicochemical, or biological parameters. As another example, multimodal data may include data in the form of images, text, audio, etc. As another example, multimodal data may include data from multiple modalities selected from medical imaging (e.g., imaging data of such a type, such as MRI, CT, or PET imaging data, each representing a different modality), electronic medical record data (e.g., electronic medical record representations of medical history, text and / or categorical data, including one or more of procedures, comorbidities, etc., each of which may represent a different modality), and biological sample data (e.g., omics data obtained from a sample, or digital pathology imaging data, and any type of omics data or digital pathology imaging data may represent a different modality). Omics data may refer to, for example, gene sequence data (e.g., gene sequence presentations, which may include the identification of the presence of one or more gene sequences or variants in a subject or a sample derived from a subject, or the presence or expression level of one or more gene expression products in a sample derived from a subject).Omics data can be selected from mRNA expression data (e.g., mRNA sequencing data, gene expression array data, RT-PCR data, digital PCR data, etc.), genomic data (e.g., genome sequencing data, e.g., whole genome sequencing data, whole exome sequencing data, targeted panel sequencing data or data derived therefrom, e.g., copy number information, variant information, mutation signature information, etc.), epigenetic data (e.g., DNA and / or histone modification data), proteomics data (e.g., data from reversed-phase protein arrays, mass spectrometry, etc.), and metabolomics data (e.g., data from mass spectrometry, targeted detection assays such as glucose, or any metabolite detection assay, including but not limited to these). The term "medical image" refers to any digital image of a biological subject. A medical image may be an image of the subject or a part of the subject. For example, a medical image may be a radiographic image, magnetic resonance imaging (MRI) image, positron emission tomography (PET) image, or ultrasound image. Any imaging techniques used in clinical settings to make decisions based on morphological features visible in images may benefit from the methods of this disclosure. As used herein, the term "digital pathology" encompasses the analysis of histopathology images (images of tissues), cytopathology images (images containing individual cells that are analyzed for their individual characteristics), hematopathology images (images of blood or blood cell samples), or anatomical pathology images (images of organs or tissues at various levels of resolution). Digital pathology images may be images of fixed biological specimens stained with one or more stains. For example, any staining method known in the art may be used, such as H&E staining (hematoxylin and eosin), immunofluorescence, silver nitrate, Romanowski-Giemsa staining, and trichrome staining. Medical images and digital pathology images are generally analyzed in contexts such as the diagnosis, prognosis, and treatment of cancer; the diagnosis, prognosis, and treatment of neurodegenerative diseases; and the diagnosis, prognosis, and treatment of cardiovascular diseases. The same applies to omics data and electronic health record data.Therefore, omics data and / or electronic health record data and / or medical image data and / or digital pathology image data are often used alone or in combination with each other to obtain diagnosis, prognosis, treatment recommendations, or patient selection (e.g., to participate in clinical trials) for many conditions including but not limited to cancer, neurological disorders, and cardiovascular diseases. Machine learning techniques for performing these tasks benefit from reproducibility, reliability, speed, and higher accuracy than analysis by trained healthcare professionals. However, prior to this disclosure, there were no multimodal machine learning techniques specifically designed to address training datasets containing missing modality pairs, and therefore no machine learning techniques have been proposed that can leverage multimodal data to the extent currently done by manual analysis. This disclosure provides a solution that can address this, thereby opening up new possibilities for multimodal machine learning-based prediction in the medical field. Accordingly, methods for processing input data related to subjects for the purpose of disease diagnosis, prognosis, monitoring, or treatment identification are also described herein, and the methods include processing such input data using the models or methods described herein. A biological sample or specimen may be any sample containing cells, whether obtained directly from a subject or from a cell or tissue culture. In embodiments, the biological sample is a sample previously obtained from a subject. The sample may be a tissue or body fluid sample (e.g., a biopsy or liquid biopsy).

[0077] Figure 1 shows a flowchart of a method for analyzing data using the transitive and commutative machine learning models described herein, and a method for training the transitive and commutative machine learning models described herein.

[0078] Next, we describe a method for training transitive and commutative machine learning models. In step 100, the training dataset is acquired by a processor. The training data may be received from a user interface, a database, a data store, a computing device, or a data acquisition device. All subsequent steps are performed by the processor that acquired the training data or a processor communicatively coupled to the processor. The training data includes data for at least three sets of modalities. Thus, step 100 may include accessing a training dataset that includes data for each of at least three sets of modalities. The training data may lack data that associates a first data for a particular input modality with a second data for a particular output modality. The training data may include, or consist of, data that associates data for pairs of modalities in at least three sets of modalities, and for each pair of first and second modalities in at least three sets of modalities, the training data included data that associates data for pairs of modalities that together form a path between the first and second modalities. In the embodiment, for each pair of first and second modalities in a set of at least three modalities, the training data includes data relating the data of the first modality to the third modality and data relating the data of the third modality to the second modality. Thus, the training data does not have to include training data relating any pair of modalities, and may include data relating missing modalities in a pair using one or more bridge (also referred to herein as “links”) modalities. The training data may include or consist of tuples (e.g., pairs) containing corresponding data for multiple modalities in a set of at least three modalities, and tuples containing data for a particular pair of modalities in a set of at least three modalities were present in the training data in less than 50%, less than 40%, less than 30%, less than 20%, less than 10%, less than 5%, or less than 1% of the number of tuples containing data for all other pairs of modalities in a set of at least three modalities.In the embodiment, the tuple did not contain a tuple containing data for a particular pair of modalities from a set of at least three modalities (i.e., one pair of modalities was completely or substantially missing from the training data).

[0079] Three or more modalities can be selected from medical or biological sample data related to the subject. For example, one or more modalities can be selected from medical imaging data, electronic medical record data, gene sequence data, and digital pathology imaging data, and optionally, medical imaging data can be selected from MRI, CT, or PET imaging data.

[0080] In step 110, the training data is used to obtain input data for a machine learning model. This may include one or more of the following steps: In step 110A, dimensionality reduction may be applied to the data of one or more modalities in the training data, for example, using PCA. In step 110B, the data of each modality in the training data may be tokenized individually using their respective predetermined tokenization schemes. In step 110C, the tokenized data obtained in step 110B may be projected onto a common feature space across modalities. This may be done using parameterized embedding functions. In step 110D, pairs of input data may be generated, each containing a first data point for a first modality and a second associated data point for a second modality. Note that the training data may also contain data (also called "tuples") that associate three or more different modalities (i.e., data from three or more modalities for the same subject or object). In such cases, multiple pairs can be sampled from the same tuple in step 110D. In step 110E, one or more modality-specific tokens may be added to the pairs obtained in step 110D to indicate the start and / or end of a modality. In step 110F, modality-specific absolute position coding may be added to the feature vectors obtained in step 110D or 110E. This can be learned concurrently with the model weights.

[0081] Model training uses pairs obtained from training data, which include first data for a first modality and associated second data for a second modality. Before providing multimodal pairs (e.g., pairs obtained in steps 110D, 110E, or 110F) as input to the model for training, the order of the two modalities in each pair may be randomly sampled between primary and secondary. Alternatively, pairs corresponding to both orders may be generated. This is called commutative modeling, which is performed in step 120. Thus, step 120 may include generating training ground truth data from the training dataset, where the ground truth data includes concatenated input data of the first and second modalities of the data pair for each of the multiple data pairs in the training data, and the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair with a predetermined probability. The predetermined probability may be 50%. This allows the model to infer the first modality from the second modality and the second modality from the first modality in step 130. Thus, the model may be trained in the first steps 120-130, in which the model is trained using commutative modeling and self-supervised learning objectives.

[0082] In step 130, pairs can be used to train a model using any variation thereof, such as causal language modeling (CLM), masked modeling (MM), and / or causal masked modeling (CMM), prefix language modeling [PLM, Raffel et al., 2019], substitution modeling [Yang et al., 2020], unified language learning [uniLM, Dong et al., 2019; UL2 / denoiser mixture, Tay et al., 2023], LLM2Vec [Behnam Ghader et al., 2024], and GIVT-Causal / GIVT-MaskGIT [Tschannen et al., 2024]. Step 130 may include taking partially masked concatenated input data for a first modality and a second modality of a data pair in a training dataset, each containing ground truth data for multiple data pairs, each containing corresponding data for a modality pair, and training a model to produce predictions for the masked input data as output, wherein the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair, and / or the ground truth data contains the first concatenated input data and the second concatenated input data for each of the multiple data pairs in the training dataset, wherein the modality of the data pair used as the first modality differs between the first concatenated input data and the second concatenated input data.

[0083] For example, in embodiments such as those shown below, the model is trained in step 130 using causally masked modeling (CMM). In this step, the model is trained to learn the conditional distribution of any pair of modalities present in the training data. Causal generative modeling, masked modeling, and / or variations thereof may use ground truth data, which includes concatenated input data for the first and second modalities of a data pair, for each of several data pairs in the training data. The modality of a data pair used as the first modality can be randomly sampled from the two modalities of the data pair. Alternatively, the ground truth data may include a first concatenated input data and a second concatenated input data for each of several data pairs in the training data, where the modality of the data pair used as the first modality differs between the first concatenated input data and the second concatenated input data, and either of these options ensures interchangeability when training the model using self-supervised goals such as causal language modeling (CLM), masked modeling (MM), and / or variations thereof. Causal masked modeling may include identifying a subset of tokens in the training dataset, where the tokens in the subset of tokens are located at multiple locations in the training dataset, appending the subset of tokens to the training dataset, and adding a mask to the multiple locations in the training dataset. Causal masked modeling may further include using an identifying representation of the masked multiple locations during training.

[0084] In step 140, the model is further trained using transitive modeling. The model weights in step 140 may be initialized to the weights learned in the first steps 120-130 using commutative modeling, and then further trained using transitive modeling in step 140. Alternatively, training using commutative and transitive modeling may be combined. Training using transitive modeling may include training the model using transitive modeling, which may include, for each of several data pairs in the training dataset, each containing first data for a first modality and corresponding second data for a second modality, predicting third data for a third modality from the set of at least three modalities corresponding to the second data using a machine learning model that takes the second data as input, predicting final data for the first modality corresponding to the predicted third data using transitive and commutative machine learning models that take the third data as input, and calculating a loss comparing the first data with the predicted final data. In such embodiments, the training data may include (i) pairs of data relating to a second modality and corresponding data relating to a third modality, and (ii) pairs of data relating to a third modality and corresponding data relating to a first modality.Training a model using transitive modeling may include: predicting a third data for a third modality in the set of at least three modalities corresponding to the second data, using a machine learning model that takes the second data as input, for each of several data pairs in the training dataset, each containing a first data for a first modality and a corresponding second data for a second modality, using a machine learning model that takes the second data as input; iteratively predicting subsequent data for further modalities in the set of at least three modalities corresponding to the predicted third data or predicted subsequent data, using a transitive and commutative machine learning model that takes the predicted third data or predicted subsequent data as input; predicting final data for the first modality corresponding to the most recently predicted iteratively predicted subsequent data using a transitive and commutative machine learning model that takes the most recently predicted iteratively predicted subsequent data as input; and calculating a loss comparing the first data with the predicted final data. In such embodiments, the training data includes (i) pairs of data relating to a second modality and corresponding data relating to a third modality, and (ii) pairs of data relating to the latest further modality and corresponding data relating to the first modality, and each pair of data relating to two modalities used in the step of iteratively predicting subsequent data relating to further modalities.In other words, the method may include, in step 100, accessing a training dataset containing first data for a particular input modality and corresponding second data for another modality, and in step 140, generating a first predicted output dataset by processing the first data and a discriminant representation of the other modality using a transitive and commutative machine learning model, wherein the first predicted output dataset corresponds to the other modality, and processing the first predicted output data and a discriminant representation of a particular target output modality using a transitive and commutative machine learning model. This may include generating a second predicted output dataset, wherein the second predicted output dataset corresponds to a specific target output modality; generating a third predicted output dataset by processing the second predicted output data and the discriminative representation of a specific input modality using a transitive and commutative machine learning model, wherein the third predicted output dataset corresponds to a specific input modality; and calculating the loss by comparing at least a portion of the third predicted output dataset with the first data.

[0085] The trained model resulting from step 140 is a transitive and commutative machine learning model in that it is a machine learning model trained to transform any of the modalities and their respective input datasets from a set of at least three modalities to output any of the other modalities and their respective datasets from a set of at least three modalities, the machine learning model is trained to transform data between two modalities by using training data that includes data pairs corresponding to a first modality and another modality, and other data pairs corresponding to another modality and a second modality, the machine learning model is trained to perform corresponding inverse modality transformations for any forward modality transformations the model is trained to perform. Steps 130 and 140 may both be described as training a machine learning model to learn a first conditional distribution (P(C|A), P(A|C)) for a modality pair (A,C), wherein the training data for the modality pair (A,C) does not include enough training data pairs to relate the data for the modality pair to parameterize the conditional distribution, and steps 130 and 140 may also be described as training a machine learning model to learn the first conditional distribution (P(C|A), P(A|C)) by predicting data for one or more additional modalities (B) that form a link between the modality pairs using the learned further conditional distribution (P(C|B)) for the modality pair (C,B) that form the link, wherein the training data for the modality pair (C,B) that form the link includes enough training data pairs to relate the data for parameterizing the further conditional distribution.

[0086] The trained model resulting from step 140 may be considered the final trained model for use on the input dataset for generative tasks, such as filling or transforming between modalities. Alternatively, the model may be further trained in step 150 for different generative tasks and / or included as part of a classification or regression model trained in step 150. The classification model may include the model obtained in step 140 (a transitive and commutative machine learning model) and a classification model trained in step 150 to take features extracted from the transitive and commutative machine learning model as input and produce classifications on the input dataset provided to the transitive and commutative machine learning model as output. The classification model can be trained in step 150 using training data that includes data from at least three sets of modalities and one or more associated ground truth classification labels. The generative model may be a model that includes or consists of the trained machine learning model obtained in step 140, which is further trained in step 150 to take an input dataset as input and produce a generated output dataset corresponding to the input dataset, where the output dataset corresponds to one modality of a set of at least three modalities that are not present in the input dataset. In other words, the generative model can be trained to take an input dataset as input and produce a generated output dataset corresponding to the input dataset, where the output dataset corresponds to one modality of a set of at least three modalities that are not present in the input dataset. Alternatively, or in addition to the above, the generative model can be trained to take an input dataset containing missing data as input and produce a generated output dataset corresponding to the missing data in the input dataset. The trained machine learning model obtained in step 140 can be used as the generative model, or the trained machine learning model obtained in step 140 can be fine-tuned in step 150 using further training data to obtain a generative model.The regression model may include the model obtained in step 140 (a transitive and commutative machine learning model) and the regression model trained in step 150 to take features extracted from the transitive and commutative machine learning model as input and generate regression values ​​of the input dataset as output. The regression model may be a Cox proportional hazards model, and the regression values ​​may be survival predictions. Alternatively, the regression values ​​may be any continuous values ​​related to the input data, such as probabilities (e.g., the probability of having or developing a disease, the probability of responding to treatment, etc.). The regression model may be trained in step 150 using training data and associated ground truth regression values, which may include data from one or more sets of at least three modalities.

[0087] For example, a machine learning model, trained in step 150, can take medical or biological sample data related to a subject as input and generate a display of recommended treatments for the subject as output. The display could be a classification. The medical or biological sample data may include one or more of the following: radiographic images, gene sequences, and microscope slides. As another example, a machine learning model can be trained in steps 130–140 and optionally further in step 150 to take data collected by a vehicle, one or more of several types of sensors optionally associated with the vehicle as input, and generate predicted sensor data for other types of sensor data associated with the vehicle as output, and optionally, the machine learning model is trained using synchronized data from pairs of several types of sensors.

[0088] In any step 160, one or more machine learning models or their arbitrary parameters obtained as a result of step 140 or 150' are provided to the user or computing device.

[0089] Training a machine learning model applies commutative modeling. Commutativity is ensured by randomly mixing the order of concatenation when appending tokens from two modalities to an input sequence, as described above. This allows the model to infer modality A from B and modality B from A. Random mixing is performed by randomly reversing the default order in which modalities are presented as input to the machine learning model, using a predetermined probability. In a fully commutative model, the probability can be 50%, meaning that pairs of input A and input B are likely to be presented as (A, B) or (B, A). This probability can be set to a value different from 50% (e.g., less than 50%) to prioritize learning in a given direction. This can be useful in certain contexts, depending, for example, on the task the model is intended to be trained for, as well as the combination of modalities intended to be used when testing and / or deploying the model. In embodiments, commutative modeling is used to pre-train a model using a self-supervised learning objective. The self-supervised learning objective may be a causal generative modeling objective. Therefore, models can be trained using commutative modeling in the context of self-supervised learning tasks such as causal generative modeling. Causal generative modeling may include one or more of causal language modeling, masked language modeling, and their derivatives.

[0090] Training a machine learning model involves applying transitive modeling. Consider an example where three modalities A, B, and C are modeled, and only the combinations (A,B) and (B,C) are present in the training data. Training a machine learning model can be done by randomly sampling available pairs in the training data (e.g., (a,b), (b,a), (b,c), or (c,b)) during training and applying transitive modeling. Transitive modeling involves predicting missing modalities in a multimodal dataset by using the learned relationships between the available modalities for the pairs of data to infer any missing data points required for a transition between two modalities (also referred to herein as bridging or link modalities; in the example above, B is the link modality between modality A and modality C). For example, consider the pair (a,b). Since the model is trained to predict modality C from B, it is possible to infer a pseudo-data point c from b (i.e., predict the value of c even if ground truth data is not available for the value c corresponding to data a). Since the model can also predict modality A from C, we use a new sample c (pseudodata point) to predict point a again, ensuring that all modalities are aligned. The same principle can be applied to the pair (b, a), from which we can infer a pseudodata point c, which can then be used to predict point b, minimizing the loss between the predicted point b and the ground truth point b, the loss relating to the prediction of the pseudodata point c from point a and the prediction of b from the pseudodata point c.

[0091] In an embodiment, acquiring or generating input data in step 110 may include generating an input dataset by tokenizing an initial dataset (step 110B). In an embodiment, acquiring or generating input data in step 110 may include tokenizing an initial dataset (step 110B) and projecting the tokenized initial dataset into a predetermined feature space using a parameterized embedding function for each modality (step 110C). A parameterized embedding function refers to a function with learned parameters, which can be implemented as one or more additional layers of a deep learning model, the parameters of which are learned when the deep learning model is trained. In an embodiment, acquiring or generating input data in step 110 may include generating an input dataset in step 110E by adding and / or appending modality-specific tokens corresponding to each single modality in the input dataset to the beginning of the optionally projected data. In an embodiment, acquiring or generating input data in step 110 may include including learned location embeddings in step 110F. Learned location embeddings are location embeddings that are learned simultaneously with the training of a machine learning model.

[0092] In one embodiment, all input data is tokenized using a predetermined selected method. After tokenization, all tokens are projected into a common feature space using a parameterized embedding function (e.g., tokenized representations of samples a, b, and c for modalities A, B, and C are projected into a shared vector space: a → [a_1,...,a_l], b → [b_1,...,b_m], and c → [c_1,...,c_n]). In embodiments, the embedding function is implemented as a linear layer with parameters learned together with other parameters of the model (i.e., the weights of all layers, including the embedding layer, are learned together). This is preferably simple. Alternative implementations of the parameterized embedding function include convolutional layers or multiple layers (i.e., an additional neural network for embedding), such as a transformer block. Such implementations typically involve more parameters for training but may be beneficial in some situations depending on the complexity of the input data. Furthermore, modality-specific tokens are added to indicate the start and end of modalities. In the embodiment, learned modality-specific absolute position coding is added to the feature vector. Then, the embeddings for each modality are concatenated when they are aligned (for example, if a and b are aligned, the input to the model is s=[a_1,...,a_l;b_1,...,b_m] or s=[b_1,...,b_m;a_1,...,a_l], and the use of both orders ensures commutativity). For example, in the case of audio-image pairs or image-text pairs describing the same concept, their token sequences can be concatenated and fed into a model (e.g., a transformer model). However, if only one modality is available, only the token sequence for that modality is used as input to the model.

[0093] In embodiments, the model is trained using causal generative modeling. Causal generative modeling refers to the task of predicting the next token in a given sequence of tokens, where a sequence of tokens s_1:L and parameters θ are given, as modeling the data using a probabilistic chain rule: log pθ(s1,.....,sL)=sum log pθ(sl|s1,....,sl-1). As described above, commutativity can be ensured in this framework by randomly mixing the order of concatenation when adding tokens from two modalities to form the input sequence. This allows the model to infer modality A from B and modality B from A. This is referred to herein as commutative modeling.

[0094] In embodiments, training a model using causal generative modeling involves using causal masked modeling. Causal masked modeling randomly masks a span (or many spans) of tokens and moves them to the end. This allows the model to still use the information after masking to predict the masked position, which is not possible with classical generative modeling. This is advantageous, though not required, for the execution of the models described herein. Indeed, models such as generative pre-trained transformers are known to consider only information from the left, ignore information from the right, and lack bidirectional context, which is important for many downstream tasks. The use of causal masked modeling allows for training a universal model that can be fine-tuned for many different downstream tasks. Other techniques that integrate masked and causal modeling can also be used, such as prefix language modeling (PLM), permutation modeling, unified language learning, LLM2Vec, and GIVT-Causal / GIVT-MaskGIT. The latter is associated with a specific architecture (Generative Infinite Vocabulary Transformer, GIVT), but other techniques are generally applicable to decoder-only language models.

[0095] In embodiments, a machine learning model may be trained to prioritize each of the commutative and / or transitive processes. Such training may include configuring the input to the machine learning model to identify a specific target output modality (for example, since it may be possible to transform any given input into the output of multiple output modalities). The training may further or alternatively include defining training techniques and / or loss functions (e.g., cross-entropy loss functions) for transforming data bidirectionally and / or across three or more types of data, and / or prioritizing them.

[0096] In embodiments, a machine learning model may be pre-trained using commutative modeling to self-supervised learning objectives (e.g., causal language modeling, masked language modeling, and / or derivatives thereof, as further described herein), but without transitive modeling. This may involve using available pairs of linked data modalities. The resulting pre-trained model can then be used as a starting point for further training using transitive modeling. In other words, a machine learning model may be trained using a first step of commutative modeling and self-supervised learning objectives, a second step in which the weights of the machine learning model are initialized to the weights learned in the first step, and then further trained using transitive modeling.

[0097] Next, the method for analyzing the data will be described. In step 100', the data on which predictions are to be made is acquired by the processor. The acquired dataset includes data from one or more modalities selected from a set of at least three modalities. The set of at least three modalities corresponds to the set of at least three modalities that the machine learning model used in step 170 below is trained to process. Step 100' may also include receiving data related to one or more modalities. Step 100' may also include acquiring or accessing input data (e.g., medical data or biological sample data) by a processor that receives input data from a database, data acquisition means (e.g., medical data image acquisition means, biological sample data acquisition means such as a microscope, and associated sequencers or computing devices), computer-readable media, a user interface, or a computing device. This may include receiving data from a user interface, a database, a data store, a computing device, or a data acquisition device. All subsequent steps are performed by the processor that acquired the data, or a processor communicatively coupled to said processor. In any step 110', the data is processed to obtain input data for a machine learning model trained as described herein. Step 110' may include one or more of the following: dimensionality reduction in step 110A', tokenization in step 110B', projection onto a common feature space in step 110C', addition of modality-specific tokens in step 110E', and addition of positional coding in step 110F'. Any of these steps can be performed as described above in relation to steps 110A, 110B, 110C, 110E, and 110F.

[0098] In step 170, the input data is processed using a trained machine learning model described herein, for example, obtained in step 140 or step 150. Thus, step 170 may also include generating an output dataset by processing the input dataset using a machine learning model, which includes a transitive and commutative machine learning model, the transitive and commutative machine learning model being a machine learning model trained to transform any of the input datasets of any of the modalities from a set of at least three modalities and each corresponding to it, to output any of the other modalities from the set of at least three modalities and each corresponding dataset, the machine learning model being trained to transform data between two modalities by using training data including data pairs corresponding to a first modality and another modality of two modalities, and other data pairs corresponding to another modality and a second modality of two modalities, the machine learning model being trained to perform corresponding inverse modality transformations with respect to any forward modality transformations the model is trained to perform. In an optional step 180, derived data may be obtained from the output of a machine learning model, such as prognosis indications, treatment selections, diagnostic indications, or selections for participation in clinical trials. This step is optional, and the output of the machine learning model obtained in step 170 may already be directly interpretable as one of these types of information. In step 190, the results of one or more of the aforementioned steps are provided to the user, for example, through a user interface. For example, the results may include classification predictions, regression predictions, predicted output data related to one or more of several modalities (e.g., the output of a generative model), trained parameters of one or more trained machine learning models, the trained machine learning models, and / or any information derived therefrom. The data store may be a public or private database.The information derived from classification prediction may include one or more of the following: a prognosis derived from a classification obtained using a machine learning model, a treatment indication derived from a classification obtained using a machine learning model, and an indication of suitability for participation in a clinical trial derived from a classification obtained using a machine learning model.

[0099] A trained machine learning model may be used to generate an output dataset for a specific target output modality using input datasets from one or more other modalities. Thus, step 100' may include determining a specific input modality for the input dataset and receiving a representation of a specific target output modality. Step 170 may include generating an output dataset by processing the input dataset and a discriminative representation of a specific target output modality using transitive and commutative machine learning models.

[0100] In embodiments, as described above, a machine learning model may be used to train and / or transform data between modalities. Thus, the model may be used to identify the output of a particular target output modality using the input dataset of a particular input modality (other than one). Once the output of a particular target output modality is identified, the output may be output (e.g., transmitted to another device or presented via an interface), the output may trigger or be used to identify an action (e.g., movement of an artificial arm or driving control in a car), and / or the output may be used to generate another result (e.g., output, may trigger another action, or may be used to identify another action). As an example, a particular target output modality may include a predicted MRI scan, which can then be post-processed to identify a potential diagnosis. The results may be generated to include a possible diagnosis, the predicted MRI scan, and one or more possible actions (e.g., determined based on the potential diagnosis). The results can be transmitted to a caregiver's device.

[0101] Collecting multimodal datasets with two pairs of modalities A and B or B and C is practically difficult. Obtaining datasets with three aligned modalities A, B, and C is even more difficult. For example, in healthcare, one patient may only have gene sequences and microscopic images, while another patient may only have gene sequences and radiographic images. This makes it difficult to integrate and combine all modalities into a large pre-trained model. This disclosure provides a solution to this unexplored problem. Embodiments of this disclosure utilize a self-supervised framework that combines rules of commutativity and translocation for transitions between different modalities with causal generative pre-training. Thus, the proposed solution can learn the relationship A→C using A→B→C in contacts where the training data contains only aligned data for pairs (A,B) and (B,C), but does not contain aligned data for pair (A,C) (or a very small number, e.g., much smaller than the other two pairs). In particular, the proposed method uses gestational pretraining, commutativity (A,B)=(B,A), and transitivity (A→B)∧(B→C)⇒(A→C) to link modalities and enable transitions between them. Considering a dataset containing only combinations of (A,B) and (B,C), the inventors found that transformers pretrained with the novel proposed method can handle any combination of modalities in fine-tuned inference time for downstream tasks, including invisible pairs (A,C) and triplets (A,B,C), while satisfying the performance of the Oracle model. By learning intra-token and inter-token relationships from different data distributions, the inventors demonstrated in the following embodiments that this method can function as a general and multi-mode feature extractor. This has far-reaching implications for safety-critical fields such as healthcare, infrastructure, and transportation. For example, in medicine, it is already difficult to obtain large datasets with a single modality to train robust foundational models, such as those used in computer vision and natural language processing. Finding datasets with two or more modalities is even more difficult.However, combining these is important because it is common practice to examine different types of data, such as radiographic images, gene sequences, and surgical samples, in order to determine the optimal course of action.

[0102] By adding or combining different modalities during pre-training, three possibilities open up: positive transfer, cross-modal understanding, and cross-modal generation. Positive transfer occurs when learning in one modality helps improve performance in another. For example, aligning images with text has been shown to result in better classification performance on ImageNet (see CLIP, SimVLM, etc.). Cross-modal understanding refers to describing one modality using another. For example, it is possible to describe an image using text (see Flamingo, etc.) or understand a protein from a sequence (see AlphaFold, etc.). Cross-modal generation allows one modality to generate one modality using the other. Examples include text-to-image generation (e.g., DALLE-2) and text-to-music generation (e.g., MusicLM). All of these models are based on either contrast learning (e.g., ALIGN, CLIP, FLORENCE), masked modeling (e.g., VisualBERT, ViLBERT, Pixel-BERT, ImageBERT, VL-BERT, VD-BERT, LXMERT, UNITER, VinVL, BeiT-3), or causal modeling (e.g., SimVLM, Flamingo, Gato, DaVinci, CM3, DALLE-1, Parti, Pali, GPT-4). This disclosure provides a novel method for training models capable of performing each of these types of tasks. The method can integrate existing methods in masked and causal modeling into a general self-supervised framework. Models trained as described herein can handle different types of modality combinations in various downstream tasks such as classification. In contrast to previous contributions that only consider randomly missing modalities, the method is designed to address situations where entire data pairs are always missing.This creates a powerful model that can be configured and trained to predict many types of data, perform any other type of generative task of interest, or handle any combination of different types of modalities in a variety of downstream tasks, including but not limited to classification and regression or similar tasks such as survival prediction.

[0103] To illustrate the types of modality transformations that can be supported by using transitive and commutative machine learning models (configured and trained according to the various embodiments disclosed herein), the following exemplary use cases are provided. In embodiments, the machine learning models described herein (i.e., transitive and commutative machine learning models) are configured to receive flattened and tokenized versions of either an MRI scan, a PET scan, or a CT scan, and to generate one of the other three types of scans. The machine learning models may be trained on pairs of MRI / CT scans and pairs of CT / PET scans. Once trained, one potential use of the model is to transform an input MRI scan into a predicted PET scan or vice versa, even when it is not possible to obtain simultaneous scans of these types. In embodiments, the machine learning models described herein (i.e., transitive and commutative machine learning models) are configured to receive embedded data collected by one of several types of sensors (e.g., cameras, sonar sensors, lidar sensors, etc.) in or on a vehicle. Pairs of data are generated by synchronizing sensor readings. Once trained, one potential use of the model is to receive an input corresponding to one type of sensor data and then generate another type of sensor data, and another potential use of the model is to receive an input corresponding to two types of sensor data (e.g., concatenated together) and then generate yet another type of sensor data. These uses can result in three or more synchronized sensor datasets. Importantly, traditional signal synchronization is prone to errors, and the more synchronizations performed, the greater the cumulative error. Therefore, by reducing the need for synchronization by using transitive and commutative machine learning models instead, it is actually possible to generate more accurate and richer datasets (and reduce the number of sensors required for a vehicle).In embodiments, the machine learning models described herein (i.e., transitive and commutative machine learning models) are configured to receive embedded data collected by one of several types of sensors (e.g., a camera, an accelerometer to detect movement to a part of the user's body, an EMG electrode to detect muscle contraction, etc.). Pairs of data are generated by synchronizing sensor readings. Once trained, one potential use of the model is to receive an input corresponding to one type of sensor data and then generate another type of sensor data, and another potential use of the model is to receive an input corresponding to two types of sensor data (e.g., concatenated together) and then generate another type of sensor data. These uses can result in three or more synchronized sensor datasets. As mentioned in the example above, conventional signal synchronization is prone to errors, and the more synchronizations performed, the greater the cumulative error. Therefore, by reducing the need for synchronization by using transitive and commutative machine learning models instead, it is actually possible to generate more accurate and richer datasets (and reduce the number of sensors required to generate signals to the limbs).

[0104] Figure 2 shows one embodiment of a system relating to this disclosure for analyzing data and / or providing a trained machine learning model. The system comprises a computing device 1 having a processor 101 and computer-readable memory 102. In the illustrated embodiment, the computing device 1 also comprises a user interface 103, which is shown as a screen but may include any other means of communicating information to the user, such as audible or visual signals. The computing device 1 is communicably connected, for example via a network, to data acquisition means 3 (e.g., medical image data acquisition means and / or omics data acquisition means and / or medical data acquisition means) such as a microscope associated with a histopathology station, a medical imaging device such as an MRI or PET scanner, or a computing device associated therewith, and / or to one or more databases or data stores 2 that store data such as medical data. One or more databases 2 may further store one or more of the following: one or more deep learning algorithms, training data, parameters (e.g., parameters of a deep learning model used to analyze data), clinical and / or sample-related information. The data acquisition means 3 can be configured to acquire medical image data, omics data, or any other type of medical data from a patient or specimen. The computing device may be a smartphone, tablet, server, personal computer, or other computing device. The computing device is configured to perform methods for analyzing data and / or training machine learning models, as described herein. In an alternative embodiment, the computing device 1 is configured to communicate with a remote computing device (not shown) configured itself to perform the method described herein. In such a case, the remote computing device may also be configured to transmit the results of the method to the computing device.Communication between computing device 1 and remote computing device may be via wired or wireless connection, and may be conducted via a local or public network 6, such as the public internet. Data acquisition means 3 and / or data store 2 may be wiredly connected to computing device 1, or, as shown in the figure, may be able to communicate via a wireless connection, such as WiFi and / or the public internet. The connection between computing device 1 and data acquisition means 3 and / or data store 2 may be direct or indirect (for example, via a remote computer).

[0105] The following are presented as examples, but should not be interpreted as limitations on the scope of the patent claims.

[0106] Examples Training multimodal foundational models is challenging due to the limited availability of multimodal datasets. While many public datasets pair images with text, few combine images with audio or text with audio. Datasets that align all three modalities simultaneously are even rarer. Critical domains such as healthcare, infrastructure, or transportation are particularly affected by the missing modalities. This makes it difficult to integrate all modalities into a large, pre-trained neural network that is either out of the box or tuned for different downstream tasks. This embodiment introduces LoReTTa (linking mOdalities with tRansitive and commutativE pre-Training sTrAtegy) to address this understood problem. The described self-supervised framework integrates causal and masked modeling with commutativity and transitivity rules. This allows for the transfer of techniques within and between modalities. As a result, the pre-trained model is better suited to exploring the true underlying joint probability distribution. Considering a dataset containing only disjoint combinations (A, B) and (B, C), LoReTTa is a relational TIFF2026516833000004.tif756 can be modeled. In particular, this embodiment demonstrates that a transformer pre-trained with LoReTTa can handle any mixture of modalities at inference, including invisible pairs (A, C) and triplets (A, B, C). This embodiment broadly evaluates the novel method on synthetic, medical, and reinforcement learning datasets. Across different domains, the novel general-purpose multimodal transformer consistently outperforms strong baselines such as GPT, BERT, and CLIP for tasks involving missing modality tuples.

[0107] method Model Architecture The LoReTTa implementation used in this embodiment uses the autoregressive transformer decoder architecture described by Radford et al.

[2019] , with several modifications as described below. This model includes modified initialization, pre-normalization, and GELU invocation. The inventors chose to use RMSNorm [Zhang et al. 2019] instead of LayerNorm [Ba et al. 2016]. Also, to reduce memory usage and runtime, StandardAttention [Vaswani et al., 2017] was replaced with FlashAttention [Dao et al. 2022]. Following Chowdhery et al. [Chowdhery et al., 2022], this model was trained without dropping out and did not use bias in either high-density grain or layer standards. While any general sequence model could function for predicting the next token, the inventors chose the transformer [Vaswani et al., 2017] in particular for its simplicity and scalability. However, theoretically, this method should also work with other backbones such as the recently proposed RWKV (Peng et al. 2023) and Hyena (Poli et al. 2023).

[0108] Data Representation: Since the transformer takes a set of tokens as input, the input data needs to be tokenized. This example uses common methods such as raw byte coding [Reed et al. 2022, Jaegle et al. 2021, Chen et al. 2020], learned lookup tables [Kudo and Richardson, 2018, Sennrich et al., 2016, Schuster and Nakajima, 2012], and vector quantization [Van Den Oord et al. 2017, Razavi et al. 2019, Ramesh et al. 2021, Agostinelli et al. 2023]. After tokenization, all tokens are projected into a common feature space using a parameterized embedding function (implemented here as a simple linear layer with learned parameters). Furthermore, modality-specific tokens are added to indicate the start and end of the modality. Modality-specific absolute position coding is added to the feature vector and learned. Then, the representations of each modality are concatenated only if they belong to the same subject / subject. For example, if there are image-audio pairs or image-text pairs that describe the same concept, their token sequences are concatenated and fed into the transformer. However, if only one modality is available, only that modality is used. In summary, the preprocessing steps used in this embodiment are described below. 1. Tokenize all samples a, b, c and project them onto a common vector space, and create a tokenized embedding [a1, ..., a l ],[b1,...,b m ], and [c1,...,c n ] obtain 2. If a and b are aligned (i.e., they relate to the same subject / subject), then they are given the formula s=[a1,...,a l ;b1,...,b m ] or s=[b1,...,b m ;a1,...,a l They are linked as ], thereby ensuring interchangeability.

[0109] Pre-training strategy: The transformer model was trained using causal language modeling (CLM). Tokens s 1:L Given a sequence of parameters θ, the data is modeled using a chain rule of probabilities given by equation (1).

number

[0110] Therefore, the model's task is to predict the next token, taking into account the sequence of previous tokens. This objective is modality-independent, as shown in GATO [Reed et al. 2022] and PALM-E [Driess et al. 2023]. The system may even learn a rich internal-world model [Lin et al. 2023, Li et al. 2023]. Compared to conventional CLM methods, the inventors ensured commutativity by randomly mixing the order of concatenation when adding tokens from two modalities to the input sequence, as shown in Figure 4a (step 2 of the preprocessing method described in the "Data Representation" section). This allows the model to infer modality A from B and modality B from A. The inventors called this technique commutative modeling. Interestingly, this simple modification has not been used in recent multimodal foundational models.

[0111] Generative pre-trained transformers are known to lack the bidirectional context crucial for many downstream tasks [Wang et al. 2022] because they consider only information from the left and ignore information from the right. Therefore, the inventors combine masked modeling (GPT-style models) and causal modeling (BERT-style models) into a single framework. This recently developed method is known in the literature under various names: Intermediate Filling (FIM) [Bavarian et al., 2022], Causal Masked Modeling (CM3) [Aghajanyan et al. 2022, Fried et al. 2023], Hybrid Unidirectional Modeling (HybUni) [Artetxe et al. 2022], and Span Breaking + Language Modeling (SCLM) [Tay et al. 2022]. All of these methods are conceptually identical, differing only in minor implementation details. Therefore, references to causally masked modeling throughout this disclosure conceptually encompass any of the above implementations and other implementations of the same concept, which involves randomly masking out a span (or span) of tokens, replacing them with sentinel tokens, and moving the original tokens to the end (as shown in Figure 4b). This modification does not alter the training process based on equation (1). However, it allows the model to predict the masked position using information to the right of the mask, which is not possible with conventional one-way causal modeling. The hybrid view not only yields better performance on downstream tasks after fine-tuning but also retains the usefulness of language modeling. In CM3Leon [Yu et al. 2023], causally masked modeling is scaled up to a large visual language model that can generate and fill both text and images. CM3Leon is trained on much more data and larger transformers than CM3, but uses the same algorithm as CM3.

[0112] Transitive Modeling Next, the inventors extend several language learning paradigms (masked, prefix, causal, causally masked) with a new technique which the inventors call transitive modeling. The above problem of enabling learning in situations where combinations are unconditionally missing requires combining modalities A and C when only separate combinations (A, B) and (B', C) are given. The inventors propose a method of applying transitive modeling by randomly sampling data points (a, b), (b, a), (b', c) or (c, b') during training. For brevity, this description considers the pair (a, b). Since the model is trained to predict modality C from B, a pseudo-data point is taken from b. TIFF2026516833000006.tif54 can be inferred. Next, the newly inferred sample is used to minimize the loss to the original sample a. Using TIFF2026516833000007.tif54, the point The transformer is trained to predict TIFF2026516833000008.tif54, ensuring that all modalities are aligned (Figure 4c). In this way, the conditional distribution of missing pairs (C,A) is modeled, which, due to commutativity, is equal to (A,C). This idea has some conceptual similarities to cycle consistency. However, this method is far more general, as it can be applied to many combinations of modalities as long as link modalities exist, as shown in Figure 4d.

[0113] In fact, upon closer examination, transitive modeling is quite different from cycle consistency. The most common version of cycle consistency was introduced in CycleGAN [Zhu et al. 2017]. Given an input a in domain / modality A, we want to generate an aligned output b in domain / modality B. The model then calculates the discriminative loss D for b and the original a and predicted values. We calculate the reconstruction loss L for TIFF2026516833000009.tif54. Different versions of cycle consistency were used in MCTN [Pham et al. 2019]: Given aligned inputs (a,b) from modalities A and B, a is Used to predict TIFF2026516833000010.tif77, and predicted TIFF2026516833000011.tif77 is Used to predict TIFF2026516833000012.tif54. Next, the reconstruction loss L is applied to a and TIFF2026516833000013.tif54 and b and Compare TIFF2026516833000014.tif77. On the other hand, the transitive modeling described herein begins with aligned modalities (a,b) and (b',c), and this transitive modeling is, Predict TIFF2026516833000015.tif613 To predict TIFF2026516833000016.tif54, use b, and the reconstruction loss is then a and We compare TIFF2026516833000017.tif54. Since this method uses commutative modeling, other directions starting from different samples (b',c) are also learned. The three methods mentioned above are summarized below.

[0114] · CycleGAN: Given a, Model TIFF2026516833000018.tif822, Calculate TIFF2026516833000019.tif1129.

[0115] · MCTN: Given (a,b), Model TIFF2026516833000020.tif721, Calculate TIFF2026516833000021.tif1134.

[0116] • LoReTTa (Method of this Disclosure): Given (a,b), Model TIFF2026516833000022.tif822, Calculate TIFF2026516833000023.tif612.

[0117] Therefore, transitive modeling as described herein is simply, TIFF2026516833000024.tif721 Extending to TIFF2026516833000025.tif728 does not add another step to cycle consistency. Instead, this technique Model TIFF2026516833000026.tif720. Intuitively, this prevents the model from "cheating" by remembering input a to reconstruct a. The technique described herein also does not use cycle loss because a is not used as input and output. Therefore, loss TIFF2026516833000027.tif613 is not periodic with respect to the model's input. However, this still ensures that the predicted modalities as a whole match the data. This has a similar flavor to cycle consistency, but as can be seen above, it is not the same. In summary, LoReTTa transposes within modalities using masked modeling, transposes between modalities using causal modeling, and vice versa thanks to commutative modeling. Furthermore, not all modality combinations need to be available at training time, as this method uses transitive modeling to predict missing modalities and connect them to the original data. In the experiments described below, we demonstrate that this method enables learning of very rich multimodal feature representations.

[0118] Experimental Evaluation Protocol The inventors empirically analyzed LoReTTa on a constructed dataset (SVL-MNIST), a real-world medical dataset (TCGA-OMICS), and an offline reinforcement learning dataset with three modalities (offline game dataset; MUGEN-GAME). For downstream tasks, the inventors selected object classification, survival prediction, and cross-modal transformation, respectively. These three tasks cover both discriminative and generative problems. The inventors evaluated the former by linear probing and the latter in zero-shot scenarios. This allows for direct evaluation of the quality of learned features. The experimental procedure was as follows: First, the inventors performed an ablation study on the constructed dataset to analyze the effects of transient modeling on different modality combinations. Second, the inventors compared LoReTTa with other methods such as masked (BERT), causal (GPT), and controlled (CLIP) modeling on the real-world medical dataset. Thirdly, the inventors compare the new model to a state-of-the-art autoregressive crossmodal transformer on an offline game dataset. The appendix lists all optimization hyperparameters and model configurations. Figure 7 shows pseudocode used for commutative and transitive modeling, as described and used in these evaluations. The following pseudocode provides further insight into how LoReTTA pre-training is implemented. It outlines the key ideas and algorithmic steps for training the model using commutative and transitive modeling. Most importantly, the code demonstrates how both are integrated into a causal modeling framework.

[0119] SVL-MNIST This new method was tested on a custom dataset derived from real-world data containing speech (A), visual (I), and language (T) modalities representing 10 different categories. The speech dataset features approximately 40,000 spectrograms from AudioMNIST [Khacef et al., 2019], the visual dataset contains 70,000 images from MNIST [LeCun et al., 2010], and the language dataset consists of 130,000 documents from WineReviews [Thoutt 2018]. The datasets are linked via their labels 0-9. This technique results in weakly aligned modalities. For clarity, this dataset is called Speech-Vision-Language-MNIST (SVL-MNIST). As shown in Figure 3c, the dataset was randomly split to have exactly the same relationships. In particular, the data was split to obtain bimodal datasets (A,I), (A,T), and (I,T), each containing 12,000 non-overlapping samples. All remaining samples are part of the unimodal datasets A, I, and T. Details of the data structure are shown in Figure 5. The SVL-MNIST dataset was split into a training set, a validation set, and a test set. The validation set is merged with the training set after hyperparameter exploration. To obtain a multimodal dataset with completely missing modality combinations, the inventors considered three datasets (I,T), (T,A), and (A,I). The first dataset (I,T) consists of 12,000 paired samples from MNIST and WineReviews. The second dataset (T,A) is constructed similarly and contains 12,000 paired samples from WineReviews and AudioMNIST. Another 12,000 samples were imported from AudioMNIST and combined with the 12,000 samples from MNIST to obtain (A,I). All remaining samples are part of the unimodal datasets I, T, and A. Except for the experiment, there are no datasets with all three modalities (I,T,A). Note that there is no overlap in the samples within each dataset; that is, all datasets have the same relationships as in Figure 3c.For fair comparison, the inventors report final results using only a fully aligned subset of the test set, because the other sets contain different samples and are of different sizes. However, the inventors maintained data partitioning with unaligned individual modalities (right side of Figure 5) to make the dataset more flexible for future experiments.

[0120] The TCGA-OMICS medical dataset contains omics (e.g., genomics, proteomics, and transcriptomics) sequencing values ​​from the Cancer Genome Atlas Program (TCGA) [Weinstein et al., 2013]. The inventors selected three maximum subsets from up to 11,000 patients, including messenger ribonucleic acid (mRNA), microribonucleic acid (miRNA), and reverse-phase protein arrays (RPPA). The inventors aligned the dataset at the patient level, obtaining approximately 7,000 patients across all three modalities. To simulate a configuration with missing modality combinations, the inventors split the dataset into two subsets, (mRNA, miRNA) and (mRNA, RPPA), each containing 3,500 non-overlapping samples. The inventors selected mRNA as the link modality because it is one of the most widely available genetic modalities. TCGA is quite unique in that it is the only publicly available medical dataset with multiple aligned modalities. Therefore, it is useful for testing the methods described herein. Further details of the data structure are shown in Figure 6. The TCGA-OMICS dataset contains 11,069 mRNAs, 10,824 miRNAs, and 7,790 RPPA samples. We aligned the dataset at the patient level, obtaining 7,030 data points across three modalities. Of these, 1,030 were used for testing. The remaining samples were part of the training and validation sets (again, the validation set was merged with the training set after hyperparameter tuning). In particular, the training set consisted of (mRNA, miRNA) and (mRNA, RPPA) datasets, each consisting of 3,000 pairs of samples. As described above, we selected mRNA as the link modality because it is typically the most abundant and common modality available in medical datasets. As described above, since the other sets contain different samples and are of different sizes, we will report the final results using only a perfectly aligned subset of the test set.However, the inventors maintained data partitioning with unaligned individual modalities (right side of Figure 6) to make the dataset more flexible for future experiments.

[0121] As a large-scale, multimodal dataset, the inventors selected MUGEN-GAME [Hayes et al. 2022], which has 375,000 naturally aligned video (V), audio (A), and text (T) samples collected by a reinforcement learning agent in the closed-world platform game CoinRun. Mugen is the main character, walking, jumping, collecting coins, killing monsters, climbing ladders, dying, and generating various sound effects overlaid with background music. All of these adventures are recorded in 3-second video clips and audio tracks. A group of human narrators provides rich text descriptions of each scene. The inventors used video games, with the most important modality in video being the concatenated modality, and examined discontinuous datasets (V,A) and (V,T) in the final experiment. MUGEN-GAME consists of 375,368 perfectly aligned samples. This was divided into training, validation, and test sets of sizes 349, 666, 12,851, and 12,851, respectively. The inventors randomly paired video and audio files, as well as video and text files. This resulted in two discontinuous training datasets of size 174,833.

[0122] Implementation Details: In the first two experiments (SVL-MNIST and TCGA-OMICS data), the inventors used an 8-layer, 8-attention-head transformer decoder with an embedding dimension of 512 and query, key, and value dimensions of 64, respectively. This compact architecture has proven effective in a variety of tasks and is widely used in the literature (see, e.g., Li et al. 2023). The inventors applied a single parameterized embedder to encode tokens and added modality-specific learned position embeddings (as described in Ramesh et al. 2021, Wang and Cho 2019, and Aghajanyan et al. 2023). For optimization, the AdamW algorithm was used with a learning rate of 6e-4, a weight decay coefficient of 0.1, and a gradient clipping of 1. The learning rate undergoes a 10x decay using cosine annealing and linear warm-up during the first few hundred steps. In the third experiment (MUGEN-GAME), the inventors use the exact same training hyperparameters and model configuration as the baseline model described in [Hayes et al. 2022]. Specifically, the transformer has 12 layers, 8 attention heads, and 768 embedding dimensions. The query, key, and value dimensions are each 96.

[0123] Models pre-trained in the language learning paradigm share a similar set of hyperparameters. This aligns with current best practices for state-of-the-art (multimodal) foundational models based on transformers. Specifically, we use an AdamW optimizer with beta = (0.9, 0.95), epsilon = 1e-8, weight decay = 0.1, and gradient clipping = 1.0. The learning rate starts at 1e-7, increases linearly to 6e-4, and gradually decays to 6e-5 according to a cosine schedule. To ensure the model sees batches containing different modalities and modality combinations during training, we accumulated batches across different data loaders. For CLIP, we used beta = (0.9, 0.98), epsilon = 1e-6, and weight decay = 0.2. The maximum learning rate starts at 5e-4 and ends at 5e-5. The batch size, warm-up steps, and total training steps for each experiment can be found in Tables 5 and 6. The above settings apply to the SVL-MNIST and TCGA-OMICS experiments. The MUGEN-GAME experiment follows the exact hyperparameters of Hayes et al. 2022, except that training was performed in 142,000 steps.

[0124] Tokenization Scheme Due to the low dimensionality of the SVL-MNIST dataset, the inventors directly encoded the raw byte stream by binning each continuous value into a dictionary of 256. TCGA-OMICS data can be very high-dimensional and are best handled in a different way. For example, mRNA expression arrays typically contain more than 20,500 entries. Therefore, the inventors first reduced the dimensions tenfold using PCA and then binned the values ​​into a dictionary of 1,000. The final input sizes are 2,048, 64, and 16, respectively, while retaining their original relative lengths. High-dimensional video and audio samples in MUGEN-GAME are tokenized using their pre-trained VQ-VAE encoders, similar to Hayes et al. 2022.

[0125] In the linear probing SVL-MNIST experiment, the backbone is frozen and a linear classifier is trained on it. SGD is used as the optimizer with the Nesterov momentum set to 0.9. The initial learning rate is 0.1, but gradually decreases to 0 during training by cosine annealing. Weight decay is not used. The batch size is 16, and the number of epochs (based on the validation set) is listed in Table 7. For the TCGA experiment, the inventors fit a Cox proportional hazards model with an elastic net penalty to the extracted features. The weights of the regularization terms L1 and L2 are both set to 0.5. Training is stopped when one of the following criteria is met: tol=1e-7 or iter=100000. The underlying optimization algorithm is based on coordinate descent, which continuously minimizes the objective function along one direction at a time. This is particularly effective for problems with many features.

[0126] theoretical analysis If a subset S sampled from a joint probability distribution exists, then modalities X1,...,X N We assume that a dataset containing X1, ..., X is sorted. N;Y). Here, Y represents the label space. This form of alignment, in which only labels are shared between modalities, is called weak alignment [Castrejon et al., 2016]. Conversely, if there is a functional relationship at the level of physical objects, the alignment is considered strong [Castrejon et al., 2016]. The definitions of strong and weak alignment also extend to datasets that do not have all combinations of modalities at once. For example, during data collection, only paired samples (x1,x2) and (x'2,x'3) may be collected, where (x'2,x'3) and x'2 do not belong to the same object / subject. Thus, instead of (X1,X2,X3), there is a dataset in the form of (X1,X2) and (X2,X3). This is different from the case where the dataset still contains some samples in the form (x1,x2,x3) considered in previous studies. To model invisible relationships (X1, X3) using a self-supervised method, techniques such as masked modeling or causal modeling cannot be simply used (because there are no aligned observations for modalities X1 and X3). The neural network needs to learn implicitly.

number

[0127] Empirically, by deriving the probability chain rule and p(x1,x2,x3)=p(x3|x2,x1)p(x2|x1)p(x1) from the probability chain rule (see [Asai and Hajishirzi, 2020, Li and Tou Ng 2022, Yanaka et al., 2021]), both unidirectional and bidirectional methods fail this task, even within a single modality, because it is difficult to model p(x3|x2,x1) due to the missing set (X1,X2,X3). This is where the proposed transitive modeling paradigm comes in. (X1,X2),(X2,X3),...,(X N-1 ,X N Let's assume we have a dataset in the format X k and X l teeth, The data is not aligned in TIFF2026516833000029.tif621. By applying causal modeling to all pairs, the transition X1→X2→…→X N-1 →X N This can be learned. Adding the proposed reciprocal modeling will result in an even stronger bidirectional relationship. Next, we interpret each modality as a node on a high-dimensional graph. This gives us a connected graph (minimum spanning tree) that can be walked from one modality to any other modality. Thus, the actual data point x i Given x i from x j By transitioning to this, pseudo-data point x j Transitive modeling can be used to sample many pairs (x i ,x j When you sample ) a new dataset (X i ,X j This yields a new transition. This is then used again in the generative pre-training process. TIFF2026516833000031.tif615 can be learned. The inventors apply this idea to all i and j. Finally, the conditional distribution between all samples is learned, and a fully connected graph is obtained. This idea also applies to more complex combinations of modalities as long as bridging modalities exist, i.e., as long as a spanning tree is present (see Figure 4d). All missing tuples (x k ,...,x l It is also possible to sample the missing joint probability distribution P(X) by sampling and training with the missing modality combination. k ,...,X l This makes it possible to approximate ).

[0128] Ideally, there would be no generalization error after pre-training. However, in practice, it is not possible to accurately recover the ground truth of missing modalities with perfect precision, even if available. For datasets (X1,X2) and (X2,X3), the trained generative model f produces the following errors:

number

[0129] Here, all e i,j is the error term of the vector value. Without loss of generality, always e i,j '=-1·e i,j Since we can write to it, we do not need to keep tracking signs of error. Assuming that f is differentiable, and using a multivariate Taylor expansion, we obtain the following:

number

[0130] Here, J f is the Jacobian of f. By iteratively applying Taylor's law, a more general error propagation formula is obtained for longer chains. Intuitively, the error from the first transition is amplified more and more. However, since we reconstruct x1 using the predicted x3, the error term e is actually 1,3 :=J f (x3|f(x2|x1))J f (x2|x1)e 1,2 +e 2,3 This can be restricted. Recall that we have a Ground Truth x1 such that the following is true.

number

[0131] During backpropagation Given TIFF2026516833000035.tif625, the reconstruction error of x1 is minimized, so the error term e 1,2 and e 2,3They must be bounded. Also, they have an error term e 1,3 Since it constitutes f and J f As long as that is the case, it must also be bounded.

[0132] Experimental evaluation results SVL-MNIST In the first experiment, the inventors trained one model for each modality (A, I, and T) using causally masked modeling (CM3). Next, the inventors trained four additional models using commutative causally masked modeling (C2M3) for bimodal datasets (A,I), (A,T), and (I,T), as well as a trimodal dataset (A,I,T). These seven models serve as unimodal, multimodal, and "upper" baselines, respectively. Next, using C2M3 with T as a link modality, the inventors trained models simultaneously for both (I,T) and (A,T), and repeated this process for all modality combinations. Next, the inventors initialized LoReTTa with C2M3 model weights to explicitly merge bimodal datasets via the link modality. This pre-training technique speeds up training compared to full LoReTTa training from scratch. After pre-training, all models were frozen and evaluated in a low-data region using only 1,000 labeled samples (100 per class). Three runs were performed, each with a newly randomized subset, and classification accuracy was reported along with the perplexity of the test set. Note that the models were trained only on the modalities they had seen during pre-training. This also means that during probing, the linear classifier would see some samples of modality pairs it had not seen. However, the pre-trained backbone itself did not see them. In this way, we can analyze how the model combines mixtures of input modalities (combinations of modalities) that it had not seen before.

[0133] When pre-trained on pairs (T,I) and (T,A) only, LoReTTa achieves a remarkably low perplexity of 5.04 for the invisible pair (I,A), as shown in Table 1, significantly outperforming the baseline non-temporal model trained on C2M3, which has a perplexity of 104.27. This trend extends to other combinations. For example, a model trained on (A,I) and (A,T) and evaluated on (I,T) has a perplexity of 3.13 (LoReTTa) compared to 9.70 (C2M3). Similarly, training on (I,T) and (I,A) and evaluating on (T,A) yields a perplexity of 11.21 (LoReTTa) versus 30.88 (C2M3). Importantly, LoReTTa consistently achieves similar complexity for both invisible and visible modality combinations, demonstrating its ability to effectively learn missing distributions. A similar trend is observed in downstream tasks (Table 2). In almost all cases, LoReTTa consistently outperforms C2M3 for unimodal datasets A, I, and T, as well as bimodal datasets (A,I), (A,T), or (I,T). Simply put, if either modality pair is missing, it is recommended to train the model with LoReTTa. This is because it uniquely integrates these modalities and improves accuracy compared to any single-modality model. Indeed, providing a small number of samples with all three modalities during linear probing further increases the classification score of models pre-trained with LoReTTa. Interestingly, the "upper" baseline trained with all aligned modalities often lags behind LoReTTa despite good complexity scores. This strongly suggests some kind of negative transfer or modality competition that LoReTTa can overcome. [Table 1] [Table 2]

[0134] TCGA-OMICSThe transformants were pre-trained with C2M3, and LoReTTa was initialized with the weights of the C2M3 model, similar to the SVL-MNIST experiment. For GPT, the inventors simply used the C2M3 algorithm but disabled causal masking and commutative switching. BERT uses the same transformer architecture; however, the inventors simply changed the one-way attention mask to a two-way attention mask. To directly demonstrate the advantages of transitive modeling, the inventors extended both GPT and BERT using the transitive modeling techniques described herein. The resulting models are named T-GPT and T-BERT. These models use transitive modeling but not commutative modeling. The CLIP-based model consists of three encoders (each using the same transformer architecture) and contrasting losses for feature alignment [Khosla et al. 2020]. After pre-training, for each model, the inventors extracted embeddings and fitted a Cox proportional hazards model using an elastic net penalty for each modality combination, utilizing all available labels. In six of the seven test sets, the LoReTTA pre-trained transformer achieved the best c-index (Table 3). Most importantly, this includes invisible modality pairs (miRNA, RPPA) and triplets (mRNA, miRNA, RPPA). The only frameworks equivalent to LoReTTA are T-BERT and CLIP on the invisible dataset (miRNA, RPPA). On the unconfirmed trimodal dataset, T-GPT and CLIP yield good results, but fall far short of LoReTTA. Overall, transitive modeling improves the performance of almost all models. Notably, LoReTTA and CLIP are the only strategies that consistently allow positive or neutral transfers, which are not always the case with other frameworks. The main drawback of contrastive learning is the need for a separate encoder for each additional modality. Furthermore, CLIP-based methods are only superior for large batch sizes. Here, the inventors compared models with the same computational constraints.To investigate how CLIP can compare to LoReTTa with additional computational power, the inventors spent limited resources on additional computational power, enabling them to scale up the CLIP experiment by increasing the batch size fivefold and the number of training steps threefold. This model, called Large CLIP (L-CLIP), consistently improves the c-index of CLIP but requires three times more VRAM than LoReTTa. As seen in Table 3, when performing trimodal prediction or two of the three types of bimodal prediction (the third type is associated with the same accuracy only with T-BERT and CLIP), none of the models achieve the same performance as LoReTTa. When using invisible pairs as part of the bimodal (M,R) or trimodal (M,I,R) input, all models fall far short of LoReTTa. This indicates that the LoReTTa method yields higher performance than the comparison models when using the same amount of computational resources, and that increasing the computational resource allocation to the comparison models improved performance, but not to the same level as LoReTTa. In other words, the data demonstrates that the method described herein enables multimodal prediction with higher accuracy than comparable methods with the same computational efficiency, even when perfectly aligned training data is unavailable. In other words, the proposed method provides a more computationally efficient way to learn from multimodal data in the presence of one or more missing tuples.

[0135] In contrast to the SVL-MNIST experiment, the 3-mode "upper bound" model is less susceptible to negative transfer or modality competition and achieves the best results for most modality combinations. However, as explained above, the availability of such perfectly aligned data is extremely rare in the field of medical data. Therefore, models like the upper bound model are practically unattainable in most cases. [Table 3]

[0136] MUGEN-GAME In recent years, interest in cross-modal power generation has been growing. While much research has focused on text-to-video or video-to-text translation, other cross-modal tasks such as video-to-audio, audio-to-video, text-to-audio, or audio-to-text have not been sufficiently investigated. In this large-scale experiment, the inventors addressed the latter challenge. The inventors used video as the link modality and examined separated datasets (video, audio) and (video, text). When the training sets were equally divided, approximately 187,500 pairs were obtained in each set. The inventors trained versions of (MM)GPT and LoReTTa using the same optimized hyperparameters and model architecture as the state-of-the-art top baseline MMGPT [Hayes et al. 2022]. Surprisingly, BERT was unable to generate long-range coherent samples, as seen in the TCGA-OMICS experiment described above (see results for mRNA). Therefore, the inventors did not use BERT in this experiment. The inventors also did not train the CLIP model. This is because it is necessary to train a text diffusion model (or equivalent generator) on a pre-trained encoder that is neither end-to-end nor simple. For GPT, the inventors trained the model to generate video from audio and text from video. In this way, the inventors implicitly taught the model how to caption audio tracks. For LoReTTa, the inventors directly model the transition between audio and text by transitive modeling. Notably, GPT trained on two bimodal datasets outperforms the audio-versus-text baseline with respect to ROUGE, showing a strong positive transfer (Table 4). LoReTTa further improves upon GPT, even matching the video-versus-text baseline with respect to METEOR and ROUGE. BLEU4 values ​​are low for both models. Since BLEU involves precise n-gram matching, these results are expected to account for strict data constraints. [Table 4]

[0137] conclusion Using LoReTTa, the inventors have introduced an innovative self-supervised learning framework for multimodal integration that learns the missing common condition distribution given link modalities. The inventors have demonstrated this theoretically and practically. In particular, the novel method combines commutativity and transitivity rules to model multimodal datasets (B,C) and (A,B,C) given two non-overlapping datasets (A,B) and (B,C). The inventors have shown that the extracted features of a transformer pre-trained with LoReTTa are highly expressive for all modality combinations, including the invisible. Although the inventors have only evaluated the novel algorithm on a dataset with three modalities, it is far more general because it can be used with any number of modalities and any mixture of modalities. As long as there is an aligned chain of modalities in the dataset, LoReTTa can learn the missing distribution P(X j ,...,X k Transition (X) for modeling i →...→X j ),...,(X i →...→X k ) can be learned (as shown in Figure 4d). To the best of our knowledge, this scenario has never been considered in the literature without training an ensemble of contrasting models based on the CLIP concept, most notably Wav2CLIP [Wu et al., 2022] and its extension ImageBind [Girdhar et al., 2023], which is far more computationally intensive as described above.

[0138] These examples demonstrate that LoReTTa can serve as a foundation for training very powerful multimodal generative and discriminative models that are useful in many research fields. For example, LoReTTa is designed to improve the quality of neural networks that serve society by training more accurate multimodal systems that can assist, for example, physicians and other practitioners in their daily work. LoReTTa is best used in settings where several modality combinations are missing and at least one connected modality is present. The method can still be used in datasets where some samples have all modality combinations. Although the inventors have not analyzed this scenario in this embodiment, LoReTTa is likely to still benefit by augmenting data points that are missing the remaining modalities. [Table 5] [Table 6] [Table 7]

[0139] References Jimmy Lei Ba,Jamie Ryan Kiros,and Geoffrey E Hinton.Layer normalization.arXiv preprint arXiv:1607.06450,2016.4 Peng Bo.Blinkdl / rwkv-lm:0.01,Aug 2021.URL https: / / doi.org / 10.5281 / zenodo.5196577 Aakanksha Chowdhery,Sharan Narang,Jacob Devlin,Maarten Bosma,Gaurav Mishra,Adam Roberts,Paul Barham,Hyung Won Chung,Charles Sutton,Sebastian Gehrmann,et al.Palm:Scaling language modeling with pathways.arXiv preprint arXiv:2204.02311,2022 TIFF2026516833000043.tif68164Alec Radford,Jeff Wu,Rewon Child,David Luan,Dario Amodei,and Ilya Sutskever.Language models are unsupervised multitask learners,2019. TIFF2026516833000044.tif40164Biao Zhang and Rico Sennrich.Root mean square layer normalization.In Advances in Neural Information Processing Systems(NeurIPS),volume 32,2019 Armen Aghajanyan,Bernie Huang,Candace Ross,Vladimir Karpukhin,Hu Xu,Naman Goyal,Dmytro Okhonko,Mandar Joshi,Gargi Ghosh,Mike Lewis,et al.Cm3:A causal masked multimodal model of the internet.arXiv preprint arXiv:2201.07520,2022 Armen Aghajanyan,Lili Yu,Alexis Conneau,Wei-Ning Hsu,Karen Hambardzumyan,Susan Zhang,Stephen Roller,Naman Goyal,Omer Levy,and Luke Zettlemoyer.Scaling laws for generative mixed-modal language models.arXiv preprint arXiv:2301.03728,2023 TIFF2026516833000045.tif45164Jean-Baptiste Alayrac,Jeff Donahue,Pauline Luc,Antoine Miech,Iain Barr,Yana Hasson,Karel Lenc,Arthur Mensch,Katherine Millican,Malcolm Reynolds,et al.Flamingo:a visual language model for few-shot learning.In Advances in Neural Information Processing Systems(NeurIPS),volume 35,pages 23716-23736,2022 TIFF2026516833000046.tif33157Mark Chen,Alec Radford,Rewon Child,Jeffrey Wu,Heewoo Jun,David Luan,and Ilya Sutskever.Generative pretraining from pixels.In International Conference on Machine Learning(ICML),pages 1691-1703.PMLR,2020 Lluis Castrejon,Yusuf Aytar,Carl Vondrick,Hamed Pirsiavash,and Antonio Torralba.Learning aligned cross-modal representations from weakly aligned data.In IEEE Conference on Computer Vision and Pattern Recognition(CVPR),pages 2940-2949,2016 Jordan Hoffmann,Sebastian Borgeaud,Arthur Mensch,Elena Buchatskaya,Trevor Cai,Eliza Rutherford,Diego de Las Casas,Lisa Anne Hendricks,Johannes Welbl,Aidan Clark,et al.An empirical analysis of compute-optimal large language model training.In Advances in Neural Information Processing Systems(NeurIPS),volume 35,pages 30016-30030,2022 Jacob Devlin,Ming-Wei Chang,Kenton Lee,and Kristina Toutanova.BERT:Pre-training of deep bidirectional transformers for language understanding.In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,Volume 1(Long and Short Papers),pages 4171-4186.Association for Computational Linguistics,2019 Thomas Hayes,Songyang Zhang,Xi Yin,Guan Pang,Sasha Sheng,Harry Yang,Songwei Ge,Qiyuan Hu,and Devi Parikh.Mugen:A playground for video-audio-text multimodal understanding and generation.In 17th European Conference on Computer Vision(ECCV),pages 431-449.Springer,2022. Alec Radford,Karthik Narasimhan,Tim Salimans,Ilya Sutskever,et al.Improving language understanding by generative pre-training,2018 Alec Radford,Jong Wook Kim,Chris Hallacy,Aditya Ramesh,Gabriel Goh,Sandhini Agarwal,Girish Sastry,Amanda Askell,Pamela Mishkin,Jack Clark,et al.Learning transferable visual models from natural language supervision.In International Conference on Machine Learning(ICML),pages 8748-8763.PMLR,2021b Rohit Girdhar,Alaaeldin El-Nouby,Zhuang Liu,Mannat Singh,Kalyan Vasudev Alwala,Armand Joulin,and Ishan Misra.Imagebind:One embedding space to bind them all.arXiv preprint arXiv:2305.05665,2023 John Jumper,Richard Evans,Alexander Pritzel,Tim Green,Michael Figurnov,Olaf Ronneberger,Kathryn Tunyasuvunakool,Russ Bates,Augustin Zidek,Anna Potapenko,et al.Highly accurate protein structure prediction with alphafold.Nature,596(7873):583-589,2021. Kudo and John Richardson.SentencePiece:A simple and language independent subword tokenizer and detokenizer for neural text processing.In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing:System Demonstrations(EMNLP),pages 66-71.Association for Computational Linguistics,2018. Lyes Khacef,Laurent Rodriguez,and Benoit Miramond.Written and spoken digits database for multimodal learning.https: / / zenodo.org / record / 3515935,2019 Prannay Khosla,Piotr Teterwak,Chen Wang,Aaron Sarna,Yonglong Tian,Phillip Isola,Aaron Maschinot,Ce Liu,and Dilip Krishnan.Supervised contrastive learning.In 33th Advances in Neural Information Processing Systems(NeurIPS),pages 18661-18673.Curran Associates,Inc.,2020 Yann LeCun,Corinna Cortes,and Christopher J.C.Burges.Mnist handwritten digit database.http: / / yann.lecun.com / exdb / mnist / ,2010 Ruixi Lin and Hwee Tou Ng.Does bert know that the is-a relation is transitive? In 60th Annual Meeting of the Association for Computational Linguistics(Volume 2:Short Papers),pages 94-99.Association for Computational Linguistics,2022 Jessy Lin,Yuqing Du,Olivia Watkins,Danijar Hafner,Pieter Abbeel,Dan Klein,and Anca Dragan.Learning to model the world with language.arXiv preprint arXiv:2308.01399,2023 TIFF2026516833000047.tif39164Andrew Jaegle,Felix Gimeno,Andy Brock,Oriol Vinyals,Andrew Zisserman,and Joao Carreira.Perceiver:General perception with iterative attention.In International Conference on Machine Learning(ICML),pages 4651-4664.PMLR,2021 Wenhui Wang,Hangbo Bao,Li Dong,Johan Bjorck,Zhiliang Peng,Qiang Liu,Kriti Aggarwal,Owais Khan Mohammed,Saksham Singhal,Subhojit Som,et al.Image as a foreign language:Beit pretraining for all vision and vision-language tasks.arXiv preprint arXiv:2208.10442,2022 Jiahui Yu,Yuanzhong Xu,Jing Yu Koh,Thang Luong,Gunjan Baid,Zirui Wang,Vijay Vasudevan,Alexander Ku,Yinfei Yang,Burcu Karagol Ayan,et al.Scaling autoregressive models for content-rich text-to-image generation.arXiv preprint arXiv:2206.10789,2022. TIFF2026516833000048.tif96164Mike Schuster and Kaisuke Nakajima.Japanese and korean voice search.In 2012 IEEE international conference on acoustics,speech and signal processing(ICASSP),pages 5149-5152.IEEE,2012. Rico Sennrich,Barry Haddow,and Alexandra” Birch.Neural machine translation of rare words with subword units.In 54th Annual Meeting of the Association for Computational Linguistics(ACL),pages 1715-1725.Association for Computational Linguistics,2016. Aditya Ramesh,Mikhail Pavlov,Gabriel Goh,Scott Gray,Chelsea Voss,Alec Radford,Mark Chen,and Ilya Sutskever.Zero-shot text-to-image generation.In International Conference on Machine Learning(ICML),pages 8821-8831.PMLR,2021. Ali Razavi,Aaron Van den Oord,and Oriol Vinyals.Generating diverse high-fidelity images with vq-vae-2.In Advances in Neural Information Processing Systems(NeurIPS),volume 32,2019 Aaron Van Den Oord,Oriol Vinyals,et al.Neural discrete representation learning.In Advances in Neural Information Processing Systems(NeurIPS),volume 30,2017. Danny Driess,Fei Xia,Mehdi SM Sajjadi,Corey Lynch,Aakanksha Chowdhery,Brian Ichter,Ayzaan Wahid,Jonathan Tompson,Quan Vuong,Tianhe Yu,et al.Palm-e:An embodied multimodal language model.arXiv preprint arXiv:2303.03378,2023. Thomas Wang,Adam Roberts,Daniel Hesslow,Teven Le Scao,Hyung Won Chung,Iz Beltagy,Julien Launay,and Colin Raffel.What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning(ICML),pages 22964-22984.PMLR,2022 Jun-Yan Zhu,Taesung Park,Phillip Isola,and Alexei A Efros.Unpaired image-to-image translation using cycle-consistent adversarial networks.In IEEE International Conference on Computer Vision(ICCV),pages 2223-2232,2017 TIFF2026516833000049.tif39164Hitomi Yanaka,Koji Mineshima,and Kentaro Inui.Exploring transitivity in neural nli models through veridicality.In 16th Conference of the European Chapter of the Association for Computational Linguistics,pages 920-934.Association for Computational Linguistics,2021 Zack Thoutt.Wine reviews:130k wine reviews with variety,location,winery,price,and description.https: / / www.kaggle.com / datasets / zynicide / wine-reviews,2018. John N Weinstein,Eric A Collisson,Gordon B Mills,Kenna R Shaw,Brad A Ozenberger,Kyle Ellrott,Ilya Shmulevich,Chris Sander,and Joshua M Stuart.The cancer genome atlas pan-cancer analysis project.Nature Genetics,45(10):1113-1120,09 2013. Alex Wang and Kyunghyun Cho.Bert has a mouth,and it must speak:Bert as a markov random field language model.arXiv preprint arXiv:1902.04094,2019. Poli,M.et al.2023b.StripedHyena:Moving Beyond Transformers with Hybrid Signal Processing Models.github.com / togethercomputer / stripedhyena.doi:10.57967 / hf / 1595.December 2023. Lieber et al.2024.Jamba:A Hybrid Transformer-Mamba Language Model.arXiv:2403.19887.Thu,28 Mar 2024 De et al.2024.Griffin:Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models.arXiv:2402.19427v1 [cs.LG] 29 Feb 2024 Tschannen et al.2023.GIVT:Generative Infinite-Vocabulary Transformers.arXiv:2312.02116.Mon,4 Dec 2023. BehnamGhader et al.2024.LLM2Vec:Large Language Models Are Secretly Powerful Text Encoders.arXiv:2404.05961v1 [cs.CL] 09 Apr 2024 Yang et al.2020.XLNet:Generalized Autoregressive Pretraining for Language Understanding.arXiv:1906.08237v2 [cs.CL] Thu,2 Jan 2020 Dong et al.2019.Unified Language Model Pre-training for Natural Language Understanding and Generation.arXiv:1905.03197.Tue,15 Oct 2019 Raffel et al.2023.Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer arXiv:1910.10683v4 [cs.LG] Tue,19 Sep 2023 Tay et al.2023.UL2:Unifying Language Learning Paradigms.arXiv:2205.05131v3 [cs.CL] Tue,28 Feb 2023 Mikel Artetxe,Jingfei Du,Naman Goyal,Luke Zettlemoyer,and Ves Stoyanov.On the role of bidirectionality in language model pre-training.arXiv preprint arXiv:2205.11726,2022. Akari Asai and Hannaneh Hajishirzi.Logic-guided data augmentation and regularization for consistent question answering.In 58th Annual Meeting of the Association for Computational Linguistics,pages 5642-5650.Association for Computational Linguistics,2020. Mohammad Bavarian,Heewoo Jun,Nikolas Tezak,John Schulman,Christine McLeavey,Jerry Tworek,and Mark Chen.Efficient training of language models to fill in the middle,2022. Daniel Fried,Armen Aghajanyan,Jessy Lin,Sida Wang,Eric Wallace,Freda Shi,Ruiqi Zhong,Scott Yih,Luke Zettlemoyer,and Mike Lewis.Incoder:A generative model for code infilling and synthesis.In 11th International Conference on Learning Representations(ICLR),2023. Ho-Hsiang Wu,Prem Seetharaman,Kundan Kumar,and Juan Pablo Bello.Wav2clip:Learning robust audio representations from clip.In 2022-2022 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),pages 4563-4567.IEEE,2022 Lili Yu,…,and Armen Aghajanyan.Scaling autoregressive multi-modal models:Pretraining and instruction tuning.Technical report,Meta AI,2023.

[0140] All references referred to herein are incorporated herein by reference for all purposes to the same extent as each individual publication or patent or patent application is specifically and individually indicated as being incorporated by reference in whole.

Claims

1. A computer implementation method, Obtain an input dataset containing data from one or more modalities selected from a set of at least three modalities, This includes generating an output dataset by processing the input dataset using a machine learning model that includes transitive and commutative machine learning models, A computer implementation method comprising a transitive and commutative machine learning model trained to transform input datasets corresponding to any of the modalities and each of them from the set of at least three modalities to output datasets corresponding to any of the other modalities and each of them from the set of at least three modalities, wherein the machine learning model is trained to transform data between the two modalities by using training data including data pairs corresponding to a first modality and another modality of two modalities, and other data pairs corresponding to the other modality and a second modality of the two modalities, wherein the machine learning model is trained to perform corresponding inverse modality transformations with respect to any forward modality transformations that the model is trained to perform.

2. The aforementioned machine learning model is a classification model, a regression model, or a generative model. The classification model includes a transitive and commutative machine learning model configured to take the input dataset as input, and a trained classification model trained to take features extracted from the transitive and commutative machine learning model as input and generate classifications of the input dataset as output. The generative model is trained to take the input dataset as input and to generate a generated output dataset corresponding to the input dataset, wherein the output dataset corresponds to one of the set of at least three modalities that are not present in the input dataset, or The method according to claim 1, wherein the regression model includes a transitive and commutative machine learning model configured to take the input dataset as input, and a trained regression model that takes features extracted from the transitive and commutative machine learning model as input and is trained to generate regression values ​​with respect to the input dataset as output, wherein optionally the regression model is a Cox proportional hazards model and the regression values ​​are survival predictions.

3. The method according to claim 1 or 2, wherein the method comprises determining a specific input modality of the input dataset and receiving a representation of a specific target output modality, and generating an output dataset is performed by processing the input dataset and the identification representation of the specific target output modality using the transitive and commutative machine learning model.

4. The transitive and commutative machine learning models are trained using causal generative modeling, masked modeling, and / or variations thereof, optionally, the variation being causal masked modeling. The aforementioned causal generative modeling, masked modeling, and / or variations thereof use ground truth data, which includes concatenated input data relating to the first modality and the second modality of each of the multiple data pairs in the training data, The method according to any one of claims 1 to 3, wherein the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair, and / or the ground truth data includes, with respect to each of the plurality of data pairs in the training data, a first concatenated input data and a second concatenated input data, and the modality of the data pair used as the first modality is different between the first concatenated input data and the second concatenated input data.

5. The aforementioned transitive and commutative machine learning models are trained using causally masked modeling, and the causally masked modeling is Identifying a subset of tokens in a training dataset, wherein the tokens in the subset of tokens are located at multiple positions within the training dataset. Adding a subset of the tokens to the training dataset, The method according to any one of claims 1 to 4, comprising adding masks to the plurality of locations in the training dataset.

6. The method of claim 5, further comprising using an identification representation of the masked plurality of locations during the training of the causally masked modeling.

7. The method according to any one of claims 1 to 6, wherein the transitive and commutative machine learning model is a sequence model, and / or the transitive and commutative machine learning model is a deep learning model, and / or the transitive and commutative machine learning model includes a sequence model selected from a recurrent neural network, a model combining a recurrent mechanism and an attention mechanism, a convolutional neural network or a convolution-based model, optionally a model combining element-wise multiplication (gating) and long convolution, or a model combining a convolution mechanism and an attention mechanism, a state-space model, or a model combining a state-space model and an attention mechanism, or a transformation-based model.

8. The method according to any one of claims 1 to 7, wherein the transitive and commutative machine learning models include a transformer model that uses self-attention.

9. Obtaining the aforementioned input dataset means Receiving the initial dataset, The method according to any one of claims 1 to 8, comprising generating the input dataset by tokenizing the initial dataset.

10. Generating the aforementioned input dataset is, The method according to claim 9, comprising one or more of the following: (i) tokenizing the initial dataset and projecting the tokenized initial dataset into a predetermined feature space using modality-parameterized embedding functions; (ii) preparing and / or adding modality-specific tokens to the tokenized and arbitrarily projected data corresponding to each single modality in the input dataset; and (iii) including learned location embeddings.

11. The method according to any one of claims 1 to 10, wherein the transitive and commutative machine learning model is trained using a first step of using commutative modeling and self-supervised learning objectives, and a second step of initializing the weights of the machine learning model to the weights learned in the first step, and is further trained using transitive modeling.

12. The first step includes taking partially masked concatenated input data for a first modality and a second modality of a data pair in a training dataset, each containing ground truth data for a plurality of data pairs, each containing corresponding data for a modality pair, and training the model to produce predictions of the masked input data as output, wherein the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair, and / or the ground truth data contains a first concatenated input data and a second concatenated input data for each of the plurality of data pairs in the training dataset, wherein the modality of the data pair used as the first modality differs between the first concatenated input data and the second concatenated input data, and / or The second step is to train the model, wherein with respect to each of a plurality of data pairs of the training dataset, each of which includes first data for a first modality and corresponding second data for a second modality among the set of at least three modalities, Using the machine learning model that takes the second data as input, predict the third data relating to the third modality of the set of at least three modalities corresponding to the second data, Optionally, iteratively predict subsequent data for further modalities of the set of at least three modalities corresponding to the predicted third data or predicted subsequent data using the transitive and commutative machine learning model that takes the predicted third data or predicted subsequent data as input, Using the transitive and commutative machine learning model that takes the third data or the most recent iteratively predicted subsequent data as input, predict the final data for the first modality corresponding to the predicted third data or the most recent iteratively predicted subsequent data, This includes training the model to calculate a loss by comparing the first data with the predicted final data, The method according to claim 11, wherein the training data includes (i) pairs of data relating to the second modality and corresponding data relating to the third modality, and (ii) pairs of data relating to the third modality and corresponding data relating to the first modality, or pairs of data relating to the latest further modality and corresponding data relating to the first modality, and each pair of data relating to each of the two modalities used in the step of iteratively predicting subsequent data relating to the further modality.

13. The aforementioned transitive and commutative machine learning models are: Accessing a training dataset containing the first data for a specific input modality and the corresponding second data for another modality, The process involves generating a first predicted output dataset by processing the first data and the identified representation of the other modality using the transitive and commutative machine learning model, wherein the first predicted output dataset corresponds to the other modality. A second predicted output dataset is generated by processing the first predicted output data and an identified representation of a specific target output modality using the transitive and commutative machine learning model, wherein the second predicted output dataset corresponds to the specific target output modality. A third predicted output dataset is generated by processing the second predicted output data and the identified representation of the specific input modality using the transitive and commutative machine learning model, wherein the third predicted output dataset corresponds to the specific input modality. The method according to any one of claims 1 to 12, which is trained using a method that includes calculating a loss by comparing at least a portion of the third predicted output dataset with the first data.

14. The aforementioned transitive and commutative machine learning models are trained using training data that lacks data relating a first data of a particular input modality to a second data of a particular output modality, and / or The method according to any one of claims 1 to 13, wherein the transitive and commutative machine learning model is trained using training data which includes data relating data of modality pairs of the set of at least three modalities, and for each pair of first and second modalities of the set of at least three modalities, the training data includes data relating data of modality pairs which together form a path between the first and second modalities, and optionally, for each pair of first and second modalities of the set of at least three modalities, the training data includes data relating data of the first modality to a third modality and data relating data of the third modality to a second modality.

15. The method according to any one of claims 1 to 14, wherein the transitive and commutative machine learning model is trained to learn a first conditional distribution (P(C|A), P(A|C)) for a pair of modalities (A,C), and with respect to the pair of modalities (A,C), the training data does not include enough pairs of training data to relate the data of the pair of modalities to parameterize the conditional distribution, and the transitive and commutative machine learning model is trained to learn the first conditional distribution (P(C|A), P(A|C)) by predicting data for one or more additional modalities (B) that form a link between the pair of modalities using a further learned conditional distribution (P(C|B)) for the pair of modalities (C,B) that form the link, and with respect to the pair of modalities (C,B) that form the link, the training data includes enough pairs of training data to relate the data of the pair of modalities to parameterize the further conditional distribution.

16. The transitive and commutative machine learning model includes, with respect to each pair of training data, a pair of data including a first data for a first modality and a corresponding second data for a second modality among the set of at least three modalities, Using the transitive and commutative machine learning model that takes the second data as input, predict the third data relating to the third modality of the set of at least three modalities corresponding to the second data, Optionally, iteratively predict subsequent data for further modalities of the set of at least three modalities corresponding to the predicted third data or predicted subsequent data using the transitive and commutative machine learning model that takes the predicted third data or predicted subsequent data as input, Using the transitive and commutative machine learning model that takes the third data or the most recent iteratively predicted subsequent data as input, predict the final data for the first modality corresponding to the predicted third data or the most recent iteratively predicted subsequent data, The model is trained using a method that includes calculating a loss by comparing the first data with the predicted final data, The method according to any one of claims 1 to 15, wherein the training data includes (i) pairs of data relating to the second modality and corresponding data relating to the third modality, and (ii) pairs of data relating to the third modality and corresponding data relating to the first modality, or pairs of data relating to the latest further modality and corresponding data relating to the first modality, and each pair of data relating to each of the two modalities used in the step of iteratively predicting subsequent data relating to the further modality.

17. The method according to any one of claims 1 to 16, wherein the transitive and commutative machine learning model is trained using training data comprising tuples (e.g., pairs) comprising corresponding data for multiple modalities of the set of at least three modalities, the tuples comprising data for a particular pair of modalities of the set of at least three modalities are present in the training data in less than 50%, less than 40%, less than 30%, less than 20%, less than 10%, less than 5%, or less than 1% of the number of tuples comprising data for all other pairs of modalities of the set of at least three modalities, and optionally, the tuples do not comprise tuples comprising data for a particular pair of modalities of the set of at least three modalities.

18. The method according to any one of claims 1 to 17, further comprising training the machine learning model.

19. The method according to any one of claims 1 to 18, wherein the one or more modalities are selected from medical or biological sample data associated with a subject, and optionally, the one or more modalities are selected from medical image data, electronic medical record data, gene sequence data, and digital pathology image data, and optionally, the medical image data is selected from MRI, CT, or PET imaging data.

20. The method according to any one of claims 1 to 19, wherein the machine learning model is configured to take medical or biological sample data associated with a subject as input and to generate an output of a display of treatment recommended for the subject, wherein optionally the display is a classification, and / or the medical or biological sample data includes one or more of radiographic images, gene sequences, and microscope slides.

21. The method according to any one of claims 1 to 18, wherein the machine learning model is configured to take as input data collected by a vehicle, one or more of a plurality of types of sensors optionally associated with the vehicle, and to produce as output predicted sensor data relating to another type of sensor data associated with the vehicle, and optionally the machine learning model is trained using synchronous data from pairs of the plurality of types of sensors.

22. A computer implementation method, Analyzing input medical or biological sample data associated with a subject using the method according to any one of claims 1 to 20, A computer implementation method comprising providing a diagnosis and / or prognosis and / or treatment recommendation regarding a subject using the output of the machine learning model, wherein the output of the machine learning model indicates the diagnosis, prognosis, or treatment recommendation.

23. A computer implementation method for training a multimodal machine learning model, Access to a training dataset containing data for each of at least three sets of modalities, A computer implementation method comprising: training a machine learning model to transform an input dataset corresponding to any of the modalities and each of them from the set of at least three modalities to an output dataset corresponding to any of the other modalities and each of them from the set of at least three modalities, wherein the machine learning model is trained to transform data between the two modalities by using training data which includes data pairs corresponding to a first modality and another modality of two modalities, and other data pairs corresponding to the other modality and a second modality of two modalities, and the machine learning model is trained to perform corresponding inverse modality transformations with respect to any forward modality transformations that the model is trained to perform.

24. Training the aforementioned machine learning model involves using causal generative modeling, masked modeling, and / or causal masked modeling, The causal generative modeling, masked modeling, and / or causal masked modeling use ground truth data from the training dataset, which includes concatenated input data relating to the first modality and the second modality of each of the multiple data pairs in the training dataset, The method according to claim 23, wherein the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair, and / or the ground truth data includes, with respect to each of the plurality of data pairs in the training data, a first concatenated input data and a second concatenated input data, and the modality of the data pair used as the first modality is different between the first concatenated input data and the second concatenated input data.

25. The method according to claim 24, wherein the training uses ground truth data from the training dataset, which includes concatenated input data relating to a first modality and a second modality of a data pair for each of a plurality of data pairs in the training data, and the modality of the data pair used as the first modality is randomly sampled from the two modalities of the data pair with a predetermined probability, optionally 50%.

26. The method according to any one of claims 23 to 25, wherein training the machine learning model involves learning a first conditional distribution (P(C|A), P(A|C)) for a modality pair (A,C), wherein with respect to the modality pair (A,C), the training data does not include sufficient training data pairs to relate the data of the modality pair for parameterizing the conditional distribution, and the method involves learning the first conditional distribution (P(C|A), P(A|C)) for a modality pair (A,C) by using the machine learning model to predict data for one or more additional modalities (B) that form a link between the modality pair, wherein the prediction uses a further learned conditional distribution (P(C|B)) for the modality pair (C,B) that form the link, and with respect to the modality pair (C,B) that form the link, the training data includes sufficient training data pairs to relate the data of the modality pair for parameterizing the further conditional distribution.

27. Training the machine learning model involves, with respect to each pair of training data, which includes a pair of data containing first data for a first modality and corresponding second data for a second modality of the set of at least three modalities, To predict a third data relating to a third modality of the set of at least three modalities corresponding to the second data using a transitive and commutative machine learning model that takes the second data as input, Optionally, iteratively predict subsequent data for further modalities of the set of at least three modalities corresponding to the predicted third data or predicted subsequent data using the transitive and commutative machine learning model that takes the predicted third data or predicted subsequent data as input, Using the transitive and commutative machine learning model that takes the third data or the most recent iteratively predicted subsequent data as input, predict the final data for the first modality corresponding to the predicted third data or the most recent iteratively predicted subsequent data, This includes calculating a loss by comparing the first data with the predicted final data, The method according to any one of claims 23 to 26, wherein the training data includes (i) pairs of data relating to the second modality and corresponding data relating to the third modality, and (ii) pairs of data relating to the third modality and corresponding data relating to the first modality, or pairs of data relating to the latest further modality and corresponding data relating to the first modality, and each pair of data relating to each of the two modalities used in the step of iteratively predicting subsequent data relating to the further modality.

28. Training the aforementioned machine learning model is Identify a subset of tokens in the training dataset, and determine that the tokens in the subset of tokens are located at multiple positions within the training dataset. The training dataset is added using a subset of the aforementioned tokens. The method according to any one of claims 23 to 27, comprising using causally masked modeling to add masks to the plurality of locations in the training dataset.

29. Training the aforementioned machine learning model is Accessing a training dataset that includes first data for a specific input modality of the at least three modalities and corresponding second data for another modality of the at least three modalities, A first predicted output dataset is generated by processing the first data and the identified representation of the other modality using a transitive and commutative machine learning model, wherein the first predicted output dataset corresponds to the other modality. A second predicted output dataset is generated by processing the first predicted output data and an identified representation of a specific target output modality using the transitive and commutative machine learning model, wherein the second predicted output dataset corresponds to the specific target output modality. A third predicted output dataset is generated by processing the second predicted output data and the identified representation of the specific input modality using the transitive and commutative machine learning model, wherein the third predicted output dataset corresponds to the specific input modality. The method according to any one of claims 23 to 28, comprising calculating a loss by comparing at least a portion of the third predicted output dataset with the first data.

30. Further includes training a further machine learning model, which includes the trained machine learning model, the further machine learning model being trained for a specific task and being a classification model, regression model, generative model, Optionally, The classification model includes the trained machine learning model and a classification model that takes features extracted from the transitive and commutative machine learning models as input and is trained to produce classifications of the input dataset provided to the trained machine learning model as output, wherein the classification model is optionally trained using training data that includes data relating to one or more of the sets of at least three modalities and associated ground truth classification labels. The generative model is trained to take an input dataset as input and to produce a generated output dataset corresponding to the input dataset, the output dataset corresponding to one of the set of at least three modalities that are not present in the input dataset, and optionally the trained machine learning model is used as the generative model, or the trained machine learning model is fine-tuned using further training data to obtain the generative model, or The generative model is trained to take an input dataset containing missing data as input and to produce a generated output dataset corresponding to the missing data in the input dataset as output, and optionally the trained machine learning model is used as the generative model, or the trained machine learning model is fine-tuned using further training data to obtain the generative model, or The method according to any one of claims 23 to 29, wherein the regression model comprises the trained machine learning model and a regression model that takes features extracted from the trained machine learning model with respect to an input dataset as input and is trained to produce regression values ​​as output with respect to the input dataset, wherein optionally the regression model is a Cox proportional hazards model, the regression values ​​are survival predictions, and / or the regression model is trained using training data that includes data with respect to one or more of the set of at least three modalities and associated ground truth regression values.

31. A system comprising one or more processors and a computer-readable memory that stores instructions for causing the processors to perform the method according to any one of claims 1 to 30.

32. A non-temporary computer-readable storage medium comprising machine-executable instructions that cause the processor to perform the method according to any one of claims 1 to 30 when executed by the processor.

Images