Selective Acquisition of Multimodal Temporal Data

Abstract

In one aspect, the method includes acquiring respective observations characterizing a state of the environment for each time step in a sequence of a plurality of time steps, the acquiring including, for each time step after a first time step in the sequence of time steps, processing a network input including the observations acquired for one or more preceding time steps to generate a plurality of acquisition decisions, acquiring the observations for the time step, the observations including data corresponding to a modality selected for acquisition at the time step and not including data corresponding to a modality not selected for acquisition at the time step, and the method further includes processing a model input including the observations for each time step in the sequence of time steps to generate the predictions.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to GR National Application No. 20220100868, filed October 21, 2022. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated by reference into the disclosure of this application. [Background technology]

[0002] This specification relates to processing data using machine learning models.

[0003] A machine learning model receives input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models, which generate an output based on the received input and the values ​​of the parameters of the model.

[0004] Some machine learning models are deep models that use multiple layers of the model to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, where each layer applies a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0005] This specification generally describes a system, implemented as a computer program on one or more computers at one or more locations, that generates predictions that characterize an environment.

[0006] According to one aspect, a method implemented by one or more computers is provided, the method including obtaining respective observations characterizing a state of an environment for each time step in a sequence of a plurality of time steps, the obtaining including, for each time step starting from a first time step in the sequence of time steps, processing a network input including the observations obtained for any predicted time steps using a selection neural network to generate a plurality of acquisition decisions, each acquisition decision corresponding to a respective modality from a set of a plurality of modalities, defining whether data corresponding to the modality is selected for acquisition at the time step, the obtaining further including obtaining observations for the time step, the observations including only data corresponding to the modality selected for acquisition at the time step, and the method further including processing a model input including the observations for each time step in the sequence of time steps using a predictive model to generate a prediction characterizing the environment.

[0007] In some implementations, the method further includes determining an acquisition cost based on each modality selected for acquisition at each time step in the sequence of time steps, determining a reward based at least in part on the acquisition cost, and training a selection neural network based on the reward using reinforcement learning techniques.

[0008] In some embodiments, each modality in the set of modalities is associated with a respective cost factor, and determining the acquisition cost includes, for each time step in the sequence of time steps, determining a respective acquisition cost for the time step based on a respective cost factor associated with each modality selected for acquisition at the time step, and determining the acquisition cost as a combination of the acquisition costs for the time steps.

[0009] In some embodiments, for each time step in the sequence of time steps, determining the acquisition cost for the time step includes determining the acquisition cost for the time step as a sum of cost factors associated with each modality selected for acquisition at the time step.

[0010] In some implementations, determining the acquisition cost as a combination of the acquisition costs for the time steps includes determining the acquisition cost as a sum over the acquisition costs for the time steps.

[0011] In some implementations, for one or more of the modalities, the cost factor of the modality is based at least in part on the resource usage required to capture the data corresponding to the modality.

[0012] In some implementations, the resource usage required to capture data corresponding to a modality characterizes at least an energy usage required to capture data corresponding to the modality.

[0013] In some implementations, the resource usage required to capture data corresponding to a modality characterizes at least an amount of time required to capture data corresponding to the modality.

[0014] In some implementations, for one or more of the modalities, the cost factor of the modality is based at least in part on the risk associated with capturing data corresponding to the modality.

[0015] In some embodiments, the environment includes a patient, and the risk associated with capturing the data corresponding to the modality is based at least in part on a medical risk to the patient resulting from capturing the data corresponding to the modality.

[0016] In some implementations, the method further includes determining a prediction error measuring an error in the prediction generated by the predictive model, and determining a reward based on both (i) the acquisition cost and (ii) the prediction error.

[0017] In some embodiments, the predictive model is a machine learning model.

[0018] In some embodiments, the predictive model comprises a neural network.

[0019] In some implementations, the method further includes training the predictive machine learning model to optimize an objective function that depends on the prediction error of the predictive machine learning model.

[0020] In some embodiments, for each time step in the sequence of time steps, the network input to the selection neural network at the time step further includes data identifying the acquisition decisions of any modalities at any preceding time steps.

[0021] In some implementations, the method further includes, for each of one or more time steps in the sequence of time steps, processing model inputs including observations for the time step and observations for one or more preceding time steps in the sequence of time steps using a predictive model to generate an intermediate prediction characterizing the environment; determining an intermediate prediction error measuring an error in the intermediate predictions generated by the predictive model; and determining a reward based at least in part on the intermediate prediction error.

[0022] In some embodiments, the set of modalities includes an imaging modality, and the data corresponding to the imaging modality includes image data.

[0023] In some embodiments, the set of modalities includes medical imaging modalities.

[0024] In some embodiments, the environment is a medical environment containing a patient.

[0025] In some embodiments, the prediction characterizing the environment includes a predicted medical diagnosis for the patient.

[0026] In some embodiments, the prediction characterizing the environment includes a prediction of a medical treatment to be applied to a patient.

[0027] In some embodiments, the method further includes determining, for each time step after the first time step in the sequence of time steps, that (i) data corresponding to modalities from the set of modalities that are selected for acquisition at the time step is included in the observations for the time step, and (ii) data corresponding to modalities from the set of modalities that are not selected for acquisition at the time step is not included in the observations for the time step.

[0028] In some embodiments, for each of one or more time steps after the first time step in the sequence of time steps, only a proper subset of the modalities in the set of modalities are selected for acquisition at the time step.

[0029] In some embodiments, the method further includes, for each time step after the first time step in the sequence of time steps, causing data to be acquired only for the modality selected for acquisition at the time step.

[0030] According to another aspect, a system is provided that includes one or more computers and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of methods described herein.

[0031] According to other aspects, one or more non-transitory computer storage media are provided that store instructions that, when executed by one or more computers, cause the one or more computers to perform operations of the methods described herein.

[0032] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0033] This specification describes a system for processing multimodal data captured over a sequence of time points to generate predictions for characterizing an environment. In many real-world scenarios, capturing data corresponding to a modality can be costly, e.g., in terms of resource consumption (e.g., energy or time consumption) or risk (e.g., medical risks resulting from exposing a patient to radiation resulting from obtaining medical images of the patient, e.g., x-ray or CT images). Furthermore, processing data corresponding to a particular modality can also be costly, e.g., in terms of computational resources (e.g., memory and computing power), for example, for high-dimensional data such as image data, video data, or audio data. The system described herein can adaptively determine which data modalities need to be acquired at each time point and can acquire fewer than all available modalities (or none at all) at a particular time point.

[0034] The system can be trained using machine learning techniques to optimize the trade-off between acquisition cost and predictive performance. In particular, the system can be trained to achieve acceptable predictive performance while minimizing acquisition cost across available modalities, thereby enabling more efficient use of resources (e.g., energy or computational resources) and reduced risk (e.g., medical risk). In some cases, the system can be trained to optimize predictive performance while encouraging (or requiring) acquisition costs to meet cost budgets.

[0035] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0036] [Figure 1] 1 shows an example of a neural network system. [Figure 2] 1 is a flowchart of an example process for generating a prediction characterizing an environment. [Figure 3] 1 is a flowchart of an exemplary process for training a selection neural network. [Figure 4] 4 is a flowchart of an exemplary process for a sub-step of one of the steps of the process of FIG. 3. [Figure 5] 4 is a flowchart of another exemplary process for a sub-step of one of the steps of the process of FIG. 3. [Figure 6] FIG. 10 is an exemplary diagram of generating a prediction using a selected neural network and determining one or more updates to parameter values ​​of the selected neural network based on the prediction. DETAILED DESCRIPTION OF THE INVENTION

[0037] Like reference symbols and designations in the various drawings indicate like elements.

[0038] 1 illustrates an example of a neural network system 100. Neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0039] The neural network system 100 includes a selection neural network 110, a predictive model 120, a data acquisition engine 130, and optionally, in some implementations, a training engine 140.

[0040] In general, a neural network system 100 is a system that acquires input data 102 generated in or about an environment at each time step over a sequence of multiple time steps and makes one or more predictions 122 that characterize one or more aspects of the environment. For example, the neural network system 100 may output a prediction 122 after acquiring input data 102 for the last time step in the sequence of multiple time steps.

[0041] At any given time step in the sequence, the input data 102 acquired by the system can potentially (but not necessarily) include data from a set of two or more available modalities. As used herein, data "modality" refers to the type of data generated, for example, using a specified sensor or diagnostic technique (e.g., a medical diagnostic technique).

[0042] The set of modalities may include any suitable modalities. Some examples of possible modalities are described below. In some implementations, the set of modalities includes one or more of these examples.

[0043] In some embodiments, the set of modalities includes an imaging modality, and the data corresponding to the imaging modality includes image data (e.g., one-dimensional (1D) image data, two-dimensional (2D) image data, three-dimensional (3D) image data, etc. The image data may include pixel value data, e.g., color or monochrome pixel value data.

[0044] For example, the collection of modalities may include one or more medical imaging modalities, such as a computed tomography (CT) modality, an ultrasound (US) modality, a magnetic resonance imaging (MRI) modality, an x-ray modality, a tissue imaging modality, an electroencephalogram (EEG) modality, an electromyogram (EMG) modality, an electrocardiogram (ECG) modality, etc.

[0045] As another example, the set of modalities can include a camera modality, eg, data corresponding to the camera modality is captured using a camera, eg, a visible spectrum camera or an infrared spectrum camera.

[0046] In some embodiments, the set of modalities can include a genetic data modality, and the data corresponding to the genetic data modality includes genetic data. The genetic data can include, for example, data defining the respective expression levels (in the subject) of each gene in the set of genes. The genetic data can be obtained by a suitable diagnostic technique, such as DNA or RNA sequencing, performed on genetic material obtained from the subject.

[0047] In some embodiments, the set of modalities can include a proteomic data modality, and the data corresponding to the proteomic data modality can include proteomic data. The proteomic data can include, for example, data defining the respective expression levels (in a subject) of each protein in the set of proteins.

[0048] In some embodiments, the set of modalities can include a blood test modality, and the data corresponding to the blood test modality can include data defining the levels of one or more components of the subject's blood, such as sodium, potassium, chloride, bicarbonate, blood urea nitrogen, magnesium, creatinine, glucose, calcium, cholesterol, etc.

[0049] In some implementations, the set of modalities may include an audio modality, and the data corresponding to the audio modality may include audio data, e.g., audio data characterizing words spoken by a person, audio data characterizing sounds made by one or more body parts of a person (e.g., the heart, digestive system, lungs, etc.), etc. The audio data may include data defining an audio waveform, such as a series of values ​​in the time and / or frequency domain that define an audio waveform.

[0050] In some embodiments, the set of modalities can include a biopsy modality, and the data corresponding to the biopsy modality can characterize a cell or tissue sample obtained from the patient by biopsy. For example, the data corresponding to the biopsy modality can include a microscopic image of the sample obtained from the patient.

[0051] In some embodiments, the collection of modalities may include modalities that measure one or more of humidity, light, air quality, sound, temperature, wind speed, pH, and the like.

[0052] In certain embodiments, the set of modalities includes at least medical imaging modalities and blood testing modalities.

[0053] In certain embodiments, the set of modalities includes at least a medical imaging modality, a blood test modality, and a biopsy modality.

[0054] In certain embodiments, the set of modalities includes at least a medical imaging modality, a blood test modality, a biopsy modality, and a genetic data modality.

[0055] In some of these implementations, these data types differ not only in feature space and dimensionality, but also in the data capturing processes and costs associated with capturing data corresponding to these data types. For example, medical images may be taken at the discretion of a physician and need to be captured at a relatively high cost, e.g., in terms of resource consumption or risk, whereas blood pressure and temperature may be monitored periodically and captured at a relatively low cost.

[0056] The environment can be any suitable environment, for example, a real-world environment, such as a medical environment, an agricultural environment, an aquaculture environment, an industrial environment, or a scientific environment.

[0057] The medical environment can include a patient, and one or more of the modalities can be modalities that generate data characterizing the patient, as described above.

[0058] The industrial environment can include, for example, a manufacturing facility (e.g., including another industrial machine used to produce manufactured goods), a chemical processing facility (e.g., including another industrial machine used for chemical processing), a data center facility (e.g., including a collection of computing units, e.g., processors, used to perform computing tasks), or an energy production facility (e.g., a nuclear power plant, a hydroelectric power plant, a solar power plant, etc.). The one or more modalities can be modalities that generate data characterizing the facility, e.g., data generated by one or more sensors located in or around the facility, e.g., sensors for measuring the condition of industrial machines or computing units in the facility. A prediction characterizing the industrial environment may include predicted values ​​of one or more properties that can be determined based on sensor values ​​or may include one or more properties measured by a sensor(s), e.g., the predicted values ​​may include predicted sensor values.

[0059] A scientific environment can include a collection of objects being studied for scientific purposes, and the objects can include, for example, plants, animals, cells, tissues, and the like.

[0060] Because an environment can evolve over time, input data 102 generated in or about an environment at a first time step can have different values ​​than input data 102 generated in or about an environment at a second time step. Thus, a collection of input data 102 acquired over a sequence of multiple time steps may, in some implementations, be referred to as a "time" sequence of input data, because the input data is arranged according to the time steps at which the input data was captured. For example, the most recent input data is the last input data in a time sequence of input data, and the oldest input data is the first input data in the time sequence.

[0061] One or more predictions 122 characterizing one or more aspects of the environment are made by the predictive model 120 based on the input data 102. The predictive model 120 may be configured as a machine learning model, which may have any suitable machine learning model architecture. For example, the predictive model 120 may be implemented as a neural network, or a decision tree, or a random forest, or a support vector machine, or a linear regression model, etc. In a particular example, the predictive model 120 may be implemented as a neural network that may include any suitable type of neural network layer (e.g., fully connected layers, attention layers, convolutional layers, etc.) in any suitable number (e.g., 5 layers, 10 layers, or 100 layers), which may be combined in any suitable configuration (e.g., as a directed graph of layers).

[0062] Next, some examples of possible predictions 122 are described.

[0063] In some embodiments, the environment is a medical environment including a patient, and the prediction defines a predicted medical procedure to be applied to the patient. For example, the prediction can include a respective score for each medical procedure in a set of medical procedures, where the medical procedure score defines a likelihood that the medical procedure should be applied to the patient. The set of medical procedures can include medical procedures corresponding to administering a medication to the patient, performing an intervention (e.g., surgery) on the patient, etc.

[0064] In some embodiments, the environment is a medical environment including a patient, and the prediction defines a predicted medical diagnosis for the patient. For example, the prediction can include a respective score for each medical diagnosis in a set of medical diagnoses, where the score of the medical diagnosis defines a likelihood of applying the medical diagnosis to the patient. The set of medical diagnoses can include, for example, diagnoses of one or more diseases, such as cancer, diabetes, heart failure, Alzheimer's disease, influenza, measles, strep throat, sepsis, etc.

[0065] In some embodiments, the environment is an agricultural environment (e.g., an environment in which crops are grown) or an aquaculture environment (e.g., an environment in which aquatic organisms are cultivated), and the prediction defines a predicted yield (e.g., measured in tons of crops or aquatic organisms) or a predicted amount of time (e.g., measured in days) until the crops or aquatic organisms in the environment are harvested.

[0066] In some embodiments, the environment is an industrial environment, and the forecast defines a projected production level of the industrial environment over a predetermined time range (e.g., an hour, a day, or a week), e.g., the number of units of product produced at a manufacturing facility, or the amount of chemical produced at a chemical processing facility, or the number of computing tasks completed at a data center facility, or the amount of energy produced at an energy production facility.

[0067] In some embodiments, the environment is a scientific environment and the prediction defines a predicted outcome of a scientific study, e.g., the health of a subject of a scientific study, e.g., the cell wall integrity of a collection of cells at the end of the study, or the weight of animals in a population of animals at the end of the study.

[0068] In these exemplary environments and many other real-world environments, capturing data corresponding to a modality often comes with a high cost, e.g., in terms of resource consumption (e.g., energy or time consumption) or in terms of risk (e.g., medical risks resulting from exposing a patient to radiation resulting from obtaining medical images of the patient, e.g., x-ray or CT images). Furthermore, processing data corresponding to a particular modality can also come with a high cost, e.g., in terms of computational resources (e.g., memory and computing power), e.g., for high-dimensional data such as image data, video data, or audio data.

[0069] Thus, while neural network system 100 may potentially receive input data corresponding to each and every modality in a set of two or more available modalities at each time step, the system may not actually do so, but instead may only acquire (and subsequently receive) input data corresponding to each modality in a proper subset of the set of two or more available modalities at each of one or more time steps, where the proper subset includes at least one modality in the set of two or more available modalities, but fewer than all of the modalities in the set.

[0070] In the example of FIG. 1, neural network system 100 may potentially receive data corresponding to a set of three data modalities that may be made available to the system: data 102A corresponding to modality A, data 102B corresponding to modality B, and data 102C corresponding to modality C.

[0071] However, as shown, the actually acquired input data 102 is multimodal data that includes only data corresponding to two of the three available data modalities, namely, data 102A corresponding to modality A and data 102B corresponding to modality B. That is, data 102C corresponding to modality C is not acquired and therefore not received by the system, and the actually acquired input data 102 does not include data 102C corresponding to modality C.

[0072] In other examples, the input data may potentially (but not necessarily) include data corresponding to a smaller (e.g., 2) or larger (e.g., 10, 100, or more) set of available modalities. Similarly, in those examples, the actually acquired input data 102 may include data corresponding to each modality in a proper subset of the smaller or larger set of available modalities.

[0073] In particular, for each time step after the first time step in the sequence of time steps, the neural network system 100 uses the selection neural network 110 to make an acquisition decision for each modality in the set of modalities that defines whether data corresponding to the modality will be selected for acquisition.

[0074] At any given time step after the first time step in the sequence of multiple time steps, the selection neural network 110 processes network inputs including (i) previous observations 112 obtained for one or more preceding time steps and, optionally, (ii) data identifying acquisition decisions for any modalities at any preceding time step, to generate multiple acquisition decisions for the given time step. Each acquisition decision corresponds to a respective modality from the set of multiple modalities and defines whether data corresponding to the modality should be selected for acquisition at the given time step. As explained further below, "observations" refer to data generated by the data acquisition engine 130 from the acquired input data 102 and provided to the selection neural network 110 and / or the predictive model 120 for further processing.

[0075] The selection neural network 110 may have any suitable neural network architecture that enables the selection neural network 110 to generate acquisition decisions from prior observations. In particular, the selection neural network may include any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) in any suitable number (e.g., 5 layers, or 10 layers, or 100 layers), which may be combined in any suitable configuration (e.g., as a directed graph of layers).

[0076] As a specific example, the selection neural network 110 and the prediction model 120 can each be configured as a respective neural network having one of the architectures described in Andrew Jaegle, et al. PerceiverIO: A general architecture for structured inputs & outputs. In International Conference on Representation Learning, 2022.

[0077] 1, at a particular time step, selection neural network 110 generates three acquisition decisions: decision A corresponding to modality A, decision B corresponding to modality B, and decision C corresponding to modality C. Specifically, decision A defines that data corresponding to modality A should be selected for acquisition at the time step, decision B defines that data corresponding to modality B should be selected for acquisition at the time step, and decision C defines that data corresponding to modality C should not be selected for acquisition at the time step. Selection neural network 110 can generate different acquisition decisions at other time steps.

[0078] Each acquisition decision can be generated deterministically, e.g., by the output of the selection neural network 110. For example, the output layer of the selection neural network 110 can include a respective neuron corresponding to each modality, and each modality is selected for acquisition only if the neuron's activation exceeds a predetermined threshold. Alternatively, each acquisition decision can be generated probabilistically, e.g., the output of the selection neural network 110 parameterizes the distribution from which the acquisition decision is sampled. For example, each acquisition decision can be a binary decision, where 0 indicates that data corresponding to the particular modality does not need to be selected for acquisition and 1 indicates that data corresponding to the particular modality does need to be selected for acquisition.

[0079] The neural network system 100 then uses the data acquisition engine 130 to implement the acquisition decisions generated by the selection neural network 110. That is, the neural network system 100 provides the acquisition decisions to the data acquisition engine 130, which causes data to be acquired only for the modalities selected for acquisition at a given time step.

[0080] 1, the data acquisition engine 130 acquires data corresponding to modality A and data corresponding to modality B according to the acquisition decisions generated by the selection neural network 110. The data acquisition engine 130 refrains from acquiring data corresponding to modality C.

[0081] In some implementations, the data acquisition engine 130 may realize (e.g., execute or enforce) the acquisition decision by passing an electronic signal to a sensor or other electronic device with environmental sensing capabilities communicatively coupled to the system to capture data corresponding to one of the selected modalities. In response to receiving the electronic signal, the sensor operates to capture data regarding or about the environment.

[0082] In some implementations, the data acquisition engine 130 can implement the acquisition decisions by generating and outputting prompts for presentation to a user via a user interface device, which can be any suitable stationary or mobile computing device, such as a desktop computer, a workstation in a medical environment, a tablet, a smartphone, or a smartwatch.

[0083] The prompt can help guide the user in capturing data according to the acquisition decisions generated by the selection neural network 110. The prompt can instruct the user which modalities of data need to be captured. For example, a prompt can be presented in a window with text inquiring that data corresponding to one of the selected modalities needs to be captured. The user can interact with a user interface device to display the selected modality and upload the data corresponding to the selected modality after the data has been captured.

[0084] The neural network system 100 then generates observations 112 for a given time step from the input data 102 acquired by the data acquisition engine 130 according to the acquisition decisions generated by the selection neural network 110 .

[0085] The observations 112 for a given time step (i) include data corresponding to modalities from the set of modalities that are selected for acquisition at the given time step, and (ii) exclude, i.e., do not include, data corresponding to modalities from the set of modalities that are not selected for acquisition at the given time step. After being generated, the observations 112 are then provided to the predictive model 120 for further processing.

[0086] In this manner, although the data potentially available to the system includes multimodal data, each corresponding to a set of modalities, only data corresponding to a proper subset of the modalities within the set of modalities may actually be selected for acquisition by neural network system 100. For example, only data corresponding to a small number of modalities within a relatively large number of modalities may be selected for acquisition by neural network system 100 and subsequently used by predictive model 120 to generate predictions 122.

[0087] By incorporating the selection neural network 110 and acquiring data according to acquisition decisions generated by the selection neural network 110, the neural network system 100 can reduce the amount of computational resources consumed by the prediction process by eliminating the need to repeatedly acquire and then process data from the entire set of modalities. Instead, at least at each of some time steps, data from only a relatively small number of selected modalities need to be acquired and then processed.

[0088] The training engine 140, when included, can train the selected neural network 110 and, optionally, the predictive model 120, and determine trained parameter values ​​for the selected neural network 110 and, optionally, the predictive model 120, thereby enabling the selected neural network 110 to generate winning decisions, thereby reducing the consumption of computational resources by the system while maintaining predictive performance, e.g., in terms of accuracy of predictions 122. Thus, in some implementations, the selected neural network 110 and the predictive model 120 can be jointly trained by the training engine 140.

[0089] 1, training engine 140 includes or has access to cost calculation engine 145. Cost calculation engine 145 is configured to calculate acquisition costs associated with modalities selected for acquisition according to acquisition decisions generated by selection neural network 110.

[0090] Thus, the training engine 140 can jointly train the selection neural network 110 with the predictive model 120 using rewards derived from acquisition costs to apply reinforcement learning techniques that optimize the trade-off between acquisition costs and predictive performance.

[0091] In particular, the training engine 140 can train the selection neural network 110 and the prediction model 120 to achieve acceptable predictive performance while minimizing acquisition costs across available modalities, thereby enabling more efficient use of resources (e.g., energy or computational resources) and reduced risks (e.g., medical risks).

[0092] The cost calculation engine 145 can be configured to calculate the acquisition cost of a modality in the collection of modalities based on any suitable criteria. The following describes some examples of possible criteria for setting the acquisition cost of a modality.

[0093] In some implementations, the acquisition cost of a modality may be based at least in part on the usage of resources (eg, energy or time) required to acquire data corresponding to the modality.

[0094] In some embodiments, the acquisition cost of a modality can be based at least in part on the amount of risk required to acquire data corresponding to the modality. For example, in a medical setting, acquiring data corresponding to a biopsy modality may incur a patient's risk of infection, while acquiring data corresponding to an x-ray modality may incur a patient's risk of exposure to unhealthy levels of radiation. The amount of risk may be determined based on statistics characterizing different outcomes (e.g., patient outcomes) once data corresponding to the modality is acquired.

[0095] In some implementations, the acquisition cost of a modality can be based at least in part on the level of disruption caused by acquiring data corresponding to the modality. For example, in an industrial environment, acquiring data corresponding to the modality may include running a diagnostic test that reduces production of an industrial facility. As another example, in a scientific environment, acquiring data corresponding to the modality may include disrupting conditions in the environment (e.g., by conducting tests on one or more subjects in the environment) in a manner that could compromise the validity or accuracy of experimental results. Training the selection neural network 110 is further described below with reference to Figures 3-6.

[0096] 2 is a flow diagram of an exemplary process 200 for generating predictions characterizing an environment. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system, such as the neural network system 100 of FIG. 1 , appropriately programmed, can perform process 200.

[0097] The environment can be any suitable environment, for example, a real-world environment, such as a medical environment, an agricultural environment, an aquaculture environment, an industrial environment, or a scientific environment.

[0098] The system repeatedly performs steps 202 and 204 to obtain respective observations characterizing the state of the environment for each time step in the sequence of time steps, i.e., the system performs one iteration of steps 202 and 204 for each time step in the sequence of time steps.

[0099] In some implementations, the number of time steps is fixed (predefined). For example, the system may generate a sequence-level prediction after a predefined number of time steps have elapsed. In other implementations, the number of time steps is flexible, and different sequences may include various numbers of time steps. For example, the system may repeatedly perform the iterations of steps 202 and 204 until a termination signal (e.g., a flag or other indicator) is received at a given time step indicating that the given time step is the last time step in the sequence. For example, if the given time step is not the last time step in the sequence, the flag may be set to a first value, and if the given time step is the last time step in the sequence, the flag may be set to a second value. The termination signal may be based on the prediction(s) generated by the system.

[0100] For each time step after the first time step in the sequence of multiple time steps, the system uses a selection neural network to process network inputs including (i) observations obtained for one or more previous time steps and, optionally, (ii) data identifying acquisition decisions for any modalities at any previous time steps, to generate multiple acquisition decisions for the time step (step 202). "Observations" refers to data generated by the data acquisition engine 130 from the acquired input data 102 and provided to the selection neural network 110 and / or the predictive model 120 for further processing. Each acquisition decision corresponds to a respective modality from the set of multiple modalities and defines whether data corresponding to the modality is selected for acquisition at the time step.

[0101] For the first time step, because there are no preceding time steps, some implementations of the system may instead provide predetermined network inputs, i.e., inputs having predetermined values, for processing by the selection neural network. Some other implementations of the system may instead acquire a default (e.g., random or predefined) set of modalities, i.e., acquire without using a selection neural network to generate any acquisition decisions for the first time step.

[0102] The system acquires observations for the time steps according to the acquisition decisions generated by the selection neural network, step 204. For example, the system can use a data acquisition engine to acquire data corresponding to each modality selected for acquisition according to the acquisition decisions and then include the acquired data in the observations.

[0103] In particular, the observations (i) include data corresponding to modalities from the set of modalities that are selected for acquisition at the time step, and (ii) do not include data corresponding to modalities from the set of modalities that are not selected for acquisition at the time step.

[0104] The selected modality may, and typically will, vary from one time step to another, in other words, the system may acquire data corresponding to different modalities at different time steps.

[0105] In some examples, for one or more time steps in a sequence of multiple time steps, the system may obtain an observation that includes data corresponding to all of the modalities in the set. In other examples, for one or more time steps, the system may obtain an observation that includes data corresponding to a proper subset of the set of modalities (and does not include data corresponding to any remaining modalities that are not in the proper subset). A "proper" subset of a set is a subset that includes one or more, but not all, of the elements in the set. In other examples, for one or more time steps, the system may obtain a null observation that does not include data corresponding to any of the modalities in the set.

[0106] After performing the iterations of steps 202 and 204 for the last time step in the sequence of multiple time steps, the system processes model inputs including observations for each time step in the sequence of time steps using the predictive model to generate predictions that characterize the environment (step 206).

[0107] An example algorithm for generating a prediction is given below. [Table 1]

[0108] In Algorithm 1, each input x is a sequence of observations x i =(x i,1 ,...,x i,T ) at each time step t, the observation x i,t is the data x corresponding to M modalities i,t =(x i,t,1 ,...,x i,t,M ) Each modality may be high-dimensional,

number

[0109] At each time step t ∈ (1,...,T), multiple acquisition decisions a t =(a t,1 ,...,a t,M ) is generated by sampling from the output of a selection neural network (called an agent in Algorithm 1).

number

number

[0110] By repeatedly executing process 200, the system can generate different predictions that characterize the same or different aspects of the environment. That is, process 200 can be executed as part of generating predictions from a sequence of observations where the desired output, i.e., the desired prediction that needs to be generated by the system from the sequence of observations, is unknown. One or more actions may be performed based on the prediction(s). For example, an agent, such as an electromechanical agent, that interacts with a real-world environment to perform a task may select one or more actions to perform in the real-world environment according to the prediction(s).

[0111] Additionally, some of the steps of process 200 may be performed as part of processing sequences of observations derived from a set of training data, i.e., sequences of observations derived from input data for which predictions need to be made by the system are known, in order to train trainable components of the system and determine trained values ​​for the parameters of those components.

[0112] 3 is a flow diagram of an exemplary process 300 for training a selection neural network. For convenience, process 300 is described as being performed by one or more computer systems at one or more locations. For example, a neural network system, such as neural network system 100 of FIG. 1 , appropriately programmed, can perform process 300.

[0113] During training, process 300 may be performed after process 200 for each training input selected from a set of training data derived from multiple time sequences of input data generated in or about an environment (e.g., one of the physical environments described above, or a computer simulation of one of these physical environments). That is, for each training input, the system performs process 200 to generate a prediction characterizing the environment using a selected neural network and according to the current values ​​of the selected neural network's parameters, and then performs process 300 to determine one or more updates to the selected neural network's parameter values ​​based on the predictions generated in process 200.

[0114] Process 300 is shown in FIG. 6, which illustrates an example of using a selection neural network to generate a prediction and determining one or more updates to parameter values ​​of the selection neural network based on the prediction.

[0115] As shown in Figure 6, at any given time step, the selection neural network ("agent") generates three acquired decisions: a first decision corresponding to the text modality, a second decision corresponding to the image modality, and a third decision corresponding to the numeric modality. To generate these acquired decisions for a given time step, the selection neural network processes network inputs including observations obtained for one or more previous time steps to generate an output π that parameterizes the distribution from which the acquired decisions are sampled.

[0116] For example, at a first time step, a first decision defines that text data should be selected for acquisition at the time step, a second decision defines that image data should be selected for acquisition at the time step, and a third decision defines that numeric data should not be selected for acquisition at the time step. Some input data may contain missing entries, as represented by the [masked] token.

[0117] After generating predictions based on the modalities selected for acquisition at each time step by the selection neural network, the system determines an acquisition cost based on each modality selected for acquisition at each time step in the sequence of time steps (step 302). In some implementations, each modality in the set of modalities is associated with a respective cost factor. In these implementations, the system can perform substeps 402-404, as described in further detail with reference to FIG. 4, to determine the acquisition cost.

[0118] FIG. 4 is a flow diagram of sub-steps 402-404 of step 302 of the process of FIG.

[0119] In embodiments in which each modality in the set of modalities is associated with a respective cost factor, the system may determine, for each time step in the sequence of time steps, a respective acquisition cost for the time step based on the respective cost factors associated with each modality selected for acquisition at the time step (step 402). For example, the acquisition cost for a time step may be calculated as either a weighted or unweighted sum of the cost factors associated with each modality selected for acquisition at the time step.

[0120] The system determines the acquisition cost as a combination of the acquisition costs for the time steps in the sequence (step 404). For example, the acquisition cost can be calculated as either a weighted or unweighted sum over the respective acquisition costs for the time steps in the sequence.

[0121] For each of one or more of the modalities, the cost factor of the modality is based at least in part on a resource usage required to capture the data corresponding to the modality. For example, the resource usage required to capture the data corresponding to the modality characterizes at least an amount of energy usage required to capture the data corresponding to the modality. As another example, the resource usage required to capture the data corresponding to the modality characterizes at least an amount of time required to capture the data corresponding to the modality.

[0122] Additionally or alternatively, for each of one or more of the modalities, the cost factor of the modality is based at least in part on a risk associated with capturing data corresponding to the modality. If the environment is a medical environment including a patient, for example, the risk associated with capturing data corresponding to the modality is based at least in part on a medical risk to the patient resulting from capturing the data corresponding to the modality.

[0123] The system determines a reward based at least in part on the acquisition cost of the selected modality (step 304). The acquisition cost is typically a numerical value, but can be included in the reward in any suitable way.

[0124] For example, the system may determine compensation based at least in part on a comparison of acquisition costs to a threshold value referred to as a "cost budget." In some implementations, the system may reduce compensation by a predefined or adaptive amount if acquisition costs exceed the cost budget. The cost budget may indicate, for example, an acceptable level of acquisition costs, e.g., an acceptable amount of energy usage, or an acceptable amount of medical risk (e.g., based on a patient's radiation exposure tolerance), or an acceptable amount of computational resources (e.g., memory and computing power) to be used to process data from acquired modalities.

[0125] In some implementations, the reward depends on both (i) the acquisition cost and (ii) the prediction error, which measures the error of the prediction generated by the predictive model. In these implementations, as shown in Figure 6, the reward can be calculated, for example, as an expected value, as follows:

number

[0126] where the expectation is for the training input (x,y), where x is the sequence of observations, y is the ground truth prediction, a represents the acquisition decision generated by the selection neural network, C(a) represents the total acquisition cost of the sequence of observations, and C m is a modality-specific cost factor,

number

[0127] Optionally, in some implementations, the system adds the intermediate prediction error to the reward, e.g., the reward calculated using Equation (1). The intermediate prediction error is used to encourage the selection neural network to reduce the prediction error. In these implementations, the system can perform substeps 502-506, as described in further detail with reference to FIG. 5, to determine the reward.

[0128] FIG. 5 is a flow diagram of sub-steps 502-506 of step 304 of the process of FIG.

[0129] For each of one or more time steps in the sequence of time steps, the system processes model inputs including observations for the time step and observations for one or more preceding time steps in the sequence of time steps using a predictive model to generate intermediate predictions that characterize the environment (step 502).

[0130] For each of one or more time steps in the sequence of time steps, the system determines an intermediate prediction error that measures the error in the intermediate predictions produced by the predictive model (step 504).

[0131] The system determines a reward based at least in part on the intermediate prediction error determined for one or more time steps (step 506). For example, the system may add the intermediate prediction error to the reward calculated using equation (1). The intermediate prediction error may be calculated, for example, as follows:

number

number

[0132] The system trains the selection neural network based on the reward using a reinforcement learning technique to adjust the values ​​of the selection neural network's parameters (step 304). In particular, the system trains the selection neural network to generate winning decisions that maximize rewards determined at least in part based on winning costs. For example, the reinforcement learning technique can be a policy gradient technique, e.g., an advantage actor critic (A2C) policy gradient technique, that applies a Gumbel parameterization to (discrete) winning decisions.

[0133] In some implementations, the system also trains the predictive model based on a reward, e.g., a reward calculated using equation (1) that depends on both the acquisition cost and the prediction error, to simultaneously adjust the values ​​of the parameters of the predictive model.

[0134] For example, the system may jointly train the predictive model and the selection neural network to update parameter values ​​of both the selection neural network and the predictive model, such as to enable the predictive model to specifically adapt to the combination of modalities frequently selected by the selection neural network. In this example, updating the parameter values ​​of the predictive model may result in a change in the reward received by the selection neural network.

[0135] Alternatively, in other embodiments, the system can train the predictive model separately from training the selected neural network (during which the parameter values ​​of the predictive model remain fixed), e.g., based on optimizing an objective function that depends on the prediction error of the predictive machine learning model.

[0136] For example, the system can pre-train a predictive model to process a sequence of masked observations and generate corresponding predictions, and then train a selected neural network to update the parameter values ​​of the selected neural network while holding the pre-trained parameter values ​​of the predictive model fixed.

[0137] The term "configured" is used herein in connection with systems and computer program components. When one or more computer systems are configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when running, causes the system to perform the operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0138] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, such as the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by or to control the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively, or in addition, the program instructions can be encoded in an artificially generated, transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver apparatus for execution by the data processing apparatus.

[0139] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or multiple computers. An apparatus may be or further include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may also optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0140] A computer program, which may be called or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, for example, files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0141] The term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and run on the same computer or computers.

[0142] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0143] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The basic elements of a computer are a central processing unit for implementing and executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated into, special-purpose logic circuitry. Typically, a computer also includes, or is operatively coupled to receive data from, transfer data to, or both of, one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks. However, such devices are not required for a computer. Furthermore, a computer may be incorporated in other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0144] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all types of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0145] To provide for user interaction, embodiments of the subject matter described herein can be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, voice input, or tactile input. Furthermore, a computer can interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other types of messages to a personal device, such as a smartphone running a messaging application, and receiving a reply message from the user in return.

[0146] A data processing device for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling the common and computationally intensive part of machine learning training or production, i.e., inference, workloads.

[0147] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the JAX framework.

[0148] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components such as, for example, a data server, or that includes middleware components such as, for example, an application server, or that includes front-end components such as a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include a local area network (LAN), a wide area network (WAN), e.g., the Internet.

[0149] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying the data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of user interaction, may be received from the device by the server.

[0150] While the specification contains many specific implementation details, these should not be construed as limiting the scope or patentability of any invention, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features of the invention that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombination.

[0151] Similarly, while operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems generally may be integrated into a single software product or packaged into multiple software products.

[0152] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method implemented by one or more computers, the method comprising: obtaining a respective observation characterizing a state of an environment for each time step in a sequence of a plurality of time steps, said obtaining including, for each time step after a first time step in said sequence of time steps: processing a network input including observations obtained for one or more prior time steps using a selection neural network to generate a plurality of acquisition decisions; Each acquisition decision corresponds to a respective modality from a set of multiple modalities and defines whether data corresponding to the modality is selected for acquisition at the time step, and the acquiring further comprises: obtaining observations for the time step, the observations (i) including data corresponding to modalities from the set of modalities that are selected for acquisition at the time step, and (ii) including no data corresponding to modalities from the set of modalities that are not selected for acquisition at the time step, the method further comprising: processing a model input including the observations for each time step in the sequence of time steps using a predictive model to generate a prediction characterizing the environment. method.

2. determining an acquisition cost based on the respective modality selected for acquisition at each time step in the sequence of time steps; determining a compensation based at least in part on said acquisition cost; training the selection neural network based on the rewards using reinforcement learning techniques; The method of claim 1 further comprising:

3. Each modality in the set of modalities is associated with a respective cost factor, and determining the acquisition cost includes: for each time step in the sequence of time steps, determining a respective acquisition cost for the time step based on the respective cost factors associated with each modality selected for acquisition at that time step; determining the acquisition cost as a combination of the acquisition costs for the time steps; The method of claim 2 , comprising:

4. For each time step in the sequence of time steps, determining the acquisition cost for the time step includes: The method of claim 3 , comprising determining the acquisition cost for the time step as a sum of the cost factors associated with each modality selected for acquisition at the time step.

5. Determining the acquisition cost as a combination of the acquisition costs for the time steps may include: A method according to any one of claims 3 to 4, comprising determining the acquisition cost as a sum over the acquisition costs for the time steps.

6. 6. The method of claim 3, wherein for one or more of the modalities, the cost factor of the modality is based at least in part on the resource usage required to capture data corresponding to the modality.

7. The method of claim 6 , wherein the resource usage required to capture data corresponding to the modality characterizes at least an energy usage required to capture data corresponding to the modality.

8. The method of any one of claims 6 to 7, wherein the resource usage required to capture data corresponding to the modality characterizes at least an amount of time required to capture data corresponding to the modality.

9. 9. The method of claim 3, wherein for one or more of the modalities, the cost factor of the modality is based at least in part on a risk associated with capturing data corresponding to the modality.

10. 10. The method of claim 9, wherein the environment includes a patient, and the risk associated with capturing data corresponding to the modality is based at least in part on a medical risk to the patient resulting from capturing data corresponding to the modality.

11. determining a prediction error measuring the error in the prediction produced by the predictive model; determining the reward based on both (i) the acquisition cost and (ii) the prediction error; The method of any one of claims 2 to 10, further comprising:

12. The method of any one of claims 1 to 11, wherein the predictive model is a machine learning model.

13. The method of claim 12 , wherein the predictive model comprises a neural network.

14. 14. The method of claim 12, further comprising training the predictive machine learning model to optimize an objective function that depends on a prediction error of the predictive machine learning model.

15. For each time step in the sequence of time steps, the network input to the selected neural network at that time step is: The method of any preceding claim, further comprising data identifying said acquisition decisions of any modality at any preceding time step.

16. For each of one or more time steps in the sequence of time steps: processing model inputs including the observations for the time step and observations for one or more preceding time steps in the sequence of time steps using the predictive model to generate intermediate predictions characterizing the environment; determining an intermediate prediction error measuring the error in the intermediate predictions produced by the predictive model; determining the reward based at least in part on the intermediate prediction error; The method of any one of claims 2 to 15, further comprising:

17. The method of any preceding claim, wherein the set of modalities comprises an imaging modality, and the data corresponding to the imaging modality comprises image data.

18. The method of claim 17 , wherein the set of modalities includes medical imaging modalities.

19. The method of any one of claims 1 to 18, wherein the environment is a medical environment containing a patient.

20. The method of claim 19 , wherein the prediction characterizing the environment comprises a predicted medical diagnosis for the patient.

21. The method of any one of claims 19 to 20, wherein the prediction characterizing the environment comprises a prediction of a medical treatment to be applied to the patient.

22. For each time step after the first time step in the sequence of time steps, 22. The method of claim 1, further comprising determining that (i) data corresponding to modalities from the set of modalities that are selected for acquisition at the time step are included in the observations for the time step, and (ii) data corresponding to modalities from the set of modalities that are not selected for acquisition at the time step are not included in the observations for the time step.

23. for each of one or more time steps after the first time step in the sequence of time steps:

23. The method of claim 22, wherein only a proper subset of the modalities in the set of modalities are selected for acquisition at the time step.

24. For each time step after the first time step in the sequence of time steps, The method of any one of claims 22 to 23, further comprising causing data to be acquired only for modalities selected for acquisition at said time step.

25. one or more computers; one or more storage devices communicatively coupled to the one or more computers, A system, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective methods described in any one of claims 1 to 24.

26. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations of any one of claims 1 to 24.

Images