Generating neural network outputs by enriching latent embeddings using self-attention and mutual attention operations

Abstract

To provide a method and a system for generating neural network output.SOLUTION: A method includes the steps of: obtaining representation of an entity as a set of data element embeddings; obtaining a set of latent embeddings; and processing, the set of data element embeddings, and the set of latent embeddings, using the neural network to generate network output characterizing the entity. The neural network includes one or more cross-attention blocks, one or more self-attention blocks, and an output block. Each cross-attention block updates each latent embedding using attention over some or all of the data element embeddings. Each self-attention block updates each latent embedding using attention over the set of latent embeddings. The output block processes one or more latent embeddings to generate the network output that characterizes the entity.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 146,161, entitled "GENERATING NEURAL NETWORK OUTPUTS BY ENRICHING LATENT EMBEDDINGS USING SELF-ATTENTION AND CROSS-ATTENTION OPERATIONS," filed February 5, 2021, and incorporated herein by reference in its entirety.

[0002] This specification relates to processing data using machine learning models. [Background technology]

[0003] A machine learning model receives input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models, which generate an output based on the received input and the values ​​of the parameters of the model.

[0004] Some machine learning models are deep models that use multiple layers of the model to generate an output in relation to a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to the received input to generate an output. Summary of the Invention [Means for solving the problem]

[0005] This specification generally describes a system implemented as a computer program on one or more computers at one or more locations that uses a neural network to generate network outputs that characterize entities.

[0006] Throughout this specification, an embedding refers to an ordered collection of numbers, for example, a vector, matrix, or other tensor of numbers.

[0007] A block refers to a group of one or more neural network layers in a neural network.

[0008] The neural network can be configured to process data element embeddings that represent any suitable type of entity. For example, the entity can include an image, an audio waveform, a point cloud (e.g., generated by a LiDAR or radar sensor), a protein, a sequence of words (e.g., forming one or more sentences or paragraphs), a video (e.g., representing a sequence of video frames), or a combination thereof.

[0009] The neural network can be configured to generate any suitable neural network output that characterizes the entities. For example, the neural network output can be a classification output, a regression output, a sequence output (i.e., comprising a sequence of output elements), a segmentation output, or a combination thereof.

[0010] According to a first aspect, a method is provided for use with a neural network to generate a network output that characterizes an entity, the method comprising the steps of: obtaining a representation of the entity as a set of data element embeddings; obtaining a set of potential embeddings; and processing (i) the set of data element embeddings and (ii) the set of potential embeddings using the neural network to generate a network output that characterizes the entity. The neural network comprises a sequence of neural network blocks, including (i) one or more mutual attention blocks, (ii) one or more self-attention blocks, and (iii) an output block. Each mutual attention block performs operations including updating each potential embedding in the set of potential embeddings using attention for some or all of the data element embeddings in the set of data element embeddings. Each self-attention block performs operations including updating each potential embedding in the set of potential embeddings using attention for the set of potential embeddings. The output block performs operations including processing one or more potential embeddings from the set of potential embeddings to generate a network output that characterizes the entity after the set of potential embeddings has been updated using one or more mutual attention blocks and one or more self-attention blocks.

[0011] In some implementations, the number of potential embeddings in the set of potential embeddings is less than the number of data element embeddings in the set of data element embeddings.

[0012] In some implementations, the number of potential embeddings in the set of potential embeddings is predefined and independent of the number of data element embeddings in the set of data element embeddings.

[0013] In some implementations, the neural network includes multiple mutual attention blocks and multiple self-attention blocks, where the mutual attention blocks and the self-attention blocks are interleaved.

[0014] In some implementations, processing one or more latent embeddings from the set of potential embeddings by the output block to generate a network output that characterizes the entity includes pooling the latent embeddings in the set of potential embeddings to generate a pooled latent embedding, and processing the pooled latent embeddings using one or more neural network layers to generate a network output that characterizes the entity.

[0015] In some implementations, pooling the potential embeddings in the set of potential embeddings includes averaging the potential embeddings.

[0016] In some implementations, the network output characterizing the entity includes a sequence of output elements, wherein processing one or more potential embeddings from the set of potential embeddings by the output block to generate the network output characterizing the entity includes, at each time step of the multiple time steps, processing (i) one or more potential embeddings from the set of potential embeddings and (ii) the output elements generated at any previous time steps to generate an output element for that time step.

[0017] In some implementations, for each self-attention block, updating each potential embedding in the set of potential embeddings using attention for the set of potential embeddings includes updating each potential embedding in the set of potential embeddings using query-key value attention for the set of potential embeddings.

[0018] In some implementations, each self-attention block performs operations that include iteratively updating each potential embedding in the set of potential embeddings using attention for the set of potential embeddings.

[0019] In some implementations, for each mutual attention block, updating each potential embedding in the set of potential embeddings using attention for some or all of the data element embeddings in the set of data element embeddings includes updating each potential embedding in the set of potential embeddings using query-key-value attention for some or all of the data element embeddings in the set of data element embeddings, including generating a respective query embedding for each potential embedding in the set of potential embeddings; generating a respective key embedding and a respective value embedding for each of a number of data element embeddings in the set of data element embeddings; and updating each potential embedding in the set of potential embeddings using the query-key-value attention for the number of data element embeddings in the set of data element embeddings based on (i) the query embedding for the potential embedding, and (ii) the key embedding and value embedding for the data element embedding.

[0020] In some implementations, the entity includes a number of units arranged in a spatial structure, where each unit is associated with location data that defines the unit's respective location in the spatial structure, and obtaining a representation of the entity as a set of data element embeddings includes: for each unit in the entity, generating a feature embedding for that unit based on features of that unit; for each unit in the entity, generating a location embedding for that unit based on the unit's location in the spatial structure; and for each unit in the entity, generating a data element embedding for that unit based on (i) the feature embedding for that unit and (ii) the location embedding for that unit.

[0021] In some implementations, for each unit in the entity, generating a data element embedding for the unit based on (i) a feature embedding for the unit and (ii) a positional embedding for the unit includes concatenating the feature embedding for the unit and the positional embedding for the unit.

[0022] In some implementations, the spatial structure is a one-dimensional (1D), two-dimensional (2D), or three-dimensional (3D) array of units.

[0023] In some implementations, generating, for each unit in the entity, a positional embedding of that unit based on the unit's position in the spatial structure includes generating, for each unit in the entity, a Fourier feature positional encoding having frequency bands that are log-linearly spaced across a predefined target frequency range.

[0024] In some implementations, the entity includes an image, with each pixel in the image defining a respective unit in the entity.

[0025] In some implementations, the entity includes an audio waveform, with each audio sample in the audio waveform defining a respective unit in the entity.

[0026] In some implementations, the entity includes a point cloud, with each point in the point cloud defining a respective unit in the entity.

[0027] In some implementations, the entity includes a protein, and each amino acid in the amino acid sequence of the protein defines a respective unit in the entity.

[0028] In some implementations, an entity includes a sequence of words, with each word in the sequence of words defining a respective unit in the entity.

[0029] In some implementations, the sequence of neural network blocks of the neural network further includes one or more selection blocks, where each selection block performs operations including: processing the set of potential embeddings and the set of data element embeddings using one or more mutual attention blocks, one or more self-attention blocks, or both, to generate a respective selection score for each data element embedding in the set of data element embeddings after the set of potential embeddings has been updated; and selecting, based on the selection scores, an appropriate subset of the set of data element embeddings to be used by one or more designated mutual attention blocks, where each designated mutual attention block updates each potential embedding in the set of potential embeddings using attention for only the data element embeddings in the selected appropriate subset of the set of data element embeddings.

[0030] In some implementations, each selection block includes (i) a parameter selection neural network and (ii) a unit selection neural network, wherein for each selection block, processing the set of potential embeddings and the set of data element embeddings to generate a respective selection score for each data element embedding in the set of data element embeddings includes processing the potential embeddings using the parameter selection neural network to generate a network output defining values ​​for a set of neural network parameters of the unit selection neural network, and processing each data element embedding in the set of data element embeddings using the unit selection neural network according to values ​​of the set of neural network parameters of the unit selection neural network to generate a selection score for the data element embedding.

[0031] In some implementations, selecting an appropriate subset of data element embeddings to be used by the one or more designated mutual attention blocks based on the selection scores includes selecting a predefined number of data element embeddings having the highest selection scores among the set of data element embeddings.

[0032] In some implementations, the method further includes calculating a task performance measure based on the network output characterizing the entity, calculating a reward based on the task performance measure, and training the selection block against a reinforcement learning objective function that depends on the reward.

[0033] In some implementations, the task performance measure includes cross-entropy classification error.

[0034] In some implementations, the reinforcement learning objective function includes the squared Bellman error.

[0035] According to a second aspect, one or more non-transitory computer storage media are provided that store instructions that, when executed by one or more computers, cause the one or more computers to perform operations of the respective method of any preceding aspect. According to a third aspect, a system is provided that includes one or more computers and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any preceding aspect.

[0036] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0037] To generate output that characterizes an entity (e.g., an image) represented as a set of embeddings (called data element embeddings), the system described herein instantiates a set of potential embeddings and processes both the data element embeddings and the potential embeddings using a neural network. The system may instantiate a predefined number of potential embeddings that are independent of the number of data element embeddings. As part of processing the data element embeddings and the potential embeddings, the neural network uses mutual attention on the set of data element embeddings to update the set of potential embeddings, thereby enriching the potential embeddings with information from the data element embeddings. Furthermore, because the number of potential embeddings is independent of the number of data element embeddings, the computational complexity of the mutual attention operation is partially decoupled from the number of data element embeddings and remains feasible even for a large number of data element embeddings. Thus, the system enables complex inputs represented by a large number of data element embeddings (e.g., each data element embedding represents a single pixel in an image) to be efficiently processed using attention operations while reducing the consumption of computational resources (e.g., memory and computing power).

[0038] Rather than updating potential embeddings using mutual attention on the complete set of data element embeddings, the system can learn to adaptively select an appropriate subset of data element embeddings for which to pay attention. As a result, the system can reduce the amount of computational resources required to perform mutual attention operations on data element embeddings while maintaining acceptable task performance (e.g., prediction accuracy) of the neural network.

[0039] The system described herein processes sets of data element embeddings representing entities using attentional operations that do not require assuming that the data element embeddings are associated with a fixed spatial arrangement. For example, the attentional operations do not rely on assuming that the data element embeddings are associated with a spatial arrangement drawn up in a one-dimensional (1D) sequence (e.g., of audio data samples) or a two-dimensional (2D) grid (e.g., of image pixels). Instead, the system is able to flexibly incorporate information about the spatial arrangement of the data element embeddings by tagging (e.g., concatenating) the data element embeddings with a positional coding and allowing the attentional operations to learn to utilize this information where relevant to generating accurate network outputs. Thus, the system can be used to process sets of data element embeddings that are not associated with a predefined spatial arrangement, e.g., sets of data elements representing point clouds or proteins, thereby making the system more broadly applicable. This flexibility also facilitates processing multimodal data, including high-bandwidth data such as video and audio data, using the same shared neural network architecture to perform multimodal processing tasks. The computational reduction provided by system implementations can be particularly important in such tasks.

[0040] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description that follows. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0041] [Figure 1] FIG. 1 is a block diagram illustrating an example neural network system capable of characterizing entities. [Figure 2] FIG. 2 is a block diagram illustrating an exemplary neural network system in greater detail. [Figure 3]FIG. 1 is a block diagram illustrating an exemplary selection neural network block included in a neural network system capable of characterizing an entity. [Figure 4] 1 is a flowchart illustrating an exemplary process for using a neural network system to characterize an entity. [Figure 5A] FIG. 1 illustrates examples of entities and units that can be characterized by a neural network system. [Figure 5B] FIG. 10 illustrates another example of entities and units that can be characterized by a neural network system. [Figure 6A] FIG. 1 illustrates an exemplary attention map generated by a neural network system capable of characterizing entities. [Figure 6B] FIG. 10 illustrates another example of an attention map generated by a neural network system capable of characterizing entities. [Figure 7] 1A-1C illustrate exemplary performance of different configurations of a neural network system capable of characterizing entities. [Figure 8A] FIG. 1 illustrates exemplary parameters of a neural network system that can characterize an entity. [Figure 8B] FIG. 10 illustrates another example of parameters of a neural network system that can characterize an entity. [Figure 9] FIG. 1 shows experimental results achieved using a neural network system. [Figure 10] FIG. 1 shows experimental results achieved using a neural network system. DETAILED DESCRIPTION OF THE INVENTION

[0042] Like reference numbers and designations in the various drawings indicate like elements.

[0043] 1 is a block diagram of an exemplary neural network system 100 capable of generating a network output 150 that characterizes an entity. Neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0044] Throughout this specification, an "entity" may include any suitable type of data. For example, an entity may include an image, an audio waveform, a point cloud (e.g., generated by a LiDAR or radar sensor), a protein, a sequence of words (e.g., forming one or more sentences or paragraphs), a video (e.g., representative of a sequence of video frames), or any other suitable type of data, or a combination thereof.

[0045] In some implementations, an entity may include multiple units arranged in a spatial structure; for example, the entity may be an image and each unit may be a pixel in the image. Each unit, or data element, in the entity may have an associated data element embedding that may characterize, for example, the unit's position in the spatial structure and / or features associated with the unit in the spatial structure. Thus, an entity may have a spatial structure, and units or data elements may have associated positions in the spatial structure. The spatial structure may correspond to a physical spatial structure, for example, a pixel in an image, or to an abstract spatial structure, for example, a time sequence of audio samples. Exemplary entities and units are described in more detail below with reference to Figures 5A and 5B.

[0046] The neural network system 100 can be configured to process (i) a representation of an entity as a set of data element embeddings 104 and (ii) a set of potential embeddings 102 (e.g., randomly initialized) to generate a network output 150 that characterizes the entity. An exemplary process for generating data element embeddings 104 representing an entity is described in more detail below with reference to FIG. 4. The network output 150 can be, for example, a classification output, a regression output, a sequence output (i.e., including a sequence of output elements), a segmentation output, or any other suitable network output, or a combination thereof.

[0047] An "embedding" may generally refer to an ordered collection of numerical values, e.g., a vector, matrix, or other tensor of numerical values. A "data element embedding" may refer to an embedding of data elements associated with a particular unit in an entity. A "latent embedding" may refer to a predefined embedding in a potential space and / or a randomly initialized embedding. In general, data element embeddings and latent embeddings may have any suitable dimensionality. In some implementations, the dimensionality of a data element embedding may differ from the dimensionality of a latent embedding.

[0048] As will be described in more detail later, in some implementations, the number of potential embeddings 102 can be less than the number of data element embeddings 104. For example, if the entity is an image with dimensions of 224×224 pixels and the number of data element embeddings is M = 50176, the number of potential embeddings can be, for example, N = 512 such that N << M. Further, in some implementations, the number of potential embeddings 102 can be predefined and independent of the number of data element embeddings 104. For example, the potential embeddings 102 can be randomly initialized, for example, using a normal distribution. As a specific example, the potential embeddings can be initialized using a truncated normal distribution with a mean of 0, a standard deviation of 0.02, and truncation limits of [-2, 2]. As just another example of how the potential embeddings 102 can be obtained, these can include, for example, a learned set of weights where each weight defines an element of the "potential array" described later. In some implementations, the number of potential embeddings 102 can be a hyperparameter of the neural network system 100.

[0049] As previously described, the neural network system 100 can be configured to process the data element embeddings 104 and latent embeddings 102 representing an entity to generate a network output 150 that characterizes the entity. More specifically, the neural network system 100 can include a neural network 160 having a sequence of one or more neural network blocks. A “neural network block” can generally refer to a group of one or more neural network layers in a neural network. The sequence of neural network blocks can include (i) one or more mutual attention blocks 120, (ii) one or more self-attention blocks 130, e.g., following the mutual attention block 120, and (iii) an output block 140. In one example illustrated in FIG. 1 , the sequence of neural network blocks can include a first mutual attention block 120, followed by a first self-attention block 130, followed by a second mutual attention block 120, followed by a second self-attention block 130, followed by an output block 140. The neural network system 100 can use a sequence of neural network blocks to process the data element embeddings 104 and the latent embeddings 102 to generate a network output 150 that characterizes an entity.

[0050] The attention blocks (e.g., mutual attention block 120 and self-attention block 130) can be configured to perform attention operations, such as updating each embedding in the first set of embeddings using attention to the second set of embeddings. Generally, updating the first set of embeddings using attention to the second set of embeddings refers to updating the first set of embeddings by applying an attention mechanism to the second set of embeddings, and there are many different possible attention mechanisms that can be used. For example, for each target embedding in the first set of embeddings, each attention block can generate a respective attention weight for each embedding in the second set of embeddings and generate a combined embedding based on the second set of embeddings and the corresponding attention weights. As a particular example, each attention block can generate the combined embedding as a weighted sum of the second set of embeddings, for example, by multiplying each embedding in the second set of embeddings by its corresponding weight and summing the weighted embeddings. Each attention block can then use the combined embedding to update the target embedding in the first set of embeddings, for example, by replacing the target embedding with the combined embedding, by adding the combined embedding to the target embedding, or in any other suitable manner.

[0051] In some implementations, the attention block may perform a query-key-value (QKV) attention operation, e.g., update each embedding in the first set of embeddings using attention to the second set of embeddings using a query (Q) embedding, a key (K) embedding, and a value (V) embedding. Specifically, each attention block may include (i) a query subnetwork, (ii) a key subnetwork, and (iii) a value subnetwork. For each target embedding in the first set of embeddings, the query subnetwork may be configured to process the target embedding in the first set of embeddings to generate a respective query embedding (Q) for the target embedding. The key subnetwork may be configured to process each embedding in the second set of embeddings to generate a respective key embedding (K) for each embedding in the second set of embeddings. Similarly, the value subnetwork may be configured to process each embedding in the second set of embeddings to generate a respective value embedding (V) for each embedding in the second set of embeddings.

[0052] Each attention block can then use the query embedding (Q), key embedding (K), and value embedding (V) to update each target embedding in the first set of embeddings with respect to the second set of embeddings. Specifically, each attention block can generate attention weights for each embedding in the second set of embeddings, e.g., as the inner (e.g., dot) product of the query embedding (Q) and each key embedding in the key embedding (K). Based on the second set of embeddings and the attention weights, each attention block can generate combined embeddings, e.g., as a linear combination of the value embeddings (V) weighted by their respective attention weights. Finally, each attention block can use the combined embeddings to update the target embeddings in the first set of embeddings, e.g., by replacing the target embeddings in the first set of embeddings with a weighted sum of the value embeddings (V).

[0053] In some implementations, the first set of embeddings and the second set of embeddings may be different sets of embeddings. In such cases, the attention operation (e.g., the QKV attention operation) may be referred to as a “mutual attention” operation. The mutual attention operation may be performed, for example, by mutual attention block 120. For example, the first set of embeddings may be a set of potential embeddings 102, and the second set of embeddings may be data element embeddings 104, and mutual attention block 120 may update each potential embedding using mutual attention for some or all of the data element embeddings in the set of data element embeddings 104.

[0054] In some implementations, the first set of embeddings and the second set of embeddings may be the same set of embeddings. In such cases, the attention operation (e.g., the QKV attention operation) may be referred to as a “self-attention” operation. The self-attention operation may be performed, for example, by self-attention block 130. For example, the first set of embeddings may be a set of potential embeddings 102, the second set of embeddings may also be a set of potential embeddings, and self-attention block 130 may update each potential embedding in the set of potential embeddings 102 using self-attention on the set of potential embeddings. In some implementations, self-attention block 130 may iteratively update each potential embedding in the set of potential embeddings using self-attention on the set of potential embeddings.

[0055] In some implementations, the neural network 160 may further include one or more selection blocks 180. The selection block 180 may select a subset of data element embeddings from the set of data element embeddings 104 for the mutual attention block 120 to pay attention to. In other words, in some implementations, the mutual attention block 120 may update each potential embedding in the set of potential embeddings 102 using mutual attention on only the subset of data element embeddings selected by the selection block 180. An exemplary selection block 180 is described in more detail below with reference to FIG. 3.

[0056] In some implementations, the mutual attention block 120 and the self-attention block 130 can be configured to perform other operations in addition to the attention operations described above. For example, in addition to implementing one or more attention neural network layers, the attention block can also include any other neural network layers (e.g., convolutional layers, fully connected layers, recurrent layers, attention layers, etc.) of any suitable number (e.g., 2 layers, 5 layers, or 10 layers) connected in any suitable configuration (e.g., as a linear sequence of layers).

[0057] In some implementations, each attention block may further include one or more normalization neural network layers that may be configured to process the embeddings (e.g., data element embeddings 104 and / or latent embeddings 102) and to change the dimensionality of the embeddings. For example, if each data element embedding includes, e.g., C channels (components) and each latent embedding includes, e.g., D channels, one or more normalization layers may process the embeddings to generate outputs that include the data element embeddings and latent embeddings with the same number of channels, e.g., C channels.

[0058] In some implementations, each attention block may further include one or more neural network layers configured to combine inputs into each attention block with outputs from each respective attention block and normalize the combination. For example, one or more neural network layers may be configured to combine potential embedding inputs into the self-attention block 130 with potential embedding outputs from the self-attention block 130 (e.g., with potential embeddings that have been updated by the self-attention block 130 using the self-attention operations described above).

[0059] In addition to the mutual attention block 120 and the self-attention block 130, the neural network 160 may further include an output block 140. The output block 140 may process the output from the last attention block in the sequence of attention blocks (e.g., from the self-attention block 130 in FIG. 1 ) to generate a network output 150 that characterizes an entity. For example, the output block 140 may pool (i.e., combine, e.g., average-pool or max-pool) the latent embeddings included in the output to generate a pooled latent embedding, e.g., a global summary vector. The output block 140 may process the pooled latent embeddings using one or more neural network layers included in the output block 140 to generate the network output 150 that characterizes an entity. For example, a single linear neural network layer may project the global summary vector to a number of target classes or categories to provide a classification output.

[0060] In some implementations, the network output 150 characterizing an entity can have a sequence of output elements. In such cases, at each time step in the sequence of time steps, the output block 140 can process the output from the last attention block in the sequence of attention blocks and the output elements generated in any previous time steps to generate the output element for that time step.

[0061] As previously described, the neural network system 100 can update each potential embedding in the set of potential embeddings 102 using mutual attention for some or all of the data element embeddings in the set of data element embeddings 104. Each time the system 100 performs a mutual attention operation, the system 100 enriches the potential embeddings 102 with information from the data element embeddings 104. Because the number of potential embeddings 102 (e.g., N) is independent of the number of data element embeddings 104 (e.g., M), the computational complexity of the mutual attention operation is partially decoupled from the number of data element embeddings 104 and remains feasible even for a large number of data element embeddings 104. Thus, the neural network system 100 can process large, complex inputs while reducing the consumption of computational resources (e.g., memory and computing power).

[0062] Furthermore, the neural network system 100 can use self-attention on the set of potential embeddings to update the set of potential embeddings enriched with information from the data element embeddings, e.g., the self-attention operation is independent of the number of data element embeddings. For example, the mutual attention operation can have a complexity of about M N, and the self-attention operation can have a complexity of about N 2It is possible to have a complexity of, where N << M. Since the computational complexity of the self-attention operation is decoupled from the number of data element embeddings, the neural network system 100 can process a large amount of input and repeatedly execute the self-attention operation without significantly increasing the computational complexity. Therefore, the neural network system 100 can exhibit high prediction accuracy while reducing the consumption of computational resources (e.g., memory and computing power).

[0063] Generally, the neural network 160 can have any suitable neural network architecture that enables the neural network 160 to perform a defined function. For example, each of the neural network 160, as well as the cross-attention block 120, self-attention block 130, output block 140, and selection block 180, can have any suitable number (e.g., 2 layers, 5 layers, or 10 layers) of any suitable neural network layers (e.g., convolutional layers, fully-connected layers, recurrent layers, attention layers, etc.) connected in any suitable configuration (e.g., as a linear sequence of layers). Also, the neural network system 100 can further include any number of neural network blocks configured to perform any suitable operations.

[0064] Although the mutual attention blocks 120 and self-attention blocks 130 are shown as interleaved in FIG. 1 , the attention blocks may be arranged in any suitable configuration. For example, the system 100 may include a sequence of attention blocks having two mutual attention blocks 120, followed by two self-attention blocks 130, followed by two mutual attention blocks 120. In another example, the system 100 may include a sequence of attention blocks having one mutual attention block 120, followed by multiple self-attention blocks 130. In general, the system 100 may include any number of attention blocks 120, 130, and / or selection blocks 180 (e.g., 5, 10, 100, etc.) arranged in any suitable configuration.

[0065] Neural network system 100 may further include a training engine capable of training neural network 160 against a set of training data over multiple training iterations. The training data may include a set of training examples, where each training example specifies (i) a training input and (ii) a target output to be produced by neural network 160 by processing the training input.

[0066] In each training iteration, the training engine may sample a batch of training examples from the training data and process the training inputs specified by the training examples using a sequence of neural network blocks included in neural network 160 to generate corresponding network outputs. Specifically, for each training input, neural network 160 processes the training input using the current model parameter values ​​of the first attention block in the sequence (e.g., mutual attention block 120 in FIG. 1 ) to generate an output from the first attention block. Neural network 160 processes the output generated by the first attention block in the sequence using the current model parameter values ​​of the second attention block in the sequence (e.g., self-attention block 130 in FIG. 1 ) to generate an output from the second attention block in the sequence. Neural network 160 processes the output generated by the last attention block in the sequence (e.g., self-attention block 130 in FIG. 1) using the current model parameter values ​​of output block 140 to generate network outputs corresponding to the training inputs.

[0067] The training engine may adjust model parameter values ​​of attention blocks 120, 130 and output block 140, as well as values ​​for latent embeddings 102, in some implementations, to optimize an objective function that measures the similarity between (i) the network output produced by neural network 160 and (ii) a target network output specified by the training examples. The objective function may be, for example, a cross-entropy objective function, a squared error objective function, or any other suitable objective function.

[0068] The training engine may compute the gradient of the objective function using, for example, backpropagation techniques. The training engine may use the gradient to update the model parameter values ​​of the attention blocks 120, 130 and the output block 140 using, for example, any suitable gradient descent optimization algorithm, such as the Adam optimization algorithm. The training engine may compute a performance measure of the neural network 160 on a set of validation data that is not used during training of the neural network 160.

[0069] As mentioned above, in some implementations, the neural network system 100 can further include one or more selection blocks 180. The training engine can train the one or more selection blocks 180 using reinforcement learning techniques, as described in more detail below with reference to Figure 3. After training, the neural network system 100 can be used to perform machine learning tasks, for example, to process inputs and generate outputs that characterize entities.

[0070] Neural network system 100 can be configured to perform any suitable machine learning task, some examples of which are described below.

[0071] In some implementations, the neural network system 100 can process the set of data element embeddings 104 representing the pixels of the image to generate a classification output 150 that includes a respective score for each object category (e.g., vehicle, pedestrian, bicyclist, etc.) in a set of possible object categories. The score for an object category can define the likelihood that the image depicts an object belonging to that object category.

[0072] In some implementations, the system 100 is capable of processing a set of data element embeddings 104 representing audio samples in an audio waveform to perform speech recognition, i.e., to generate an output 150 that defines a sequence of phonemes, graphemes, characters, or words that correspond to the audio waveform.

[0073] In some implementations, the system 100 can process a set of data element embeddings 104 representing words in a word sequence to perform a natural language processing task, such as topic classification or summarization. To perform topic classification, the system 100 can generate a network output 150 including a respective score for each topic category (e.g., sports, business, science, etc.) in a set of possible categories. The score for the topic category can define the likelihood that the word sequence is related to the topic category. To perform summarization, the system can generate a network output 150 including an output sequence of words that has a shorter length than the input sequence of words and captures important or relevant information from the input sequence of words.

[0074] In some implementations, the system 100 is capable of performing a neural machine translation task, for example, processing a set of data element embeddings 104 representing sequences of text, e.g., words, phrases, characters, or word fragments in one language, to generate a network output 150, which may be a translation of the sequence of text into another language, i.e., a sequence of text in the other language that is a translation of the input sequence of text. As a particular example, the task may be a multilingual machine translation task in which the system 100 is configured to translate between a number of different source language-target language pairs. In this example, the source language text may be augmented with an identifier indicating the target language into which the neural network should translate the source language text.

[0075] In some implementations, the system 100 is capable of performing audio processing tasks. For example, if the data element embeddings 104 represent verbal speech, the output 150 generated by the system 100 may be a score for each fragment of text of the set of text fragments, where each score represents an estimated likelihood that the fragment of text is a correct transcription of the utterance. As another example, if the data element embeddings 104 represent verbal speech, the output 150 generated by the system 100 may indicate whether a particular word or phrase (a "hot word") was uttered in the utterance. As another example, if the data element embeddings 104 represent verbal speech, the output 150 generated by the system 100 may identify the natural language in which the utterance was spoken.

[0076] In some implementations, the system 100 is capable of performing natural language processing or natural language understanding tasks, such as entailment tasks, paraphrasing tasks, text similarity tasks, sentiment tasks, sentence completion tasks, grammaticality tasks, etc., that operate on sets of data element embeddings 104 that represent text in several natural languages.

[0077] In some implementations, the system 100 is capable of performing text-to-speech tasks in which the data element embeddings 104 represent text in a natural language or features of text in a natural language, and the network output 150 is a spectrogram, waveform, or other data that defines the audio of the spoken text in the natural language.

[0078] In some implementations, the system 100 is capable of performing health prediction tasks where the data element embeddings 104 represent data derived from electronic health record data about a patient and the output 150 is a prediction related to the patient's future health, such as a predicted treatment to be prescribed for the patient, the likelihood that the patient will experience an adverse health event, or a predicted diagnosis for the patient.

[0079] In some implementations, the system 100 is capable of performing a text generation task where the data element embeddings 104 represent a sequence of text and the output 150 is another sequence of text, e.g., a completion of an input sequence of text, a response to a question posed in the input sequence, or a sequence of text about a topic specified by the first sequence of text. As another example, the data element embeddings 104 can represent data other than text, e.g., an image, and the output sequence 150 can be text that describes the data represented by the data element embeddings 104.

[0080] In some implementations, the system 100 is capable of performing an image generation task where the data element embeddings 104 represent conditioned inputs and the output 150 is a sequence of intensity value inputs for pixels of the image.

[0081] In some implementations, the system 100 is capable of performing agent control tasks where the data element embeddings 104 represent a sequence of one or more observations or other data characterizing the state of an environment, and the output 150 defines an action to be performed by an agent in response to the most recent data in the sequence. The agent can be, for example, a real-world or simulated robot, a control system for an industrial facility, or a control system that controls different types of agents.

[0082] In some implementations, the system 100 is capable of performing genomics tasks in which the data element embeddings 104 represent fragments of DNA sequences or other molecular sequences, and the output 150 is an embedding of the fragment for use in, or output for, a downstream task, for example, by utilizing unsupervised learning techniques on a dataset of DNA sequence fragments. Examples of downstream tasks include promoter site prediction, methylation analysis, predicting the functional effect of non-coding variants, etc.

[0083] In some implementations, the system 100 may perform a protein modeling task, for example, where the data element embeddings 104 represent proteins and the network output 150 characterizes the proteins. For example, the network output 150 may characterize the predicted stability of the proteins or the predicted structure of the proteins.

[0084] In some implementations, the system 100 is capable of performing point cloud processing tasks, for example, where the data element embeddings 104 represent a point cloud (e.g., generated by a LiDAR or radar sensor) and the network output 150 characterizes the type of object represented by the point cloud.

[0085] In some implementations, system 100 is capable of performing a combination of multiple individual machine learning tasks, i.e., system 100 is configured to perform multiple different individual machine learning tasks, such as two or more of the machine learning tasks described above. For example, system 100 can be configured to perform multiple individual natural language understanding tasks, and the data element embeddings 104 processed by the neural network include identifiers for the individual natural language understanding tasks to be performed on the data element embeddings.

[0086] Neural network system 100 is described in more detail below with reference to FIG.

[0087] 2 is a block diagram of a more detailed example neural network system 200 (e.g., neural network system 100 in FIG. 1 ). Neural network system 200 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0088] The neural network system 200 can include a sequence of neural network blocks, such as one or more attention blocks, an output block, and, optionally, one or more selection blocks. A particular example is shown in FIG. 2, in which the neural network system 200 includes a first mutual attention block 220a, followed by a first self-attention block 230a, followed by a second mutual attention block 220b, followed by a second self-attention block 230b, followed by an output block 240.

[0089] As described above with reference to FIG. 1, the neural network system 200 can be configured to process a set of data element embeddings 204 representing an entity (e.g., an image) and a set of latent embeddings 202 (e.g., randomly initialized) to generate a network output characterizing the entity. The set of data element embeddings 204 (e.g., a "byte array") can have dimensions of M×C, where M is the number of data element embeddings and C is the number of channels in each data element embedding. For example, if the entity includes an image, M can be the number of pixels in the image and C can be the number of channels per pixel. The set of latent embeddings 202 (e.g., a "latent array") can have dimensions of N×D, where N is the number of latent embeddings and D is the number of channels in each latent embedding. In some implementations, N is predefined and independent of M. In some implementations, N << M. In some implementations, N and D are hyperparameters that can be selected according to available computing resources.

[0090] Each attention block can be configured to update each embedding in the first set of embeddings using attention for some or all of the embeddings in the second set of embeddings. Specifically, each cross-attention block 220a, 220b can be configured to generate a cross-attention output 206 by updating each latent embedding in the set of latent embeddings 202 using cross-attention for some or all of the data element embeddings in the set of data element embeddings 204. Similarly, each self-attention block 230a, 230b can be configured to generate a self-attention output 208 by updating each latent embedding in the set of latent embeddings 202 using self-attention for the set of latent embeddings.

[0091] As described above with reference to Figure 1, the mutual attention and self-attention operations can be implemented as query-key-value (QKV) attention. For example, each of the attention blocks can use a query sub-network to generate a query embedding ("Q"), a key sub-network to generate a key embedding ("K"), and a value sub-network to generate a value embedding ("V"). The mutual attention blocks 220a, 220b can use the query embedding, key embedding, and value embedding to perform the mutual attention operations, and the self-attention blocks 230a, 230b can use the query embedding, key embedding, and value embedding to perform the self-attention operations.

[0092] Output block 240 may receive the output from the last attention block in the sequence (e.g., self-attention output 208 from self-attention block 230b) and process the output to generate a network output that characterizes the entity. Specifically, output block 240 may include one or more neural network layers (e.g., "mean" in FIG. 2) configured to pool the latent embeddings to generate a pooled latent embedding. Output block 240 may process the pooled latent embeddings using one or more additional neural network layers to generate an output that characterizes the entity.

[0093] As described above with reference to FIG. 1 , the attention blocks 220, 230, and the output block 240 can each have a respective set of model parameters that can be trained by a training engine. In some implementations, the neural network system 200 can share model parameter values ​​between different neural network blocks. For example, the system 200 can share model parameter values ​​between the first mutual attention block 220 a and the second mutual attention block 220 b. Similarly, the system 200 can share model parameter values ​​between the first self-attention block 230 a and the second self-attention block 230 b. In some implementations, the system 200 can refrain from sharing model parameter values ​​between the first mutual attention block 220 a and any other neural network blocks in the system 200.

[0094] Sharing model parameter values ​​among different neural network blocks can improve the performance of the trained neural network system 200, for example, by reducing the likelihood of overfitting and, in some cases, by reducing the total number of model parameters in the system 200. As a result, the system 200 may require less training data and / or fewer training iterations to achieve a threshold level of performance (e.g., predictive accuracy).

[0095] As mentioned above, in some implementations, the neural network system 200 may further include one or more selection blocks (e.g., selection block 180 in FIG. 1). Each selection block may be configured to select a subset of data element embeddings from the set of data element embeddings 204 for some or all of the mutual attention blocks (e.g., blocks 220a, 220b) to pay attention to. In such cases, the mutual attention block may update each potential embedding in the set of potential embeddings 202 using mutual attention on only the subset of data element embeddings selected by the selection block.

[0096] As a particular example, in some implementations, an entity may be a one-second-long video with an image resolution of 224x224 pixels with 24 frames per second. In such a case, the number of data element embeddings may be quite large, e.g., approximately 1.2 million. If it is desired to reduce the number of data element embeddings, the system may subsample or generate embeddings of video patches that extend across space and / or time. In some implementations, the system 200 may use a selection block to adaptively select a subset of data element embeddings from the approximately 1.2 million data element embeddings for the mutual attention block to pay attention to. An exemplary selection block is described in more detail below.

[0097] 3 is a block diagram of an example selection block 300 included in a neural network system (e.g., neural network system 100 in FIG. 1 or neural network system 200 in FIG. 2) used to generate a network output that characterizes an entity. Selection block 300 is an example of a system implemented as a computer program on one or more computers at one or more locations where the systems, components, and techniques described below are implemented.

[0098] The selection block 300 may be configured to process a set of data element embeddings 304 representing entities (e.g., images) and a set of potential embeddings 302 (e.g., randomly initialized) to select an appropriate subset of data element embeddings from the set of data element embeddings 304. The neural network system may provide the subset of data element embeddings to one or more mutual attention blocks to pay attention to. Specifically, the one or more mutual attention blocks may update each potential embedding in the set of potential embeddings using mutual attention on only the subset of data element embeddings selected by the selection block 300.

[0099] In some implementations, the system may use only one selection block to select a subset of data element embeddings and provide that subset to one or more mutual attention blocks for attention. In some implementations, the system may use multiple selection blocks to select a subset of data element embeddings. For example, each selection block may select a different subset of data element embeddings from the set of data element embeddings. In some implementations, the system may provide each subset selected by a respective selection block to a respective mutual attention block for attention, for example, several mutual attention blocks may pay attention to different subsets of data element embeddings. The selection block 300 may select any number of data element embeddings, for example, less than 50%, less than 25%, less than 10%, or less than 5% of the complete set of data element embeddings 304.

[0100] The selection block 300 can select a subset of data element embeddings by using (i) a parameter selection neural network 310 and (ii) a unit selection neural network 320, each of which is described in more detail below.

[0101] After the neural network system updates the set of potential embeddings 302 using one or more mutual attention blocks, one or more self-attention blocks, or both (e.g., as described above with reference to FIGS. 1 and 2 ), the parameter selection neural network 310 may process the potential embeddings 302 to generate a network output that defines values ​​for unit selection neural network parameters 360. For example, in some implementations, the unit selection neural network 320 may include one or more fully connected neural network layers, each of which may have a corresponding tensor (e.g., a matrix) of unit selection neural network parameters. Thus, the output 360 from the parameter selection neural network 310 may include, for example, an ordered collection of numerical values ​​that define the parameter values ​​for these fully connected neural network layers of the unit selection neural network 320.

[0102] The unit selection neural network 320 may process each data element embedding in the set of data element embeddings 304 according to the values ​​of the unit selection neural network parameters (e.g., generated by the parameter selection neural network 310) to generate a selection score 380 for each data element embedding 304. In one example, the selection score 380 for each data element embedding may be a one-dimensional value, such as a reinforcement learning Q-value.

[0103] Based on the selection scores 308, the selection block 300 may select a subset of data element embeddings from the complete set of data element embeddings 304. For example, the selection block may select one or more data element embeddings with the highest selection scores from the set of data element embeddings 304. In another example, the selection block 300 may select a predefined subset of data element embeddings from the set of data element embeddings 304, for example, the 5% of data element embeddings with the highest selection scores.

[0104] Thus, the selection block 300 may adaptively select an appropriate subset of data element embeddings from the complete set of data element embeddings 304. For example, if the entity is a one second long video with an image resolution of 224x224 pixels, the selection block 300 may select a subset of data element embeddings that corresponds to, for example, the maximum value of 10,000 pixels of the video.

[0105] As described above with reference to FIG. 1 , the training engine may adjust model parameter values ​​of one or more attention blocks and an output block, for example, using supervised learning techniques. The training engine may further train one or more selection blocks, for example, using reinforcement learning techniques. For example, the training engine may train the selection block 300 by iteratively adjusting model parameter values ​​of the selection block 300 by iteratively backpropagating the gradient of a reinforcement learning objective function through the selection block 300. The reinforcement learning function may be, for example, a squared Bellman error objective function or any other suitable reinforcement learning objective function. In some implementations, the training engine may compute an overall loss of the neural network (e.g., of one or more attention blocks, an output block, and a selection block) as a linear combination of a supervised loss and a reinforcement learning loss.

[0106] To train the selection block 300, in each training iteration, the training engine may use a neural network to process the set of data element embeddings and the set of potential embeddings to generate a network output that characterizes the entity (e.g., as described above with reference to FIG. 1). The training engine may compute task performance measures based on the network output that characterizes the entity. For example, the training engine may evaluate the same objective function used by the training engine to train the attention block and the output block, as described above with reference to FIG. 1. In other words, the training engine may train the selection block 300 directly on the same classification signal used to train the rest of the neural network, as described above with reference to FIG. 1.

[0107] Then, at each training iteration, the training engine can calculate a reward based on the task performance measure. The reward can be any suitable function of the task performance measure. In another example, the task performance measure can characterize the prediction error (e.g., by cross-entropy loss), and the reward can be the negative of the task performance measure, such that a lower prediction error results in a higher reward.

[0108] Based on the reward, the training engine can train the selection block 300 against a reinforcement learning objective function that depends on the reward. The reinforcement learning objective function can encourage the selection of data element embeddings that result in an increase in the reward received by the neural network.

[0109] As a specific example, the training engine may use a squared Bellman error objective function, E[(RQ(x i ;φ(z l ))) 2 ] (1) It is possible to train the selection block 300 by minimizing Q(xi ;·) is the data element embedded x i is the output of the selection block 300 for φ(z l ) is the output from the lth self-attention block, and R is a task performance measure (e.g., negative cross-entropy classification error).

[0110] After training the selection block 300, the neural network system can use the selection block 300 to select a subset of data element embeddings from the complete set of data element embeddings 304. The system can provide the subset of data element embeddings to one or more mutual attention blocks to pay attention to. Rather than updating each potential embedding using mutual attention on the complete set of data element embeddings, the one or more mutual attention blocks can update each potential embedding on only the subset of data element embeddings adaptively selected by the selection block 300. As a result, the system can reduce the amount of computational resources required to perform mutual attention operations on the data element embeddings while maintaining acceptable task performance (e.g., prediction accuracy).

[0111] An exemplary process for using a neural network system to generate a network output that characterizes an entity is now described in more detail.

[0112] 4 is a flowchart of an exemplary process 400 for using a neural network system to characterize an entity. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as neural network system 100 in FIG. 1 or neural network system 200 in FIG. 2, can perform process 400.

[0113] The system obtains 402 a representation of the entity as a set of data element embeddings. The entity may include, for example, a number of units arranged in a spatial structure (as a one-dimensional (1D) array, a two-dimensional (2D) array, or a three-dimensional (3D) array of units), where each unit is associated with position data that defines the unit's respective position in the spatial structure. For example, the entity may include an image, where each pixel in the image may define a respective unit in the entity. In another example, the entity may include an audio waveform, where each audio sample in the audio waveform, or, for example, in a mel spectrogram of the audio waveform, may define a respective unit in the entity. In yet another example, the entity may include a point cloud, where each point in the point cloud may define a respective unit in the entity. In yet another example, the entity may include a protein, where each amino acid in the amino acid sequence of the protein may define a respective unit in the entity. In yet another example, an entity may include a sequence of words, with each word in the sequence of words defining a respective unit in the entity. Exemplary entities and units are described in more detail below with reference to Figures 5A and 5B.

[0114] Obtaining the representation of the entity as a set of data element embeddings may include, for each unit in the entity, generating a data element embedding for that unit. For each unit in the entity, the system may generate a data element embedding based on the feature embedding and the positional embedding for that unit.

[0115] The system can generate a feature embedding based on the features of the unit. For example, if the entity is an image (e.g., image 520 in FIG. 5) and the unit is a pixel in the image (e.g., pixel 525 in FIG. 5), the system can obtain a feature embedding by selecting a patch of the image around the pixel and concatenating the patches into a vector. In another example, if the entity is an audio waveform (e.g., audio waveform 540 in FIG. 5) and the unit is an audio sample (e.g., audio sample 545 in FIG. 5), the system can obtain a feature embedding at a time point by selecting the amplitude of the audio sample at that time point.

[0116] The system can generate a location embedding based on the location of the unit in the spatial structure. For example, if the spatial structure of an entity is one-dimensional, e.g., if the entity is a sequence of words (e.g., word sequence 530 in FIG. 5) and the unit is a word in the sequence (e.g., word 535 in FIG. 5), the system can generate a location embedding based on the index of the word in the sequence of words. In another example, if the entity is a two-dimensional array of pixels, the system can generate a location embedding based on the x- and y-coordinates of the pixels in the array of pixels. In yet another example, if the entity is a point cloud (e.g., point cloud 510 in FIG. 5) and the unit is a point in the point cloud (e.g., point 515 in FIG. 5), the system can generate a location embedding based on the x-, y-, and z-coordinates of the point in the point cloud.

[0117] In general, for each unit in the entity, the system can generate a location embedding as any suitable function of the unit's location in the spatial structure. In some implementations, the location embedding can be, for example, a Fourier feature location encoding with frequency bands spaced logarithmically across a predefined target frequency range. In general, the Fourier feature location encoding can be a coding that maps input coordinates in one or more dimensions to Fourier coefficients of a Fourier series with frequencies (“bands”) spaced logarithmically (linearly on a logarithmic scale) across a predefined target frequency range. In some other implementations, the location encoding can be fixed or learned. The location embedding of units is described in more detail below with reference to Figures 5A and 5B.

[0118] After obtaining the feature embeddings and location embeddings, the system can generate a corresponding data element embedding for each unit in the entity, for example, by concatenating the feature embeddings of that unit with the location embedding of that unit. In some other implementations, the corresponding data element embedding may be generated by adding the feature embeddings of the unit with the location embedding of that unit.

[0119] The system obtains a set of potential embeddings 404. In some implementations, the number of potential embeddings in the set of potential embeddings may be less than the number of data element embeddings in the set of data element embeddings. In some implementations, the number of potential embeddings in the set of potential embeddings may be predefined and independent of the number of data element embeddings in the set of data element embeddings.

[0120] The system processes (406) (i) the set of data element embeddings and (ii) the set of potential embeddings using a neural network to generate a network output that characterizes the entity.

[0121] The system may include a sequence of neural network blocks having (i) one or more mutual attention blocks, (ii) one or more self-attention blocks, and (iii) an output block. For example, the system may include multiple mutual attention blocks and multiple self-attention blocks, where the mutual attention blocks and the self-attention blocks are interleaved.

[0122] The sequence of neural network blocks may further include one or more selection blocks, each configured to select an appropriate subset of data element embeddings from the complete set of data element embeddings (i.e., a subset of the set that does not include the set itself). For example, after the set of potential embeddings has been updated using one or more mutual attention blocks, one or more self-attention blocks, or both, the selection block may process the set of potential embeddings and the set of data element embeddings to generate a respective selection score for each data element embedding. Based on the selection scores, the selection block may select an appropriate subset of the set of data element embeddings (e.g., a predefined number of data element embeddings with the highest selection scores) for use by one or more designated mutual attention blocks. Each designated mutual attention block may, for example, update each potential embedding in the set of potential embeddings using mutual attention to only the data element embeddings in the selected appropriate subset of the set of data element embeddings, instead of the complete set of data element embeddings.

[0123] Each selection block may include (i) a parameter selection neural network and (ii) a unit selection neural network. For each selection block, processing the set of potential embeddings and the set of data element embeddings to generate a respective selection score for each data element embedding in the set of data element embeddings may include processing the potential embeddings with the parameter selection neural network to generate network outputs defining values ​​for a set of neural network parameters of the unit selection neural network, and processing each data element embedding in the set of data element embeddings with the unit selection neural network to generate a selection score for the data element embedding according to the values ​​of the set of neural network parameters of the unit selection neural network.

[0124] In some implementations, the system is capable of computing a task performance measure (e.g., cross-entropy classification error) based on network outputs characterizing the entities, computing a reward based on the task performance measure, and training a selection block against a reinforcement learning objective function (e.g., squared Bellman error) that depends on the reward.

[0125] Each mutual attention block may update each potential embedding in the set of potential embeddings using attention for some or all of the data element embeddings in the set of data element embeddings. This may include, for example, generating a respective query embedding for each potential embedding in the set of potential embeddings, generating a respective key embedding and a respective value embedding for each of multiple data element embeddings in the set of data element embeddings, and updating each potential embedding in the set of potential embeddings using query-key-value attention for some or all of the data element embeddings in the set of data element embeddings, including updating each potential embedding in the set of potential embeddings using query-key-value attention for multiple data element embeddings in the set of data element embeddings based on (i) the query embedding for the potential embedding, and (ii) the key embedding and value embedding for the data element embedding.

[0126] Each self-attention block may update (e.g., iteratively) each potential embedding in the set of potential embeddings using attention for the set of potential embeddings. This may include, for example, updating each potential embedding in the set of potential embeddings using query-key-value attention for the set of potential embeddings.

[0127] After the set of latent embeddings has been updated using one or more mutual attention blocks and one or more self-attention blocks, the output block can process one or more latent embeddings from the set of latent embeddings to generate a network output that characterizes the entity. This can include pooling, e.g., averaging, the latent embeddings in the set of latent embeddings to generate a pooled latent embedding, and processing the pooled latent embedding using one or more neural network layers to generate a network output that characterizes the entity.

[0128] In some implementations, the network output can include a sequence of output elements. In such cases, processing one or more potential embeddings from the set of potential embeddings by the output block to generate a network output that characterizes the entity can include, at each time step of multiple time steps, processing (i) one or more potential embeddings from the set of potential embeddings and (ii) the output elements generated at any previous time steps to generate an output element for that time step.

[0129] Exemplary entities and units are described in more detail below.

[0130] Figure 5A shows example entities and units 500 that can be characterized by a neural network system (e.g., system 100 in Figure 1 or system 200 in Figure 2). Although only two types of entities are shown in Figure 5A, the neural network system can be used to characterize entities of any suitable type or modality.

[0131] As previously mentioned, a neural network can obtain and process a representation of an entity as a set of data element embeddings (e.g., together with latent embeddings) to generate a network output that characterizes the entity. An entity can include multiple units arranged in a spatial structure, and each unit in the entity can be associated with position data that defines the unit's respective position in the spatial structure. The spatial structure can be, for example, a one-dimensional (1D), two-dimensional (2D), or three-dimensional (3D) array of units.

[0132] 5A , in one example, the entity may include an image 520, and each pixel 525 in the image 520 may define a respective unit in the entity, e.g., the pixels 525 in the image 520 may be arranged as a two-dimensional array of units. In another example, the entity may include a point cloud 510, and each point 515 in the point cloud 510 may define a respective unit in the entity, e.g., the points 515 in the point cloud 510 may be arranged as a three-dimensional array of units. The pixels 525 and points 515 may be associated with position data (e.g., coordinates defining the respective positions of the pixels 525 and point cloud 515 in their respective spatial structures).

[0133] In some implementations, the neural network system can associate location and modality specific features with each unit 515, 525 in the entity 510, 520. For example, for each unit in the entity, the neural network system can generate a feature embedding for the unit based on the features of the unit and a location embedding for the unit based on the location of the unit in the spatial structure. In some implementations, the system can generate the location embedding by generating a Fourier feature location encoding with frequency bands spaced log-linearly across a predefined target frequency range. For example, the Fourier encoding can be: [sin(f k πx d ),cos(f k πx d )] (2) where frequency f k is the kth band of a bank of equally spaced frequencies between 1 and μ / 2, where μ / 2 can be, for example, the Nyquist frequency corresponding to the target sampling rate of μ, and x dis the value of units along the d-th dimension of the entity (e.g., d=2 for images and d=3 for videos). d can have values ​​of [-1, 1] for each dimension in the entity. In some implementations, the system uses the raw position values ​​x to generate the final representation of the position. d can be concatenated to yield a positional encoding of size d(2K+1). Thus, for each unit in the entity, the neural network system can generate a corresponding data element embedding for that unit based on the feature embedding and the positional embedding, e.g., by concatenating the feature embedding and the positional embedding.

[0134] As described above with reference to FIGS. 1 and 2, neural network systems are capable of processing data element embeddings and potential embeddings that represent entities using attention operations. Attention operations do not require assuming that data element embeddings are associated with a fixed spatial arrangement. For example, attention operations do not rely on assuming that data element embeddings are associated with a spatial arrangement, such as in a two-dimensional array of image pixels 525. Rather, neural network systems can flexibly incorporate information about the spatial arrangement of data element embeddings by tagging (e.g., concatenating) the data element embeddings with a positional coding and allowing attention operations to learn to utilize this information where relevant to generating accurate network outputs. Thus, neural network systems can be used to process sets of data element embeddings that are not associated with a predefined spatial arrangement, such as sets of data elements representing point clouds 510 or images 520, thereby making the system more widely applicable.

[0135] Figure 5B shows another example 500 of entities and units that can be characterized by a neural network system (e.g., system 100 in Figure 1 or system 200 in Figure 2). Although only three types of entities are shown in Figure 5B, the neural network system can be used to characterize entities of any suitable type, e.g., any suitable modality.

[0136] As shown in FIG. 5B , in one example, an entity may include a sequence of words 530, where each word 535 in the sequence of words 530 may define a respective unit in the entity. In another example, an entity may include an audio waveform 540, where each audio sample 545 in the audio waveform 540 may define a respective unit in the entity. In yet another example, an entity may include a protein 550, where each amino acid 555 in the amino acid sequence of the protein 550 may define a respective unit in the entity. In some implementations, an entity may include a mixture of different modalities. For example, an entity may include a video including an image 520, as shown in FIG. 5A , and an audio waveform 540, as shown in FIG. 5B .

[0137] Implementations of the systems described herein are capable of processing multimodal data for multimodal entities. That is, as previously described, the characterized entities may include combinations of different types of data, such as image or video data and audio data, image or video data and language data, somatosensory input data (sensor data that senses a physical agent's real-world environment, such as sensing touch data, pressure data, motion data, temperature data, or vibration data) and motor feedback data (i.e., control data that controls the movement of a physical agent). In the case of multimodal data, each type or domain of data may use a different positional embedding. In particular, each different positional embedding may have the correct dimensionality for that type of data, e.g., 3D for video data or point cloud data, 2D for image data, and 1D for audio data. The positional embedding may be a Fourier feature positional embedding, or a fixed positional embedding, or a learned positional embedding. In the case of multimodal data, each type or domain of data may be associated with one or more modality-specific features, i.e., embeddings. These may be fixed or learned, and because they are modality specific, they can be used by the system to identify modalities. Thus, units of a multimodal entity may be tagged with both location and modality-specific features (embeddings).

[0138] When a multimodal entity is processed by the system, units or data elements of different modalities may be combined. More specifically, data element embeddings for different modalities may be combined, for example, by fusing byte arrays for the different modalities into a combined byte array having the same number of channels for each modality, for example, by concatenating a learned modality-specific encoding to each data element embedding. In some implementations, modality-specific encodings may be combined with positional encodings.

[0139] The network output for a multimodal entity may be as described above. For example, if the network output is a classification output for a classification task (e.g., defining a score for each category of a set of possible categories), it may be the same as described above, except that the network output is generated based on the multimodal data embeddings provided as input. As such, the machine learning task performed by the system, e.g., classification, may be performed better, e.g., more accurately. For example, a classification task may be performed on a combination of video and (corresponding) audio data to obtain a more accurate classification result. As another example, the machine learning task may be based on processing data of different modalities, e.g., in a task of combining video or image data with language data, e.g., textual data, to determine whether an image or video is described by a particular caption.

[0140] 6A shows an example attention map 600 generated by a neural network system capable of characterizing entities (e.g., system 100 in FIG. 1 or system 200 in FIG. 2). In this example, the neural network system includes eight mutual attention blocks.

[0141] The first image on the left is the original image (e.g., entity) characterized by the neural network system. The second image is the attention map generated by the first mutual attention block of the neural network system. The third image is the attention map generated by the second mutual attention block of the neural network system. The last image is the attention map generated by the eighth mutual attention block of the neural network system.

[0142] 6B shows an example attention map 600 generated by a neural network system capable of characterizing an entity (e.g., system 100 in FIG. 1 or system 200 in FIG. 2). In this example, the entity being characterized is the first image in FIG. 6A, and the neural network system also includes eight mutual attention blocks.

[0143] The top panel is an overview of the attention map generated by the first mutual attention block of the neural network system. The middle panel is an overview of the attention map generated by the second mutual attention block of the neural network system. The bottom panel is an overview of the attention map generated by the eighth mutual attention block of the neural network system. The attention map is capable of scanning the input image using a tartan-like pattern across a range of spatial frequencies.

[0144] FIG. 7 illustrates exemplary performance of different configurations of a neural network system 700 (eg, system 100 in FIG. 1 or system 200 in FIG. 2) capable of characterizing entities.

[0145] Specifically, FIG. 7 shows the performance of the neural network system as a function of the number of mutual attentions and the respective placement of the mutual attention blocks relative to the other neural network blocks in the neural network system.

[0146] In "interleaved," the mutual attention layers are spaced throughout the network (due to reentrant processing), while in "initial," all mutual attention is placed at the beginning of the network, followed by all potential self-attention layers. All mutual attention layers except the first are shared, and self-attention is also shared (e.g., using eight blocks of six self-attention modules). The results are top-1 validation accuracy (in %) on ImageNet (higher is better).

[0147] 8A shows example parameters of a neural network system 800 (e.g., system 100 in FIG. 1 or system 200 in FIG. 2) capable of characterizing entities. All plots show top-1 accuracy (higher is better).

[0148] Specifically, Figure 8A shows the effect of model hyperparameters on the performance of a neural network system. Increasing the number of potential embeddings, the number of self-attentions per block, and the number of mutual attentions generally improves the performance of the neural network system. In some cases, increasing the number of channels in each potential embedding can also improve the performance of the neural network system.

[0149] 8B shows another example of parameters of a neural network system 800 (e.g., system 100 in FIG. 1 or system 200 in FIG. 2) that can characterize an entity. All plots show top-1 accuracy (higher is better).

[0150] Specifically, Figure 8B shows the effect of the latent embedding initialization scale and Fourier feature position-coding parameters on the performance of the neural network system. In general, increasing the number of bands and maximum resolution (up to Nyquist) improved performance. In some cases, the same effect can be observed whether linearly or logarithmically spaced position-coding bands are used.

[0151] 9 shows experimental results achieved using the neural network system described herein. In particular, table 910 shows the top-1 validation accuracy (in %) of the neural network system described herein (e.g., "Perceiver") and alternative neural network systems. It can be appreciated that the neural network system described herein provides significantly better performance than alternative systems without relying on domain-specific architectural assumptions. Table 920 also shows the top-1 validation accuracy. It will be appreciated that the neural network system described herein provides significantly better performance than alternative systems while using learned positional encoding or Fourier features.

[0152] 10 shows experimental results achieved using the neural network system described herein. In particular, table 1010 shows the top-1 classification accuracy (in %) of the neural network system described herein (e.g., "Perceiver") and an alternative neural network system. Table 1020 shows the performance of the neural network system described herein for video and audio-only experiments. It will be appreciated that the neural network system described herein significantly outperforms the alternative neural network systems.

[0153] This specification uses the term "configured" in connection with system components and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when activated, causes the system to perform that operation or action. One or more computer programs configured to perform a particular operation or action means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform that operation or action.

[0154] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, or in computer hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or a combination of one or more of these. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to an appropriate receiving device for execution by the data processing apparatus.

[0155] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or multiple computers. The apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations of these.

[0156] A computer program, which may be called or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or other data, for example, in one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperative files, for example, files storing one or more modules, subprograms, or portions of code. A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and connected together by a data communications network.

[0157] The term "engine" is used herein to broadly refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and running on the same computer or computers.

[0158] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., FPGAs or ASICs, or a combination of special purpose logic circuitry and one or more programmed computers.

[0159] A computer suitable for running a computer program may be based on a general-purpose or specialized microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for executing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated in special-purpose logic circuitry. Typically, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to such mass storage devices, or both. However, a computer need not have such devices. Additionally, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0160] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, non-volatile media, and non-volatile memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0161] To enable interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) monitor or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to enable interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, voice input, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0162] A data processing device for implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit for handling the typical, computationally intensive part of machine learning training or machine learning production, i.e., the inference, workload.

[0163] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0164] Embodiments of the subject matter described herein can be implemented in a computing system including back-end components, e.g., as a data server, or in a computing system including middleware components, e.g., as an application server, or in a computing system including front-end components, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be connected to each other by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0165] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between clients and servers arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to user devices, e.g., for the purpose of displaying the data to and receiving user input from users interacting with the devices acting as clients. Data generated at the user devices, e.g., results of user interactions, may be received from the devices at the server.

[0166] While the specification contains many specific implementation details, these should not be construed as limitations on the scope of the invention or on the scope to which the claims may be directed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Also, some features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as acting in a certain combination, and may even initially be claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and the claimed combination may be directed to a subcombination or a variation of a subcombination.

[0167] Similarly, although operations may be depicted in the figures or recited in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged as multiple software products.

[0168] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, nor a sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0169] 100, 200, 700 Neural Network System 102 Latent Embedding 104 Data Element Embedding 120 Mutual Attention Block 130 Self-Attention Block 140, 240 output blocks 150 Network output, classification output, output, output sequence 160 Neural Networks 180 Selection Block 202 Latent Embedding 204 Data Element Embedding 206 Mutual attention output 208 Self-attention output 220a First Interactive Attention Block 220b Second Interactive Attention Block 230a First self-attention block 230b Second Self-Attention Block 302 Potential Embedding 304 Data Element Embedding 500 Entities and Units 510 Point Clouds, Entities 515 points, unit 520 images, entities 525 pixels, units 530 word sequence 535 words 540 Audio Waveforms 545 audio samples 550 Protein 555 amino acids 600 Caution Map

Claims

1. A method executed by one or more data processing devices for using a neural network to generate network outputs characterizing a multimodal entity to perform a multimodal processing task, the multimodal entity comprising a combination of at least two different types of data, the method comprising: obtaining a representation of the multimodal entity as a set of data element embeddings that combine data element embeddings of the at least two different types of data, the at least two different types of data comprising one or more of video data, image data, point cloud data, audio data, or text data; obtaining a set of potential embeddings; processing (i) the set of data element embeddings and (ii) the set of latent embeddings using the neural network to generate the network output that characterizes the multimodal entity; Including, the neural network comprises a sequence of neural network blocks comprising: (i) one or more mutual attention blocks; (ii) one or more self-attention blocks; and (iii) an output block; each mutual attention block performs operations including updating each potential embedding in the set of potential embeddings using attention for some or all of the data element embeddings in the set of data element embeddings; each self-attention block performs operations including updating each potential embedding in the set of potential embeddings using attention for the set of potential embeddings; the output block performs operations including processing one or more latent embeddings from the set of latent embeddings to generate the network output that characterizes the multimodal entity after the set of latent embeddings has been updated using the one or more mutual attention blocks and the one or more self-attention blocks.

2. The multimodal entity comprises a combination of video and audio data, the multimodal processing task comprises a classification or segmentation task, and the network output comprises a classification or segmentation of the multimodal entity; or the multimodal entities include data other than textual data, and the network output includes text describing the multimodal entities; or the multimodal entity comprises audio data representing an utterance, and the network output comprises a score for each fragment of a set of text fragments, each score representing an estimated likelihood that the text fragment is a correct transcription of the utterance, or indicating whether a particular word or phrase was uttered in the utterance; or the multimodal entities represent a sequence of one or more observations characterizing a state of a real-world environment, and the network outputs define actions to be performed by a real-world robot in response to the most recent data in the sequence; The method of claim 1.

3. The method of claim 1 or 2, wherein the number of potential embeddings in the set of potential embeddings is less than the number of data element embeddings in the set of data element embeddings.

4. 4. The method of claim 1, wherein the number of potential embeddings in the set of potential embeddings is predefined and independent of the number of data element embeddings in the set of data element embeddings.

5. 5. The method of claim 1, wherein the neural network comprises a plurality of mutual attention blocks and a plurality of self-attention blocks, the plurality of mutual attention blocks and the plurality of self-attention blocks being interleaved.

6. processing, by the output block, one or more latent embeddings from the set of latent embeddings to generate the network output characterizing the multimodal entity, pooling the potential embeddings in the set of potential embeddings to generate a pooled potential embedding; and processing the pooled latent embeddings using one or more neural network layers to generate the network output that characterizes the multimodal entity.

7. The method of claim 6 , wherein pooling the potential embeddings in the set of potential embeddings comprises averaging the potential embeddings.

8. the network output characterizing the multimodal entity comprises a sequence of output elements, and the step of processing by the output block one or more latent embeddings from the set of latent embeddings to generate the network output characterizing the multimodal entity includes, at each time step of a plurality of time steps, processing one or more latent embeddings from the set of latent embeddings to generate an output element for that time step:

6. The method of claim 1, comprising processing (i) the one or more potential embeddings from the set of potential embeddings, and (ii) output elements generated in any preceding time step.

9. For each self-attention block, updating each potential embedding in the set of potential embeddings using attention for the set of potential embeddings comprises:

9. The method of claim 1, comprising updating each potential embedding in the set of potential embeddings using a query-key value attention for the set of potential embeddings.

10. Each self-attention block consists of 10. The method of claim 1, wherein the method performs operations including iteratively updating each potential embedding in the set of potential embeddings using attention on the set of potential embeddings.

11. For each mutual attention block, updating each potential embedding in the set of potential embeddings using attention for some or all of the data element embeddings in the set of data element embeddings comprises: updating each potential embedding in the set of potential embeddings using query-key value attention for some or all of the data element embeddings in the set of data element embeddings; generating a respective query embedding for each potential embedding in the set of potential embeddings; generating, for each of a plurality of data element embeddings in the set of data element embeddings, a respective key embedding and a respective value embedding; 11. The method of claim 1, comprising: updating each potential embedding in the set of potential embeddings using a query-key-value attention for the multiple data element embeddings in the set of data element embeddings based on (i) the query embedding for the potential embedding, and (ii) the key embedding and value embedding for the data element embedding.

12. the multimodal entity comprises a plurality of units arranged in a spatial structure, each unit associated with position data defining a respective position of the unit in the spatial structure; obtaining the representation of the multimodal entity as the set of data element embeddings, for each unit in the multimodal entity, generating a feature embedding of the unit based on the features of the unit; generating, for each unit in the multimodal entity, a positional embedding of the unit based on the position of the unit in the spatial structure; 12. The method of claim 1, comprising: for each unit in the multimodal entity, generating a data element embedding for the unit based on (i) the feature embedding for the unit, and (ii) the position embedding for the unit.

13. generating, for each unit in the multimodal entity, the data element embedding of the unit based on (i) the feature embedding of the unit, and (ii) the position embedding of the unit, comprising: The method of claim 12 , comprising concatenating the feature embedding of the unit and the location embedding of the unit.

14. 14. The method of claim 12 or 13, wherein the spatial structure is a one-dimensional (1D), two-dimensional (2D), or three-dimensional (3D) array of units.

15. generating, for each unit in the multimodal entity, the location embedding of the unit based on the location of the unit in the spatial structure, 15. The method of claim 14, comprising generating, for each unit in the multimodal entity, a Fourier feature location encoding having frequency bands spaced log-linearly over a predefined target frequency range.

16. The method of claim 15, wherein the sequence of neural network blocks of the neural network further comprises one or more selection blocks; Each selection block is processing the set of potential embeddings and the set of data element embeddings after the set of potential embeddings has been updated using one or more mutual attention blocks, one or more self-attention blocks, or both, to generate a respective selection score for each data element embedding in the set of data element embeddings; selecting an appropriate subset of the set of data element embeddings for use by one or more designated mutual attention blocks based on the selection scores; 16. The method of claim 1, wherein each designated mutual attention block updates each potential embedding in the set of potential embeddings using attention for only data element embeddings in the selected appropriate subset of the set of data element embeddings.

17. Each selection block comprises: (i) a parameter selection neural network; and (ii) a unit selection neural network; For each selection block, processing the set of potential embeddings and the set of data element embeddings to generate the respective selection scores for each data element embedding in the set of data element embeddings comprises: processing the latent embeddings using the parameter selection neural network to generate a network output that defines values ​​for a set of neural network parameters of the unit selection neural network; and processing each data element embedding in the set of data element embeddings using the unit selection neural network according to the values ​​of the set of neural network parameters of the unit selection neural network to generate the selection score for the data element embedding.

18. selecting an appropriate subset of the data element embeddings to be used by one or more designated mutual attention blocks based on the selection scores, 18. The method of claim 16 or 17, comprising selecting a predefined number of the data element embeddings having the highest selection scores among the set of data element embeddings.

19. calculating a task performance measure based on the network output characterizing the multimodal entity; assessing a reward based on the task performance measure; training the selection block against a reinforcement learning objective function that depends on the reward; 19. The method of any one of claims 16 to 18, further comprising:

20. 20. The method of claim 19, wherein the task performance measure comprises a cross-entropy classification error.

21. 21. The method of claim 19 or 20, wherein the reinforcement learning objective function comprises a squared Bellman error.

22. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations of any one of claims 1 to 21.

23. one or more computers; one or more storage devices communicatively coupled to the one or more computers; A system comprising:

22. A system, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of each of the methods described in any one of claims 1 to 21.

Images