Processing network input using partitioned attention

By using partition attention mechanism in neural networks, the multimodal representation entanglement problem is solved, the performance of multimodal tasks is improved, and the output accuracy is maintained in singlemodal tasks, ensuring that the model is in the absence of modality. Validity.

CN119096250APending Publication Date: 2024-12-06GDM HOLDING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380025622.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-07
Filing Date
2023-03-07
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing attention-based neural networks have representation entanglement problems in multimodal processing, resulting in the inability to operate the features of each modal independently, affecting the execution of contrast learning and single-modal tasks.

Method used

Ensure model modal specificity in multimodal tasks by using partition attention mechanisms in neural networks, updating its latent vector set for each modality and updating the fusion latent vector set with information from all modalities, ensuring model modality specificity, thus in multimodal tasks Improve performance and maintain effectiveness in single-modal tasks.

Benefits of technology

It realizes effective comparison pre-training and downstream performance improvement in multimodal tasks, while ensuring the accuracy of output in singlemodal tasks, and can generate accurate output even if a certain mode is missing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096250A_ABST
    Figure CN119096250A_ABST
Patent Text Reader

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing network inputs using neural networks that implement partition attention.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 317,538, filed on March 7, 2022. The disclosure of the prior application is considered part of and incorporated by reference into the disclosure of the present application. Background Art

[0003] This specification relates to processing input using a neural network.

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs from received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network, that is, the next hidden layer or output layer. Each layer of the network generates an output from the received inputs based on the current value inputs of the corresponding set of parameters. Summary of the invention

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations that processes a network input partitioned into multiple disjoint partitions to generate a network output for a machine learning task.

[0006] For example, the network input can be a single unimodal tensor, and each disjoint partition can be a different non-overlapping region of the tensor.

[0007] As another example, the network input may be a multimodal input comprising a plurality of different modalities, and each partition may be a different one of the plurality of modalities.

[0008] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.

[0009] Attention-based models are attractive for multimodal processing because inputs from multiple modalities can be concatenated and fed into a single backbone network — thus requiring very little fusion engineering. However, the resulting representations are fully entangled throughout the network, which can be problematic in several scenarios.

[0010] For example, contrastive learning has been shown to be an effective technique for leveraging unlabeled data to improve downstream performance on a variety of tasks. During training, multimodal contrastive self-supervised learning requires independent features of each modality to operate on, otherwise the learning collapses. However, because the representations are entangled, there are no independent features suitable for use as input to the contrast.

[0011] Furthermore, at inference time, these models cannot effectively perform unimodal tasks or process input when one modality is missing.

[0012] This specification describes techniques for controlling how inputs from each modality are routed within an attention-based neural network in order to keep the portion of the model's internal representation modality specific, i.e., based only on data from a single modality. Specifically, this specification describes techniques for each modality that use attention only on the latent vectors for that modality (and not other modalities) to update the set of latent vectors for that modality, while using information from all modalities to update the set of fused latent vectors.

[0013] This allows the system to effectively incorporate contrastive pre-training in order to improve performance on a variety of downstream multimodal tasks. Additionally, this allows the system to efficiently perform unimodal inference, or more generally, generate accurate output for a task even when data from one modality is missing.

[0014] The details of one or more embodiments of the subject matter of this specification are set forth in the drawings and description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a diagram of an example neural network system.

[0016] Figure 2 is a flow chart of an example process for processing network input using a neural network.

[0017] Figure 3 An example architecture of an attention block is shown.

[0018] Figure 4 Another example architecture of an attention block is shown.

[0019] Figure 5 Yet another example architecture of an attention block is shown.

[0020] Figure 6 The operation of the output block is shown.

[0021] Figure 7 Various uses of neural networks are shown, both during training and during inference.

[0022] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0023] Figure 1is a diagram of an example neural network system 100. Neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0024] The neural network system 100 processes a network input 102 partitioned into a plurality of disjoint partitions 104 to generate a network output 112 for a machine learning task.

[0025] For example, the network input 102 may be an image, and the disjoint partitions may be different non-overlapping regions of the image. In this example, the network output 112 may be an object detection output that identifies regions in the image that depict objects, an image segmentation output that assigns a corresponding class label to each pixel in the image, or an image classification output that classifies the image into one or more object classes from a set of object classes.

[0026] As another example, the network input 102 can be a video, and the disjoint partitions can be different sets of video frames from the video or different space-time regions from the video. In this example, the network output 112 can be any suitable video understanding output, such as an action recognition output that identifies one or more actions being performed in the video, a topic classification output that identifies one or more topics to which the video relates, an object detection output that identifies regions in a video frame that depict an object, a video segmentation output that assigns a corresponding class label to each pixel in one or more video frames of the video, etc.

[0027] As another example, the network input 102 may be other inputs from sensors that measure characteristics of the physical environment. Disjoint partitions may be, for example, different sets of measurements that are distinct in time or measurements of different regions in space. The sensor may be any appropriate sensor, and by way of example only, the network input 102 may include measurements of light, temperature, radar, sonar, LIDAR, tactile feedback, resistance, voltage, current, acceleration, chemical composition, and the like. In this example, the network output 112 may be any output that characterizes the physical environment, such as a recognition output that identifies one or more characteristics of the environment, a categorized output that assigns corresponding category labels to each environment in a series of environments, and the like. The task may include, for example, selecting an action to be performed by the mechanical agent in response to the identification of a characteristic or the assignment of a category, or may be used to detect or predict a fault condition or other condition.

[0028] As another example, the network input 102 may be input from a sensor that measures the position or state of the mechanical agent, i.e., proprioception or posture. Disjoint partitions may be, for example, different measurements of different parts of the mechanical agent (e.g., actuators, limbs) or different sets of temporally distinct measurements. In this example, the network output 112 may be any output that characterizes the state of the mechanical agent, a categorized output that assigns a corresponding class label to each part of the mechanical agent, or a series of postures of the mechanical agent.

[0029] As another example, the network input 102 can be a multimodal input including multiple different modalities, and each partition can be a different modality in the multiple modalities. In general, a multimodal processing task can correspond to any task previously described for any data type that constitutes a multimodal combination. For example, when a task is applied to multimodal data that combines data for which a task has been previously described and data of another type, the accuracy of the previously described task can be improved. For example, when processing data of multiple different types (modalities), the detection or classification of objects or events can be improved.

[0030] For example, multimodal input may include two or more of any of the above inputs, such as images, video, text, and audio.

[0031] In these cases, the machine learning task may be any suitable multimodal task, and the network output 112 may include any suitable multimodal task output.

[0032] For example, when the input 102 includes a video or image and text, the network output 112 may be a sequence of text that answers questions posed by the text about the video or image, or an edited video or image that has been edited as described in the text.

[0033] As another example, when the input 102 includes video and audio, the audio can be an audio soundtrack for the video, and the network output 112 can be an output for a classification task, such as classifying the subject of the video, classifying audio event classes in the audio soundtrack, classifying one or more types of objects making noise in the audio, classifying one or more actions being performed in the video, etc.

[0034] As another example, when input 102 includes video and audio, the audio can be an audio track for the video, and network output 112 can be an output for a speech recognition task, e.g., speech present in the audio track can be transcribed, for a speech isolation task, e.g., isolated speech signals for one or more speakers in the audio track can be output, and the like.

[0035] In particular, the system 100 takes a network input 102 and uses a partitioning engine 104 to determine a plurality of disjoint partitions of the network input.

[0036] If the network input 102 is a multi-modal input, the multiple disjoint partitions may be different modalities of the input.

[0037] If the network input 102 is a tensor of a single modality, the multiple disjoint partitions 104 may be different parts of the tensor.

[0038] For each partition 104, the system 100 generates from the partition a corresponding set of partitioned potential word-grams 106. For example, the system 100 can process the partition using a corresponding encoder neural network to generate potential word-grams.

[0039] As used in this specification, a "word-gram" is a vector or other ordered collection of numerical values ​​having fixed dimensionality, ie, the number of values ​​in the ordered set is constant across different word-grams.

[0040] The system 100 also generates a set of fused potential word-grams 108 .

[0041] For example, the fused potential word-grams 108 may each consist of predetermined values ​​or may be learned during training.

[0042] The system 100 processes the corresponding set of potential word-grams 106 for each partition and the set of fused potential word-grams 108 using a neural network 110 to generate a network output 112 that represents the network input 102 .

[0043] Neural network 110 includes a sequence of neural network blocks 120 including (i) one or more attention blocks 130 and (ii) an output block 140 .

[0044] Each attention block 130 uses partitioned attention to update the latent word-gram, i.e., a corresponding attention mechanism is used for each partition and fused word-gram.

[0045] The following will refer to Figure 2 and 3 Describe in more detail updating latent tokens using partitioned attention.

[0046] When updating the potential word-unit, each attention block 130 may also perform additional operations, such as one or more of a normalization operation, a skip connection operation, a feed-forward layer operation, etc.

[0047] After updating the respective sets of potential word-grams 106 and fusing the potential word-grams 108 using the one or more attention blocks 130 , the output block 140 processes at least one or more of the potential word-grams 106 , 108 to generate a network output 112 that represents the network input 102 .

[0048] In some implementations, the output block 140 generates a single network output (from at least one or more of the potential word-grams 106, 108), which is then used as the network output 112. For example, the output block 140 may generate a fused output generated only from one or more fused word-grams (rather than any word-grams of any partition) or a combined ("global") output generated from all potential word-grams as a single output. As another example, when a partition-specific output is needed, e.g., when data from other partitions is missing, the output block 140 may generate a partition output for a specified partition as a single output, which is generated only from one or more word-grams in the word-grams of the partition (rather than any word-grams of the fused word-grams or any other partitions).

[0049] In some other implementations, the output block 140 generates multiple candidate network outputs and then combines the candidate network outputs, for example, averaging the network outputs, to generate the final network output 112 .

[0050] For example, output block 140 may generate two or more of the following: a fused output generated only from one or more fused terms in the fused term (and not any terms in any partition), a respective partition output for each partition generated only from one or more terms in the term-grams of a partition (and not any terms in the fused term or any other partition), or a combined ("global") output generated from all potential term-grams.

[0051] Reference below Figure 6 The operations performed by output block 140 to generate output are described.

[0052] Figure 2 is a flow chart of an example process 200 for processing a network input to generate a network output. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to the present specification, such as Figure 1 The neural network system 100 depicted in FIG. 1 may perform process 200 .

[0053] The system obtains network input (step 202).

[0054] The system determines a plurality of disjoint partitions of the network input (step 204 ).

[0055] For each partition, the system generates a corresponding set of potential word-grams for the partition from the partition (step 206 ).

[0056] For example, for each partition, the system can process the partition of the network input using the partition's encoder neural network to generate a corresponding set of latent tokens for the partition.

[0057] As a specific example, the system can divide the partition into patches and then process each patch using the partitioned encoder neural network to generate latent tokens.

[0058] When each partition is from the same modality, the system can use the same encoder neural network for each modality. When the partitions are from different modalities, the system can use different encoder neural networks for different partitions. For example, the encoder neural network can include any of a convolutional neural network including only a single neural network layer, such as a single linear layer or a single convolutional layer, a multilayer perceptron (MLP), or a shallow neural network.

[0059] Optionally, the system can modify each potential word-gram generated by processing a partition of the network input using the encoder neural network using a positional encoding, for example, by summing or averaging each potential word-gram with the corresponding positional encoding. The positional encoding typically identifies for each patch of each partition the position of the patch within the partition and to which partition the patch belongs. For example, the system can arrange the potential word-grams of all modalities (and optionally the fused potential word-grams) as a sequence and assign to each potential word-gram a positional encoding corresponding to the potential position in the sequence.

[0060] The position encoding may be learned in conjunction with the training of the neural network, or may be predetermined, for example, it may be a sinusoidal or Fourier position encoding. Alternatively, the position encoding may be a combination of a learned position encoding and a predetermined position encoding, such as a sum.

[0061] The system generates a set of fused potential word-grams (step 208). For example, the set of fused potential word-grams can be jointly learned during the training of the neural network. As described above, the system can also optionally utilize position encoding to modify the fused potential word-grams.

[0062] The system processes the corresponding set of potential word-grams for each partition and fuses the set of potential word-grams using a neural network to generate a network output that represents the network input (step 210 ).

[0063] As described above, a neural network typically comprises a sequence of neural network blocks, which comprises (i) one or more attention blocks and (ii) an output block.

[0064] Each attention block updates the latent tokens, i.e., the set of latent tokens for each partition and the fused latent tokens.

[0065] As part of updating the latent tokens, for each partition, the block updates the corresponding set of latent tokens for the partition by applying, for the partition, a corresponding attention mechanism that only pays attention to the corresponding set of latent tokens for the partition.

[0066] The attention block also updates the set of fused latent tokens by applying a corresponding fused attention mechanism that attends to the corresponding set of latent tokens for each partition in the partition and the set of fused latent tokens.

[0067] An example of the corresponding attention mechanism of the attention block is referenced below Figure 3 describe.

[0068] Another example of a corresponding attention mechanism for an attention block is referenced below Figure 4 describe.

[0069] Another example of a corresponding attention mechanism for an attention block is given below Figure 5 describe.

[0070] After updating the corresponding sets of potential word-grams and the fused potential word-grams using the one or more attention blocks, the output block processes at least one or more of the fused potential word-grams to generate a network output representing the network input.

[0071] In some implementations, the output block generates the network output by applying a criss-cross attention mechanism.

[0072] An example of generating network outputs through crisscross attention is given below. Figure 6 describe.

[0073] Figure 3 An example architecture 300 of the attention block 130 is shown.

[0074] exist Figure 3 In the example of , the input is a multimodal input including two modalities: audio and video. Therefore, the input to the attention block 130 includes a set of potential word-grams 302 for the audio modality, a fused potential word-gram 304, and a set of potential word-grams 306 for the video modality.

[0075] exist Figure 3 In the example of , the corresponding attention mechanism of each partition and fused latent token is the same self-attention mechanism, i.e., the corresponding attention mechanism of the partition and fused latent token share parameters and can be calculated in parallel. However, the attention block incorporates masking to regulate the information flow between partitions. Specifically, the attention block incorporates masking to ensure that for each partition, only tokens from that partition are used (not tokens from any other partition or fused latent tokens) to update the partition token, while all tokens and latent tokens from all partitions are used to update the latent fused token.

[0076] Specifically, in Figure 3In the example, for each partition, the corresponding attention mechanism for the partition is a self-attention mechanism with a partition-specific mask that restricts the attention mechanism to only pay attention to the corresponding set of potential tokens of the partition, and for the fused latent tokens, the corresponding attention mechanism is a self-attention mechanism that is not masked to allow the attention mechanism to pay attention to the corresponding set of potential tokens of each partition and the corresponding set of fused latent tokens in the partition.

[0077] Specifically, the attention block can be constructed by transforming the masked binary tensor Introduced into each calculation of the attention weight, for example, in each attention head of the attention block 130, to implement the above attention mechanism. The masked tensor is applied to the standard attention operation , which becomes ,in It is a word The value vector of It is a word Query With attention applied Keys for positions The attention weights between yes:

[0078] in are constant values, such as the dimensions of the query, the key vector, and the value vector, and where if is one of the fused words, then is equal to 1, otherwise only and When from the same partition, is equal to 1, otherwise it is equal to 0.

[0079] Therefore, each latent word-gram of each partition is updated using only the latent word-gram of that partition, while the fused latent word-gram is updated using the fused latent word-gram and the latent word-grams of all partitions.

[0080] Figure 4 Another example architecture 400 of the attention block 130 is shown.

[0081] exist Figure 4 In the example of , the input is a multimodal input including two modalities. Therefore, the input to the attention block 130 includes a set of potential word-grams for the first modality, a fused potential word-gram, and a set of potential word-grams for the second modality.

[0082] exist Figure 4 In the example above, the corresponding attention mechanisms for each partition and fused latent token are different self-attention mechanisms.

[0083] Specifically, the attention block updates the latent tokens of each partition using the corresponding self-attention mechanism that only pays attention to the latent tokens in that partition.

[0084] For the fused latent tokens, the self-attention mechanism includes two separate mechanisms: (i) a self-attention mechanism that updates the fused latent token by self-attention on each fused latent token; and (ii) a cross-attention mechanism that uses the fused latent tokens (as updated by (i)) to cross-attention into the partitioned latent tokens after those tokens have been updated by their corresponding self-attention mechanisms.

[0085] Therefore, each latent word-gram of each partition is updated using only the latent word-gram of that partition, while the fused latent word-gram is updated using the fused latent word-gram and the latent word-grams of all partitions.

[0086] Structure as Figure 4 The attention blocks shown allow different types of self-attention to be used for each partition in the partition, e.g., to exploit different structures in the underlying data. Figure 4 As shown, one partition may correspond to the audio modality and another partition may correspond to the video modality. The system may use one attention mechanism for the audio modality and a different attention mechanism for the video modality.

[0087] exist Figure 4 In the example of , the system uses 2D SwinTransformer blocks as the attention mechanism for the audio modality due to the 2D structure of the input space of the audio spectrogram. For the video modality, the system uses 3D SwinTransformer blocks as the attention mechanism to exploit the 3D structure of the input space of the video frame.

[0088] In general, a Swin Transformer block differs from a standard self-attention block in that, for each token, the Swin Transformer block applies the self-attention operation only to nearby tokens instead of to all tokens in a partition. A 2D Swin Transformer block applies the self-attention operation to the (a, b) patch of nearby tokens, i.e., a 2-dimensional (2D) patch. On the other hand, a 3D Swin Transformer block applies the self-attention operation to the (a, b, c) patch of nearby tokens, i.e., a 3-dimensional (3D) patch. The Swin Transformer block is described in more detail in Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXivpreprint arXiv:2111.09883, 2021. Figure 5 Yet another example architecture 500 of the attention block 130 is shown.

[0089] Similar to Figure 3 and Figure 4 In the example of , the input is a multimodal input including two modalities. Therefore, the input to the attention block includes a set of potential word units for the first modality, a fused potential word unit, and a set of potential word units for the second modality.

[0090] exist Figure 5 In the example of Fig. 2, the corresponding attention mechanism for each partition and fused latent tokens is based on the Hierarchical Perceptron (HiP) attention block.

[0091] The HiP attention block operates by dividing the input to the block into multiple groups, and for each group, applying cross attention to the set of learned query vectors for the group to generate a set of latent vectors, and then operating self-attention only within the latent vectors of the group. The latent vectors are then merged to generate the input of the next network component, such as the next HiP attention block. By arranging such blocks in a hierarchical architecture, a hierarchical perceptron neural network can fuse groups together to aggregate information and global reasons about the input. HiP blocks are described in more detail in Joao Carreira, Skanda Koppula, Daniel Zoran, Adria Recasens, Catalin Ionescu, Olivier Henaff, Evan Shelhamer, Relja Arandjelovic, Matt Botvinick, Oriol Vinyals, et al. Hierarchical perceiver. arXiv preprint arXiv:2202.10890, 2022. The example architecture 500 includes separate HiP attention blocks for each partition. That is, the corresponding attention mechanism for each partition is implemented as a separate HiP block for each partition.

[0092] Therefore, the attention block 130 processes the potential word-grams of each partition using the partitioned HiP attention block to update the potential word-grams of the partition.

[0093] For fused word-grams, the attention block 130 includes two separate HiP blocks. The block 130 processes the fused potential word-gram using only the first HiP block to update the fused potential word-gram. After the first HiP block, the attention block 130 concatenates the partitioned potential word-grams with the fused potential word-grams to generate a combined sequence, and processes the combined sequence using the HiP block to update the fused potential word-grams instead of the partitioned potential word-grams.

[0094] Therefore, the corresponding attention mechanism for the fused latent word-gram is implemented as a first HiP block operating on the fused latent word-gram and a second HiP block operating on the combined sequence.

[0095] Therefore, each latent word of each partition is updated using only the latent word of that partition, i.e., by being processed by the HiP block of the partition, while the fused latent word is updated using the fused latent word and the latent words of all partitions, i.e., with the help of the combined sequence processed by the second HiP block of the fused latent word.

[0096] The system can achieve hierarchical fusion across the entire input by modifying the number of blocks used by the HiP blocks within different attention blocks 130. For example, when the neural network 110 includes a sequence of five attention blocks 130, the number of groups can be as follows: (32; 4; 1; 1; 1). Thus, in this example, the HiP blocks within the last three attention blocks 130 operate on the entire set of potential tokens received as input by the HiP blocks.

[0097] Figure 6 An example 600 of operations performed by the output block 140 is shown.

[0098] exist Figure 6 In the example of , the output block 140 generates the network output by utilizing cross attention.

[0099] Specifically, in Figure 6 In the example of , for any given input, the output block 140 can generate one or more of three different types of outputs 630: (1) a partition-specific output generated using only the latent tokens of a single partition, (2) a fused output generated using only the fused latent tokens, or (3) a global output generated using all latent tokens, i.e., the fused latent tokens and the latent tokens of all partitions.

[0100] In some cases, output block 140 is configured to generate only a fused output or a global output, but not both.

[0101] Similarly, in some cases, after training, the output block 140 is only configured to generate one or more of a fused output or a global output, rather than any partition-specific output.

[0102] Thus, the output block 140 maintains a corresponding set of learned query vectors 610 for each partition, fused latent tokens, and global outputs, and a corresponding output head 620 for each output 630. The learned query vectors 610 are learned jointly with the training of the neural network 110.

[0103] To generate the output for a given partition, the output block 140 uses the learned query vector of the partition to apply cross attention to the potential tags of the partition to update the learned query vector. The output block 140 can then use a partition-specific header that applies one or more appropriate learned transformations to one or more updated learned query vectors to map the learned query vector to the output of the machine learning task. For example, the output block 140 can process each learned query vector using one or more linear layers, activation function layers, softmax layers, or some combination of the above.

[0104] To generate the fused output, the output block 140 uses the learned query vector of the fused potential word to apply cross attention to the fused potential word to update the learned query vector. Then, the output block 140 can use a fusion head that applies one or more appropriate learning transformations to one or more updated learning query vectors to map the learning query vector to the output of the machine learning task. For example, the output block 140 can use one or more linear layers, activation function layers, softmax layers, or some combination of the above to process each learning query vector.

[0105] To generate a global output, the output block 140 applies cross attention to all potential tags using the learned query vector for the global output to update the learned query vector. The output block 140 can then use a global head that applies one or more appropriate learned transformations to one or more updated learned query vectors to map the learned query vectors to the output of the machine learning task. For example, the output block 140 can process each learned query vector using one or more linear layers, activation function layers, softmax layers, or some combination of the above.

[0106] exist Figure 6 In the example of , the task is a classification task that requires classifying the input into one or more of 527 categories. Therefore, each output is a 1 x 527 vector of scores, such as probabilities. Similarly, each set of learned query vectors includes a single query vector, and each output head maps the corresponding single updated learned query vector to a 1 x 527 vector of scores.

[0107] Thus, partition-specific outputs depend only on a single given partition, while fused and global outputs incorporate information from the entire input.

[0108] Before using the neural network 110 to perform a machine learning task, the training system trains the neural network 110 to perform the task, i.e., determines training values ​​for the parameters of the neural network, i.e., training values ​​for the attention blocks and output blocks in the sequence, and optionally, an encoder neural network for generating potential word units. For example, the training system can train the neural network from scratch on the training data of the task using conventional machine learning techniques to minimize the loss function of the task, such as cross entropy loss, negative log-likelihood loss, etc. As another example, the training system can first pre-train the neural network on an unsupervised objective and then fine-tune the neural network on the training data of the task. As yet another example, the training system can train the neural network on both the unlabeled data and the training data of the task through semi-supervised learning.

[0109] During training, the training system can incorporate any number of techniques to improve the speed, effectiveness, or both of the training process. For example, the system can use dropout, label smoothing, or both to reduce overfitting. As another example, the system can perform training using a distributed architecture that trains multiple instances of a neural network in parallel.

[0110] Furthermore, as described above, the system can first pre-train a neural network on a large unsupervised dataset through self-supervised or unsupervised learning, and then fine-tune the neural network on task-specific training data to optimize the loss function for the task.

[0111] In particular, due to their architecture, neural networks are particularly suitable for training using unsupervised or self-supervised objectives, either as a pre-training step or as an auxiliary loss for supervised training, even when the input to the neural network is multimodal.

[0112] Additionally, after training, the neural network 110 may be used to process two inputs that include only a single partition, for example, a unimodal input and a multimodal input.

[0113] Figure 7 Examples of various uses of neural network 110 are shown.

[0114] like Figure 7 As shown, a neural network referred to as “Zorro” in the figure can be used for both multimodal inference 740 and unimodal inference 730 after training, even though the neural network is only trained to perform multimodal tasks or unimodal tasks (but not both).

[0115] To use the neural network for multimodal inference 740, i.e., where the input includes two or more modalities, each modality being assigned to a different partition, the system uses the output block to generate a fused output, a global output, or both. Further optionally, the system can also generate corresponding partition-specific outputs for each partition. The system can then use the fused output or the global output as the network output, or when multiple outputs are generated, combine the outputs as described above to generate the network output.

[0116] To use a neural network for unimodal inference 730, i.e., where the input includes only one modality, and that modality is assigned to a single partition, the system may proceed as follows.

[0117] Specifically, the system may receive a network input that includes only a first modality among multiple modalities. As described above, the system may generate a set of potential word-grams for a partition corresponding to the first modality. The system may then use a neural network to process the set of potential word-grams for the partition corresponding to the first modality in the following manner: for each attention block, update the corresponding set of potential word-grams for the partition corresponding to the modality by applying a corresponding attention mechanism to the partition that only pays attention to the set of potential word-grams for the partition. After updating the set of potential word-grams for the corresponding modality using one or more attention blocks, the system may use an output block to process one or more potential word-grams from the set of potential word-grams for the corresponding modality to generate a network output that represents the network input. Therefore, the neural network does not need to perform the operation of the attention block for other modalities or for fused word-grams.

[0118] During training of the neural network, the system may use the neural network to perform both contrast pre-training (unsupervised tasks) 710 and supervised training 720 .

[0119] To perform supervised training on a given training example that includes a training input and a label, the system can use a neural network to generate each of the fused outputs and modality-specific outputs (and optionally, global outputs) of the training input. The system can then calculate the corresponding losses between the labels and all generated outputs, and use the gradients of all corresponding losses to update the parameters of the neural network. This can give a richer training signal than using only a single multimodal output, and can also allow the neural network to effectively perform unimodal inference after training.

[0120] Multimodal contrastive methods learn representations by aligning multiple modalities into a common embedding space. In contrast to unimodal methods, multimodal contrastive methods use different modalities as views instead of producing multiple views of the data. An important requirement is that the two components of the neural network that generate the embeddings do not share information. If information is shared across modalities, self-supervised training can easily collapse or converge to a trivial solution.

[0121] Conventional models for multimodal perception typically produce a single output for multiple inputs. This is sufficient for supervised applications, but prevents the use of these contrastive techniques. However, the ability of neural networks to generate both unimodal and multimodal outputs enables the effective use of self-supervised contrastive losses.

[0122] For training with a contrastive loss, such as a noise contrastive estimation loss, the system applies a final linear projection (different for each modality) to the output of the cross-attention performed by the output block for that modality for each modality to produce a final embedding vector for each modality. The system can then use these final embedding vectors to compute a contrastive loss, such as a noise contrastive estimation loss or other contrastive learning loss that measures the similarity between embeddings. To train any parameters specific to the fused representation or output (e.g., fused cross-attention or fused weights if the model has separate weights for each modality), the system can add a corresponding fused contrastive loss for each modality. The fused loss for a given modality is a contrastive loss that takes as input the final fused embedding computed from the fused latent vector as described above and the final embedding for the given modality. For example, when there are two input modalities, the overall contrastive loss can be the sum or weighted sum between the contrastive loss between the two input modalities and the fused contrastive loss of the two input modalities.

[0123] When learning the fused latent vector during supervised, unsupervised, or self-supervised training, the system can also update the fused latent vector, for example, by back-propagating the gradient of the loss function through the neural network into the fused latent vector.

[0124] When the system also trains one or more encoder neural networks jointly with the training of the neural network, the system can also update the encoder neural network by back-propagating the gradient of the loss function through the neural network into the encoder neural network.

[0125]

[0126] Table 1

[0127] Table 2

[0128] Tables 1 and 2 show the performance of the described neural network ("Zorro") on various tasks that require processing audio (A), video (V), or multi-modal inputs that require processing A and V, and under various self-supervised pre-training ("pre-training") and supervised pre-training ("Sup. Pre-training"). In Tables 1 and 2, when processing both audio and video, the task is audio-video classification, and when processing only audio, the task is audio classification. As can be seen from Tables 1 and 2, Zorro achieves results that match or exceed various baselines (all models not labeled "Zorro" in the table) on various tasks under various regimes in terms of both top-1 accuracy ("top-1") and top-5 accuracy ("top-5") . That is, the described architecture is flexible enough to generate high-quality results in any of a variety of training regimes without requiring modifications to the architecture. In addition, after training, the described architecture can be effectively used to perform unimodal processing even if the training data is multimodal.

[0129] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0130] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, for example, one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing device or for controlling the operation of a data processing device. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical signal, optical signal, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0131] The term "data processing apparatus" refers to data processing hardware and covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the apparatus may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0132] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files storing one or more modules, subroutines, or portions of code. A computer program may be deployed to execute on one computer located at one site or on multiple computers distributed across multiple sites and interconnected by a data communications network.

[0133] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way or at all, and it may be stored on storage devices in one or more locations. Thus, for example, an index database may include multiple data collections, each of which may be organized and accessed differently.

[0134] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and run on the same computer or on multiple computers.

[0135] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit such as an FPGA or ASIC, or by a combination of a dedicated logic circuit and one or more programmed computers.

[0136] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by or incorporated into a special logic circuit. Typically, the computer will also include one or more mass storage devices, or be operably coupled to receive data from or transmit data to or to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, the computer does not need to have such a device. In addition, the computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0137] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0138] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device, such as a smart phone running a messaging application, and receiving a response message from the user in return.

[0139] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production, such as inference, workloads.

[0140] Machine learning models can be implemented and deployed using a machine learning framework such as the TensorFlow framework or the Jax framework.

[0141] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0142] The computing system may include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to a user device, for example, to display data to a user of the device interacting with the client and to receive user input therefrom. Data generated at the user device, such as the result of a user interaction, may be received from the device at the server.

[0143] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope that can be claimed, but rather as descriptions of features that can be specific to a particular embodiment of a particular invention. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0144] Similarly, although operations are depicted in a particular order in the drawings and recited in the claims, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0145] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0146] A number of statements describing examples of the subject matter described herein are now set forth.

[0147] 1. A method performed by one or more computers, the method comprising: Get the first network input; determining a plurality of disjoint partitions of the first network input; For each partition, generating a corresponding set of potential word-grams for the partition from the partition; Generate a set of fused latent word units; processing the corresponding set of latent word units for each partition using a neural network and fusing the set of latent word units to generate a network output representing the first network input, The neural network includes a sequence of neural network blocks, the sequence of neural network blocks including: (i) one or more attention blocks and (ii) an output block, wherein each attention block performs an operation, the operation comprising: for each partition, updating the corresponding set of latent tokens of the partition by applying, for that partition, a corresponding attention mechanism that only attends to the corresponding set of latent tokens of the partition; and updating the set of fused latent word-grams by applying a corresponding fused attention mechanism that attends to the corresponding set of latent word-grams for each of the partitions and the set of fused latent word-grams; and The output block performs operations, including: After updating the respective sets of potential word-grams and fusing the potential word-grams using the one or more attention blocks, at least one or more of the potential word-grams are processed to generate a network output representing the first network input.

[0148] 2. The method of Example 1, wherein each disjoint partition corresponds to a different modality from a plurality of modalities.

[0149] 3. The method of Example 2, wherein the modality comprises an image modality, a video modality, or both.

[0150] 4. A method according to Example 2 or Example 3, wherein the modality includes an audio modality.

[0151] 5. A method according to any one of Examples 2 to 4, wherein the modality includes a text modality.

[0152] 6. The method according to any one of Examples 2 to 5, further comprising: receiving a second input including only a first modality of the plurality of modalities; generating a set of partitioned potential word-grams corresponding to the first modality; The set of potential word units corresponding to the partition of the first modality is processed using a neural network, including: For each attention block, updating the set of latent tokens of the partition corresponding to the modality by applying, for the partition, a corresponding attention mechanism that only attends to the corresponding set of latent tokens of the partition; and After updating the set of potential word-grams for the corresponding modality using the one or more attention blocks, one or more potential word-grams from the set of potential word-grams for the corresponding modality are processed to generate a network output representing a second network input.

[0153] 7. A method according to any of the preceding examples, wherein the fused latent tokens are learned during training of the neural network.

[0154] 8. The method of any preceding example, wherein, for each partition, generating from the partition a corresponding set of potential word-grams for the partition comprises: Partitions of the first network input are processed using a partitioned encoder neural network to generate corresponding sets of partitioned latent tokens.

[0155] 9. The method of example 8, wherein, for each partition, generating a corresponding set of potential word-grams for the partition from the partition further comprises: Each latent token generated by processing a partition of the first network input using an encoder neural network is modified using a positional encoding.

[0156] 10. The method according to example 8 or example 9, wherein, for each partition, generating a corresponding set of potential word-grams for the partition from the partition further comprises: The latent word-grams generated by processing the partitions of the first network input using an encoder neural network are augmented with one or more learned word-grams.

[0157] 11. The method of any preceding example, wherein: For each partition, the corresponding attention mechanism for the partition is a self-attention mechanism with a partition-specific mask that restricts the attention mechanism to only attend to the corresponding set of latent tokens of the partition; and For fused latent tokens, the corresponding attention mechanism is a self-attention mechanism, which is not masked to allow the attention mechanism to pay attention to the corresponding set of latent tokens for each partition in the partition and the corresponding set of fused latent tokens.

[0158] 12. The method of Example 11, wherein corresponding attention mechanisms for partitioning and fusing latent tokens share parameters.

[0159] 13. The method of any preceding example, wherein processing at least one or more of the fused potential word-grams to generate a network output representing the first network input comprises: A fused network output is generated based on one or more latent word-grams selected only from the fused latent word-grams.

[0160] 14. The method of any preceding example, wherein processing at least one or more of the fused potential word-grams to generate a network output representing the first network input comprises: An overall network output is generated based on one or more latent word-grams selected from the set of latent word-grams for each of the partitions and the fused latent word-grams.

[0161] 15. The method of any preceding example, wherein processing at least one or more of the fused potential word-grams to generate a network output representing the first network input comprises: generating two or more candidate network outputs, the two or more candidate network outputs including at least one candidate network output generated using one or more of the fused potential word-grams; and The two or more candidate network outputs are combined to generate a final network output.

[0162] 16. A method for training a neural network performed by one or more computers, the method comprising: Get the first network input; determining a plurality of disjoint partitions of the first network input; For each partition, generating a corresponding set of potential word-grams of the partition from the partition; Generate a set of fused latent word units; processing the corresponding set of latent tokens for each partition and fusing the set of latent tokens using a neural network to generate a network output representing the first network input, The neural network comprises a sequence of neural network blocks, the sequence of neural network blocks comprising: (i) one or more attention blocks and (ii) an output block, Each attention block performs an operation, which includes: For each partition, updating the corresponding set of latent word-grams of the partition by applying, for the partition, a corresponding attention mechanism that only pays attention to the corresponding set of latent word-grams of the partition; and updating the set of fused latent tokens by applying a corresponding fused attention mechanism that attends to the corresponding set of latent tokens for each of the partitions and the set of fused latent tokens; and The output block performs operations, including: After updating the respective sets of potential word-grams and the fused potential word-grams using the one or more attention blocks, at least one or more of the fused potential word-grams are processed to generate a network output representing the first network input.

[0163] The network output is compared to an expected network output for the first network input to determine a loss and the neural network is trained based on the loss.

[0164] 17. The method of Example 16, wherein generating a corresponding set of potential word units for the partitions and / or generating a fused set of potential word units comprises: processing the first network input using an encoder neural network, the encoder neural network being trained to generate a corresponding set of potential word units for each partition and / or generating a fused set of potential word units; and Wherein, training the neural network based on the loss includes training the encoder neural network.

[0165] 18. The method of Example 16 or 17, further comprising pre-training a neural network using an unsupervised dataset using a self-supervised or unsupervised learning objective.

[0166] 19. The method of Example 16, Example 17 or 18, wherein determining the loss includes using an unsupervised loss function or a self-supervised loss function as an auxiliary loss function.

[0167] 20. A method performed by one or more computers, the method comprising: Get the first unimodal network input; The unimodal input is processed using a neural network trained according to any one of Examples 16 to 19 to generate a network output representing the first unimodal network input.

[0168] 21. A system comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method according to any one of Examples 1 to 20.

[0169] 22. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations of the corresponding method described in any one of Examples 1 to 20.

Claims

1. A method executed by one or more computers, the method include: Get the first network input; determining a plurality of disjoint partitions of the first network input; For each partition, generating a corresponding set of potential word-grams for the partition from the partition; Generate a set of fused latent word units; processing the corresponding set of potential word-grams for each partition and the set of fused potential word-grams using a neural network to generate a network output representing the first network input, wherein the neural network comprises a sequence of neural network blocks, the sequence of neural network blocks comprising: (i) one or more attention blocks and (ii) an output block, Each attention block performs an operation, including: For each partition, updating the corresponding set of potential word-grams of the partition by applying, for the partition, a corresponding attention mechanism that only pays attention to the corresponding set of potential word-grams of the partition; and updating the set of fused latent word-grams by applying a corresponding fused attention mechanism that attends to the corresponding set of latent word-grams and the set of fused latent word-grams for each of the partitions; and The output block performs an operation, wherein the operation includes: After the corresponding sets of the potential word-grams and the fused potential word-grams are updated using the one or more attention blocks, at least one or more of the fused potential word-grams are processed to generate the network output representing the first network input.

2. The method according to claim 1, in, Each disjoint partition corresponds to a different modality from the plurality of modalities.

3. The method according to claim 2, in, The modality includes an image modality, a video modality, or both.

4. The method according to claim 2 or claim 3, in, The modalities include an audio modality.

5. The method according to any one of claims 2 to 4, in, The modality includes a text modality.

6. The method according to any one of claims 2 to 5, further comprising: include: receiving a second input including only a first modality of the plurality of modalities; generating a set of partitioned potential word-grams corresponding to the first modality; Processing the set of potential word-grams of the partition corresponding to the first modality using the neural network includes: For each attention block, updating the set of potential word-grams of the partition corresponding to the modality by applying, for the partition, a corresponding attention mechanism that only pays attention to the corresponding set of potential word-grams of the partition; as well as After the set of potential word-grams for the corresponding modality is updated using the one or more attention blocks, one or more potential word-grams from the set of potential word-grams for the corresponding modality are processed to generate a network output representing the second network input.

7. A method according to any preceding claim, in, The fused latent word-grams are learned during training of the neural network.

8. A method according to any preceding claim, in, For each partition, the corresponding set of potential word-grams generated from the partition includes: The partition of the first network input is processed using the partition's encoder neural network to generate a corresponding set of the potential word-grams for the partition.

9. The method according to claim 8, in, For each partition, generating a corresponding set of potential word-grams for the partition from the partition further comprises: Each potential word-gram generated by processing the partition of the first network input using the encoder neural network is modified using a positional encoding.

10. The method according to claim 8 or claim 9, in, For each partition, generating a corresponding set of potential word-grams for the partition from the partition further comprises: The latent word-grams generated by processing the partition of the first network input using the encoder neural network are augmented with one or more learned word-grams.

11. A method according to any preceding claim, in: For each partition, the corresponding attention mechanism for that partition is a self-attention mechanism with a partition-specific mask that restricts the attention mechanism to only attend to the corresponding set of potential tokens for that partition; as well as For the fused potential word-grams, the corresponding attention mechanism is an unmasked self-attention mechanism to allow the attention mechanism to pay attention to the corresponding set of the potential word-grams of each partition in the partition and the corresponding set of the fused potential word-grams.

12. The method according to claim 11, in, The corresponding attention mechanisms for the partitioning and the fused latent tokens share parameters.

13. A method according to any preceding claim, in, Processing at least one or more of the fused potential word-grams to generate the network output representing the first network input includes: A fused network output is generated based on one or more potential word-grams selected only from the fused potential word-grams.

14. A method according to any preceding claim, in, Processing at least one or more of the fused potential word-grams to generate the network output representing the first network input includes: An overall network output is generated based on one or more potential word-grams selected from the fused potential word-gram and the set of potential word-grams for each of the partitions.

15. A method according to any preceding claim, in, Processing at least one or more of the fused potential word-grams to generate the network output representing the first network input includes: generating two or more candidate network outputs, the two or more candidate network outputs including at least one candidate network output generated using one or more of the fused potential word-grams; and The two or more candidate network outputs are combined to generate a final network output.

16. A method for training a neural network executed by one or more computers, the method include: Get the first network input; determining a plurality of disjoint partitions of the first network input; For each partition, generating a corresponding set of potential word-grams for the partition from the partition; Generate a set of fused latent word units; processing the corresponding set of potential word-grams for each partition and the set of fused potential word-grams using a neural network to generate a network output representing the first network input, wherein the neural network comprises a sequence of neural network blocks, the sequence of neural network blocks comprising: (i) one or more attention blocks and (ii) an output block, Each attention block performs an operation, including: For each partition, updating the corresponding set of potential word-grams of the partition by applying, for the partition, a corresponding attention mechanism that only pays attention to the corresponding set of potential word-grams of the partition; and updating the set of fused latent word-grams by applying a corresponding fused attention mechanism that attends to the corresponding set of latent word-grams and the set of fused latent word-grams for each of the partitions; and The output block performs an operation, wherein the operation includes: processing at least one or more of the fused potential word-grams to generate the network output representing the first network input after the corresponding set of the potential word-grams and the fused potential word-grams are updated using the one or more attention blocks; and The network output is compared to an expected network output for the first network input to determine a loss and the neural network is trained based on the loss.

17. The method according to claim 16, in, Generating the corresponding sets of potential word-grams for the partitions includes processing the first network input using an encoder neural network trained to generate the corresponding sets of potential word-grams for each partition.

18. The method according to claim 17, wherein: in, Training the neural network based on the loss includes training the encoder neural network.

19. The method according to any one of claims 16 to 18, in, The set of fused potential word-grams is learned during the training, and wherein training the neural network based on the loss comprises updating the set of fused potential word-grams.

20. The method of any one of claims 16 to 19, further comprising pre-training the neural network using an unsupervised dataset using a self-supervised learning objective or an unsupervised learning objective.

21. A method according to any one of claims 16 to 20, in, Determining the loss includes using an unsupervised loss function or a self-supervised loss function as an auxiliary loss function.

22. A method executed by one or more computers, the method include: Get the first unimodal network input; The unimodal input is processed using a neural network trained according to any one of claims 16 to 21 to generate a network output representing the first unimodal network input.

23. A system comprising one or more computers and one or more storage devices, wherein the one or more storage devices store instructions, which, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method according to any one of claims 1 to 22.

24. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 22.