Neural network with feedforward spatial transformation units
A feedforward neural network with a simple spatial interaction mechanism addresses the computational inefficiency of transformers by reducing self-attention, allowing for efficient deployment and performance comparable to transformers with lower resource usage.
Patent Information
- Application Number
- JP2023560898
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-05-14
- Filing Date
- 2022-05-16
- Publication Date
- 2024-12-09
- Estimated Expiration
- 2042-05-16
AI Technical Summary
Existing neural networks, particularly transformers, require significant computational resources and are not easily deployable on edge devices due to their reliance on self-attention mechanisms, which are computationally intensive.
Implementing a feedforward neural network with a simple spatial interaction mechanism that eliminates or reduces self-attention, using static parameterization to facilitate deployment on various computing devices, including edge devices and data centers, by integrating information across positions within the sequence through feedforward spatial transformations.
The proposed neural network achieves performance comparable to or exceeding that of transformers while using less computational resources, enabling efficient deployment on a variety of devices and reducing latency in inference tasks.
Smart Images

Figure 0007596559000006 
Figure 0007596559000007 
Figure 0007596559000008
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 63 / 189,013, filed May 14, 2021, which is incorporated by reference herein in its entirety.
[0002] This specification relates to using neural networks to perform machine learning tasks on network inputs. [Background technology]
[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e. the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of its respective set of parameters. Summary of the Invention [Means for solving the problem]
[0004] Described herein is a system implemented as a computer program on one or more computers at one or more locations that performs machine learning tasks on a network input using a neural network including multiple neural network blocks, at least one of which has a feedforward spatial transformation unit. The feedforward spatial transformation unit is a unit that receives a sequence of vectors as input and applies a feedforward spatial transformation that integrates information across positions in the sequence, such that each spatially transformed input vector at a given position depends not only on the transformed input vector at the given position but also on each input vector at multiple different positions. A "feedforward" spatial transformation is one that does not use any recursion-based or attention-based operations.
[0005] Particular embodiments of the subject matter described herein can be implemented to realize one or more of the following advantages.
[0006] We describe a system that performs sequence tasks, i.e. tasks that require the processing of a sequence of inputs, using a feed-forward neural network with a simple spatial interaction mechanism, while achieving results that are comparable to or exceed those of Transformers, a state-of-the-art class of sequence-processing neural networks that requires a complex and computationally intensive self-attention mechanism to be applied iteratively across the input sequence.
[0007] More specifically, by eliminating or substantially reducing the capacity allocated to self-attention within the neural network, the neural network described herein achieves this superior performance while consuming much less computational resources than the Transformer and can be more easily deployed on a variety of computing devices, such as on edge devices or in data centers, than the Transformer. For example, because the operations performed by the spatial interaction mechanism described herein use static parameterization (as opposed to the Transformer, which uses a dynamically parallelized spatial interaction mechanism), the operations can be much more easily mapped to machine learning hardware accelerators, such as GPUs, TPUs, or other ASICs, that are configured to perform matrix-vector multiplication in hardware. As a result, the neural network described herein can be easily deployed on devices equipped with one or more such accelerators and used to perform inference with low latency.
[0008] Furthermore, the system disclosed herein utilizing a feedforward neural network with a simple spatial interaction mechanism achieves better performance (e.g., accuracy) for any given number of parameters / FLOPs (floating point operations per second) than existing feedforward neural network architectures.
[0009] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 illustrates an exemplary neural network system. [Diagram 2] FIG. 1 illustrates an exemplary architecture of a neural network. [Diagram 3] FIG. 2 is a flow diagram of an example process for processing an input sequence using one of multiple blocks in a neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Like reference numbers and designations in the various drawings indicate like elements.
[0012] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that performs a machine learning task on a network input and generates a network output for the machine learning task.
[0013] The machine learning task can be any machine learning task that operates on a sequence of inputs, the network inputs, to generate a network output for the network inputs.
[0014] Some examples of machine learning tasks that the system can be configured to perform are:
[0015] As another example, the task may be an audio processing task. For example, if the input to the neural network is a sequence representing a verbal utterance, the output may be a classification output that classifies the verbal utterance into one or more categories from a set of categories. For example, if the input to the neural network is a sequence representing a verbal utterance, the output generated by the neural network may indicate whether a particular word or phrase (a "hot word") was spoken within the utterance. As another example, if the input to the neural network is a sequence representing a verbal utterance, the output generated by the neural network may identify the natural language used to speak the utterance.
[0016] As another example, the task may be a natural language processing or understanding task, such as an entailment task, a paraphrase task, a text similarity task, a sentiment task, a sentence completion task, a grammaticality task, etc., that operates on a sequence of text written in some natural language to generate a classification output that classifies the text into one or more categories from a set of categories.
[0017] As another example, the task may be a health prediction task where the input is a sequence derived from a patient's electronic health record data and the output is a prediction related to the patient's future health, such as a predicted treatment to be prescribed for the patient, the likelihood of the patient experiencing an adverse health event, or a predicted diagnosis for the patient.
[0018] As another example, a task may be an agent control task where the input is a sequence of observations or other data characterizing the state of the environment and the output defines an action to be performed by an agent in response to the most recent data in the sequence. An agent may be, for example, a real-world or simulated robot, a control system for an industrial facility, or a control system that controls different kinds of agents.
[0019] As another example, the task may be a genomics task where the input is a sequence representing fragments of DNA sequence or other molecular sequence and the output is an embedding of the fragments for use in a downstream task, e.g., by using unsupervised learning techniques on a dataset of DNA sequence fragments, or the output of that downstream task. Examples of downstream tasks include promoter site prediction, methylation analysis, predicting the functional impact of non-coding variants, etc.
[0020] As another example, the task may be a computer vision task where the input is an image or point cloud and the output is a computer vision output for the image or point cloud, e.g., a classification output including a respective score for each category of a plurality of categories, each score representing the likelihood that the image or point cloud contains an object belonging to that category.
[0021] When the input is an image or point cloud, the neural network may include an embedding sub-network that generates a respective embedding for each of multiple patches of the image or point cloud, and the input to a first block of the neural network may be a sequence including each respective embedding (and optionally one or more additional embeddings at predetermined locations that are subsequently used, e.g., to generate the output), where each patch includes intensity values of pixels in a different region of the input image.
[0022] In some cases, the machine learning task is a combination of multiple individual machine learning tasks, i.e., the system is configured to perform multiple different individual machine learning tasks, such as two or more of the machine learning tasks described above. For example, the system can be configured to perform multiple individual natural language understanding tasks, with the network input including an identifier for the individual natural language understanding task to be performed on the network input.
[0023] 1 illustrates an exemplary neural network system 100. Neural network system 100 is one example of a system, implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described below may be implemented.
[0024] The neural network system 100 can receive an input 102 and perform a machine learning task on the input 102 to generate an output 152 .
[0025] As explained above, the neural network system 100 can perform any of a variety of tasks that involve operating on an input 102, which is a sequence of inputs.
[0026] Neural network system 100 includes a neural network 150 that includes a sequence of blocks 110. Each block implements a collection of learned operations and operates on a respective input sequence that includes a respective input vector at each location of one or more locations.
[0027] That is, each block 110 operates on an input sequence 104 and produces a corresponding output sequence 134 .
[0028] Specifically, input sequence 104 has a respective input at each of a plurality of input positions in the input order, and output sequence 134 has a respective output at each of those positions in the input order, i.e., the block generates a respective output for each input position in input sequence 104.
[0029] In general, the input sequence 104 to a given block 110 can be any intermediate sequence of data generated by a neural network when performing a machine learning task on the input 102.
[0030] For example, the neural network 150 may include an embedding sub-network that includes one or more layers that generate a sequence of embeddings from the network input 102.
[0031] For example, when the input 102 is an image or point cloud, the neural network 150 may include an embedding sub-network that generates a respective embedding for each of a plurality of patches of the image or point cloud, and the input to a first block of the neural network 150 may be a sequence including each respective embedding. Each patch includes intensity values of pixels in different regions of the input image. For example, for each patch, the embedding sub-network may process the intensity values of the pixels in the patch using a learned transform to generate an embedding for the patch.
[0032] As another example, when the input 102 is a natural language sequence, the neural network 150 can include an embedding sub-network that generates a respective embedding for each of the text tokens in the natural language sequence, and the input to the first block of the neural network 150 can be a sequence including each respective embedding. The text tokens can include, for example, characters or other text symbols, word fragments, or words from the natural language sequence. For example, the embedding sub-network can map each token in a lexicon of tokens to a learned embedding for the token.
[0033] Optionally, in any of the above examples, when generating the sequence of embeddings, the embedding sub-network may append embeddings of placeholder inputs, e.g., predefined or learned "class" embeddings of given "class" inputs that are later used to generate the network outputs.
[0034] Thus, the input sequence 104 to the first block 110 in the neural network 150 comprises an embedded (i.e., numerical) representation of the network input 102, generated by the embedding sub-network.
[0035] The input sequence 104 to every block 110 after the first block may be the output sequence 134 generated by the preceding block.
[0036] The neural network 150 may also include an output sub-network that processes one or more of the vectors in the output sequence 134 generated by the final block 110 in the sequence to generate an output 152 of the machine learning task.
[0037] In an implementation in which the input sequence 104 to the first block 110 includes an embedding of class inputs, the output sub-network can process vectors from the output sequence 134 that correspond to the class inputs, i.e., vectors in the same positions as the embedding of the class inputs, to generate outputs 152 for the machine learning task.
[0038] In some other implementations, when the input sequence 104 to the first block 110 does not include an embedding of a class input, the output sub-network receives the output sequence 134 generated by the final block 110, applies a pooling operation, such as global average pooling, across the output vectors to generate a pooled output vector, and then processes the pooled output vectors to generate an output 152 of the machine learning task.
[0039] The output sub-network may have any suitable architecture that enables the sub-network to map vectors to the output 152 of the machine learning task. For example, when the task is a classification task, the output sub-network may include one or more fully connected layers, e.g., a linear layer, optionally followed by a softmax layer. When the output is a regression task, the output sub-network may include one or more fully connected layers, followed by a different type of output layer suitable for the regression task, e.g., a linear layer, a sigmoid output layer, etc.
[0040] To generate an output sequence 134 from an input sequence 104, each block 110 includes a spatial transformation unit 160 that applies a feed-forward spatial transformation to the input sequence to the unit that integrates information across multiple locations within the input sequence to the unit.
[0041] The operations performed by block 110 are described in more detail below with reference to FIGS.
[0042] FIG. 2 illustrates an example architecture 200 of the neural network 150.
[0043] As shown in FIG. 2, neural network 150 includes a sequence of L blocks 110 .
[0044] Each block 110 obtains an input sequence for that block, including a respective input vector at each of a plurality of positions. Each input vector has a first number d_model channels, i.e., d_model entries. That is, a vector with d_model dimensions can be considered as having d_model channels, with each entry of the vector being in one of the d_model channels.
[0045] More specifically, a first block 110 in the sequence of L blocks receives as input a sequence of input embeddings 202 (i.e., generated by the embedding sub-network described above) that includes a respective input embedding at each of a plurality of positions.
[0046] Each block after the first receives as input the output sequence produced by the preceding block 110 .
[0047] The output sequence 204 produced by the final block 110 may be provided, for example, as an input to the output sub-network described above.
[0048] To generate an output sequence from an input sequence, each block 110 applies a first set of transformations to each input vector at that position to generate, for each position, a respective transformed input vector at that position. Each respective transformed input vector has a second number d_ffn channels, i.e. has d_ffn entries. In general, the second number d_ffn channels is greater than the first number d_model channels. The first set of transformations is applied per channel, i.e., such that each transformed input vector at a given position depends only on the respective input vector at that given position, and not on any input vector at any other position.
[0049] As shown in the example of FIG. 2, the first set of transforms includes a normalization (“normalization”) operation 210 , a channel projection (“channel projection”) operation 220 , and a non-linear activation (“activation”) operation 230 .
[0050] That is, block 110 applies a normalization operation 210 to each respective input vector to generate a respective normalized input vector for each position, i.e., normalizes each input vector to generate a corresponding normalized input vector.
[0051] For each location, block 110 then applies a channel projection operation 220 to each normalized input vector at that location to generate an initial respective transformed input vector for that location having a second number of channels. In other words, block 110 projects each normalized input vector to a higher dimensionality by applying a first projection matrix to the normalized input vector.
[0052] For each position, block 110 applies a nonlinear activation operation 230, i.e., an activation function, to the initial respective transformed input vector for that position to generate a respective transformed input vector for that position. The activation function may be any suitable non-linear elementwise activation function, e.g., ReLU or GeLU.
[0053] In other words, given a matrix X containing the vectors in the input sequence, block 110 derives a matrix Z of the transformed input vectors as: Z = σ((norm(X))U) where norm(X) is the normalization operation 210, U is the first projection matrix, and σ represents the activation function.
[0054] Block 110 then generates a respective spatially transformed input vector at each of the multiple locations using a spatial transformation unit that applies a feedforward spatial transformation that integrates information across the multiple locations, i.e., the spatial transformation unit applies a feedforward spatial transformation that integrates information across the multiple locations, such that each spatially transformed input vector at a given location depends not only on the transformed input vector at the given location, but also on each transformed input vector at multiple different locations, without using any recursion-based or attention-based operations.
[0055] In other words, given a matrix Z of transformed input vectors, block 110 generates a matrix Z of spatially transformed input vectors
[0056]
number
[0057] of,
[0058]
number
[0059] where s represents the operation of the spatial transformation unit.
[0060] In the example of FIG. 2, the spatial transformation unit is a spatial gating unit 240 that, in addition to applying a feedforward spatial transformation, also applies a gating mechanism as part of generating the spatially transformed input vector.
[0061] More specifically, FIG. 2 illustrates an example of the operations performed by spatial gating unit 240.
[0062] In the example of Fig. 2, the spatial gating unit 240 applies a split operation 242 to generate, for each location, a respective first partial vector that includes a first subset of the second number of channels of the respective transformed input vector of that location, and a respective second partial vector that includes a second subset of the second number of channels of the respective transformed input vector of that location. That is, the unit 240 "splits" each transformed input vector into two along the channel dimension. For example, one partial vector can include the first half of the channels and the other partial vector can include the remaining half of the channels.
[0063] The spatial gating unit 240 then applies normalization 244 to each respective first partial vector, i.e., normalizes each first partial vector, to generate a respective normalized first partial vector for each position.
[0064] The spatial gating unit 240 then applies a feedforward spatial transformation 246 ("spatial projection") to each respective normalized first partial vector, which combines information across each respective normalized first partial vector at multiple locations, to generate a respective spatially transformed partial vector for each location.
[0065] As a particular example, unit 240 may determine the product of (i) a spatial transformation matrix and (ii) a matrix of each respective normalized first partial vector, and then add a bias term to this product to generate each respective spatially transformed partial vector for each location.
[0066] The spatial gating unit 240 then generates a respective spatially transformed input vector at each of the multiple locations from at least each respective spatially transformed partial vector and each respective second partial vector. For example, the spatial gating unit 240 can perform gating between each respective spatially transformed partial vector and each respective second partial vector of the multiple locations. That is, for each location, the unit 240 can determine an element-wise product 248 of (i) the respective spatially transformed partial vector of that location and (ii) the respective second partial vector of that location.
[0067] In other words, in this example,
[0068]
number
[0069] where f W,b (norm(Z2))=W(norm(Z2))+b, where Z1 is the matrix of the second subvector and Z2 is the matrix of the first subvector,
[0070]
number
[0071] denotes element-wise multiplication, norm(Z2) denotes the normalization operation, W is the spatial transformation matrix, and b is the bias term.
[0072] In some implementations, at the beginning of training, the training system can initialize W to a value close to zero, e.g., a randomly selected value within a threshold distance of zero, and b to a number of ones, i.e., at the beginning of training, f W,b (norm(Z2)) is approximately equal to 1, and s(Z) is approximately equal to Z1. Initializing these values in this way ensures that each block behaves like a regular FFN (feed-forward neural network) that does not mix information across vectors in the early stages of training, and only injects spatial information bit by bit across tokens during learning. This can improve the stability of the training process.
[0073] Thus, in some implementations (including the implementation shown in FIG. 2), unit 240 (and neural networks more generally) do not include any self-attention operations.
[0074] State-of-the-art Transformer neural networks typically use a multi-headed self-attention mechanism to perform spatial aggregation of information across inputs in a sequence. "Self-attention" refers to the operation of aggregating spatial information across tokens in a sequence by, for each token, applying attention across all tokens in the sequence to generate an updated token at that position. Multi-headed self-attention performs multiple instances of this operation in parallel and then combines the outputs of each to generate the final output of the operation. This type of spatial aggregation is dynamically parameterized based on the input representation, i.e., the weights in the weighted sum for each position depend on how many inputs there are in the input sequence and on the values in each input vector. Self-attention, and especially multi-headed attention, can consume a significant amount of computational power of a Transformer neural network.
[0075] By replacing multi-head self-attention with the spatial transformation described above, neural network 150 achieves results that rival or exceed those of Transformer neural networks while consuming much fewer computational resources and can be more easily deployed on a variety of computing devices, such as on edge devices or in data centers.
[0076] In some other implementations, unit 240 incorporates a self-attention mechanism in generating a respective spatially transformed input vector at each of the multiple locations. Specifically, in these implementations, unit 240 may apply a self-attention mechanism to the input sequence of the block to generate a respective attended input vector at each of the multiple locations, and then, for each location, determine a sum of (i) the respective spatially transformed partial vector for that location and (ii) the respective attended input vector for that location to generate a respective combined vector for that location. Unit 240 may then determine an element-wise product of (i) the respective combined vector for that location and (ii) the respective second partial vector for that location to generate a spatially transformed vector.
[0077] Therefore, to apply the self-attention mechanism to the input sequence of a block, unit 240 can optionally first normalize the input sequence and then apply a linear transformation to generate a respective query, key, and value for each input vector in the input sequence. Unit 240 then generates, for each position, a respective attention weight for each position in the sequence from the query for that position and the keys for all positions, and then calculates a weighted sum of the values for those positions, i.e., weighting each value with the corresponding attention weight. Optionally, if the dimensionality of the queries, keys, and values is too large, unit 240 can project each weighted sum to have a dimensionality equal to the subvector after the spatial transformation.
[0078] The inclusion of a self-attention mechanism can improve the performance of neural networks for certain tasks, such as processing long sequences of text containing several different sentences, or natural language tasks that require matching information between text located in different sentences.
[0079] Even when blocks 110 include a self-attention mechanism, the self-attention mechanism can be a "tiny" mechanism that is significantly more computationally efficient than the attention mechanism used by state-of-the-art Transformer neural networks because of the presence of unit 240. For example, the self-attention mechanism can be a single-head mechanism rather than the multi-head mechanism used by Transformer neural networks. In addition, the dimensionality of the query, key, and value can be significantly smaller than the dimensionality used by Transformer neural networks. In other words, even when blocks include a self-attention mechanism, each block 110 nevertheless consumes significantly less computational resources than a given block in a state-of-the-art Transformer neural network.
[0080] 2 represents an implementation in which unit 240 "splits" the transformed input vector. However, in some other implementations, unit 240 does not perform this "split" but instead operates directly on the transformed input vector.
[0081] In these implementations, unit 240 may apply normalization to each respective transformed input vector to generate a respective normalized transformed input vector for each location, and then apply a feed-forward spatial transformation to each respective transformed input vector that combines information across each respective transformed input vector at multiple locations to generate a respective spatially transformed vector for each location.
[0082] For example, applying a feedforward spatial transformation to each respective transformed input vector may include determining the product of (i) a spatial transformation matrix and (ii) a matrix of each respective transformed input vector, and adding a bias term to the product to generate each respective spatially transformed vector for each location.
[0083] Unit 240 may then generate a respective spatially transformed input vector at each of the multiple locations from at least each respective spatially transformed vector and each respective transformed input vector. For example, unit 240 may determine, for each location, an element-wise product of (i) the respective spatially transformed vector for that location and (ii) the respective transformed input vector for that location.
[0084] When unit 240 also applies self-attention, unit 2140 may apply the self-attention mechanism to the input sequence of the block to generate a respective attended input vector at each of the multiple locations, and for each location, determine the sum of (i) the respective spatially transformed vector for that location and (ii) the respective attended input vector for that location, and determine the element-wise product of (i) the respective combined vector for that location and (ii) the respective transformed input vector for that location, to generate a respective combined vector for that location.
[0085] Block 110 then generates its output sequence by applying, for each location, a second set of transforms to the respective spatially transformed input vector at that location to generate a respective output vector at that location. The second set of transforms is applied per channel, i.e., such that the output vector at a given location depends only on the respective spatially transformed input vector at that given location and not on any respective spatially transformed input vector at any other location.
[0086] In the example of FIG. 2, the second set of transforms includes a channel projection (“channel projection”) operation 260 and a skip connection 270 .
[0087] That is, applying the second set of transformations to each input vector at that location includes applying a channel projection operation 260, i.e., applying a second projection matrix, to each spatially transformed input vector at that location to generate an initial respective output vector at that location having a first number of channels, i.e., to reduce the number of channels from d_ffn, or from half d_ffn (or another fraction of d_ffn) to d_model when a ``split'' operation is included.
[0088] Applying the second set of transformations then involves applying a skip connection 270 to the initial respective output vector at each location to add the initial respective output vector at that location to the respective input vector to that location to generate a respective output vector for that location.
[0089] Thus, as described above with reference to FIG. 2, each block 110 first projects its input into a higher dimensionality, i.e., a larger number of channels, then applies a feedforward spatial transformation in that higher dimensional space, and then projects the output of the feedforward spatial transformation back into the original dimensionality.
[0090] In other words, the matrix Y consisting of the output vectors in the output sequence is
[0091]
number
[0092] where V is the second projection matrix.
[0093] As can be seen from the above, when self-attention is not included, the parameterization of the spatial transformation is static, i.e., it does not depend on the input, but is fixed after the neural network is trained. That is, the projection matrix has a fixed number of dimensions and is fixed after the training is completed. To allow for a fixed size matrix, when the system operates on variable length data, e.g., when the input sequence is variable length, the system can pad the data with zero vectors or garbage vectors so that each sequence operated by the neural network 150 has a fixed size. For example, the system can pad the original network input to the system with zero vectors, and the embedding sub-network can add a predefined embedding after the embedding of the input in the network input to generate a fixed length sequence of embeddings.
[0094] Because the spatial projection matrix has a fixed dimensionality, the spatial transformation maintains the location information throughout the neural network. Thus, unlike a transformer neural network, the embeddings generated by the embedding sub-network do not need to encode location information, i.e., information that identifies the location of a given embedding within the sequence of embeddings. In other words, the embeddings within the sequence of embeddings are not generated using any location embeddings or any other location information, which further simplifies the architecture of the neural network 150.
[0095] 3 is a flow diagram of an exemplary process 300 for processing an input sequence using one of a number of blocks in a neural network. For convenience, the process 300 is described as being performed by a system of one or more computers located at one or more locations. For example, a neural network system including a sequence of blocks appropriately programmed in accordance with this specification, such as the neural network system 100 of FIG. 1, can perform the process 300.
[0096] During processing of a network input by the neural network, the block obtains an input sequence for the block including a respective input vector at each of a plurality of locations, each input vector having a first number d_model channels, i.e., d_model entries (step 302).
[0097] For each position, the block applies a first set of transforms to a respective input vector at that position to generate a respective transformed input vector at that position, each respective transformed input vector having a second number d_ffn channels, i.e., d_ffn entries (step 304). In general, the second number of channels is greater than the first number of channels. The first set of transforms is applied channel-by-channel, i.e., such that each transformed input vector at a given position depends only on the respective input vector at that given position and not on any input vector at any other position.
[0098] The block then generates a respective spatially transformed input vector at each of the multiple locations (step 306). Generating the spatially transformed input vectors includes applying a feedforward spatial transformation that integrates information across the multiple locations, such that each spatially transformed input vector at a given location depends not only on the transformed input vector at the given location, but also on each transformed input vector at multiple different locations. The feedforward spatial transformation does not use any recursion-based or attention-based operations.
[0099] The block then generates a sequence of outputs for the block by applying, for each location, a second set of transforms to the respective spatially transformed input vector at that location to generate a respective output vector at that location (step 308). The second set of transforms is applied per channel, i.e., such that the output vector at a given location depends only on the respective spatially transformed input vector at that given location and not on any respective spatially transformed input vector at any other location.
[0100] Before using a neural network to perform a machine learning task, a training system trains the neural network to perform the task, i.e., to determine post-training values of the neural network's parameters, i.e., parameters of blocks in a sequence, output sub-networks, and embedding sub-networks. For example, the training system can train the neural network from scratch using traditional machine learning techniques to minimize a loss function of the task, e.g., cross-entropy loss, negative log-likelihood, etc., on the training data of the task. As another example, the training system can first pre-train a mixer neural network on an unsupervised objective, and then fine-tune the mixer neural network on the training data of the task. As yet another example, the training system can train the mixer neural network on both unlabeled data and the training data of the task through semi-supervised learning.
[0101] During training, the training system can incorporate any number of techniques to improve the speed, effectiveness, or both of the training process. For example, the system can use dropout, label smoothing, or both to reduce overfitting. As another example, the system can perform training using a distributed architecture that trains multiple instances of the neural network in parallel. Furthermore, as described above, the system can first pre-train the neural network through unsupervised learning on a large unsupervised dataset to, for example, minimize the BERT loss or other unsupervised loss, and then fine-tune the neural network on task-specific training data to optimize the loss function for that task.
[0102] The term "configured" is used herein with respect to systems and computer program components. A system of one or more computers is configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or action. One or more computer programs are configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.
[0103] The embodiments and functional operations of the subject matter described herein can be implemented as digital electronic circuitry, as tangibly embodied computer software or firmware, as computer hardware including the structures disclosed herein and their structural equivalents, or as a combination of one or more of them. The embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.
[0104] The term "data processing apparatus" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). An apparatus may optionally also include, in addition to hardware, code that creates an execution environment for a computer program, such as code constituting a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.
[0105] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data, for example in one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple linked files, for example files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.
[0106] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular manner, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.
[0107] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and running on the same computer or computers.
[0108] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
[0109] A computer suitable for executing a computer program can be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated in special-purpose logic circuitry. In general, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data therefrom or transfer data thereto, or both. However, a computer need not have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.
[0110] Suitable computer-readable media for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM disks and DVD-ROM disks.
[0111] To enable interaction with a user, the embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can input to the computer. Other types of devices can also be used to enable interaction with the user, for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input. In addition, the computer can also interact with the user by sending documents to and receiving documents from a device being used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message in return from the user.
[0112] A data processing device for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling common computationally intensive parts of machine learning training or production, i.e., inference, workloads.
[0113] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.
[0114] The embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user may interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.
[0115] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, e.g., a result of a user interaction, may be received at the server from the device.
[0116] Although this specification contains many specific details of implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in the context of separate embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination, and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.
[0117] Similarly, although operations are depicted in the figures and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequence shown, or that all of the operations shown be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.
[0118] Specific embodiments of the present subject matter have been described above. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. As an example, the processes depicted in the appended figures do not necessarily require the particular order shown, or sequence, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]
[0119] 100 Neural Network System 102 Input 104 Input Series 110 block, first block, last block 134 Output series 150 Neural Networks 152 Output 160 Spatial Transformation Unit 200 Architecture 202 Input embedding sequence 204 Output series 210 Normalization operations 220 Channel Projection Calculation 230 Nonlinear Activation Operations 240 Spatial Gating Unit 242 Division Operation 244 Normalization 246 Feedforward spatial transformation 248 Element-wise product 260 Channel Projection Calculation 270 Skip Connection 300 processes 2140 units
Claims
1. 1. A system for performing a machine learning task on a network input to generate a network output, comprising: one or more computers; and one or more storage devices that store instructions that, when executed by the one or more computers, cause the one or more computers to: A neural network configured to perform the machine learning task, the neural network comprising a plurality of blocks, each block comprising: obtaining an input sequence for the block comprising a respective input vector at each location of a plurality of locations, each input vector having a first number of channels; applying, for each location, a first set of transforms to the respective input vector at the location to generate a respective transformed input vector at the location, each respective transformed input vector having a second number of channels; generating a respective spatially transformed input vector at each of the plurality of locations, the generating comprising applying a feedforward spatial transformation that integrates information across the plurality of locations; and generating, for each location, a sequence of outputs for the block by applying a second set of transforms to the respective spatially transformed input vectors at the location to generate a respective output vector at the location; a neural network configured to perform operations including: A system that implements the above.
2. The neural network comprises: one or more output layers configured to process one or more of each of the respective output vectors in the output sequence of a final block of the plurality of blocks to generate the network output. The system of claim 1 further comprising:
3. applying the spatial transformation generating, for each location, a respective first partial vector comprising a first subset of the second number of channels of the respective transformed input vector for the location, and a respective second partial vector comprising a second subset of the second number of channels of the respective transformed input vector for the location; applying a normalization to each said respective first partial vector to generate a respective normalized first partial vector for each location; applying a feedforward spatial transformation to each of the respective normalized first partial vectors to generate a respective spatially transformed partial vector for each location, the feedforward spatial transformation combining information across each of the respective normalized first partial vectors at the multiple locations; generating a respective spatially transformed input vector at each of the plurality of locations from at least each of the respective spatially transformed partial vectors and each of the respective second partial vectors; The system of claim 1 , comprising:
4. applying a feedforward spatial transformation to each said respective normalized first sub-vector; determining a product of (i) a spatial transformation matrix and (ii) a matrix of each said respective normalized first sub-vector; adding a bias term to said product to generate each of said respective spatially transformed subvectors for each location; The system of claim 3, comprising:
5. Generating the respective spatially transformed input vector at each of the plurality of locations comprises, for each location: determining an element-wise product of (i) said respective spatially transformed partial vectors of said locations and (ii) said respective second partial vectors of said locations; The system of claim 3, comprising:
6. generating the respective spatially transformed input vectors at each of the plurality of locations, applying a self-attention mechanism to the input sequence of the block to generate a respective attended input vector at each location of the plurality of locations; For each position, determining a sum of (i) a spatially transformed partial vector of each of the locations and (ii) an attended input vector of each of the locations to generate a combined vector for each of the locations; determining an element-wise product of (i) said respective combined vectors of said locations and (ii) said respective second partial vectors of said locations; The system of claim 3, comprising:
7. applying the spatial transformation applying a normalization to each said respective transformed input vector to generate a respective normalized transformed input vector for each position; applying a feedforward spatial transformation to each of the respective transformed input vectors to generate a respective spatially transformed vector for each location; generating, from at least each of the respective spatially transformed vectors and each of the respective transformed input vectors, the respective spatially transformed input vectors at each of the plurality of locations; The system of claim 1 , comprising:
8. applying a feedforward spatial transformation to each said respective transformed input vector; determining the product of (i) a spatial transformation matrix and (ii) a matrix of each said respective transformed input vector; adding a bias term to said product to generate each said respective spatially transformed vector for each location; The system of claim 7, comprising:
9. Generating the respective spatially transformed input vector at each of the plurality of locations comprises, for each location: determining an element-wise product of (i) said respective spatial transformed vectors of said locations and (ii) said respective transformed input vectors of said locations; The system of claim 7, comprising:
10. generating the respective spatially transformed input vectors at each of the plurality of locations, applying a self-attention mechanism to the input sequence of the block to generate a respective attended input vector at each location of the plurality of locations; For each position, determining a sum of (i) a spatially transformed vector for each of the locations and (ii) an attended input vector for each of the locations to generate a combined vector for each of the locations; (i) determining an element-wise product of said respective combined vectors for said locations and (ii) said respective transformed input vectors for said locations; The system of claim 7, comprising:
11. applying, for each location, a first set of transformations to the respective input vector at the location; applying a normalization to each said respective input vector to generate a respective normalized input vector for each position; applying a first projection matrix to the respective normalized inputs at the locations to generate, for each location, an initial respective transformed input vector for the location having the second number of channels; applying, for each location, an activation function to the initial respective transformed input vector for the location to generate the respective transformed input vector for the location; The system of claim 1 , comprising:
12. applying, for each location, a second set of transformations to the respective input vector at the location; applying a second projection matrix to the respective spatially transformed input vectors at the locations to generate an initial respective output vector for the locations having the first number of channels. The system of claim 1 , comprising:
13. applying, for each location, a second set of transformations to the respective input vector at the location; adding the initial respective output vector of the location to the respective input vector to the location to generate the respective output vector for the location. The system of claim 12, comprising:
14. 2. The system of claim 1, wherein the input sequence of a first block of the plurality of blocks is a sequence of embeddings representing the network input.
15. The system of claim 14 , wherein the embedding is not generated using any positional embedding.
16. 15. The system of claim 14, wherein the network input is an image, and the sequence of embeddings comprises a respective embedding representing each patch of a plurality of patches from the image.
17. The machine learning task operates on a network input, which is a sequence of inputs, to generate a network output for the network inputs, the machine learning task comprising: an audio processing task, wherein the network input is a sequence representing a verbal utterance and the network output is a classification output that classifies the verbal utterance into one or more categories from a set of categories; a health prediction task, where the network input is a sequence obtained from a patient's electronic health record data and the network output is a predicted diagnosis for that patient; an agent control task, wherein the network inputs are sequences of observations or other data characterizing the state of an environment, and the network outputs define actions to be taken by an agent in response to the most recent data in the sequences; A genomics task, wherein the network input is a sequence representing fragments of a DNA sequence or other molecular sequence, and the network output is an embedding of the fragments for use in a downstream task, or the output of the downstream task; or A computer vision task, wherein the network input is an image or a point cloud, and the output is a computer vision output for the image or the point cloud. The system of claim 1 , comprising:
18. 2. The system of claim 1, wherein the machine learning task is an image classification task, the network input is an image, and the network output is a classification output including a respective score for each category of a plurality of categories, each score representing a likelihood that the image contains an object belonging to that category.
19. One or more computer readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to implement the neural network of any one of claims 1 to 18.
20. 1. A method implemented by one or more computers, comprising: receiving a network input; processing said network inputs using a neural network according to any one of claims 1 to 18 to generate network outputs; A method comprising:
Citation Information
Patent Citations
Spatial transformation module
JP2018514856A
Attention-based decoder-only sequence transduction neural networks
WO2019084551A1
