Neural network with linear cycles and feed-forward units

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing sequence data using a neural network having a linear loop and a feed-forward unit. In one aspect, a system is provided that performs a machine learning task on a network input to generate a network output. The system includes one or more computers and one or more storage devices that store instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network configured to perform machine learning tasks. The neural network comprises a plurality of layer blocks, each layer block comprising: (i) a linear cyclic layer, and (ii) one or more feed-forward layers. Each layer block is configured to perform operations including: receiving an input sequence of layer blocks; and generating an output sequence of the layer blocks.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 441,434, filed on January 26, 2023. The disclosure of that prior application is considered part of and incorporated by reference into the disclosure of this application. Background Art

[0002] This specification relates to using machine learning models to process data.

[0003] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate an output based on the received input and the values ​​of the model parameters.

[0004] Some machine learning models are deep models that use multiple layers to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations that uses a neural network to generate a network output in response to received network input.

[0006] The examples of neural networks described in this specification use linear recurrent and feedforward units to process, generate, or otherwise manipulate sequence data (e.g., long sequence data) to perform machine learning tasks. Specifically, this specification introduces a linear recurrent layer, referred to as a "linear recurrent unit" (or LRU), which can be effectively and efficiently used as the main neural network layer of a deep sequence model - the neural network described in this article is such a model. In the described example, the neural network includes a series of layer blocks, for example, in a residual configuration, each layer block includes: (i) a linear recurrent layer (or LRU), and (ii) one or more feedforward layers. The linear recurrent layer implements the linear cycle of the layer block. The feedforward layer implements nonlinearity for the layer block using, for example, a multilayer perceptron (MLP) and / or a gated linear unit (GLU). Feedforward layers are neural network layers that do not implement recurrent, for example, so that their output at the corresponding position (or time step) does not depend on their output at any previous position (or time step). In other words, a feedforward layer is a layer of a neural network that transmits information unidirectionally - from one layer to the next - without any feedback. The following paper provides a brief review of feed-forward neural networks: Sazli, Murat H. “A brief review of feed-forward neural networks.” Communications Facultyof Sciences University of Ankara Series A2-A3 Physical Sciences andEngineering 50.01 (2006).

[0007] By offloading nonlinearities to feedforward layers, i.e., avoiding the use of nonlinear loops, neural networks can achieve performance on machine learning tasks that is unattainable by currently available neural network architectures (e.g., existing recurrent neural network (RNN) architectures), particularly on tasks involving long-range reasoning. For example, neural networks can implement deep models that include a large number of layer block sequences (or stacks), e.g., 2 or more layer blocks, 5 or more layer blocks, 10 or more layer blocks, 20 or more layer blocks, 50 or more layer blocks, 100 or more layer blocks, 200 or more layer blocks, 500 or more layer blocks, 1000 or more layer blocks, to efficiently process long program sequences. A long program sequence generally refers to a sequence of 2,000 or more data elements, e.g., 4,000 or more data elements, 8,000 or more data elements, 16,000 or more data elements, 32,000 or more data elements, 64,000 or more data elements, or 128,000 or more data elements.

[0008] This specification also provides several modifications to linear recurrent layers (LRUs), such as diagonalization, stable exponential parameterization, normalization, and parallelization, which can improve the stability of neural networks during training, for example, by mitigating vanishing and exploding gradients, as well as significantly improve the speed and accuracy of neural networks at inference time.

[0009] These and other aspects of the subject matter described in this specification are summarized below.

[0010] In a first aspect, a system for performing a machine learning task on a network input to generate a network output is provided. The system includes one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to implement: a neural network configured to perform the machine learning task, the neural network including a plurality of layer blocks, each layer block including: (i) a linear recurrent layer, and (ii) one or more feed-forward layers, wherein each layer block is configured to perform operations including: receiving an input sequence for the layer block, the input sequence including a corresponding input at each of a plurality of input positions; and generating an output sequence for the layer block, the output sequence including a corresponding output at each of a plurality of output positions, the generating including: using the linear recurrent layer of the layer block to process the input sequence to generate an initial output sequence, the initial output sequence including a corresponding initial output at each of the plurality of input positions; and using the one or more feed-forward layers of the layer block to process the initial output sequence to generate the output sequence.

[0011] In some implementations of the system, the plurality of layer blocks are organized into a sequence, and for each layer block that is not the first layer block in the layer block sequence, the input sequence for the layer block is the output sequence of the previous layer block in the layer block sequence.

[0012] In some implementations of the system, the network input is an input sequence for the first layer block in the sequence of layer blocks.

[0013] In some implementations of the system, the neural network further includes an encoder configured to perform operations including: receiving the network input; and processing the network input to generate an input sequence for the first layer block in the sequence of layer blocks.

[0014] In some implementations of the system, the encoder is a linear encoder.

[0015] In some implementations of the system, the neural network further includes one or more output layers configured to perform operations including: receiving an output sequence of a last layer block in the sequence of layer blocks; and processing the output sequence of the last layer block to generate the network output.

[0016] In some implementations of the system, the one or more output layers are linear layers.

[0017] In some implementations of the system, the linear recurrent layer of each layer block is parameterized by multiple matrices, and using the linear recurrent layer of the layer block to process the input sequence to generate the initial output sequence includes: for each input position: obtaining a current hidden state (in the case of a first input position, the current hidden state can be initialized in any way, for example, initialized to a default value, such as zero); calculating a first matrix-vector product between: (i) the first matrix of the linear recurrent layer, and (ii) the current hidden state; receiving an input for the input position; calculating a second matrix-vector product between: (i) the second matrix of the linear recurrent layer, and (ii) the input for the input position; and adding the first matrix-vector product and the second matrix-vector product to generate an updated hidden state.

[0018] In some implementations of the system, the first matrix of each linear recurrent layer is a diagonal matrix parameterized by a plurality of eigenvalues.

[0019] In some implementations of the system, each eigenvalue has a magnitude less than one.

[0020] In some implementations of the system, the eigenvalues ​​include complex eigenvalues.

[0021] In some implementations of the system, each complex eigenvalue is represented as an exponential function parameterized by a complex variable.

[0022] In some implementations of the system, the linear recurrent layer of each layer block is further parameterized by a normalization parameter, and wherein using the linear recurrent layer of the layer block to process the input sequence to generate the initial output sequence further includes, for each input position: calculating the element-wise product between: (i) the normalization parameter of the linear recurrent layer, and (ii) the second matrix-vector product, and wherein adding the first matrix-vector product and the second matrix-vector product to generate the updated hidden state includes: adding the first matrix-vector product and the element-wise product to generate the updated hidden state.

[0023] In some implementations of the system, using the linear recurrent layer of the layer block to process the input sequence to generate the initial output sequence further includes: for each input position: computing a third matrix-vector product between: (i) the third matrix of the linear recurrent layer, and (ii) the updated hidden state; computing a fourth matrix-vector product between: (i) the fourth matrix of the linear recurrent layer, and (ii) the input for the input position; and adding the third matrix-vector product and the fourth matrix-vector product to generate the initial output for the input position.

[0024] In some implementations of the system, the neural network has been trained to perform the machine learning task on multiple training examples, each training example comprising: (i) a corresponding training network input, and (ii) a corresponding target network output, and wherein the training comprises: processing the training network input for each training example using the neural network to generate a training network output that is an estimate of the corresponding target network output for the training example; determining a gradient of an objective function that depends on the training network output and the target network output for each training example (e.g., the objective function may comprise a corresponding similarity measure between the training network output and the target network output for each training example); and using the gradients of the objective function to update corresponding parameters of one or more of the linear recurrent layers.

[0025] In some implementations of the system, for each updated linear recurrent layer, the training further includes initializing the complex variable of each complex eigenvalue parameterizing the linear recurrent layer to approximately zero.

[0026] In some implementations of the system, for each updated linear recurrent layer, the training further includes: initializing the normalization parameters parameterizing the linear recurrent layer using the eigenvalues ​​of the first matrix of the linear recurrent layer.

[0027] In some implementations of the system, for each layer block, the number of input positions of the layer block is equal to the number of output positions of the layer block.

[0028] In some implementations of the system, the one or more feed-forward layers of each layer block include: (i) a multilayer perceptron, (ii) a gated linear unit, or (iii) both.

[0029] In some implementations of the system, the network input is a long program train.

[0030] In some implementations of the system, the network input includes two thousand or more data elements.

[0031] In some implementations of the system, the data element is a word-gram.

[0032] In some implementations of the system, the linear recurrent layer of each layer block is configured to perform a parallel scan on the input sequence to generate the initial output sequence.

[0033] In some implementations of the system, the parallel scanning is a work-efficient parallel scanning.

[0034] In some implementations of the system, the layer block is implemented by multiple processors operating in parallel.

[0035] In some implementations of the system, the network input includes data to be compressed, and the network output includes a compressed version of the data.

[0036] In some implementations of the system, the network input includes audio data representing an audio waveform, and the network output includes a spectrogram of the audio waveform.

[0037] In some implementations of the system, the network input includes: (i) intensity values ​​of pixels of an input image, or (ii) encoded representations of the intensity values ​​of the pixels of the input image, and the network output is: (i) an image classification output that classifies the input image into one of a plurality of object categories; (ii) an object detection output that specifies the coordinates of one or more bounding boxes in the input image that are predicted to contain objects; or (iii) a segmentation output that classifies each pixel in the input image into one of a plurality of categories.

[0038] In some implementations of the system, the machine learning task is an agent control task for an agent in an environment, the network input is a sequence of observations characterizing the state of the environment, and the network output defines an action to be performed by the agent in response to the most recent observation in the sequence.

[0039] In some implementations of the system, the machine learning task is a health status prediction task, the network input comprises data derived from a patient's electronic health record, and the network output is a prediction output defining: (i) a predicted treatment that should be taken for the patient, (ii) the likelihood that the patient will experience an adverse health status event, or (iii) a predicted diagnosis for the patient.

[0040] In a second aspect, one or more computer storage media are provided. The one or more computer storage media store instructions that, when executed by one or more computers, cause the one or more computers to implement the neural network of any one of the above systems.

[0041] In a third aspect, a method performed by one or more computers is provided, comprising: receiving a network input; and processing the network input using a neural network according to any one of the above systems to generate a network output corresponding to the network input.

[0042] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.

[0043] Transformers are generally efficient and scalable, but can be prohibitively expensive to train and deploy on long sequences. This is because the standard self-attention mechanism of the Transformer is sensitive to sequence length. It expands and contracts quadratically, and involves In contrast, recurrent neural networks (RNNs) scale linearly with sequence length during inference. , and is generally faster than the Transformer even for moderate sequence lengths. That being said, traditional RNNs and their gated variants can be difficult (or infeasible) to train in practice. Specifically, traditional RNNs that implement nonlinear recurrence (e.g. using tanh or sigmoid activation functions) suffer from the vanishing gradient problem, which occurs when the gradients of the recurrent weights become extremely small as they backpropagate through the RNN. The vanishing gradient problem generally inhibits RNNs from learning long-range dependencies in sequential data. As a result, RNNs have currently been replaced by other models.

[0044] This specification describes linear recurrent layers, called "linear recurrent units" (or LRUs), which, in the implementations described herein, overcome the vanishing gradient problem, are highly parallelizable (e.g., so as to be used for efficient parallel scanning), are capable of learning long-range interactions in data, and can be used in deep, high-performance, and efficient neural network models. For example, the neural network architectures described in this specification can perform machine learning tasks on sequence data involving 16,000 or more interacting tokens, such as machine learning tasks in the Long Range Arena (LRA) such as PathFinder and PathX. The performance of neural networks in selection tasks in the LRA is provided herein, showing significant improvements over conventional RNN architectures.

[0045] The examples of neural networks described in this article provide a general framework for theoretically sound and computationally efficient long-range reasoning using stacks of linear recurrent layers interleaved with feed-forward layers. The neural network architecture described in this article can be applied to a wide range of machine learning tasks, such as language modeling, natural language processing (e.g., text classification), machine translation, reading comprehension, question answering, video processing, and computer vision.

[0046] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1A is a block diagram of an example system configured to perform machine learning tasks using a neural network.

[0048] Figure 1B is a block diagram of an example residual layer block.

[0049] Figure 1C is a diagram of an example linear recurrent layer that implements linear recurrence.

[0050] Figure 2A is a flow chart of an example process for processing network inputs to generate network outputs using a neural network.

[0051] Figure 2B is a flowchart of an example process for processing an input sequence using a residual layer block to generate an output sequence.

[0052] Figure 2C is a flowchart of an example process for processing an input sequence using a linear recurrent layer to generate an initial output sequence.

[0053] Figure 3A is a block diagram of an example training engine configured to train a neural network to perform a machine learning task.

[0054] Figure 3B is a flowchart of an example process for training a neural network to perform machine learning tasks.

[0055] Figures 4A to 4C Shown are the eigenvalues ​​of the transfer matrix of the linear recurrent layer after Glorot initialization.

[0056] Figure 4D Shown are the eigenvalues ​​of the diagonal transfer matrix of the linear recurrent layer after equivalent spectral initialization.

[0057] Figures 5A to 5C is an experimental plot showing the relationship between training loss, training accuracy, and test accuracy versus training iterations for a neural network trained to perform the PathX task.

[0058] Figure 6 An example JAX implementation of a linear recurrent layer is shown.

[0059] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0060] Recurrent neural networks (RNNs) have played an important role since the rise of deep machine learning and are a natural choice when modeling sequential data. However, while these neural networks have powerful theoretical properties, such as Turing completeness, they are difficult to train in practice. Specifically, RNNs suffer from the vanishing and exploding gradient problem, which makes it difficult for these models to learn long-range dependencies in the data. Several techniques have been developed to try to alleviate this problem, including orthogonal RNNs, unitary RNNs, and gating mechanisms such as long short-term memory (LSTM) and gated recurrent units (GRU). Despite this, due to the sequential nature of the computation of these models, their optimization speed remains slow, which introduces many difficulties when scaling.

[0061] In recent years, Transformers have become increasingly prominent in sequence modeling tasks and have achieved success in a wide range of applications, such as large language models (LLMs). Compared to traditional (non-linear) recurrent layers, attention layers are generally easier to scale and parallelize during training, and they do not suffer from the vanishing gradient problem. However, one problem with attention layers is that their computational and memory costs scale quadratically with the length of the sequence. Therefore, deploying a Transformer on a long sequence can be expensive. RNNs scale linearly with sequence length. , which is generally faster than the Transformer at inference time even for moderate sequence lengths.

[0062] Inspired by these problems, this specification introduces a linear recurrent layer, called a "Linear Recurrent Unit" (or LRU), which, in the implementation described below, overcomes the vanishing and exploding gradient problems and can be used for deep, high-performance, and efficient neural network models, and is capable of learning long-range interactions in sequence data. Neural networks utilizing the linear recurrent layers described herein can provide memory savings, lower latency, and higher throughput in other systems where memory is limited and computational resources are scarce, such as smartphones, smartwatches, personal computers (PCs), laptops, tablets, wearable devices, edge computing devices, small computers, etc. For example, since their computational and memory costs are as low as scalability, so they can be implemented in systems where other neural network models (such as Transformers) are impractical or infeasible.

[0063] The example of a neural network utilizing linear recurrent layers illustrates a number of advantages and theoretical insights related to sequence data processing and RNNs in general, including but not limited to one or more of the following:

[0064] Linear loop.Removing nonlinearity from the loop, i.e., using linear loops, provides a significant improvement in inference accuracy. This paper shows that stacking linear loop layers interleaved with feed-forward layers can model complex nonlinear sequence-to-sequence mappings without involving nonlinearity in the loop. In general, abandoning nonlinearity in the loop does not harm the expressiveness of RNNs and brings several advantages, such as being able to directly control the rate at which gradients vanish or explode, enabling parallelization of training and inference, and so on.

[0065] Complex diagonal circulant matrix. Dense linear recurrent layers (e.g., those with depthwise connections) can be reparameterized to a complex diagonal form. This complex diagonal form reduces the number of parameters in the linear recurrent layer while also speeding up training and inference, for example, because diagonal matrices are computationally cheaper to compute than dense matrices. This is achieved without compromising the expressiveness of the neural network or its initialization characteristics. Additionally, diagonally parameterized linear recurrent layers allow for highly parallel unrolling of the loop, for example, using work-efficient parallel scans, to significantly speed up training and inference.

[0066] Stability index parameterization. Using an exponential parameterization for the diagonal transfer matrix has many benefits, such as achieving stability during training, which in turn allows modifying the initialization distribution, e.g., to facilitate long-range reasoning and improve performance.

[0067] Standardization. Normalizing the hidden activations in the forward pass can further improve the performance of neural networks on tasks involving long-range dependencies.

[0068] These and other features associated with the systems and methods disclosed in this specification are described in more detail below.

[0069] Figure 1A is a block diagram of an example system 100 configured to perform machine learning tasks using a neural network 200. System 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0070] At a high level, the system 100 is configured to: receive network input 102; and processing the network input 102 using the neural network 200 to generate a network output 104. Here, is a function representing the parameter model of the neural network 200, which is composed of the network parameters Generally speaking, the network parameters Included are the corresponding (learnable) parameters for each neural network layer in neural network 200. The learnable parameters associated with linear recurrent layer 222 are described in detail herein.

[0071] Neural network 200 is configured to perform a machine learning task on network input 102 to generate network output 104. A machine learning task can be any machine learning task that (i) operates on network input 102 as an input sequence, (ii) generates network output 104 as an output sequence, or (iii) both. For convenience, examples of machine learning tasks that system 100 can perform using neural network 200 are described at the end of this specification.

[0072] It is particularly noteworthy that the neural network 200 can efficiently process network inputs 102 comprising long program sequences. A long program sequence generally refers to a sequence of 2,000 or more data elements, such as 4,000 or more data elements, 8,000 or more data elements, 16,000 or more data elements, 32,000 or more data elements, 64,000 or more data elements, or 128,000 or more data elements. For example, the neural network 200 can perform machine learning tasks, such as machine learning tasks in long-range arenas (LRAs) such as PathFinder and PathX, on network inputs 102 comprising 16,000 or more interacting word sequences. Reference Figures 3A to 5C The performance of neural network 200 on selected machine learning tasks in LRA is described. Here, various modifications and reparameterizations of linear recurrent layer 222 are described step by step to improve the inference accuracy and other performance metrics of neural network 200 on long-range tasks. Details of specific tasks in LRA, such as training and test datasets, are provided by Yi Tay, et al., “Long-Range Arena: A Benchmark for Efficient Transformers.” arXiv preprint arXiv:2011.04006 (2020).

[0073] Note that in implementations where network input 102 is a sequence of inputs, neural network 200 may receive all of network input 102 simultaneously, or may receive each of network input 102 sequentially, e.g., in a streaming fashion. Similarly, in implementations where network output 104 is a sequence of outputs, neural network 200 may generate all of network output 104 simultaneously, or may generate each of network output 104 sequentially. As one example, in some implementations, neural network 200 may be used for batch (or offline) inference. In these cases, neural network 200 may receive and process each of network input 102 simultaneously, and then generate each of network output 104 simultaneously. As another example, in some implementations, neural network 200 may be used for online inference. In these cases, network input 102 may be a data stream including a corresponding input at each of a plurality of time steps, and network output 104 may include a corresponding output for each time step. Then, at each time step, neural network 200 may receive an input for that time step in network input 102 and process the input to generate a corresponding output for that time step in network output 104 .

[0074] refer to Figure 1A , the neural network 200 includes a sequence (or stack) of layer blocks, each layer block including one or more neural network layers. More specifically, the sequence of layer blocks includes: an input layer block 210, a plurality of intermediate layer blocks 220-1 to 220-N, and an output layer block 230. In this example, a skip connection (e.g., an identity skip connection) places the intermediate layer block 220 in a residual configuration, forming a residual network 201 of residual layer blocks 220-1 to 220-N. In general, the residual configuration enables the neural network 200 to implement deep models, such as having tens, hundreds, thousands, or tens of thousands of neural network layers, which are easier to train and approach better accuracy as the number of layers increases. For example, the residual network 201 may include 2 or more layer blocks, 5 or more layer blocks, 10 or more layer blocks, 20 or more layer blocks, 50 or more layer blocks, 100 or more layer blocks, 200 or more layer blocks, 500 or more layer blocks, 1000 or more layer blocks, etc. That being said, in some implementations, there may be no skip connection, such that the output layer block 230 does not receive the output of the input layer block 210.

[0075] The input layer block 210 is configured to: receive the network input 102; and process the network input 102 to generate an input sequence 202 for the first residual layer block 220-1 in the residual network 201. In some implementations, the input layer block 210 includes one or more linear neural network layers. Generally speaking, a linear layer is a layer of the neural network 200 for which each of the multiple values ​​output by the layer is a linear function of the multiple values ​​of the layer input. In some implementations, the input layer block 210 is an encoder, such as a linear encoder. The encoder 210 can encode the network input 102 into a sequence representing an embedding of the network input 102. As an example, if the network input 102 is a sequence of data elements (e.g., a word sequence), the encoder 210 can encode each data element in the sequence into the corresponding embedding of the data element. In other implementations, for example, when the network input 102 is a pre-embedded sequence, the network input 102 can be directly processed by the first residual layer block 220-1 in the residual network 201, that is, the input layer block 210 can be omitted or bypassed. The end of this specification describes various types of network inputs 102 that can be processed by the neural network 200.

[0076] The first residual layer block 220-1 in the residual network 201 is configured to receive an input sequence 202, for example, an input sequence generated as an output by the input layer block 210 or the network input 102 itself; and process the input sequence 202 to generate an output sequence 206. Each residual layer block 220 in the residual network 201 after the first residual layer block 220-1 is configured to: receive a corresponding input sequence 202, i.e., a sequence generated as an output by the previous residual layer block 220 in the residual network 201; and process the corresponding input sequence 202 to generate a corresponding output sequence 206 as an output. Figure 1B The operation of a single residual layer block 220 is described in more detail.

[0077] The output layer block 230 is configured to: receive the output sequence 206-N generated by the last residual layer block 220-N in the residual network 201; and process the output sequence 206-N to generate the network output 104. In this example, due to the skip connection, the output sequence 206-N of the last residual layer block 220-N is also added to the output of the input layer block 210. In some implementations, the output layer block 230 includes one or more linear neural network layers. For example, the output layer block 230 can generate a classification output as the network output 104 by shrinking the output sequence of the last residual layer block 220-N. In some implementations, the output layer block 230 is a decoder, such as a linear decoder. The decoder 230 can decode the output sequence of the last residual layer block 220-N into the network output 104. As an example, if the output sequence of the last residual layer block 220-N is a sequence of embeddings of data elements (e.g., a sequence of embeddings of word units), the decoder 230 can decode each embedding in the sequence into a corresponding data element. The various different types of network outputs 104 that may be generated by neural network 200 are described at the end of this specification.

[0078] In some implementations, the neural network 200 does not include any attention layers (e.g., self-attention layers). In other words, the input layer block 210, each residual layer block 220, and the output layer block 230 do not include any attention layers. For example, in some implementations, the layer blocks 210, 220, and 230 in the neural network 200 are not Transformer layer blocks that implement a multi-head attention mechanism. As described above, the attention layer is scalability and can be a bottleneck for long sequences. Neural network 200 can utilize linear recurrent and feedforward units without an attention mechanism to process long sequences, thereby speeding up computation and reducing memory costs. Further details on the multi-head attention mechanism implemented by the Transformer model are provided in the following reference: Vaswani, Ashish, et al. “Attention is all you need,” Advances in neural information processing systems 30 (2017).

[0079] System 100 can be implemented in any suitable location, such as on a user device (e.g., a mobile device), or on one or more computers in a data center. The modularity of system 100 allows multiple devices to implement individual components of system 100 separately from one another. For example, the various layers of neural network 200 can be executed on different devices and can send their outputs and / or inputs to one another (e.g., via telecommunications). A user can interact with system 100, for example, by providing queries to system 100 via an interface (e.g., a graphical user interface or an application programming interface (API)). Specifically, the user can provide a query comprising: (i) a request to generate network output 104 for network input 102, and (ii) network input 102. In response to receiving the query, system 100 can process network input 102 using neural network 200 to generate network output 104 responsive to the request, and then provide network output 104 to the user, e.g., for display on the user's user device or for storage in a data storage device. In some cases, system 100 may transmit network output 104 to a user device of a user, such as by means of a data communications network (eg, the Internet).

[0080] Figure 2A is a flow chart of an example process 300 for processing network input 102 using neural network 200 to generate network output 104. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system appropriately programmed according to the present specification (e.g., Figure 1A The system 100 may perform the process 300 by implementing the neural network 200.

[0081] The neural network 200 receives the network input 102 (310).

[0082] In some implementations, the network input 102 is a long program sequence. For example, the network input 102 can be a long program sequence including two thousand or more data elements. In some implementations, the data elements are word elements.

[0083] The neural network 200 processes the network input 102 using the input layer block 210 to generate an input sequence 202 (320) for the first residual layer block 220-1 in the residual network 201.

[0084] In some implementations, the input layer block 210 includes one or more linear neural network layers. In some implementations, the input layer block 210 is an encoder, such as a linear encoder.

[0085] For each residual layer block 220-1 to 220-N in the residual network 201:

[0086] The residual layer block 220 receives the corresponding input sequence 202 (330) generated as output by the previous layer block in the neural network 200.

[0087] The residual layer block 220 processes the corresponding input sequence 202 to generate a corresponding output sequence 206 as output (340).

[0088] The neural network 200 uses the output layer block 230 to process the output sequence 206 of the last residual layer block 220-N in the residual network 201 to generate the network output 104 (350).

[0089] In some implementations, the output layer block 230 includes one or more linear neural network layers. In some implementations, the output layer block 230 is a decoder, such as a linear decoder.

[0090] Figure 1B is a block diagram of an example residual layer block 220. Residual layer block 220 includes an (optional) normalization layer 221, a linear recurrent layer 222, and one or more feed-forward layers 224-1 through 224-M. Residual layer block 220 is configured to: receive an input sequence 202; and process input sequence 202 to generate an output sequence 206 as an output. In more detail, residual layer block 220 processes input sequence 202 using a normalization layer 221 to normalize input sequence 202. Normalization layer 221 can implement batch normalization (BatchNorm), layer normalization (LayerNorm), or other normalization schemes, which generally improve the stability of residual layer block 220. Residual layer block 220 then processes input sequence 202 using a linear recurrent layer 222 to generate an initial output sequence 204. Residual layer block 220 then processes initial output sequence 204 using a feed-forward layer 224 to generate output sequence 206.

[0091] Input sequence Has length and includes at each of the plurality of input locations (i) a corresponding input Each input of the input sequence 202 is typically dimensional vector. In an implementation involving the normalization layer 221, the input sequence 202 may also be normalized For example, in the case of batch normalization, the inputs are normalized over mini-batches to have a fixed mean and variance, whereas with layer normalization, all inputs are normalized together.

[0092] Initial output sequence Also has length And at the input position ( ) includes the corresponding initial output ( ). Each initial output of the initial output sequence 204 is typically dimensional vector. The initial output can have the same ( ) or different ( ) dimension. The initial output sequence 204 is generated by the linear cycle layer 222 by implementing a linear cycle (e.g., using parallel scanning) on ​​the input sequence 202, which will be referred to below. Figure 1C Describe in more detail.

[0093] Output sequence Has length And in multiple output locations Each of the outputs includes a corresponding Each output is typically dimensional vector. These outputs can have the same ( ) or different dimensions ( ). In addition, the output sequence 206 may have the same ( ) or different ( ) length. Figure 1B In the example of , the residual layer block 220 includes a skip connection (e.g., an identity skip connection) that adds the input sequence 202 to the output of the feed-forward layer 224 to generate the output sequence 206. In these cases, the input 202 and output 206 sequences typically have the same length and include vectors of the same dimension. The feed-forward layer 224 implements (unidirectional) nonlinearity on the initial output sequence 204 using, for example, a nonlinear activation function (such as tanh and sigmoid activation functions). For example, in some implementations, the feed-forward layer 224 can include: (i) a multilayer perceptron (MLP), (ii) a gated linear unit (GLU), or (iii) both.

[0094] Note that in implementations where the network input 102 is a data stream (e.g., for online inference), each input position ( ) may correspond to corresponding time steps (or timestamps) in the network input 102. The residual layer block 220 may then sequentially generate each output in the output sequence 206 at the corresponding time step The residual layer block 220 may then forward the output at time step t to the next layer block in the neural network 200 for further processing. In these cases, the length of the sequence may actually be close to , or a very large number of elements.

[0095] Figure 2Bis a flow diagram of an example process 340 for processing an input sequence 202 using a residual layer block 220 to generate an output sequence 206 .

[0096] The input sequence 202 includes a corresponding input at each of a plurality of input positions. The output sequence 206 includes a corresponding output at each of a plurality of output positions. In some implementations, the number of input positions is equal to the number of output positions.

[0097] In some implementations, the residual layer block 220 first processes the input sequence 202 using the normalization layer 221 to normalize the input sequence 202 (342).

[0098] The residual layer block 220 then processes the input sequence 202 using a linear recurrent layer 222 to generate an initial output sequence 204 (344). The initial output sequence 204 includes a corresponding initial output at each input position.

[0099] The residual layer block 220 processes the initial output sequence 204 using one or more feed-forward layers 224 to generate an output sequence 206 (346).

[0100] In some implementations, the feed-forward layer 224 includes: (i) a multi-layer perceptron (MLP), (ii) a gated linear unit (GLU), or (iii) both.

[0101] Figure 1C is a schematic diagram of an example linear recurrent layer 222 or LRU, which implements a linear recurrent on the input sequence 202 to generate an initial output sequence 204. For each input position, the operation of the linear recurrent layer 222 can be concisely summarized by the following two equations:

[0102] as well as,

[0103] As shown in Equation (1) and Equation (2), the linear recurrent layer 222 is composed of a set of matrices Parameterization. The set of matrices includes: the first matrix (also called transfer matrix), the second matrix , the third matrix and the fourth matrix Note that the linear recurrent layer 222 can be additionally parameterized by bias parameters that bias the hidden state and / or the initial output. This is equivalent to adding a bias vector to equation (1) and / or equation (2). However, due to the linearity of the recurrent relationship, these parameters can be equivalently incorporated into the feedforward layer 224 before and / or after the linear recurrent layer 222 (see Figure 1B ).

[0104] From the initial hidden state Initially, for each input position ( ), the linear cycle layer 222 receives the input for this position ( ) and obtain (e.g., from memory) the current hidden state ( ). The hidden state is usually dimensional vector that represents the information currently held by the linear recurrent layer 222 at a specific position. Note that the initial hidden state ( ) can be initialized in any way during inference and training, e.g., as default or random values, hyperparameters, etc. A common initialization for the initial hidden state is , but you can also choose others, such as .

[0105] Referring to Equation (1), the linear recurrent layer 222 calculates the first matrix-vector product between the following terms: (i) the first matrix ( ), and (ii) the current hidden state ( ). The linear recurrent layer 222 calculates the second matrix-vector product between the following terms: (i) the second matrix ( ) and (ii) for the input position ( The linear recurrent layer 222 then adds the first matrix-vector product to the second matrix-vector product to generate an updated hidden state ( ).

[0106] Referring to Equation (2), the linear recurrent layer 222 calculates the third matrix-vector product between the following terms: (i) the third matrix ( ), and (ii) the updated hidden state ( ). The linear recurrent layer 222 calculates the fourth matrix-vector product between the following terms: (i) the fourth matrix ( ), and (ii) input for the input position ( The linear recurrent layer 222 then adds the third matrix-vector product to the fourth matrix-vector product to generate an initial output for the input position ( ).

[0107] Since the operations described above with reference to equations (1) and (2) are linear, they can be efficiently parallelized by the linear recurrent layer 222 using parallel scans until . This makes the neural network 200 and the linear recurrent layer 222 suitable for use in parallel computing systems. This is in contrast to nonlinear RNNs, where computations are performed sequentially, thereby hindering parallelization. An example of a parallel scan (e.g., a parallel prefix scan) that the linear recurrent layer 222 can perform on the input sequence 202 to generate the initial output sequence 204 is described in more detail below. Further details related to parallel scanning are provided by the following literature: Eric Martin and Chris Cundy, “Parallelizing Linear Recurrent NeuralNets Over Sequence Length,” arXiv preprint arXiv:1709.04057 (2017), and Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman, “Simplified StateSpace Layers for Sequence Modeling,” arXiv preprint arXiv:2208.04933 (2022).

[0108] Nevertheless, as previously mentioned, the linear recurrent layer 222 can also process each input in the input sequence 202 in sequence via the linear recurrent relationship in equations (1) and (2), which may be relevant when the neural network 200 processes the network input 102 as a data stream (e.g., for online inference). In these cases, each input position ( ) may correspond to corresponding time steps in the network input 102. For example, at each time step, the linear recurrent layer 222 may receive an input for that time step ( ) and get the current hidden state ( ), where the hidden state represents the information held by the linear recurrent layer 222 between the time steps. The linear recurrent layer 222 can then process the input at the time step and the current hidden state, as outlined in equations (1) and (2), to generate an updated hidden state ( ) and the initial output for that time step ( ). The linear recurrent layer 222 then repeats this process for each subsequent time step.

[0109] Figure 2C is a flow diagram of an example process 344 for processing the input sequence 202 using the linear recurrent layer 222 to generate the initial output sequence 204 .

[0110] The linear recurrent layer 222 consists of a set of matrices Parameterized, the set of matrices includes: the first (transfer) matrix ( ), the second matrix ( ), the third matrix ( ) and the fourth matrix ( ). The input sequence 202 includes a corresponding input at each of a plurality of input positions. The initial output sequence 204 includes a corresponding initial output at each input position.

[0111] For each input location:

[0112] The linear recurrent layer 222 obtains the current hidden state (410).

[0113] The linear recurrent layer 222 computes a first matrix-vector product between: (i) the first matrix, and (ii) the current hidden state.

[0114] The linear recurrent layer 222 receives input for the input position (430).

[0115] The linear recurrent layer 222 computes a second matrix-vector product between: (i) the second matrix, and (ii) the input for the input position (440).

[0116] The linear recurrent layer 222 adds the first matrix-vector product to the second matrix-vector product to generate an updated hidden state (450).

[0117] The linear recurrent layer 222 computes a third matrix-vector product between: (i) the third matrix, and (ii) the updated hidden state (460).

[0118] The linear recurrent layer 222 computes a fourth matrix-vector product between: (i) the fourth matrix, and (ii) the input for the input position (470).

[0119] The linear recurrent layer 222 adds the first matrix-vector product to the second matrix-vector product to generate an initial output for the input position (480).

[0120] 3 is a block diagram of an example training engine 500 configured to train neural network 200 to perform machine learning tasks. Training engine 500 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0121] First, a general process for training a neural network 200 on common machine learning tasks is described. Following this, benchmark results of the neural network 200 trained on various tasks in the Long Range Arena (LRA) are provided. The benchmark results are accompanied by a gradual modification of the parameterization and initialization scheme of the linear recurrent layer 222, which improves various performance metrics of the neural network 200 during training and inference. As a related example, it is well known that RNNs can be difficult to train in practice due to the vanishing gradient problem, which occurs when the gradients of the recurrent weights become extremely small when backpropagating through the RNN. The linear recurrent layer 222 can use a (stable) exponential parameterization of the transfer matrix to mitigate (or completely eliminate) this effect.

[0122] First, the training engine 500 initializes the network parameters of the neural network 200 ( ), for example, parameters of the normalization layer 221, parameters of the linear recurrent layer 222, parameters of the feedforward layer 224, and / or parameters of other neural network layers in the neural network 200. The training engine 500 then obtains a training dataset related to a machine learning task (e.g., a task in LRA). 510. The training data set 510 includes training examples 512. The training dataset 510 may include any suitable number of training examples 512 for a machine learning task, e.g. or more training examples, or more training examples, or more training examples, or more training examples, or more training examples, or more training examples, etc. Each training example 512 includes: (i) a corresponding training network input ( ) 502, and (ii) the corresponding target network output ( ) 503. The training engine 500 then performs a training on the training data set 510 (or one or more batches of the training data set 510 ( ) training examples 512) to perform machine learning tasks. Generally speaking, the training engine 500 trains the neural network 200 to produce a target network output 503 in response to its training network input 502. That is, the training engine 500 uses the neural network 200 to process the training network input 502 of each training example 512 to generate a training network output 504, the output is an estimate of the target network output 503 for the training example 512.

[0123] The training engine 500 then optimizes the objective function 520, which depends on the training 504 and target 503 network outputs for each training example 512 in the training data set 510 (or a batch of training examples 512 in the training data set 510). Specifically, the training engine 500 minimizes (or maximizes) the objective function 520 with respect to the network parameters of the neural network 200, such as . Broadly speaking, the training engine 500 typically uses an objective function 520 that causes each training network output 504 to conform to its respective target network output 503, for example, as measured by some error or similarity metric between the two. In view of this, the objective function 520 can be (or can include) any appropriate objective function for the training dataset 510 and the machine learning task that the neural network 200 is trained to perform. For example, the objective function 520 can include a mean squared error loss or a mean absolute error loss for a regression task, a binary cross entropy loss or a hinge loss for a binary classification task, a categorical cross entropy loss for a multi-class classification task, a Kullback-Leibler divergence loss for a generative or reinforcement learning task, a MinMax loss for an image segmentation task, and the like.

[0124] To optimize the objective function 520, the training engine 500 uses, for example, backpropagation to determine the gradient 530 of the objective function 520 with respect to the network parameters of the neural network 200. The training engine 500 then uses the gradient 530 to update the network parameters of the neural network 200, for example, the parameters of the normalization layer 221, the parameters of the linear recurrent layer 222, the parameters of the feedforward layer 224, and / or the parameters of the neural network layers in the input layer block 210 and / or the output layer block 230. For example, the training engine 500 can use a stochastic gradient descent method (such as implicit update, momentum method, Adam, RMSProp, AdaGrad, etc.) with a specific learning rate and / or weight decay to update the network parameters with a value that optimizes the objective function 520. The training engine 500 can perform any suitable number of training iterations to optimize the objective function 520, for example, or more training iterations, or more training iterations, or more training iterations, or more training iterations, or more training iterations, or more training iterations, etc.

[0125] After training, training engine 500 can then use the test dataset to evaluate how well neural network 200 performs a machine learning task, for example, to benchmark the inference accuracy of neural network 200.

[0126] In this specification, neural network 200 is benchmarked against LRA, a set of machine learning tasks designed to test the ability of a neural network model to perform long sequence modeling. Transformers typically fail to complete most of these tasks, e.g., with inference accuracy no better than random guessing. In contrast, neural network 200 described herein generally performs well on these tasks, e.g., with inference accuracy exceeding random guessing.

[0127] For each experiment, the neural network 200 includes six residual layer blocks 220, each of which is based on Figure 1B Each residual layer block 220 includes: a batch normalization layer 221, a linear recurrent layer 222, a multilayer perceptron (MLP) for the feedforward layer 224, and a residual skip connection. All experiments were repeated three times - the means and standard errors are reported in Tables 1 and 2 below. The neural network 200 was trained using the AdamW optimizer. A small learning rate was used for the parameters of the linear recurrent layer 222, for example, about arrive , and no weight decay was used. More details on the experimental setup are provided by Antonio Orvieto, et al., “Resurrecting Recurrent Neural Networks for Long Sequences,” arXiv preprint arXiv:2303.06349 (2023).

[0128] Below, we describe several modifications to the linear recurrent layer 222 that provide progressively improved performance on the LRA task and other tasks involving long-range reasoning for the neural network 200. These modifications are summarized as follows: (i) using the linear recurrent layer 222 to implement linear recurrence (as opposed to nonlinear recurrence in some known systems), (ii) diagonalizing the transition matrix to speed up training and inference of the linear recurrent layer 222 (without affecting expressiveness or initialization distribution), (iii) changing the parameterization and initialization distribution of the linear recurrent layer 222 to achieve stability and improve long-range modeling, and (iv) providing a normalization strategy for the hidden activations of the linear recurrent layer 222.

[0129] Table 1 shows that removing nonlinearities from RNNs improves inference accuracy for most LRA tasks. The linear recurrent layer 222 is expressive when coupled with the nonlinear feedforward layer 224, outperforming tuned nonlinear RNN variants of the same architecture. In general, interleaving the linear recurrent layer 222 with the nonlinear feedforward layer 224 (e.g., Figures 1A to 1C ) is sufficient to approximate highly nonlinear systems. Table 1: Effect of removing nonlinearities from recurrent layers on inference accuracy. Shows results from sCIFAR, ListOps, Text, and Retrieval tasks in the Long Range Arena (LRA). Bold indicates the best inference accuracy.

[0130] Another advantage of linear recurrence is that the linear recurrence layer 222 can take advantage of many properties of linear algebra. Figure 1C As shown in , the loop relation of equation (1) can be initialized using To expand, as follows:

[0131] Transfer Matrix ( ) is the root cause of the vanishing (and exploding) gradient problem in RNNs. Although the hidden state in nonlinear RNNs is projected onto the compact image of the activation function, but as As the hidden states of the linear recurrent layer 222 increase, they may vanish or explode exponentially. This phenomenon can be understood by using eigenvalue (spectral) analysis. In the presence of any small perturbation in its entries, the transfer matrix is diagonalizable, that is, it can be parameterized as ,in is an invertible matrix, and is the eigenvalue ( ), where When the transfer matrix is ​​symmetric , the eigenvalues ​​and eigenvectors are real numbers. On the other hand, in the asymmetric case , the eigenvalues ​​and eigenvectors are complex numbers. Substitute into equation (3) and multiply both sides by , we get:

[0132] in ,and The linear recurrent layer 222 then calculates the initial output as:

[0133] in , and the linear recurrent layer 222 calculates The real part of ( ). Therefore, instead of learning , the linear recurrent layer 222 can equivalently learn the set of matrices ,in 、 and is complex valued, and is the diagonal transfer matrix of eigenvalues. For ease of description, the hidden state, the second matrix, and the third matrix can be relabeled in canonical form as 、 and , keeping in mind that these quantities are typically complex-valued in the diagonal parameterization of the linear recurrent layer 222. In some implementations, the diagonal transfer matrix ( ) has a magnitude less than one, which ensures the stability of the linear cycle layer 222. Specifically, at position The component of the hidden state at The norm of Therefore, a sufficient condition to ensure the stability of the linear circulation layer 222 (for example, such that and its gradient will not explode) is for all Both .

[0134] As a noteworthy application of diagonal parameterization, the linear recurrent layer 222 can compute the recurrent relations in Equations (4) and (5) in parallel, for example using parallel scans, to significantly speed up the training and inference of the neural network 200. In particular, since the transfer matrix ( ) is diagonal, so the parallelization time is approximately ) and involves The linear recurrent layer 222 can use a work-efficient algorithm to implement parallel scanning (e.g., see Figure 6 ). Thus, for a diagonal transfer matrix, use The total computational cost of parallel scanning with 4 processors is approximately More details on efficient parallel (prefix) scanning are provided by Ladner, Richard E., and Michael J. Fischer. “Parallel prefix computation,” Journal of the ACM (JACM) 27.4 (1980): 831-838. In other implementations, the linear recurrent layer 222 may use other algorithms to implement parallel scanning, which may provide more parallelism (but may not be as efficient), such as the algorithm proposed by Hillis, W. Daniel, and Guy L. Steele Jr., “Data parallel algorithms,” Communications of the ACM 29.12 (1986): 1170-1183. As the design space increases with the sequence length, the number of parallel (prefix) scans may be increased by 1. There are many different algorithms that can be implemented by the linear recurrent layer 222 to compute parallel scans. The generalization technique used to construct such an algorithm is as follows:

[0135] To implement parallel scanning, such as parallel prefix scanning, the linear recurrent layer 222 may first precompute the corresponding input tuple for each input position. ,as follows:

[0136] Which initialize ,and The linear recurrent layer 222 can then perform prefix computation on the input tuples, which generates a corresponding hidden tuple for each input position. ,as follows:

[0137] in is a binary associative operator for prefix calculation. The associative operator (or prefix operator) implements the loop relation in equation (1). It and Perform the following operations:

[0138] Therefore, the hidden tuple for each input position is Include the updated hidden state for that input position , which the linear recurrent layer 222 can then use to compute the initial output for that position ( ), as shown in equation (5).

[0139] With the help of prefix operator and equation (8), the hidden tuple can be expanded into groups of consecutive positions, starting from position To location (in ),as follows:

[0140] Furthermore, since prefix operators are associative, the following also holds:

[0141] for Equation (10) implies that a continuous group of positions can be decomposed into multiple continuous subgroups. This allows the linear recurrent layer 222 to compute the groups of hidden tuples in parallel ( ), and then combine them to compute the hidden state, as shown in Equation (7), hence the term “parallel scan” or “parallel prefix scan”.

[0142] For example, consider a length of The linear recurrent layer 222 can calculate the hidden tuple as: 、 、 and ,in . Introducing additional items Broke right The dependency of , thus allowing the linear recurrent layer 222 to compute both in parallel. This parallelization can significantly reduce the number of sequence steps performed by the linear recurrent layer 222 when the sequence length is large, because the parallel time scales logarithmically with the sequence length. In addition, it is suitable for multiple ( The invention is implemented in parallel using multiple) processors (e.g., multiple cores of an integrated circuit).

[0143] This process can also be used by the training engine 500 to efficiently parallelize the calculation of the gradients of the hidden states during training. For example, consider Derivatives with respect to the eigenvalues ​​of the diagonal transfer matrix:

[0144] for Equation (11) means that the gradient of a set of consecutive positions can be decomposed into the gradients of its consecutive subgroups. This allows the training engine 500 to calculate the hidden tuples ( ) and then combine them.

[0145] Returning to the example above, where the length of input sequence 202 is The training engine 500 can calculate the gradient of the hidden tuple as follows:

[0146] as well as,

[0147] Again, introduce additional items Broke right dependencies, thereby allowing the training engine 500 to compute the gradient of each in parallel.

[0148] The diagonal reparameterization described above generally reduces the total number of parameters of the linear recurrent layer 222 without limiting expressiveness, while providing computational speedups for both training and inference of the neural network 200. This is due, at least in part, to the fact that: (i) the computational power of diagonal matrices is computationally cheap (speeding up both training and inference), and (ii) associative scans can be used to parallelize the unrolled linear loops, resulting in faster training and inference. Table 2 shows that the diagonal transfer matrix improves the inference accuracy of the neural network 200 on tasks like ListOps and sCIFAR, while also reducing training and inference time on all LRA tasks. It is particularly well-suited for implementation in parallel computing systems.

[0149] refer to Figures 4A to 4D , the training engine 500 can randomly initialize the transfer matrix of the linear recurrent layer 222 in a stable manner. Figures 4A to 4C Shown are the 、 and After the matrix of size is initialized with Glorot (off-diagonal) transfer matrix Example of eigenvalues ​​of . Each entry of is independently sampled from a Gaussian distribution with mean 0 and variance . The eigenvalue is a complex number ( asymmetric) and is represented on the complex plane. The black circle is the unit disk . Figure 4D shows the diagonal transfer matrix ( ) eigenvalues. Under Glorot initialization, the transfer matrix The spectrum is at least approximately from The ESI is similarly applied to the diagonal transfer matrix . Using exponentials of complex variables , the training engine 500 is able to The median radius is and Specifically, in order to make the linear cycle layer 222 learn long-range dependencies and avoid gradient disappearance, the training engine 500 can be used to uniformly sample the rings between the circles. and Choose appropriate values ​​to initialize the eigenvalues ​​of the diagonal transfer matrix to be close to 1, for example, such that and ,or and ,etc.

[0150] ESI also recommends parameterizing the linear recurrent layer 222 in terms of the exponential of the complex variable:

[0151] in and are learnable parameters (instead of The choice of this “exponential parameterization” decouples the amplitude and the frequency of the oscillation, thus stabilizing optimization via stochastic gradient descent methods, for example.

[0152] Another benefit of exponential parameterization is that the linear recurrent layer 222 can be trained by limiting the learnable parameters To see this, consider initializing ,in Therefore, to maintain stability during training, the linear recurrent layer 222 can use an exponential or other positive nonlinearity to parameterize the diagonal transfer matrix:

[0153] in is the vector of logarithmic eigenvalue magnitudes, and is the vector of eigenvalue phases, corresponding to the learnable parameters. For example, the exponential nonlinearity in Implementation , and in Implementation This “stable exponential parameterization” is often helpful for training the linear recurrent layer 222 for long-range tasks. In addition, to reduce the oscillation of the eigenvalues ​​during training, the linear recurrent layer 222 can also reparameterize the eigenvalue phase in a logarithmic form as ,in is a learnable parameter of the phase. The training engine 500 can then initialize the phase of the eigenvalues ​​to a thin slice close to 0, for example, within the scope of within the scope of within the scope, etc. Table 2: Inference accuracy of the neural network 200 using the linear recurrent layer 222 with a diagonal transfer matrix under different parameterizations of the diagonal transfer matrix. Results from the sCIFAR, ListOps, and Pathfinder tasks in the Long Range Arena (LRA) are shown. Bold indicates the parameterization option with the best inference accuracy.

[0154] Finally, to gain additional stability during training, e.g., to stabilize the training loss, the linear recurrent layer 222 can be normalized by the parameter The normalization parameter is further parameterized, and the normalization parameter is initialized element by element by the training engine 500 as The cyclic relation of equation (1) is then normalized (in diagonal form) to:

[0155] in represents the element-wise product. The normalization parameter allows the linear recurrent layer 222 to adaptively scale the input fed to the corresponding feature direction Similar to the above, in order to improve stability, the linear cycle layer 222 can also re-parameterize the normalization parameter in logarithmic form as ,in is the variable of the normalization parameter. In this case, the training engine 500 initializes the variable of the normalization parameter element by element to . Regularization can improve performance on tasks that benefit from initialization close to the unit disk, such as sCIFAR and Pathfinder.

[0156] Figures 5A to 5C Shown are the effects of normalization parameters and using a small phase at initialization on the PathX task. Figure 5A is a plot showing the relationship between training loss and training iterations for neural network 200. Figure 5B is a plot showing the training accuracy of neural network 200 versus training iterations. Figure 5C is a plot showing the test accuracy of neural network 200 versus training iterations. Figures 5A to 5C Each of the 100k training iterations shows the mean and standard error of three independent runs. Without normalization, the neural network 200 exhibits higher loss values ​​at initialization and quickly converges to suboptimal values ​​where both training and test accuracy are random. Adding the normalization parameter helps. The training loss is lower at initialization, and the AdamW optimizer is able to escape the suboptimal region. Training accuracy also improves. Finally, reducing the initialization phase (i.e., adjusting range) improves convergence on the training dataset while also generalizing on the test dataset.

[0157] Figure 3B is a flow chart of an example process 600 for training a neural network 200 to perform a machine learning task. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training engine appropriately programmed according to the present specification, such as Figure 3A The training engine 500 can execute process 600.

[0158] The training engine initializes the network parameters of the neural network 200 ( ), including corresponding parameters (610) of one or more of the linear recurrent layers 222.

[0159] For example, when the first (transfer) matrix ( ) is parameterized in a diagonal form (e.g., as shown in Equation (4)), the training engine can use equivalent spectral initialization (ESI) to initialize and The diagonal transfer matrix on the ring between The eigenvalues ​​of ). In these cases, the diagonal transfer matrix can be parameterized according to an exponential parameterization (e.g., as shown in Equation (13)) or a stable exponential parameterization (e.g., as shown in Equation (14)). The training engine can be trained by and Select appropriate values ​​to initialize the eigenvalues ​​of the diagonal transfer matrix to be close to 1. The training engine can also initialize the phase of the eigenvalues ​​to a thin slice close to 0. In addition, when the linear recurrent layer 222 is further normalized by the parameter When parameterized (e.g., as in Equation (15)), the training engine can initialize the normalization parameters element-wise as .

[0160] The training engine may use various initialization schemes to initialize the second matrix (B), the third matrix (C), and the fourth matrix (D) of the linear recurrent layer 222. For example, the training engine may use Glorot initialization to initialize the second matrix (B) and the third matrix (C) of the linear recurrent layer 222. The training engine may use random initialization to initialize the fourth matrix (D) of the linear recurrent layer 222.

[0161] The training engine obtains a training data set 510 (620) related to the machine learning task. The training data set 510 includes a plurality of training examples 512, each of which includes: (i) a corresponding training network input 502, and (ii) a corresponding target network output 503.

[0162] The training engine processes the training network input 502 for each training example 512 using the neural network 200 to generate a training network output 504 that is an estimate of the corresponding target network output 503 for the training example 512 (630).

[0163] The training engine determines the gradient 530 of the objective function 540, which depends on the training network output 504 and the target network output 503 for each training example 512 (640).

[0164] For example, the training engine uses backpropagation to determine the gradient 530 of the objective function 540 with respect to the network parameters of the neural network 200.

[0165] The training engine uses the gradient 530 of the objective function 520 to update the network parameters of the neural network 200, including corresponding parameters of one or more of the linear recurrent layers 222 (650).

[0166] For example, the training engine may use a stochastic gradient descent method (e.g., implicit update, momentum method, Adam, RMSProp, AdaGrad, etc.) with a specific learning rate and / or weight decay to update the network parameters with the value of the optimized objective function 520.

[0167] Figure 6 An example JAX implementation of a linear recurrent layer 222 (also known as a "linear recurrent unit" (or LRU)) is shown. See J. Bradbury et al., "JAX: Composable Transformations of Python+Numpy Programs" (2018) for details on the JAX code. The hidden state of the linear recurrent unit 222 ( ) is composed of a sequence length Input Driven by the following formula (and parallelized using associative scans): The (initial) output is at each input position The places are calculated as follows: In the JAX code, and The Glorot initialization is followed, where B is additionally scaled by a factor of 2 to account for halving the state variance by taking the real part of the output projection. is random and in Multiply each position element-wise by . Initialization is performed according to Equivalent Spectral Initialization (ESI), where the phase is (possibly) constrained to thin slices.

[0168] In summary, the following describes examples of machine learning tasks that system 100 can perform using neural network 200.

[0169] As an example, the machine learning task can be a neural machine translation task. For example, if the network input 102 of the neural network 200 is a text sequence in one (e.g., natural) language, such as a sequence of words, phrases, characters, or word chunks, the network output 104 generated by the neural network 200 can be a translation of the text sequence into another (e.g., natural) language, that is, the text sequence in the other language is a translation of the input text sequence. As a specific example, the machine learning task can be a multilingual machine translation task, in which the neural network 200 is configured to translate between multiple different source language-target language pairs. In this example, the source language text can be enhanced with an identifier that indicates the target language into which the neural network 200 should translate the source language text.

[0170] As another example, the machine learning task may be an audio processing task. For example, if the network input 102 of the neural network 200 is a sequence representing a spoken utterance, the network output 104 generated by the neural network 200 may be a score for each text segment in the set of text segments, each score representing an estimated likelihood that the text segment is a correct transcription of the utterance. As another example, if the network input 102 of the neural network 200 is a sequence representing a spoken utterance, such as data recorded by a microphone, the network output 104 generated by the neural network 200 may indicate whether a particular word or phrase ("hot word") was spoken in the utterance. As another example, if the network input 102 of the neural network 200 is a sequence representing a spoken utterance, the network output 104 generated by the neural network 200 may identify the natural language in which the utterance was spoken.

[0171] As another example, a machine learning task can be a natural language processing or understanding task that operates on a text sequence of a certain natural language, such as an entailment task, a paraphrase task, a text similarity task, a sentiment task, a sentence completion task, a grammatical legitimacy task, etc.

[0172] As another example, the machine learning task can be a text-to-speech task, where the network input 102 is natural language text (e.g., a sequence of tokens as described above) or features of natural language text, and the network output 104 is a spectrogram, waveform, or other data defining audio of the text spoken in the natural language.

[0173] As another example, the machine learning task can be a health status prediction task, where the network input 102 is a sequence derived from a patient's electronic health record data, and the network output 104 is a prediction related to the patient's future health status, such as a predicted treatment that should be taken for the patient, the likelihood that the patient will experience an adverse health status event, or a predicted diagnosis for the patient.

[0174] As another example, the machine learning task can be a text generation task, where the network input 102 is a text sequence and the network output 104 is another text sequence, such as a completion of the input text sequence, a response to a question posed in the input sequence, or a text sequence about a topic specified by the first text sequence. As another example, the network input 102 of the text generation task can be an input other than text, such as an image, and the network output 104 can be text describing the network input 102.

[0175] As another example, the machine learning task may be an image generation task, where the network input 102 is a conditional input, such as text, a low-resolution image, or a partial image, and the network output 104 is an input sequence of intensity values ​​of pixels of the image.

[0176] As another example, the machine learning task can be an image processing task. For example, the network input 102 can be the intensity values ​​of the pixels of the image (or an encoded representation of the intensity values ​​generated by the encoder neural network), and the network output 104 can be (i) an image classification output that classifies the image into one of multiple object categories; (ii) an object detection output, i.e., a sequence of coordinates of one or more bounding boxes in the specified image that are predicted to contain objects; or (iii) a segmentation output that classifies each pixel in the image into one of multiple categories.

[0177] As another example, a machine learning task can be an agent control task, where the network input 102 is a sequence of observations or other data representing the state of an environment, and the network output 104 defines an action to be performed by the agent in response to the most recent data in the sequence. The agent can be, for example, a real-world or simulated robot, a control system for an industrial facility, or a control system that controls different types of agents.

[0178] As another example, the machine learning task can be a genomics task, where the network input 102 is a sequence representing a fragment of a DNA sequence or other molecular sequence, and the network output 104 is an embedding of the fragment for use in a downstream task (e.g., by utilizing unsupervised learning techniques on a dataset of DNA sequence fragments), or an output for use in a downstream task. Examples of downstream tasks include, but are not limited to, promoter site prediction, methylation analysis, predicting the functional effects of non-coding variants, and the like.

[0179] In some cases, the machine learning task is a combination of multiple separate machine learning tasks, i.e., system 100 is configured to perform multiple different separate machine learning tasks, such as two or more of the machine learning tasks mentioned above. For example, system 100 can be configured to perform multiple separate natural language understanding tasks, where network input 102 includes identifiers of the separate natural language understanding tasks to be performed on network input 102.

[0180] System 100 can use neural network 200 to generate network output 104 comprising an output sequence in response to network input 102 comprising an input sequence. Examples of sequence-to-sequence machine learning tasks that can be performed by system 100 are described below.

[0181] In some implementations, either or both of the network input 102 and the network output 104 can include word-gram sequences. That is, the network input 102 can include an input word-gram sequence, and the network output 104 can include an output word-gram sequence.

[0182] For example, in some cases, the input and output tokens each represent a word, a word chunk, or a character in a natural language. A word chunk can be a subword (a portion of a word), and can also be a separate letter or character. As used herein, "character" includes Chinese characters and other similar characters, as well as morpheme characters, syllabic characters, and the like.

[0183] Some of these implementations of system 100 can be used for natural language tasks, such as providing natural language responses to natural language inputs, for example, for question and answer or for text completion. In some implementations, network input 102 can represent natural language text, and network output 104 can represent the same natural language text, for example, a longer text item. For example, network input 102 can represent natural language text, and network output 104 can represent the same text, but with missing parts of the text added or filled in. For example, network output 104 can represent a predicted completion of the text represented by network input 102. Such applications can be used, for example, to provide an automatic completion function, for example, for searches based on natural language. In some implementations, network input 102 can represent natural language text, for example, to ask a question or define a topic, and network output 104 can represent the same or different natural language text, which is a response to the question or a response about a specified topic.

[0184] As another example, network input 102 may represent a first text item, while network output 104 may represent a second, shorter text item, e.g., the second text item may be a summary of a paragraph of the first text item. As another example, network input 102 may represent a first text item, while network output 104 may represent a simplification of the first text item. As another example, network input 102 may represent a first text item, while network output 104 may represent a certain aspect of the first text item. For example, network output 104 may represent the results of an inference task, a paraphrase task, a text similarity task, a sentiment analysis task, a sentence completion task, a grammatical legitimacy task, a syntactic analysis task (e.g., componential syntactic analysis), or the results of any natural language understanding task based on a natural language text sequence, e.g., to generate an output that classifies or predicts certain attributes of the text. For example, some implementations of system 100 may be used to identify the natural language of the first text item (or, when the input sequence is audio, to identify the natural language of spoken words, as described below).

[0185] Some implementations of system 100 can be used for neural machine translation. In such applications, input tokens can represent words, word chunks, or characters in a first natural language, while output tokens can represent words, word chunks, or characters in a second, different natural language. In other words, network input 102 can represent input text in a first language, while network output 104 can represent a translation of the input text into a second language.

[0186] Some implementations of the system 100 can be used for automatic code generation. In such applications, input tokens can represent words, word fragments, or characters in a first natural language, and output tokens can represent instructions in a computer programming or token language, or instructions for controlling an application to perform a task (e.g., constructing a data item such as an image or a web page).

[0187] Some implementations of system 100 may be used for speech recognition. In such applications, network input 102 may represent spoken words, and network output 104 may represent a conversion of the spoken words into a machine-written representation (e.g., text). Input tokens may then include tokens representing audio data input (e.g., a waveform representing audio in the time domain or time-frequency domain) that includes the spoken words. Output tokens may represent words, chunks, characters, or graphemes of a machine-written (e.g., text) representation of the spoken input, i.e., a transcription of the spoken input.

[0188] Some implementations of the system 100 can be used for handwriting recognition. In such applications, the network input 102 can represent handwritten words, syllables, or characters, and the network output 104 can represent the conversion of the network input 102 into a machine-written representation (e.g., text). The input tokens can then include tokens representing portions of handwriting, while the output tokens can represent words, chunks, characters, or graphemes of a machine-written (e.g., text) representation of the spoken input.

[0189] Some implementations of the system 100 can be used for text-to-speech conversion. In such applications, the network input 102 can represent text, and the network output 104 can represent the conversion of text into spoken words. The input tokens can then include tokens representing words, chunks, or graphemes of the text, while the output tokens can represent portions of audio data used to generate speech corresponding to the text, for example, tokens representing portions of a speech waveform in the time domain or the time-frequency domain, or phonemes.

[0190] In some implementations, the network input 102 and the network output 104 may each include a sequence of tokens representing different input and / or output modalities. For example, the network input 102 may represent natural language text, while the network output 104 may represent an image or video corresponding to the text, or vice versa. Generally speaking, a token may represent image or video features, and a sequence of these tokens may represent an image or video. There are many ways to use tokens to represent images (or videos). As an example, an image (or video) may be represented as a sequence of regions of interest (RoIs) in the image (or video), optionally including one or more tokens for global image features. For example, an encoder neural network may be used to encode the image to extract RoI features; optionally (but not necessarily), the token may also include data, such as a positional encoding, indicating the location of the RoI in the image. As another example, a token may encode the color or intensity value of an image pixel. As another example, some image processing neural network systems (e.g., autoregressive systems) naturally represent images as a sequence of image features.

[0191] In some implementations, at least one of the network input 102 or the network output 104 is a sequence representing an image or video, and the word-gram represents the image or video. For example, the network input 102 can be a text sequence, the input word-gram can represent a word, word chunk, or character, and the network output 104 can include an output word-gram representing the image or video, e.g., described by the text, or providing a visual answer to a question posed by the text, or providing a visualization of the subject of the text. In another example, the network input 102 can include a sequence of input word-grams representing an image or video (e.g., captured from the real world using a camera), and the output word-gram can represent a word, word chunk, or text character, e.g., as a description or representation of the image or video, either by providing an answer to a question posed visually by the image or video, or by providing information about the subject in the image or video.

[0192] In other implementations, both network input 102 and network output 104 may represent images or videos, and both input and output tokens may represent corresponding images or videos. In such implementations, system 100 may be configured to perform image or video transformations. For example, network input 102 and network output 104 may represent the same image or video in different styles, e.g., one may be an image and the other a sketch of the image; or different styles of the same garment.

[0193] In some implementations, network input 102 represents data to be compressed, e.g., image data, text data, audio data, or any other type of data, and network output 104 represents a compressed version of the data. Input and output tokens can each include any representation of the data to be compressed / decompressed, e.g., symbols or embeddings generated / decoded by a corresponding neural network.

[0194] In an implementation, network input 102 represents a sequence of actions to be performed by an agent, for example, a mechanical agent in a real-world environment implementing actions to perform a mechanical task. Network output 104 can include modified action sequences, for example, action sequences in which operating parameters such as motion speed or power consumption have bounded values; or action sequences in which crossing safety or other boundaries is unlikely. Both input and output tokens can then represent the actions to be performed.

[0195] In some implementations, the network input 102 represents a sequence of health condition data, and the network output 104 may include a sequence of predicted treatments. The input tokens may represent any aspect of a patient's health condition, such as data from blood and other medical tests on the patient and / or other patient data; and the output tokens may represent, for example, diagnostic information related to the patient's disease state and / or a recommended treatment for the patient and / or the likelihood of an adverse health condition event for the patient.

[0196] In some implementations, the network input 102 represents a time series, and the output sequence can include a continuation of the time series. For example, the network input 102 can be a sequence representing the output of a power plant (e.g., a solar or wind power plant), or a sequence representing power consumption, and the network output 104 can provide a forecast of the power generated or consumed. As another example, the network input 102 can be a sequence representing traffic levels on one or more roads, and the network output 104 can provide a forecast of future traffic.

[0197] System 100 can process and / or perform machine learning tasks on various types of data using neural network 200. Examples of different types of data that can be processed by neural network 200 are described below.

[0198] In implementations where network input 102 includes a sequence of inputs, each of network input 102 may be a data element embedding. As used herein, an embedding refers to an ordered collection of values, such as a vector, matrix, or other tensor of values.

[0199] For example, the data element embeddings may represent pixels of an image, and the network output 104 may include a classification output, e.g., including a corresponding score for each object category in a set of possible object categories, thereby defining the likelihood that the image depicts an object belonging to that object category.

[0200] In some implementations, the data elements are embedded in audio samples representing an audio waveform, and the system 100 is configured to perform speech recognition, ie, generate a network output 104 defining a sequence of phonemes, graphemes, characters, or words corresponding to the audio waveform.

[0201] In some implementations, the data element embeddings represent words in a word sequence, and the system 100 is configured to perform natural language processing tasks, such as topic classification or summarization. To perform topic classification, the network output 104 may include a corresponding score for each topic category in a set of possible category categories. For example, the score for a topic category may define the likelihood that the word sequence belongs to that topic category.

[0202] In some implementations, the system 100 is configured to perform audio processing tasks. For example, if the data element embedding represents a spoken utterance, the network output 104 may be a score for each text segment in the set of text segments, each score representing an estimated likelihood that the text segment is a correct transcription of the utterance. As another example, if the data element embedding represents a spoken utterance, the network output 104 may indicate whether a particular word or phrase ("hot word") was spoken in the utterance. As another example, if the data element embedding represents a spoken utterance, the network output 104 may identify the natural language in which the utterance was spoken.

[0203] In some implementations, system 100 can perform an image generation task where the data element embedding represents a conditional input, such as text, and the network output 104 defines an input sequence of intensity values ​​for image pixels.

[0204] In some implementations, as further described below, the system 100 can perform an agent control task, wherein the data elements embed a sequence representing one or more observations and / or other data representing an environment state, and the network output 104 includes a policy output. The agent can be, for example, a real-world or simulated mechanical agent (such as a robot or vehicle), a control system for an industrial facility, or a control system that controls different types of agents.

[0205] In some implementations, system 100 may perform point cloud processing tasks, e.g., where data elements are embedded representing a point cloud (e.g., generated by a lidar or radar sensor) and network output 104 characterizes, e.g., the type of object represented by the point cloud.

[0206] In some cases, system 100 is configured to perform a combination of multiple individual machine learning tasks, such as two or more of the machine learning tasks mentioned above.

[0207] Neural network 200 can process data element embeddings representing any suitable type of entity. For example, an entity can include an image, an audio waveform, a point cloud (e.g., generated by a lidar or radar sensor), a protein, a sequence of words (e.g., forming one or more sentences or paragraphs), a video (e.g., representing a sequence of video frames), or a combination thereof, e.g., multimodal data. Network output 104 can represent the entity or perform a processing task on the entity.

[0208] Implementations of system 100 can process multimodal data for multimodal entities. Such entities can include a combination of different types of data, such as image or video data and audio data, image or video data and language data, somatosensory input data (sensor data that senses the real-world environment of a physical agent, such as touch, pressure, motion, temperature, or vibration data), and motion feedback data (i.e., control data used to control the movement of a physical agent). When multimodal entities are processed by system 100, embeddings of data elements of different modalities can be combined.

[0209] The network output 104 for a multimodal entity can be as described above. For example, when the network output 104 is a classification output of a classification task (e.g., defining a score for each category in a set of possible categories), this can be as described above, except that the network output 104 is generated based on the multimodal data embedding provided as network input 102. As a result, the machine learning task (e.g., a classification task) performed by the system 100 can be performed better, e.g., more accurately. For example, a classification task can be performed on a combination of video and (corresponding) audio data to obtain more accurate classification results. As another example, the machine learning task can be a task based on processing data of different modalities, for example, in a task combining video or image data and language data (e.g., text data) to determine whether an image or video is described by a particular subtitle.

[0210] This specification uses the term "configured" in conjunction with system and computer program components. With respect to a system of one or more computers that are configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform those operations or actions. With respect to one or more computer programs that are configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operations or actions.

[0211] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0212] The term "data processing equipment" refers to data processing hardware and encompasses all kinds of equipment, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The equipment may also be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0213] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language (including compiled or interpreted languages ​​or declarative or procedural languages); and it may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0214] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0215] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry (e.g., an FPGA or an ASIC), or by a combination of special purpose logic circuitry and one or more programmed computers.

[0216] A computer suitable for executing a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or random access memory, or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a dedicated logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to receive data from or transfer data to or both. However, a computer need not have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0217] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0218] To provide for user interaction, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide for user interaction; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. Furthermore, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Furthermore, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving a response message from the user in return.

[0219] A data processing device used to implement a machine learning model may also include, for example, dedicated hardware accelerator units for processing general-purpose and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0220] Machine learning models can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework).

[0221] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0222] A computing system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The relationship of client and server arises by virtue of computer programs running on respective computers that establish a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, such as the results of a user interaction, may be received at the server from the device.

[0223] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be unique to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed as such, in some cases one or more features from the claimed combination may be deleted from the combination, and the claimed combination may involve a subcombination or a variant of a subcombination.

[0224] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0225] Specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to: A neural network configured to perform the machine learning task, wherein the neural network includes a plurality of layer blocks, each layer block including: (i) a linear recurrent layer, and (ii) one or more feed-forward layers, Each layer block is configured to perform the following operations: receiving an input sequence for the layer block, the input sequence comprising a respective input at each of a plurality of input positions; and Generating an output sequence for the layer block, the output sequence including a corresponding output at each of a plurality of output positions, the generating comprising: processing the input sequence using the linear recurrent layer of the layer block to generate an initial output sequence comprising a respective initial output at each of the plurality of input positions; and The initial output sequence is processed using the one or more feed-forward layers of the layer block to generate the output sequence.

2. The system of claim 1, wherein: The plurality of layer blocks are organized into sequences, and For each layer block in said sequence of layer blocks that is not the first layer block, the input sequence for the layer block is the output sequence of the previous layer block in said sequence of layer blocks.

3. The system of claim 2, wherein: The network input is an input sequence for the first layer block in the sequence of layer blocks.

4. The system of claim 2, wherein: The neural network further includes an encoder configured to perform operations comprising: receiving the network input; and The network input is processed to generate an input sequence for the first layer block in the sequence of layer blocks.

5. The system of claim 4, wherein: The encoder is a linear encoder.

6. The system according to any one of claims 2 to 5, wherein: The neural network further includes one or more output layers, the one or more output layers being configured to perform operations including: receiving an output sequence of a last layer block in said sequence of layer blocks; and The output sequence of the final layer block is processed to generate the network output.

7. The system of claim 6, wherein: The one or more output layers are linear layers.

8. A system as claimed in any preceding claim, wherein: The linear recurrent layer of each layer block is parameterized by a plurality of matrices, and wherein processing the input sequence using the linear recurrent layer of the layer block to generate the initial output sequence comprises, for each input position: Get the current hidden state; Compute a first matrix-vector product between: (i) the first matrix of the linear recurrent layer, and (ii) the current hidden state; receiving the input for the input location; computing a second matrix-vector product between: (i) a second matrix of the linear recurrent layer, and (ii) the input for the input position; and The first matrix-vector product and the second matrix-vector product are added to generate an updated hidden state.

9. The system of claim 8, wherein: The first matrix of each linear recurrent layer is a diagonal matrix parameterized by a plurality of eigenvalues.

10. The system of claim 9, wherein: Each eigenvalue has a magnitude less than one.

11. The system according to any one of claims 9 to 10, wherein: The eigenvalues ​​include complex eigenvalues.

12. The system of claim 11, wherein: Each complex eigenvalue is represented as an exponential function parameterized by a complex variable.

13. The system of any one of claims 8 to 12, wherein: The linear recurrent layer of each layer block is further parameterized by a normalization parameter, and wherein processing the input sequence using the linear recurrent layer of the layer block to generate the initial output sequence further comprises, for each input position: computing the element-wise product between: (i) the normalization parameter of the linear recurrent layer, and (ii) the second matrix-vector product, and Adding the first matrix-vector product and the second matrix-vector product to generate the updated hidden state includes: The first matrix-vector product is added to the element-wise product to generate the updated hidden state.

14. A system as claimed in any one of claims 8 to 13, wherein Processing the input sequence using the linear recurrent layer of the layer block to generate the initial output sequence further comprises, for each input position: Compute a third matrix-vector product between: (i) the third matrix of the linear recurrent layer, and (ii) the updated hidden state; computing a fourth matrix-vector product between: (i) a fourth matrix of the linear recurrent layer, and (ii) the input for the input position; as well as The third matrix-vector product and the fourth matrix-vector product are added to generate the initial output for the input position.

15. The system of any one of claims 8 to 14, wherein: The neural network has been trained to perform the machine learning task on a plurality of training examples, each training example comprising: (i) a corresponding training network input, and (ii) a corresponding target network output, and wherein the training comprises: Processing the training network input of each training example using the neural network to generate a training network output, the training network output being an estimate of the corresponding target network output for the training example; determining a gradient of an objective function that depends on the training network output and the target network output for each training example; and The gradient of the objective function is used to update corresponding parameters of one or more of the linear recurrent layers.

16. A system as claimed in claim 15 when dependent on claim 12, wherein The training further includes, for each updated linear recurrent layer: The complex variable for each complex eigenvalue parameterizing the linear recurrent layer is initialized to near zero.

17. A system as claimed in claim 15 or 16 when dependent on claims 13 and 9, wherein The training further includes, for each updated linear recurrent layer: The normalization parameters parameterizing the linear recurrent layer using the eigenvalues ​​of the first matrix of the linear recurrent layer are initialized.

18. A system as claimed in any preceding claim, wherein For each layer block, the number of input positions of the layer block is equal to the number of output positions of the layer block.

19. A system as claimed in any preceding claim, wherein: The one or more feed-forward layers of each layer block include: (i) a multilayer perceptron, (ii) a gated linear unit, or (iii) both.

20. A system as claimed in any preceding claim, wherein The network input is a long program sequence.

21. The system of claim 20, wherein: The network input includes two thousand or more data elements.

22. The system of claim 21, wherein: The data elements are word-grams.

23. A system as claimed in any preceding claim, wherein: The linear recurrent layer of each layer block is configured to perform parallel scanning on the input sequence to generate the initial output sequence.

24. The system of claim 23, wherein: The parallel scanning is a highly efficient parallel scanning.

25. The system of any one of claims 23 to 24, wherein: The layer blocks are implemented by multiple processors operating in parallel.

26. A system as claimed in any one of the preceding claims, wherein: The network input comprises data to be compressed, and the network output comprises a compressed version of the data.

27. The system of any one of claims 1 to 25, wherein: The network input comprises audio data representing an audio waveform, and the network output comprises a spectrogram of the audio waveform.

28. The system of any one of claims 1 to 25, wherein: The network input comprises: (i) intensity values ​​of pixels of an input image, or (ii) encoded representations of the intensity values ​​of the pixels of the input image, and The network output is: (i) an image classification output that classifies the input image into one of a plurality of object categories; (ii) an object detection output that specifies the coordinates of one or more bounding boxes in the input image that are predicted to contain objects; or (iii) a segmentation output that classifies each pixel in the input image into one of the plurality of categories.

29. The system of any one of claims 1 to 25, wherein: The machine learning task is an agent control task for an agent in an environment. The network input is a sequence of observations representing the state of the environment, and The network output defines the action to be performed by the agent in response to the most recent observation in the sequence.

30. The system of any one of claims 1 to 22, wherein: The machine learning task is a health status prediction task, The network input includes data derived from the patient's electronic health record, and The network output is a prediction output defining: (i) a predicted treatment that should be taken for the patient, (ii) a likelihood that the patient will experience an adverse health condition event, or (iii) a predicted diagnosis for the patient.

31. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to implement the neural network of any preceding claim.

32. A method performed by one or more computers, the method comprising: Receive network input; as well as The network input is processed using a neural network as claimed in any one of claims 1 to 30 to generate a network output for the network input.

Images