Multitask recurrent neural network

Abstract

To provide methods, systems, and apparatuses, including computer programs encoded on computer storage media, relating to multi-task recurrent neural networks.SOLUTION: A method includes maintaining data specifying, for a recurrent neural network, a separate internal state for each of a plurality of memory regions; receiving a current input; identifying a particular memory region of a memory access address defined by the current input; selecting, from the internal states specified in the maintained data, the internal state for the particular memory region; processing, in accordance with the selected internal state for the particular memory region, the current input in a sequence of inputs using the recurrent neural network to generate an output defining a probability distribution of a predicted memory access address, and update the selected internal state of the particular memory region; and associating the updated selected internal state with the particular memory region in the maintained data.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] background This specification relates to recurrent neural networks. [Background technology]

[0002] A neural network is a machine learning model that uses the operations of multiple layers to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers located between the input and output layers. The output of each hidden layer is used as input to the next layer, e.g., the next hidden layer or the output layer.

[0003] Each layer of a neural network specifies one or more transformation operations to be performed on the inputs to the layer. Some neural network layers have operations called neurons. Each neuron receives one or more inputs and produces outputs that are received by other neural network layers. Often, each neuron receives inputs from other neurons and provides outputs to one or more other neurons.

[0004] Each layer generates one or more outputs using the current values ​​of the set of parameters for that layer. Training a neural network involves successively making forward passes over the inputs, calculating gradients, and updating the current values ​​of the set of parameters for each layer. Once the neural network is trained, it can make predictions in a production system using the final set of parameters.

[0005] Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from previous time steps when calculating the output at the current time step. This allows a recurrent neural network to exhibit dynamic behavior over time. That is, a recurrent neural network can aggregate all the information received up to the current time step and collect long-term dependencies of a sequence of data. Summary of the Invention [Means for solving the problem]

[0006] overview This specification describes a multi-tasking recurrent neural network, and more particularly, a multi-tasking recurrent neural network that improves time series prediction in a microprocessor.

[0007] A circuit implementing a multitasking recurrent neural network includes recurrent neural network cells ("RNN cells") configured to perform the operations of the recurrent neural network. That is, the RNN cells are configured to perform the transformation operations defined by each layer in the recurrent neural network. The circuit also includes state registers. Each state register (i) is configured to store internal state data for the recurrent neural network and (ii) corresponds to a respective task that the neural network is trained to perform. State data is data that persists across multiple time steps, i.e., data retained after processing one input in a sequence to be used in processing the next input in the sequence. For example, the state data may be stored in a recurrent neural network to store data for long and short time steps. For a linear-memory (LSTM) network, the state data may include a hidden state h and a cell state c for each LSTM unit in the network.

[0008] The circuit receives a current input in a sequence of inputs and sequence identification data. The circuit identifies a specific task based on the sequence identification data. The circuit retrieves internal state data from a state register corresponding to the identified task. The circuit provides the retrieved internal state data and the current input to an RNN cell, causing the RNN cell to process the retrieved internal state data and the current input to generate a neural network output and updated internal state data for the identified task. The circuit receives the neural network output and the updated internal state for the identified task from the RNN cell. Finally, the circuit writes the updated internal state data to the state register corresponding to the identified task.

[0009] The above-described circuit can be used to more accurately predict memory accesses in a computing system according to a method described below. The circuit can maintain data specifying a separate internal state for each of a plurality of memory regions for a recurrent neural network trained to predict memory access addresses of a computer program in one of the memory regions, receive a current input in a sequence of inputs defining a sequence of previous memory access addresses of the computer program, identify a particular memory region of the memory access address defined by the current input in the sequence of inputs, select an internal state for the particular memory region from the internal states specified in the maintained data, process the current input in the sequence of inputs using the recurrent neural network according to the selected internal state for the particular memory region to (i) generate an output defining a probability distribution of predicted memory access addresses that are future memory access addresses of the computer program in the particular memory region, and (ii) update the selected internal state for the particular memory region, and associate the updated selected internal state with the particular memory region in the maintained data.

[0010] In some embodiments, one or more computer-readable storage media are encoded with instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations, including methods and steps, according to any embodiment or aspect described herein.

[0011] In some embodiments, the system comprises one or more computers and one or more storage devices that store instructions that, when executed by the one or more computers, cause the computers to perform methods and steps according to any embodiment or aspect described herein.

[0012] The subject matter described herein can be implemented in certain embodiments to achieve one or more of the following advantages: Implementing a recurrent neural network in hardware that includes storing recurrent neural network states in state registers coupled to the recurrent neural network allows the recurrent neural network to perform different tasks in rapid succession, known as multitasking. More specifically, each state register can store a state for each task. The recurrent neural network can obtain a first state from a first state register to process a first input sequence, and before completing processing of the first input sequence, can obtain a second state from a second state register to process a second, different input sequence. For example, a hardware circuit implementing a recurrent neural network can predict memory accesses, cache misses, and branches simply by obtaining the state corresponding to one of the tasks from the respective state registers. Storing state data in the state registers By storing the input sequence, the number of sequence dependencies that the RNN must compute at each time step is reduced, reducing the overall latency by the length of the input sequence.

[0013] This specification also describes ways in which classification of input sequences and partitioning by latent factors can be performed. In data prefetching, one example of partitioning by latent factors is separating an interleaved stream of memory access addresses in different regions of memory. Each of these regions may have different memory access patterns, and the machine learning model can more effectively know if the stream is separated before being used as input to the recurrent neural network. Another example of partitioning by latent factors in data prefetching is separating the stream of memory accesses by the data structure that caused these memory accesses. Again, doing so can more effectively train the machine learning model.

[0014] One or more embodiments of the subject matter herein are described in detail in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 illustrates an example of a recurrent neural network implemented in hardware. [Figure 2] FIG. 1 illustrates an example of a recurrent neural network system having a state register and a multiplexer. [Figure 3] 1 is a flowchart illustrating an example of a process for processing a multi-tasking recurrent neural network using hardware circuitry. [Figure 4] FIG. 1 illustrates an example of a computing system configured to prefetch data from memory. [Figure 5] 1 is a flowchart illustrating an example of a process for predicting future memory access addresses of a computer program executing on a computing system. DETAILED DESCRIPTION OF THE INVENTION

[0016] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description This specification describes a multi-tasking recurrent neural network, and more particularly, a multi-tasking recurrent neural network that improves time series prediction in a microprocessor.

[0017] Figure 1 illustrates a hardware-implemented recurrent neural network 100. A recurrent neural network typically includes an input layer, one or more hidden layers, and an output layer.

[0018] The input layer of a recurrent neural network is typically configured to receive a sequence of inputs, including a respective input at each time step, from 0 to the current time step N, and to process the sequence of inputs to generate a feature representation of the sequence.

[0019] The hidden layers of a recurrent neural network are recurrent layers, meaning that they maintain a state between inputs in a sequence and update the current state as part of generating the output for the current time step. Traditional recurrent neural networks have hidden layers that calculate the output and next state for a layer, where the next state is based on the previous state and the current input. The state is maintained across time steps and updates the current state as part of generating the output for the input sequence. Capture dependencies.

[0020] A particular type of recurrent neural network is the long-short-term memory (LSTM) network. The hidden layer in an LSTM network contains one LSTM unit for each input in the input sequence. Each LSTM unit provides outputs to other LSTM units in its own layer and to LSTM units in the next hidden or output layer. Each LSTM unit contains a hidden state h, a cell state c, an input gate i, a forget gate f, and an output gate o. Each of these three gates can be thought of as an individual neuron that computes a weighted sum activation. Intuitively, the input gate controls the extent to which new values ​​flow into the cell, the forget gate controls the extent to which values ​​remain in the cell, and the output gate controls the extent to which values ​​in the cell are used to compute the LSTM unit's output activation. That is, these gates control what the LSTM unit "remembers," or how its state is affected by previous states.

[0021] The output layer of a recurrent neural network is typically configured to receive input from the last hidden layer and generate an output, e.g., a prediction, at each time step. The output layer may be a softmax layer, a logistic layer, a linear layer, or the like. The softmax output layer generates a discrete probability distribution over the possible outputs for the time step; that is, each possible output is associated with a discrete point number. The logistic output layer generates parameters, e.g., the mean, log scale, or mixture weights of the distribution over the possible outputs for the time step. Additionally, the linear layer projects the output of the last hidden layer into a suitable output space. In some implementations, the recurrent neural network does not have an output layer; that is, the output of the last hidden layer is the output of the recurrent neural network.

[0022] Training data for a recurrent neural network includes sequences of inputs and labels for those sequences of inputs. Each label corresponds to a desired or expected output at a particular time step for a particular sequence of inputs. Training a recurrent neural network involves setting the network weights, e.g., the weights of the input layer, RNN cells, and output layer, to minimize the discrepancy between the actual output and the predicted output for a given sequence of inputs. If there is a discrepancy between the actual output and the predicted output for a given sequence of inputs, the weights of the recurrent neural network can be updated, e.g., using conventional backpropagation over time training techniques. Other techniques known in the art can also be used to update the weights of a recurrent neural network.

[0023] Recurrent neural network 100 is a hardware implementation of a recurrent neural network that includes one state register per hidden layer and one RNN cell per hidden layer. In some implementations, each RNN cell is a processing unit in a systolic array of processing units. Each processing unit performs the row multiplication and other operations of the recurrent neural network layer in hardware, for example, by including a set of multiply-accumulate units that collectively perform the row multiplication and other circuitry that applies any necessary gating functions and performs element-wise multiplication.

[0024] At time step N, input unit 110 receives input N in a sequence of inputs and generates a feature representation of that input. RNN cells 120 and 130 process the feature representation using both the fixed parameters of the recurrent neural network and the state data stored in state registers for each hidden layer. The output of RNN cell 130 is then provided to output unit 140, which generates an output for time step N+1. RNN cells 120 and 130 also update the state data in their respective state registers to reflect input N. This state data reflects all previous inputs and states. Capture.

[0025] 2 illustrates an example of a recurrent neural network system 200 that utilizes state registers in different ways. State registers 210a-210n can each store data specifying a different internal state of the recurrent neural network. Each internal state, and therefore each state register, corresponds to a different task the neural network is trained to perform. For example, a first state register can store data specifying an internal state corresponding to a prediction of a memory access address, i.e., a loading and storing address, while a second state register can store data specifying an internal state corresponding to a prediction of a branch in a computer program. Meanwhile, other state registers can store data specifying an internal state corresponding to, for example, a prediction of a future program instruction or a cache miss.

[0026] The RNN selector 220, which is a multiplexer, can select a particular internal state that the recurrent neural network uses at a particular time based on the sequence identification data. The sequence identification data specifies the state register, and therefore the internal state, to be loaded into the RNN computational cell 230. The sequence identification data may be based on a user input that specifies a particular task that the RNN computational cell 230 should perform, for example, predicting a memory access address (as opposed to a branch, program instruction, or cache miss). Alternatively, the sequence identification data may be based on an observation about the current input. For example, the sequence identification data may be based on the observation that the current input is a cache address. In such a situation, the sequence identification data may specify that an internal state corresponding to the task of predicting a cache miss should be loaded into the RNN computational cell 230.

[0027] The recurrent neural network system 200 also includes an input unit 240 that is loaded with a set of fixed parameters. The input unit 240 is capable of receiving a current input, generating a feature representation of the current input, and providing the feature representation to the RNN computational cell 230.

[0028] Using the selected state, feature representation, and set of fixed weights maintained within the RNN computational cell 230, the RNN computational cell 230 can generate a prediction and next state for the task. If the RNN has multiple hidden layers, the RNN computational cell 230 can provide the generated prediction to other RNN computational cells, which may have their own state registers and RNN selectors. The RNN system may also optionally include an output unit. Meanwhile, the RNN computational cell 230 can provide the next state for the task to the appropriate state register 210a-210n, overwriting the previous state of that register. Training a multitasking RNN network such as the one described above requires training the RNN to perform each task.

[0029] Alternatively, state registers 210a-210n can each store data specifying a separate internal state for each of multiple memory regions for a recurrent neural network trained to predict memory access addresses of a computer program within multiple memory regions. By dividing memory accesses by memory region in this manner, memory access predictions can be improved because memory accesses within a single memory region typically follow a more predictable pattern than memory accesses across two or more memory regions. That is, RNN computational cell 230 generates more accurate predictions when predicting memory accesses within a specific memory region rather than across the entire address space.

[0030] 3 is a flowchart illustrating an example of a process 300 for operating a multitasking recurrent neural network. Process 300 is performed by an appropriately programmed circuit. The circuit includes an RNN cell configured to perform the operations of the recurrent neural network, a plurality of state registers, and other circuit components. Each state register (i) is configured to store internal state data for the recurrent neural network and (ii) corresponds to a respective one of a plurality of tasks that the neural network is trained to perform.

[0031] The circuit receives 310 a current input in a sequence of inputs and sequence identification data. The input sequence includes a respective input at each of a plurality of time steps. The time steps may or may not be consecutive. The sequence identification data specifies a task, and therefore a state register, whose internal state is loaded into the RNN cell.

[0032] The circuit identifies a particular task to be performed based on the sequence identification data (320). The sequence identification data may be based on a user input specifying a particular task for the RNN cell to perform, such as predicting a memory access address (as opposed to a branch, a program instruction, or a cache miss). Alternatively, the sequence identification data may be based on an observation about the current input. For example, the sequence identification data may be based on an observation that the current input is a cache address. In such a situation, the sequence identification data may specify that an internal state corresponding to the task of predicting a cache miss should be loaded into the RNN cell.

[0033] The circuit retrieves internal state data from a state register corresponding to the identified task (330). The circuit provides the retrieved internal state data and current input to the RNN cell and causes the RNN cell to process the retrieved internal state data and current input to generate a neural network output and updated internal state data for the identified task (340).

[0034] The circuit receives the neural network output and the updated internal state for the identified task from the RNN cell (350). Finally, the circuit writes the updated internal state to the state register corresponding to the identified task (360).

[0035] The example recurrent neural network described with reference to Figures 1 and 2 and the example process described with reference to Figure 3 can be used to predict future memory access addresses in a computing system.

[0036] A computer program causes a computing system to perform operations on data stored in various memory locations within the computing system. Typically, a computing system must load data from main memory or a local cache into local registers before it can perform an operation on the data. However, loading data from main memory is particularly time-consuming because main memory is typically large and physically distant from the local registers. Thus, computer programs that periodically cause a computing system to retrieve data from main memory take longer to execute than computer programs that do not. By prefetching data from main memory and storing it in an earlier local cache before it is needed, a computing system can reduce the runtime of a computer program. Recurrent neural networks can be used to predict data that should be prefetched.

[0037] FIG. 4 shows a process of prefetching data from memory and storing the data in a local cache. 4 illustrates an example computing system 400 configured to store computer program instructions. The computing system 400 is capable of executing computer program instructions using one or more processors (not shown). In particular, the computing system 400 is capable of storing the computer program in an instruction memory and then executing the instructions in an order defined by the computer program.

[0038] The computing system 400 includes a program counter 410. The program counter, or instruction address register, is a register that stores a value indicating the next computer program instruction to be executed. That is, the program counter 410 stores the memory address where the next computer program instruction to be executed is stored.

[0039] When an instruction cycle begins, computing system 400 receives the next computer program instruction from the address specified by program counter 410. The address stored in program counter 410 corresponds to a location in instruction memory 422, which is a block of memory within memory 420.

[0040] Typically, a given computer program instruction specifies an operation, such as a load, store, add, subtract, NOR, branch, etc., one or more source registers, and a destination register. The computing system 400 performs the specified operation on data stored in the source register and stores the result in the destination register. For example, a computer program instruction may specify that data stored in register A should be added to data stored in register B, and that the result should be stored in register C.

[0041] Typically, a computing system has a limited number of local registers, so that data on which an operation is to be performed is loaded into these local registers only when needed. However, fetching data from memory takes time and slows down the execution of a computer program. One solution to this problem is to predict data on which future operations will be performed, prefetch that data, and store it in faster local memory, such as local cache 430. Because a local cache is generally physically closer to the system and smaller than main memory, fetching data from a local cache is faster than fetching data from main memory.

[0042] However, local caches are also limited in size, and a computing system can only store a small subset of main memory in the local cache at any given time. Therefore, computing system 400 can advantageously predict a subset of future memory access addresses and store data from these addresses in the local cache. If the computing system makes accurate predictions, the computing system can execute computer program instructions faster.

[0043] The prediction subsystem 440 of the computing system 400 is configured to receive a sequence of previous program counter addresses and corresponding memory access addresses and to predict future memory access addresses. The prediction subsystem 440 includes a neural network subsystem 442. The neural network subsystem 442 may be the recurrent neural network system described with reference to FIG. 2. In addition to the components described with reference to FIG. 2, the neural network subsystem 442 includes a memory region classifier 444 configured to determine a specific memory region for each memory access from multiple memory regions. Memory accesses within one memory region typically include two or more memory regions. By separating memory accesses by memory region, memory access prediction can be improved because memory accesses across multiple memory regions follow a more predictable pattern. The importance of dividing memory into regions is explained in more detail with reference to Figure 5.

[0044] The prediction subsystem 440 can be implemented on the same computer on which the computer program executes, or it can be implemented on a different computer.

[0045] After prediction subsystem 440 predicts a particular memory access address, computing system 400 prefetches data from that address and stores it in local cache 430. Prediction subsystem 440 continues to predict memory access addresses as the computer program executes, i.e., computing system 400 continues to prefetch data from these addresses. It is faster to fetch data from local cache 430 than to fetch data from data memory 424. This allows computing system 400 to execute computer programs faster.

[0046] Although this specification primarily focuses on data prefetching, the recurrent neural network system described herein can be configured to perform other time series predictions in a microprocessor, for example, the system can be used to prefetch instructions from an instruction memory.

[0047] The system can also be used to optimize memory disambiguation, which is a set of techniques used by computing systems that execute memory access instructions out of order, by detecting or predicting dependencies between memory operations to increase memory access efficiency and speed.

[0048] The recurrent neural network system described herein can also be used to optimize cache replacement policies. Caches have a limited size, and when a computing system places data in a cache, it typically needs to remove a corresponding amount of data from the cache. A recurrent neural network system can predict the best data, e.g., data that is least likely to be used by the computing system in the near future, and remove it from the cache. Alternatively, a recurrent neural network system can predict data that is most likely to be used in the near future and ensure that the data remains in the cache. For example, in a "least recently used" cache, i.e., a cache with a replacement policy that specifies that the least recently used data in the cache should be removed in favor of newer data, the system can update an "age bit" for data that is predicted to be used in the near future. By updating the age bit, it is possible to ensure that the data is not used in the near future. Indeed, data is never replaced by new data.

[0049] 5 is a flow chart illustrating an example process 500 for predicting future memory access addresses of a computer program executing on a computing system. The recurrent neural network system described with reference to FIG. 2 can perform the example process 500. Alternatively, process 500 can be implemented in software. For convenience, process 500 will be described as being performed by a suitably programmed system of one or more computers at one or more locations.

[0050] The system maintains (510) data specifying a separate internal state for each of a plurality of memory regions for a recurrent neural network trained to predict memory access addresses of a computer program within the plurality of memory regions.

[0051] The system receives 520 a current input in a sequence of inputs defining a sequence of previous memory access addresses of the computer program. The sequence of inputs may be a sequence of previous program counter addresses and corresponding delta values. Each delta value defines the difference between a respective first memory access address and a respective second memory access address. The first memory access address defines the memory address accessed when the instruction indicated by the corresponding program counter address is executed. For example, a particular program counter address may indicate a memory address that stores an instruction that, when executed, fetches data from memory. In such a case, the memory address of the data is the first memory access address. The second memory access address is the memory address accessed before the first memory address is accessed. Each previous memory access address is an address within one of multiple memory regions in the main memory of the computing system. The system can determine the memory region by identifying memory pages touched by the memory access. Alternatively, the system can determine the memory region by applying a clustering algorithm, such as a k-means clustering algorithm, to the memory accesses.

[0052] In some implementations, the system generates an input representation of the current input. For example, in an implementation where the current input is a program counter address and a delta value, the system can map the program counter address and the delta value to a numeric embedding in a high-dimensional space that serves as the input representation. This embedding is a high-dimensional real-valued vector.

[0053] The system can generate the embedding using a lookup table generated by a trained neural network. For example, a program counter address and delta value pair can be mapped to an index, and the index can be used to look up the embedding in the lookup table. Alternatively, the system can generate the embedding by providing the program counter address and delta value directly to the trained neural network.

[0054] The system identifies (530) a particular memory region of the memory access address defined by the current input in the sequence of inputs.

[0055] The system selects (540) an internal state for a particular memory region from the internal states specified in the retained data.

[0056] The system processes a current input in the sequence of inputs using a recurrent neural network according to the selected internal state for the particular memory region to generate an output and update the selected internal state (550). The output defines a probability distribution of predicted memory access addresses, which are future memory access addresses of the computer program within the particular memory region.

[0057] The system can set a threshold criterion and determine whether any of the probabilities in the distribution meet the threshold criterion. For example, the threshold criterion can be a probability of 0.5, i.e., a 50% chance that a particular memory address is a future memory access address, i.e., that a computer program will cause a computing system to access data in the memory address. The system can dynamically adjust the threshold criterion. For example, the system can set a higher threshold when memory bandwidth utilization is high and a lower threshold when memory bandwidth utilization is low.

[0058] In response to determining that a particular probability in the probability distribution does not meet the threshold criterion, the system may fetch data from the memory access associated with that probability and store the data in a local cache. In some cases, the system may determine that multiple probabilities meet the threshold criterion and prefetch data from multiple memory addresses associated with those probabilities. Doing so increases the likelihood that correct data is prefetched, thereby reducing the runtime of a computer program running on the computing system.

[0059] The system is typically configured to predict the next memory access address within the identified memory region. However, it is possible to train a recurrent neural network to predict memory access addresses further into the future. For example, a recurrent neural network can be trained to predict memory access addresses two steps ahead (N+2), four steps ahead (N+4), or eight steps ahead (N+8) within the identified memory region. Unlike sequential memory access addresses, a recurrent neural network can be trained to predict two steps ahead memory access addresses, for example, by providing the recurrent neural network with delta values ​​calculated using the memory access address at time step N and the memory access address at time step N-2.

[0060] In some implementations, instead of prefetching data as described above, the system can insert a fetch instruction into the computer program. Execution of the fetch instruction causes data to be fetched from a future memory address associated with a probability that meets the threshold criterion. The fetch instruction needs to be inserted into the computer program before a future instruction of the computer program that would otherwise cause data to be fetched from memory. In this way, when the future instruction is executed, the data required by the future instruction is already present in the local cache. Execution of the future instruction results in a cache hit instead of a cache miss, and the system does not need to fetch the data from memory. This allows the computer program to run faster because fetching data from memory can be very time-consuming. The system can insert two or more fetch instructions into the computer program if two or more probabilities meet the threshold criterion.

[0061] In some implementations, the system may provide additional features to the recurrent neural network. For example, the system may indicate whether the most recent access was a load or a store. The system may also provide the recurrent neural network with the current cache occupancy, which may affect the threshold criteria as described above.

[0062] Finally, the system associates (560) the updated selected internal state with a particular memory region within the persisted data.

[0063] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by, or to control the operation of, a data processing apparatus. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or any of these. Alternatively, or in addition, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0064] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include an off-the-shelf or custom-built parallel processing subsystem, e.g., a GPU or other type of dedicated processing subsystem. An apparatus may also be or further include dedicated logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). An apparatus may optionally, in addition to hardware, include code that creates an execution environment for a computer program, such as processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations of these.

[0065] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, either as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a single file dedicated to the program, or in multiple coordinated files (e.g., one or more files of code). The computer program may be stored in a file that contains other programs or data (e.g., one or more scripts stored in a markup language document), in a file that contains modules, subprograms, or portions of the program, or in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0066] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides an output distinct from an input. An engine may be a coded block of functionality, such as a library, platform, software development kit ("SDK"), or object. Each engine may be implemented on any suitable type of computing device, such as a server, mobile phone, tablet computer, notebook computer, music player, e-reader, laptop or desktop computer, PDA, smartphone, or other stationary or mobile device that includes one or more processors and computer-readable media. Additionally, two or more of the engines may be implemented on the same computing device or on different computing devices.

[0067] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0068] A computer suitable for executing a computer program may be based on a general-purpose or special-purpose microprocessor or both, or on any kind of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Typically, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks, or is operatively coupled to receive data from or transfer data to, the one or more mass storage devices, or both. However, a computer need not have such devices. Furthermore, a computer may be implemented in a variety of ways, for example, as a mobile phone, a portable information It may be embedded in another device such as a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive).

[0069] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices, including semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0070] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse, trackball, or presence-sensitive display or other surface, through which the user can provide input to the computer. Other types of devices may be used to provide for user interaction as well; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; input from the user may be received in any form, including acoustic input, voice input, or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. A computer may also interact with a user by sending text messages or other types of messages to a personal device, such as a smartphone, running a messaging application, and receiving a reply message from the user.

[0071] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of possible features specific to particular embodiments of particular inventions. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as operative in a combination, and may even initially be claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0072] Similarly, while operations are shown in a particular order in the figures, it should not be understood that such operations need to be performed in the particular order shown, or sequential order, or that all of the shown operations need to be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may typically be integrated into a single software product or packaged into multiple software products.

[0073] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method performed using an application specific hardware integrated circuit implementing a recurrent neural network (RNN), comprising: generating a feature representation of a current input to the RNN; selecting a particular internal state of the RNN from a state register of the application specific hardware integrated circuit; processing the feature representation in hardware using the particular internal state selected from the state register and an RNN cell of the application specific hardware integrated circuit; and using a set of layer weights of the RNN to generate i) a prediction for a particular task and ii) a next state of the RNN for the particular task in response to processing the feature representation.

2. The method of claim 1 , wherein the feature representation represents an embedding vector derived from an embedding space of the RNN based on the current input to the RNN.

3. 2. The method of claim 1, wherein generating a prediction for the particular task comprises predicting memory access addresses representing future memory access addresses of a program used to process data input to the RNN.

4. 4. The method of claim 3, wherein the predicted memory access address is one of a plurality of addresses identifying a memory location across a plurality of memory regions of the application-specific hardware integrated circuit that implements the RNN.

5. i) the particular internal state of the RNN is one of a plurality of internal states; ii) each internal state of the plurality of internal states corresponds to a respective different task that the RNN is trained to perform.

6. 2. The method of claim 1, wherein selecting the particular internal state of the RNN comprises selecting the particular internal state of the RNN from the state register of the application-specific hardware integrated circuit based on a sequence identifying data specifying the state register.

7. The method of claim 1 , comprising providing the prediction to another RNN cell of the application specific hardware integrated circuit.

8. 1. A system comprising: an application specific hardware integrated circuit implementing a recurrent neural network (RNN); and one or more storage devices storing instructions that, when executed by a processor of the application specific hardware integrated circuit, cause operations to be performed, the operations comprising: generating a feature representation of a current input to the RNN; selecting a particular internal state of the RNN from a state register of the application specific hardware integrated circuit; processing the feature representation in hardware using the particular internal state selected from the state register and an RNN cell of the application specific hardware integrated circuit; and generating, in response to processing the feature representation, i) a prediction for a particular task and ii) a next state of the RNN for the particular task using a set of layer weights of the RNN.

9. The system of claim 8 , wherein the feature representation represents an embedding vector derived from an embedding space of the RNN based on the current input to the RNN.

10. 9. The system of claim 8, wherein generating a prediction for the particular task comprises predicting memory access addresses representing future memory access addresses of a program used to process data input to the RNN.

11. 11. The system of claim 10, wherein the predicted memory access address is one of a plurality of addresses identifying a memory location across a plurality of memory regions of the application-specific hardware integrated circuit that implements the RNN.

12. i) the particular internal state of the RNN is one of a plurality of internal states; 9. The system of claim 8, wherein ii) each internal state of the plurality of internal states corresponds to a respective different task that the RNN is trained to perform.

13. 9. The system of claim 8, wherein selecting the particular internal state of the RNN includes selecting the particular internal state of the RNN from the state register of the application-specific hardware integrated circuit based on a sequence that identifies data that specifies the state register.

14. 9. The system of claim 8, wherein the operation further comprises providing the prediction to another RNN cell of the application specific hardware integrated circuit.

15. The present invention includes instructions that, when executed by a processor of an application specific hardware integrated circuit implementing a recurrent neural network (RNN), cause the processor to perform operations, the operations including: generating a feature representation of a current input to the RNN; selecting a particular internal state of the RNN from a state register of the application specific hardware integrated circuit; processing the feature representation in hardware using the particular internal state selected from the state register and an RNN cell of the application specific hardware integrated circuit; and generating, in response to processing the feature representation, i) a prediction for a particular task and ii) a next state of the RNN for the particular task using a set of layer weights of the RNN.

Images