Multi-task recurrent neural networks

JP2024119850A5Active Publication Date: 2024-11-21GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024083470
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-01-30
Filing Date
2024-05-22
Publication Date
2024-11-21
Estimated Expiration
2039-11-15

AI Technical Summary

Technical Problem

Existing neural networks struggle with efficiently performing multiple tasks in quick succession due to sequence dependencies, leading to increased latency in time series predictions, particularly in microprocessors.

Method used

Implementing a multi-tasking recurrent neural network (RNN) with state registers to store internal states for different tasks, allowing the network to perform tasks in parallel by retrieving and updating states from dedicated registers, reducing sequence dependencies and latency.

Benefits of technology

The multi-tasking RNN reduces latency by enabling simultaneous processing of multiple tasks, improving time series predictions, such as memory access, cache misses, and branch predictions, by storing and retrieving task-specific states efficiently.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To provide methods, systems, and apparatuses, including computer programs encoded on computer storage media, relating to multi-task recurrent neural networks.SOLUTION: A method includes maintaining data specifying, for a recurrent neural network, a separate internal state for each of a plurality of memory regions; receiving a current input; identifying a particular memory region of a memory access address defined by the current input; selecting, from the internal states specified in the maintained data, the internal state for the particular memory region; processing, in accordance with the selected internal state for the particular memory region, the current input in a sequence of inputs using the recurrent neural network to generate an output defining a probability distribution of a predicted memory access address, and update the selected internal state of the particular memory region; and associating the updated selected internal state with the particular memory region in the maintained data.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] background This specification relates to recurrent neural networks. [Background technology]

[0002] A neural network is a machine learning model that uses the operation of multiple layers to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers located between the input layer and the output layer. The output of each hidden layer is used as the input to the next layer, e.g. the next hidden layer or the output layer.

[0003] Each layer of a neural network specifies one or more transformation operations to be performed on the inputs to the layer. Some neural network layers have operations called neurons. Each neuron receives one or more inputs and produces outputs that are received by other neural network layers. Often, each neuron receives inputs from other neurons, and each neuron provides outputs to one or more other neurons.

[0004] Each layer generates one or more outputs using the current values ​​of the set of parameters for that layer. Training the neural network involves successively making forward passes over the inputs, computing gradients, and updating the current values ​​of the set of parameters for each layer. Once the neural network is trained, it can be used to make predictions in a production system using the final set of parameters.

[0005] Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from previous time steps when computing the output at the current time step. This allows a recurrent neural network to exhibit dynamic behavior in time. That is, a recurrent neural network can aggregate all the information received up to the current time step and collect long-term dependencies of a sequence of data. Summary of the Invention [Means for solving the problem]

[0006] overview This specification describes a multi-tasking recurrent neural network, and more particularly, a multi-tasking recurrent neural network that improves time series prediction in a microprocessor.

[0007] A circuit implementing a multitasking recurrent neural network includes recurrent neural network cells ("RNN cells") configured to perform the operations of the recurrent neural network. That is, the RNN cells are configured to perform the transformation operations defined by each layer in the recurrent neural network. The circuit also includes state registers. Each state register (i) is configured to store internal state data for the recurrent neural network and (ii) corresponds to a respective task that the neural network is trained to perform. State data is data that persists across multiple time steps, i.e., data retained after processing one input in a sequence to be used in processing the next input in the sequence. For example, a recurrent neural network may store state data for long and short inputs. For a linear memory (LSTM) network, the state data may include a hidden state h and a cell state c for each LSTM unit in the network.

[0008] The circuit receives a current input in a sequence of inputs and sequence identification data. The circuit identifies a particular task based on the sequence identification data. The circuit retrieves internal state data from a state register corresponding to the identified task. The circuit provides the retrieved internal state data and the current input to an RNN cell to cause the RNN cell to process the retrieved internal state data and the current input to generate a neural network output and updated internal state data for the identified task. The circuit receives the neural network output and the updated internal state for the identified task from the RNN cell. The circuit finally writes the updated internal state data to the state register corresponding to the identified task.

[0009] The circuit described above can be used to more accurately predict memory accesses in a computing system according to the method described below. The circuit can hold data specifying a separate internal state for each of a plurality of memory regions for a recurrent neural network trained to predict memory access addresses of a computer program in one of the memory regions, receive a current input in a sequence of inputs defining a sequence of previous memory access addresses of the computer program, identify a particular memory region of the memory access address defined by the current input in the sequence of inputs, select an internal state for the particular memory region from the internal states specified in the held data, process the current input in the sequence of inputs with the recurrent neural network according to the selected internal state for the particular memory region to (i) generate an output defining a probability distribution of predicted memory access addresses that are future memory access addresses of the computer program in the particular memory region, and (ii) update the selected internal state for the particular memory region, and associate the updated selected internal state with the particular memory region in the held data.

[0010] In some embodiments, one or more computer-readable storage media are encoded with instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform operations including methods and steps according to any embodiment or aspect described herein.

[0011] In some embodiments, the system comprises one or more computers and one or more storage devices that store instructions that, when executed by the one or more computers, cause the computers to perform methods and steps according to any embodiment or aspect described herein.

[0012] The subject matter described herein can be implemented in certain embodiments to achieve one or more of the following advantages: An implementation of a recurrent neural network in hardware that includes storing recurrent neural network states in state registers coupled to the recurrent neural network allows the recurrent neural network to perform different tasks in quick succession, known as multitasking. More specifically, each state register can store a state for a respective task. The recurrent neural network can obtain a first state from a first state register to process a first input sequence, and obtain a second state from a second state register to process a second, different input sequence before completing processing of the first input sequence. For example, a hardware circuit implementing a recurrent neural network can predict memory accesses, cache misses, and branches simply by obtaining a state corresponding to one of the tasks from the respective state registers. Storing state data in the state registers can be used to predict memory accesses, cache misses, and branches. By storing the sequence dependencies, the number of sequence dependencies that the RNN must compute at each time step is reduced, reducing the overall latency by the length of the input sequence.

[0013] This specification also describes how classification of input sequences and partitioning by latent factors may be performed. In data prefetching, one example of partitioning by latent factors is to separate an interleaved stream of memory access addresses in different regions of memory. Each of these regions may have different memory access patterns, and the machine learning model can learn more effectively if the stream is partitioned before being used as an input to the recurrent neural network. Another example of partitioning by latent factors in data prefetching is to separate the stream of memory accesses by the data structure that caused these memory accesses. Again, by doing so, the machine learning model can be trained more effectively.

[0014] One or more embodiments of the subject matter herein are described in detail in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 illustrates an example of a recurrent neural network implemented in hardware. [Diagram 2] FIG. 1 illustrates an example of a recurrent neural network system having a state register and a multiplexer. [Diagram 3] 1 is a flowchart illustrating an example of a process for processing a multi-task recurrent neural network using hardware circuitry. [Figure 4] FIG. 1 illustrates an example of a computing system configured to prefetch data from memory. [Diagram 5] 1 is a flowchart illustrating an example of a process for predicting future memory access addresses of a computer program executed on a computing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description This specification describes a multi-tasking recurrent neural network, and more particularly, a multi-tasking recurrent neural network that improves time series prediction in a microprocessor.

[0017] Figure 1 illustrates a hardware-implemented recurrent neural network 100. A recurrent neural network typically includes an input layer, one or more hidden layers, and an output layer.

[0018] The input layer of a recurrent neural network is typically configured to receive a sequence of inputs, including respective inputs at each time step, from 0 to the current time step, N, and to process the sequence of inputs to generate a feature representation of the sequence.

[0019] The hidden layers of a recurrent neural network are recurrent layers, meaning that they maintain state between inputs in a sequence and update the current state as part of generating the output for the current time step. Traditional recurrent neural networks have hidden layers that calculate the output and next state for the layer, where the next state is based on the previous state and the current input. The state is maintained across time steps and updates the state of the input sequence. Capture dependencies.

[0020] A particular type of recurrent neural network is the long short-term memory (LSTM) network. The hidden layer in an LSTM network contains one LSTM unit for each input in the input sequence. Each LSTM unit provides outputs to other LSTM units in its own layer and to LSTM units in the next hidden or output layer. Each LSTM unit contains a hidden state h, a cell state c, an input gate i, a forget gate f, and an output gate o. Each of these three gates can be thought of as an individual neuron that computes a weighted sum activation. Intuitively, the input gate controls the extent to which new values ​​flow into the cell, the forget gate controls the extent to which values ​​stay in the cell, and the output gate controls the extent to which values ​​in the cell are used to compute the LSTM unit's output activation. That is, these gates control what the LSTM unit "remembers," or the manner in which its state is affected by previous states.

[0021] The output layer of a recurrent neural network is typically configured to receive inputs from the last hidden layer and generate an output, e.g., a prediction, at each time step. The output layer may be a softmax layer, a logistic layer, a linear layer, or the like. The softmax output layer generates a discrete probability distribution over the possible outputs for the time step. That is, each possible output is associated with a discrete point number. The logistic output layer generates parameters, e.g., the mean, log scale, or mixture weights of the distribution over the possible outputs for the time step. And, the linear layer projects the output of the last hidden layer into a suitable output space. In some implementations, the recurrent neural network does not have an output layer, i.e., the output of the last hidden layer is the output of the recurrent neural network.

[0022] Training data for a recurrent neural network includes sequences of inputs and labels for these sequences of inputs. Each label corresponds to a desired or expected output at a particular time step for a particular sequence of inputs. Training a recurrent neural network involves setting the weights of the network, e.g., the weights of the input layer, the RNN cells, and the output layer, to minimize the discrepancy between the actual output and the predicted output for a given sequence of inputs. If there is a discrepancy between the actual output and the predicted output for a given sequence of inputs, the weights of the recurrent neural network can be updated, e.g., using conventional backpropagation over time training techniques. Other techniques known in the art can also be used to update the weights of the recurrent neural network.

[0023] Recurrent neural network 100 is a hardware implementation of a recurrent neural network that includes one state register per hidden layer and one RNN cell per hidden layer. In some implementations, each RNN cell is a processing unit in a systolic array of processing units. Each processing unit performs the row multiplication and other operations of the recurrent neural network layer in hardware, for example by including a set of multiply-accumulate units that collectively perform the row multiplication and other circuitry that applies any necessary gating functions and performs element-wise multiplication.

[0024] At time step N, input unit 110 receives input N in a sequence of inputs and generates a feature representation of that input. RNN cells 120 and 130 process the feature representation using both the fixed parameters of the recurrent neural network and the state data stored in state registers for each hidden layer. The output of RNN cell 130 is then provided to output unit 140, which generates an output for time step N+1. RNN cells 120 and 130 also update the state data in their respective state registers to reflect input N. This state data reflects all previous inputs and states. Capture.

[0025] FIG. 2 illustrates an example of a recurrent neural network system 200 that utilizes state registers in different ways. Each of the state registers 210a-210n can store data that specifies a different internal state of the recurrent neural network. Each internal state, and therefore each state register, corresponds to a different task that the neural network is trained to perform. For example, a first state register can store data that specifies an internal state that corresponds to a prediction of a memory access address, i.e., loading and storing addresses, while a second state register can store data that specifies an internal state that corresponds to a prediction of a branch of a computer program. Meanwhile, other state registers can store data that specifies an internal state that corresponds to, for example, a prediction of a future program instruction or a cache miss.

[0026] The RNN selector 220, which is a multiplexer, can select a particular internal state that the recurrent neural network uses at a particular time based on the sequence identification data. The sequence identification data specifies the state registers, and therefore the internal states to be loaded into the RNN computational cell 230. The sequence identification data may be based on a user input that specifies a particular task that the RNN computational cell 230 should perform, for example, predicting a memory access address (as opposed to a branch, program instruction, or cache miss). Alternatively, the sequence identification data may be based on an observation about the current input. For example, the sequence identification data may be based on an observation that the current input is a cache address. In such a situation, the sequence identification data may specify that an internal state corresponding to the task of predicting a cache miss should be loaded into the RNN computational cell 230.

[0027] The recurrent neural network system 200 also includes an input unit 240 that is loaded with a set of fixed parameters. The input unit 240 is capable of receiving a current input, generating a feature representation of the current input, and providing the feature representation to the RNN computational cell 230.

[0028] Using the selected state, the feature representation, and a set of fixed weights held in the RNN computational cell 230, the RNN computational cell 230 can generate a prediction and a next state for the task. If the recurrent neural network has multiple hidden layers, the RNN computational cell 230 can provide the generated prediction to other RNN computational cells, which may have their own state registers and RNN selectors. The recurrent neural network system may also optionally include an output unit. Meanwhile, the RNN computational cell 230 can provide the appropriate state register 210a-210n with the next state for the task, overwriting the previous state of that register. Training a multitasking recurrent neural network as described above requires training the recurrent neural network to perform each task.

[0029] Alternatively, state registers 210a-210n can each store data specifying a separate internal state for each of a plurality of memory regions for a recurrent neural network trained to predict memory access addresses of a computer program in a plurality of memory regions. By dividing memory accesses by memory region in this manner, memory access prediction can be improved because memory accesses within one memory region typically follow a more predictable pattern than memory accesses across two or more memory regions. That is, RNN computational cell 230 produces more accurate predictions when predicting memory accesses in a particular memory region rather than across the entire address space.

[0030] 3 is a flow chart illustrating an example of a process 300 for processing a multitasking recurrent neural network. The process 300 is performed by a suitably programmed circuit. The circuit comprises an RNN cell configured to perform the operations of the recurrent neural network, a number of state registers, and other circuit components. Each state register (i) is configured to store internal state data for the recurrent neural network and (ii) corresponds to a respective one of a number of tasks that the neural network is trained to perform.

[0031] The circuit receives 310 a current input in a sequence of inputs and sequence identification data. The input sequence includes a respective input at each of a number of time steps. The time steps may or may not be consecutive. The sequence identification data specifies a task, and therefore a state register, whose internal state is loaded into the RNN cell.

[0032] The circuit identifies (320) a particular task to be performed based on the sequence identification data. The sequence identification data may be based on a user input that specifies a particular task for the RNN cell to perform, e.g., predict a memory access address (as opposed to a branch, a program instruction, or a cache miss). Alternatively, the sequence identification data may be based on an observation about the current input. For example, the sequence identification data may be based on an observation that the current input is a cache address. In such a situation, the sequence identification data may specify that an internal state corresponding to the task of predicting a cache miss should be loaded into the RNN cell.

[0033] The circuitry retrieves internal state data from a state register corresponding to the identified task (330). The circuitry provides the retrieved internal state data and the current input to the RNN cell and causes the RNN cell to process the retrieved internal state data and the current input to generate a neural network output and updated internal state data for the identified task (340).

[0034] The circuit receives the neural network output and the updated internal state for the identified task from the RNN cell (350). The circuit finally writes the updated internal state to the state register corresponding to the identified task (360).

[0035] The example recurrent neural network described with reference to Figures 1 and 2 and the example process described with reference to Figure 3 can be used to predict future memory access addresses in a computing system.

[0036] A computer program causes a computing system to perform operations on data stored in various memory locations within the computing system. Typically, the computing system must load data from main memory or a local cache into local registers before it can perform operations on the data. However, loading data from main memory is particularly time-consuming because main memory is typically large and physically separate from the local registers. Thus, computer programs that periodically cause a computing system to retrieve data from main memory take longer to execute than computer programs that do not. By pre-fetching data from main memory and storing it in an earlier local cache before it is needed, a computing system can reduce the runtime of a computer program. Recurrent neural networks can be used to predict data that should be pre-fetched.

[0037] FIG. 4 shows a block diagram of a method for prefetching data from memory and storing the data in a local cache. 4 illustrates an example of a computing system 400 configured to store computer program instructions. The computing system 400 is capable of executing computer program instructions using one or more processors, not shown. In particular, the computing system 400 is capable of storing the computer program in an instruction memory and then executing the instructions in an order defined by the computer program.

[0038] The computing system 400 includes a program counter 410. The program counter, or instruction address register, is a register that stores a value indicating the next computer program instruction to be executed. That is, the program counter 410 stores the memory address at which the next computer program instruction to be executed is stored.

[0039] When an instruction cycle begins, the computing system 400 receives the next computer program instruction from an address specified by a program counter 410. The address stored in the program counter 410 corresponds to a location in an instruction memory 422, which is a block of memory in memory 420.

[0040] Typically, a given computer program instruction specifies an operation, e.g., load, store, add, subtract, NOR, branch, one or more source registers, and a destination register. The computing system 400 performs the specified operation on data stored in the source register and stores the result in the destination register. For example, a computer program instruction may specify that data stored in register A should be added to data stored in register B, and that the result should be stored in register C.

[0041] Typically, a computing system has a limited number of local registers, so that data on which an operation is to be performed is loaded into these local registers only when necessary. However, fetching data from memory takes time and slows down the execution of a computer program. One solution to this problem is to predict data on which an operation will be performed in the future, prefetch the data, and store it in a faster local memory, such as the local cache 430. Because the local cache is generally physically closer to the system and smaller than the main memory, fetching data from the local cache is faster than fetching data from the main memory.

[0042] However, the local cache is also limited in size, and the computing system can only store a small subset of the main memory in the local cache at any given time. Thus, the computing system 400 can advantageously predict a subset of future memory access addresses and store data from these addresses in the local cache. If the computing system makes accurate predictions, the computing system can execute computer program instructions faster.

[0043] The prediction subsystem 440 of the computing system 400 is configured to receive a sequence of previous program counter addresses and corresponding memory access addresses and predict future memory access addresses. The prediction subsystem 440 includes a neural network subsystem 442. The neural network subsystem 442 may be a recurrent neural network system as described with reference to FIG. 2. In addition to the components described with reference to FIG. 2, the neural network subsystem 442 includes a memory region classifier 444 configured to determine a specific memory region for each memory access from multiple memory regions. A memory access in one memory region typically has two or more memory regions. Since memory accesses across multiple memory regions follow a more predictable pattern than memory accesses across multiple memory regions, separating memory accesses by memory region can improve memory access prediction. The importance of dividing memory into regions is explained in more detail with reference to FIG. 5.

[0044] The prediction subsystem 440 can be implemented on the same computer on which the computer program executes, or it can be implemented on a different computer.

[0045] After the prediction subsystem 440 predicts a particular memory access address, the computing system 400 prefetches data from that address and stores it in the local cache 430. The prediction subsystem 440 continues to predict memory access addresses as the computer program executes, i.e., the computing system 400 continues to prefetch data from these addresses. It is faster to fetch data from the local cache 430 than to fetch data from the data memory 424. This allows the computing system 400 to execute the computer program faster.

[0046] Although this specification primarily focuses on data prefetching, the recurrent neural network system described herein can be configured to perform other time series predictions in a microprocessor, for example, the system can be used to prefetch instructions from an instruction memory.

[0047] The system can also be used to optimize memory disambiguation, which is a set of techniques used by computing systems that execute memory access instructions out of order, to detect or predict dependencies between memory operations to increase memory access efficiency and speed.

[0048] The recurrent neural network system described herein can also be used to optimize cache replacement policies. Caches are limited in size, and when a computing system places data in a cache, it typically needs to remove a corresponding amount of data from the cache. The recurrent neural network system can predict the best data, e.g., data that is least likely to be used by the computing system in the near future, to be removed from the cache. Alternatively, the recurrent neural network system can predict data that is most likely to be used in the near future and ensure that the data stays in the cache. For example, in a "least recently used" cache, i.e., a cache that has a replacement policy that specifies that the least recently used data in the cache should be removed in favor of new data, the system can update an "age bit" for data that is predicted to be used in the near future. Updating the age bit ensures that the data is not used in the near future. Indeed, data is never replaced by new data.

[0049] Figure 5 is a flow chart illustrating an example of a process 500 for predicting future memory access addresses of a computer program executed on a computing system. The recurrent neural network system described with reference to Figure 2 may perform the example of process 500. Alternatively, process 500 may be implemented in software. For convenience, process 500 will be described as being performed by a suitably programmed system of one or more computers at one or more locations.

[0050] The system maintains (510) data specifying a separate internal state for each of a plurality of memory regions for a recurrent neural network trained to predict memory access addresses of a computer program in the plurality of memory regions.

[0051] The system receives (520) a current input in a sequence of inputs that defines a sequence of previous memory access addresses of the computer program. The sequence of inputs may be a sequence of previous program counter addresses and corresponding delta values. Each delta value defines a difference between a respective first memory access address and a respective second memory access address. The first memory access address defines a memory address accessed when an instruction indicated by the corresponding program counter address is executed. For example, a particular program counter address may indicate a memory address that stores an instruction that, when executed, fetches data from memory. In such a case, the memory address of the data is the first memory access address. The second memory access address is a memory address accessed before the first memory address is accessed. Each previous memory access address is an address within one of a plurality of memory regions in a main memory of the computing system. The system may determine the memory regions by identifying memory pages that the memory access touches. Alternatively, the system may determine the memory regions by applying a clustering algorithm, e.g., a k-means clustering algorithm, to the memory accesses.

[0052] In some implementations, the system generates an input representation of the current input. For example, in an implementation where the current input is a program counter address and a delta value, the system can map the program counter address and the delta value to a numeric embedding in a high-dimensional space that serves as the input representation. The embedding is a high-dimensional real-valued vector.

[0053] The system can generate the embeddings using a lookup table generated by the trained neural network. For example, a program counter address and delta value pair can be mapped to an index, and the index can be used to look up the embedding in the lookup table. Alternatively, the system can generate the embeddings by providing the program counter address and delta value directly to the trained neural network.

[0054] The system identifies (530) a particular memory region of a memory access address defined by a current input in a sequence of inputs.

[0055] The system selects (540) an internal state for a particular memory region from the internal states specified in the retained data.

[0056] The system processes a current input in the sequence of inputs using a recurrent neural network according to the selected internal state for the particular memory region to generate an output and update the selected internal state (550). The output defines a probability distribution of predicted memory access addresses, which are future memory access addresses of the computer program in the particular memory region.

[0057] The system can set a threshold criterion and determine whether any of the probabilities in the distribution meet the threshold criterion. For example, the threshold criterion can be a probability of 0.5, i.e., a 50% probability that a particular memory address is a future memory access address, i.e., that a computer program will cause a computing system to access data in the memory address. The system can dynamically adjust the threshold criterion. For example, the system can set a higher threshold when memory bandwidth utilization is high and a lower threshold when memory bandwidth utilization is low.

[0058] In response to determining that a particular probability in the probability distribution does not meet the threshold criterion, the system can fetch data from a memory access associated with that probability and store the data in a local cache. In some cases, the system may determine that multiple probabilities meet the threshold criterion and prefetch data from multiple memory addresses associated with those probabilities. Doing so increases the likelihood that correct data is prefetched. This reduces the runtime of a computer program executing on a computing system.

[0059] The system is typically configured to predict the next memory access address in the identified memory region. However, the recurrent neural network can be trained to predict memory access addresses further into the future. For example, the recurrent neural network can be trained to predict memory access addresses two (N+2), four (N+4), or eight (N+8) ahead in the identified memory region. The recurrent neural network can be trained to predict memory access addresses two ahead, as opposed to sequential memory access addresses, by providing the recurrent neural network with delta values ​​calculated using the memory access address at time step N and the memory access address at time step N-2, respectively.

[0060] In some implementations, instead of pre-fetching data as described above, the system can insert a fetch instruction into the computer program. Execution of the fetch instruction causes data to be fetched from a future memory address associated with a probability that meets the threshold criterion. The fetch instruction needs to be inserted into the computer program before a future instruction of the computer program where data would otherwise be fetched from memory. In this way, when the future instruction is executed, the data required by the future instruction is already present in the local cache. Execution of the future instruction results in a cache hit instead of a cache miss, and the system does not need to fetch the data from memory. This allows the computer program to run faster, since fetching data from memory is very time consuming. The system can insert two or more fetch instructions into the computer program if two or more probabilities meet the threshold criterion.

[0061] In some implementations, the system also provides other features to the recurrent neural network. For example, the system can indicate whether the most recent access was a load or a store. The system can also provide the recurrent neural network with the current cache occupancy. The current cache occupancy can affect the threshold criteria as described above.

[0062] Finally, the system associates (560) the updated, selected internal state with a particular memory region within the persisted data.

[0063] Embodiments of the subject matter and functional operations described herein may be realized in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, or in one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be realized as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or any combination thereof. Alternatively, or in addition, the program instructions may be encoded on an artificially generated, propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0064] The term "data processing apparatus" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or further include an off-the-shelf or custom parallel processing subsystem, such as a GPU or other type of dedicated processing subsystem. The apparatus may also be or further include a dedicated logic circuit component, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the apparatus may optionally include code that creates an environment for the execution of a computer program, such as a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0065] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, either as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a single file dedicated to that program, or in multiple coordinated files (e.g., one or more files of code). A computer program may be stored in a file that stores modules, subprograms or portions of the program, in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0066] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides an output that is distinct from an input. An engine may be a coding block of functionality, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine may be implemented on any suitable type of computing device, such as a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-reader, a laptop or desktop computer, a PDA, a smartphone, or other stationary or mobile device that includes one or more processors and a computer-readable medium. Additionally, two or more of the engines may be implemented on the same computing device or on different computing devices.

[0067] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0068] A computer suitable for executing a computer program may be based on a general purpose or special purpose microprocessor or both, or on any kind of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by, or incorporated in, special purpose logic circuitry. Typically, a computer also includes one or more mass storage devices for storing data, e.g., magnetic disks, magneto-optical disks or optical disks, or is operatively coupled to receive data from or transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices. Additionally, a computer may be implemented in a variety of ways, e.g., a mobile phone, a portable information device, a The portable information processing device may be embedded in another device, such as a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (eg, a universal serial bus (USB) flash drive).

[0069] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices, including semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0070] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse, trackball, or presence-sensitive display or other surface, through which the user can provide input to the computer. Other types of devices may be used to provide for user interaction as well; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; input from the user may be received in any form, including acoustic input, speech input, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone, running a messaging application, and receiving a response message from the user.

[0071] Although the specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination, and may even be initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0072] Similarly, although operations are shown in a particular order in the figures, it should not be understood that such operations need to be performed in the particular order shown, or sequential order, to achieve desirable results, or that all of the shown operations need to be performed. In some situations, multitasking and parallel processing may be advantageous. Furthermore, it should be understood that the separation of various system modules and components in the above-described embodiments does not require such separation in all embodiments, and that the program components and systems described may typically be integrated into a single software product or packaged into multiple software products.

[0073] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method implemented using an application specific hardware integrated circuit implementing a recurrent neural network (RNN), comprising: generating a feature representation of a current input to the RNN; selecting a particular internal state of the RNN from a state register of the application specific hardware integrated circuit; processing said feature representation in hardware using said particular internal state selected from said state register and an RNN cell of said application specific hardware integrated circuit; and using a set of weights of a layer of the RNN to generate i) a prediction for a particular task, and ii) a next state of the RNN for the particular task, in response to processing the feature representation.

2. The method of claim 1 , wherein the feature representation represents an embedding vector derived from an embedding space of the RNN based on the current input to the RNN.

3. 2. The method of claim 1, wherein generating a prediction for the particular task comprises predicting memory access addresses that represent future memory access addresses of a program used to process data input to the RNN.

4. 4. The method of claim 3, wherein the predicted memory access address is one of a plurality of addresses identifying a memory location across a plurality of memory regions of the application specific hardware integrated circuit that implements the RNN.

5. i) the particular internal state of the RNN is one of a plurality of internal states; 2. The method of claim 1, wherein ii) each internal state of the plurality of internal states corresponds to a respective different task that the RNN is trained to perform.

6. 2. The method of claim 1, wherein selecting the particular internal state of the RNN comprises selecting the particular internal state of the RNN from the state register of the application-specific hardware integrated circuit based on a sequence that identifies data that specifies the state register.

7. The method of claim 1 , comprising providing the prediction to another RNN cell of the application specific hardware integrated circuit.

8. 1. A system comprising: an application specific hardware integrated circuit implementing a recurrent neural network (RNN); and one or more storage devices storing instructions that, when executed by a processor of the application specific hardware integrated circuit, cause operations to be performed, the operations comprising: generating a feature representation of a current input to the RNN; selecting a particular internal state of the RNN from a state register of the application specific hardware integrated circuit; processing said feature representation in hardware using said particular internal state selected from said state register and an RNN cell of said application specific hardware integrated circuit; and using a set of weights of a layer of the RNN to generate i) a prediction for a particular task, and ii) a next state of the RNN for the particular task, in response to processing the feature representation.

9. The system of claim 8 , wherein the feature representation represents an embedding vector derived from an embedding space of the RNN based on the current input to the RNN.

10. 9. The system of claim 8, wherein generating a prediction for the particular task comprises predicting memory access addresses that represent future memory access addresses of a program used to process data input to the RNN.

11. 11. The system of claim 10, wherein the predicted memory access address is one of a plurality of addresses identifying a memory location across a plurality of memory regions of the application specific hardware integrated circuit that implements the RNN.

12. i) the particular internal state of the RNN is one of a plurality of internal states; 9. The system of claim 8, wherein ii) each internal state of the plurality of internal states corresponds to a respective different task that the RNN is trained to perform.

13. 9. The system of claim 8, wherein selecting the particular internal state of the RNN includes selecting the particular internal state of the RNN from the state register of the application-specific hardware integrated circuit based on a sequence that identifies data that specifies the state register.

14. 10. The system of claim 8, wherein the operations further include providing the prediction to another RNN cell of the application specific hardware integrated circuit.

15. The invention includes instructions that, when executed by a processor of an application specific hardware integrated circuit implementing a recurrent neural network (RNN), cause the processor to perform operations, the operations including: generating a feature representation of a current input to the RNN; selecting a particular internal state of the RNN from a state register of the application specific hardware integrated circuit; processing said feature representation in hardware using said particular internal state selected from said state register and an RNN cell of said application specific hardware integrated circuit; and generating, in response to processing the feature representation, i) a prediction for a particular task, and ii) a next state of the RNN for the particular task, using a set of weights of a layer of the RNN.