COMPUTER-IMPLEMENTED METHOD, SYSTEM, ONE OR MORE NON-TRANSITIOUS COMPUTER STORAGE MEDIA FOR COMPUTER SYSTEM PREDICTION MODELS USING MACHINE LEARNING

Abstract

Computer-implemented method comprising: Receiving (210) a sequence of previous program counter addresses (110) of a computer program and corresponding delta values, each delta value defining a difference between a respective first memory address (122) and a respective second memory address (124), the first memory address being a memory address from which data was retrieved when an instruction pointed to by the corresponding program counter address (110) was executed, and the second memory address (124) being a memory address from which data was retrieved before data was retrieved from the first memory address (122); generating (220) an input representation based on the sequence of program counter addresses (110) and their corresponding delta values; Providing (230) the input representation as input to a recurrent neural network (300); Receiving (240) from the recurrent neural network (300) an output defining a probability distribution of future delta values, each probability in the distribution representing a probability that execution of a future instruction of the computer program will cause data to be retrieved from a future memory address equal to (i) a respective first memory address (122) from which data was retrieved when an instruction pointed to by a last program counter address (110) in the sequence executed, and (ii) the future delta value corresponding to the probability; Determining that one or more probabilities in the distribution satisfy a threshold criterion; and automatically inserting one or more retrieval instructions into the computer program, wherein execution of the one or more fetch instructions causes data to be fetched from the one or more future memory addresses associated with the one or more probabilities satisfying the criterion, and wherein the one or more fetch instructions are inserted into the computer program before future instructions of the computer program.

Description

TECHNICAL FIELD

[0001] This specification relates to techniques for prefetching data from a memory using neural networks and a computer-implemented method, system, one or more non-transitory computer storage media for computer system prediction models with machine learning. BACKGROUND

[0002] Data prefetchers use hardware and / or software to predict future memory addresses from which data will be fetched based on a previous history of memory accesses.

[0003] Neural networks are machine learning models that employ multiple layers of operations to predict one or more outputs from one or more inputs. Neural networks typically contain one or more hidden layers sandwiched between an input layer and an output layer. The output of each layer is used as input to another layer, such as the next hidden layer or the output layer.

[0004] Each layer of a neural network specifies one or more transformation operations to be performed on the input to that layer. Some layers of a neural network contain operations called neurons. Each neuron receives one or more inputs and produces an output, which is then passed through another layer of a neural network. Often, each neuron receives inputs from other neurons, and each neuron provides an output to one or more other neurons.

[0005] The architecture of a neural network defines which layers are included in the network and their properties, as well as how the neurons in each layer of the network are connected. In other words, the architecture defines which layers provide their output as input to which other layers and how the output is provided.

[0006] Each layer generates one or more outputs using the current values ​​of a parameter set for that layer. Training the network thus involves continuously performing a feedforward pass on the input, calculating gradient values, and updating the current values ​​of the parameter set for each layer. Once a neural network is trained, the final parameter set can be used to make predictions in a production system.

[0007] Peled, Leeor; Weiser, Uri; Etsion, Yoav: Towards Memory Prefetching with Neural Networks: Challenges and Insights. In: Computing Research Repository (CoRR), March 19, 2018, pp. 1–13, describes the use of a neural network for memory prefetching. A conceptual neural network prefetcher is described and shown that, by focusing on semantic locality, this prefetcher can learn different memory access patterns that cannot be covered by other state-of-the-art prefetchers.

[0008] Peled, Leeor [et al.]: Semantic Locality and Context-Based Prefetching Using Reinforcement Learning. In: SIGARCH Comput. Archit. News, 43, 13.06.2015, 35, pp. 285-297 describes the concept of semantic locality, which uses the inherent semantics of programs to characterize access relationships. It shows how semantic locality can fundamentally capture the relationships between data elements independent of the actual data layout and that semantic locality goes beyond spatio-temporal aspects.

[0009] One objective of the methods and systems described herein is to provide highly accurate cues for memory prefetching using neural networks. SUMMARY

[0010] The problem is solved by the subject matter of the independent patent claims.

[0011] This specification refers to techniques for prefetching data from memory using neural networks.

[0012] A system for prefetching data from memory receives a sequence of previous program counter addresses of a computer program and corresponding delta values. Each delta value defines a difference between a respective first memory address and a respective second memory address. The first memory address is a memory address that was accessed when an instruction pointed to by the corresponding program counter address was executed. The second memory address is a memory address that was accessed before the first memory address was accessed. For example, the second memory address may be a memory address that was accessed immediately previously, or it may be a memory address that was accessed several or more memory operations previously.

[0013] The system then converts the sequence of program counter addresses into an input representation. The system provides the input representation as input to a recurrent neural network. In some implementations, the input representation is a sequence of numerical embeddings in a high-dimensional embedding space. In other implementations, the input representation is a set of pairs of a program counter address and a delta value that are "k-means clustered."

[0014] The system receives an output from the recurrent neural network that defines a probability distribution over future delta values. Each probability in the distribution represents a probability that the execution of a future instruction of the computer program will cause data to be fetched from a future memory address equal to: (i) a respective first memory address accessed when an instruction pointed to by a last program counter address in the sequence was executed, and (ii) the future delta value corresponding to the probability.

[0015] The subject matter described in this specification may be implemented in certain embodiments to realize one or more of the following advantages. The described system can more accurately predict future memory addresses from which data will be retrieved using a recurrent neural network to detect long-term dependencies in a sequence of program counter addresses and corresponding delta values. The system can prefetch data from the future memory addresses it predicts and store that data in a local cache. Correct prefetching reduces the number of cache misses in a computing system. Because retrieving data from a local cache is faster than retrieving data from main memory, the runtime of computer programs executing in the computing system is reduced.Using a recurrent neural network as described in this specification, the system can detect even complex and irregular memory access patterns and therefore reduce the running time of computer programs, even when conventional prefetching approaches would fail to provide a significant improvement.

[0016] Reducing the number of cache misses also reduces the energy consumption of the computing system because retrieving data from main memory is more energy intensive than retrieving data from a local cache.

[0017] The recurrent neural network can be trained offline. In other words, the recurrent neural network can be trained before being implemented in a computing system. Once implemented in the computing system, the parameters of the recurrent neural network remain static. Offline training is advantageous because it separates the computational overhead associated with training from the computational overhead associated with executing the computer program itself. This further reduces the runtime of the computer program.

[0018] The recurrent neural network can be configured to receive and predict deltas instead of memory addresses. A delta is the difference between two memory addresses. Deltas are easier to predict than memory addresses because physical address spaces are very large and sparse. On the other hand, the number of deltas in a given computer program is more limited. If the system predicts a delta, it can calculate a future memory address associated with that delta by simple subtraction. Predicting deltas and calculating future memory addresses from these deltas is more accurate than directly predicting future memory addresses. Consequently, the described system can more accurately prefetch data from main memory, which reduces the runtime of computer programs in the computing system.

[0019] Inputs to the recurrent neural network can also be grouped. The advantage of grouping pairs of program counter addresses and delta values ​​is that memory access patterns in groups are more predictable than memory access patterns across an entire set of program counter addresses and delta values. Therefore, grouping inputs increases the prediction accuracy of the described system, which further reduces the runtime of computer programs executing in the computing system.

[0020] Finally, in the next time step, the recurrent neural network can predict several future memory addresses from which data can be retrieved. Each prediction is made with a certain confidence. The described system can prefetch data from all future memory addresses that meet a given confidence threshold. This increases the probability that the correct data will be prefetched. This again reduces the runtime of computer programs executing in the computing system.

[0021] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Further features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a diagram of an example system for prefetching data from memory. Fig. Figure 2 is a flowchart of an example process for prefetching data from memory. Fig. Figure 3 is a flowchart of an example process for generating high-dimensional embeddings from a sequence of previous program counter addresses and their corresponding delta value. Fig. 4 shows an example computer program. Fig. Figure 5 illustrates an example prediction subsystem for predicting future memory addresses.

[0022] Similar reference symbols and designations in the various drawings indicate similar elements. DETAILED DESCRIPTION

[0023] A computer program causes a computing system to operate on data stored at various memory locations within the computing system. Generally, the computing system must load data from additional locations into local registers before it can operate on that data. The computing system can load data from, for example, a local cache or main memory. Loading data from main memory is particularly slow because main memory is generally large and distant from local registers. Therefore, computer programs that regularly cause computing systems to retrieve data from main memory will run slower than they otherwise would. By prefetching data from main memory and storing it in a faster local cache before it is needed, computing systems can reduce the runtime of computer programs.

[0024] Data prefetchers use hardware and / or software to predict future memory addresses from which data will be retrieved based on a past history of memory accesses. This specification specifically addresses techniques for prefetching data from memory using neural networks.

[0025] Fig. 1 is a diagram of an example computing system 100.

[0026] Computing system 100 executes a computer program by executing computer program instructions of the computer program using one or more processors. Specifically, computing system 100 stores computer program instructions in an instruction memory and then executes the instructions in an arrangement defined by the computer program.

[0027] Computing system 100 includes a program counter 110. A program counter, or instruction address register, is a register that stores a value that "points" to the next computer program instruction to be executed. In other words, program counter 110 stores an address at which the next computer program instruction to be executed is located. Because computer program instructions are typically stored in consecutive locations in memory, program counter 110 includes a digital counter 115 that increments the address stored in program counter 110 each time a new instruction is executed.

[0028] In some cases, computing system 100 does not execute the computer program in the same order in which it is stored in memory. For example, a particular group of machine code instructions may correspond to a conditional instruction, such as an "if-then-else" instruction or a function call. If the condition associated with the conditional instruction is met, the computing system executes one instruction sequence. If the condition associated with the conditional instruction is not met, the computing system executes a different instruction sequence. If the current instruction sequence is different from the instruction sequence following the conditional instruction, computing system 100 must write a new value to program counter register 110. The new value is the address at which the first instruction in the new instruction sequence is located.

[0029] When an instruction cycle begins, computing system 100 fetches the next computer program instruction from the address specified by program counter 110. The address stored in program counter 110 corresponds to a location in instruction memory 122, which is a block of memory in memory 120. In some implementations, memory 120 is volatile memory such as static random access memory (SRAM) or dynamic random access memory (DRAM). In another implementation, memory 120 is non-volatile memory such as read-only memory (ROM), flash memory, or magnetic memory. In yet another implementation, memory 120 is some combination of volatile and non-volatile memory.

[0030] Typically, a given computer program instruction specifies an operation, e.g., load, store, add, subtract, NOR, branch, etc., one or more source registers, and a destination register. Computing system 100 performs the specified operation on the data stored in the source register and stores the result in the destination register. For example, a computer program instruction may specify that data stored in register A should be added to data stored in register B and that the result should be stored in register C.

[0031] The source registers are typically of limited size, so data to be worked on is generally stored in memory and only loaded into the source registers when needed. However, retrieving data from memory is time-consuming and slows down the execution of computer programs. One solution to this problem is to predict data that will be worked on in the future, pre-fetch the data, and store it in a faster local memory such as a local cache 130. Retrieving data from a local cache is faster than retrieving data from main memory because the local cache is generally both physically closer to the system than main memory and smaller than main memory.Because a local cache is smaller than main memory, the computing system can only store a small subset of main memory in a local cache at a time. Consequently, the computing system 100 can predict a subset of memory addresses from which data will be fetched in the future and store that data in a local cache. If the computing system makes correct predictions, the computing system can execute computer program instructions more quickly.

[0032] A prediction subsystem 140 of computing system 100 is configured to predict data that computing system 100 will operate on in the future. After prediction subsystem 140 predicts a particular memory address from which data might be retrieved in the future, computing system 100 prefetches that data from main memory and stores it in local cache 130. Prediction subsystem 140 continues to predict memory addresses—and computing system 100 continues to prefetch data from those memory addresses—while the computer program is executing.

[0033] In some implementations, the prediction subsystem 140 is implemented in hardware directly within the computing system 100. In other implementations, the prediction subsystem 140 is implemented in one or more computers located at other locations, or in software, ie, as one or more computer programs within the computing system 100.

[0034] In some implementations, a register 142 receives and stores sequences of program counter addresses and corresponding memory addresses from program counter 110 and instruction memory 122, respectively. The memory addresses are memory addresses that were accessed when instructions referenced by their corresponding program counter addresses were executed. For example, a particular memory address may be a memory address from which data was fetched when a load instruction was executed.

[0035] In other implementations, register 142 receives and stores sequences of program counter addresses and corresponding delta values. Each delta value in the sequence defines a difference between a respective first memory address and a respective second memory address. The first memory address is a memory address accessed when an instruction pointed to by the corresponding program counter address was executed, while the second memory address is a memory address accessed before the first memory address was accessed. For example, the first and second memory addresses may be memory addresses accessed when consecutive load instructions were executed. The first and second memory addresses may also be memory addresses accessed when non-consecutive load instructions were executed.When consecutive memory addresses are used to calculate deltas, the system predicts future memory addresses that will be accessed at time N+1. When non-consecutive memory addresses are used to calculate deltas, the system does not predict the immediately following memory address that will be accessed. Instead, it predicts further into the future. For example, when a first memory access and a fifth memory access are used to calculate deltas, the system will predict future memory addresses that will be accessed at time N+4. In some implementations, the first and second memory addresses are addresses that specify locations in data memory 124.

[0036] In implementations where register 142 stores program counter addresses and corresponding memory addresses, prediction subsystem 140 calculates delta values ​​by subtracting subsequent memory addresses.

[0037] The prediction subsystem 140 includes a neural network subsystem 144. The neural network subsystem 144 may be implemented in the same computer in which the computer program is executed, or it may be implemented in a different computer.

[0038] The neural network subsystem 144 receives a sequence of program counter addresses and corresponding delta values. The neural network subsystem 144 processes the sequence of program counter addresses and corresponding delta values ​​to predict memory addresses from which data will be fetched in the future as a result of the execution of the computer program.

[0039] The neural network subsystem 144 repeatedly generates these predictions as it receives new program counter address and delta value pairs.

[0040] In response to the repeated predictions generated by the neural network subsystem 144, the computing system 100 repeatedly retrieves data from the data storage 124. The predictions specify memory addresses in the data storage 124 from which the data should be retrieved. The computing system 100 then stores the data in the local cache 130. It is faster to retrieve data from the local cache 130 than it is to retrieve data from the data storage 124. This results in faster execution of computer programs by the computing system 100.

[0041] Although this specification essentially describes prefetching data from memory, the techniques described here can be applied to other types of predictive decision making in computing systems. For example, the techniques described here can be used to prefetch instructions from instruction memory.

[0042] The systems and methods described in this application can also be used to optimize memory disambiguation. Memory disambiguation is a set of techniques employed by computing systems that execute memory access instructions out of order. The techniques detect or predict dependencies between memory operations to increase the efficiency and speed of memory accesses.

[0043] The techniques described here can also be used to optimize cache replacement strategies. Because the cache is of limited size, when a computing system places data in the cache, it must generally remove a corresponding amount of data from the cache. The systems and methods described in this application can predict the best data to remove from the cache, e.g., the data least likely to be used by the computing system in the near future. Alternatively, the systems and methods can predict the data most likely to be used in the near future and ensure that this data remains in the cache. For example, the systems and methods described here canA cache that has a swap strategy that specifies that the least recently used data in the cache should be removed in favor of new data can update an "age bit" for data that is predicted to be used in the near future. Updating the age bit ensures that the data is not replaced by new data.

[0044] Fig. 2 is a flowchart of an example process 200 for prefetching data from memory. For convenience, the process 200 is described as being performed by a system of one or more computers located at one or more locations. For example, a neural network subsystem, such as the neural network subsystem 144 of Fig. 1, which is suitably programmed, perform process 200.

[0045] First, the system receives a sequence of previous program counter addresses of a computer program and corresponding delta values ​​(210). As described above with reference to Fig. 1, program counter addresses are addresses that point to computer program instructions to be executed. These computer program instructions are typically stored in an instruction memory. Meanwhile, each delta value defines a difference between a respective first memory address and a respective second memory address. The first memory address is a memory address that was accessed when an instruction pointed to by its corresponding program counter address was executed. For example, a particular program counter address may point to a memory address storing an instruction that, when executed, retrieves data from memory. The memory address of this data is the first memory address. The second memory address is a memory address that was accessed before the first memory address was accessed.

[0046] In some implementations, the system receives the sequence of previous program counter addresses and corresponding delta values. In other implementations, the system only receives a sequence of previous program counter addresses and corresponding memory addresses. In such a case, the system calculates the delta values ​​by subtracting adjacent memory addresses in the received sequence.

[0047] The system then creates an input representation based on the sequence of program counter addresses and their corresponding delta values ​​(220).

[0048] In some implementations, the system constructs the input representation by mapping the sequence of program counter addresses and their corresponding delta values ​​to a sequence of numerical embeddings in a high-dimensional space. The embeddings are high-dimensional real-valued vectors. In some implementations, the embeddings for a program counter address and a corresponding delta value are each 128-dimensional. The system can concatenate the embeddings to form a 256-dimensional embedding. In other implementations, the embeddings for a program counter address and a corresponding delta value are each 64-dimensional, 256-dimensional, or 512-dimensional.

[0049] The embeddings represent points in a high-dimensional space. Embeddings that are "close" to each other in the space are associated with computer program instructions that retrieve data from similar memory locations. For example, two different program counter addresses may have significantly different raw values, but may correspond to a function that regularly calls another function. In such a case, the program counter addresses may have similar embeddings.

[0050] In some implementations, the system generates the sequence of embeddings using a lookup table generated by a trained neural network. In other implementations, the system generates the embeddings by directly supplying the sequence of program counter addresses and delta values ​​to the trained neural network. Further input representations are described with reference to Fig. 3 described in detail.

[0051] The system then provides the sequence of numerical embeddings as input to a recurrent neural network (230). Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input according to current values ​​of a corresponding set of parameters.

[0052] Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the network's internal states from a previous time step when calculating an output at a current time step. This allows the neural network to exhibit dynamic temporal behavior. In other words, the recurrent neural network summarizes all the information it has received up to a current time step and can capture long-term dependencies in data sequences.

[0053] In some implementations, the recurrent neural network is a long-short-term memory neural network (LSTM neural network) containing two LSTM memory layers. Each LSTM layer contains multiple units. The first LSTM layer receives a multidimensional input representing a sequence of program counter addresses and corresponding delta values. In a first time step, the first LSTM layer receives an input representing an "oldest" program counter address and its corresponding delta value. In other words, it first receives an input representing a program counter address corresponding to a program instruction that was executed first in time. In a second time step, the first LSTM layer receives an input representing a program counter address corresponding to a program instruction that was executed second in time.At a third time step, the first LSTM layer receives an input representing a program counter address corresponding to a program instruction that was executed third in time. This pattern continues for all time steps.

[0054] The number of units in each LSTM layer corresponds to the number of dimensions in the multidimensional input.

[0055] The recurrent neural network also has a second LSTM layer. The units in the second LSTM layer receive input from the units in the first LSTM layer. This is analogous to having multiple layers in a feedforward neural network and allows for greater modeling flexibility with relatively fewer parameters.

[0056] The recurrent neural network is structured and trained to detect long-term dependencies, such as patterns, in the sequence of inputs. Specifically, the recurrent neural network can detect long-term dependencies in the sequence of program counter addresses and corresponding delta values ​​to predict future memory addresses from which data will be retrieved. Each LSTM unit in the recurrent neural network 300 includes a hidden state h, a cell state c, an input gate i, a forget gate f, and an output gate o. Each of the three gates can be considered as individual neurons that compute weighted sum activations.Intuitively, the input gate controls the extent to which a new value flows into the cell, the forget gate controls the extent to which a value remains in the cell, and the output gate controls the extent to which the value in the cell is used to calculate the output activation of the LSTM unit. In other words, these gates control what the LSTM unit "remembers," or how its state is affected by previous states.

[0057] The final layer in the recurrent neural network is a soft-max layer. The soft-max layer has several neurons that receive input from the second LSTM layer. The soft-max layer processes its input to generate a probability distribution of outputs representing future delta values.

[0058] In some cases, a neural network model directly outputs a probability distribution over delta values. In other cases, a neural network model outputs a probability distribution over indices. The indices must then be converted to delta values ​​in a similar way to how delta values ​​are mapped to embeddings.

[0059] The recurrent neural network learns offline through supervised learning. In other words, the recurrent neural network can be trained before being implemented in a computing system whose prefetching will be controlled by the neural network. Once implemented in the computing system, the parameters of the recurrent neural network remain static. Offline training is advantageous because it separates the computational overhead associated with training from the computational overhead associated with executing the computer program itself. This reduces the runtime of the computer program.

[0060] Generally, training data for a neural network contains a set of inputs and labels for those inputs. The labels correspond to desired or expected outputs produced for the corresponding input. For example, the labels may contain actual delta values ​​observed when a computing system executes a computer program.

[0061] The system trains the neural network to correctly predict outputs by determining trained values ​​for the neural network's weights. Specifically, the system trains the neural network to adjust the neural network's weights to minimize the discrepancy between the actual output of the neural network and the desired or expected output for a given set of inputs. The measure of the discrepancy is known as a loss function.

[0062] In some cases, the loss function for training is the cross-entropy function, i.e., the log-loss function. Based on the probability distribution of the outputs generated by the soft-max layer, the negative log-probability that the neural network will produce the correct output can be calculated.

[0063] If the cross-entropy function indicates an error between the output generated by the recurrent neural network and the desired or expected output for a given set of inputs, the weights of the recurrent neural network can be updated, e.g., using a conventional backpropagation through time training technique. Other techniques known in the art and other loss functions can also be used to update the weights of the recurrent neural network.

[0064] As described above, the system receives an output from the recurrent neural network that defines a probability distribution of future delta values ​​(240). The system can limit the set of possible delta values ​​to the most frequently occurring delta values ​​to keep the recurrent neural network fast. For example, the system can limit the set of possible delta values ​​to the 50,000 most frequently occurring delta values.

[0065] Each probability in the distribution represents a probability that the execution of a future instruction of the computer program will cause data to be fetched from a future memory address equal to (i) a respective first memory address accessed when an instruction pointed to by a last program counter address in the sequence was executed, and (ii) the future delta value corresponding to the probability. The computer program is the same computer program associated with the program counter addresses.

[0066] In some implementations, the system may set a threshold criterion and determine whether any probabilities in the distribution satisfy that threshold criterion. For example, the threshold criterion may be a probability of 0.5, i.e., a 50% probability that a particular future memory address will be accessed when a future instruction of the computer program is executed. The system may dynamically adjust the threshold criterion. For example, the system may set a higher threshold when memory bandwidth usage is high and a lower threshold when memory bandwidth usage is low. In some implementations, the system may set a higher threshold if it more accurately predicts future memory addresses.

[0067] In response to determining that a particular probability in the probability distribution meets the threshold criterion, the system may retrieve data from the memory address associated with that probability and store that data in a local cache.

[0068] In some implementations, the system may determine that multiple probabilities meet the threshold criterion and prefetch data from the multiple memory addresses associated with those probabilities.

[0069] This increases the probability that the correct data is prefetched. This reduces the runtime of the computer program running in the computing system.

[0070] The system is typically configured to predict memory addresses accessed at time step N+1, i.e., memory addresses accessed when the next computer program instruction is executed. However, it is possible to train the recurrent neural network to predict further ahead in time steps. For example, the recurrent neural network can be trained to predict a memory address accessed in N+2, N+4, or N+8. The recurrent neural network can be trained to predict memory addresses accessed, for example, in N+2 by providing the recurrent neural network with deltas computed using second memory addresses accessed in N-2.

[0071] In a further implementation, instead of prefetching data as described above, the system may insert a fetch instruction into the computer program. Execution of the fetch instruction causes data to be fetched from a future memory address associated with a probability that meets the threshold criterion. The fetch instruction must be inserted into the computer program before the future instruction of the computer program that would otherwise cause the data to be fetched from memory. In this way, the data required by the future instruction will already be in a local cache when the future instruction is executed. Execution of the future instruction will then result in a cache hit instead of a cache miss, and the system will not need to fetch data from memory.This makes the computer program run faster because retrieving data from memory is time-consuming.

[0072] More than one retrieval instruction may be inserted into the computer program if more than one probability meets the threshold criterion.

[0073] In some implementations, the system provides additional features to the recurrent neural network. For example, the system may provide an indication of whether the last memory access was a load or a store. The system may also provide the current cache usage to the recurrent neural network. The current cache usage may influence the threshold criterion, as described above.

[0074] Fig. 3 is a flowchart of an example process for generating the high-dimensional embeddings from the sequence of previous program counter addresses and their corresponding delta value. For convenience, the process 300 is described as being performed by a system of one or more computers located at one or more locations. For example, a neural network subsystem, such as the neural network subsystem 144 of Fig. 1, which is suitably programmed, perform process 300.

[0075] First, the sequence of previous program counter addresses and their corresponding delta values ​​are supplied to a neural network (310). For convenience and clarity, this neural network is referred to as the embedding neural network. The embedding neural network may precede and merge with the recurrent neural network 300 described with reference to Fig. 3. In an implementation in which the embedding neural network is connected to the recurrent neural network 300, the two neural networks are trained end-to-end. In other words, training inputs, i.e., sequences of previous program counter addresses and corresponding delta values, are supplied to the embedding neural network. Then, the combined neural networks process the inputs to generate outputs that are compared with desired or expected outputs, as described above. If any discrepancy exists, this discrepancy is used to update the parameters of the two neural networks through stochastic gradient descent with backpropagation. The parameters of the recurrent neural network 300 are updated first, followed by the parameters of the embedding neural network, i.e.,by backpropagating gradients from the recurrent neural network into the embedding neural network.

[0076] Through training, the embedding neural network learns parameters that result in similar pairs of program counter address and delta value that possess similar embeddings. In other words, two embeddings that are geometrically close to each other in the high-dimensional embedding space should be programmatically similar. For example, two different program counter addresses may each correspond to a function that regularly calls another. In such a case, the two program counter addresses may correspond to high-dimensional embeddings that are geometrically close to each other. As another example, two embeddings may also be similar if their delta values ​​each correspond to memory addresses that store data in a particular linked list or ordering.

[0077] Then the embedding neural network processes the sequence of program counter addresses and their corresponding delta values ​​(320).

[0078] Finally, the system receives as output from the embedding neural network a sequence of numerical embeddings in a high-dimensional embedding space (330). The embeddings are high-dimensional real-valued vectors.

[0079] In some implementations, the embedding neural network can be represented by a simple lookup table once it is trained. Program counter address and delta value pairs can be mapped to indices, and the indices can be used to look up embeddings in the lookup table. Such a representation is possible because the embedding neural network is static after it has been trained. In other words, once trained, the embedding neural network will always output the same high-dimensional embedding, i.e., a specific program counter address and delta value pair, for a given input.

[0080] The sequence of previous program counter addresses and delta values ​​can be represented in other ways.

[0081] In some implementations, the sequence of previous program counter addresses and delta values ​​may be k-means grouped.

[0082] A k-means clustering is a vector quantization technique that aims to divide n observations, e.g., pairs of program counter address and delta value, into k clusters, where each observation belongs to the cluster with the nearest mean or centroid, which serves as a prototype of the cluster. This results in a partitioning of the data space. Initially, the pairs of program counter address and delta value must be represented as points or vectors in a space such that the system can calculate distances between the points. As discussed with reference to Fig. As described in Section 3, the program counter address and delta value pairs can be mapped to high-dimensional embeddings in a high-dimensional embedding space. In some implementations, the program counter address can represent an x-coordinate, while the delta value can represent a y-coordinate.

[0083] First, the system must choose the number of clusters k. K is an important parameter that can affect the quality of the input representation. The system can choose an appropriate value for K through trial and error. Then, the system must choose arbitrary means or centroids for each of the k clusters. The system then calculates the distance from each observation—for example, each point representing a pair of program counter address and delta value—to each centroid. Each point is then assigned to the cluster with the closest centroid. Finally, new centroids are calculated based on the new set of points in each cluster. This process is repeated until no points are assigned to new clusters.

[0084] Each program counter address and delta value pair can then be represented as a distance to the centroid of its cluster. These distances can be normalized and used as input to the recurrent neural network 300. For a more detailed discussion of k-means clustering, see, for example, MacQueen, J. "Some Procedures for Classifying and Analyzing Multivariate Observations." Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, 281-297, University of California Press, Berkeley, California, 1967.

[0085] The advantage of grouping program counter address and delta value pairs is that data fetch patterns in groups are much more predictable than data fetch patterns across the entire set of program counter addresses and delta values.

[0086] In other implementations, the sequence of program counter addresses and the corresponding delta values ​​are represented as binary strings. The recurrent neural network 300 can learn patterns of bit reversals. Different bits can be weighted more or less heavily to account for the fact that mispredicting bit 31 of a future memory address would result in a large error, while mispredicting bit 0 of a future memory address would result in a small error.

[0087] Fig. Figure 4 illustrates an example computer program 400. The computer program 400 is written in a fictitious assembly language. Assembly languages ​​are low-level programming languages ​​for computers or other programmable devices. Unlike high-level programming languages, e.g., C++, Java, Python, Ruby, etc., assembly languages ​​are specific to specific computer architectures. High-level language instructions are converted into assembly language instructions by compilers. Assembly language instructions closely correspond to machine language instructions.

[0088] Assembly language instructions generally specify an instruction code, meaning the operation to be performed; source registers, meaning the registers that store the data to be operated on; and a destination register, meaning the register in which the result of the operation is to be stored. Assembly language operations include load, store, add, subtract, NOR, compare, branch, jump, etc.

[0089] Load operations move data from memory to a register. Store operations move data from a register to memory. Add, subtract, NOR, and compare operations are arithmetic and logic operations performed on the data stored in the specified source register. Compare, branch, and jump operations correspond to conditional statements and function calls in high-level programming languages.

[0090] Computer program 400 is executed in a processor with two registers R1 and R2 and four memory addresses mem1, mem2, mem3, and mem4. Computer program 400 calculates the sum of the values ​​stored in the four memory addresses.

[0091] First, in instructions 401 and 402, the values ​​stored in mem1 and mem2 are loaded into R1 and R2, respectively. In instruction 403, R1 is added to R2 and stored in R1. After instruction 403, R1 stores the sum of (1) the value stored in mem1 and (2) the value stored in mem2.

[0092] In instruction 404, the value stored in mem3 is loaded into R2. In instruction 405, R1 is added back to R2 and stored in R1. After instruction 405, R1 stores the sum of (1) the value stored in mem1, (2) the value stored in mem2, and (3) the value stored in mem3.

[0093] In instruction 406, the value stored in mem4 is loaded into R2. In instruction 407, R1 is added back to R2 and stored in R1. After instruction 407, R1 stores the sum of (1) the value stored in mem1, (2) the value stored in mem2, (3) the value stored in mem3, and (4) the value stored in mem4. Finally, in instruction 408, R1 is stored in mem1, overwriting the previous value stored in mem1.

[0094] The computer program 400 has four load instructions: instruction 401, instruction 402, instruction 404, and instruction 406. Each of these four instructions has a program counter address and a corresponding delta value that can be provided to a system for predicting future memory addresses.

[0095] The program counter address for instruction 401 is PC. The corresponding delta value is simply mem1, because no memory address access occurs before instruction 401. The program counter address for instruction 402 is PC + 1. The corresponding delta value is mem1 - mem2. The program counter address for instruction 404 is PC + 3. The corresponding delta value is mem3 - mem2. Finally, the program counter address for instruction 406 is PC + 5. The corresponding delta value is mem4 - mem3. The four pairs of program counter address and delta value can be used as input to a prediction system, e.g., the prediction subsystem 140 of Fig. 1, will be delivered.

[0096] Fig. 5 illustrates an example prediction subsystem 500 for predicting future memory addresses from which data is expected to be fetched. The prediction subsystem 500 is implemented directly in hardware in the computing system in which the prefetching occurs.

[0097] Register 510 receives and stores sequences of previous program counter addresses and corresponding delta values.

[0098] A neural network subsystem 520 of the prediction subsystem 500 has a parameter memory 522, an arithmetic / logic unit (ALU) 524, and a controller 526. The parameter memory 522 stores the parameters of a long short-term memory network (LSTM), which is a type of recurrent neural network. For example, the parameter memory 522 stores the input, forget, output, and cell weights of each LSTM layer of the LSTM network. The parameter memory 522 also stores the biases of each layer and the cell states and hidden states of each LSTM cell in each LSTM layer. In some implementations, the parameter memory 522 is volatile memory such as static random access memory (SRAM) or dynamic random access memory (DRAM). In further implementations, the parameter memory 522 is a non-volatile memory such asa read-only memory (ROM), a flash memory, or a magnetic memory. In further implementations, the parameter memory 522 is a specific combination of volatile and non-volatile memory. For example, the weights of each LSTM layer, which do not change after the LSTM network is trained, can be stored in non-volatile memory, while the cell state and hidden state of each LSTM unit, which change frequently, can be stored in volatile memory.

[0099] The ALU 524 calculates the input, forget, and output gates of each LSTM unit. It also calculates the cell state and hidden state of each LSTM unit. An ALU is a combinational circuit that performs arithmetic and bitwise operations on binary numbers. The inputs to an ALU are the data to be operated on, called operands, and an instruction code that specifies the operation to be performed. The output of the ALU is the result of the performed operation. The controller 526 controls the inputs that the ALU 524 receives from the parameter memory 522. It also controls the operation that the ALU 524 performs.

[0100] The ALU 524 receives as input the parameters stored in parameter memory 522 and the sequence of program counter addresses and corresponding delta values ​​stored in register 510. The ALU 524 calculates the outputs of each LSTM unit. For example, the ALU 524 calculates the input, forget, output, and hidden gates of each LSTM cell.

[0101] The controller 526 may cause the ALU 524 to write calculated cell states and hidden states to the parameter memory 522 such that these cell and hidden states may be used to calculate outputs at a next time step.

[0102] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuitry, in tangible computer software or computer firmware, in computer hardware incorporating the structures disclosed in this specification and their structural equivalents, or in combinations of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded in a tangible non-transitory storage medium for execution by or for controlling the operation of a data processing device.The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more thereof. Alternatively or additionally, the program instructions may be encoded in a synthetically generated propagating signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiving device for execution by a data processing device.

[0103] The term "data processing device" refers to data processing hardware and includes all types of devices, facilities, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may also be or further include off-the-shelf or purpose-built parallel processing subsystems, e.g., a GPU or another type of special-purpose processing subsystem. The device may also be or further include special-purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, e.g.,Code that forms processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0104] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be provided in any form containing, as a stand-alone program or as a module, a component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be encoded in a section of a file containing further programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such asFiles that store one or more modules, subroutines, or sections of code. A computer program may be provided to run on one computer or on multiple computers located at one location or distributed across multiple locations and connected by a data communications network.

[0105] As used in this specification, a "machine" or a "software machine" refers to a software-implemented input / output system that provides an output that is different from its input. A machine can be a coded functional block such as a library, a platform, a software development kit ("SDK"), or an object. Each machine can be implemented in any suitable type of computing device, such as servers, mobile phones, tablet computers, notebook computers, music players, e-book readers, laptop or desktop computers, PDAs, smartphones, or other stationary or portable devices that include one or more processors and computer-readable media. Additionally, two or more machines can be implemented in the same computing device or in different computing devices.

[0106] The processes and logic sequences described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic sequences may also be performed by special-purpose logic circuitry, such as an FPGA or an ASIC, or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0107] Computers suitable for executing a computer program may use general-purpose microprocessors and / or special-purpose microprocessors or any other type of central processing unit as a basis. Generally, a central processing unit will receive instructions and data from read-only memory and / or random access memory. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Generally, a computer will also include or be operatively coupled to one or more mass storage devices for receiving data from one or more mass storage devices for storing data, e.g.A computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0108] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM disks and DVD-ROM disks.

[0109] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented in a computer having a display device, e.g., a CRT (cathode ray tube) monitor or an LCD (liquid crystal display) monitor for displaying information to the user, and a keyboard and pointing device, e.g., a mouse, a trackball, or a presence-sensitive display or other surface with which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; e.g., feedback provided to the user may be any form of sensory feedback, e.g.,optical feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form that includes auditory, verbal, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; e.g., by sending web pages to an internet browser on a user's device in response to requests received from the internet browser. Furthermore, a computer can interact with a user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving reply messages from the user.

Claims

[1] Computer-implemented method comprising: Receiving (210) a sequence of previous program counter addresses (110) of a computer program and corresponding delta values, each delta value defining a difference between a respective first memory address (122) and a respective second memory address (124), the first memory address being a memory address from which data was retrieved when an instruction pointed to by the corresponding program counter address (110) was executed, and the second memory address (124) being a memory address from which data was retrieved before data was retrieved from the first memory address (122); generating (220) an input representation based on the sequence of program counter addresses (110) and their corresponding delta values; Providing (230) the input representation as input to a recurrent neural network (300); Receiving (240) from the recurrent neural network (300) an output defining a probability distribution of future delta values, each probability in the distribution representing a probability that execution of a future instruction of the computer program will cause data to be retrieved from a future memory address equal to (i) a respective first memory address (122) from which data was retrieved when an instruction pointed to by a last program counter address (110) in the sequence executed, and (ii) the future delta value corresponding to the probability; Determining that one or more probabilities in the distribution satisfy a threshold criterion; and automatically inserting one or more retrieval instructions into the computer program, wherein execution of the one or more fetch instructions causes data to be fetched from the one or more future memory addresses associated with the one or more probabilities satisfying the criterion, and wherein the one or more fetch instructions are inserted into the computer program before future instructions of the computer program. [2] The method of claim 1, further comprising: in response to determining that the one or more probabilities in the distribution satisfy the threshold criterion, retrieving data from the one or more future memory addresses associated with the one or more probabilities satisfying the criterion; and Storing the data in a local cache (130). [3] The method of claim 1, further comprising: Comparing memory addresses from which data is retrieved as a result of the execution of future instructions of the computer program with the probability distribution of future delta values; and Updating the parameters of the recurrent neural network (300) based on the comparison. [4] The method of claim 1, wherein the operations are performed in a microprocessor, wherein the recurrent neural network (300) is implemented in the microprocessor, and wherein the method further comprises: Retrieving data from one or more future memory addresses associated with one or more probabilities in the distribution that satisfy a threshold criterion; and Storing the data in a local buffer (130) in the microprocessor. [5] The method of claim 1, further comprising: Determining that a memory address associated with a probability satisfying a threshold criterion is to be accessed in an arrangement different from an arrangement in which a corresponding future instruction of the computer program appears; and Changing the order of the corresponding future instruction of the computer program. [6] The method of claim 1, further comprising: Determining that data from a memory address associated with a probability that satisfies the threshold criterion is present in a local buffer (130), and Updating an age bit for the data in a local cache (130), wherein the age bit indicates how long ago the data was used, and wherein the local cache (130) is a least recently used local cache (130). [7] A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations that include: Receiving (210) a sequence of previous program counter addresses (110) of a computer program and corresponding delta values, each delta value defining a difference between a respective first memory address (122) and a respective second memory address (124), the first memory address being a memory address from which data was retrieved when an instruction pointed to by the corresponding program counter address (110) was executed, and the second memory address being a memory address (124) from which data was retrieved before data was retrieved from the first memory address (122); generating (220) an input representation based on the sequence of program counter addresses (110) and their corresponding delta values; Providing (230) the input representation as input to a recurrent neural network (300); Receiving (240) from the recurrent neural network (300) an output defining a probability distribution of future delta values, each probability in the distribution representing a probability that execution of a future instruction of the computer program will cause data to be retrieved from a future memory address equal to (i) a respective first memory address (122) from which data was retrieved when an instruction pointed to by a last program counter address (110) in the sequence executed, and (ii) the future delta value corresponding to the probability; Determining that one or more probabilities in the distribution satisfy a threshold criterion; and automatically inserting one or more retrieval instructions into the computer program, wherein execution of the one or more fetch instructions causes data to be fetched from the one or more future memory addresses associated with the one or more probabilities satisfying the criterion, and wherein the one or more fetch instructions are inserted into the computer program before future instructions of the computer program. [8] The system of claim 7, wherein the operations further comprise: in response to determining that the one or more probabilities in the distribution satisfy the threshold criterion, retrieving data from the one or more future memory addresses associated with the one or more probabilities satisfying the criterion; and Storing the data in a local cache (130). [9] The system of claim 7, wherein the operations further comprise: Comparing memory addresses from which data is retrieved as a result of the execution of future instructions of the computer program with the probability distribution of future delta values; and Updating the parameters of the recurrent neural network (300) based on the comparison. [10] The system of claim 7, wherein the operations are performed in a microprocessor, wherein the recurrent neural network (300) is implemented in the microprocessor, and wherein the method further comprises: Retrieving data from one or more future memory addresses associated with one or more probabilities in the distribution that satisfy a threshold criterion; and Storing the data in a local buffer (130) in the microprocessor. [11] One or more non-transitory computer storage media encoded with instructions that, when executed by the one or more computers, cause the one or more computers to perform operations that include: Receiving (210) a sequence of previous program counter addresses (110) of a computer program and corresponding delta values, each delta value defining a difference between a respective first memory address (122) and a respective second memory address (124), the first memory address (122) being a memory address from which data was retrieved when an instruction pointed to by the corresponding program counter address (110) was executed, and the second memory address (124) being a memory address from which data was retrieved before data was retrieved from the first memory address (122); generating (220) an input representation based on the sequence of program counter addresses (110) and their corresponding delta values; Providing (230) the input representation as input to a recurrent neural network (300); and Receiving (240) from the recurrent neural network (300) an output defining a probability distribution of future delta values, each probability in the distribution representing a probability that execution of a future instruction of the computer program will cause data to be retrieved from a future memory address equal to (i) a respective first memory address (122) from which data was retrieved when an instruction pointed to by a last program counter address (110) in the sequence executed, and (ii) the future delta value corresponding to the probability; Determining that one or more probabilities in the distribution satisfy a threshold criterion; and automatically inserting one or more fetch instructions into the computer program, wherein execution of the one or more fetch instructions causes data to be retrieved from the one or more future memory addresses associated with the one or more probabilities satisfying the criterion, and wherein the one or more fetch instructions are inserted into the computer program before the future instructions of the computer program. [12] One or more non-transitory computer storage media according to claim 11, wherein the operations further comprise: in response to determining that the one or more probabilities in the distribution satisfy the threshold criterion, retrieving data from the one or more future memory addresses associated with the one or more probabilities satisfying the criterion; and Storing the data in a local cache (130). [13] One or more non-transitory computer storage media according to claim 11, wherein the operations further comprise: Comparing memory addresses from which data is retrieved as a result of the execution of future instructions of the computer program with the probability distribution of future delta values; and Updating the parameters of the recurrent neural network (300) based on the comparison. [14] One or more non-transitory computer storage media according to claim 11, wherein the operations are performed in a microprocessor, wherein the recurrent neural network (300) is implemented in the microprocessor, and wherein the operations further comprise: Retrieving data from one or more future memory addresses associated with one or more probabilities in the distribution that satisfy a threshold criterion; and Storing the data in a local buffer (130) in the microprocessor.

Images