Computer system prediction machine learning model

Through the recurrent neural network to identify the long-term dependence of program counter address and incremental value sequence, predict future memory addresses and prefetch data in local cache, solve the problem of low prefetch efficiency in complex memory access mode in the prior art. Improves the running speed and energy efficiency of computer programs.

CN112136142BActive Publication Date: 2024-11-29GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980026107.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-05-30
Publication Date
2024-11-29
Estimated Expiration
2039-05-30

AI Technical Summary

Technical Problem

The prior art is difficult to accurately identify complex and irregular memory access patterns when predicting and prefetching memory addresses, resulting in higher running time and energy consumption of computer programs.

Method used

Recurrent neural networks are used to identify long-term dependencies from program counter addresses and incremental value sequences, predict future memory addresses through offline training, and prefetch data in the local cache to reduce the number of cache misses.

Benefits of technology

Improve prediction accuracy and efficiency, reduce the running time and energy consumption of computer programs, and reduce the frequency of data extraction from main memory.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112136142B_ABST
    Figure CN112136142B_ABST
Patent Text Reader

Abstract

Methods, systems and apparatus for prefetching data from a memory using a neural network, including a computer program encoded on a computer storage medium. An example system receives a sequence of prior program counter addresses and corresponding increment values ​​of a computer program. The system creates an input representation based on the sequence. The system provides the input representation as an input to a recurrent neural network. The system receives an output from the recurrent neural network, the output defining a probability distribution of future increment values. Each probability in the distribution represents the likelihood that execution of a future instruction of the computer program will result in fetching data from a particular future memory address.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] This specification relates to techniques for prefetching data from memory using a neural network.

[0002] A data prefetcher utilizes hardware and / or software to predict a future memory address and fetch data from that memory address based on past memory access history.

[0003] A neural network is a machine learning model that uses multiple layers of operations to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers between the input layer and the output layer. The output of each layer is used as the input to another layer, which can be, for example, the next hidden layer or the output layer.

[0004] Each neural network layer specifies one or more transformation operations to be performed on the inputs to that layer. Some neural network layers have operations called neurons. Each neuron receives one or more inputs and produces outputs that are received by another neural network layer. Typically, each neuron receives inputs from other neurons, and each neuron provides outputs to one or more other neurons.

[0005] The architecture of a neural network specifies which layers are included in the network, their properties, and how the neurons in each layer of the network are connected. In other words, the architecture specifies which layers provide their output as input to which other layers and how they provide output.

[0006] Each layer generates one or more outputs using the current values ​​of a set of parameters for that layer. Therefore, training the network consists of continuously passing the input forward, computing gradient values, and updating the current values ​​of a set of parameters for each layer. Once the neural network is trained, the final set of parameters can be used to make predictions in a production system. Summary of the invention

[0007] This specification relates to techniques for prefetching data from memory using a neural network.

[0008] A system for prefetching data from a memory receives a sequence of previous program counter addresses and corresponding increment values ​​of a computer program. Each increment value defines a difference between a corresponding first memory address and a corresponding second memory address. The first memory address is a memory address accessed when the instruction pointed to by the corresponding program counter address is executed. The second memory address is a memory address accessed before the first memory address is accessed. For example, the second memory address may be a memory address that was just accessed before, or may be a memory address that was accessed by several or more previous memory operations.

[0009] The system then converts the sequence of program counter addresses into an input representation. The system provides the input representation as input to the recurrent neural network. In some embodiments, the input representation is a sequence of digital embeddings in a high-dimensional embedding space. In other embodiments, the input representation is a set of program counter address and increment value pairs that are operated through a "k-means clustering" operation.

[0010] The system receives an output from a recurrent neural network that defines a probability distribution of future increment values. Each probability in the distribution represents a likelihood that execution of a future instruction of the computer program will result in fetching data from a future memory address that is equal to: (i) the corresponding first memory address accessed when executing the instruction pointed to by the most recent program counter address in the sequence, plus (ii) the future increment value corresponding to the probability.

[0011] The subject matter described in this specification may be implemented in specific embodiments to achieve one or more of the following advantages. The described system may more accurately predict future memory addresses from which data will be extracted by using a recurrent neural network to identify sequential long-term dependencies on program counter addresses and corresponding increment values. The system may pre-fetch data from its predicted future memory addresses and store the data in a local cache. Accurate pre-fetching reduces the number of cache misses in a computing system. Because extracting data from a local cache is faster than extracting data from main memory, the running time of a computer program executed on a computing system is reduced. Even when conventional pre-fetching methods cannot achieve significant improvements, the system may still be able to identify complex and irregular memory access patterns and thereby reduce the running time of a computer program by using a recurrent neural network as described in this specification.

[0012] Reducing the number of cache misses also reduces the energy consumption of the computing system because fetching data from main memory is energy intensive compared to fetching data from a local cache.

[0013] Recurrent neural networks can be trained offline. In other words, the recurrent neural network can be trained before implementation in a computing system. After implementation in the computing system, the parameters of the recurrent neural network remain static. Offline training is advantageous because it separates the computational overhead associated with training from the computational overhead associated with running the computer program itself. This further reduces the running time of the computer program.

[0014] The recurrent neural network can be configured to receive and predict increments instead of memory addresses. An increment is the difference between two memory addresses. Increments are easier to predict than memory addresses because the physical address space is very large and sparse. On the other hand, the number of increments in a given computer program is more limited. Once the system predicts an increment, the future memory address associated with that increment can be calculated by simple subtraction. Predicting increments and calculating future memory addresses based on these increments is more accurate than directly predicting future memory addresses. Therefore, the described system can pre-fetch data from main memory more accurately, thereby reducing the running time of the computer program on the computing system.

[0015] The inputs to the recurrent neural network may also be clustered. An advantage of clustering pairs of program counter addresses and increment values ​​is that memory access patterns within a group are more predictable than memory access patterns within the entire set of program counter addresses and increment values. Thus, input clustering increases the prediction accuracy of the described system, which further reduces the runtime of a computer program executed on the computing system.

[0016] Finally, the recurrent neural network can predict multiple future memory addresses from which data is to be extracted in the next time step. Each prediction is made with a specific confidence level. The described system can pre-fetch data from all future memory addresses that meet a predetermined threshold confidence level. In this way, the probability of pre-fetching the correct data is increased. And, this reduces the running time of a computer program executed on the computing system.

[0017] The details of one or more embodiments of the subject matter of the present specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a diagram of an example system for prefetching data from a memory.

[0019] Figure 2 is a flow chart of an example process for prefetching data from memory.

[0020] Figure 3 is a flow chart of an example process for generating a high-dimensional embedding from a sequence of prior program counter addresses and their corresponding increment values.

[0021] Figure 4 An example computer program is depicted.

[0022] Figure 5 An example prediction subsystem for predicting future memory addresses is depicted.

[0023] The same reference numbers and names in different drawings represent the same elements. DETAILED DESCRIPTION

[0024] A computer program can cause a computing system to operate on data stored in various memory locations of the computing system. Typically, a computing system must load data into local registers before operating on data in other locations. A computing system can load data, for example, from a local cache or from main memory. Loading data from main memory is particularly slow because, typically, main memory is large and physically far away from the local registers. Therefore, a computer program that causes a computing system to fetch data from main memory typically executes more slowly than it would otherwise. A computing system can reduce the running time of a computer program by pre-fetching data from main memory and storing the data in a faster local cache until it is needed.

[0025] The data prefetcher utilizes hardware and / or software to predict future memory addresses and thereby fetch data from the memory addresses based on past memory access history. The present specification relates, in particular, to techniques for prefetching data from memory using a neural network.

[0026] Figure 1 is a diagram of an example computing system 100 .

[0027] The computing system 100 runs the computer program by executing the computer program instructions of the computer program using one or more processors. In particular, the computing system 100 stores the computer program instructions in an instruction memory and then executes the instructions in the order defined by the computer program.

[0028] The computing system 100 has a program counter 110. The program counter, or instruction address register, is a register that stores a value that "points to" the next computer program instruction to be executed. In other words, the program counter 110 stores the address at which the next computer program instruction to be executed is located. Because computer program instructions are typically stored in sequential locations in memory, the program counter 110 includes a digital counter 115. The digital counter 115 increments the address stored in the program counter 110 each time a new instruction is executed.

[0029] In some cases, computing system 100 does not execute computer programs in the same order in which they are stored in memory. For example, a particular set of machine code instructions may correspond to a conditional statement, such as an "if-then-else" statement or a function call. If the condition associated with the conditional statement is met, the computing system executes one instruction sequence. If the condition associated with the conditional statement is not met, the computing system executes a different instruction sequence. If the current instruction sequence is different from the instruction sequence following the conditional statement, computing system 100 must write a new value to a register of program counter 110. The new value is the address at which the first instruction in the new instruction sequence is located.

[0030] When an instruction cycle begins, computing system 100 fetches the next computer program instruction from the address specified by program counter 110. The address stored in program counter 110 corresponds to a location in instruction memory 122, which is a memory block in memory 120. In some embodiments, memory 120 is a volatile memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In another embodiment, memory 120 is a non-volatile memory, such as read-only memory (ROM), flash memory, or magnetic storage. In yet another embodiment, memory 120 is some combination of volatile and non-volatile memory.

[0031] Typically, a given computer program instruction specifies an operation, such as load, store, add, subtract, NOR, branch, etc., one or more source registers and a destination register. The computing system 100 performs the specified operation on the data stored in the source registers and stores the result in the destination register. For example, a computer program instruction may specify that the data stored in register A should be added to the data stored in register B, and the result should be stored in register C.

[0032] The source registers are usually of a certain size, so the data to be operated on usually must be stored in the memory and loaded into the source registers only when needed. However, extracting data from the memory is very time-consuming and reduces the execution speed of the computer program. One solution to this problem is to predict the data that will need to be operated in the future, pre-fetch the data, and store it in a faster local memory such as the local cache 130. Extracting data from the local cache is faster than extracting data from the main memory, because the local cache is usually physically closer to the system than the main memory and is also smaller than the main memory. Since the local cache is smaller than the main memory, the computing system can only store a small part of the main memory in the local cache at any given time. Therefore, the computing system 100 can predict a subset of the memory addresses from which data will need to be extracted in the future, and store the data in the local cache. If the computing system makes accurate predictions, the computing system can execute computer program instructions faster.

[0033] The prediction subsystem 140 of the computing system 100 is configured to predict data that the computing system 100 will operate on in the future. After the prediction subsystem 140 predicts a particular memory address from which data may be fetched in the future, the computing system 100 pre-fetches the data from the main memory and stores it in the local cache 130. As the computer program is executed, the prediction subsystem 140 continues to predict memory addresses, and the computing system 100 continues to pre-fetch data from those memory addresses.

[0034] In some embodiments, prediction subsystem 140 is implemented in hardware directly on computing system 100. In other embodiments, prediction subsystem 140 is implemented on one or more computers located elsewhere, or in software, i.e., as one or more computer programs on computing system 100.

[0035] In some embodiments, register 142 receives and stores a sequence of program counter addresses and corresponding memory addresses from program counter 110 and instruction memory 122, respectively. A memory address is a memory address that is accessed when the instruction pointed to by its corresponding program counter address is executed. For example, a particular memory address may be a memory address from which data is extracted when a load instruction is executed.

[0036] In other embodiments, register 142 receives and stores a sequence of program counter addresses and corresponding delta values. Each delta value in the sequence defines the difference between the corresponding first memory address and the corresponding second memory address. The first memory address is the memory address accessed when the instruction pointed to by the corresponding program counter address is executed, and the second memory address is the memory address accessed before the first memory address is accessed. For example, the first and second memory addresses may be memory addresses accessed when executing continuous load instructions. The first and second memory addresses may also be memory addresses accessed when executing non-continuous load instructions. If continuous memory addresses are used to calculate the increment, the system will predict the future memory address accessed at time N+1. If non-continuous memory addresses are used to calculate the increment, the system will not predict the next memory address that needs to be accessed. Instead, it will predict further in the future. For example, if the first memory access and the fifth memory access are used to calculate the increment, the system will predict the future memory address accessed at time N+4. In some embodiments, the first and second memory addresses are addresses of locations in the specified data memory 124.

[0037] In embodiments where register 142 stores a program counter address and a corresponding memory address, prediction subsystem 140 calculates the increment value by subtracting consecutive memory addresses.

[0038] The prediction subsystem 140 includes a neural network subsystem 144. The neural network subsystem 144 may be implemented on the same computer that executes the computer program, or may be implemented on a different computer.

[0039] The neural network subsystem 144 receives a sequence of program counter addresses and corresponding increment values. The neural network subsystem 144 processes the sequence of program counter addresses and corresponding increment values ​​to predict future memory addresses from which data will be retrieved as a result of the execution of the computer program. The neural network subsystem 144 repeatedly generates these predictions as the neural network subsystem 144 receives new program counter address and increment value pairs.

[0040] In response to the repeated predictions generated by the neural network subsystem 144, the computing system 100 repeatedly pre-fetches data from the data memory 124. The predictions specify a memory address in the data memory 124 from which to fetch the data. The computing system 100 then stores the data in the local cache 130. It is faster to fetch data from the local cache 130 than from the data memory 124. This causes the computing system 100 to execute the computer program faster.

[0041] Although the present specification mainly describes prefetching data from memory, the techniques described herein can be applied to other types of prediction decisions in computing systems. For example, the techniques described herein can be used to prefetch instructions from instruction memory.

[0042] The systems and methods described in this application can also be used to optimize memory disambiguation. Memory disambiguation is a set of techniques used by computing systems to execute memory access instructions out of order. These techniques detect or predict dependencies between memory operations to improve memory access efficiency and speed.

[0043] The techniques described herein can also be used to optimize cache replacement policies. Because caches are finite in size, when a computing system places data into a cache, it typically must remove a corresponding amount of data from the cache. The systems and methods described in this application can predict the best data to remove from a cache, for example, data that a computing system is least likely to use in the near future. Alternatively, the systems and methods can predict the data that is most likely to be used in the near future and ensure that the data remains in the cache. For example, in a "least recently used" cache, i.e., a cache with a replacement policy that specifies that the least recently used data in the cache should be removed in favor of new data, the systems and methods described herein can update an "age bit" for data that is predicted to be used in the near future. Updating the age bit ensures that the data is not replaced by new data.

[0044] Figure 2 is a flow chart of an example process 200 for pre-fetching data from a memory. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed neural network subsystem, e.g., Figure 1 The neural network subsystem 144 can perform processing 200.

[0045] First, the system obtains a sequence of previous program counter addresses and corresponding increment values ​​of the computer program (210). Figure 1 As described, the program counter address is the address pointing to the computer program instruction to be executed. Those computer program instructions are usually stored in the instruction memory. At the same time, each increment value defines the difference between the corresponding first memory address and the corresponding second memory address. The first memory address is the memory address accessed when the instruction pointed to by the corresponding program counter address is executed. For example, a specific program counter address may point to the memory address of the stored instruction, which can extract data from the memory when executed. The memory address of the data is the first memory address. The second memory address is the memory address accessed before accessing the first memory address.

[0046] In some embodiments, the system receives a sequence of previous program counter addresses and corresponding increment values. In other embodiments, the system only receives a sequence of previous program counter addresses and corresponding memory addresses. In this case, the system calculates the increment value by subtracting the adjacent memory addresses in the received sequence.

[0047] Next, the system creates an input representation based on the sequence of program counter addresses and their corresponding increment values ​​(220).

[0048] In some embodiments, the system creates an input representation by mapping a sequence of program counter addresses and their corresponding incremental values ​​to a digital embedding sequence in a high-dimensional space. The embedding is a high-dimensional real-valued vector. In some embodiments, the embedding of the program counter address and the corresponding incremental value is 128-dimensional. The system can connect the embeddings in series to form a 256-dimensional embedding. In other embodiments, the embedding of the program counter address and the corresponding incremental value is 64-dimensional, 256-dimensional, or 512-dimensional.

[0049] Embeddings represent points in a high-dimensional space. Embeddings that are "close" to each other in the space are associated with computer program instructions that extract data from similar memory regions. For example, two different program counter addresses may have very different raw values ​​but may correspond to one function that regularly calls another function. In this case, the program counter addresses may have similar embeddings.

[0050] In some embodiments, the system generates the embedding sequence using a lookup table generated by the trained neural network. In other embodiments, the system generates the embedding by providing the sequence of program counter addresses and increment values ​​directly to the trained neural network. Figure 3 Other input representations are described in more detail.

[0051] Next, the system provides the digital embedding sequence as input to a recurrent neural network (230). A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as input to the next layer, which is either the next hidden layer or the output layer. Each layer of the network generates an output from the received input based on the current values ​​of a corresponding set of parameters.

[0052] Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can utilize some or all of the network's internal state from the previous time step when computing the output at the current time step. This allows the neural network to display dynamic temporal behavior. In other words, a recurrent neural network summarizes all the information it has received up to the current time step and is able to capture long-term dependencies on the data sequence.

[0053] In some embodiments, the recurrent neural network is a long short-term memory (LSTM) neural network including two long short-term memory (LSTM) memory layers. Each LSTM layer includes a plurality of units. The first LSTM layer receives a multidimensional input representing a sequence of program counter addresses and corresponding increment values. At a first time step, the first LSTM layer receives an input representing the "oldest" program counter address and its corresponding increment value. In other words, first receive an input representing the program counter address corresponding to the program instruction first executed in time. At a second time step, the first LSTM layer receives an input representing the program counter address corresponding to the program instruction executed for the second time in time. At a third time step, the first LSTM layer receives an input representing the program counter address corresponding to the program instruction executed for the third time in time. This pattern continues to all time steps.

[0054] The number of units in each LSTM layer corresponds to the number of dimensions in the multi-dimensional input.

[0055] The recurrent neural network also has a second LSTM layer. The units in the second LSTM layer receive input from the units in the first LSTM layer. This is similar to having multiple layers in a feedforward neural network and allows for greater modeling flexibility with relatively few parameters.

[0056] The recurrent neural network is structured and trained so that it can recognize long-term dependencies, e.g., patterns, on a sequence of inputs. Specifically, the recurrent neural network can recognize long-term dependencies on a sequence of program counter addresses and corresponding increment values, thereby predicting future memory addresses from which data is to be extracted.

[0057] Each LSTM cell in the recurrent neural network 300 includes a hidden state h, a cell state c, an input gate i, a forget gate f, and an output gate o. Each of these three gates can be thought of as a separate neuron activated for computing a weighted sum. Intuitively, the input gate controls the extent to which new values ​​flow into the cell, the forget gate controls the extent to which values ​​are retained in the cell, and the output gate controls the extent to which the values ​​in the cell are used to compute the output activation of the LSTM cell. In other words, these gates control what the LSTM cell "remembers" or how its state is affected by the previous state.

[0058] The last layer in the recurrent neural network is the soft-max layer. The soft-max layer has multiple neurons that receive input from the second LSTM layer. The soft-max layer processes its input to generate a probability distribution of outputs representing future incremental values.

[0059] In some cases, the neural network model directly outputs a probability distribution over delta values. In other cases, the neural network model outputs a probability distribution over indices. The indices must then be converted to delta values ​​in much the same way that delta values ​​are mapped to embeddings.

[0060] Recurrent neural networks learn offline through supervised learning. In other words, the recurrent neural network can be trained before implementation in the computing system that the neural network will control the prefetching of the computing system. After implementation in the computing system, the parameters of the recurrent neural network remain static. Offline training is advantageous because it separates the computational overhead associated with training from the computational overhead associated with running the computer program itself. This reduces the running time of the computer program.

[0061] Typically, training data for a neural network includes a set of inputs and labels for those inputs. The labels correspond to the desired or expected outputs generated for the corresponding inputs. For example, the labels may include the actual incremental values ​​observed when a computing system executes a computer program.

[0062] The system trains the neural network by determining training values ​​for the neural network weights to accurately predict outputs. More specifically, the system trains the neural network to set the neural network weights so as to minimize the difference between the actual neural network output and the desired or expected output for a particular set of inputs. The measure of the difference is called a loss function.

[0063] In some cases, the loss function for training is the cross entropy function, also known as the logarithmic loss function. Given the probability distribution of the outputs generated by the soft-max layer, one can calculate the negative log probability that the neural network generated the correct output.

[0064] For a given set of inputs, if the cross entropy function represents the error between the output generated by the recurrent neural network and the desired or expected output, the weights of the recurrent neural network can be updated, for example, using traditional back propagation through time training techniques. Other techniques and other loss functions known in the art can also be used to update the weights of the recurrent neural network.

[0065] As described above, the system receives an output from the recurrent neural network that defines a probability distribution of future increment values ​​(240). The system can limit the set of possible increment values ​​to the most common increment values ​​in order to maintain a faster speed of the recurrent neural network. For example, the system can limit the set of possible increment values ​​to the 50,000 most common increment values.

[0066] Each probability in the distribution represents a likelihood that execution of a future instruction of the computer program will result in fetching data from a future memory address that is equal to (i) the corresponding first memory address accessed when the instruction pointed to by the most recent program counter address in the sequence is executed, plus (ii) a future increment value corresponding to the probability. The computer program is the same computer program associated with the multiple program counter addresses.

[0067] In some embodiments, the system can set a threshold criterion and determine whether any probability in the distribution meets the threshold criterion. For example, the threshold criterion can be a probability of 0.5, that is, a 50% probability of accessing a specific future memory address when future instructions of the computer program are executed. The system can dynamically adjust the threshold criterion. For example, the system can set a higher threshold when memory bandwidth utilization is high and a lower threshold when memory bandwidth utilization is low. In some embodiments, the system can set a higher threshold when the system predicts future memory addresses more accurately.

[0068] In response to determining that a particular probability in the probability distribution does meet the threshold criteria, the system can extract data from the memory address associated with the probability and store the data in the local cache.

[0069] In some embodiments, the system can determine that multiple probabilities meet the threshold criteria and pre-fetch data from multiple memory addresses associated with these probabilities. In doing so, the probability of pre-fetching the correct data is increased, reducing the running time of the computer program executed on the computing system.

[0070] The system is typically configured to predict the memory address that will be accessed at time step N+1, i.e., the memory address that will be accessed when the next computer program instruction is executed. However, the recurrent neural network can be trained to predict other previous time steps. For example, the recurrent neural network can be trained to predict the memory address that will be accessed at time N+2, N+4, or N+8. For example, the recurrent neural network can be trained to predict the memory address that will be accessed at N+2 by providing the recurrent neural network with an increment calculated using the second memory address accessed at N-2.

[0071] In another embodiment, instead of pre-fetching data as described above, the system can insert a fetch instruction into the computer program. Execution of the fetch instruction results in fetching data from a future memory address associated with a probability of meeting a threshold criterion. The fetch instruction must be inserted into the computer program before a future instruction of the computer program that causes the data to be fetched from memory. In this way, when the future instruction is executed, the data required by the future instruction is already in the local cache. The execution of the future instruction will result in a cache hit, rather than a cache miss, and the system does not need to fetch data from memory. This allows the computer program to run faster, because fetching data from memory is time-consuming.

[0072] If more than one probability meets the threshold criteria, more than one fetch instruction may be inserted into the computer program.

[0073] In some embodiments, the system provides additional features to the recurrent neural network. For example, the system can provide an indication of whether the most recent memory access was a load or a store. The system can also provide the current cache occupancy to the recurrent neural network. As described above, the current cache occupancy can affect the threshold criteria.

[0074] Figure 3 is a flow chart of an example process for generating a high-dimensional embedding from a sequence of prior program counter addresses and corresponding increment values. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed neural network subsystem, e.g., Figure 1 The neural network subsystem 144 can perform processing 300.

[0075] First, a sequence of previous program counter addresses and their corresponding increment values ​​is provided to a neural network (310). For convenience and clarity, this neural network will be referred to as an embedded neural network. The embedded neural network can be found in reference Figure 3The embodiment in which the embedded neural network is connected to the recurrent neural network 300 described above is preceded by and connected to it. In an embodiment in which the embedded neural network is connected to the recurrent neural network 300, the two neural networks are trained end-to-end. In other words, the training input, i.e., the sequence of prior program counter addresses and corresponding increment values, is provided to the embedded neural network. The combined neural network then processes the input to generate outputs, which are compared to the desired or expected outputs as described above. If there are any differences, the differences are used to update the parameters of the two neural networks by stochastic gradient descent with back propagation. Using the gradients backpropagated from the recurrent neural network to the embedded neural network, the parameters of the recurrent neural network 300 are updated first, and then the parameters of the embedded neural network are updated.

[0076] Through training, the embedding neural network learns parameters that produce similar program counter addresses and increment value pairs with similar embeddings. In other words, in a high-dimensional embedding space, two embeddings that are geometrically close to each other should be procedurally similar. For example, two different program counter addresses may both correspond to a function that regularly calls another function. In this case, the two program counter addresses may correspond to high-dimensional embeddings that are geometrically close to each other. As another example, two embeddings may also be similar if their increment values ​​both correspond to memory addresses that store data in a specific linked list or a specific array.

[0077] Next, a sequence of neural network processing program counter addresses and their corresponding increment values ​​is embedded (320).

[0078] Finally, the system receives a sequence of digital embeddings in a high-dimensional embedding space from the embedding neural network as output (330). The embeddings are high-dimensional real-valued vectors.

[0079] In some embodiments, once the embedding neural network is trained, it can be represented by a simple lookup table. Program counter address and increment value pairs can be mapped to indices, and the indices can be used to look up embeddings in the lookup table. This representation is possible because the embedding neural network is static after training. In other words, once trained, given a specific input, i.e., a specific program counter address and increment value pair, the embedding neural network will always output the same high-dimensional embedding.

[0080] The sequence of previous program counter addresses and increment values ​​may be represented in other ways.

[0081] In some implementations, the sequence of prior program counter addresses and increment values ​​may be "k-means clustered."

[0082] K-means clustering is a vector quantization method that aims to partition n observation points, for example, program counter address and increment value pairs, into k clusters, where each observation point belongs to the cluster with the nearest mean or centroid that serves as the prototype of the cluster. This results in a partitioning of the data space. First, the program counter address and increment value pairs must be represented as points or vectors in space so that the system can calculate the distance between the points. For example, Figure 3 As described, the program counter address and increment value pair can be mapped to a high-dimensional embedding in a high-dimensional embedding space. In some embodiments, the program counter address can represent an x-coordinate, and the increment value can represent a y-coordinate.

[0083] First, the system must choose k, which represents the number of clusters. k is an important parameter that affects the quality of the input representation. The system can choose an appropriate value of K by trial and error. Next, the system must choose an arbitrary mean or centroid for each of the k clusters. The system then calculates the distance from each observation point, for example, each point representing a pair of program counter address and increment value, to each centroid. Each point is then assigned to the cluster whose centroid is closest to it. Finally, new centroids are calculated based on a new set of points in each cluster. The process is repeated until no points are assigned to new clusters.

[0084] Each program counter address and increment value pair can then be represented as a distance to the centroid of its population. These distances can be normalized and used as input to the recurrent neural network 300. For a more detailed discussion of k-means clustering, see, for example, MacQueen, J., Some methods for classification and analysis of multivariate observations, Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1: Statistics, 281-297, University of California Press, Berkeley, CA, 1967.

[0085] An advantage of clustering pairs of program counter addresses and increment values ​​is that the data fetch patterns within a cluster are more predictable than the data fetch patterns within the entire set of program counter addresses and increment values.

[0086] In other embodiments, the sequence of program counter addresses and corresponding increment values ​​is represented as a binary string. The recurrent neural network 300 can learn a bit flip pattern. Different bits can be weighted more or less heavily to account for the fact that incorrectly predicting bit 31 of a future memory address will result in a large miss while incorrectly predicting bit 0 of a future memory address will result in a small miss.

[0087] Figure 4 An example computer program 400 is depicted. The computer program 400 is written in a fictitious assembly language. Assembly language is a low-level programming language for a computer or other programmable device. In contrast to high-level programming languages, such as C++, Java, Python, Ruby, etc., assembly language is specialized for a particular computer architecture. A compiler converts high-level programming language instructions into assembly language instructions. Assembly language instructions correspond closely to machine language instructions.

[0088] Assembly language instructions usually specify an opcode, which is the operation to be performed; a source register, which is the register that stores the data to be operated on; and a destination register, which is the register that stores the result of the operation. Assembly language operations include load, store, add, subtract, NOR, compare, branch, jump, etc.

[0089] Load operations move data from memory to registers. Store operations move data from registers to memory. Addition, subtraction, or-not, and comparison operations are arithmetic and logic operations performed on data stored in specified source registers. Comparison, branch, and jump operations correspond to conditional statements and function calls in high-level programming languages.

[0090] The computer program 400 is executed on a processor having two registers R1 and R2 and four memory addresses mem1, mem2, mem3 and mem4. The computer program 400 calculates the sum of the values ​​stored in the four memory addresses.

[0091] First, in instructions 401 and 402, the values ​​stored in mem1 and mem2 are loaded into R1 and R2, respectively. In instruction 403, R1 is added to R2 and stored in R1. After instruction 403, R1 stores the sum of (1) the value stored in mem1 and (2) the value stored in mem2.

[0092] In instruction 404, the value stored in mem3 is loaded into R2. In instruction 405, R1 is added to R2 again and stored in R1. After instruction 405, R1 stores the sum of (1) the value stored in mem1, (2) the value stored in mem2, and (3) the value stored in mem3.

[0093] In instruction 406, the value stored in mem4 is loaded into R2. In instruction 407, R1 is added to R2 again and stored in R1. After instruction 407, R1 stores the sum of (1) the value stored in mem1, (2) the value stored in mem2, (3) the value stored in mem3, and (4) the value stored in mem4. Finally, in instruction 408, R1 is stored in mem1, overwriting the previous value stored in mem1.

[0094] Computer program 400 has four load instructions: instruction 401, instruction 402, instruction 404, and instruction 406. Each of these four instructions has a program counter address and a corresponding increment value that can be provided to the system for predicting future memory addresses.

[0095] The program counter address of instruction 401 is PC. The corresponding increment value is only mem1 because there is no memory address access before instruction 401. The program counter address of instruction 402 is PC+1. The corresponding increment value is mem1-mem2. The program counter address of instruction 404 is PC+3. The corresponding increment value is mem3-mem2. Finally, the program counter address of instruction 406 is PC+5. The corresponding increment value is mem4-mem3. Four program counter address and increment value pairs can be provided as input to the prediction system, for example, Figure 1 The prediction subsystem 140.

[0096] Figure 5 An example prediction subsystem 500 is depicted for predicting future memory addresses from which data may be fetched. The prediction subsystem 500 may be implemented in hardware directly in the computing system where the fetching occurs.

[0097] Register 510 receives and stores a sequence of previous program counter addresses and corresponding increment values.

[0098] The neural network subsystem 520 of the prediction subsystem 500 has a parameter memory 522, an arithmetic logic unit (ALU) 524, and a controller 526. The parameter memory 522 stores the parameters of a long short-term memory (LSTM) network, which is a recurrent neural network. For example, the parameter memory 522 stores the input, forgetting, output, and unit weights of each LSTM layer of the LSTM network. The parameter memory 522 also stores the bias of each layer and the unit state and hidden state of each LSTM unit in each LSTM layer. In some embodiments, the parameter memory 522 is a volatile memory, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM). In other embodiments, the parameter memory 522 is a non-volatile memory, such as a read-only memory (ROM), flash memory, or magnetic memory. In other embodiments, the parameter memory 522 is a combination of volatile and non-volatile memory. For example, the weights of each LSTM layer do not change after training the LSTM network and can be stored in a non-volatile memory, while the cell state and hidden state of each LSTM unit that change frequently can be stored in a volatile memory.

[0099] ALU 524 calculates the input gate, forget gate, and output gate of each LSTM cell. It also calculates the cell state and hidden state of each LSTM cell. ALU is a combinational circuit used to perform arithmetic and bitwise operations on binary numbers. The input of ALU is the data to be operated on, called operands, and the operation code indicating the operation to be performed. The output of ALU is the result of the operation performed. Controller 526 controls the input received by ALU 524 from parameter memory 522. In addition, it also controls the operations performed by ALU 524.

[0100] ALU 524 receives as input the parameters stored in parameter memory 522 and the sequence of program counter addresses and corresponding increment values ​​stored in register 510. ALU 524 calculates the output of each LSTM unit. For example, ALU 524 calculates the input gate, forget gate, output gate, and hidden gate of each LSTM unit.

[0101] Controller 526 may cause ALU 524 to write the computed cells and hidden states to parameter memory 522 so that these cells and hidden states may be used to compute the output in the next time step.

[0102] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing device or for controlling the operation of a data processing device. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more thereof. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal, for example, on a machine-generated electrical, optical, or electromagnetic signal, the generation of which is intended to encode information for transmission to a suitable receiving device for execution by a data processing device.

[0103] The term "data processing apparatus" refers to data processing hardware and includes all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or also include an off-the-shelf or customized parallel processing subsystem, such as a GPU or another special processing subsystem. The apparatus may also be or also include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof.

[0104] A computer program, referred to or described as a program, software, software application, application, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as in one or more scripts in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files that store one or more modules, subroutines or portions of code. A computer program may be deployed to execute on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0105] As used in this specification, an "engine" or "software engine" refers to a software-implemented input / output system that provides an output that is different from the input. An engine can be a coded functional block, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine can be implemented on any suitable type of computing device including one or more processors and a computer-readable medium, such as a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smart phone, or other fixed or portable device. In addition, two or more engines can be implemented on the same computing device, or on different computing devices.

[0106] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs or ASICs, or by a combination of special purpose logic circuits and one or more programmed computers.

[0107] Computers suitable for executing computer programs can be based on general or special microprocessors or both or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for fulfilling or executing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated into the special logic circuit. Typically, the computer also includes or is operably coupled to one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk or an optical disk, to receive data therefrom or to transmit data thereto, or both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, to name a few, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive.

[0108] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0109] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display)) for displaying information to the user, as well as a keyboard and pointing device, such as a mouse, trackball, or presence-sensitive display or other surface through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone, running a messaging application, and receiving a response message from the user.

[0110] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or on the scope of the claimed protection, but rather as descriptions of specific features of specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even claimed first, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0111] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0112] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes described in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for pre-fetching data from a memory, characterized in that The method comprises: receiving a sequence of prior program counter addresses and corresponding increment values ​​of a computer program, wherein each increment value defines a difference between a corresponding first memory address and a corresponding second memory address, wherein the first memory address is a memory address from which data is extracted when an instruction pointed to by the corresponding program counter address is executed, and wherein the second memory address is a memory address from which data is extracted before the data is extracted from the first memory address; generating an input representation based on said sequence of program counter addresses and their corresponding increment values; providing the input representation as input to a recurrent neural network; receiving from the recurrent neural network an output defining a probability distribution of future increment values, wherein each probability in the distribution represents a likelihood that execution of a future instruction of the computer program will result in fetching data from a future memory address that is equal to (i) a corresponding first memory address from which data would be fetched when the instruction pointed to by a most recent program counter address in the sequence is executed, plus (ii) the future increment value corresponding to the probability; determining that one or more probabilities in the distribution defined by the outputs of the recurrent neural network satisfy a threshold criterion; responsive to determining that the one or more probabilities in the distribution satisfy the threshold criterion, modifying the computer program by automatically inserting one or more extraction instructions into the computer program based on the output from the recurrent neural network, wherein execution of the one or more fetch instructions results in fetching data from one or more future memory addresses associated with the one or more probabilities of satisfying the threshold criteria, and wherein the one or more fetch instructions are inserted into the computer program before the future instruction of the computer program; and Executing the modified computer program.

2. The method according to claim 1, characterized in that Executing the modified computer program comprises: executing the one or more fetch instructions to cause fetching of data from one or more of the future memory addresses associated with the one or more probabilities of satisfying the criteria; wherein the fetched data is stored in a local cache; and The future instruction is executed to cause at least a corresponding portion of the fetched data to be obtained from the local cache.

3. The method according to claim 1, characterized in that Further including: comparing a memory address from which data is to be extracted as a result of executing future instructions of said computer program to said probability distribution of future increment values; and Parameters of the recurrent neural network are updated based on the comparison.

4. The method according to claim 1, characterized in that The method is performed on a microprocessor, wherein the recurrent neural network is implemented on the microprocessor, and wherein executing the modified computer program comprises: executing the one or more extract instructions to extract data from one or more future memory addresses associated with one or more probabilities in the distribution that satisfy a threshold criterion; and The data is stored in a local cache on the microprocessor.

5. The method according to claim 1, characterized in that: Generating an input representation based on the sequence of program counter addresses and their corresponding increment values ​​comprises: The sequence of program counter addresses and their corresponding increment values ​​is mapped to a digital embedding sequence in a high dimensional embedding space.

6. The method according to claim 5, characterized in that Mapping the sequence of program counter addresses and their corresponding increment values ​​to a digital embedding in a high-dimensional embedding space comprises: mapping the program counter address and increment value to an index using a pre-specified vocabulary; and The corresponding embedding is retrieved from the lookup table using the index.

7. The method according to claim 5, characterized in that Further including: Divide the digital embedding input sequence into multiple groups, where each group has a centroid point; Normalize each digit embedding relative to the centroid of its corresponding cluster; and The normalized digital embedding is provided as input to the recurrent neural network.

8. The method according to claim 1, characterized in that Further including: determining that memory addresses associated with probabilities of satisfying a threshold criterion should be accessed in an order different from the order in which corresponding future instructions of the computer program appear; and Changing the order of the corresponding future instructions of the computer program.

9. The method according to claim 1, characterized in that: Further including: determining that data from a memory address associated with a probability of satisfying the threshold criterion is present in a local cache, and An age bit of the data in a local cache is updated, wherein the age bit indicates when the data was most recently used, and wherein the local cache is a least recently used local cache.

10. A system for pre-fetching data from a memory, characterized in that The system includes one or more computers and one or more storage devices storing instructions. When the instructions are executed by the one or more computers, the one or more computers are caused to perform the following operations, including: receiving a sequence of prior program counter addresses and corresponding increment values ​​of a computer program, wherein each increment value defines a difference between a corresponding first memory address and a corresponding second memory address, wherein the first memory address is a memory address from which data is extracted when an instruction pointed to by the corresponding program counter address is executed, and wherein the second memory address is a memory address from which data is extracted before the data is extracted from the first memory address; generating an input representation based on said sequence of program counter addresses and their corresponding increment values; providing the input representation as input to a recurrent neural network; receiving from the recurrent neural network an output defining a probability distribution of future increment values, wherein each probability in the distribution represents a likelihood that execution of a future instruction of the computer program will result in fetching data from a future memory address that is equal to (i) a corresponding first memory address from which data would be fetched when the instruction pointed to by a most recent program counter address in the sequence is executed, plus (ii) the future increment value corresponding to the probability; determining that one or more probabilities in the distribution defined by the outputs of the recurrent neural network satisfy a threshold criterion; responsive to determining that the one or more probabilities in the distribution satisfy the threshold criterion, modifying the computer program by automatically inserting one or more extraction instructions into the computer program based on the output from the recurrent neural network, wherein execution of the one or more fetch instructions results in fetching data from one or more future memory addresses associated with the one or more probabilities of satisfying the threshold criteria, and wherein the one or more fetch instructions are inserted into the computer program before the future instruction of the computer program; and Executing the modified computer program.

11. The system according to claim 10, characterized in that Executing the modified computer program comprises: executing the one or more fetch instructions to cause fetching of data from one or more of the future memory addresses associated with the one or more probabilities of satisfying the criteria; wherein the fetched data is stored in a local cache; and The future instruction is executed to cause at least a corresponding portion of the fetched data to be obtained from the local cache.

12. The system according to claim 10, characterized in that The operations further include: comparing a memory address from which data is to be extracted as a result of executing future instructions of the computer program with the probability distribution of future increment values; and Parameters of the recurrent neural network are updated based on the comparison.

13. The system according to claim 10, characterized in that The operations are performed on a microprocessor, wherein the recurrent neural network is implemented on the microprocessor, and wherein executing the modified computer program comprises: executing the one or more extract instructions to extract data from one or more future memory addresses associated with one or more probabilities in the distribution that satisfy a threshold criterion; and The data is stored in a local cache on the microprocessor.

14. The system according to claim 10, characterized in that Generating an input representation based on the sequence of program counter addresses and their corresponding increment values ​​comprises: The sequence of program counter addresses and their corresponding increment values ​​is mapped to a digital embedding sequence in a high dimensional embedding space.

15. The system according to claim 10, characterized in that Mapping the sequence of program counter addresses and their corresponding increment values ​​to a digital embedding in a high-dimensional embedding space comprises: mapping the program counter address and increment value to an index using a pre-specified vocabulary; and The corresponding embedding is retrieved from the lookup table using the index.

16. One or more non-transitory computer storage media encoded with instructions, characterized in that: When the instructions are executed by one or more computers, the one or more computers are prompted to perform the following operations, including: receiving a sequence of prior program counter addresses and corresponding increment values ​​of a computer program, wherein each increment value defines a difference between a corresponding first memory address and a corresponding second memory address, wherein the first memory address is a memory address from which data is extracted when an instruction pointed to by the corresponding program counter address is executed, and wherein the second memory address is a memory address from which data is extracted before the data is extracted from the first memory address; generating an input representation based on said sequence of program counter addresses and their corresponding increment values; providing the input representation as input to a recurrent neural network; receiving from the recurrent neural network an output defining a probability distribution of future increment values, wherein each probability in the distribution represents a likelihood that execution of a future instruction of the computer program will result in fetching data from a future memory address that is equal to (i) a corresponding first memory address from which data would be fetched when the instruction pointed to by a most recent program counter address in the sequence is executed, plus (ii) the future increment value corresponding to the probability; determining that one or more probabilities in the distribution defined by the outputs of the recurrent neural network satisfy a threshold criterion; responsive to determining that the one or more probabilities in the distribution satisfy the threshold criterion, modifying the computer program by automatically inserting one or more extraction instructions into the computer program based on the output from the recurrent neural network, wherein execution of the one or more fetch instructions results in fetching data from one or more future memory addresses associated with the one or more probabilities of satisfying the threshold criteria, and wherein the one or more fetch instructions are inserted into the computer program before the future instruction of the computer program; and Executing the modified computer program.