Cache replacement method, device, electronic device and medium based on reinforcement learning

Abstract

The present application provides a cache replacement method, device, electronic device and medium based on reinforcement learning, which relates to the field of computer technology. The method includes: in response to receiving a data release request, obtaining the target program register value and the target page number in the data release request, obtaining the replacement priority corresponding to the target program register value, and the replacement priority corresponding to the target page number, and obtaining the replacement priority of the target data block from all replacement priorities; if the storage space of the target cache is full, obtaining a replacement priority list; according to the comparison results of all replacement priorities in the replacement priority list, determining and deleting the replacement data block from the stored data blocks; storing the target data block in the address corresponding to the replacement data block, and storing the replacement priority of the target data block in the replacement priority list. The method of the present application has a wide range of applications and high cache replacement accuracy.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a cache replacement method, device, electronic device and medium based on reinforcement learning. Background Art

[0002] Cache replacement means that when the cache is full and a release request is received, some data blocks need to be removed when the data to be released by the release request needs to be loaded into the cache to leave cache space for storing the data to be released by the release request.

[0003] In the related art, a fixed replacement rule is usually set according to the empirical data of the program, and cache replacement is performed according to the fixed replacement rule.

[0004] However, the replacement rules obtained by the method based on the related art are only applicable to some programs, but not to all programs, and the method has the problem of low cache replacement accuracy. Summary of the invention

[0005] The embodiments of the present application provide a cache replacement method, device, electronic device and medium based on reinforcement learning to solve the problem in the prior art that cache replacement cannot be adaptively performed according to different programs.

[0006] In a first aspect, an embodiment of the present application provides a cache replacement method, comprising: in response to receiving a data release request, obtaining a target program register value and a target page number in the data release request, wherein the data release request is used to store a target data block in a target cache; obtaining a replacement priority corresponding to the target program register value and a replacement priority corresponding to the target page number, and obtaining the replacement priority of the target data block from all replacement priorities; if the storage space of the target cache is full, obtaining a replacement priority list; the replacement priority list includes the replacement priority of at least one stored data block in the target cache; according to the comparison result of all replacement priorities in the replacement priority list, determining and deleting a replacement data block from the stored data blocks; storing the target data block in an address corresponding to the replacement data block, and storing the replacement priority of the target data block in the replacement priority list.

[0007] In a second aspect, an embodiment of the present application provides a cache replacement device, the device comprising: a first acquisition module, for acquiring a target program register value and a target page number in response to receiving a data release request, the data release request being used to store a target data block into a target cache; a second acquisition module, for acquiring a replacement priority corresponding to the target program register value and a replacement priority corresponding to the target page number, and acquiring the replacement priority of the target data block from all replacement priorities; a third acquisition module, for acquiring a replacement priority list if the storage space of the target cache is full; the replacement priority list includes the replacement priority of at least one stored data block in the target cache; a first determination module, for determining and deleting a replacement data block from the stored data blocks based on a comparison result of all replacement priorities in the replacement priority list; a first storage module, for storing the target data block into an address corresponding to the replacement data block, and storing the replacement priority of the target data block in the replacement priority list.

[0008] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method of the first aspect.

[0009] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method of the first aspect.

[0010] In an embodiment of the present application, the replacement priority list is dynamically updated through the target program register value (Program Counter, PC) and the target page number value (Page Number, PN) of the data release request. Based on the dynamically updated replacement priority list, it is possible to adaptively determine the replacement data block that is compatible with the current program during program operation, thereby improving the accuracy of cache replacement. By jointly predicting the replacement data block through the PC value and the PN value, during program operation, the best cache replacement strategy can be adaptively selected during operation according to the program's own memory access characteristics, thereby achieving the purpose of accurately obtaining replacement data blocks for different programs. The cache replacement method of this embodiment has a good cache replacement effect for different programs.

[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 This is a schematic diagram of an application scenario of a cache replacement method provided in an embodiment of the present application;

[0014] Figure 2 It is a flow chart of the steps of a cache replacement method provided by an embodiment of the present application;

[0015] Figure 3 is a flowchart of another cache replacement method provided by an embodiment of the present application;

[0016] Figure 4 is a flowchart of another cache replacement method provided by an embodiment of the present application;

[0017] Figure 5 is a flowchart of another cache replacement method provided by an embodiment of the present application;

[0018] Figure 6 is a flowchart of another cache replacement method provided by an embodiment of the present application;

[0019] Figure 7 This is a flowchart of another cache replacement method provided by an embodiment of the present application;

[0020] Figure 8 It is a schematic diagram of an L3 cache directory structure and signal flow provided by an embodiment of the present application;

[0021] Fig. 9 This is a flowchart of another cache replacement method provided by an embodiment of the present application;

[0022] Fig.10 is a block diagram of a cache replacement device provided by an embodiment of the present invention;

[0023] Fig.11 is a block diagram of an electronic device provided by an embodiment of the present invention;

[0024] Fig.12 is a block diagram of another electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, the term "and / or" in the specification and claims is used to describe the association relationship of associated objects, indicating that three kinds of relationships can exist, for example, A and / or B can be represented: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the front and back associated objects are a kind of "or" relationship. In the embodiment of the present application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0027] Figure 1 is a schematic diagram of an application scenario of a cache replacement method based on reinforcement learning provided in an embodiment of the present application, with reference to Figure 1, the application scenarios include at least the central processing unit (CPU), L1 cache, L2 cache and L3 cache. Cache is a first-level memory located between the main memory and the CPU, usually composed of static random-access memory (SRAM). The cache access speed is fast, but the capacity is much smaller than the main memory, the purpose is to improve the memory access speed. Modern processors usually adopt a three-level cache architecture including L1 cache, L2 cache and L3 cache, among which L3 cache is the last-level cache. When the CPU is running, it first obtains the required data from the L1 cache. If the L1 cache does not have the required data, it will search from the L2 cache. If the L2 cache does not have the required data, it will search from the L3 cache. If the required data is not found in any of the three levels of cache, it means that the data is missing in the cache, and the data to be obtained needs to be searched from the last level cache (L3 cache) from the main memory. The time cost of the last level cache to search for data from the main memory can be more than hundreds of beats, while the data access between the L2 cache and the L3 cache only requires a dozen to dozens of beats. Therefore, when data in L3 is missing, it takes more time to search for data from the main memory, which affects the processing efficiency of the CPU. In order to improve the processing efficiency of the processor, the cache hit rate needs to be improved. However, the storage space of the cache is limited, and only a part of the data can be stored in the cache. In order to improve the cache hit rate, it is necessary to obtain a cache replacement method with high accuracy, so that when the storage space corresponding to the index of the data to be stored in the last-level cache is full, the most suitable stored data block is determined, and the data block required by the CPU is used to replace the determined stored data block, and then the data block required by the CPU is stored in the cache to improve the cache hit rate.

[0028] In the related art, cache replacement can be performed through processor cache replacement methods such as the Least Recently Used (LRU) algorithm and the Static Re-reference Interval Prediction (SRRIP) algorithm. However, these methods manually set fixed replacement rules based on empirical data. Such fixed replacement rules are effective for some programs, but only for programs that obtain empirical data. The cache replacement results for other programs are inaccurate.

[0029] In order to improve the accuracy of cache replacement, in response to receiving a data release request, an embodiment of the present application obtains the target program register value and the target page number in the data release request, and the data release request is used to store the target data block in the target cache; obtains the replacement priority corresponding to the target program register value, and the replacement priority corresponding to the target page number, and obtains the replacement priority of the target data block from all replacement priorities; if the storage space of the target cache is full, obtains a replacement priority list; the replacement priority list includes the replacement priority of at least one stored data block in the target cache; according to the comparison results of all replacement priorities in the replacement priority list, determines and deletes the replacement data block from the stored data blocks; stores the target data block in the address corresponding to the replacement data block, and stores the replacement priority of the target data block in the replacement priority list. This embodiment realizes the dynamic update of the replacement priority list, and based on the dynamically updated replacement priority list, it can adaptively determine the replacement data block adapted to the current program during the program running process, thereby improving the accuracy of cache replacement. By jointly predicting the replacement data blocks through the PC value and the PN value, during the program running process, the best cache replacement strategy can be adaptively selected during the running process according to the program's own memory access characteristics, thereby achieving the purpose of accurately obtaining replacement data blocks for different programs. The cache replacement method based on reinforcement learning provided in this embodiment has a good cache replacement effect on different programs.

[0030] It should be noted that the application scenario of the above three-level cache is only an exemplary description of the embodiment of the present application, and the method of the present application is also applicable to application scenarios of other levels of cache.

[0031] The cache replacement method based on reinforcement learning provided by the embodiment of the present application is described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0032] Figure 2 is a flowchart of a cache replacement method based on reinforcement learning provided by an embodiment of the present application, such as Figure 2 As shown, the method may include:

[0033] Step 101 : In response to receiving a data release request, obtaining a target program register value and a target page number in the data release request.

[0034] The data release request is used to store the target data block into the target cache. The data release request carries the target program register value and the target page number. The upper cache can release unnecessary data to the lower cache for storage. This process is implemented based on the data release (Release) request. That is, the data release request is used to release the target data block that needs to be released from the upper cache level of the target cache to the target cache of the next level. For example, if the target cache is the L3 cache, the data release request is used to release the target data block that needs to be released from the L2 cache to the target cache. Among them, the target program register value and the target page number are respectively the program register value and the page number of the target data block to be released by the data release request. Among them, the target PC value is used to store the address of the current pre-execution instruction; the target PN value is the page number of the cache block that the program wants to access in the page table.

[0035] Step 102, obtaining a replacement priority corresponding to the target program register value and a replacement priority corresponding to the target page number, and obtaining a replacement priority of the target data block from all replacement priorities.

[0036] The replacement priority (Insertion-age) of the data block stored in the target cache is positively correlated with the probability of the stored data block being replaced when the storage space of the target cache is full. That is, the greater the replacement priority, the easier it is for the stored data block corresponding to the replacement priority to be replaced. For example, a first list is obtained, and the first list is used to store at least one first action (Action) information corresponding to each PC value, and the first action information includes at least status information and replacement priority. The status information can be a hit (Hit), a miss (Miss), or directly stored in the memory (Bypass). Among them, a hit means that the lower-level cache has a data block to be accessed by the upper-level cache; a miss means that the lower-level cache does not have a data block to be accessed by the upper-level cache. Further, the first action information can also include an action value function value (Q value). For example, a PC value can correspond to 9 first action information. For example, the target program register value is the i-th one among the preset program register values. The target register value is represented as PC(i), and the first action information corresponding to the target register value PC(i) can be shown in Table 1:

[0037]

[0038] Where N represents the replacement priority corresponding to the state information that is not hit and is directly released to the memory. Q(i) 11 、Q(i) 12 、Q(i) 13 、Q(i) 14Respectively represent the Q values ​​corresponding to the actions with a hit and replacement priority of 0, a hit and replacement priority of 1, a hit and replacement priority of 2, and a hit and replacement priority of 3 in the action information (Action) corresponding to the target register value PC(i). Q(i) 15 、Q(i) 16 、Q(i) 17 、Q(i) 18 、Q(i) 19 They respectively represent the Q values ​​corresponding to the action information (Action) corresponding to the target register value PC(i), which are miss and replacement priority 0, miss and replacement priority 1, miss and replacement priority 2, miss and replacement priority 3, and miss and directly released to memory.

[0039] The target state information of the data release request can be obtained, and then all replacement priorities corresponding to the target state information can be obtained from the first list shown in Table 1. These replacement priorities are replacement priorities corresponding to the target program register value. For example, if the target state information of the data release request is a hit, then the target program register value PC i The corresponding replacement priorities include: 0, 1, 2, 3.

[0040] For example, a second list is obtained, and the second list is used to store at least one second action information corresponding to each PN value, and the second action information includes at least state information and replacement priority. The state information can be hit, miss, or directly stored in memory (Bypass).

[0041] Furthermore, the second action information may also include an action value function value (Q value). For example, one PN value may correspond to 9 first action information. For example, the target page number is the i-th one in the preset program register value, and the target page number is represented as PN(i). The second table corresponding to the target register value PN(i) may be shown in Table 2:

[0042]

[0043] Where N represents the replacement priority corresponding to the state information that is not hit and is directly released to the memory. Q(i) 21 、Q(i) 22 、Q(i) 23 、Q(i) 24 , respectively represent the Q values ​​corresponding to the actions corresponding to the target page number PN(i) that are hit and have a replacement priority of 0, hit and have a replacement priority of 1, hit and have a replacement priority of 2, and hit and have a replacement priority of 3. Q(i) 25 、Q(i)26 、Q(i) 27 、Q(i) 28 、Q(i) 29 They respectively represent the Q values ​​corresponding to the action information (Action) corresponding to the target page number PN(i), which are miss and replacement priority 0, miss and replacement priority 1, miss and replacement priority 2, miss and replacement priority 3, and miss and directly released to the memory.

[0044] The target state information of the data release request can be obtained, and then all replacement priorities corresponding to the target state information can be obtained from the second list shown in Table 2. These replacement priorities are replacement priorities corresponding to the target page number. For example, if the target state information of the data release request is hit, the replacement priorities corresponding to the target page number PN(i) include: 0, 1, 2, 3.

[0045] For example, the number of items in the first list and the second list can be 8192 items each. In the first list, each item corresponds to a program register value, and in the second list, each item corresponds to a page number. In this embodiment, the number of bits of the program register value and the page number only needs 13 bits (8192 items), but the program register value and the page number in the program actually have more bits. In this embodiment, the lower 13 bits of the real program register value and the upper 13 bits of the real page number are intercepted.

[0046] The first list and the second list can have more items (greater than 8192 items). If there are more items, the prediction effect will be more accurate, but the area overhead will be very large and the training speed will be slower. The number of items in the first list and the second list can be different from 8192. The number of items in the first list and the second list can be set according to the accuracy requirements of the prediction effect, the overhead limit, and the training speed.

[0047] In this embodiment, the replacement priorities shown include 5 types, namely 0, 1, 2, 3, and a replacement priority N corresponding to a miss and direct release to the memory. A miss and direct release to the memory can be understood as the highest replacement priority, that is, it is replaced immediately without waiting for the next data block to come, but it is replaced only when there is no storage location in the target cache. It should be noted that it is necessary to consider direct release to the memory only when there is a miss. If it is a hit, it means that the target cache now has this data block and does not need to replace it.

[0048] Step 103: If the storage space of the target cache is full, obtain a replacement priority list.

[0049] The replacement priority list includes a replacement priority of at least one stored data block in the target cache.

[0050] For example, data in the replacement priority list is recorded in a replacement priority memory, which may be a static random access memory (replacer_Static Random-Access Memory, replacer SRAM) that records replacement priorities of data blocks in the replacement priority list for users.

[0051] Step 104: Determine and delete a replacement data block from the stored data blocks according to the comparison results of all replacement priorities in the replacement priority list.

[0052] The replacement priority is positively correlated with the probability of the stored data block being evicted from the target cache. For example, all replacement priorities in the replacement priority list are compared to obtain the maximum replacement priority, and the stored data block corresponding to the maximum replacement priority is determined as the replacement data block, which is deleted from the target cache.

[0053] Processor cache designs often use a multi-way set-connected structure. A data block can be placed in different ways with the same index number (Set), and an index number value can include multiple ways (Ways). The data release request includes the index number value of the target data block. According to the index number value of the target data block, the stored data blocks with the same index number value as the target data block in the stored data blocks of the target cache are determined. If the stored data blocks with the same index number value as the target data block have occupied all the ways under the index number value of the target data block, it is determined that the storage space of the target cache is full. In this case, the determined stored data blocks are evicted from the stored data blocks with the same index number value as the target data block to release cache space for the target data block.

[0054] Furthermore, the replacement priority list includes the index number (Set) value and replacement priority of the stored data block. Each stored data block has a unique index number value, and an index number value may include multiple ways (Ways). A data block can be placed in different ways of the same index number. For example, if an index number value has 16 ways, it means that the data block corresponding to this index number can be placed in any of the 16 ways, and 16 data blocks can be stored under this index number.

[0055] For example, the replacement priorities of each stored data block with the same index number value as the target data block are obtained, and the largest replacement priority is selected from them, and the stored data block corresponding to the largest replacement priority is determined as the replacement data block to be evicted. For example, the index number for storing the target data block carried by the data release request is Set(i), and the index number has 16 ways, and when the data release request is received, each way has a stored data block. That is, there are 16 stored data blocks under the index number, which means that the storage space of the target cache is full. From the replacement priority list, the replacement priorities of the 16 stored data blocks are obtained and compared, and then the stored data block corresponding to the largest replacement priority is determined as the replacement data block, and it is evicted from the target cache. Furthermore, if there are multiple largest replacement priorities, the way number of the stored data block corresponding to the largest replacement priority can be obtained, and the stored data block with the smallest way number can be determined as the replacement data block, and it is evicted from the target cache. For example, the stored data blocks with path number 1 and path number 3 have the highest replacement priority, and the replacement priorities of these two stored data blocks are both 3, so the stored data block with path number 1 is determined as the replacement data block and deleted from the target cache.

[0056] Step 105: Store the target data block into an address corresponding to the replacement data block, and store the replacement priority of the target data block into a replacement priority list.

[0057] After the replacement data block is deleted, the cache slice corresponding to the replacement data block in the target cache is in an idle state, and the target data block can be written to the address corresponding to the replacement data block.

[0058] The replacement priority of the target data block is stored in the replacement priority list to obtain an updated replacement priority list. When a new data release request is subsequently received and the target cache is full, a new replacement data block is determined and deleted based on the comparison results of all replacement priorities in the updated replacement priority list to write the target data block corresponding to the new data release request to the address of the new replacement data block.

[0059] Preferably, the method of this embodiment is implemented at the hardware level.

[0060] In an embodiment of the present application, the replacement priority corresponding to the target data block to be released by the data release request is dynamically obtained according to the target PC value and the target PN value of the data release request. When the target data block is stored in the address corresponding to the replacement data block, the replacement priority of the target data block is stored in the replacement priority list, thereby realizing the dynamic update of the replacement priority list. Based on the dynamically updated replacement priority list, it is possible to adaptively determine the replacement data block adapted to the current program during the program running process, thereby improving the cache replacement accuracy. The method of this embodiment can predict the replacement data block through the PC value and the PN value together. During the program running process, it can adaptively select the best cache replacement strategy during the running process according to the program's own memory access characteristics, thereby achieving the purpose of accurately obtaining replacement data blocks for different programs. The cache replacement method based on reinforcement learning in this embodiment has a good cache replacement effect for different programs.

[0061] In this embodiment, after receiving a data release request, if the storage space of the target register is full, based on the comparison result of the replacement priority in the replacement priority list, the replacement data block can be accurately determined from the stored data blocks, the replacement data block is deleted, and the target data block is stored in the address corresponding to the replacement data block, thereby realizing accurate determination of the replacement data block and storage of the target data block corresponding to the data release request.

[0062] Figure 3 is a flowchart of a cache replacement method based on reinforcement learning provided in an embodiment of the present application, referring to Figure 3 , the method may include the following steps:

[0063] Step 201 , in response to receiving a data release request, obtaining a target program register value and a target page number in the data release request.

[0064] The method of this step has been described in the aforementioned step 101 and will not be repeated here.

[0065] Step 202: Obtain target status information of the data release request.

[0066] The target status information includes: release hit, release miss, or storing the target data block into memory (by pass).

[0067] For example, step 202 may include sub-steps:

[0068] Sub-step 2021, obtain the meta list.

[0069] The meta list includes program register values ​​and page numbers of data blocks stored in the target cache.

[0070] Sub-step 2022, determining the target state information of the data release request according to the target program register value, the target page number, and the meta list.

[0071] The target state information of the data release request is determined according to the target PC value, the target PN value, and the PC value and PN value of the stored data block stored in the meta list. If the PC value and PN value that are the same as the target PC value and the target PN value exist in the meta list, the target state information is determined to be a hit, otherwise it is a miss.

[0072] Step 203: Obtain a first list.

[0073] The first list includes a plurality of program register values, one program register value has corresponding at least one preset first action information, and the action information includes state information and replacement priority.

[0074] Among them, the first action information corresponding to the same program register value, and the state information and replacement priority included in the first action information can refer to the description of the aforementioned Table 1, and will not be repeated here.

[0075] Step 204: Obtain at least one first action information corresponding to the target program register value from the first list, and obtain target first action information having the same state information as the target state information from the at least one first action information corresponding to the target program register value.

[0076] For example, if the target program register value is PC(i), then at least one first action information corresponding to the target program register value includes Action1 to Action9 in Table 1. For example, if the target state information of the data release request is hit, and in the first action information, the state information included in Action1 to Action4 is hit, then the target first action information having the same state information as the target state information includes four types of Action1 to Action4.

[0077] Step 205: determine the replacement priority in the target first action information as the replacement priority corresponding to the register value of the target program.

[0078] For example, the target first action information includes four types, Action 1 to Action 4, and the replacement priorities corresponding to the target PC value PC(i) include: 0, 1, 2, 3.

[0079] Step 206, obtaining a second list.

[0080] The second list includes multiple page numbers, and one page number has at least one corresponding preset second action information, and the second action information includes state information and replacement priority. The second action information corresponding to the same page number, and the state information and replacement priority included in the second action information can refer to the description of the aforementioned Table 2, which will not be repeated here.

[0081] Step 207: Acquire at least one second action information corresponding to the target page number from the second list, and acquire target second action information having the same state information as the target state information from the at least one second action information corresponding to the target page number.

[0082] For example, if the target page number is PN(i), then at least one second action information corresponding to the target page number includes Action1 to Action9 in Table 2. For example, if the target state information of the data release request is hit, and in the second action information, the state information included in Action1 to Action4 is hit, then the target second action information having the same state information as the target state information includes four types of Action1 to Action4.

[0083] Step 208: determine the replacement priority in the target second action information as the replacement priority corresponding to the target page number.

[0084] For example, the target second action information includes four types, Action 1 to Action 4, and the replacement priorities corresponding to the target PC value PN(i) include: 0, 1, 2, 3.

[0085] Step 209, obtaining and replacing the action value function value corresponding to the priority.

[0086] Specifically, obtaining and replacing the action value function value corresponding to the priority includes: obtaining the action value function value of the replacement priority corresponding to the target program register value, and the action value function value of the replacement priority corresponding to the target page number.

[0087] For example, referring to Table 1, the replacement priorities corresponding to the target program register values ​​include the replacement priorities in Action 1 to Action 4, and the action value function values ​​of the replacement priorities corresponding to the target program register values ​​include: Q(i) 11 、Q(i) 12 、Q(i) 13 、Q(i) 14 For example, referring to Table 2, the replacement priority corresponding to the target page number includes the replacement priorities in Action 1 to Action 4, and the action value function value of the replacement priority corresponding to the target page number includes: Q(i) 21 、Q(i) 22、Q(i) 23 、Q(i) 24 .

[0088] Step 210, obtaining the largest first action-value function value from the action-value function values ​​of the replacement priority corresponding to the register value of the target program.

[0089] For example, the action value function values ​​of the replacement priority corresponding to the target program register value include: Q(i) 11 、Q(i) 12 、Q(i) 13 、Q(i) 14 , where Q(i) 13 The maximum value of the first action value function is Q(i) 13 .

[0090] Step 211, obtaining the maximum second action-value function value from the action-value function values ​​of the replacement priority corresponding to the target page number.

[0091] For example, the action value function values ​​of the replacement priority corresponding to the target page number include: Q(i) 21 、Q(i) 22 、Q(i) 23 、Q(i) 24 . Among them, Q(i) 12 The maximum value of the second action value function is Q(i) 12 .

[0092] Step 212, compare the first action value function value and the second action value function value to obtain the maximum action value function value, and determine the replacement priority corresponding to the maximum action value function value as the replacement priority of the target data block.

[0093] For example, the first action value function value is Q(i) 13 , the value of the second action value function is Q(i) 12 , where Q(i) 13 >Q(i) 12 , refer to Table 1, Q(i) 13 The corresponding priority is 2, so the replacement priority of the target data block of the data release request is 2.

[0094] Step 213: If the storage space of the target cache is full, obtain a replacement priority list.

[0095] The replacement priority list includes the replacement priority of at least one data block stored in the target cache. The method of this step has been described in the aforementioned step 103 and will not be repeated here.

[0096] Step 214: Determine and delete a replacement data block from the stored data blocks according to the comparison results of all replacement priorities in the replacement priority list.

[0097] The method of this step has been described in the aforementioned step 104 and will not be repeated here.

[0098] Step 215: Store the target data block into the address corresponding to the replacement data block, and store the replacement priority of the target data block into a replacement priority list.

[0099] The method of this step has been described in the aforementioned step 105 and will not be repeated here.

[0100] In this embodiment, according to the target state information of the data release request, the replacement priority corresponding to the target PC value is obtained from the first list corresponding to the target PC value, and the first action value function value with the largest action value function value is obtained from it; the replacement priority corresponding to the target PN value is obtained from the second list corresponding to the target PN value, and then the second action value function value with the largest action value function value is obtained from it; the maximum value is obtained from the first action value function value and the second action value function value, and the replacement priority corresponding to the maximum value is determined as the replacement priority of the target data block, and the replacement priority of the target data block is stored in the replacement priority list, so as to realize the dynamic update of the replacement priority list. Based on the replacement priority list dynamically updated by the method of this embodiment, a cache replacement method based on reinforcement learning can be realized, and the replacement data block adapted to the current program can be adaptively determined during the program running process, thereby improving the cache replacement accuracy. The target PC value and the target PN value of the data release request are used to predict the replacement data block together. During the program running process, the best cache replacement strategy can be adaptively selected during the running process according to the program's own memory access characteristics, so as to achieve the purpose of accurately obtaining replacement data blocks for different programs.

[0101] Reference Figure 4 , after step 215, further comprising the following steps:

[0102] Step 216, obtaining an evaluation list.

[0103] Among them, the evaluation list includes at least one evaluation queue, one evaluation queue corresponds to a preset index number, and the evaluation queue is used to store data block information of the stored data block stored in the preset index number, and the data block information includes the program register value, page number, action information, and action value function value of the data block.

[0104] The evaluation queue is a first-in-first-out queue. Furthermore, each evaluation queue may include the following seven pieces of information: program register value, page number, status information, target action information selected from the first list and the second list, and action value function value.

[0105] For example, a preset number of index numbers (for example, 64) are selected from all index numbers for the cache, and the preset number of index numbers is the preset index number in this step. The preset number of index numbers constitutes the sample set (Sampleset) in this application, and model training is performed based on the sample set to obtain a reinforcement learning model for cache replacement prediction. Among them, a preset index number has a corresponding evaluation queue, and the evaluation queue stores program register values ​​(PC values), page numbers (PN values), action information (Action), and action value function values ​​(Q values). Among them, the action information Action includes state information and replacement priority. For example, in the L3 cache of the preset processor, there are a total of 4096 indexes, which need to be represented by 12 bits. 64 index number samples are selected from them to form a sample set for training the reinforcement learning model.

[0106] For example, an index number whose high 6 bits and low 6 bits add up to 31 is selected and determined as the preset index number constituting the sample set. The index number samples obtained based on this method are relatively uniform. Further, the number of data block information stored in the evaluation queue is equal to the preset number threshold. Among them, the preset number threshold can be set according to user needs, for example, it can be set to 32. Specifically, the evaluation queue is used to store the data block information of a preset number (for example, 32) of stored data blocks that have been recently stored in the target cache. In one embodiment, the number of preset index numbers is 64, and the evaluation queue corresponding to a preset index number stores the data block information of 32 stored data blocks that have been recently stored in the target cache. The data block information includes: program register value (PC value), page number (PN value), address (addr), action information (Action), Q value (Q-Value), reward value (reward), and status information. The status information includes hit or miss.

[0107] Step 217, obtaining the target index number and target status information of the data release request, and data block information of the target data block.

[0108] The target status information includes: hit, miss, or storing the target data block into the memory.

[0109] The data release request carries the target PC value, target PN value, and target Set. The target Set is obtained by parsing the data release request. The target state information of the data release request is obtained according to the target PC value, target PN value, and meta list.

[0110] Step 218: When the target index number is the same as any preset index number, a target evaluation queue having the same preset index number as the target index number is obtained from the evaluation list.

[0111] The preset index number is a sample set for reinforcement learning in this embodiment. If the target index number is the same as any preset index number, it means that the target index number belongs to one of the sample sets. For example, after obtaining the target evaluation queue with the same preset index number and the target index number from the evaluation list, it also includes: storing the data block information of the target data block in the target evaluation queue.

[0112] Step 219: If the storage space of the target evaluation queue is full, obtain the first data block information of the first data block first stored in the target evaluation queue.

[0113] For example, the target evaluation queue is used to store data block information of stored data blocks. When the number of stored data blocks reaches a preset number threshold, it is determined that the storage space of the target evaluation queue is full. For example, the preset number threshold is 32. When the data block information of 32 stored data blocks is stored in the target evaluation queue, it is determined that the storage space of the target evaluation queue is full. For example, the stored data blocks stored in the target evaluation queue in chronological order are marked as: data block 0, data block 1 to data block 31, and data block 0 is the first data block first stored in the target evaluation queue.

[0114] For example, the data block information also includes a reward value; after step 219, the following steps are also included:

[0115] Step 220, obtaining the second data block information of the second data block from the target evaluation queue, and extracting the action value function value in the second data block information.

[0116] The second data block information is the data block information stored in the target evaluation queue after the first data block information. For example, the data block information of the stored data blocks stored in the target evaluation queue in chronological order is marked as: data block 0, data block 1 to data block 31, then data block 0 is the first data block first stored in the target evaluation queue, and data block 1 is the second data block stored in the target evaluation queue after the first data block.

[0117] Step 221, obtaining an updated action-value function value of the first data block according to the reward value of the first data block, the action-value function value of the first data block, and the action-value function value of the second data block.

[0118] For example, obtain the ratio of the action value function value Q(head) of the second data block to the first hyperparameter, obtain a first summation result of the ratio and the reward value reward of the first data block, obtain the difference between the first summation result and the action value function value Q(old) of the first data block, obtain the product of the difference and the second hyperparameter; obtain a second summation result of the product and the action value function value Q(old) of the first data block, and determine the second summation result as the updated action value function value Q(new) of the first data block.

[0119] For example, the first hyperparameter is 3, the second hyperparameter is 50, and the update action value function value Q(new) of the first data block can be obtained according to the following method:

[0120] Q(new)=Q(old)+50[reward+Q(head) / 3-Q(old)]

[0121] For example, the formula for obtaining the updated action value function value Q(new) can be further optimized. Specifically, the two hyperparameters 50 and 3 can be optimized through multiple experiments.

[0122] Step 222, using the updated action value function value of the first data block, replaces the action value function value in the first list corresponding to the program register value, state information and replacement priority of the first data block.

[0123] The first list includes multiple PC values, each PC value has at least one corresponding action information, and each action information includes state information, replacement priority, and action value function value. The action information, state information, replacement priority, and action value function value corresponding to the PC value in the first list can refer to Table 1 in the aforementioned embodiment, and will not be repeated here.

[0124] After obtaining the updated action value function value of the first data block, use the updated action value function value to replace the action value function value in the action information in the first list whose program register value, status information, and replacement priority correspond to the program register value, status information, and replacement priority of the first data block respectively.

[0125] For example, the PC value of the first data block is PC(j), its status information is hit, the replacement priority is 0, and the updated action value function value is Q(new). In the first list, the action value function value corresponding to the action information with PC value PC(j), status information is hit, and replacement priority is 0 is Q(j). 11 , then use Q(new) to replace Q(j) 11 .

[0126] Step 223: Use the update action value function value of the first data block to replace the action value function value in the second list corresponding to the page number, status information and replacement priority of the first data block.

[0127] The second list includes multiple PN values, each PN value has at least one corresponding action information, and each action information includes state information, replacement priority, and action value function value. The action information, state information, replacement priority, and action value function value corresponding to the PN value in the first list can refer to Table 2 in the aforementioned embodiment, and will not be repeated here.

[0128] After obtaining the updated action value function value of the first data block, use the updated action value function value to replace the action value function value in the action information in the first list whose program register value, status information, and replacement priority correspond to the program register value, status information, and replacement priority of the first data block respectively.

[0129] For example, the PN value of the first data block is PN(j), its status information is hit, the replacement priority is 0, and the updated action value function value is Q(new). In the first list, the action value function value corresponding to the action information with the PC value of PC(j), the status information of hit, and the replacement priority of 0 is Q(j). 21 , then use Q(new) to replace Q(j) 21 .

[0130] Step 224: Delete the first data block information of the first data block from the target evaluation queue.

[0131] The data block information of the first data block is deleted from the target evaluation queue, thereby obtaining the data block information for storing the target data block from the target evaluation queue whose storage space is full.

[0132] Step 225: store the data block information of the target data block into the target evaluation queue.

[0133] Specifically, the data block information of the target data block is stored in the target evaluation queue under the address of the first data block. The data block information of the target data block is stored in the target evaluation queue so that in a subsequent processing process, based on the data block information of the target data block and the data block information of other data blocks in the target evaluation queue, the action value function values ​​in the first list and the second list are dynamically updated.

[0134] For example, refer to Figure 5 , the method further comprises the steps of:

[0135] Step 226, in response to receiving the data acquisition request, obtain the index number, program register value, page number and status information of the data acquisition request.

[0136] The data acquisition request includes an immediate data acquisition request (Acquire) from the upper cache to the lower cache for a data block, and a pre-fetch request (Hint) from the upper cache to the lower cache, indicating that the data block may be used later. The data acquisition request is parsed to obtain the index number, program register value, and page number of the data acquisition request. According to its program register value and page number, as well as the meta list, the status information of the data acquisition request is obtained.

[0137] Step 227: If the index number of the data acquisition request is the same as any preset index number, obtain third data block information having the same index number, program register value, page number and status information as the data acquisition request from the evaluation list.

[0138] Specifically, the evaluation list includes a plurality of evaluation queues, one evaluation queue includes data block information of at least one data block, and the data block information includes at least an index number, a program register value, a page number, and status information.

[0139] From the plurality of evaluation queues, third data block information is obtained, the index number of which is the same as the index number of the data acquisition request, the program register value of which is the same as the program register value of the data acquisition request, the page number of which is the same as the page number of the data acquisition request, and the status information of which is the same as the status information of the data acquisition request. Further, the data block corresponding to the third data block information can be represented as a third data block.

[0140] Step 228, if the third data block information is the data block information first stored in the evaluation queue to which the third data block information belongs, and the reward value of the third data block information is not equal to the preset initial reward value, then update the reward value in the third data block information according to the status information of the data acquisition request and the first reward value update strategy.

[0141] The initial reward value is a reward value preset for the evaluation queue. The first third data block stored is the data block to be evicted. When a data acquisition request is received, even if the storage space of the evaluation queue is full, it is not necessary to delete the data block information of the data block from the evaluation queue. However, when a data release request is received, it is necessary to delete the data block information of the first stored data block from the evaluation queue to leave storage space for storing the data block information of the new data block. After receiving the data release request, the data block information of the data block that needs to be deleted from the evaluation queue is the third data block information of the third data block in this step.

[0142] For example, in the evaluation queue to which the third data block belongs, the stored data blocks stored in chronological order are marked as: data block 0, data block 1 to data block 31, and data block 0 is the first to be stored in the evaluation queue to which the third data block belongs. When a data release request is received and the evaluation queue is full, the data block to be evicted is data block 0, which is the third data block, and its data block information is the third data block information of the third data block.

[0143] Reference Figure 6 , step 228 may include the following sub-steps:

[0144] Sub-step 2281, obtaining, in the preset action information, the first replacement priority of the action information whose all status information is hit, and the second replacement priority of the action information whose all status information is miss.

[0145] For example, in one embodiment, there are 9 preset action information, specifically including Action1 to Action9, where the status information of Action1 to Action4 is hit, and the first replacement priorities of Action1 to Action4 are 0, 1, 2, and 3 respectively. The status information of Action5 to Action9 is hit, and the second replacement priorities of Action5 to Action9 are 0, 1, 2, 3, and N respectively, where N>3.

[0146] In this embodiment, the first replacement priority of all action information with hit status information includes 0, 1, 2, and 3, and the second replacement priority of all action information with miss status information includes 0, 1, 2, 3, and N. Further, the correspondence between the preset action information and the priority can refer to the description of Table 1 or Table 2.

[0147] Sub-step 2282, if the status information of the data acquisition request is hit, and the replacement priority of the third data block is equal to the maximum value of the first replacement priority, then the reward value in the third data block information is updated to R1; R1>0, R1 represents the first preset reward value. For example, R1 can be set to 28000.

[0148] The third data block is the data block to be evicted. If the reward value of the third data block is equal to the preset initial reward value, it means that the reward value of the first data block has not changed from being stored in the evaluation queue to being evicted from the evaluation queue. In this case, if the status information of the data acquisition request is a hit, the reward value in the third data block information is updated to R1.

[0149] Sub-step 2283, if the status information of the data acquisition request is a hit, and the replacement priority of the third data block is not equal to the maximum value of the first replacement priority, then the reward value in the third data block information is updated to -R1.

[0150] If the reward value in the third data block information is equal to the preset initial reward value, it means that the reward value of the third data block corresponding to the reward value has not been changed when it is to be expelled from the evaluation queue.

[0151] Sub-step 2284, if the status information of the data acquisition request is a miss, and the replacement priority of the third data block is equal to the maximum value of the second replacement priority, then the reward value in the third data block information is updated to R2; R2>0, R2 represents the second preset reward value.

[0152] Wherein, R2>0, R2 represents the second preset reward value. For example, R2 can be equal to R1, and both are set to 28000.

[0153] Sub-step 2285, if the status information of the data acquisition request is a miss, and the replacement priority of the third data block is not equal to the maximum value of the second replacement priority, then the reward value in the third data block information is updated to -R2.

[0154] For example, the first reward value update strategy in sub-steps 2282 to 2285 may be as shown in Table 3:

[0155]

[0156] It should be noted that the first preset reward value and the second preset reward value in the aforementioned embodiment can be set to 28000 or other values, which are not limited here.

[0157] Step 229, if the third data block information is not the first data block information stored in the evaluation queue to which the third data block information belongs, or the reward value of the third data block information is not equal to the preset initial reward value, then update the reward value in the third data block information according to the status information of the data acquisition request and the second reward value update strategy.

[0158] Reference Figure 7 , step 229 may include the following sub-steps:

[0159] Sub-step 2291, if the data acquisition request is immediate acquisition, and the status information of the data acquisition request is hit, then the reward value in the third data block information is updated to R3.

[0160] Wherein, R3>0, R3 represents the third preset reward value. For example, R3 can be set to 20000.

[0161] Sub-step 2292, if the data acquisition request is immediate acquisition and the status information of the data acquisition request is miss, then the reward value in the third data block information is updated to -R3.

[0162] If the acquisition request is an Acquire request, and its status information is a miss, the third data block information of the corresponding third data block is not first stored in the evaluation queue, or when it is about to be evicted, its reward value is not equal to the preset initial reward value, then its reward value is updated to a negative number. For example, its reward value is updated to -20000.

[0163] Sub-step 2293, if the data acquisition request is pre-acquisition and the status information of the data acquisition request is hit, then the reward value in the third data block information is updated to R4.

[0164] Wherein, R4>0, R4 represents the fourth preset reward value. The value of R4 may be 5000.

[0165] Sub-step 2294, if the data acquisition request is a pre-acquisition and the status information of the data acquisition request is a miss, then the reward value in the third data block information is updated to -R4.

[0166] The second reward value update strategy from sub-step 2291 to sub-step 2292 is further described below in conjunction with Table 4:

[0167]

[0168] It should be noted that the third preset reward value in the aforementioned embodiment can be set to 20000 or other values, and the second preset reward value can be set to 5000 or other values, which are not limited here. For example, an upper limit value of the reward value can be set, for example, it can be set to 42000, and the upper limit value is set as the initial value of the reward value. Based on the initial value, the training speed and training effect of the model can be improved.

[0169] Figure 8 Schematic diagram of the structure and signal flow of the L3 cache self-directory provided in the embodiment of the present application. In the embodiment of the present application, according to the requested program register value, page number, and Figure 8 The self-directory shown obtains the cache replacement strategy through reinforcement learning.

[0170] Reference Figure 8 In the self-directory of the L3 cache, there are three types of tables, namely, a first list for storing action value function values ​​corresponding to program register values, a second list for storing action value function values ​​corresponding to page numbers, and an evaluation queue (EQ) for storing data block information of some stored data blocks as samples. The first list can be marked as PC_Q value list (Q-value Table of PC), and the second list can be marked as PC_Q value list (Q-value Table of PN).

[0171] Furthermore, the evaluation list is used to record the data block information of a preset number (e.g., 32 times) of stored data blocks under the preset index number in the preset number of accesses (e.g., 32 times) closest to the current moment in the historical time period as the preset index sample of the index number. The first list and the second list are respectively used to record the values ​​of the action value function (Q-value) of the data block stored in the L3 cache corresponding to the PC or PN value. Based on Figure 8 From the directory shown, the method of the embodiment of the present application performs cache replacement prediction based on reinforcement learning through two parts: training and query.

[0172] Specifically, when the data block information in the evaluation queue exceeds 32, the first block information added will be expelled, and the program register value and page number of the data block information will be queried. According to the program register value and page number of the data block information, the action evaluation function value (Q-value) in the first list and the second list corresponding to the data block information will be updated. When a new data block is added to the L3 cache, it is necessary to query the first list and the second list to find the replacement priority corresponding to the largest action evaluation function value, and determine this replacement priority as the replacement priority of the data block in the cache.

[0173] In this embodiment, by collecting the historical access information of the program register value and page number corresponding to the data block in the cache, it is dynamically predicted whether a data block will be used in a preset time period in the future through the historical access information. The replacement priority of the data block is obtained, and the data block that may be used is determined according to the replacement priority, and it is driven out of the L3 cache, so that more useful data blocks are retained in the L3 cache as much as possible, thereby improving the cache hit rate, thereby improving the memory access speed of the processor.

[0174] For example, the method of this embodiment can be applied to the last level cache (L3 cache) of the preset processor. The L3 cache of the preset processor adopts a directory consistency protocol, and the cache adopts a TileLink bus protocol. The request to access L3 can be divided into 6 categories according to the request type and the hit situation: Acquire Hit, Acquire Miss, Hint Hit, Hint Miss, Release Hit, and Release Miss.

[0175] Among them, immediate acquisition (Acquire) can be divided into acquisition of data (AcquireBlock) and acquisition of permissions (AcquirePerm) in the specific protocol, and release (data release request) can be divided into release request (Release) and release of data (Release Data). In this embodiment, these classification information may not be processed differently.

[0176] In one embodiment, Figure 8 Both the first and second lists in the table have 8192 different pc / pn values. Each item stores the Q-Value corresponding to 9 different Actions. The 9 Actions are Action1 to Action9. Among them, Action1 is a hit and the replacement priority is 0; Action2 is a hit and the replacement priority is 1; Action3 is a hit and the replacement priority is 2; Action4 is a hit and the replacement priority is 3; Action5 is a miss and the replacement priority is 0; Action6 is a miss and the replacement priority is 1; Action7 is a miss and the replacement priority is 2; Action8 is a miss and the replacement priority is 3; Action9 is a miss and Bypass. Miss and Bypass means that this block is not stored in L3 and is released directly to the memory.

[0177] Figure 8 The evaluation list in includes an evaluation queue set for each sample in the sample set, and each evaluation queue has data block information of 32 stored data blocks. Among them, is a first-in-first-out queue, and each entry contains 7 parts of information: the program register value, page number, status information of this data release request, the target action information selected from the first list and the second list, the action value function value, and the reward value of the data block corresponding to the address of the data release request.

[0178] based on Figure 8As shown in the self-directory, the cache replacement method based on reinforcement learning in this embodiment may include two steps of training and querying. The following is a further exemplary description of these two steps.

[0179] The query method includes: sending a data release request to the L3 cache carrying a program register value and a page number at the same time, and after obtaining status information (hit or miss) of the data release request, sending the program register value, the page number and the status information to a first list and a second list to determine what replacement priority should be selected for this data block.

[0180] Specifically, the first list is indexed by the program register value, the second list is indexed by the page number value, and the range of the action information (Action) is determined by the status information (hit or miss). For example, if it is a hit, the action information determined is the first four action information of the 9 Actions listed in the above embodiment, and if it is a miss, the action information determined is the last five action information of the 9 Actions listed in the above embodiment, and then the action value evaluation function values ​​corresponding to these action information are compared to select the Action corresponding to the largest action value evaluation function value. Then, the action value evaluation function values ​​of the action information selected from the first list and the second list are compared, and the action information corresponding to the largest action value evaluation function value is determined as the target action information corresponding to the data release request, and the replacement priority in the target action information is determined as the replacement priority of the block of the data release request after it is stored in L3.

[0181] The training process may include the following method: selecting 64 index number samples from all indexes (eg, 8192 different indexes), and training accesses to the selected 64 index number samples.

[0182] Specifically, when a data release request corresponding to the index number sample comes to L3, the status information of the data release request is queried first. Then, in the evaluation list, at the front of the target evaluation queue corresponding to the index number of the data release request, a new database information is inserted, and the inserted data block information is the data block information corresponding to the data release request information.

[0183] If the data block information in the target evaluation queue is full of 32, the first inserted data block information will be evicted. When evicting, check the program register value, page number, action information, and reward value of the evicted data block information, and update the action value evaluation function value corresponding to the program register value and action information in the data block information in the first list according to this information, and update the action value evaluation function value corresponding to the page number and action information in the data block information in the second list.

[0184] For example, the method for obtaining the updated action value evaluation function value Q(new) is as follows:

[0185] The update formula is as follows (where Q(head) is the action value evaluation function value of the next entry to be evicted in this eq):

[0186] Q(new) = Q(old) + 50[reward + Q(head) / 3 - Q(old)]

[0187] Among them, Q(old) is the unupdated action-value function evaluation value in the first data block information, reward is the reward value in the first data block information, Q(head) is the second data block information stored after the first database information of the first database is stored in the target evaluation queue, and the data block corresponding to the second data block information can be recorded as the second data block.

[0188] When a data acquisition request (Acquire or Hint) of a certain index number sample comes to the L3 cache, after knowing the status information (hit / miss) of the data acquisition request, check whether there is an address corresponding to the data acquisition request in the evaluation list. If there is, according to the preset reward value update strategy described in the above embodiment, the reward value in the third data block information of the third data block with the same program register value, page number and status information as the data acquisition request in the evaluation queue with the same index as the data acquisition request is updated.

[0189] If a data block does not generate a reward value when it is evicted, and its reward value is the same as the preset reward value (for example, the preset reward value is 0), then update its reward value according to the following rules: if it hits and the action information is Action4, or misses and the action information is Action9, then update its reward value to 28000; if it hits and the action information is not Action4, or misses and the action information is not Action9, then update its reward value to -28000. Among them, if a data block does not generate a reward value when it is about to be evicted, it means that this address has not been accessed in the past period of time and is relatively useless. For such a data block, if it was assigned a high replacement priority when it was stored and is easy to be replaced, it means that the previous cache replacement decision was correct and a reward should be given, and its reward value is updated to a preset positive value (for example, 28000); on the contrary, if it is not assigned the highest priority, it means that the previous cache replacement decision is problematic and needs to be punished, and its reward value is updated to a preset negative value (for example, -28000) to correct the strategy).

[0190] Fig. 9A flowchart of a cache replacement method based on reinforcement learning provided in an embodiment of the present application is provided. Fig. 9 , the method may include the following steps:

[0191] Step S1, obtaining a data release request and the release data that the data release request needs to release to the L3 cache.

[0192] The data release request carries a target program register value, a target page number and a target index number.

[0193] Step S2: acquiring target status information of the data release request according to the data release request and the meta list.

[0194] For example, Figure 8 The meta list in includes the program register value and the page number of the stored data block. According to the target program register value, the target page number, and the meta list of the data release request, the target state information of the data release request is obtained.

[0195] Step S3, according to the target program register value and target page number in the data release request, and the target state information of the data release request, obtain the replacement priority corresponding to the target program register value from the first list, and obtain the replacement priority corresponding to the target page number from the second list.

[0196] Reference Figure 8 The first list includes at least one program register value and action information corresponding to each register program value, wherein one register program value corresponds to 9 types of action information, namely action information 0 to action information 8. The action information includes status information, replacement priority and action value function value. For details, please refer to Table 1, which will not be repeated here.

[0197] Reference Figure 8 The second list includes at least one page number and action information corresponding to each page number, wherein one page number corresponds to 9 types of action information, from action information 0 to action information 8, and the action information includes status information, replacement priority and action value function value. For details, please refer to Table 1, which will not be repeated here.

[0198] Step S4, obtaining target action information of the data release request from the replacement priority corresponding to the target program register value and the replacement priority corresponding to the target page number.

[0199] The data of the target data block is Figure 8 The method of obtaining the replacement priority of the target data block in this step has been described in the aforementioned steps 203 to 212, and will not be repeated here.

[0200] The target action information includes the replacement priority of the target data block, the target state information of the data release request, and the action value function value.

[0201] Step S5, storing the target priority in the target action information into the replacement memory, and storing the released data into the L3 cache.

[0202] Figure 8 The replacement memory is used to store the replacement priority of the data block stored in the L3 cache.

[0203] Step S6: If the target index number of the data release request is an index number in the preset sample set, then obtain a target evaluation queue in the evaluation list that has the same index number as the data release request.

[0204] The index number sample is the preset index number in the aforementioned embodiment.

[0205] The method of this step has been described in the aforementioned step 218 and will not be repeated here.

[0206] Step S7, determine whether the target evaluation queue is full, if yes, proceed to step S8, otherwise proceed to step S9.

[0207] The method of this step has been explained in the aforementioned step 219 and will not be repeated here.

[0208] Step S8: inserting the address of the data acquisition request, the target state information and the target action information into the target evaluation queue.

[0209] If the target evaluation queue is not full, the target state information and target action information can be directly inserted into the target evaluation queue.

[0210] Step S9, deleting the first data block information of the first data block stored first from the target evaluation queue, and then inserting the target state information and the target action information into the position where the first data block information is located in the target evaluation queue.

[0211] Reference Figure 8 The evaluation list includes 64 evaluation queues, one evaluation queue corresponds to one index number sample, and the index number samples corresponding to the 64 evaluation queues are index number sample 1 to index number sample 63. The evaluation queue corresponding to each index number sample includes data block information of 32 data blocks stored in sequence, data block 0 to data block 31. Among them, data block 0 is the first data block, and the data block information of data block 0 is the first data block information of the first data block.

[0212] Step S10, obtain the updated action value function value of the first data block according to the reward value of the first data block, the action value function value of the first data block, and the action value function value of the second data block, update the action value function value in the first data block information, and update the updated action value function value to the first list and the second list.

[0213] The method of this step has been explained in the aforementioned steps 220 to 221 and will not be repeated here.

[0214] Step S11, obtaining a data acquisition request. If the index number of the data acquisition request is an index number in the preset sample set, then obtaining the status information of the data acquisition request according to the data acquisition request and the meta list.

[0215] The method of this step has been described in the aforementioned step 226 and will not be repeated here.

[0216] Step S12: updating the action evaluation function value of the third data block information in the evaluation list according to the state information of the data acquisition request and the preset reward value update strategy.

[0217] The third data block is data block information having the same index number, status information, program register value and page number as the data acquisition request. The method of this step has been described in the aforementioned steps 208, 209, and the sub-steps of step 208 and the sub-steps of step 209, and will not be repeated here.

[0218] The method of this embodiment is applicable to cache architectures such as the Inclusive architecture, the Non-inclusive structure, and the Exclusive structure. The Inclusive structure indicates that the lower cache contains all the contents of the upper cache; the Non-inclusive structure indicates that the lower cache contains part of the contents of the upper cache; and the Exclusive structure indicates that the lower cache does not contain any contents of the upper cache.

[0219] In the related art, cache replacement algorithms such as LRU and SRRIP perform cache replacement based on a fixed replacement strategy, which is unchanged regardless of the type of program. This method lacks program adaptability and may be effective for some programs but completely ineffective for other programs. The method of this embodiment dynamically adjusts the replacement strategy according to different programs, and has a good adaptability to different programs. In the related art, although algorithms such as Dynamic Re-reference Interval Prediction (DRRIP) and Hawkeye have certain program adaptability, the adjustment strategy is too simple, relies on fewer parameters, is not reliable, and has poor results. This embodiment utilizes PC value and PN value, two parameters that have been proven to be very effective for cache replacement, combined with reinforcement learning strategy, to continuously generate reward values ​​for updating the action value function value, so as to modify the model according to the reward value, so that the model has a more powerful prediction ability.

[0220] In this embodiment, by using the two cache parameters of program register value and page number, and using reinforcement learning strategy, the cache replacement result is made more accurate and reliable, thereby affecting the processor performance in two aspects: the overall cache hit rate can be improved, thereby reducing the processor's memory access latency, thereby improving the processor's performance; even if the overall cache hit rate does not change much, by improving the hit rate of individual memory access requests in the critical path, the processor's memory access latency can be effectively reduced and the processor performance can be improved. In the presence of multiple prefetchers, in some SPEC programs that are sensitive to memory access, the method of this embodiment can significantly improve the cache hit rate and improve processor performance. Among them, the SPEC program is the main program used to test the performance of processor chips.

[0221] refer to Fig.10, which shows a cache replacement device provided by an embodiment of the present application, the cache replacement device 30 includes: a first acquisition module 301, used to respond to receiving a data release request, obtain the target program register value and the target page number in the data release request, the data release request is used to store the target data block in the target cache; a second acquisition module 302, used to obtain the replacement priority corresponding to the target program register value, and the replacement priority corresponding to the target page number, and obtain the replacement priority of the target data block from all replacement priorities; a third acquisition module 303, used to obtain a replacement priority list if the storage space of the target cache is full; the replacement priority list includes the replacement priority of at least one stored data block in the target cache; a first determination module 304, used to determine and delete the replacement data block from the stored data blocks according to the comparison results of all replacement priorities in the replacement priority list; a first storage module 305, used to store the target data block in the address corresponding to the replacement data block, and store the replacement priority of the target data block in the replacement priority list.

[0222] Optionally, the second acquisition module 302 includes: a first acquisition sub-module, used to acquire and replace the action value function value corresponding to the priority; a second acquisition sub-module, used to acquire the maximum first action value function value from the action value function value of the replacement priority corresponding to the target program register value; a third acquisition sub-module, used to acquire the maximum second action value function value from the action value function value of the replacement priority corresponding to the target page number; a fourth acquisition sub-module, used to compare the first action value function value and the second action value function value to obtain the maximum action value function value, and determine the replacement priority corresponding to the maximum action value function value as the replacement priority of the target data block.

[0223] Optionally, the second acquisition module 302 includes: a fifth acquisition sub-module, used to acquire target status information of a data release request; the target status information includes: hit, miss, or storing the target data block in memory; a sixth acquisition sub-module, used to acquire a first list; the first list includes multiple program register values, and a program register value has at least one corresponding preset first action information, and the action information includes status information and replacement priority; a seventh acquisition sub-module, used to acquire at least one first action information corresponding to the target program register value from the first list, and acquire target first action information having the same status information and target status information from at least one first action information corresponding to the target program register value; a first determination sub-module, used to determine the replacement priority in the target first action information as the replacement priority corresponding to the target program register value.

[0224] Optionally, the second acquisition module 302 includes: an eighth acquisition sub-module, used to obtain target status information of the data release request; the target status information includes: hit, miss, or storing the target data block in the memory; a ninth acquisition sub-module, used to obtain the second list; the second list includes multiple page numbers, one page number has at least one corresponding preset second action information, and the second action information includes status information and replacement priority; the tenth acquisition sub-module, used to obtain at least one second action information corresponding to the target page number from the second list, and obtain target second action information with the same status information and target status information from at least one second action information corresponding to the target page number; a second determination sub-module, used to determine the replacement priority in the target second action information as the replacement priority corresponding to the target page number.

[0225] Optionally, the device 30 method also includes: a fourth acquisition module, used to obtain an evaluation list; the evaluation list includes at least one evaluation queue, one evaluation queue corresponds to a preset index number, and the evaluation queue is used to store data block information of a stored data block stored in the preset index number; the data block information includes the program register value, page number, replacement priority, status information, and action value function value of the data block; a fifth acquisition module, used to obtain the target index number and target status information of the data release request, and the data block information of the target data block; the target status information includes: hit, miss, or storing the target data block in the memory; a sixth acquisition module, used to obtain a target evaluation queue with the same preset index number and target index number from the evaluation list when the target index number is the same as any preset index number; a first storage module, used to store the data block information of the target data block into the target evaluation queue.

[0226] Optionally, the device 30 also includes: a seventh acquisition module, used to obtain the first data block information of the first data block first stored in the target evaluation queue before storing the data block information of the target data block into the target evaluation queue if the storage space in the target evaluation queue is full; and a first deletion module, used to delete the first data block information of the first data block from the target evaluation queue.

[0227] Optionally, the data block information also includes a reward value; the device 30 also includes: an eighth acquisition module, which is used to obtain the second data block information of the second data block from the target evaluation queue after obtaining the data block information of the first data block first stored in the target evaluation queue, and extract the action value function value in the second data block information; wherein the second data block information is the data block information stored in the target evaluation queue after the first data block information; a ninth acquisition module, which is used to obtain the updated action value function value of the first data block according to the reward value of the first data block, the action value function value of the first data block, and the action value function value of the second data block; a first replacement module, which is used to replace the action value function value corresponding to the program register value, status information and replacement priority of the first data block in the first list with the updated action value function value of the first data block; a second replacement module, which is used to replace the action value function value corresponding to the page number, status information and replacement priority of the first data block in the second list with the updated action value function value of the first data block.

[0228] Optionally, the device 30 also includes: a tenth acquisition module, which is used to obtain the index number, program register value, page number and status information of the data acquisition request in response to receiving the data acquisition request; an eleventh acquisition module, which is used to obtain the third data block information that is the same as the index number, program register value, page number and status information of the data acquisition request from the evaluation list if the index number of the data acquisition request is the same as any preset index number; a first update module, which is used to update the reward value in the third data block information according to the status information of the data acquisition request and the first reward value update strategy if the third data block information is the data block information first stored in the evaluation queue to which the third data block information belongs, and the reward value of the third data block information is not equal to the preset initial reward value; a second update module, which is used to update the reward value in the third data block information according to the status information of the data acquisition request and the second reward value update strategy if the third data block information is not the data block information first stored in the evaluation queue to which the third data block information belongs, or the reward value of the third data block information is not equal to the preset initial reward value.

[0229] Optionally, the first update module includes: an eleventh acquisition submodule, which is used to acquire, in the preset action information, a first replacement priority of action information in which all state information is hit, and a second replacement priority of action information in which all state information is not hit; a first update submodule, which is used to update the reward value in the third data block information to R1 if the state information of the data acquisition request is a hit and the replacement priority of the third data block is equal to the maximum value of the first replacement priority; R1>0, R1 represents the first preset reward value; a second update submodule, which is used to update the reward value in the third data block information to R1 if the state information of the data acquisition request is a hit and the replacement priority of the third data block is equal to the maximum value of the first replacement priority; R1>0, R1 represents the first preset reward value; If the replacement priority is not equal to the maximum value in the first replacement priority, the reward value in the third data block information is updated to -R1; the third update submodule is used to update the reward value in the third data block information to R2 if the status information of the data acquisition request is a miss and the replacement priority of the third data block is equal to the maximum value in the second replacement priority; R2>0, R2 represents the second preset reward value; the fourth update submodule is used to update the reward value in the third data block information to -R2 if the status information of the data acquisition request is a miss and the replacement priority of the third data block is not equal to the maximum value in the second replacement priority.

[0230] Optionally, the first update module includes: a fifth update sub-module, which is used to update the reward value in the third data block information to R3 if the data acquisition request is an immediate acquisition and the status information of the data acquisition request is a hit; R3>0, R3 represents a third preset reward value; a sixth update sub-module, which is used to update the reward value in the third data block information to -R3 if the data acquisition request is an immediate acquisition and the status information of the data acquisition request is a miss; a seventh update sub-module, which is used to update the reward value in the third data block information to R4 if the data acquisition request is a pre-acquisition and the status information of the data acquisition request is a hit; R4>0, R4 represents a fourth preset reward value; an eighth update sub-module, which is used to update the reward value in the third data block information to -R4 if the data acquisition request is a pre-acquisition and the status information of the data acquisition request is a miss.

[0231] Optionally, the device 30 also includes: a twelfth acquisition module, used to obtain a meta-list; the meta-list includes the program register value and page number of the data block stored in the target cache; a thirteenth acquisition module, used to determine the target status information of the data release request based on the target program register value, the target page number, and the meta-list.

[0232] According to the target PC value and target PN value of the data release request, the replacement priority corresponding to the target data block to be released by the data release request is dynamically obtained. When the target data block is stored in the address corresponding to the replacement data block, the replacement priority of the target data block is stored in the replacement priority list, thereby realizing the dynamic update of the replacement priority list. Based on the dynamically updated replacement priority list, it is possible to adaptively determine the replacement data block adapted to the current program during the program running process, thereby improving the cache replacement accuracy. The method of this embodiment predicts the replacement data block through the target PC value and target PN value of the data release request. During the program running process, it can adaptively select the best cache replacement strategy during the running process according to the program's own memory access characteristics, thereby achieving the purpose of accurately obtaining replacement data blocks for different programs. The cache replacement method based on reinforcement learning of this embodiment has a good cache replacement effect for different programs.

[0233] Fig.11 4 is a block diagram of an electronic device 400 according to an exemplary embodiment. For example, the electronic device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc. Fig.11 , the electronic device 400 may include one or more of the following components: a processing component 402, a memory 404, a power component 406, a multimedia component 408, an audio component 410, an input / output (I / O) interface 412, a sensor component 414, and a communication component 416. The processing component 402 generally controls the overall operation of the electronic device 400. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above method. The memory 404 is used to store various types of data to support the operation of the electronic device 400. The power component 406 provides power to various components of the electronic device 400. The multimedia component 408 includes a screen that provides an output interface between the electronic device 400 and the user.

[0234] The audio component 410 is used to output and / or input audio signals. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. The I / O interface 412 provides an interface between the processing component 402 and the peripheral interface module. The sensor component 414 includes one or more sensors for providing various aspects of state evaluation for the electronic device 400. The communication component 416 is used to facilitate the wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 can access a wireless network based on a communication standard. The electronic device 400 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to implement a cache replacement method based on reinforcement learning provided in an embodiment of the present application.

[0235] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the above instructions can be executed by a processor 420 of an electronic device 400 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0236] Fig.12 is a block diagram of an electronic device 500 according to an exemplary embodiment. For example, the electronic device 500 may be provided as a server. Fig.12 , the electronic device 500 includes a processing component 522, which further includes one or more processors, and a memory resource represented by a memory 532 for storing instructions that can be executed by the processing component 522. The application stored in the memory 532 may include one or more modules, each of which corresponds to a set of instructions. In addition, the processing component 522 is configured to execute instructions to execute a cache replacement method based on reinforcement learning provided in an embodiment of the present application. The electronic device 500 may also include a power supply component 526 configured to perform power management of the electronic device 500, a wired or wireless network interface 550 configured to connect the electronic device 500 to a network, and an input / output (I / O) interface 555.

[0237] An embodiment of the present application also provides a computer program product, including a computer program, which implements a cache replacement method based on reinforcement learning when the computer program is executed by a processor.

[0238] Those skilled in the art will easily think of other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes the common knowledge or customary technical means in the technical field that are not disclosed in the present application. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present application are indicated by the following claims. It should be understood that the present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is limited only by the attached claims. The above is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0239] The above is a detailed introduction to the cache replacement method, device, electronic device and medium based on reinforcement learning provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A cache replacement method based on reinforcement learning, characterized in that: include: In response to receiving a data release request, obtaining a target program register value and a target page number in the data release request, wherein the data release request is used to store a target data block into a target cache; Acquire a replacement priority corresponding to the target program register value and a replacement priority corresponding to the target page number, and dynamically acquire a replacement priority corresponding to a target data block to be released by the data release request according to the target program register value and the target page number of the data release request; If the storage space of the target cache is full, obtaining a replacement priority list; the replacement priority list includes a replacement priority of at least one stored data block in the target cache; Determining and deleting a replacement data block from the stored data blocks according to a comparison result of all replacement priorities in the replacement priority list; The target data block is stored in an address corresponding to the replacement data block, and the replacement priority of the target data block is stored in the replacement priority list.

2. The method according to claim 1, characterized in that Obtaining the replacement priority of the target data block from all replacement priorities includes: Get and replace the action value function value corresponding to the priority; Obtaining a maximum first action-value function value from the action-value function values ​​of the replacement priority corresponding to the register value of the target program; Obtaining the maximum second action value function value from the action value function values ​​of the replacement priority corresponding to the target page number; The first action value function value and the second action value function value are compared to obtain a maximum action value function value, and the replacement priority corresponding to the maximum action value function value is determined as the replacement priority of the target data block.

3. The method according to claim 1, characterized in that The obtaining of the replacement priority corresponding to the target program register value includes: Obtaining target status information of the data release request; the target status information includes: hit, miss, or storing the target data block into the memory; Acquire a first list; the first list includes a plurality of program register values, each of the program register values ​​has corresponding at least one preset first action information, the action information includes state information and replacement priority; Acquire at least one first action information corresponding to the target program register value from the first list, and acquire target first action information having the same state information as the target state information from the at least one first action information corresponding to the target program register value; The replacement priority in the target first action information is determined as the replacement priority corresponding to the register value of the target program.

4. The method according to claim 1, characterized in that: Obtaining a replacement priority corresponding to the target page number, including: Obtaining target status information of the data release request; the target status information includes: hit, miss, or storing the target data block into the memory; Acquire a second list; the second list includes a plurality of page numbers, each of the page numbers has corresponding at least one preset second action information, the second action information includes state information and replacement priority; Acquire at least one second action information corresponding to the target page number from the second list, and acquire target second action information having the same state information as the target state information from the at least one second action information corresponding to the target page number; The replacement priority in the target second action information is determined as the replacement priority corresponding to the target page number.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining an evaluation list; the evaluation list includes at least one evaluation queue, one evaluation queue corresponds to a preset index number, and the evaluation queue is used to store data block information of a stored data block stored in the preset index number; the data block information includes a program register value, a page number, a replacement priority, a state information, and an action value function value of the data block; Acquire the target index number and target status information of the data release request, and data block information of the target data block; the target status information includes: hit, miss, or storing the target data block into memory; When the target index number is the same as any preset index number, obtaining a target evaluation queue having the same preset index number as the target index number from the evaluation list; The data block information of the target data block is stored in the target evaluation queue.

6. The method according to claim 5, characterized in that Before storing the data block information of the target data block into the target evaluation queue, the method further includes: If the storage space in the target evaluation queue is full, obtaining the first data block information of the first data block first stored in the target evaluation queue; The first data block information of the first data block is deleted from the target evaluation queue.

7. The method according to claim 6, characterized in that The data block information also includes a reward value; after obtaining the first data block information of the first data block first stored in the target evaluation queue, it also includes: Obtaining second data block information of the second data block from the target evaluation queue, and extracting the action value function value in the second data block information; wherein the second data block information is data block information stored in the target evaluation queue after the first data block information; Obtaining an updated action-value function value of the first data block according to the reward value of the first data block, the action-value function value of the first data block, and the action-value function value of the second data block; Using the updated action value function value of the first data block, replacing the action value function value corresponding to the program register value, state information and replacement priority of the first data block in the first list; The update action value function value of the first data block is used to replace the action value function value corresponding to the page number, status information and replacement priority of the first data block in the second list.

8. The method according to claim 7, characterized in that The method further comprises: In response to receiving a data acquisition request, acquiring an index number, a program register value, a page number, and status information of the data acquisition request; If the index number of the data acquisition request is the same as any preset index number, obtaining third data block information having the same index number, program register value, page number and status information as the data acquisition request from the evaluation list; If the third data block information is the data block information first stored in the evaluation queue to which the third data block information belongs, and the reward value of the third data block information is not equal to the preset initial reward value, then updating the reward value in the third data block information according to the status information of the data acquisition request and the first reward value update strategy; If the third data block information is not the data block information first stored in the evaluation queue to which the third data block information belongs, or the reward value of the third data block information is not equal to the preset initial reward value, the reward value in the third data block information is updated according to the status information of the data acquisition request and the second reward value update strategy.

9. A cache replacement device based on reinforcement learning, characterized in that: The device comprises: A first acquisition module, configured to, in response to receiving a data release request, acquire a target program register value and a target page number in the data release request, wherein the data release request is used to store a target data block into a target cache; A second acquisition module is used to acquire a replacement priority corresponding to the target program register value and a replacement priority corresponding to the target page number, and dynamically acquire a replacement priority corresponding to a target data block to be released by the data release request according to the target program register value and the target page number of the data release request; A third acquisition module, configured to acquire a replacement priority list if the storage space of the target cache is full; the replacement priority list includes a replacement priority of at least one stored data block in the target cache; A first determining module, configured to determine and delete a replacement data block from the stored data blocks according to a comparison result of all replacement priorities in the replacement priority list; The first storing module is used to store the target data block into an address corresponding to the replacement data block, and store the replacement priority of the target data block into the replacement priority list.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 8.

Images