Delay cache replacement method and device, equipment, storage medium and product

Abstract

The invention discloses a delayed cache replacement method and device, equipment, a storage medium and a product, and relates to the technical field of network transmission, and the method comprises the steps: determining a cache execution time node based on a delayed cache strategy; based on the value joint standard mathematical model, determining the content value of the cache content; determining a plurality of expelling candidate sets based on the content value of the cached content; inputting the plurality of expelling candidate sets into a cache decision model to obtain output target expelling content; and based on the cache execution time node, deleting the target eviction content from the cache content, and caching the corresponding missing content in the request information of the client. Through the mode, the delay cache model is provided, the time for acquiring data in the distributed system is considered, the cache accuracy is improved, the joint standard model is provided, a unified measurement standard is set for overall access and exit, the unification of an access strategy standard and a cache replacement strategy standard is ensured, and the cache efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of network transmission technology, and in particular to a delayed cache replacement method, device, equipment, storage medium and product. Background Art

[0002] Currently, due to the lack of unified metrics for cache admission and cache replacement, low-value content may be cached in the cache space of distributed clients, and low-value content is constantly replaced with low-value content in the cache space, resulting in frequent eviction of content, which in turn affects cache performance. Assuming that the future access potential of requested content and cached content is evaluated using different criteria, this means that there may be allowed content with a lower actual access probability than the evicted content, or non-allowed content with a higher actual access probability than the content cached in the distributed server, which often leads to problems of repeated access and deletion of demand content, affecting cache efficiency.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide a delayed cache replacement method, device, equipment, storage medium and product, aiming to solve the technical problem of cache inefficiency caused by inconsistent value measurement standards for cache access and exit in the prior art.

[0005] To achieve the above object, the present application provides a delayed cache replacement method, the method comprising:

[0006] Based on the delayed cache strategy, determine the cache execution time node;

[0007] Determine the content value of cached content based on a value-joint standard mathematical model;

[0008] Determining a plurality of eviction candidate sets based on the content value of the cached content;

[0009] Inputting the multiple eviction candidate sets into a cache decision model to obtain output target eviction content;

[0010] Based on the cache execution time node, the target evicted content is deleted from the cache content, and the corresponding missing content in the request information of the client is cached.

[0011] In one embodiment, the step of determining the content value of the cached content based on the value-joint standard mathematical model includes:

[0012] Based on the value joint standard mathematical model, determining a first corresponding relationship between the frequency of cached content, the size of cached content and the value of the content;

[0013] Based on the frequency of cached content in the sliding time window, the size of the cached content, and the first corresponding relationship, the content value of the cached content is obtained.

[0014] In one embodiment, the step of determining a plurality of eviction candidate sets based on the content value of the cached content includes:

[0015] Based on the content value of the cached content, the cached content is divided into a high-value cached content set and a low-value cached content set;

[0016] A plurality of groups of low-value cache content are selected from the low-value cache content set to generate a plurality of eviction candidate sets, and missing content corresponding to the eviction candidate sets is determined in the request information, wherein the number of low-value cache content in the eviction candidate set is greater than or equal to the number of missing content corresponding to the eviction candidate set.

[0017] In one embodiment, the step of inputting the multiple eviction candidate sets into a cache decision model to obtain output target eviction content includes:

[0018] Determine a second correspondence between the frequency of missing content, the size of missing content and the value of content based on the value joint standard mathematical model;

[0019] Obtaining the content value of the missing content based on the frequency of missing content within the sliding time window, the size of the missing content, and the second corresponding relationship;

[0020] Determine a cache state corresponding to the eviction candidate set based on the content value of the missing content corresponding to the eviction candidate set and the content value of the low-value cached content in the eviction candidate set;

[0021] Based on the cache state corresponding to the eviction candidate set and the policy function of the cache decision model, obtaining the cache action corresponding to the eviction candidate set;

[0022] Based on the cache action corresponding to the eviction candidate set, a target eviction candidate set is determined, and the low-value cache content in the target eviction candidate set is used as the target eviction content.

[0023] In one embodiment, the method further comprises:

[0024] Determining a reward function of the cache decision model based on a byte hit rate within a preset period;

[0025] Based on the cache state, determining a state value function;

[0026] Determining an action-value function based on the state-value function and the reward function;

[0027] Determining an action advantage function based on the action value function and the state value function;

[0028] Based on the action advantage function, updating the global network parameters of the cache decision model within the preset period;

[0029] Based on the updated global network parameters, the policy function of the cache decision model is updated.

[0030] In one embodiment, the step of determining the cache execution time node based on the delayed cache strategy includes:

[0031] Based on the time sequence of the request information from the client, the request information is sorted to generate a request queue;

[0032] Obtaining the critical time of the request queue, wherein the critical time at least includes the data request time, the request generation time period, and the last frame reception time;

[0033] The last frame receiving time is used as the cache execution time node under the delayed cache strategy.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a delay cache replacement device, the delay cache replacement device comprising:

[0035] The delayed cache module is used to determine the cache execution time node based on the delayed cache strategy;

[0036] An eviction decision module, for determining the content value of cached content based on a value-joint standard mathematical model;

[0037] The eviction decision module is further configured to determine a plurality of eviction candidate sets based on the content value of the cached content;

[0038] The eviction decision module is further used to input the multiple eviction candidate sets into the cache decision model to obtain output target eviction content;

[0039] A cache replacement module is used to delete the target evicted content from the cache content based on the cache execution time node, and cache the corresponding missing content in the request information of the client.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a delayed cache replacement device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the delayed cache replacement method as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the delayed cache replacement method described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the delayed cache replacement method described above are implemented.

[0043] The present application provides a delayed cache replacement method, which determines the cache execution time node based on the delayed cache strategy; determines the content value of the cache content based on the value joint standard mathematical model; determines multiple eviction candidate sets based on the content value of the cache content; inputs multiple eviction candidate sets into the cache decision model to obtain the output target eviction content; based on the cache execution time node, deletes the target eviction content from the cache content, and caches the corresponding missing content in the client's request information. The present application proposes a delayed cache model, which takes into account the time to obtain data in a distributed system, utilizes the cache information in the acquisition gap, improves the cache accuracy, and proposes a joint standard model to set a unified value measurement standard for overall access and access, improves cache efficiency, and solves the technical problem of cache inefficiency caused by inconsistent value measurement standards for cache access and access. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 This is a flow chart of Embodiment 1 of the delayed cache replacement method of the present application;

[0047] Figure 2 A schematic diagram of a request queue of the delayed cache replacement method provided in Embodiment 2 of the present application;

[0048] Figure 3 This is a flow chart of Embodiment 2 of the delayed cache replacement method of the present application;

[0049] Figure 4 This is a schematic diagram of the module structure of the delayed cache replacement device according to an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the device structure of the hardware operating environment involved in the delayed cache replacement method in the embodiment of the present application.

[0051] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of the embodiment of the present application is: based on the delayed caching strategy, determining the cache execution time node; based on the value joint standard mathematical model, determining the content value of the cache content; based on the content value of the cache content, determining multiple eviction candidate sets; inputting the multiple eviction candidate sets into the cache decision model to obtain the output target eviction content; based on the cache execution time node, deleting the target eviction content from the cache content, and caching the corresponding missing content in the client's request information.

[0055] The present application provides a solution and proposes a delayed caching model, which takes the time for acquiring data in a distributed system into consideration, utilizes the cache information in the acquisition gap, improves the cache accuracy, and proposes a joint standard model to set a unified value measurement standard for overall access and exit, thereby improving cache efficiency and solving the technical problem of cache inefficiency caused by inconsistent value measurement standards for cache access and exit.

[0056] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a delay cache replacement device, etc., and this embodiment does not specifically limit this. The following takes the delay cache replacement device as an example to illustrate this embodiment and the following embodiments.

[0057] The present application embodiment provides a delayed cache replacement method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the delayed cache replacement method of the present application.

[0058] In this embodiment, the delayed cache replacement method includes steps S10 to S50:

[0059] Step S10, determining a cache execution time node based on a delayed cache strategy;

[0060] It should be noted that the client in this embodiment refers to a distributed client. The distributed client obtains the missing content from each data node, and in the process of obtaining it, it will also receive multiple data requests from the user. This embodiment fully considers the information carried by the request content received in this gap. Based on this feature, a delayed caching method is adopted to utilize the cache information in the acquisition gap to improve the cache efficiency and accuracy. The delayed caching strategy is the delayed caching method adopted in this embodiment.

[0061] In a feasible implementation, step S10 may include steps S101 to S103:

[0062] Step S101, sorting the request information based on the time sequence of the client's request information to generate a request queue;

[0063] It should be noted that the request information refers to a large amount of request information generated by the client during the data acquisition period, and the time sequence of the request information refers to the order in which the request information is generated.

[0064] It can be understood that the time sequence of all request information is obtained, and the request information is arranged in time sequence into a request queue, usually arranging the request information generated earlier in front and the request information generated later in the back.

[0065] Step S102, obtaining the key time of the request queue, wherein the key time at least includes the data request time, the request generation time period and the last frame reception time;

[0066] It should be noted that the key time refers to the required relevant time information, including at least the data request time, the request generation time period and the last frame reception time. The data request time is the time point when the data in the request data block is issued, the request generation time period is the time period for generating the request, and the last frame reception time is the time point when the last frame of data in the data block is received.

[0067] Step S103: taking the last frame receiving time as the cache execution time node under the delayed cache strategy.

[0068] It is understandable that a large amount of request information generated by the client during data acquisition usually needs to be marked so as to subsequently determine the cache execution time node. Figure 2 Schematic diagram of the request queue. In the request queue, the distributed client sends a request for data block C at time t1, and the last frame of data block C is received at time t2. During this period, within the time period T, the client generates a large number of data requests (A, B).

[0069] It should be noted that the cache execution time node is the time node for executing the cache strategy. The cache strategy refers to the cache replacement strategy, which evicts low-value content in the cache and stores other higher-value content.

[0070] It is understandable that when to choose to evict cached content affects the efficiency of the cache. If the delay time node is too late, the client may have obtained the required data, but there is no available space in the local cache to cache it; if the delay time node is too early, the information in the request queue is not obtained completely, resulting in the decision that the data to be evicted may be requested again. Therefore, in order to avoid such problems, the local cache is evicted when the last frame of the requested content arrives locally, that is, the cache replacement strategy is executed at the moment the last frame is received. Refer to Figure 2 , set the cache execution time node at t2, save the data request information before t2, and execute the cache replacement strategy based on the saved information.

[0071] In the specific implementation, the request information generated by the distributed client is arranged in chronological order as a request queue, and the last frame reception time is found as the cache execution time node to realize delayed caching.

[0072] Step S20, determining the content value of the cached content based on the value joint standard mathematical model;

[0073] It should be noted that, since the metrics of cache admission and cache replacement are usually not unified, low-value content may be constantly replaced with low-value content in the client's cache space, resulting in frequent content eviction. Therefore, this embodiment proposes a value joint standard mathematical model to unify the value metrics of cache admission and cache removal, which can effectively ensure that the content cached in the distributed server reaches a higher future access value.

[0074] In addition, it should be noted that the future access value of cached content is usually related to three key factors: the size, frequency, and timeliness of the content. The most intuitive attribute that reflects the future access value is the access frequency of historical accesses. Generally speaking, the higher the frequency of access to each content does not mean that its value is higher. For the same access frequency, smaller content has higher value because smaller content occupies less space. Therefore, the single-byte access frequency can be used to reflect the content access value. In addition, the continuous accumulation of the frequency of each content will lead to an increase in the frequency of the content, which will lead to a misjudgment of the frequency of the content. Therefore, a certain time span and request span need to be guaranteed.

[0075] It is understandable that a sliding window based on the cycle time δt can be added to calculate the access frequency to ensure that the value of the cached content is available. The unified standards are:

[0076]

[0077] In the formula, a represents the coefficient of variation, τ i Indicates the size of content i, represents the total size of the content in the request information during the period from δt to t, F t-δt Indicates the total number of requests, F i,f represents the number of accesses to content i in the f-th request information during the period from t-δt to t. If the f-th request is not for content i, then F i,f When the number of requests is small, the user sends fewer requests during the acquisition period. When the number of requests is small, the value of the requested content is very close, and it is difficult to separate high-value content from cached content. Therefore, this embodiment optimizes the traditional calculation method, calculates the content value based on the request frequency of the content and the size of the content, and constructs a value joint standard mathematical model. For example, the request frequency of the content (within the sliding window δt) is divided by the size of the content, which can easily distinguish high-value content from low-value content. At this time, the value of the content is determined only by its size and frequency, and is not affected by the number of requests from other content.

[0078] In a feasible implementation, step S20 may include: determining a first correspondence between the frequency of cached content, the size of cached content and the content value based on the value joint standard mathematical model; and obtaining the content value of the cached content based on the frequency of cached content within a sliding time window, the size of the cached content and the first correspondence.

[0079] It should be noted that the cached content is the content currently in the cache. The first correspondence between the frequency of the cached content, the size of the cached content and the content value, i.e., the calculation relationship of the content value of the cached content, is determined according to the value joint standard mathematical model, as shown below:

[0080]

[0081] In the formula, represents the content value of cached content i, i represents the frequency of cached content i in the sliding window δt, τ i Represents the size of cache content i. Substituting the relevant data into the above first corresponding relationship, the content value of all cached contents can be calculated.

[0082] Step S30, determining a plurality of eviction candidate sets based on the content value of the cached content;

[0083] It should be noted that the eviction candidate set is a set of content that may be evicted. Since the essence of the cache replacement strategy is to select low-value content as evicted content by comparing the value of cached content in distributed servers, the eviction candidate set is usually generated based on cached content with lower value.

[0084] In a feasible implementation, step S30 may include steps S301 to S303:

[0085] Step S301, dividing the cache content into a high-value cache content set and a low-value cache content set based on the content value of the cache content;

[0086] It should be noted that the high-value cache content set is a collection of high-value cache content, and the high-value cache content is the high-value content in the cache content. The low-value cache content set is a collection of low-value cache content, and the low-value cache content is the low-value content in the cache content. High-value cache content and low-value cache content can be judged by setting a value threshold. The cache content with a content value greater than or equal to the value threshold is the high-value cache content, and the cache content with a content value less than the value threshold is the low-value cache content.

[0087] It is understandable that, according to the content value of the cached content, all cached content is divided into high-value cached content and low-value cached content, all high-value cached content constitutes a high-value cached content set, and all low-value cached content constitutes a low-value cached content set.

[0088] Step S302, selecting multiple groups of low-value cache content from the low-value cache content set, generating multiple eviction candidate sets, and determining the missing content corresponding to the eviction candidate sets in the request information, wherein the number of low-value cache content in the eviction candidate set is greater than or equal to the number of missing content corresponding to the eviction candidate set.

[0089] It should be noted that multiple groups of low-value cache content are selected from the low-value cache content set, and each group of low-value cache content is used as an eviction candidate set. The missing content corresponding to the eviction candidate set is the content requested by the client that is missing in the current cache content after the content in the eviction candidate set is evicted. Generally speaking, the number of low-value cache content in the eviction candidate set needs to be greater than or equal to the number of missing content corresponding to the eviction candidate set.

[0090] It is understandable that since the access value of each content needs to be recalculated each time it is compared, and the two calculations may result in the same value, it is easy to affect the calculation speed and reduce the selection efficiency of the expelled content. In addition, since high-value content is far more valuable than other content at certain times, whether its value is updated in real time or not has no effect on the selection of the expelled content, because. Therefore, in order to improve the calculation speed and selection efficiency of the expelled content, this embodiment divides all cached content in the storage space into two groups (set1 and set2) according to the content value. Set1 is a low-value cache content set, which contains content with lower value and occupies ζ% of all content, and set2 is a high-value cache content set, which contains the remaining content with higher value. Set1 is usually used to set the expulsion candidate set. When selecting expelled content, only the value of the content in the expulsion candidate set needs to be calculated each time, which can significantly reduce the amount of calculation, and the range of expulsion options is much smaller. However, since the requested content is dynamic, the value of the content is usually time-sensitive. In order to ensure that the low-value content is in the expulsion candidate set, it is necessary to regularly update the two sets of set1 and set2, and in order to avoid frequent expulsion of the content entering the cache, the content entering the cache is stored in the high-value cache content set set2.

[0091] Step S40, inputting the multiple eviction candidate sets into a cache decision model to obtain output target eviction content;

[0092] It should be noted that the target eviction content refers to the content that is finally decided to be evicted. In this embodiment, multiple eviction candidate sets are set, and a set that needs to be evicted is finally directly decided, and all the content in the set needs to be evicted. The cache decision model is a model constructed in this embodiment for deciding the target eviction content. Its input is all the eviction candidate sets determined in step S30, and its output is the target eviction content. It can be obtained by training using a reinforcement learning algorithm, and no specific limitation is made to this.

[0093] Step S50: based on the cache execution time node, the target evicted content is deleted from the cache content, and the corresponding missing content in the request information of the client is cached.

[0094] It is understandable that the missing content corresponding to the client's request information is the content requested by the client that is missing from the current cache content. The cache replacement strategy is executed according to the cache execution time node, the target expelled content is deleted, and the corresponding missing content is cached.

[0095] It should be understood that when determining the target expulsion content, it is also necessary to consider the value of the missing content after deleting the target expulsion content. The value of the missing content also needs to be calculated according to the value joint standard mathematical model to ensure that admission and exit have a unified value measurement standard.

[0096] This embodiment provides a delayed cache replacement method, which determines the cache execution time node based on the delayed cache strategy; determines the content value of the cache content based on the value joint standard mathematical model; determines multiple eviction candidate sets based on the content value of the cache content; inputs multiple eviction candidate sets into the cache decision model to obtain the output target eviction content; based on the cache execution time node, deletes the target eviction content from the cache content, and caches the corresponding missing content in the client's request information. A delayed cache model is proposed, which takes into account the time of obtaining data in a distributed system, utilizes the cache information in the acquisition gap, improves the cache accuracy, and proposes a joint standard model to set a unified value measurement standard for the overall access and access, thereby improving cache efficiency.

[0097] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , the step S40 may include steps S401 to S403:

[0098] Step S401, based on the value joint standard mathematical model, determining a second correspondence between the frequency of missing content, the size of missing content and the content value, and obtaining the content value of the missing content based on the frequency of missing content in the sliding time window, the size of the missing content and the second correspondence;

[0099] It should be noted that the second corresponding relationship between the frequency of missing content, the size of missing content and the content value, i.e., the calculation relationship of the content value of missing content, is determined according to the value joint standard mathematical model, as shown below:

[0100]

[0101] In the formula, represents the content value of missing content i, i represents the frequency of missing content i within the sliding window δt, τ i Represents the size of missing content i. Substituting the relevant data into the second corresponding relationship above, the content value of all missing content can be calculated.

[0102] Step S402, determining a cache state corresponding to the eviction candidate set based on the content value of the missing content corresponding to the eviction candidate set and the content value of the low-value cached content in the eviction candidate set;

[0103] It should be noted that the delayed cache model and the value joint standard mathematical model can greatly improve the performance of the existing cache replacement strategy, and can solve the problems of low flexibility and low accuracy of the traditional cache replacement strategy. In order to make more effective use of these two models, this embodiment proposes an overall framework that can integrate the two models, which includes four parts: initialization, information collection, cache decision and model update. Initialization: In the initialization stage, LRU is used as the eviction strategy, and all request contents in the distributed server are allowed to initialize the global network and establish a local network, wherein the selection probability of each action is equal at the time of initialization. Information collection: When the request queue length recorded by the server is sufficient to meet the two sliding time windows, the request information is collected during the data acquisition period and attached to the historical request information, and the historical request information is divided into two groups of high value and low value, and the low value set is selected as the cache content eviction candidate set. Cache decision: Based on the information collection stage, the eviction candidate set obtained is used as the input content of the cache decision model, and the cache decision model is executed to make a cache decision, and the eviction content decided by the model is returned, and then the content to be evicted is selected and deleted. Model update: To maintain the validity of the model, the global network parameters of the model are updated at intervals of T seconds.

[0104] In addition, it should be noted that this embodiment uses the A3C (Asynchronous Advantage Actor-Critic) model as the main component of the cache decision model. Compared with the traditional reinforcement learning cache strategy, the A3C model used in this embodiment can introduce Actor-Critic into multiple threads for simultaneous training, with high training efficiency, and is suitable for selecting eviction content. In addition, based on the value-joint standard mathematical model, the state space of the cache strategy is reasonably quantified, and a reward function based on the hit rate is separately designed, which improves the learning ability of the algorithm itself and improves the overall performance of the model.

[0105] It can be understood that A3C is a reinforcement learning algorithm, including state space, action space, reward, etc. State space: When the first frame of missing content is obtained, multiple content combinations are randomly selected in set1 to generate candidate eviction sets (each candidate eviction set has M content), whose total size is greater than the size of the missing content. The sum of the content value of the low-value cached content and the content value of the missing content in each candidate eviction set is set as the cache state s t , which can be expressed as Action space: The length of the action space is m+1. Considering the value joint standard mathematical model, the available actions include two types. The first type is a t = 0, indicating that the requested content is not allowed to enter the cache, the second type, namely a t∈[1,2,...,m], which means that the missing content is allowed to enter the cache, and the ath t The candidate eviction set will be expelled. The action with the highest probability is selected as the action to be executed in practice. The probability of the action comes from the strategy function π(a|s; θ) = P(a = a t |s=s t ). Bonus: Higher byte hit rate means better cache performance, so action a t The reward function r t is defined as t The byte hit rate in the following sliding time window after that.

[0106] Step S403, obtaining a cache action corresponding to the eviction candidate set based on the cache state corresponding to the eviction candidate set and the policy function of the cache decision model;

[0107] It can be understood that according to the policy function π(a|s;θ)=P(a=a t |s=s t ) and cache status s t , determine the cache action a corresponding to each candidate eviction set t .

[0108] Step S404: determine a target eviction candidate set based on the cache action corresponding to the eviction candidate set, and use the low-value cache content in the target eviction candidate set as the target eviction content.

[0109] It can be understood that, according to the probability of each candidate eviction set being evicted, a candidate eviction set that needs to be evicted is finally determined, and all low-value cache contents in the candidate eviction set will be evicted.

[0110] In a feasible implementation, the delayed cache replacement method also includes: determining a reward function of the cache decision model based on a byte hit rate within a preset period; determining a state value function based on the cache state; determining an action value function based on the state value function and the reward function; determining an action advantage function based on the action value function and the state value function; updating the global network parameters of the cache decision model within the preset period based on the action advantage function; and updating the strategy function of the cache decision model based on the updated global network parameters.

[0111] It should be noted that the A3C algorithm contains a global network and some working networks. The global network structure is the same as each working network, that is, the Actor-Critic structure. The only difference is that the global network does not need to be trained and is only used to store the parameters of the Actor-Critic structure. The Actor network and the Critic network constitute the global network. The goal of the Actor network is to optimize the policy function π(a|s; θ), and its goal is to select better actions. The Critic network aims to improve the accuracy of the value function to evaluate the action selection strategy of the Actor network.

[0112] In addition, it should be noted that this embodiment defines the state value function V(s t ) and the action-value function Q(s t ,a t ). Using the n-step sampling method, the state value function can be expressed as V(s t )=E π [r t +γ t+1 +…+γ n V(s t+n )], which represents the state space s t The value evaluation of the eviction candidate set under the state space s t It can be expressed as According to the mathematical model of value joint standard, r t is the reward function, and γ is the decay factor during training, which depends on the training process. Using the n-step sampling method, the action value function can be expressed as Q(s t ,a t )=r t +γ t+1 +…+γ n V(s t+n ), represents the state space s t Next, perform action a t The value assessment of the decision. The action advantage function is: A(s t ,a t )=Q(s t ,a t )-V(s t ), where V(s t ) is Q(s t ,a t ) is the expectation of all actions, which represents the average value that can be obtained under the eviction action in all eviction candidate sets. t ,a t )>0, action a t The value of the eviction candidate set will be higher than the value of the eviction candidate set in state st Other actions below.

[0113] It can be understood that the Critic network has a fully connected layer whose input is the current eviction candidate state s t , and the output is the value function V in the current state E (s; ω), the parameter ω is given by the formula The Actor network uses a policy function π with parameters θ to generate cache actions and interact with the entire cache state modeled. The Actor network also has a fully connected layer, the input is the current eviction candidate state, and the output is the policy function π(a|s;θ), and the parameters θ are given by the formula Update. According to the calculation formula of dθ and dω, the estimation accuracy of the cache strategy value function can be improved, and the probability of selecting a high-reward action can be increased. Parameters ω and θ are global network parameters of the cache decision model. The preset period refers to the set update period T. At each preset period, the global network parameters of the cache decision model are updated, thereby updating the strategy function of the cache decision model and obtaining a more accurate decision.

[0114] This embodiment provides a delayed cache replacement method and proposes a delayed cache model, which takes the time of acquiring data in a distributed system into consideration, utilizes the cache information in the acquisition gap, improves the cache accuracy, and proposes a value joint standard mathematical model to set a unified value measurement standard for the overall access and exit, thereby improving the cache efficiency. In addition, a cache framework based on reinforcement learning is proposed, which sets the reward function as the byte hit rate and uses the A3C model to realize cache decisions. By constructing a state space, an action space, and a reward mechanism, the efficiency and accuracy of the cache are improved.

[0115] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the delayed cache replacement method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0116] This application also provides a delayed cache replacement device, please refer to Figure 4 , the delay cache replacement device comprises:

[0117] The delay cache module 10 is used to determine the cache execution time node based on the delay cache strategy.

[0118] The eviction decision module 20 is used to determine the content value of the cached content based on the value-joint standard mathematical model.

[0119] The eviction decision module 20 is further configured to determine a plurality of eviction candidate sets based on the content value of the cached content.

[0120] The eviction decision module 20 is further configured to input the multiple eviction candidate sets into a cache decision model to obtain output target eviction content.

[0121] The cache replacement module 30 is used to delete the target evicted content from the cache content based on the cache execution time node, and cache the corresponding missing content in the request information of the client.

[0122] In a feasible implementation manner, the eviction decision module 20 is further used to determine a first corresponding relationship between the frequency of cached content, the size of cached content and the value of the content based on the value joint standard mathematical model;

[0123] Based on the frequency of cached content in the sliding time window, the size of the cached content, and the first corresponding relationship, the content value of the cached content is obtained.

[0124] In a feasible implementation manner, the eviction decision module 20 is further configured to divide the cache content into a high-value cache content set and a low-value cache content set based on the content value of the cache content;

[0125] A plurality of groups of low-value cache content are selected from the low-value cache content set to generate a plurality of eviction candidate sets, and missing content corresponding to the eviction candidate sets is determined in the request information, wherein the number of low-value cache content in the eviction candidate set is greater than or equal to the number of missing content corresponding to the eviction candidate set.

[0126] In a feasible implementation manner, the eviction decision module 20 is further used to determine a second corresponding relationship between the frequency of missing content, the size of missing content and the value of content based on the value joint standard mathematical model;

[0127] Obtaining the content value of the missing content based on the frequency of missing content within the sliding time window, the size of the missing content, and the second corresponding relationship;

[0128] Determine a cache state corresponding to the eviction candidate set based on the content value of the missing content corresponding to the eviction candidate set and the content value of the low-value cached content in the eviction candidate set;

[0129] Based on the cache state corresponding to the eviction candidate set and the policy function of the cache decision model, obtaining the cache action corresponding to the eviction candidate set;

[0130] Based on the cache action corresponding to the eviction candidate set, a target eviction candidate set is determined, and the low-value cache content in the target eviction candidate set is used as the target eviction content.

[0131] In a feasible implementation manner, the eviction decision module 20 is further used to determine a reward function of the cache decision model based on a byte hit rate within a preset period;

[0132] Based on the cache state, determining a state value function;

[0133] Determining an action-value function based on the state-value function and the reward function;

[0134] Determining an action advantage function based on the action value function and the state value function;

[0135] Based on the action advantage function, updating the global network parameters of the cache decision model within the preset period;

[0136] Based on the updated global network parameters, the policy function of the cache decision model is updated.

[0137] In a feasible implementation manner, the delay cache module 10 is further used to sort the request information based on the time sequence of the request information of the client to generate a request queue;

[0138] Obtaining the critical time of the request queue, wherein the critical time at least includes the data request time, the request generation time period, and the last frame reception time;

[0139] The last frame receiving time is used as the cache execution time node under the delayed cache strategy.

[0140] The delayed cache replacement device provided by the present application adopts the delayed cache replacement method in the above embodiment, which can solve the technical problem of cache inefficiency caused by inconsistent value measurement standards for cache access. Compared with the prior art, the beneficial effects of the delayed cache replacement device provided by the present application are the same as the beneficial effects of the delayed cache replacement method provided by the above embodiment, and other technical features in the delayed cache replacement device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0141] The present application provides a delayed cache replacement device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the delayed cache replacement method in the above-mentioned embodiment one.

[0142] Reference below Figure 5, which shows a schematic diagram of the structure of a delay cache replacement device suitable for implementing the embodiment of the present application. The delay cache replacement device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The delayed cache replacement device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0143] like Figure 5 As shown, the delay cache replacement device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the delay cache replacement device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the delay cache replacement device to communicate wirelessly or wired with other devices to exchange data. Although the delay cache replacement device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0144] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0145] The delayed cache replacement device provided by the present application adopts the delayed cache replacement method in the above embodiment, which can solve the technical problem of cache inefficiency caused by inconsistent value measurement standards for cache access. Compared with the prior art, the beneficial effects of the delayed cache replacement device provided by the present application are the same as the beneficial effects of the delayed cache replacement method provided by the above embodiment, and other technical features in the delayed cache replacement device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0146] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0147] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0148] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the delayed cache replacement method in the above-mentioned embodiment.

[0149] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0150] The computer-readable storage medium may be included in the delay cache replacement device; or may exist independently without being assembled into the delay cache replacement device.

[0151] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the delayed cache replacement device, the delayed cache replacement device: determines a cache execution time node based on a delayed cache strategy; determines a content value of cache content based on a value-joint standard mathematical model; determines multiple eviction candidate sets based on the content value of the cache content; inputs the multiple eviction candidate sets into a cache decision model to obtain output target eviction content; deletes the target eviction content from the cache content based on the cache execution time node, and caches the corresponding missing content in the request information of the client.

[0152] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0153] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0154] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0155] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned delayed cache replacement method, and can solve the technical problem of cache inefficiency caused by inconsistent value metrics for cache access. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the delayed cache replacement method provided by the above-mentioned embodiment, and will not be repeated here.

[0156] The present application also provides a computer program product, including a computer program, which implements the steps of the delayed cache replacement method as described above when executed by a processor.

[0157] The computer program product provided by the present application can solve the technical problem of cache inefficiency caused by inconsistent value measurement standards for cache access. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the delayed cache replacement method provided by the above embodiment, which will not be described in detail here.

[0158] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A delayed cache replacement method, characterized in that: The method comprises: Based on the delayed cache strategy, determine the cache execution time node; Determine the content value of cached content based on a value-joint standard mathematical model; Determining a plurality of eviction candidate sets based on the content value of the cached content; Inputting the multiple eviction candidate sets into a cache decision model to obtain output target eviction content; Based on the cache execution time node, the target evicted content is deleted from the cache content, and the corresponding missing content in the request information of the client is cached.

2. The method according to claim 1, characterized in that The step of determining the content value of the cached content based on the value joint standard mathematical model includes: Based on the value joint standard mathematical model, determining a first corresponding relationship between the frequency of cached content, the size of cached content and the value of the content; Based on the frequency of cached content in the sliding time window, the size of the cached content, and the first corresponding relationship, the content value of the cached content is obtained.

3. The method according to claim 1, characterized in that The step of determining a plurality of eviction candidate sets based on the content value of the cached content comprises: Based on the content value of the cached content, the cached content is divided into a high-value cached content set and a low-value cached content set; A plurality of groups of low-value cache content are selected from the low-value cache content set to generate a plurality of eviction candidate sets, and missing content corresponding to the eviction candidate sets is determined in the request information, wherein the number of low-value cache content in the eviction candidate set is greater than or equal to the number of missing content corresponding to the eviction candidate set.

4. The method according to claim 3, characterized in that The step of inputting the multiple eviction candidate sets into the cache decision model to obtain output target eviction content includes: Determine a second correspondence between the frequency of missing content, the size of missing content and the value of content based on the value joint standard mathematical model; Obtaining the content value of the missing content based on the frequency of missing content within the sliding time window, the size of the missing content, and the second corresponding relationship; Determine a cache state corresponding to the eviction candidate set based on the content value of the missing content corresponding to the eviction candidate set and the content value of the low-value cached content in the eviction candidate set; Based on the cache state corresponding to the eviction candidate set and the policy function of the cache decision model, obtaining the cache action corresponding to the eviction candidate set; Based on the cache action corresponding to the eviction candidate set, a target eviction candidate set is determined, and the low-value cache content in the target eviction candidate set is used as the target eviction content.

5. The method according to claim 4, characterized in that The method further comprises: Determining a reward function of the cache decision model based on a byte hit rate within a preset period; Based on the cache state, determining a state value function; Determining an action-value function based on the state-value function and the reward function; Determining an action advantage function based on the action value function and the state value function; Based on the action advantage function, updating the global network parameters of the cache decision model within the preset period; Based on the updated global network parameters, the policy function of the cache decision model is updated.

6. The method according to any one of claims 1 to 5, characterized in that The step of determining the cache execution time node based on the delayed cache strategy includes: Based on the time sequence of the request information from the client, the request information is sorted to generate a request queue; Obtaining the critical time of the request queue, wherein the critical time at least includes the data request time, the request generation time period, and the last frame reception time; The last frame receiving time is used as the cache execution time node under the delayed cache strategy.

7. A delayed cache replacement device, characterized in that: The device comprises: A delayed cache module is used to determine the cache execution time node based on the delayed cache strategy; An eviction decision module, for determining the content value of cached content based on a value-joint standard mathematical model; The eviction decision module is further configured to determine a plurality of eviction candidate sets based on the content value of the cached content; The eviction decision module is further used to input the multiple eviction candidate sets into the cache decision model to obtain output target eviction content; A cache replacement module is used to delete the target evicted content from the cache content based on the cache execution time node, and cache the corresponding missing content in the request information of the client.

8. A delayed cache replacement device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the delayed cache replacement method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the delayed cache replacement method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the delayed cache replacement method according to any one of claims 1 to 6 are implemented.

Images