Delay cache replacement method, device, equipment, storage medium and product

Abstract

The application discloses a delay cache replacement method and device, equipment, storage medium and product, and relates to the technical field of network transmission. The method comprises the following steps: determining a cache execution time node based on a delay cache strategy; determining the content value of cache content based on a value joint standard mathematical model; determining a plurality of eviction candidate sets based on the content value of the cache content; inputting the plurality of eviction candidate sets into a cache decision model to obtain output target eviction content; and deleting the target eviction content from the cache content based on the cache execution time node, and caching corresponding missing content in the request information of a client. Through the above method, a delay cache model is proposed, the time for obtaining data in a distributed system is considered, the cache accuracy is improved, a joint standard model is proposed, a unified measurement standard is set for the overall access and exit, the uniformity of the access strategy and the cache replacement strategy standard is ensured, and the cache efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network transmission, in particular to a delay cache replacement method and device, equipment, storage medium and product. BACKGROUND

[0002] At present, due to the non-uniformity of cache admission and cache replacement metrics, low-value content may be cached in the cache space of the distributed client, and low-value content is constantly replaced by low-value content in the cache space, resulting in frequent eviction of content, which in turn affects the performance of the cache. Assuming that the future access potential of the requested content and the cached content is evaluated using different standards, this means that there may be allowed content with lower actual access probability than the evicted content, or non-allowed content with higher actual access probability than the cached content in the distributed server, which often leads to the problem of repeated access and demand content being deleted, affecting cache efficiency.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a delay cache replacement method, device, equipment, storage medium and product, which aims to solve the technical problem of low efficiency of the cache caused by the non-uniformity of the value measurement standard of the cache admission and cache replacement in the prior art.

[0005] To achieve the above purpose, the present application provides a delay cache replacement method, which comprises:

[0006] determining a cache execution time node based on a delay cache strategy;

[0007] determining the content value of the cache content based on a value joint standard mathematical model;

[0008] determining a plurality of eviction candidate sets based on the content value of the cache content;

[0009] inputting the plurality of eviction candidate sets into a cache decision model to obtain the target eviction content output;

[0010] based on the cache execution time node, deleting the target eviction content from the cache content, and caching the corresponding missing content in the request information of the client.

[0011] In an embodiment, the step of determining the content value of the cache content based on the value joint standard mathematical model comprises:

[0012] determining a first corresponding relationship between the frequency of the cache content, the size of the cache content and the content value based on the value joint standard mathematical model;

[0013] obtaining a content value of the cache content based on the frequency of the cache content, the size of the cache content and the first corresponding relationship.

[0014] In an embodiment, the step of determining a plurality of eviction candidate sets based on the content value of the cache content comprises:

[0015] dividing the cache content into a high-value cache content set and a low-value cache content set based on the content value of the cache content;

[0016] selecting a plurality of groups of low-value cache content in the low-value cache content set, generating a plurality of eviction candidate sets, and determining the missing content corresponding to the eviction candidate set in the request information, wherein the number of low-value cache content in the eviction candidate set is greater than or equal to the number of missing content corresponding to the eviction candidate set.

[0017] In an embodiment, the step of inputting the plurality of eviction candidate sets into the cache decision model to obtain the output target eviction content comprises:

[0018] determining a second corresponding relationship between the frequency of the missing content, the size of the missing content and the content value based on the value joint standard mathematical model;

[0019] obtaining a content value of the missing content based on the frequency of the missing content, the size of the missing content and the second corresponding relationship within the sliding time window;

[0020] determining the cache state corresponding to the eviction candidate set based on the content value of the missing content corresponding to the eviction candidate set and the content value of the low-value cache content in the eviction candidate set;

[0021] obtaining the cache action corresponding to the eviction candidate set based on the cache state corresponding to the eviction candidate set and the policy function of the cache decision model;

[0022] determining a target eviction candidate set based on the cache action corresponding to the eviction candidate set, and taking the low-value cache content in the target eviction candidate set as the target eviction content.

[0023] In an embodiment, the method further comprises:

[0024] determining a reward function of the cache decision model based on the byte hit rate within a preset period;

[0025] determining a state value function based on the cache state;

[0026] determining an action value function based on the state value function and the reward function;

[0027] determine an action advantage function based on the action value function and the state value function;

[0028] update global network parameters of the cache decision model in the preset period based on the action advantage function;

[0029] update a policy function of the cache decision model based on the updated global network parameters.

[0030] In an embodiment, the step of determining a cache execution time node based on the delayed cache strategy comprises:

[0031] sort the request information based on the time sequence of the request information of the client to generate a request queue;

[0032] obtain a key time of the request queue, the key time comprising at least a data request time, a request generation time period, and a last frame receiving time;

[0033] take the last frame receiving time as the cache execution time node under the delayed cache strategy.

[0034] In addition, to achieve the above-mentioned purpose, the present application further provides a delayed cache replacement device, which comprises:

[0035] a delayed cache module configured to determine a cache execution time node based on a delayed cache strategy;

[0036] a eviction decision module configured to determine a content value of cache content based on a value joint standard mathematical model;

[0037] the eviction decision module is further configured to determine a plurality of eviction candidate sets based on the content value of the cache content;

[0038] the eviction decision module is further configured to input the plurality of eviction candidate sets into a cache decision model to obtain an output target eviction content;

[0039] a cache replacement module configured to delete the target eviction content from the cache content based on the cache execution time node, and cache corresponding missing content in the request information of the client.

[0040] In addition, to achieve the above-mentioned purpose, the present application further provides a delayed cache replacement device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the delayed cache replacement method as described above.

[0041] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program realizes the steps of the delay cache replacement method when executed by a processor.

[0042] In addition, to achieve the above object, the application further provides a computer program product, which comprises a computer program, and the computer program realizes the steps of the delay cache replacement method when executed by a processor.

[0043] The application provides a delay cache replacement method, determines a cache execution time node based on a delay cache strategy, determines a content value of cache content based on a value joint standard mathematical model, determines a plurality of eviction candidate sets based on the content value of the cache content, inputs the plurality of eviction candidate sets into a cache decision model to obtain target eviction content as output, deletes the target eviction content from the cache content based on the cache execution time node, and caches corresponding missing content in request information of a client. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the application and, together with the specification, serve to explain the principles of the application.

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0046] Figure 1 Flowchart of the delay cache replacement method according to the first embodiment of the application;

[0047] Figure 2 Request queue diagram of the delay cache replacement method according to the second embodiment of the application;

[0048] Figure 3 Flowchart of the delay cache replacement method according to the second embodiment of the application;

[0049] Figure 4 Module structure diagram of the delay cache replacement device according to the embodiment of the application;

[0050] Figure 5 The device structure schematic diagram of the hardware running environment involved in the delay cache replacement method in the embodiments of the present application.

[0051] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0053] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] The main solution of the embodiments of the present application is: based on the delay cache strategy, the cache execution time node is determined; based on the value joint standard mathematical model, the content value of the cache content is determined; based on the content value of the cache content, a plurality of eviction candidate sets are determined; the plurality of eviction candidate sets are input into the cache decision model to obtain the output target eviction content; based on the cache execution time node, the target eviction content is deleted from the cache content, and the corresponding missing content in the request information of the cache client is cached.

[0055] The present application provides a solution, proposes a delay cache model, considers the time of obtaining data in a distributed system, utilizes the cache information in the gap, improves the cache accuracy, and proposes a joint standard model, sets a unified value measurement standard for the overall access and exit, improves the cache efficiency, and solves the technical problem of low cache efficiency caused by the non-uniform value measurement standard of cache access and exit.

[0056] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a delay cache replacement device, etc., and the present embodiment does not make specific limitation. The following will take the delay cache replacement device as an example to describe the present embodiment and the following embodiments.

[0057] The embodiments of the present application provide a delay cache replacement method, which is described with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the delay cache replacement method of the present application.

[0058] In the present embodiment, the delay cache replacement method comprises steps S10-S50:

[0059] Step S10, based on the delay cache strategy, the cache execution time node is determined;

[0060] It should be noted that the client in the embodiment refers to a distributed client. The distributed client obtains missing content from various data nodes, and receives multiple data requests of the user in the process of obtaining. The embodiment fully considers the information carried by the request content received in the gap, and based on the feature, uses a delayed cache mode to utilize the cache information in the gap to improve cache efficiency and accuracy. The delayed cache strategy is the delayed cache mode used in the embodiment.

[0061] In a possible implementation, the step S10 can include steps S101-S103:

[0062] In step S101, the request information is sorted based on the time sequence of the request information of the client, and a request queue is generated.

[0063] It should be noted that the request information refers to a large amount of request information generated by the client during data acquisition, and the time sequence of the request information refers to the sequence of the generation time of the request information.

[0064] It can be understood that the time sequence of all the request information is arranged in the request queue in the time sequence, and the request information generated first is arranged in front, and the request information generated later is arranged behind.

[0065] In step S102, the key time of the request queue is obtained, and the key time at least includes the data request time, the request generation time period, and the last frame receiving time.

[0066] It should be noted that the key time refers to the required related time information, and at least includes the data request time, the request generation time period, and the last frame receiving time. The data request time refers to the time point of issuing the data in the request data block, the request generation time period refers to the time period of generating the request, and the last frame receiving time refers to the time point of receiving the last frame data of the data block.

[0067] In step S103, the last frame receiving time is taken as the cache execution time node under the delayed cache strategy.

[0068] It can be understood that a large amount of request information generated by the client during data acquisition usually needs to be marked so as to determine the cache execution time node subsequently. Figure 2 A schematic diagram of the request queue is shown in the request queue (request queue). The distributed client issues a request for the data block C at t1, and the last frame data of the data block C is received at t2. During this period, the client generates a large amount of data requests (A, B) in the T time period.

[0069] It should be noted that the cache execution time node is the time node of executing the cache strategy, and the cache strategy refers to the cache replacement strategy, which replaces the content with low value in the cache with other content with higher value.

[0070] It can be understood that when the content in the cache is replaced, the efficiency of the cache is affected. If the delayed time node is too late, the client may have obtained the required data, but there is no available space in the local cache to cache it; if the delayed time node is too early, the information in the request queue is not complete enough, and the data that needs to be replaced may be requested again. Therefore, in order to avoid such problems, the last frame data of the requested content is selected to perform the local cache replacement operation, that is, the cache replacement strategy is executed at the last frame receiving time, and the cache execution time node is set at t2, the data request information before t2 is saved, and the cache replacement strategy is executed according to the saved information. Figure 2

[0071] In a specific implementation, the request information generated by the distributed client is arranged in time sequence as a request queue, the last frame receiving time is found as the cache execution time node, and the delayed cache is realized.

[0072] In step S20, the content value of the cache content is determined based on a value joint standard mathematical model.

[0073] It should be noted that since the measurement standards of cache admission and cache replacement are usually not unified, low-value content may be continuously replaced with low-value content in the cache space of the client, resulting in frequent replacement of content. Therefore, the value joint standard mathematical model is proposed in this embodiment, which unifies the value measurement standards of cache admission and cache replacement, and can effectively ensure that the content cached in the distributed server has a relatively high future access value.

[0074] In addition, it should be noted that the future access value of the cache content is usually related to three key factors of size, frequency and timeliness of the content. The most intuitive attribute reflecting the future access value is the access frequency of historical access. Generally speaking, the higher the frequency of each content being accessed, the higher its value is not. For the same access frequency, smaller content has a higher value because smaller content occupies smaller space, and therefore, single-byte access frequency can be used to reflect content access value. In addition, the continuous accumulation of the frequency of each content will lead to an increase in the frequency of the content, which will lead to a misjudgment of the content frequency degree, and therefore, a certain time span and request span need to be ensured.

[0075] ​It can be understood that a sliding window based on a period of time δt can be added to calculate the access frequency to ensure that the value of the cache content is available. The value of the content i is measured in time t The unified standard is:

[0076]

[0077] In the formula, a represents the coefficient of variation, τ i represents the size of the content i, represents the total size of the content in the request information during δt to t, F t-δt represents the total number of requests, F i,f represents the access frequency of content i in the fth request information during t-δt to t, and if the fth request is not content i, F i,f equals 0, which means that when the request amount is small, the user sends fewer requests during the acquisition period. When the number of requests is small, the value of the requested content is very close, and it is difficult to separate the high-value content from the cached content, therefore, the embodiment optimizes the traditional calculation method, calculates the content value according to the request frequency of the content and the size of the content, and constructs a value joint standard mathematical model, for example: the request frequency of the content (in the sliding window δt) divided by the size of the content, which can easily distinguish high-value content and low-value content. At this time, the value of the content is only determined by its size and frequency, and is not affected by the number of requests from other contents.

[0078] In a possible implementation, step S20 can include: determining a first correspondence relationship between the frequency of the cache content, the size of the cache content, and the content value of the cache content based on the value joint standard mathematical model; and obtaining the content value of the cache content based on the frequency of the cache content in the sliding time window, the size of the cache content, and the first correspondence relationship.

[0079] It should be noted that the cache content is the content currently in the cache. The first correspondence relationship between the frequency of the cache content, the size of the cache content, and the content value of the cache content is the calculation relationship of the content value of the cache content, which is determined according to the value joint standard mathematical model, as follows:

[0080]

[0081] In the formula, represents the content value of the cache content i, i represents the frequency of the cache content i in the sliding window δt, τ i represents the size of the cache content i. By substituting the related data into the above first correspondence relationship, the content values of all cache contents can be calculated.

[0082] Step S30, determining a plurality of eviction candidate sets based on the content values of the cache contents;

[0083] It should be noted that the eviction candidate set is a set of contents that can be evicted. Since the essence of the cache replacement strategy is to select the low-value contents as the evicted contents by comparing the values of the cache contents in the distributed server, the eviction candidate set is usually generated according to the low-value cache contents.

[0084] In a feasible implementation, step S30 can include steps S301-S303:

[0085] Step S301, dividing the cache contents into a high-value cache content set and a low-value cache content set based on the content values of the cache contents.

[0086] It should be noted that the high-value cache content set is a set of high-value cache contents, the high-value cache content is a high-value content in the cache contents, the low-value cache content set is a set of low-value cache contents, and the low-value cache content is a low-value content in the cache contents. The high-value cache content and the low-value cache content can be determined by setting a value threshold. The cache content with a content value greater than or equal to the value threshold is a high-value cache content, and the cache content with a content value less than the value threshold is a low-value cache content.

[0087] It can be understood that all the cache contents are divided into high-value cache contents and low-value cache contents according to the content values of the cache contents. All the high-value cache contents constitute the high-value cache content set, and all the low-value cache contents constitute the low-value cache content set.

[0088] Step S302, selecting a plurality of groups of low-value cache contents in the low-value cache content set, generating a plurality of eviction candidate sets, and determining the missing contents corresponding to the eviction candidate sets in the request information. The number of low-value cache contents in the eviction candidate set is greater than or equal to the number of missing contents corresponding to the eviction candidate set.

[0089] It should be noted that a plurality of groups of low-value cache contents are selected in the low-value cache content set. Each group of low-value cache contents is an eviction candidate set. The missing contents corresponding to the eviction candidate set are the contents requested by the client that are missing in the current cache contents after the contents in the eviction candidate set are evicted. Generally, the number of low-value cache contents in the eviction candidate set needs to be greater than or equal to the number of missing contents corresponding to the eviction candidate set.

[0090] It can be understood that, since the access value of each content needs to be recalculated at each comparison, and the two calculations before and after may come to the same value, it is easy to affect the calculation speed and reduce the selection efficiency of the evicted content. In addition, since the high-value content is much more valuable than other content at some time, whether its value is updated in real time or not has no effect on the selection of the evicted content, because. Therefore, in order to improve the calculation speed and selection efficiency of the evicted content, the embodiment divides all cached contents in the storage space into two sets (set1 and set2) according to the content value. Set1, i.e. the low-value cached content set, contains low-value contents, accounting for ζ% of all contents, and set2, i.e. the high-value cached content set, contains the remaining high-value contents. Set1 is usually used to set the eviction candidate set, and when selecting the evicted content, the value of the content in the eviction candidate set needs to be calculated each time, which can significantly reduce the calculation amount, and the range of eviction options is much smaller. But because the requested content is dynamic, the value of the content is usually time-sensitive, in order to ensure that the low-value content is in the eviction candidate set, set1 and set2 need to be updated regularly, and in order to avoid the content entering the cache being frequently evicted, the content entering the cache is stored in the high-value cached content set set2.

[0091] Step S40, inputting the plurality of eviction candidate sets into a cache decision model to obtain an output target evicted content;

[0092] It should be noted that the target evicted content refers to the content that needs to be evicted after the final decision. The embodiment sets multiple eviction candidate sets, and finally directly decides a set of contents that need to be evicted, all contents in the set need to be evicted. The cache decision model is a model constructed by the embodiment for deciding the target evicted content, the input of which is all eviction candidate sets determined in step S30, and the output is the target evicted content, which can be trained by using a reinforcement learning algorithm, and is not specifically limited.

[0093] Step S50, based on the cache execution time node, deleting the target evicted content from the cache content, and caching the corresponding missing content in the request information of the client.

[0094] It can be understood that the corresponding missing content in the request information of the client is the content requested by the client that is missing in the current cache content. According to the cache replacement strategy according to the cache execution time node, the target evicted content is deleted, and the corresponding missing content is cached.

[0095] It should be understood that when determining the target evicted content, the value of the missing content after deleting the target evicted content also needs to be considered, and the value of the missing content also needs to be calculated according to the value joint standard mathematical model, so as to ensure that the admission and the exit have a unified value measurement standard.

[0096] This embodiment provides a delayed cache replacement method. Based on a delayed cache strategy, a cache execution time node is determined. Based on a value-joint standard mathematical model, the content value of the cached content is determined. Based on the content value of the cached content, multiple eviction candidate sets are determined. The multiple eviction candidate sets are input into a cache decision model to output target eviction content. Based on the cache execution time node, the target eviction content is deleted from the cache content, and the corresponding missing content in the client's request information is cached. A delayed cache model is proposed that takes into account the time it takes to retrieve data in a distributed system, utilizing cached information in the retrieval gap to improve cache accuracy. Furthermore, a joint standard model is proposed that sets a unified value metric for overall access and exit, thereby improving cache efficiency.

[0097] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 3 , the step S40 may include steps S401 to S403:

[0098] Step S401: Determine a second correspondence between the frequency of missing content, the size of missing content, and content value based on the value joint standard mathematical model, and obtain the content value of the missing content based on the frequency of missing content, the size of missing content, and the second correspondence within a sliding time window.

[0099] It should be noted that the second corresponding relationship between the frequency of missing content, the size of missing content, and the content value, i.e., the calculation formula for the content value of missing content, is determined according to the value joint standard mathematical model, as shown below:

[0100]

[0101] Where, represents the content value of missing content i, i represents the frequency of missing content i within the sliding window δt, τ i Represents the size of missing content i. Substituting the relevant data into the second correspondence above, the content value of all missing content can be calculated.

[0102] Step S402, determining a cache state corresponding to the eviction candidate set based on the content value of the missing content corresponding to the eviction candidate set and the content value of the low-value cached content in the eviction candidate set;

[0103] It should be noted that the delay cache model and the value joint standard mathematical model can greatly improve the performance of the existing cache replacement strategy, and can solve the problem of low flexibility and low accuracy of the traditional cache replacement strategy. In order to more effectively utilize the two models, the embodiment proposes an overall framework that can integrate the two models, which includes four parts: initialization, information collection, cache decision and model update. Initialization: In the initialization stage, LRU is used as the eviction strategy, and all request contents in the distributed server are allowed to initialize the global network and establish the local network, wherein the selection probability of each action is equal in the initialization. Information collection: When the request queue length recorded by the server is sufficient to meet the two sliding time windows, collect the request information during the data acquisition period and append it to the historical request information, divide the historical request information into two groups of high value and low value, and select the low value set as the cache content eviction candidate set. Cache decision: Based on the collection information stage, the obtained eviction candidate set is used as the input content of the cache decision model, the cache decision model is executed to make cache decision, and the eviction content of the model decision is returned, and then the content to be evicted is selected and deleted. Model update: In order to maintain the effectiveness of the model, the global network parameters of the model are updated at intervals of T seconds.

[0104] In addition, it should be noted that the embodiment selects A3C (Asynchronous Advantage Actor-Critic) model as the main component of the cache decision model. Compared with the traditional reinforcement learning cache strategy, the A3C model used in the embodiment can introduce Actor-Critic into multiple threads for training at the same time, which is high in training efficiency and suitable for selecting eviction content. In addition, based on the value joint standard mathematical model, the state space of the cache strategy is reasonably quantized, and a reward function based on the hit rate is separately designed, which improves the learning ability of the algorithm itself and improves the overall performance of the model.

[0105] It can be understood that A3C is a reinforcement learning algorithm, including state space, action space, reward, etc. State space: when the first frame of missing content is obtained, a plurality of content combinations are randomly selected in set1 to generate a candidate eviction set (each candidate eviction set has M contents), and the total size is greater than the size of the missing content. The sum of the content value of the low value cache content in each candidate eviction set and the content value of the missing content is set as the cache state s t , which can be expressed as Action space: the length of the action space is m+1, considering the value joint standard mathematical model, the available actions include two types, the first one is a t =0, which means that the requested content is not allowed to enter the cache, and the second one is a t∈ [1, 2, …, m] means that the missing content is allowed to enter the cache, and the a t th candidate eviction set will be evicted. The action with the highest probability is selected as the action to be executed in practice, and the probability of the action comes from the policy function π(a|s; θ) = P(a = a t |s = s t ). Reward: Higher byte hit rate means better cache performance, therefore, the reward function r t of the action a t is defined as the byte hit rate in the following sliding time window after a t .

[0106] Step S403, based on the cache state corresponding to the eviction candidate set and the policy function of the cache decision model, obtaining the cache action corresponding to the eviction candidate set;

[0107] It can be understood that according to the policy function π(a|s; θ) = P(a = a t |s = s t ) and the cache state s t , the cache action a t corresponding to each candidate eviction set is determined.

[0108] Step S404, based on the cache action corresponding to the eviction candidate set, determining a target eviction candidate set, and taking the low-value cache content in the target eviction candidate set as the target eviction content.

[0109] It can be understood that according to the probability of each candidate eviction set being evicted, the candidate eviction set that needs to be evicted is finally determined, and all the low-value cache content in the candidate eviction set will be evicted.

[0110] In a possible implementation, the delayed cache replacement method further includes: determining a reward function of the cache decision model based on the byte hit rate in a preset period; determining a state value function based on the cache state; determining an action value function based on the state value function and the reward function; determining an action advantage function based on the action value function and the state value function; updating global network parameters of the cache decision model in the preset period based on the action advantage function; and updating the policy function of the cache decision model based on the updated global network parameters.

[0111] It should be noted that the A3C algorithm includes a global network and some working networks, the global network structure is the same as each working network, that is, the Actor-Critic structure, the only difference is that the global network does not need to be trained, and is only used to store the parameters of the Actor-Critic structure, and the Actor network and the Critic network constitute the global network. The goal of the Actor network is to optimize the policy function π(a|s; θ), which aims to select better actions, and the Critic network aims to improve the accuracy of the value function to evaluate the action selection strategy of the Actor network.

[0112] In addition, it should be noted that the embodiment defines a state value function V(s t ) and an action value function Q(s t ,a t ). Using the n-step sampling method, the state value function can be expressed as V(s t ) = E π [r t + γ t+1 + … + γ n V(s t+n )], which represents the value evaluation of the eviction candidate set in the state space s t , wherein the state space s t can be expressed as According to the calculation of the value joint standard mathematical model, r t is the reward function, and γ is the decay factor in the training process, which is specific to the training process. Using the n-step sampling method, the action value function can be expressed as Q(s t ,a t ) = r t + γ t+1 + … + γ n V(s t+n ), which represents the value evaluation of the decision of performing action a t in the state space s t . The action advantage function is A(s t ,a t ) = Q(s t ,a t )-V(s t ), wherein V(s t ) is the expectation of Q(s t ,a t ) for all actions, which represents the average value that can be obtained by all eviction candidate sets under the eviction action. When A(s t ,a t )>0, the value of action a t will be higher than the average value that can be obtained by the eviction candidate set in state st other actions under the Critic network.

[0113] It can be understood that the Critic network has a fully connected layer, the input of which is the current eviction candidate state s t , and the output is the value function V E (s; ω) in the current state, and the parameter ω is updated by the formula The Actor network adopts a policy function π with a parameter θ to generate a cache action and interact with the entire cache state modeled. The Actor network also has a fully connected layer, the input of which is the current eviction candidate state, and the output is the policy function π(a|s; θ), and the parameter θ is updated by the formula According to the calculation formula of dθ and dω, the estimation accuracy of the cache policy value function can be improved, and the probability of selecting a high return action can be improved. The parameter ω and the parameter θ are global network parameters of the cache decision model. The preset period refers to the set update period T, and every interval of the preset period, the global network parameters of the cache decision model are updated, so that the policy function of the cache decision model is updated, and more accurate decisions are obtained.

[0114] The embodiment provides a delay cache replacement method, proposes a delay cache model, considers the time of obtaining data in a distributed system, utilizes cache information in the obtaining gap, improves cache accuracy, proposes a value joint standard mathematical model, sets a unified value measurement standard for overall access and exit, improves cache efficiency, in addition, proposes a cache framework based on reinforcement learning, sets a reward function as a byte hit rate, uses an A3C model to realize cache decision, and through construction of a state space, an action space and a reward mechanism, improves cache efficiency and accuracy.

[0115] It should be noted that the above examples are only used for understanding the present application and do not constitute a limitation on the delay cache replacement method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0116] The present application also provides a delay cache replacement device, please refer to Figure 4 The delay cache replacement device comprises:

[0117] A delay cache module 10 is configured to determine a cache execution time node based on a delay cache strategy.

[0118] An eviction decision module 20 is configured to determine a content value of cache content based on a value joint standard mathematical model.

[0119] The eviction decision module 20 is further configured to determine a plurality of eviction candidate sets based on the content value of the cache content.

[0120] The eviction decision module 20 is further configured to input the plurality of eviction candidate sets into a cache replacement decision model to obtain output target eviction content.

[0121] The cache replacement module 30 is configured to delete the target eviction content from the cache content and cache corresponding missing content in the request information of the client based on the cache execution time node.

[0122] In an implementation, the eviction decision module 20 is further configured to determine, based on the value joint standard mathematical model, a first corresponding relationship between a frequency of cache content, a size of the cache content, and a content value of the cache content.

[0123] The content value of the cache content is obtained based on the frequency of the cache content, the size of the cache content, and the first corresponding relationship within a sliding time window.

[0124] In an implementation, the eviction decision module 20 is further configured to divide the cache content into a high-value cache content set and a low-value cache content set based on the content value of the cache content.

[0125] A plurality of groups of low-value cache content are selected from the low-value cache content set to generate a plurality of eviction candidate sets, and corresponding missing content of the eviction candidate sets is determined in the request information. The number of low-value cache content in the eviction candidate set is greater than or equal to the number of corresponding missing content of the eviction candidate set.

[0126] In an implementation, the eviction decision module 20 is further configured to determine, based on the value joint standard mathematical model, a second corresponding relationship between a frequency of missing content, a size of the missing content, and a content value of the missing content.

[0127] The content value of the missing content is obtained based on the frequency of the missing content, the size of the missing content, and the second corresponding relationship within a sliding time window.

[0128] The cache state corresponding to the eviction candidate set is determined based on the content value of the corresponding missing content of the eviction candidate set and the content value of the low-value cache content in the eviction candidate set.

[0129] The cache action corresponding to the eviction candidate set is obtained based on the cache state corresponding to the eviction candidate set and a strategy function of the cache decision model.

[0130] The target eviction candidate set is determined based on the cache action corresponding to the eviction candidate set, and low-value cache content in the target eviction candidate set is taken as the target eviction content.

[0131] In a possible implementation, the eviction decision module 20 is further configured to determine a reward function of the cache decision model based on a byte hit rate in a preset period.

[0132] determine a state value function based on the cache state;

[0133] determine an action value function based on the state value function and the reward function;

[0134] determine an action advantage function based on the action value function and the state value function;

[0135] update global network parameters of the cache decision model in the preset period based on the action advantage function;

[0136] update a policy function of the cache decision model based on the updated global network parameters.

[0137] In a possible implementation, the delayed cache module 10 is further configured to sort request information of a client based on a time sequence of the request information, and generate a request queue;

[0138] obtain a critical time of the request queue, the critical time including at least a data request time, a request generation time period, and a last frame receiving time;

[0139] take the last frame receiving time as a cache execution time node under a delayed cache strategy.

[0140] The delayed cache replacement device provided in the application adopts the delayed cache replacement method in the above embodiments, and can solve the technical problem of low cache efficiency caused by non-uniform value measurement standards of cache admission and exclusion. Compared with the prior art, the delayed cache replacement device provided in the application has the same beneficial effects as the delayed cache replacement method provided in the above embodiments, and other technical features in the delayed cache replacement device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0141] The application provides a delayed cache replacement device, which comprises at least one processor and a memory connected with the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the delayed cache replacement method in the above embodiment one.

[0142] The following refers to Figure 5, which shows a schematic diagram of the structure of a delay cache replacement device suitable for implementing embodiments of the present application. The delay cache replacement device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The delayed cache replacement device shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present application.

[0143] like Figure 5 As shown, the delay cache replacement device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the delay cache replacement device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. The communication device 1009 can allow the delay cache replacement device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a delay cache replacement device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or have alternatively.

[0144] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0145] The delay cache replacement device provided by the present application adopts the delay cache replacement method in the above embodiments, and can solve the technical problem of low cache efficiency caused by the non-uniform value measurement standard of cache admission and exclusion. Compared with the prior art, the delay cache replacement device provided by the present application has the same beneficial effects as the delay cache replacement method provided by the above embodiments, and other technical features in the delay cache replacement device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0146] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0147] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0148] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the delay cache replacement method in the above embodiments.

[0149] The computer readable storage medium provided in the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination thereof.

[0150] The computer readable storage medium described above may be contained in a delay cache replacement device, or may exist separately without being assembled into the delay cache replacement device.

[0151] The computer readable storage medium described above carries one or more programs, which, when executed by the delay cache replacement device, cause the delay cache replacement device to: determine a cache execution time node based on a delay cache strategy; determine a content value of cache content based on a value joint standard mathematical model; determine a plurality of eviction candidate sets based on the content value of the cache content; input the plurality of eviction candidate sets into a cache decision model to obtain an output target eviction content; delete the target eviction content from the cache content based on the cache execution time node, and cache corresponding missing content in the request information of the client.

[0152] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0153] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0154] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0155] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above-mentioned delay cache replacement method, and can solve the technical problem of low cache efficiency caused by the non-uniform value measurement standard of cache admission and exclusion. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the delay cache replacement method provided by the above-mentioned embodiments, and will not be described here.

[0156] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the delayed cache replacement method as described above.

[0157] The computer program product provided by the application can solve the technical problem of low cache efficiency caused by the non-uniform value measurement standard of cache admission and exclusion. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the delayed cache replacement method provided by the above-mentioned embodiments, and are not described here.

[0158] The above only describes some embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or direct / indirect application in other related technical fields within the technical concept of the application, using the content of the application specification and drawings, are included in the patent protection scope of the application.

Claims

1. A method of delayed cache replacement, the method comprising: The method comprises: Determine the cache execution time node based on the delayed cache strategy; Determine the content value of cached content based on a value-joint standard mathematical model; determining a plurality of eviction candidate sets based on content value of the cached content; Inputting the multiple eviction candidate sets into a cache decision model to obtain output target eviction content; Based on the cache execution time node, the target evicted content is deleted from the cache content, and the corresponding missing content in the client's request information is cached; The step of determining a plurality of eviction candidate sets based on the content value of the cached content includes: Based on the content value of the cached content, the cached content is divided into a high-value cached content set and a low-value cached content set; multiple groups of low-value cached content are selected from the low-value cached content set to generate multiple eviction candidate sets, and missing content corresponding to the eviction candidate sets is determined in the request information, where the number of low-value cached content in the eviction candidate sets is greater than or equal to the number of missing content corresponding to the eviction candidate sets; The step of inputting the multiple eviction candidate sets into the cache decision model to obtain output target eviction content includes: Based on the value joint standard mathematical model, the correspondence between the frequency of missing content, the size of missing content and the content value is determined; based on the frequency of missing content, the size of missing content and the correspondence between the frequency of missing content, the size of missing content and the content value within a sliding time window, the content value of the missing content is obtained; based on the content value of the missing content corresponding to the eviction candidate set and the content value of the low-value cached content in the eviction candidate set, the cache state corresponding to the eviction candidate set is determined; based on the cache state corresponding to the eviction candidate set and the policy function of the cache decision model, the cache action corresponding to the eviction candidate set is obtained; based on the cache action corresponding to the eviction candidate set, a target eviction candidate set is determined, and the low-value cached content in the target eviction candidate set is used as the target eviction content.

2. The method of claim 1, wherein, The step of determining the content value of the cached content based on the value joint standard mathematical model includes: Determining a first correspondence between a frequency of cached content, a size of cached content, and a value of the content based on the value association standard mathematical model; The content value of the cached content is obtained based on the frequency of the cached content in the sliding time window, the size of the cached content, and the first corresponding relationship.

3. The method of claim 1, wherein, The method further comprises: Determining a reward function of the cache decision model based on a byte hit rate within a preset period; Determining a state-value function based on the cache state; Determining an action-value function based on the state-value function and the reward function; Determining an action advantage function based on the action-value function and the state-value function; Based on the action advantage function, updating the global network parameters of the cache decision model within the preset period; Based on the updated global network parameters, the policy function of the cache decision model is updated.

4. The method of any one of claims 1 to 3, wherein, The step of determining the cache execution time node based on the delayed cache strategy includes: sequencing the request information based on a time sequence of the request information, to generate a request queue; obtaining a key time of the request queue, the key time including at least a data request time, a request generation time period, and a last frame receiving time; taking the last frame receiving time as a cache execution time node under a delayed cache strategy.

5. A delay cache replacement apparatus, characterized by, The device comprises: a delayed cache module configured to determine a cache execution time node based on a delayed cache strategy; a eviction decision module configured to determine a content value of cache content based on a value joint standard mathematical model; the eviction decision module is further configured to determine a plurality of eviction candidate sets based on the content value of the cache content; the eviction decision module is further configured to input the plurality of eviction candidate sets into a cache decision model to obtain an output target eviction content; a cache replacement module configured to delete the target eviction content from the cache content and cache corresponding missing content in the request information based on the cache execution time node; the eviction decision module is further configured to divide the cache content into a high-value cache content set and a low-value cache content set based on the content value of the cache content; selecting a plurality of groups of low-value cache content in the low-value cache content set to generate a plurality of eviction candidate sets, and determining corresponding missing content of the eviction candidate sets in the request information, wherein the number of low-value cache content in the eviction candidate set is greater than or equal to the number of corresponding missing content of the eviction candidate set; the eviction decision module is further configured to determine a corresponding relationship between the frequency of missing content, the size of missing content, and the content value based on the value joint standard mathematical model; obtaining the content value of the missing content based on the frequency of the missing content in the sliding time window, the size of the missing content, and the corresponding relationship between the frequency of the missing content, the size of the missing content, and the content value; determining the cache state corresponding to the eviction candidate set based on the content value of the corresponding missing content of the eviction candidate set and the content value of the low-value cache content in the eviction candidate set; obtaining the cache action corresponding to the eviction candidate set based on the cache state corresponding to the eviction candidate set and a strategy function of the cache decision model; determining a target eviction candidate set based on the cache action corresponding to the eviction candidate set, and taking the low-value cache content in the target eviction candidate set as the target eviction content.

6. A delay cache replacement device, characterized by, The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the delayed cache replacement method according to any one of claims 1 to 4.

7. A storage medium, characterized by The storage medium is a computer-readable storage medium, and the storage medium stores a computer program, which is executed by a processor to implement the steps of the delayed cache replacement method according to any one of claims 1 to 4.

8. A computer program product, characterised in that, The computer program product comprises a computer program, which is executed by a processor to implement the steps of the delayed cache replacement method according to any one of claims 1 to 4.

Images