Content caching method and device, equipment and medium

Abstract

The invention provides a content caching method and device, equipment and a medium. Comprising the steps of determining a preset state space, an identification space, an initial strategy and a reward rule; determining a corresponding storage identifier from the identifier space according to the initial strategy, and determining a new state according to the storage identifier and the state space; according to the new state and the reward rule, determining a total reward value of the initial strategy within the first preset time; determining a target reward expectation, and adjusting parameters of the initial strategy according to the total reward value and the target reward expectation to obtain an update strategy; according to the updating strategy and the reward rule, determining an updated reward total value in a second preset time, and comparing the updated reward total value with a preset reward threshold range to obtain a comparison result; according to a comparison result, adjusting parameters of the updating strategy to obtain a target strategy, and according to the target strategy, determining target user content; and caching the target user content. Therefore, the edge caching accuracy of the user content is improved.

Description

Technical Field

[0001] The present application relates to the field of mobile edge caching, and in particular to a content caching method, apparatus, device and medium. Background Art

[0002] With the development of mobile Internet, edge caching technology has become a key means to improve the quality of user experience. By placing servers with storage functions at the edge of the network, frequently requested content can be cached, thereby reducing the content delivery pressure of base stations and reducing the latency of user request acquisition.

[0003] In the current content caching solution, the cache is located at the edge base station or D2D user terminal. The storage capacity of these cache devices is limited and cannot cache all possible content. They can only cache content that is likely to be requested by future users.

[0004] However, existing content caching solutions have the problem of low accuracy of cached content. Summary of the invention

[0005] The present application provides a content caching method, apparatus, device and medium, which are used to solve the problem of low accuracy of cached content in existing content caching solutions.

[0006] In a first aspect, the present application provides a content caching method, the method comprising:

[0007] Determine the preset state space, identification space, initial strategy, and reward rules. The state in the state space includes historical user content. The identification space is used to store the storage identifier of the historical user content in the storage space. The initial strategy is used to determine the storage identifier corresponding to the current next state. The reward rule is used to determine the reward value of the state.

[0008] According to the initial strategy, a corresponding storage identifier is determined from the identifier space, and a new state is determined according to the storage identifier and the state space;

[0009] According to the new state and reward rules, determine the total reward value of the initial strategy within the first preset time;

[0010] Determine the target reward expectation, and adjust the parameters of the initial strategy according to the total reward value and the target reward expectation to obtain the updated strategy;

[0011] Determine the total value of the update reward within the second preset time according to the update strategy and reward rules, and compare the total value of the update reward with the preset reward threshold range to obtain a comparison result;

[0012] According to the comparison results, the parameters of the update strategy are adjusted to obtain the target strategy, and the target user content is determined according to the target strategy;

[0013] Cache the target user content.

[0014] In this embodiment, according to the initial strategy, the corresponding storage identifier is determined from the identifier space, and according to the storage identifier and the state space, the new state is determined, including:

[0015] Determine the historical user content corresponding to the storage identifier, and determine the corresponding state of the historical user content in the state space;

[0016] The corresponding state is determined as the new state.

[0017] In this embodiment, according to the new state and the reward rule, determining the total reward value of the initial strategy within the first preset time includes:

[0018] Determine a first preset time, and within the first preset time, repeatedly execute the steps of determining the target storage identifier from the identifier space according to the initial strategy, and determining the new state according to the target storage identifier and the state space, to obtain all new states within the first preset time;

[0019] According to the reward rules, the reward values ​​corresponding to all new states are determined, and based on the reward values, the total reward value is determined.

[0020] In this embodiment, the target reward expectation is determined, and the parameters of the initial strategy are adjusted according to the total reward value and the target reward expectation to obtain an updated strategy, including:

[0021] According to the initial strategy and identification space, the states in the state space are traversed to obtain all states, and the reward values ​​corresponding to all states are determined according to the reward rules;

[0022] Determine the total reward value based on all reward values, and determine the expected value corresponding to the total reward value;

[0023] Based on the expected value, determine the target reward expectation;

[0024] According to the total reward value and the target reward expectation, the parameters of the initial strategy are adjusted to obtain the updated strategy.

[0025] In this embodiment, according to the total reward value and the target reward expectation, the parameters of the initial strategy are adjusted to obtain an updated strategy, including:

[0026] Determine the loss function of the total reward value and the expected target reward, and determine the loss value corresponding to the loss function;

[0027] Determine the gradient value of the loss function, and adjust the parameters of the initial strategy according to the gradient value to obtain the adjusted strategy parameters;

[0028] According to the adjusted policy parameters, the update strategy is determined.

[0029] In this embodiment, according to the comparison result, the parameters of the update strategy are adjusted to obtain the target strategy, and according to the target strategy, the target user content is determined, including:

[0030] Determine the comparison result of updating the total reward value and the reward threshold range;

[0031] If the comparison result shows that the total value of the update reward does not exceed the reward threshold range, the parameters of the update strategy are not adjusted, and the update strategy is determined as the target strategy;

[0032] If the comparison result is that the total updated reward value exceeds the reward threshold range, the loss value of the total updated reward value and the target reward expectation is determined, and the gradient value of the loss value is determined;

[0033] According to the gradient value, the parameters of the update strategy are adjusted to obtain an adjusted update strategy, and according to the adjusted update strategy and reward rule, the total value of the update reward within the next second preset time is determined;

[0034] Repeat the steps of determining the comparison result of the total update reward value and the reward threshold range until the parameters of the update strategy are adjusted according to the gradient value to obtain an adjusted update strategy, and determining the total update reward value within the next second preset time according to the adjusted update strategy and the reward rule, until the adjusted update strategy is determined to be the target strategy;

[0035] Determine the target user content based on the target strategy.

[0036] In this embodiment, determining target user content according to the target strategy includes:

[0037] Determine a target strategy, determine a target storage identifier from the identifier space, and determine a target state corresponding to the target storage identifier;

[0038] The historical user content included in the target state is determined, and the target user content is determined based on the historical user content.

[0039] In a second aspect, the present application provides a content caching device, the device comprising:

[0040] A preset determination module is used to determine a preset state space, an identification space, an initial strategy, and a reward rule. The state in the state space includes historical user content. The identification space is used to store the storage identification of the historical user content in the storage space. The initial strategy is used to determine the storage identification corresponding to the current next state. The reward rule is used to determine the reward value of the state.

[0041] A state determination module, used to determine the corresponding storage identifier from the identifier space according to the initial strategy, and determine the new state according to the storage identifier and the state space;

[0042] A total value determination module, used to determine the total value of the reward of the initial strategy within a first preset time according to the new state and the reward rule;

[0043] The strategy determination module is used to determine the target reward expectation and adjust the parameters of the initial strategy according to the total reward value and the target reward expectation to obtain an updated strategy;

[0044] A result determination module, used to determine the total value of the update reward within the second preset time according to the update strategy and the reward rule, and compare the total value of the update reward with a preset reward threshold range to obtain a comparison result;

[0045] A content determination module is used to adjust the parameters of the update strategy according to the comparison result, obtain the target strategy, and determine the target user content according to the target strategy;

[0046] The cache module is used to cache the target user content.

[0047] In a third aspect, the present application provides a device, comprising: a processor, and a memory communicatively connected to the processor;

[0048] Memory stores computer-executable instructions;

[0049] The processor executes the computer-executable instructions stored in the memory to implement the method of the present application.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method of the present application.

[0051] The content caching method, apparatus, device and medium provided by the present application determine a preset state space, an identification space, an initial strategy and a reward rule, wherein the state in the state space includes historical user content, the identification space is used to store the storage identification of the historical user content in the storage space, the initial strategy is used to determine the storage identification corresponding to the current next state, and the reward rule is used to determine the reward value of the state; according to the initial strategy, the corresponding storage identification is determined from the identification space, and the new state is determined according to the storage identification and the state space; according to the new state and the reward rule, the total reward value of the initial strategy within a first preset time is determined; the target reward expectation is determined, and according to the total reward value and the target reward expectation, the parameters of the initial strategy are adjusted to obtain an updated strategy; according to the updated strategy and the reward rule, the total updated reward value within a second preset time is determined, and the updated reward total value is compared with the preset reward threshold range to obtain a comparison result; according to the comparison result, the parameters of the updated strategy are adjusted to obtain a target strategy, and according to the target strategy, the target user content is determined; and the target user content is cached.

[0052] In this way, through the mobile edge caching algorithm based on meta-reinforcement learning, the time-varying characteristics of the popularity of user requested content are effectively responded to, and the accuracy of user content in the edge cache is improved, so that when users request to obtain content, a fast and accurate response can be achieved, which reduces the user's content acquisition delay, improves the accuracy of the content, and better satisfies the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] Figure 1 A flowchart of a content caching method provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of a content caching method provided in an embodiment of the present application;

[0056] Figure 3 A diagram of a model training process of a content caching method provided in an embodiment of the present application;

[0057] Figure 4 A model testing process diagram of a content caching method provided in an embodiment of the present application;

[0058] Figure 5 A flowchart of another content caching method provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of the structure of a content caching device provided in an embodiment of the present application;

[0060] Figure 7 This is a structural block diagram of a device for executing a content caching method according to an embodiment of the present application.

[0061] The above-mentioned drawings have shown clear embodiments of the present application, which will be described in more detail below; these drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0062] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0063] Edge caching is a caching technology deployed at the edge of the network. It stores frequently used data and content close to the user to improve access speed and reduce network traffic. When a user requests data or content, the edge cache can respond quickly and return it to the user without having to access a remote server or download the content from the network again; edge caching may be implemented differently, but the basic principle is to improve access speed and reduce network traffic by storing frequently used data and content close to the user.

[0064] Markov Decision Process (MDP) is a mathematical model of sequential decision making, which is used to simulate the random strategies and rewards that can be achieved by intelligent agents in an environment where the system state has Markov properties. MDP is constructed based on a set of interacting objects, namely, intelligent agents and environments, and its elements include states, actions, strategies, and rewards. In the simulation of MDP, the intelligent agent will perceive the current system state, take actions on the environment according to the strategy, thereby changing the state of the environment and receiving rewards. The accumulation of rewards over time is called returns, and the future state depends only on the current state and has nothing to do with the past state.

[0065] In the current research on edge caching, the cache is located at the edge base station or D2D user terminal. The storage capacity of these cache devices is limited and they cannot cache all possible content. They can only cache content that is likely to be requested by future users. Content popularity reflects the popularity of content within a certain area over a period of time. However, with the continuous enrichment and diversification of network content, the continuous strengthening of user mobility, and the high time-variability of user request preferences, prediction-based methods are difficult to cope with high time-variability content changes. Although deep reinforcement learning methods do not require prediction, they can directly change the cache strategy through environmental feedback. However, in a dynamically changing environment, before the environmental attributes change, the agent can only interact with the environment for a limited number of times, and the sample efficiency of deep reinforcement learning methods is low, and it is difficult to adjust the strategy to adapt to environmental changes through samples generated by limited interactions.

[0066] In order to solve the above problems, an embodiment of the present application proposes a content caching method, which divides the long-term mobile edge caching environment into several sub-environments with shorter time scales, thereby defining the Markov decision process problem within each sub-environment as a task, and defining the corresponding state space, action space, and reward rules, so that according to the interaction between the base station and the state environment, the new state of the base station in the state space is determined, and the reward value corresponding to the new state is determined according to the reward rule, so that the policy parameters of the initial policy are updated according to the reward value to obtain the target policy, so as to determine the next state corresponding to the base station in the state space according to the target policy, and determine the user content corresponding to the new state, so as to perform edge caching on the user content, thereby achieving rapid response to the content requested by the user, and improving the accuracy of the edge cache content.

[0067] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] Figure 1 A flow chart of a content caching method provided in an embodiment of the present application. Figure 1 As shown, the content caching method may include the following steps:

[0069] S110. Determine a preset state space, identification space, initial strategy, and reward rules. The state in the state space includes historical user content. The identification space is used to store the storage identifier of the historical user content in the storage space. The initial strategy is used to determine the storage identifier corresponding to the current next state. The reward rule is used to determine the reward value of the state.

[0070] Among them, the state space can be understood as the set of all possible states, that is, the state space is a set that contains all possible states in the environment; the state can be expressed as in, is the estimated content popularity information, is the user's historical request content, that is, a state can be understood as a historical request content; all historical request contents obtained by a base station constitute the state space corresponding to the base station, which can be expressed as O i , all the historical request contents of users acquired by all base stations constitute the global state space corresponding to all base stations, expressed as S, S = O1 × O2 × ... × O N .

[0071] The identification space can also be understood as the action space. The identification can be understood as the index identification of the historical request content in the cache space, which is used to determine the corresponding historical request content in the space. The action space is a set that includes all actions that can be performed in a specific state. The action can be to replace the user's historical request content stored in the storage space of the current edge cache with other historical request content. For example, each base station caches W content, and the indexes of these contents in the cache space are {1,…,W}. The action of each base station is to replace the historical request content in the storage space with the newly determined historical request content. Then the action space of the base station can be expressed as A i =[0,W], cache update action of the base station when When , the action of the base station is to determine the corresponding historical request content according to the index in the cache space and replace the historical request content in the edge cache storage space; When , it means no replacement is performed; the global action space composed of all actions of all base stations can be expressed as A = A1 × A2 × ... × A N .

[0072] The strategy is used by the base station to determine the corresponding action from the action space according to the current state, so as to execute the action and obtain a new state. The initial strategy can be set by the user or randomly generated by the base station, so that the target strategy can be obtained by adjusting the strategy parameters of the initial strategy later. The base station determines the new state and its corresponding target historical request content according to the target strategy, and replaces the currently cached historical request content in the edge cache storage space with the target historical request content, thereby improving the prediction accuracy of the content cache, and realizing that when the user requests to obtain content, the requested content required by the user can be provided quickly and accurately, thereby improving user satisfaction.

[0073] The reward rule can be understood as a reward function. Minimizing the global average user acquisition time delay of the edge network system is the common goal pursued by all base stations. Therefore, all base stations share a reward function, which can be expressed as Among them, NU represents the total number of users, N represents the number of base stations, and U represents the number of users under each base station. Represents the acquisition time delay of the uth user in base station i at time t+1, that is, the time it takes for the user to initiate a request and obtain data. It is used to measure the global average user acquisition time delay by calculating the average of the negative values ​​of the acquisition time delay of all users in all base stations. The goal of the base station is to minimize this global average user acquisition time delay, so the larger the value of the reward function, the better the system performance.

[0074] Based on this, by determining the preset state space, identification space, initial strategy, and reward rules, the edge cache environment with a longer time scale and high time variability can be divided into several short-term, relatively stable sub-environments, and the Markov decision process problem within each sub-environment is defined as a task, and different tasks are obtained, thereby realizing a mobile edge cache algorithm based on meta-reinforcement learning. According to the interaction between the base station and the preset task, the target historical request content to be replaced is determined.

[0075] S120: According to the initial strategy, determine the corresponding storage identifier from the identifier space, and determine the new state according to the storage identifier and the state space.

[0076] The storage identifier is an identifier of the historical request content in the storage space, and can also be understood as a content storage index, which is used to determine the corresponding historical request content.

[0077] The new state is the action determined from the action space according to the initial strategy, and the corresponding new state after executing the action.

[0078] Based on this, the initial strategy is used to determine the action corresponding to the current state from the preset action space, that is, to determine the corresponding new storage identifier, so as to execute the action and obtain a new state corresponding to the new storage identifier, so as to subsequently determine the reward value corresponding to the new state according to the preset reward rules, and then adjust the strategy parameters of the initial strategy according to the reward value to determine the target strategy.

[0079] S130: Determine the total reward value of the initial strategy within the first preset time according to the new state and reward rules.

[0080] The first preset time is a preset time period, which can be understood as the supporting stage of the task.

[0081] The total reward value is the total reward value determined by determining and executing actions from the action space through the initial strategy within the first preset time period, thereby continuously changing the current state, and determining the reward value corresponding to each new state according to the reward rules, thereby determining the total reward value based on the reward values ​​corresponding to all new states.

[0082] Based on this, by continuously determining and executing actions according to the initial strategy within the first preset time period, the new state is continuously changed, and the total reward value of the new state is determined according to the reward rules, so that the strategy of the initial strategy can be adjusted according to the total reward value to obtain the target strategy.

[0083] S140, determining the target reward expectation, and adjusting the parameters of the initial strategy according to the total reward value and the target reward expectation to obtain an updated strategy.

[0084] The target reward expectation is a predetermined reward expectation used to adjust the strategy parameters, and can also be understood as a target reward value.

[0085] Based on this, by determining the target reward expectation, the strategy parameters of the initial strategy are adjusted according to the total reward value and the target reward expectation of the initial strategy within the first preset time to obtain an adjusted updated strategy.

[0086] S150. Determine the total value of the update reward within the second preset time according to the update strategy and reward rules, and compare the total value of the update reward with the preset reward threshold range to obtain a comparison result.

[0087] The second preset time is a preset time period, which can be understood as the query phase of the task.

[0088] The total reward value is updated within the second preset time period. The action is determined and executed from the action space by updating the strategy, thereby continuously changing the current state, and determining the reward value corresponding to each new state according to the reward rule, thereby determining the total reward value according to the reward values ​​corresponding to all new states.

[0089] The preset reward threshold range is a pre-set threshold range corresponding to the total reward value. By comparing the updated total reward value with the reward threshold range, it is determined whether the updated total reward value exceeds the reward threshold range, and the corresponding parameter adjustment method is determined based on the comparison result.

[0090] Based on this, by continuously determining and executing actions according to the update strategy within the second preset time period, the new state is continuously changed, and the total update reward value of the new state is determined according to the reward rules, and the total update reward value is compared with the preset reward threshold range, so as to subsequently determine the corresponding update strategy parameter adjustment method based on the comparison results.

[0091] S160: According to the comparison result, the parameters of the update strategy are adjusted to obtain a target strategy, and according to the target strategy, the target user content is determined.

[0092] The target policy may be understood as the finalized user historical request content that needs to be stored in the edge cache storage space; the target user content is the historical request content that needs to be stored in the edge cache storage space.

[0093] Based on this, the policy parameters of the update policy are adjusted through the comparison result, so as to obtain the target policy, so as to determine the target user content to be stored according to the target policy.

[0094] S170: Cache the target user content.

[0095] Based on this, by storing the target user content in the edge cache storage space, the prediction accuracy of the content that the user will request is improved, so that when the user requests to obtain the historical request content, the user's needs can be accurately met based on the content pre-stored in the edge cache storage space, thereby improving the speed and accuracy of user response.

[0096] Based on the feasible implementation of S120 above, the present application further provides a process of determining the storage identifier according to the initial strategy to determine the current corresponding new state:

[0097] Determine the historical user content corresponding to the storage identifier, and determine the corresponding state of the historical user content in the state space;

[0098] The corresponding state is determined as the new state.

[0099] Based on this, since the state in the state space includes historical request content, and the storage identifier represents the storage index of the historical request content in the storage space, the corresponding historical request content can be determined through the storage identifier, and then the corresponding state can be determined, and the state can be determined as the new state obtained after the action is executed.

[0100] Based on the feasible implementation of S130 above, the present application further provides a process of determining the reward value of the new state through the reward rule, and determining the corresponding total reward value according to all states determined within the first preset time:

[0101] Determine a first preset time, and within the first preset time, repeatedly execute the steps of determining the target storage identifier from the identifier space according to the initial strategy, and determining the new state according to the target storage identifier and the state space, to obtain all new states within the first preset time;

[0102] According to the reward rules, the reward values ​​corresponding to all new states are determined, and based on the reward values, the total reward value is determined.

[0103] Based on this, by continuously determining and executing actions according to the initial strategy within the first preset time, the corresponding new state is obtained, and the reward value corresponding to each new state is determined according to the reward rules, thereby determining the total reward value.

[0104] Based on the feasible implementation of S140 above, the present application further provides a process of determining the reward values ​​corresponding to all states by traversing all states in the state space, and further determining the target reward expectation:

[0105] According to the initial strategy and identification space, the states in the state space are traversed to obtain all states, and the reward values ​​corresponding to all states are determined according to the reward rules;

[0106] Determine the total reward value based on all reward values, and determine the expected value corresponding to the total reward value;

[0107] Based on the expected value, determine the target reward expectation;

[0108] According to the total reward value and the target reward expectation, the parameters of the initial strategy are adjusted to obtain the updated strategy.

[0109] Among them, traversal means to continuously determine the storage identifier and its corresponding historical request content from the identifier space according to the initial strategy, so as to determine the new state corresponding to the historical request content, and continuously realize the state transfer until all the states in the state space are determined.

[0110] The expected value is the mathematical expectation, which is the probability of each possible outcome in the experiment multiplied by the sum of its results. It is one of the most basic mathematical characteristics, which reflects the average value of the random variable. The expected value corresponding to the total reward value obtained by traversing all states is determined as the target reward expectation, so that the strategy parameters can be adjusted later according to the total reward value and the target reward expectation.

[0111] Based on this, by traversing all states in the state space, the reward values ​​corresponding to all states are determined according to the reward rules, and the target reward expectation is determined according to the expected total reward value, so as to adjust the parameters of the initial strategy later.

[0112] Based on the feasible implementation of S140 above, the present application further provides a process of adjusting the strategy parameters through the gradient value of the loss function:

[0113] Determine the loss function of the total reward value and the expected target reward, and determine the loss value corresponding to the loss function;

[0114] Determine the gradient value of the loss function, and adjust the parameters of the initial strategy according to the gradient value to obtain the adjusted strategy parameters;

[0115] According to the adjusted policy parameters, the update strategy is determined.

[0116] Among them, the loss function is a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event; in applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function; that is, the smaller the loss value between the total reward value and the target reward expectation, the closer the total reward value is to the target reward expectation; for example, the loss function can be expressed as

[0117] The gradient value is the value obtained by calculating the gradient of the loss value. For example, the gradient value can be

[0118]

[0119] The adjusted strategy parameters can be expressed as

[0120] Based on this, by determining the loss value between the total reward value and the target reward expectation and calculating the gradient of the loss value, the adjusted strategy parameters are determined according to the strategy parameters and the gradient value to obtain the updated strategy.

[0121] Based on the feasible implementation of S160 above, the present application further provides a process of determining a corresponding target strategy by determining whether the total value of the updated reward is within a reward threshold range:

[0122] Determine the comparison result of updating the total reward value and the reward threshold range;

[0123] If the comparison result shows that the total value of the update reward does not exceed the reward threshold range, the parameters of the update strategy are not adjusted, and the update strategy is determined as the target strategy;

[0124] If the comparison result is that the total updated reward value exceeds the reward threshold range, the loss value of the total updated reward value and the target reward expectation is determined, and the gradient value of the loss value is determined;

[0125] According to the gradient value, the parameters of the update strategy are adjusted to obtain an adjusted update strategy, and according to the adjusted update strategy and reward rule, the total value of the update reward within the next second preset time is determined;

[0126] Repeat the steps of determining the comparison result of the total update reward value and the reward threshold range until the parameters of the update strategy are adjusted according to the gradient value to obtain an adjusted update strategy, and determining the total update reward value within the next second preset time according to the adjusted update strategy and the reward rule, until the adjusted update strategy is determined to be the target strategy;

[0127] Determine the target user content based on the target strategy.

[0128] Among them, the loss value is the loss value determined by the loss function based on the total updated reward value and the target reward expectation. The loss function can be expressed as The gradient value is the value obtained by calculating the gradient of the loss value.

[0129] The adjusted update strategy can be expressed as

[0130] Based on this, by comparing the updated reward total value with the reward threshold range, it is determined whether the updated reward total value is within the reward threshold range. If it is already within the range, it represents that the policy parameters of the update strategy meet the preset parameter conditions, and the updated strategy can be used as the target strategy; if it is not within the range, it represents the need to continue to adjust the policy parameters of the update strategy within the second preset time, that is, by continuously determining the gradient value corresponding to the updated reward total value and the expected loss value of the target reward, and adjusting the policy parameters of the update strategy according to the gradient value, until the updated reward total value is determined to be within the reward threshold range. The update strategy is determined as the target strategy.

[0131] Based on the feasible implementation of S160 above, the present application further provides a process of determining a target storage identifier through a target policy, thereby determining the corresponding target user content according to the target storage identifier:

[0132] Determine a target strategy, determine a target storage identifier from the identifier space, and determine a target state corresponding to the target storage identifier;

[0133] The historical user content included in the target state is determined, and the target user content is determined based on the historical user content.

[0134] Based on this, by determining the target strategy, the target storage identifier is determined from the identifier space according to the target strategy, thereby determining the state corresponding to the target storage identifier, and determining the state as the new state. The state transfer is realized according to the target strategy, and the content stored in the edge cache storage space is replaced according to the historical user content corresponding to the new state, thereby achieving accurate prediction of the user request content.

[0135] See also Figure 2 , Figure 2 A schematic diagram of a content caching method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, meta-reinforcement learning can be used to determine the mobile edge cache in a non-stationary environment, where S is the first preset time of the task, i.e., the support phase, and Q is the second preset time, i.e., the query phase. S and the subsequent connected Q constitute the task time of a task.

[0136] See also Figure 3 , Figure 3 A model training process diagram of a content caching method provided in an embodiment of the present application, such as Figure 3As shown in the figure, in the meta-training phase, M tasks Ti (i = 1, 2, ... M) are randomly selected. In the support phase of each task, the initial strategy and the environment interact to generate a set of K trajectories, and the proxy loss function of the support phase is calculated. Then, the initial strategy is fine-tuned based on the VPG learner, as shown in formula (1). The updated strategy calculates the loss in the query phase of the task, and the losses in the query phase of the M tasks are summed. The initial strategy is updated based on the PPO meta-learner. The above operation is repeated to continuously update the initial strategies of multiple tasks until the cumulative profit of the strategy on the query data after fine-tuning on all tasks tends to be stable. The training is stopped and the optimized initial trading strategy is used in the test phase.

[0137] See also Figure 4 , Figure 4 A model test process diagram of a content caching method provided in an embodiment of the present application, such as Figure 4 As shown, the above-trained meta-reinforcement learning mobile edge cache model can be used to test in the meta-testing phase.

[0138] In this embodiment, in order to solve the problem that the user request preference presents the characteristic of high time variability and the prediction-based method is difficult to cope with the problem of high time variability of content changes, the long-term mobile edge cache environment can be divided into several sub-environments with shorter time scales, and the Markov decision process problem in each sub-environment is defined as a task, so that the action is determined and executed according to the initial strategy through the preset state space, identification space, i.e., action space, initial strategy, and reward rule, so as to obtain a new state, and the reward value corresponding to the new state is determined according to the reward rule, so as to determine the total reward value of all states within the first preset time, and determine the target reward expectation corresponding to the total reward value by traversing all states, so as to determine the gradient value corresponding to the loss value of the total reward value and the target reward expectation, and adjust the policy parameters according to the gradient value, so as to determine the update strategy, so as to compare the updated reward total value corresponding to the update strategy with the preset reward threshold range, so as to adjust the parameters of the update strategy according to the comparison result, and determine the target strategy.

[0139] In this way, by determining the target strategy, the target storage identifier is determined from the identification space according to the target strategy, and the corresponding state and its historical request content are determined according to the target storage identifier, so that the historical request content is determined as the target historical request content and stored in the edge cache storage space, thereby improving the accuracy of prediction of the content that the user will request to obtain, and achieving that when the user requests to obtain the historical request content, the user's needs can be accurately met according to the content pre-stored in the edge cache storage space, thereby improving the speed and accuracy of user response.

[0140] Figure 5A flowchart of another content caching method provided in an embodiment of the present application. Figure 5 As shown, the content caching method may include the following steps:

[0141] S510. Divide the long-term mobile edge cache environment into sub-environments with shorter time scales. Due to the short time scale of each sub-environment, the Markov decision process problem within each sub-environment is defined as a task.

[0142] Based on this, different tasks are obtained, and each task is divided into a support phase and a query phase in chronological order.

[0143] S520: construct a meta-reinforcement learning model, which consists of a learner and a meta-learner.

[0144] Among them, the meta-reinforcement learning model can be understood as an intelligent agent in the Markov decision process, using the PPO meta-learner to learn the common initial strategies for multiple tasks.

[0145] The learner is the VPG (Vanilla Policy Gradients) learner, and the meta-learner is the PPO (Proximal Policy Optimization) meta-learner.

[0146] S530, using the trained meta-learner in the support phase of the test phase, optimizing and updating the learner parameters of the initialized model, and then performing performance evaluation on the updated learner parameters on the query set of the test set.

[0147] Based on this, the optimization update is to determine the multiple states and their corresponding reward values ​​obtained by the interaction between the model and the environment in the support phase, so as to determine the total reward value and the expected loss value of the target reward and calculate the gradient, so as to fine-tune the initial strategy based on the support data of each task.

[0148] Performance evaluation is to compare the total reward value of the fine-tuned strategy in the query phase with the preset reward range to determine whether the total reward value of the fine-tuned strategy is stable. In this embodiment, in order to take into account the time-varying characteristics of content popularity, a mobile edge cache model based on meta-reinforcement learning can be constructed, which consists of a VPG learner and a PPO meta-learner. In the training phase, multiple tasks are randomly selected. For each task, the total reward value and the expected loss value of the target reward are determined and the gradient is calculated by determining the multiple states and their corresponding reward values ​​obtained by the model interacting with the environment in the support phase, so as to fine-tune the initial strategy on the support data of each task, and then use the fine-tuned strategy to calculate the loss on the query data of the task, and finally calculate the average proxy loss on the query data, so as to update the initial strategy parameters based on the PPO meta-learner and obtain the target strategy.

[0149] Figure 6 A structural diagram of a content caching device 600 provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the content caching device 600 includes: a preset determination module 610, a state determination module 620, a total value determination module 630, a strategy determination module 640, a result determination module 650, a content determination module 660, and a cache module 670.

[0150] The preset determination module 610 is used to determine the preset state space, identification space, initial strategy, and reward rules. The state in the state space includes historical user content. The identification space is used to store the storage identification of the historical user content in the storage space. The initial strategy is used to determine the storage identification corresponding to the current next state. The reward rule is used to determine the reward value of the state.

[0151] A state determination module 620, for determining a corresponding storage identifier from the identifier space according to the initial strategy, and determining a new state according to the storage identifier and the state space;

[0152] A total value determination module 630, for determining the total value of the reward of the initial strategy within the first preset time according to the new state and the reward rule;

[0153] A strategy determination module 640 is used to determine the target reward expectation and adjust the parameters of the initial strategy according to the total reward value and the target reward expectation to obtain an updated strategy;

[0154] A result determination module 650 is used to determine the total value of the update reward within the second preset time according to the update strategy and the reward rule, and compare the total value of the update reward with the preset reward threshold range to obtain a comparison result;

[0155] The content determination module 660 is used to adjust the parameters of the update strategy according to the comparison result to obtain the target strategy, and determine the target user content according to the target strategy;

[0156] The cache module 670 is used to cache target user content.

[0157] In the embodiment of the present application, the state determination module 620 may also be specifically used for:

[0158] Determine the historical user content corresponding to the storage identifier, and determine the corresponding state of the historical user content in the state space;

[0159] The corresponding state is determined as the new state.

[0160] In the embodiment of the present application, the total value determination module 620 may also be specifically used for:

[0161] Determine a first preset time, and within the first preset time, repeatedly execute the steps of determining the target storage identifier from the identifier space according to the initial strategy, and determining the new state according to the target storage identifier and the state space, to obtain all new states within the first preset time;

[0162] According to the reward rules, the reward values ​​corresponding to all new states are determined, and based on the reward values, the total reward value is determined.

[0163] In the embodiment of the present application, the policy determination module 640 may also be specifically used for:

[0164] According to the initial strategy and identification space, the states in the state space are traversed to obtain all states, and the reward values ​​corresponding to all states are determined according to the reward rules;

[0165] Determine the total reward value based on all reward values, and determine the expected value corresponding to the total reward value;

[0166] Based on the expected value, determine the target reward expectation;

[0167] According to the total reward value and the target reward expectation, the parameters of the initial strategy are adjusted to obtain the updated strategy.

[0168] In the embodiment of the present application, the policy determination module 640 may also be specifically used for:

[0169] Determine the loss function of the total reward value and the expected target reward, and determine the loss value corresponding to the loss function;

[0170] Determine the gradient value of the loss function, and adjust the parameters of the initial strategy according to the gradient value to obtain the adjusted strategy parameters;

[0171] According to the adjusted policy parameters, the update strategy is determined.

[0172] In the embodiment of the present application, the content determination module 660 may also be specifically used for:

[0173] Determine the comparison result of updating the total reward value and the reward threshold range;

[0174] If the comparison result shows that the total value of the update reward does not exceed the reward threshold range, the parameters of the update strategy are not adjusted, and the update strategy is determined as the target strategy;

[0175] If the comparison result is that the total updated reward value exceeds the reward threshold range, the loss value of the total updated reward value and the target reward expectation is determined, and the gradient value of the loss value is determined;

[0176] According to the gradient value, the parameters of the update strategy are adjusted to obtain an adjusted update strategy, and according to the adjusted update strategy and reward rule, the total value of the update reward within the next second preset time is determined;

[0177] Repeat the steps of determining the comparison result of the total update reward value and the reward threshold range until the parameters of the update strategy are adjusted according to the gradient value to obtain an adjusted update strategy, and determining the total update reward value within the next second preset time according to the adjusted update strategy and the reward rule, until the adjusted update strategy is determined to be the target strategy;

[0178] Determine the target user content based on the target strategy.

[0179] In the embodiment of the present application, the content determination module 660 may also be specifically used for:

[0180] Determine a target strategy, determine a target storage identifier from the identifier space, and determine a target state corresponding to the target storage identifier;

[0181] The historical user content included in the target state is determined, and the target user content is determined based on the historical user content.

[0182] Figure 7 This is a schematic diagram of the structure of the device provided in the embodiment of the present application. Figure 7 As shown, the device 700 includes:

[0183] The device 700 may include a processor 701 with one or more processing cores, a memory 702 with one or more computer-readable storage media, a communication component 703 and other components. The processor 701 , the memory 702 and the communication component 703 are connected via a bus 704 .

[0184] In a specific implementation process, at least one processor 701 executes the computer execution instructions stored in the memory 702, so that at least one processor 701 executes the above message processing method.

[0185] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0186] In the above Figure 7In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0187] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0188] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0189] In some embodiments, a computer program product is further provided, including a computer program or instructions, which implement the steps in any of the above-mentioned content caching methods when executed by a processor.

[0190] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0191] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0192] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of computer-executable instructions are stored. The computer-executable instructions can be loaded by a processor to execute the steps in any one of the content caching methods provided in the embodiments of the present application.

[0193] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0194] According to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program comprises computer instructions stored in a computer-readable storage medium.

[0195] Since the instructions stored in the storage medium can execute the steps in any content caching method provided in the embodiments of the present application, the beneficial effects that can be achieved by any content caching method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0196] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0197] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A content caching method, characterized in that: The method comprises: Determine a preset state space, an identification space, an initial strategy, and a reward rule, wherein the state in the state space includes historical user content, the identification space is used to store the storage identification of the historical user content in the storage space, the initial strategy is used to determine the storage identification corresponding to the current next state, and the reward rule is used to determine the reward value of the state; According to the initial strategy, determining a corresponding storage identifier from the identifier space, and determining a new state according to the storage identifier and the state space; Determine the total reward value of the initial strategy within a first preset time according to the new state and the reward rule; Determine a target reward expectation, and adjust the parameters of the initial strategy according to the total reward value and the target reward expectation to obtain an updated strategy; Determine the total value of the update reward within the second preset time according to the update strategy and the reward rule, and compare the total value of the update reward with a preset reward threshold range to obtain a comparison result; According to the comparison result, adjusting the parameters of the update strategy to obtain a target strategy, and determining the target user content according to the target strategy; The target user content is cached.

2. The method according to claim 1, characterized in that The step of determining a corresponding storage identifier from the identifier space according to the initial strategy, and determining a new state according to the storage identifier and the state space includes: Determine the historical user content corresponding to the storage identifier, and determine a corresponding state of the historical user content in the state space; The corresponding state is determined as the new state.

3. The method according to claim 1, characterized in that Determining the total reward value of the initial strategy within the first preset time according to the new state and the reward rule includes: Determine the first preset time, and repeat the steps of determining the target storage identifier from the identifier space according to the initial strategy, and determining the new state according to the target storage identifier and the state space within the first preset time, to obtain all the new states within the first preset time; According to the reward rule, the reward values ​​corresponding to all the new states are determined, and according to the reward values, the total reward value is determined.

4. The method according to claim 1, characterized in that: The determining of the target reward expectation and adjusting the parameters of the initial strategy according to the total reward value and the target reward expectation to obtain an updated strategy include: According to the initial strategy and the identification space, the states in the state space are traversed to obtain all the states, and according to the reward rule, the reward values ​​corresponding to all the states are determined; Determine a total reward value based on all reward values, and determine an expected value corresponding to the total reward value; Determining the target reward expectation according to the expected value; According to the total reward value and the target reward expectation, the parameters of the initial strategy are adjusted to obtain an updated strategy.

5. The method according to claim 4, characterized in that The adjusting the parameters of the initial strategy according to the total reward value and the target reward expectation to obtain an updated strategy includes: Determine a loss function of the total reward value and the target reward expectation, and determine a loss value corresponding to the loss function; Determining a gradient value of the loss function, and adjusting the parameters of the initial strategy according to the gradient value to obtain the adjusted strategy parameters; The update strategy is determined according to the adjusted strategy parameters.

6. The method according to claim 1, characterized in that The step of adjusting the parameters of the update strategy according to the comparison result to obtain a target strategy, and determining target user content according to the target strategy includes: Determine the comparison result of the updated reward total value and the reward threshold range; If the comparison result is that the total value of the update reward does not exceed the reward threshold range, then the parameters of the update strategy are not adjusted, and the update strategy is determined as the target strategy; If the comparison result is that the updated reward total value exceeds the reward threshold range, determining a loss value between the updated reward total value and the target reward expectation, and determining a gradient value of the loss value; According to the gradient value, the parameters of the update strategy are adjusted to obtain the adjusted update strategy, and according to the adjusted update strategy and the reward rule, the total value of the next update reward within the second preset time is determined; Repeating the steps of determining the comparison result of the updated reward total value and the reward threshold range to adjusting the parameters of the update strategy according to the gradient value to obtain the adjusted update strategy, and determining the next updated reward total value within the second preset time according to the adjusted update strategy and the reward rule, until the adjusted update strategy is determined to be the target strategy; According to the target strategy, target user content is determined.

7. The method according to claim 6, characterized in that Determining target user content according to the target strategy includes: Determine the target strategy, determine a target storage identifier from the identifier space, and determine a target state corresponding to the target storage identifier; The historical user content included in the target state is determined, and the target user content is determined according to the historical user content.

8. A content caching device, characterized in that: The device comprises: A preset determination module, used to determine a preset state space, an identification space, an initial strategy, and a reward rule, wherein the state in the state space includes historical user content, the identification space is used to store the storage identification of the historical user content in the storage space, the initial strategy is used to determine the storage identification corresponding to the current next state, and the reward rule is used to determine the reward value of the state; A state determination module, configured to determine a corresponding storage identifier from the identifier space according to the initial strategy, and determine a new state according to the storage identifier and the state space; A total value determination module, used to determine the total value of the reward of the initial strategy within a first preset time according to the new state and the reward rule; A strategy determination module, used to determine the target reward expectation, and adjust the parameters of the initial strategy according to the total reward value and the target reward expectation to obtain an updated strategy; A result determination module, used to determine the total value of the update reward within the second preset time according to the update strategy and the reward rule, and compare the total value of the update reward with a preset reward threshold range to obtain a comparison result; A content determination module, configured to adjust the parameters of the update strategy according to the comparison result to obtain a target strategy, and determine target user content according to the target strategy; A cache module is used to cache the target user content.

9. A device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions can be called by a processor to execute the method according to any one of claims 1 to 7.

Images