Cache decision model training method and device, equipment, medium and program product

Abstract

The invention provides a cache decision model training method and device, equipment, a medium and a program product, and relates to the technical field of quantum.The cache decision model training method includes the steps that according to a cache decision model state space, an action space and a reward function, through a quantum search algorithm, a Bellman equation of a reinforcement learning algorithm is iteratively calculated according to the state space, the action space and the reward function; each cache action corresponds to an accumulated reward value until the cache decision model converges; wherein under the condition that a target cache action in a first historical request state is executed in a current iteration round, if an accumulated reward value corresponding to the target cache action in the current iteration round is greater than or equal to a maximum accumulated reward value corresponding to all cache actions in the first historical request state in a previous iteration round, the target cache action in the first historical request state is executed in the previous iteration round; if yes, the probability of selecting the target cache action is improved through quantum operation. According to the method, the quantum search algorithm is utilized, the probability of follow-up action selection is improved, the convergence speed of the reinforcement learning algorithm is increased, and the training time of the cache decision model is shortened.

Description

Technical Field

[0001] The present application relates to the field of quantum technology, and in particular to a cache decision model training method, device, equipment, medium and program product. Background Art

[0002] Banking business data is the top priority of banking work. However, most of the current bank data is sent by branches to various outlets. Every day, staff spend a lot of time distributing data, and it is difficult to ensure the accuracy of the data. Today, 5G networks bring users a better mobile network experience and propose three major application scenarios, including enhanced mobile bandwidth, massive machine-type communications, and ultra-high reliability and low-latency communications. Today, more and more people enjoy the convenience of life brought by mobile Internet, and mobile Internet has completely changed people's lives and promoted the continuous progress of society. The edge caching technology in 5G networks is used as a core technology to reduce content transmission delays and alleviate network traffic burdens. When 5G networks are introduced to banks, the cumbersome steps of data distribution by branches can be alleviated.

[0003] Considering that the data required by users of each network point is different, and the primary concern of users when using wireless networks is how to obtain the required data as quickly as possible. Since the status of user request data is unknown, traditional mathematical optimization and game theory methods are not applicable to such realistic situations. Therefore, in the prior art, a data cache decision model is established based on the Markov decision process, and a reinforcement learning method is applied to solve it to complete the training of the cache decision model.

[0004] However, since the spatial dimension of the cache decision model solved using the reinforcement learning method is high, the reinforcement learning algorithm converges too slowly, making it difficult to actually deploy the cache decision model in a production environment. Summary of the invention

[0005] The present application provides a cache decision model training method, device, equipment, medium and program product to solve the technical problems in the prior art that the spatial dimension of the cache decision model solved by the reinforcement learning method is high, the reinforcement learning algorithm converges too slowly, and the cache decision model is difficult to actually deploy in a production environment.

[0006] In a first aspect, the present application provides a cache decision model training method, comprising:

[0007] A cache decision model is constructed according to multiple historical request state data of user requesting target data in multiple historical time periods, multiple cache action history data in each of the historical request states, and historical data of transmission delay of the user requesting the target data after executing each of the cache actions, wherein the cache action indicates whether the first base station caches the target data, and the cache decision model includes a state space, an action space, and a reward function, wherein the state space includes the multiple historical request states, and the action space includes the multiple cache action history data in each of the historical request states; the lower the transmission delay of the user requesting the target data, the greater the reward value of the reward function;

[0008] According to the state space, the action space, and the reward function, a quantum search algorithm is used to iteratively calculate the cumulative reward value corresponding to each cache action under each historical request state using the Bellman equation of the reinforcement learning algorithm until a cache action with the largest cumulative reward value corresponding to each historical request state is determined according to a preset convergence condition, wherein, in the case of executing a target cache action under the first historical request state in the current iteration round: if the cumulative reward value corresponding to the target cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all the cache actions under the first historical request state in the previous iteration round, the probability of selecting the target cache action is increased through quantum operation.

[0009] In a second aspect, the present application provides a cache decision method, which is implemented based on a cache decision model. The cache decision model is trained using the cache decision model training method described in the first aspect. The cache decision method includes:

[0010] Get the request status data of the user's requested target data;

[0011] Determining, by the cache decision model, a target cache action in the cache actions corresponding to the request state according to the request state, wherein the cache action indicates whether the first base station caches the target data;

[0012] Control the first base station to perform the target caching action.

[0013] In a third aspect, the present application provides a cache decision model training device, the device comprising:

[0014] A model generation module is used to: construct a cache decision model according to multiple historical request state data of user requesting target data in multiple historical time periods, multiple cache action history data under each of the historical request states, and historical data of transmission delay of the user requesting the target data after executing each of the cache actions, wherein the cache action indicates whether the first base station caches the target data, and the cache decision model includes a state space, an action space, and a reward function, wherein the state space includes the multiple historical request states, and the action space includes the multiple cache action history data under each of the historical request states; the lower the transmission delay of the user requesting the target data, the greater the reward value of the reward function;

[0015] A quantum computing module is used to: iteratively calculate the cumulative reward value corresponding to each cache action under each historical request state according to the state space, the action space, and the reward function through a quantum search algorithm and the Bellman equation of a reinforcement learning algorithm, until the cache action with the largest cumulative reward value corresponding to each historical request state is determined according to a preset convergence condition, wherein, in the case of executing a target cache action under the first historical request state in the current iteration round: if the cumulative reward value corresponding to the target cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all the cache actions under the first historical request state in the previous iteration round, the probability of selecting the target cache action is increased through quantum operation.

[0016] In a fourth aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0017] The memory stores computer-executable instructions;

[0018] The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

[0019] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0020] In a sixth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0021] The cache decision model training method, device, equipment, medium and program product provided by the present application, since the goal of the reinforcement learning algorithm is to find the cache action with the largest cumulative reward value corresponding to each historical request state in the cache decision model, after selecting a cache action under a historical request state in the current iteration round, if the cumulative reward value corresponding to the cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all cache actions under the historical request state in the previous iteration round, it means that the cache action selected in the current iteration round is a good action, otherwise it is a bad action. Since this method uses a quantum search algorithm and improves the probability of the intelligent agent selecting a good action through quantum operations, it can accelerate the convergence speed of the reinforcement learning algorithm and reduce the training time of the cache decision model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0023] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a cache decision model training method flow chart provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of the quantum search algorithm flow provided in the embodiment of the present application;

[0026] Figure 4 A schematic flow chart of a method for calculating a first reward value of a target cache action by using a reward function provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of a cache decision method flow chart provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of the structure of a cache decision model training device provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of the structure of the electronic device provided in this application.

[0030] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0032] It should be noted that the cache decision model training method, device, equipment, medium and program product provided in this application can be used in the field of quantum technology, and can also be used in any field outside the field of quantum technology. The application field of the cache decision model training method, device, equipment, medium and program product in this application is not limited.

[0033] First, the nouns involved in this application are explained in detail:

[0034] Markov decision process: Markov decision process is a mathematical framework for modeling decision problems in which an agent maximizes a certain cumulative reward by taking a series of actions in a random environment. MDP consists of four basic elements: a set of states, a set of actions, state transition probabilities, and a reward function. In MDP, future states depend only on the current state and action, and have nothing to do with past states and actions. This property is called "Markov property".

[0035] Popularity: Popularity generally refers to the degree to which something is widely accepted or used over a specific period of time. In different fields, popularity may have different specific definitions and measurement methods. For example, on social media, popularity can be measured by the number of likes, shares, or comments.

[0036] Reinforcement learning: Reinforcement learning is a machine learning method in which an agent learns how to take actions to maximize cumulative rewards by interacting with the environment. Unlike supervised learning, reinforcement learning does not rely on labeled data, but learns policies through trial and error and feedback (rewards or penalties).

[0037] Zipf distribution: Zipf distribution is a probability distribution that is often used to describe certain types of data sets where a few items appear very frequently, while most items appear very infrequently. It is a special form of power-law distribution and is often used to describe word frequency distribution in natural language, city population distribution, etc. According to Zipf's law, in a ranked frequency distribution, the frequency is inversely proportional to its rank.

[0038] Figure 1The following is a schematic diagram of an application scenario provided by an embodiment of the present application, such as Figure 1 As shown in the figure, taking banking business as an example, the system network model is a two-layer network architecture consisting of a macro base station of a bank branch and small base stations of N outlets within its coverage area. The small base stations of each outlet can share business data with each other through wireless transmission, which is called homogeneous cache. When the small base station layer cannot meet the needs of the outlet users, the macro base station will serve them, which is called heterogeneous cache. The set of small base stations at each outlet is represented as , the cache space of each small base station is .

[0039] Furthermore, the process of obtaining business data by branch users is as follows:

[0040] When the branch users have business data When making a request, the site user first goes to the small base station associated with its site Check whether the business data is cached in the local cache , set a binary cache variable Small base station Whether the business data is cached ,like Small base station Cached ,on the contrary Indicates that it is not cached;

[0041] If a small base station Not cached , users at each site need to obtain service data from small base stations at other sites through collaborative caching , through a binary indicator variable Can users at representative sites obtain service data from small base stations at other sites through collaborative caching? ;

[0042] If the branch users cannot obtain business data through the above two methods , then a request is made to the branch's macro base station, which then sends the service data to the branch's macro base station. Transmitted to the network users.

[0043] Therefore, the transmission delay of the network users through local cache, network cooperative communication and obtaining service data from the branch macro base station can be expressed as:

[0044]

[0045] Where n represents the number of all small base stations in the network, Indicates the popularity of business data, F represents all business data contained in the current network, and binary cache variables Small base station Whether the business data is cached , Can users at representative sites obtain service data from small base stations at other sites through collaborative caching? .also, Defined as getting business data from local cache The transmission delay of Indicates obtaining service data from small base stations at other sites The transmission delay of Indicates obtaining service data from branch macro base stations The transmission delay of ;

[0046] Furthermore, with minimizing the transmission delay of the network users to obtain the service data as the optimization goal, an optimization problem is established, and the optimization problem is expressed as follows:

[0047]

[0048] The restriction condition indicates the cache space limitation of each small base station.

[0049] As described in the background technology, in the prior art, Figure 1The cache decision of the system network model shown is usually to establish a data cache decision model based on the Markov decision process, and apply the reinforcement learning method to solve it, complete the training of the cache decision model, and then deploy the cache decision model to the production environment to complete the cache decision of business data. However, the spatial dimension of the cache decision model solved by the reinforcement learning method is high, and the reinforcement learning algorithm converges too slowly, making it difficult to actually deploy the cache decision model in the production environment. Therefore, in order to solve the problem of slow convergence of reinforcement learning algorithm, the technical concept of this application is to accelerate the convergence speed of reinforcement learning algorithm through quantum search algorithm. Specifically, first obtain multiple historical time periods: multiple historical request states of user request target data, and cache actions taken under each historical request state (indicating whether the base station caches the target data, and if so, the base station caches the target data), execute the historical data of transmission delay of user request target data after each cache action, build a cache decision model, and iteratively calculate the cumulative reward value of each cache action under each historical request state through quantum search algorithm and Bellman equation of reinforcement learning algorithm. During the iteration process, if a cache action under a historical request state is selected in the current iteration round, if the cumulative reward value corresponding to the cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all cache actions under the historical request state in the previous iteration round, then the probability of selecting the cache action is increased through quantum operation. Stop iteration when the cache decision model meets the convergence conditions.

[0050] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0051] An embodiment of the present application provides a cache decision model training method, which can be applied to a server. Figure 2 A schematic diagram of a cache decision model training method flow chart provided in an embodiment of the present application, such as Figure 2 As shown, the method includes:

[0052] S201, constructing a cache decision model according to multiple historical request status data of user request target data in multiple historical time periods, multiple cache action historical data under each historical request status, and historical data of transmission delay of user request target data after executing each cache action;

[0053] Specifically, the cache action indicates whether the first base station caches the target data. The cache decision model includes a state space, an action space, and a reward function. The state space includes multiple historical request state data of users requesting target data in multiple historical time periods, and the action space includes multiple cache action historical data under each historical request state. The lower the transmission delay of the user's request for target data, the greater the reward value of the reward function.

[0054] The historical request status data may be, for example, frequency data of user requests for target data in multiple historical time periods, or paths of user requests for target data, etc. Since the status of user requests for target data is affected by factors such as the popularity of the target data, there may be multiple historical request states in the above multiple historical periods. Similarly, the first base station in the embodiment of the present application may be understood as Figure 1 For small and medium-sized base stations, under different historical request states, the small base stations will also perform different caching actions. The caching action is for the small base station to cache the target data, or not cache the target data. After the small base station performs different caching actions, the transmission delay of the user's request for the target data will also change accordingly. In the actual business process, after the user requests the target data, the server will record the request status data of the user's request for the target data. Similarly, the server will also record the data of the small base station performing different caching actions, as well as the transmission delay data of the user's request for the target data after the small base station performs different caching actions. Before starting training, it is necessary to extract the above historical data from the server within a historical period through big data methods, and divide the historical period into multiple historical time periods.

[0055] S202. Based on the state space, action space, and reward function, a quantum search algorithm is used to iteratively calculate the cumulative reward value corresponding to each cache action under each historical request state using the Bellman equation of the reinforcement learning algorithm, until the cache action with the largest cumulative reward value corresponding to each historical request state is determined according to a preset convergence condition.

[0056] Specifically, the core of the reinforcement learning algorithm in this step is the Bellman equation, which uses a reward function to calculate the cumulative reward value corresponding to each cache action. The cumulative reward value corresponding to each cache action is more generally described as the Q value corresponding to each cache action, which can be set to 0 in the initial stage. In an iterative round, the agent can select a historical request state and execute a cache action in the historical request state. The reward value of the cache action is given according to the change in the transmission delay of the user's request target data in the system network model caused by the cache action. The lower the transmission delay of the user's request target data, the greater the reward value of the reward function. After executing the cache action, it transfers to the next historical request state. The next historical request state corresponds to multiple cache actions, and different cache actions correspond to different cumulative reward values. The Bellman equation uses the reward value of the cache action and the maximum cumulative reward value corresponding to the cache action in the next historical request state to update the cumulative reward value of the cache action. The reinforcement learning algorithm updates the cumulative reward value corresponding to each cache action by continuously iterating the above steps. When the preset convergence conditions are met, for example, when the cumulative reward value of the cache action with the largest cumulative reward value changes less than the preset threshold in different iteration rounds, the cache decision model is considered to have converged and the iteration is stopped. At this time, the cache decision model training is completed.

[0057] However, in practical applications, considering the target data, the request state of the target data, and the number of cache actions under different request states, the spatial dimension of the reinforcement learning algorithm is relatively high, and the time to reach the preset convergence condition may be relatively long. Therefore, the embodiment of the present application adopts a quantum search algorithm to accelerate the convergence speed of the reinforcement learning algorithm.

[0058] Specifically, the principle of the quantum search algorithm is that, in the case of executing a target cache action in the first historical request state in the current iteration round: if the cumulative reward value corresponding to the target cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all cache actions in the first historical request state in the previous iteration round, then the probability of selecting the target cache action is increased through quantum operations. It can be understood that if the cumulative reward value corresponding to the target cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all cache actions in the first historical request state in the previous iteration round, considering that the goal of the reinforcement learning algorithm is to find the cache action with the largest cumulative reward value, it is obvious that this action is a good action, then the quantum search algorithm can perform quantum operations to increase the probability of the agent subsequently selecting the target cache action, thereby accelerating the convergence speed of the reinforcement learning algorithm.

[0059] The cache decision model training method provided in this embodiment aims to find the cache action with the largest cumulative reward value corresponding to each historical request state in the cache decision model. After selecting a cache action under a historical request state in the current iteration round, if the cumulative reward value corresponding to the cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all cache actions under the historical request state in the previous iteration round, it means that the cache action selected in the current iteration round is a good action, otherwise it is a bad action. Since this method uses a quantum search algorithm and improves the probability of the intelligent agent selecting a good action through quantum operations, it can accelerate the convergence speed of the reinforcement learning algorithm and reduce the training time of the cache decision model.

[0060] A possible working principle of step S201 is described in detail below.

[0061] Quantum superposition theory is very effective in dealing with high-complexity computing problems. The computing unit of quantum computing is a quantum bit, and each quantum bit consists of two basis states. and The key operation in quantum computing for an n-bit quantum system is the superposition of the n quantum bits. Transformation. In quantum computing systems, The transformation is achieved by applying a quantum circuit to the target state to convert its amplitude and phase. Specifically, in the embodiment of the present application, a series of quantum gates (matrix operations) can be used to simulate the quantum circuit process. The transformation will act on n quantum states at the same time. Subsequently, the state observation of the quantum state will cause it to collapse into a certain state with a certain probability. This process is called quantum parallelism, so quantum parallelism can be used to improve the learning efficiency of reinforcement learning algorithms.

[0062] An entangled quantum action can be composed of a set of mutually orthogonal basis vectors. These basis vectors are also called "eigenactions", and each eigenaction corresponds to a classical action in the reinforcement learning algorithm. The mutual mapping process between classical actions and entangled actions is as follows:

[0063]

[0064] in, corresponds to the quantum state The probability amplitude, It means that the system collapses to The probability of, and M is the number of actions of the decision-making system. Since quantum computing is based on quantum bit operations, the following uses an n-bit quantum system as an example to introduce the acceleration effect of quantum parallelism on reinforcement learning algorithms.

[0065] The selection of the number of quantum bits n in the quantum search algorithm should satisfy , so that the dimension of the basis vector is sufficient to represent all actions in the reinforcement learning process. Therefore, the following conversion relationship can be obtained:

[0066]

[0067] in, and Initialized by the mean The goal of the quantum search algorithm is to search for the optimal strategy , that is, find the cache action with the largest cumulative reward value under each historical request state. Therefore, according to the above formula, the action-state mapping relationship in the traditional reinforcement learning algorithm is converted into a mapping based on quantum collapse: When the action When evaluated, the system will be evaluated with probability Collapse into a quantum state .

[0068] In the quantum search algorithm, quantum search helps the reinforcement learning algorithm to converge to the optimal strategy faster. The transformed quantum iterative algorithm performs parallel operations on the quantum entangled state. The specific process is as follows:

[0069]

[0070] in,

[0071]

[0072]

[0073] Matrix operations The role of To symbolize, matrix For n Vihadma Gate, is the outer product operation of quantum bits. Quantum iteration performs a structured search of the action space through a series of matrix operations. Compared with the unstructured search process in traditional reinforcement learning algorithms, Iteration will achieve significant efficiency improvements.

[0074] Figure 3 A schematic diagram of the quantum search algorithm flow provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, the method includes:

[0075] S301, initializing the action space to a quantum entangled state;

[0076] Specifically, the action space at the end of the cache decision is initialized to a quantum entangled state , and initialize. Quantum entanglement allows the system to represent multiple actions simultaneously in a superposition state. This initialization allows the algorithm to explore the action space in parallel, rather than relying on trying them one by one. Therefore, through parallel exploration, the algorithm can evaluate more action combinations in a shorter time, thereby accelerating convergence.

[0077] After that, the iterative process is performed, and in each iteration round, the following operations are performed:

[0078] S302, mapping the first historical request state to a quantum entangled state;

[0079] In this step, by mapping the first historical request state to a quantum state, the system can use the characteristics of quantum superposition and entanglement to represent and process complex state information. This mapping enhances the system's ability to express state information, allowing the algorithm to better use historical information for decision-making.

[0080] S303, determining a target cache action according to the collapse probability of the quantum entangled state;

[0081] The collapse probability of the quantum entangled state is used to select the most likely target cache action. This process uses the probability amplitude of the quantum state to guide the cache action selection. By utilizing the probabilistic characteristics of the quantum entangled state, the algorithm can more effectively explore and utilize the action space and improve the efficiency of selecting the optimal action.

[0082] Specifically, in steps S302 and S303, the agent observes the first historical request status Mapping to quantum entangled states: And the collapse probability Decide on an action .

[0083] S304, after executing the target cache action, calculating the first reward value of the target cache action through the reward function, and determining the second reward value according to the second historical request state of the cache decision model, where the second reward value is the maximum value of the accumulated reward values ​​corresponding to each cache action in the second historical request state;

[0084] S305, updating the cumulative reward value of the target cache action through the Bellman equation according to the first reward value and the second reward value;

[0085] Specifically, execute the target cache action Then, the first reward value of the target cache action is calculated through the reward function , observe the second historical request status , and determine the second reward value, and update the cumulative reward value of the target cache action according to the Bellman equation, which is:

[0086]

[0087] in, The cumulative reward value after the target cache action is updated, is the first historical request status, is the second historical request status, Cache actions for the target, is the learning rate, is the reward function, is the discount rate, is the second reward value, It is a cache action in the second historical request state. It should be noted that when starting the training of the cache decision model, it is necessary to set the initial iteration time period, the initial user request state of the target data, the discount rate, the learning rate, the number of first base stations in the network and the number of target data, and the initial cumulative reward value of each cache action.

[0088] S306, determining whether the cumulative reward value corresponding to the target cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all cache actions in the first historical request state in the previous iteration round;

[0089] Specifically, this step is used to determine whether the target cache action is a good action.

[0090] If yes, execute step S307;

[0091] If not, execute step S308;

[0092] S307, executing Grover iteration to amplify the probability amplitude of the target cache action;

[0093] Specifically, execute The iteration is:

[0094]

[0095] In this step, the Grover iteration is performed to amplify the probability amplitude of the target cache action, thereby increasing the probability of the subsequent system selecting the target cache action. Since the quantum search algorithm utilizes quantum parallelism and amplifies the probability amplitude of the target cache action, the time complexity of the reinforcement learning algorithm converged by the quantum search algorithm is low, shortening the training time of the cache decision model.

[0096] S308, determining whether a preset convergence condition is met;

[0097] If the preset convergence condition is met, the iteration is stopped; if the preset convergence condition is not met, step S302 is executed.

[0098] The reward function of the cache decision model is explained below.

[0099] Figure 4 A flowchart of a method for calculating a first reward value of a target cache action by using a reward function is provided in an embodiment of the present application, such as Figure 4 As shown, the method includes:

[0100] S401, calculating a reduction value of the transmission delay of the user requesting the target data in each historical time period according to the historical data of the transmission delay of the user requesting the target data after executing the target caching action;

[0101] Optionally, after executing the target caching action, in each historical time period, the first base station obtains the historical data of the transmission delay of the target data from the second base station, subtracts the actual historical data of the transmission delay of the target data obtained by the first base station, and calculates the reduction value of the transmission delay of the user requesting the target data in each historical time period, wherein each second base station communicates with multiple first base stations. In the embodiment of the present application, the first base station can be understood as Figure 1 The second base station can be understood as a small base station in Figure 1 In the macro base station, taking the banking system as an example, a macro base station communicates with multiple small base stations. By considering the transmission delay between the macro base station and the small base station, the reward function is more in line with the network architecture where the macro base station communicates with multiple small base stations. Specifically, Represents the target data. The reduction value of the transmission delay of the user requesting the target data in each historical time period can be calculated by the following formula:

[0102]

[0103] Indicates small base station Get target data from macro base station The transmission delay of It means small base station Get the target data Get the actual transmission delay of the target data, which is expressed as follows:

[0104]

[0105] in, Used to determine small base stations Can service data be obtained from other small base stations? Specifically, if Small base station Cached ,on the contrary Indicates that it is not cached; Can users at representative sites obtain target data from small base stations at other sites through collaborative caching? ,like =1 means that the small base station can obtain target data from small base stations in other nodes through collaborative caching , otherwise =0 means that the small base station cannot obtain target data from small base stations in other nodes through collaborative caching .

[0106] From small base stations Obtaining the actual transmission delay formula for obtaining target data shows that if the small base station caches the target data, and if the small base station can obtain the target data from other small base stations, the actual transmission delay historical data of the small base station obtaining the target data will be reduced. The actual transmission delay historical data of the small base station obtaining the target data is determined based on whether the small base station caches the target data, whether the small base station can obtain the target data from other small base stations, and the transmission delay historical data of the small base station obtaining the target data from the macro base station, so that the reward function is more in line with the scenarios of business data transmission between macro base stations and small base stations, and between small base stations.

[0107] S402, calculating an average value of the reduction value in multiple historical time periods according to the reduction value of the transmission delay of the user requesting the target data in each historical time period;

[0108] In this step, by calculating the average value of the delay reduction value in multiple historical time periods, the effect of the target cache action on the delay reduction value of the cache decision model in a long period of time can be determined.

[0109] Specifically, the reduction values ​​of the transmission delay of the target data requested by the user in each historical time period may be accumulated and averaged, so that the average value of the reduction values ​​may be calculated.

[0110] S403. Calculate a first reward value using a reward function. The higher the average value of the reduction value is, the higher the first reward value is.

[0111] Specifically, a linear reward function may be set, and the first reward value may be directly proportional to the average value of the reduction value; a nonlinear reward function may be set, such as a square or exponential function, to give a higher reward for a larger average value of the reduction value.

[0112] The following embodiment of the present application describes the process of obtaining historical request status data of user request target data.

[0113] Since users at the network point send data requests for the target data according to the Zipf distribution, that is, data with high user demand will be frequently requested, and data with low user demand will be requested less frequently, therefore, the popularity of the target data directly affects the user's request status for the target data. In the embodiment of the present application, multiple historical request status data of users requesting the target data in multiple historical time periods can be obtained based on the popularity of the target data. By analyzing the popularity of the target data, the user's demand for the target data can be more accurately analyzed, and the user's request status for the target data can be more accurately determined.

[0114] Optionally, the popularity of the target data in each historical time period is determined according to the ratio of the number of times the target data is requested from the first base station to the total number of times the data is requested from the first base station in each historical time period, which can be expressed by the following formula:

[0115]

[0116] In this formula, Represents the target data. The popularity of the target data among users can be determined by the ratio of the number of times users request the target data to the total number of times users request all data in each time period, so that the popularity of the target data can be accurately determined, thereby more accurately evaluating the status of users' requests for the target data.

[0117] Optionally, the average expected value of user requests for target data can be determined based on the popularity of the target data in each historical time period, and the average expected value of user requests for target data in each historical time period can be used as the historical request status data in each historical time period. Specifically, the average expected value can be understood as the average number of times the target data is requested by users in the historical time period. The popularity calculation formula can be:

[0118]

[0119] is the skew parameter of the Zipf distribution, and It can describe the skewness of the popularity of target data. Increasing from 0 to 1 means that the important target data is getting less and less, and more and more concentrated on a small number of target data. The size is In the embodiment of the present application, the skewness of the target data can be determined according to the popularity, and then the average expected value of the user's request for the target data can be calculated. By taking the average expected value of the user's request for the target data as the request state and establishing a cache decision model according to the request state, the optimal cache strategy for the target data under different user's request expectations for the target data can be determined.

[0120] Based on the above training method, after the cache decision model is trained to meet the preset convergence conditions, it can be applied to the actual production environment to make cache decisions. Figure 5 A schematic diagram of a cache decision method flow chart provided in an embodiment of the present application, wherein the method is implemented based on a cache decision model trained by the aforementioned training method, such as Figure 5 As shown, the method includes:

[0121] S501, obtaining request status data of target data requested by the user;

[0122] S502: Determine, by the cache decision model, a target cache action in the cache actions corresponding to the request state according to the request state, the cache action indicating whether the first base station caches the target data;

[0123] S503: Control the first base station to execute a target caching action.

[0124] This method can determine the current cache action of the first base station, that is, whether the first base station caches the target data, based on the real-time request status of the user requesting the target data in the system network through a pre-trained cache decision model. Through the above training method, it can be understood that since the reward function of the cache decision model gives a reward value based on the transmission delay of the user requesting the target data, the cache decision model can be trained to determine the target cache action with the largest cumulative reward value based on the corresponding cache action under the current real-time request state, so this method can reduce the transmission delay of the user requesting the target data.

[0125] Figure 6 A schematic diagram of the structure of the cache decision model training device provided in the embodiment of the present application, such as Figure 6 As shown, the device 60 includes:

[0126] The model generation module 601 is used to: construct a cache decision model according to multiple historical request state data of user request target data in multiple historical time periods, multiple cache action history data in each historical request state, and historical data of transmission delay of user request target data after executing each cache action, wherein the cache action indicates whether the first base station caches the target data, and the cache decision model includes a state space, an action space, and a reward function, wherein the state space includes multiple historical request states, and the action space includes multiple cache action history data in each historical request state; the lower the transmission delay of the user request target data, the greater the reward value of the reward function;

[0127] The quantum computing module 602 is used to: according to the state space, action space, and reward function, through the quantum search algorithm, the Bellman equation of the reinforcement learning algorithm is used to iteratively calculate the cumulative reward value corresponding to each cache action under each historical request state, until the cache action with the largest cumulative reward value corresponding to each historical request state is determined according to the preset convergence condition, wherein, in the case of executing a target cache action under the first historical request state in the current iteration round: if the cumulative reward value corresponding to the target cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all cache actions under the first historical request state in the previous iteration round, the probability of selecting the target cache action is increased through quantum operation. The cache decision model training device provided in the embodiment of the present application can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0128] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0129] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that at least one processor 701 executes the above method.

[0130] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0131] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0132] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0133] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0134] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0135] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0136] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0137] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0138] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0141] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0142] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0143] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A cache decision model training method, characterized in that: The method comprises: A cache decision model is constructed according to multiple historical request state data of user requesting target data in multiple historical time periods, multiple cache action history data in each of the historical request states, and historical data of transmission delay of the user requesting the target data after executing each of the cache actions, wherein the cache action indicates whether the first base station caches the target data, and the cache decision model includes a state space, an action space, and a reward function, wherein the state space includes the multiple historical request states, and the action space includes the multiple cache action history data in each of the historical request states; the lower the transmission delay of the user requesting the target data, the greater the reward value of the reward function; According to the state space, the action space, and the reward function, a quantum search algorithm is used to iteratively calculate the cumulative reward value corresponding to each cache action under each historical request state using the Bellman equation of the reinforcement learning algorithm until a cache action with the largest cumulative reward value corresponding to each historical request state is determined according to a preset convergence condition, wherein, in the case of executing a target cache action under the first historical request state in the current iteration round: if the cumulative reward value corresponding to the target cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all the cache actions under the first historical request state in the previous iteration round, the probability of selecting the target cache action is increased through quantum operation.

2. The method according to claim 1, characterized in that: The method of iteratively calculating the cumulative reward value corresponding to each cache action under each of the historical request states by the quantum search algorithm using the Bellman equation of the reinforcement learning algorithm until the cache action with the largest cumulative reward value corresponding to each of the historical request states is determined according to a preset convergence condition, includes: Initializing the action space to a quantum entangled state; Perform the iterative process, and in each iteration round, perform the following operations: Mapping the first historical request state to the quantum entangled state; Determining the target cache action according to the collapse probability of the quantum entangled state; After executing the target cache action, calculating a first reward value of the target cache action by using the reward function, and determining a second reward value according to a second historical request state of the cache decision model, wherein the second reward value is a maximum value of cumulative reward values ​​corresponding to each of the cache actions in the second historical request state; According to the first reward value and the second reward value, updating the cumulative reward value of the target cache action by using the Bellman equation; Determine whether the cumulative reward value corresponding to the target cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all the cache actions in the first historical request state in the previous iteration round; If yes, perform Grover iteration to enlarge the probability amplitude of the target cache action; The iteration process stops until the cache action with the largest cumulative reward value corresponding to each historical request state is determined according to the preset convergence condition.

3. The method according to claim 2, characterized in that The calculating the first reward value of the target cache action by using the reward function includes: Calculate the reduction value of the transmission delay of the user requesting the target data in each historical time period according to the historical data of the transmission delay of the user requesting the target data after executing the target caching action; According to the reduction value of the transmission delay of the user requesting the target data in each of the historical time periods, calculating the average value of the reduction value in the multiple historical time periods; The first reward value is calculated by the reward function, and the higher the average value of the reduction value is, the higher the first reward value is.

4. The method according to claim 3, characterized in that The calculating, according to the historical data of the transmission delay of the user requesting the target data after the target caching action is executed, the reduction value of the transmission delay of the user requesting the target data in each historical time period includes: After executing the target caching action, within each of the historical time periods, the first base station obtains the historical data of the transmission delay of the target data from the second base station, subtracts the actual historical data of the transmission delay of the target data obtained by the first base station, and calculates the reduction value of the transmission delay of the user requesting the target data in each of the historical time periods, wherein each of the second base stations communicates with multiple first base stations.

5. The method according to claim 4, characterized in that Before building the cache decision model based on multiple historical request status data of user requests for target data in multiple historical time periods, multiple cache action history data under each of the historical request states, and historical data of transmission delay of the user requesting the target data after executing each of the cache actions, the method further includes: Obtaining the transmission delay history data of the target data requested by the user after executing each of the caching actions, wherein the transmission delay history data of the target data requested by the user includes: after executing each of the caching actions, the first base station obtains the transmission delay history data of the target data from the second base station, and the first base station obtains the actual transmission delay history data of the target data, wherein the actual transmission delay history data of the target data obtained by the first base station is determined according to whether the first base station caches the target data, whether the first base station can obtain the target data from other first base stations, and the transmission delay history data of the target data obtained by the first base station from the second base station; if the first base station caches the target data, and if the first base station can obtain the target data from other first base stations, the actual transmission delay of the first base station obtaining the target data will be reduced.

6. The method according to claim 1, characterized in that Before building the cache decision model based on multiple historical request status data of user requests for target data in multiple historical time periods, multiple cache action history data under each of the historical request states, and historical data of transmission delay of the user requesting the target data after executing each of the cache actions, the method further includes: According to the popularity of the target data, a plurality of historical request status data of the user requesting the target data within the plurality of historical time periods are obtained.

7. The method according to claim 6, characterized in that Acquiring, according to the popularity of the target data, a plurality of historical request status data of the user requesting the target data within the plurality of historical time periods, including: Determine the popularity of the target data in each of the historical time periods according to a ratio of the number of requests for the target data from the first base station to all the number of requests for data from the first base station in each of the historical time periods; The historical request status data in each of the historical time periods is determined according to the popularity of the target data in each of the historical time periods.

8. The method according to claim 7, characterized in that The determining the historical request status data in each of the historical time periods according to the popularity of the target data in each of the historical time periods includes: According to the popularity of the target data in each of the historical time periods, the average expected value of users requesting the target data is determined, and the average expected value of users requesting the target data in each of the historical time periods is used as the historical request status data in each of the historical time periods.

9. A cache decision method, characterized in that: The cache decision method is implemented based on a cache decision model, wherein the cache decision model is trained using the cache decision model training method according to any one of claims 1 to 8, and the cache decision method includes: Get the request status data of the user's request target data; Determining, by the cache decision model, a target cache action in the cache actions corresponding to the request state according to the request state, wherein the cache action indicates whether the first base station caches the target data; Control the first base station to perform the target caching action.

10. A cache decision model training device, characterized in that: The device comprises: A model generation module is used to: construct a cache decision model according to multiple historical request state data of user requesting target data in multiple historical time periods, multiple cache action history data under each of the historical request states, and historical data of transmission delay of the user requesting the target data after executing each of the cache actions, wherein the cache action indicates whether the first base station caches the target data, and the cache decision model includes a state space, an action space, and a reward function, wherein the state space includes the multiple historical request states, and the action space includes the multiple cache action history data under each of the historical request states; the lower the transmission delay of the user requesting the target data, the greater the reward value of the reward function; A quantum computing module is used to: iteratively calculate the cumulative reward value corresponding to each cache action under each historical request state according to the state space, the action space, and the reward function through a quantum search algorithm and the Bellman equation of a reinforcement learning algorithm, until the cache action with the largest cumulative reward value corresponding to each historical request state is determined according to a preset convergence condition, wherein, in the case of executing a target cache action under the first historical request state in the current iteration round: if the cumulative reward value corresponding to the target cache action in the current iteration round is greater than or equal to the maximum cumulative reward value corresponding to all the cache actions under the first historical request state in the previous iteration round, the probability of selecting the target cache action is increased through quantum operation.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Images