An edge caching method based on causal reinforcement learning

Abstract

The present invention discloses an edge caching method based on causal reinforcement learning. In the edge caching system, popular content can be cached near the edge of the network, such as in a base station, which can greatly reduce the repeated traffic of the network and shorten the transmission delay, but how to optimize the cached content in the base station is a key issue. Therefore, the method of the present invention utilizes the observed data and considers the influence of the hidden state, and optimizes the file content cached in the base station to maximize the hit rate and reduce storage overhead and delay. Compared with other methods that do not utilize observed data or do not consider hidden states, the method of the present invention can greatly improve the hit rate at the initial moment.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communications, and in particular relates to an edge caching method based on causal reinforcement learning. Background Art

[0002] At present, the widespread application of the Internet of Things has brought about a rapid increase in service demand for existing mobile networks, resulting in a rapid increase in mobile data traffic. Intelligent applications brought about by advances in artificial intelligence technology can improve people's daily lives. However, these applications are highly dependent on computing, storage, and communication resources. If content is obtained from a remote data center (such as a cloud server), it may not meet the low latency and diverse application requirements for content access.

[0003] To solve these problems, it is necessary to introduce advanced network architecture and new data transmission technologies. In particular, edge caching is considered a promising technology that can reduce the burden of backhaul traffic. For network operators, in an edge caching system, popular content can be cached near the edge of the network, such as in a base station, which can greatly reduce the large amount of duplicate traffic of content transmitted by the backhaul network and shorten the transmission delay.

[0004] In edge caching systems, a hot research issue is the selection of cached content. In particular, when users generate content requests, we should consider designing a suitable cache content framework to adapt to the dynamic network environment. Most existing cache systems rely on rule-based cache replacement schemes, such as FIFO, LRU, LFU, etc. At the same time, some machine learning techniques have been adopted in the cache replacement problem, such as LSTM and reinforcement learning methods (such as DQN). However, some existing learning-based methods require users to send their personal data to a central server, which may cause serious privacy and security issues for the user base.

[0005] Based on the above situation, this method establishes an edge caching model with a user group privacy protection mechanism, and proposes an edge caching method based on causal reinforcement learning. Compared with other edge caching methods, this method is based on the environment model and can reasonably use the observation data. Therefore, it can learn a more accurate environment model at the initial moment, obtain a higher hit rate, and lower storage overhead and latency. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide an edge caching method based on causal reinforcement learning. The method models the edge cache content replacement problem as a model-based partially observable Markov decision process for an edge intelligent system, and adopts causal reinforcement learning to utilize observation data to achieve the effect of improving the cache hit rate at the initial moment.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] An edge caching method based on causal reinforcement learning, the method comprising the following steps:

[0009] Step S1: for an edge intelligent system having a cloud server, N base stations and N user groups, construct an edge cache model, where one base station corresponds to one user group;

[0010] Step S2: Model the edge cache model as a reinforcement learning problem of a partially observable Markov decision process, model the file requests of N user groups as environments, model each base station as an agent, determine its state set, observation set, action set, reward function and final optimization goal; optimize the agent g corresponding to one of the base stations int Edge caching strategy, the intelligent agents g corresponding to the other base stations prv Interaction data with user groups as observation data;

[0011] Step S3: For the reinforcement learning problem of the partially observable Markov decision process, the edge caching strategy based on causal reinforcement learning is optimized by using the observed data and considering the hidden state.

[0012] Furthermore, the edge cache model in step S1 is:

[0013] A cloud server serves N base stations at the same time. The total number of all files in the cloud server is recorded as M, the set of all files is recorded as F, and the mth file is recorded as f m , f m ∈F,m=0,1,...,M are all of the same size, B bits; each base station serves a corresponding user group, there is no overlap between user groups and they will not change over time; the storage capacity of the N base stations is equal, denoted by S station bits, and S station <M·B; the user group itself has no storage capacity;

[0014] The total service time of the cloud server is T. N user groups will generate a request for a certain file at each time t∈T, denoted as r t , and r t ∈F; requests r generated by the user group t It will not be uploaded to its corresponding base station, and the user group's request is unknown to the base station;

[0015] Each of the N base stations is equipped with a user group request prediction mechanism, which can predict the user group's request r in real time. t , the request prediction mechanism is: at time t, the base station first performs user group request prediction and obtains the prediction result If the file stored in the base station at this moment Contains The corresponding file is Then the base station will Transmit to user groups;

[0016] If the storage file of the base station at time t Not included Right now The base station generates a request Request a file from the cloud server. After receiving the request, the cloud server will send the corresponding file at time t+1. Transmitted to the base station, the user group's request r t+1 =r t , the user group request prediction result of the base station

[0017] If the base station's storage file at time t+1 Included Then the base station sends the file to Otherwise, it will continue to request the file from the cloud server at time t+2 until the storage file of the base station is Included Until then, the total waiting time of the user group during this period is recorded as n delay .

[0018] Furthermore, in the edge cache model of the present invention, the user's request is always unknown to the base station, and the base station can only observe its own current storage files, so this is a partially observable environment; and the storage state of the base station and the user group's request at the next moment are only related to the storage state of the base station and the user group's request at the current moment, and have nothing to do with all previous moments, so the model can be modeled as a partially observable Markov decision process, and then solved using the reinforcement learning method. Specifically, the present invention models the file request of the user group as an environment, and models each base station as an intelligent agent. The present invention mainly considers the edge cache optimization problem of one of the base stations, and the intelligent agent corresponding to the base station is denoted as g int , and the rest of the agents are denoted as g prv , the data they interact with the user group is only used as observation data to assist the agent g int Cache strategy optimization.

[0019] Intelligent Agent int The state set, observation set, action set, and reward function in the edge caching strategy are as follows:

[0020] State Set: A state set is defined as the storage state of a base station and the requests of a user group:

[0021] S={s t |t=0,1,2,...T} (22)

[0022] The state at time t is defined as

[0023]

[0024] Observation Set: An observation set is defined as the storage status of a base station.

[0025] O={o t |t=0,1,2,...T} (24)

[0026] The observation at time t is defined as

[0027]

[0028] Action Set: Action set is defined as the actions of the base station

[0029] A={a t |t=0,1,2,...T} (26)

[0030] At time t, the base station can request the cloud server to add a file, delete a file, or neither add nor delete a file. The action at time t is defined as:

[0031] a t =[a1,a2,a3,...a M ] (27)

[0032] a t Each element in takes a value of -1, 0, or 1. m =-1, m = 1, 2, ... M means that the base station wants to delete the mth file f from its own storage space m If a m =1,m=1,2,...M means the base station wants to request the mth file f from the cloud server m And store it in its own storage space; if a m =0,m=1,2,...M, it means that the base station neither adds the file nor deletes the file;

[0033] Reward function: The reward function consists of three parts: the reward function u at time t t Expressed as

[0034]

[0035] Among them, w1, w2 and w3 are the weights of the three parts respectively.

[0036] w1+w2+w3=1 (29) and ht represents the hit rate at time t, which is defined as follows

[0037]

[0038] is the number of files stored in the base station, n delay Indicates waiting delay;

[0039] Finally, the optimization problem of edge caching can be summarized as maximizing the total reward in each round, expressed as:

[0040]

[0041] Furthermore, the specific steps of step S3 include:

[0042] Step S3.1: Get agent g prv observational data of the environment and intervention data of its own interaction with the environment;

[0043] Step S3.2, estimating a true environment model based on the observation data and intervention data obtained in step S3.1;

[0044] Step S3.3: According to the environment model estimated in step S3.2, the base station whose cache strategy needs to be optimized is trained using the Actor-Critic method based on the environment model to obtain the optimal edge cache strategy.

[0045] Furthermore, the model established by the present invention can obtain the intelligent agent g int Intervention data D that interacts with its corresponding user group int , and the rest of the agents g prv Observation data D of interactions with its corresponding user group obs ;

[0046] Since the intelligent agent g prv The cache strategy π prv (a t |h t ,s t ) is not necessarily optimal, so its strategy π prv (a t |h t ,s t ) may contain confounding variables and contains observation data D of τ rounds obs is obtained from a partially observable Markov decision process with the following distribution:

[0047] D obs ~p init ,p trans ,p obs ,π prv(32)

[0048] Among them, p init represents the initial distribution probability of the state, p trans represents the state transition probability, p obs represents the observation probability, π prv Represents agent g prv strategy;

[0049] The specific form of the observation data is:

[0050]

[0051] D i =(o0,a0,...,o T ) (34)

[0052] Contains k rounds of intervention data D int is a random strategy π(a t |h t ) int D is collected from a standard partially observable Markov decision process int ~p init ,p trans ,p obs ,π (35)

[0053] Among them, p init represents the initial distribution probability of the state, p trans represents the state transition probability, p obs represents the observation probability, π represents the agent g int Strategy; Except for strategy π, the other three probabilities in formula (14) are the same as those in formula (11);

[0054] The specific form of intervention data is:

[0055]

[0056] D i =(o0,a0,...,o T ) (37)

[0057] In order to simplify the symbols, an auxiliary variable I∈{0,1} is added to distinguish the two strategies, that is, i=0 represents the observed data:

[0058] π(a t |h t ,s t ,i=0)=π prv (a t |h t ,s t) (38)

[0059] If i=1, it means intervention data:

[0060] π(a t |h t ,s t ,i=1)=π std (a t |h t ) (39).

[0061] Furthermore, in step S3.2, estimating the real environment model based on the observation data and intervention data obtained in step S3.1 specifically includes:

[0062] Estimating the true environmental model from observational data and intervention data is divided into two steps:

[0063] a. Learning of environmental models: fitting a latent probability model based on observed data and intervention data Specifically, the hidden state s of the actual partially observable Markov decision process is t ∈S is replaced by a latent variable z t ∈Z, where Z is the discrete latent space of this environment model; thus, the learning of the environment model is expressed as the following standard maximum likelihood problem:

[0064]

[0065] Where Q is the set of the following probability models:

[0066]

[0067] Where q(i) represents the proportion of observed data and intervention data, q(z0) represents the distribution probability of the hidden state, q(o0|z0) represents the observation probability of the hidden state, and q(a t |h t ,z t ,i) represents the strategy of the agent. If i=1, then:

[0068] q(a t |h t ,z t ,i=1)=q(a t |h t ,i=1) (42)

[0069] q(z t+1 |a t ,z t ) represents the transition probability of the hidden state, q(o t+1 |z t+1) represents the observation probability of the hidden state at time t+1;

[0070] b. Inference of environmental model: will restore As an estimator of transition probabilities for a standard partially observable Markov decision model, The potential probability model learned from step a above using the forward algorithm is derived from;

[0071] The observation data D obs This will affect the intervention transfer model in the following ways Specifically include: the learned model and There must be the same probability q(z0), q(o t |z t ) and q(z t+1 |z t ,a t ), only the agent’s strategy q(a t |h t ,z t , i=0) and q(a t |h t ,z t , i=1) can have some differences to distinguish the two schemes. The final result is equivalent to the observation data playing a regularization role on the intervention data.

[0072] The beneficial effects of the present invention are:

[0073] An edge caching method based on causal reinforcement learning is proposed, which can make full use of observation data to improve the accuracy of environmental model learning. Compared with the original caching method, the method of the present invention can obtain a higher hit rate, lower storage overhead and latency at the initial moment. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flow chart of the method of the present invention;

[0075] Figure 2 is a schematic diagram of an edge cache model of the present invention;

[0076] Figure 3 It is the JS divergence diagram of the environment model learned by the method of the present invention and the real environment model;

[0077] Figure 4 A comparison chart of the total reward value obtained in each round of the method of the present invention. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] Example 1

[0080] See also Figure 1-Figure 4 , this embodiment provides an edge caching method based on causal reinforcement learning, comprising the following steps:

[0081] Step S1: construct an edge cache model for an edge intelligent system having a cloud server, two base stations, and two user groups. This step specifically includes:

[0082] In an edge intelligent system, there is a cloud server, two base stations and two user groups. The cloud server provides file retrieval services to the user groups through the base stations.

[0083] Assume that the storage capacity of the cloud server is large enough to store all files. The total number of all files is denoted as m, the set of all files is denoted as F, and the i-th file is denoted as f. i , f i ∈F,i=0,1,...,m are all of the same size, which is B bits.

[0084] The base station has only a certain storage capacity, denoted as S station bits, and S station <m·B, and the user group itself has no storage capacity.

[0085] The total service time of the cloud server is T. The user group will generate a request for a file at each time t∈T, denoted as r t , and r t ∈F. For privacy reasons, the requests r generated by the user group t It will not be uploaded to the base station, and the user group's requests are always unknown to the base station.

[0086] Therefore, the base station is equipped with a user group request prediction mechanism, which can predict the user group's request r in real time. t What is it.

[0087] At time t, the base station will first predict the user group request and get the prediction result If the file stored in the base station at this moment Contains The corresponding file is Then the base station will Transmit to user groups;

[0088] On the contrary, if the storage file of the base station at time t Not included Right now Then the base station will generate a request When requesting a file from a cloud server, please note that: Not necessarily equal to After receiving the request, the cloud server will send the corresponding file to is transmitted to the base station, and the user group's request at this time is the same as at time t, that is, r t+1 =r t , the prediction result of the user group request of the base station will not change, that is,

[0089] If the base station's storage file at time t+1 Included Then the base station will send the file to Otherwise, it will continue to request the file from the cloud server at time t+2 until the storage file of the base station is Included Until then, the total waiting time of the user group during this period is recorded as n delay .

[0090] Step S2: Model the edge cache model constructed in step S1 as a reinforcement learning problem of a partially observable Markov decision process, and determine its state set, observation set, action set, reward function and final optimization goal. This step specifically includes:

[0091] Due to user group requests t It is unknowable to the base station, so the edge cache model can be modeled as a partially observable Markov decision model, with the base station as the intelligent agent. Next, the state set, observation set, action set and reward function will be defined respectively.

[0092] State Set: A state set is defined as the storage state of a base station and the requests of a user group:

[0093] S={s t |t=0,1,2,...T} (43)

[0094] The state at time t is defined as

[0095]

[0096] Observation Set: An observation set is defined as the storage status of a base station.

[0097] O={o t |t=0,1,2,...T} (45)

[0098] The observation at time t is defined as

[0099]

[0100] Action Set: Action set is defined as the actions of the base station

[0101] A={a t |t=0,1,2,...T} (47)

[0102] At time t, the base station can request the cloud server to add a file, delete a file, or neither add nor delete a file. The action at time t is defined as:

[0103] a t =[a1,a2,a3,...a m ] (48)

[0104] a t Each element in can have a value of -1, 0, or 1. i =-1, i = 1, 2, ... m means that the base station wants to delete the i-th file f from its own storage space i If a i =1,i=1,2,...m means that the base station wants to request the i-th file f from the cloud server i And store it in its own storage space; if a i =0, i=1,2,...m, it means that the base station neither adds the file nor deletes the file.

[0105] Reward function: The reward function consists of three parts: the reward function u at time t t Expressed as

[0106]

[0107] Among them, w1, w2 and w3 are the weights of the three parts respectively.

[0108] w1+w2+w3=1 (50)

[0109] And h t represents the hit rate at time t, which is defined as follows

[0110]

[0111] in addition is the number of files stored in the base station, n delay Indicates a wait delay.

[0112] Finally, the optimization problem of edge caching can be summarized as maximizing the total reward in each round, expressed as:

[0113]

[0114] Step S3: For the reinforcement learning problem of the partially observable Markov decision process established in step S2, an edge caching method based on causal reinforcement learning is proposed by using observed data and considering hidden states;

[0115] Step S3.1: Obtain the observation data of other agents and the intervention data of their own interaction with the environment. This step specifically includes:

[0116] The model established by this method can obtain two data sets of partially observable Markov decision processes, namely, observation data D obs and intervention data D int , containing τ rounds of observation data D obs is a state information that can be obtained from a t Intelligent agent g prv Collected, the agent can observe the hidden state, so its strategy π prv (a t |h t ,s t ) may contain confounding variables, and the observed data D obs is obtained from a partially observable Markov decision process with the following distribution:

[0117] D obs ~p init ,p trans ,p obs ,π prv (53)

[0118] Among them, p init represents the initial distribution probability of the state, p trans represents the state transition probability, p obs represents the observation probability, π prv Represents the strategy of the agent.

[0119] The specific form of the observation data is:

[0120]

[0121] D i =(o0,a0,...,o T ) (55)

[0122] In addition, it contains k rounds of intervention data D intis a random strategy π(a t |h t ) int D is collected from a standard partially observable Markov decision process int ~p init ,p trans ,p obs ,π (56)

[0123] Among them, p init represents the initial distribution probability of the state, p trans represents the state transition probability, p obs represents the observation probability, and π represents the strategy of the agent. Except for the strategy π, the other three probabilities in equation (56) are the same as those in equation (53).

[0124] The specific form of intervention data is:

[0125]

[0126] D i =(o0,a0,...,o T ) (58)

[0127] In order to simplify the symbols, an auxiliary variable I∈{0,1} is added to distinguish the two strategies, that is, i=0 represents the observed data:

[0128] π(a t |h t ,s t ,i=0)=π prv (a t |h t ,s t ) (59)

[0129] If i=1, it means intervention data:

[0130] π(a t |h t ,s t ,i=1)=π std (a t |h t ) (60)

[0131] Step S3.2: Estimate the real environment model based on the observation data and intervention data obtained in step S3.1. This step specifically includes:

[0132] Estimating the true environmental model from observation data and intervention data can be divided into the following two steps:

[0133] a. Learning of environmental model: In the first step, a potential probability model needs to be fitted based on the observed data and intervention data Specifically, the hidden state s of the actual partially observable Markov decision process is t ∈S is replaced by a latent variable z t ∈Z, where Z is the discrete latent space of the environment model. Therefore, the learning of the environment model can be expressed as the following standard maximum likelihood problem:

[0134]

[0135] Where Q is the set of the following probability models:

[0136]

[0137] Where q(i) represents the proportion of observed data and intervention data, q(z0) represents the distribution probability of the hidden state, q(o0|z0) represents the observation probability of the hidden state, and q(a t |h t ,z t ,i) represents the strategy of the agent. If i=1, then:

[0138] q(a t |h t ,z t ,i=1)=q(a t |h t ,i=1) (62)

[0139] q(z t+1 |a t ,z t ) represents the transition probability of the hidden state, q(o t+1 |z t+1 ) represents the observation probability of the hidden state at time t+1.

[0140] b. Inference of the environment model: In the second step, we will recover As an estimator of transition probabilities in standard partially observable Markov decision models. The potential probability model learned from step a above can be simply forward-learned Derived from .

[0141] The observation data D obs This will affect the intervention transfer model in the following ways Specifically include: the learned model and There must be the same probability q(z0), q(o t |z t) and q(z t+1 |z t ,a t ), only the agent’s strategy q(a t |h t ,z t , i=0) and q(a t |h t ,z t , i=1) can have some differences to distinguish the two schemes. The final result is equivalent to the observation data playing a regularization role on the intervention data.

[0142] Step S3.3, according to the environment model estimated in step S3.2, the agent is trained using the Actor-Critic method to obtain the optimal strategy;

[0143] The Actor-Critic method consists of two networks, the Actor network and the Critic network. The Actor network uses a policy function to generate actions and interact with the environment. The Critic network uses a value function to evaluate the Actor's performance and guide the Actor's subsequent behavior. The specific algorithm flow is as follows:

[0144] (1) Randomly initialize the values ​​Q corresponding to all states and actions;

[0145] (2) Iterate from 1 to T:

[0146] (2.1) Initialize S as the first state of the current state sequence and obtain its feature vector Φ(S);

[0147] (2.2) Use Φ(S) as input in the Actor network, output action A, and obtain the next state S' and feedback R based on action A;

[0148] (2.3) Use Φ(S) and Φ'(S) as input in the Critic network to obtain Q-value outputs V(S) and V'(S);

[0149] (2.4) Calculate the TD error: δ = R + γV(S') - V(S);

[0150] (2.5) Using the mean square error loss function ∑(R+γV(S')-V(S,w)) 2 As the gradient update of the Critic network parameter w;

[0151] (2.6) Update the Actor network parameters θ:

[0152] In order to verify the performance advantage of the above method, this embodiment provides a specific example process, which specifically includes:

[0153] (1) Experimental parameter setting

[0154] In order to prove the effectiveness of the method of this embodiment, two other methods are selected for comparison. One is a method that does not use observation data at all, which is recorded as method one; the second is a method that uses observation data but does not consider hidden states, which is recorded as method two; the method of the present invention uses both observation data and hidden states.

[0155] This embodiment stipulates that the total number of files under this experimental condition is m=3, and the duration of each round is T=20 moments. The probability of the user group requesting these three files is p user =[0.5,0.4,0.1], and does not change over time. w1 = 1, and w2 = w3 = 0.1. In addition, it is assumed that the accuracy of the prediction mechanism of the base station is 80%, and the probability of predicting the other two files is 10% respectively. In addition, we stipulate that the size of the observation data is τ = 256 rounds, and the size of the intervention data considers 7 cases, namely k = 4, 8, 16, 32, 64, 128, 256 rounds.

[0156] (2) The effect of the method of the present invention on improving the accuracy of environmental model learning

[0157] In this embodiment, the JS divergence between the learned environment model and the real environment model is as follows: Figure 3 As shown in the figure, it can be clearly seen that the solid line part is the average value of the JS divergence of the three methods, and the shaded part is its standard deviation. Compared with the method without using observed data, the JS divergence values ​​of the other two methods using observed data are much lower, and the method of the present invention can obtain the lowest JS divergence because the hidden state is taken into account. The lower the JS divergence value, the closer the learned environment model is to the real environment.

[0158] (3) The effect of the method of the present invention on improving the total round reward obtained by the agent

[0159] In this embodiment, the average round reward value of the agent is as follows: Figure 4 As shown in the figure, the solid line in the figure represents the average of the total reward value of the round, and the shaded part represents the standard deviation of the reward value. It can be clearly seen from the figure that the reward value of the method of the present invention at the initial moment is much higher than that of the other two methods, while the reward value of method 2 that uses observation data but does not consider the hidden state is slightly lower than that of the method of the present invention, and the reward value of method 1 that does not use observation data at all is the lowest.

[0160] In summary, the method of the present invention can learn the real environment model more accurately, and then according to the learned environment model, the intelligent agent can obtain a higher reward value at the initial moment, that is, a higher hit rate, lower storage overhead and waiting delay.

[0161] The present invention is not described in detail, and all of them are well-known technologies to those skilled in the art.

[0162] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. An edge caching method based on causal reinforcement learning, characterized in that: The method comprises the following steps: Step S1: for an edge intelligent system having a cloud server, N base stations and N user groups, construct an edge cache model, where one base station corresponds to one user group; Step S2: Model the edge cache model as a reinforcement learning problem of a partially observable Markov decision process, model the file requests of N user groups as environments, model each base station as an agent, determine its state set, observation set, action set, reward function and final optimization goal; optimize the agent g corresponding to one of the base stations int The edge caching strategy of the other base stations corresponds to the intelligent agent g prv Interaction data with user groups as observation data; Step S3: for the reinforcement learning problem of the partially observable Markov decision process, using the observed data and considering the hidden state, optimizing the edge caching strategy based on causal reinforcement learning; The edge cache model in step S1 is: A cloud server serves N base stations at the same time. The total number of all files in the cloud server is recorded as M, the set of all files is recorded as F, and the mth file is recorded as f m , f m ∈F,m=0,1,...,M are all of the same size, B bits; each base station serves a corresponding user group, there is no overlap between user groups and they will not change over time; the storage capacity of the N base stations is equal, denoted by S station bits, and S station <M·B; the user group itself has no storage capacity; The total service time of the cloud server is T. N user groups will generate a request for a certain file at each time t∈T, denoted as r t , and r t ∈F; requests r generated by the user group t It will not be uploaded to its corresponding base station, and the user group's request is unknown to the base station; Each of the N base stations is equipped with a user group request prediction mechanism, which can predict the user group's request r in real time. t , the request prediction mechanism is: at time t, the base station first performs user group request prediction and obtains the prediction result If the file stored in the base station at this moment Contains The corresponding file is Then the base station will Transmit to user groups; If the storage file of the base station at time t Not included Right now The base station generates a request Request a file from the cloud server. After receiving the request, the cloud server will send the corresponding file at time t+1. Transmitted to the base station, the user group's request r t+1 =r t , the user group request prediction result of the base station If the base station's storage file at time t+1 Included Then the base station sends the file to Otherwise, it will continue to request the file from the cloud server at time t+2 until the storage file of the base station is Included Until then, the total waiting time of the user group during this period is recorded as n delay ; In step S2, agent g int The state set, observation set, action set, and reward function in the edge caching strategy are as follows: State Set: A state set is defined as the storage state of a base station and the requests of a user group: S={s t |t=0,1,2,...T} (1) The state at time t is defined as Observation Set: An observation set is defined as the storage status of a base station. O={o t |t=0,1,2,...T} (3) The observation at time t is defined as Action Set: Action set is defined as the actions of the base station A={a t |t=0,1,2,...T} (5) At time t, the base station can request the cloud server to add a file, delete a file, or neither add nor delete a file. The action at time t is defined as: <h2 style=";text-align:left;direction:ltr">a<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> (a1,a2,a3,...a)<h2 style=";text-align:left;direction:ltr"> M <h2 style=";text-align:left;direction:ltr"> ] (6) a t Each element in takes a value of -1, 0, or 1. m =-1, m = 1, 2, ... M means that the base station wants to delete the mth file f from its own storage space m If a m =1,m=1,2,...M means the base station wants to request the mth file f from the cloud server m And store it in its own storage space; if a m =0,m=1,2,...M, it means that the base station neither adds the file nor deletes the file; Reward function: The reward function consists of three parts: the reward function u at time t t Expressed as Among them, w1, w2 and w3 are the weights of the three parts respectively. w1+w2+w3=1 (8) And h t represents the hit rate at time t, which is defined as follows is the number of files stored in the base station, n delay Indicates waiting delay; Finally, the optimization problem of edge caching is summarized as maximizing the total reward in each round, which can be expressed as: The specific steps of step S3 include: Step S3.1: Get agent g prv observational data of the environment and intervention data of its own interaction with the environment; Step S3.2, estimating a true environment model based on the observation data and intervention data obtained in step S3.1; Step S3.3: According to the environment model estimated in step S3.2, the base station whose cache strategy needs to be optimized is trained using an Actor-Critic method based on the environment model to obtain an optimal edge cache strategy; In step S3.1, agent g prv The cache strategy π prv (a t |h t ,s t ) contains τ rounds of observation data D obs is obtained from a partially observable Markov decision process with the following distribution: D obs ~p init ,p trans ,p obs ,π prv (11) Among them, p init represents the initial distribution probability of the state, p trans represents the state transition probability, p obs represents the observation probability, π prv Represents agent g prv strategy; The specific form of the observation data is: D i =(o0,a0,...,o T ) (13) Contains k rounds of intervention data D int is a random strategy π(a t |h t ) int Collected from a standard partially observable Markov decision process D int ~p init ,p trans ,p obs ,π (14) Among them, p init represents the initial distribution probability of the state, p trans represents the state transition probability, p obs represents the observation probability, π represents the agent g int Strategy; Except for strategy π, the other three probabilities in formula (14) are the same as those in formula (11); The specific form of intervention data is: D i =(o0,a0,...,o T ) (16) Add an auxiliary variable I∈{0,1} to distinguish the two strategies, that is, i=0 represents the observed data: π(a t |h t ,s t ,i=0)=π prv (a t |h t ,s t ) (17) If i=1, it means intervention data: π(a t |h t ,s t ,i=1)=π std (a t |h t ) (18); In step S3.2, estimating the real environment model based on the observation data and intervention data obtained in step S3.1 specifically includes: a. Learning of environmental models: fitting a latent probability model based on observed data and intervention data Specifically, the hidden state s of the actual partially observable Markov decision process is t ∈S is replaced by a latent variable z t ∈Z, where Z is the discrete latent space of the environment model; thus, the learning of the environment model is expressed as the following standard maximum likelihood problem: Where Q is the set of the following probability models: Where q(i) represents the proportion of observed data and intervention data, q(z0) represents the distribution probability of the hidden state, q(o0|z0) represents the observation probability of the hidden state, and q(a t |h t ,z t ,i) represents the strategy of the agent. If i=1, then: q(a t |h t ,z t ,i=1)=q(a t |h t ,i=1) (21) q(z t+1 |a t ,z t ) represents the transition probability of the hidden state, q(o t+1 |z t+1 ) represents the observation probability of the hidden state at time t+1; b. Inference of environmental model: will restore As an estimator of transition probabilities for a standard partially observable Markov decision model, The potential probability model learned from step a above using the forward algorithm is derived from; Observation data D obs The intervention transfer model was affected by Specifically include: the learned model and There must be the same probability q(z0), q(o t |z t ) and q(z t+1 |z t ,a t ), only the agent’s strategy q(a t |h t ,z t , i=0) and q(a t |h t ,z t , i=1) can have some differences to distinguish the two solutions.

Images