A Collaborative Cache Update Method for Edge Regions Based on Multi-Agent Deep Reinforcement Learning

Abstract

This invention relates to edge caching scenarios and the field of multi-agent deep reinforcement learning, specifically a collaborative cache update method for edge regions based on multi-agent deep reinforcement learning. This update method establishes that edge cache nodes within the collaborative cache region can share cached content, and models the edge region collaborative cache update problem as a distributed partially observable Markov decision process, thereby improving edge cache hit rate and cache utilization. Based on this, a multi-agent deep reinforcement learning algorithm combining multi-timescale recurrent neural networks and graph convolutional neural networks is proposed. Multi-timescale RNN layers process partially observable information, inferring multi-level features of local observations based on historical information. Graph convolutional layers integrate the feature vectors of each agent's local region, utilizing the latent features generated by convolutional layers with progressively increasing receptive fields to learn collaboration between agents, and addressing the model scalability issue.

Description

Technical Field

[0001] This invention relates to edge caching scenarios and the field of multi-agent deep reinforcement learning, specifically to an edge region collaborative cache update method based on multi-agent deep reinforcement learning. Background Technology

[0002] According to a forecast report by the International Telecommunication Union, global mobile data traffic will reach 4,394 exabytes per month by 2030. Networks will face enormous pressure and severe challenges, which will easily cause network congestion and excessive network latency, affecting the quality of user service experience and placing higher demands on edge caching technology.

[0003] Due to limited cache space and diverse content on edge cache nodes, single-edge cache node caching schemes are often insufficient to meet practical requirements. To more effectively utilize limited storage resources and improve the diversity of cached content and the hit rate of the entire cache region, researchers have proposed a collaborative caching scheme among multiple edge cache nodes. By sharing cached content across multiple edge cache nodes, user request costs and access latency are minimized.

[0004] While edge cache nodes share cached content, their caching resources are often relatively limited for massive amounts of content. Identifying popular content and even predicting future content requests to update the edge cache accordingly is an effective way to improve cache utilization and hit rate. Due to the complexity of edge scenarios, the diversity of requests, the finiteness of resources, and network constraints, machine learning-based solutions are more effective than traditional strategies in solving complex edge caching problems. Many researchers use machine learning-based methods to address edge collaborative caching problems. Most proactive caching strategy research focuses on improving content popularity and the accuracy of user request prediction to improve cache hit rate. However, in edge scenarios, due to rapid content updates and dynamic changes in user needs, content popularity has time-varying characteristics, and the content request process has temporal locality. Deep learning models often struggle to predict accurately, and strategies based solely on prediction for cache deployment are no longer suitable for dynamic edge caching environments.

[0005] Some researchers have used deep reinforcement learning algorithms to alleviate the difficulties of the aforementioned methods, but many studies have failed to consider the following three aspects. First, most studies assume that content popularity is known or follows a Zipf distribution. This assumption implicitly uses an Independent Reference Model (IRM), meaning it assumes that the content library and popularity are static. However, in the real world, content popularity is dynamic and should be assumed to be unknown. Second, some studies assume that edge agents have a global observation of the environment, modeling edge collaborative caching as a Markov Decision Process (MDP). In reality, each edge cache node rarely obtains complete state information when making caching decisions, such as its own historical information and information from other nodes; that is, the state is only partially observable. Third, most research assumes a static and stable learning environment, designing and optimizing caching strategies for individual edge cache nodes using single-agent deep reinforcement learning algorithms. This approach is often unsuitable for collaborative scenarios, frequently resulting in locally optimal caching strategies. Furthermore, it fails to consider the dynamic and unstable nature of the environment in collaborative caching scenarios, where the environment changes with each agent's strategy and state. For each agent, the decisions of other agents influence its own strategy, making the environment prone to convergence. Additionally, it neglects model scalability issues. Some research addresses these problems by using multi-agent deep reinforcement learning algorithms, but abstracting the relationships and roles between edge cache nodes remains challenging. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an edge collaborative cache update method based on multi-agent deep reinforcement learning using multi-timescale cyclic graph convolutional Q-networks (MRGQN) to improve edge cache hit rate and effectively reduce user access latency. This method focuses on perceiving dynamically changing environments, achieving dynamic adaptation of collaborative caching strategies at the rapidly changing and resource-constrained network edge, increasing the influence between nodes, fully leveraging the advantages of cache content sharing, and improving cache hit rate and cache utilization throughout the entire edge collaborative region.

[0007] To achieve the above objectives, the technical solution specifically adopted by the present invention is as follows:

[0008] A collaborative cache update method for edge regions based on multi-agent deep reinforcement learning is proposed. This method assumes that edge cache nodes within the collaborative cache region can share cached content, and models the edge region collaborative cache update problem as a distributed partially observable Markov decision process (Dec-POMDP), including the following steps:

[0009] Step S1: Design the edge area collaborative caching system architecture as cloud-edge-user terminal, and construct the edge area collaborative caching system model; the edge area collaborative caching system has four types of devices: cloud data center, regional core node, edge cache node and user equipment. The cloud data center is deployed by a remote cloud server, the regional core node and the edge cache node are deployed by existing macro base stations and micro base stations, and the user equipment is the various mobile network devices used by users.

[0010] Step S2: Simulate the user request process using the Shot Noise Model (SNM) to construct a user request model;

[0011] This invention, based on the time-varying characteristics of content popularity in edge scenarios, uses a shot noise model (SNM)—a request model capable of describing the dynamic changes in content popularity—to simulate the user request process. SNM is a "file-level" business model that can realistically describe the dynamic changes in the file library and the arrival rate of each file request within a region, and can be used to evaluate the performance of caching networks.

[0012] Step S3: Construct a content service model for an edge region collaborative caching system

[0013] The process of setting up content services in an edge region collaborative caching system is discretely divided into a series of small, identical processes, and the time required for each step is a unit of time, i.e., a time step. For each time step, the following three phases are repeated: content request phase, content distribution phase, and cache update phase.

[0014] Step S4: Construct an optimization objective function with the goal of maximizing the edge cache hit rate within the edge collaborative caching region;

[0015] The optimization objective function is as follows:

[0016]

[0017]

[0018] c i，f ∈{0,1}; 1≤i≤I; 1≤f≤F.

[0019] Where I represents the number of edge cache nodes within the edge collaborative caching region, N represents the limited cache capacity of each edge cache node, and F represents the total amount of content stored in the cloud data center. and These represent edge cache nodes s respectively. i The local cache node hit rate and the adjacent cache node hit rate, c i，f It is a 0-1 variable representing the content d f At edge cache nodes iThe cache status.

[0020] Step S5: Construct a distributed partially observable Markov decision process (Dec-POMDP)

[0021] The collaborative cache update problem in edge regions is modeled as a distributed partially observable Markov decision process (Dec-POMDP), with the cache update phase serving as the first phase at each time step t, and three elements are constructed: observation space, action space, and reward.

[0022] Furthermore, the specific process of step S1 is as follows:

[0023] S1.1, Let s0 represent the cloud data center, where s0 stores all content that meets user needs, D = {d1, d2, ..., d...}. F The total number of contents is F;

[0024] S1.2 Divide the edge cache nodes into several edge collaborative cache regions according to their geographical location. Each edge collaborative cache region has I (I≥2) edge cache nodes.

[0025] S1.3, Let S = {s1, s2, ..., s} I} represents the edge cache nodes within the edge collaborative caching region. Each edge cache node has a finite cache capacity N, and its corresponding cache state is C = {C1, C2, ..., C}. I}, where C i ={c i，1 c i，2 c i，F}, c i，f ∈{0,1},c i，f =1 represents edge cache node s i Content d was cached f ;

[0026] S1.4 Deploy a regional core node s in each edge collaborative caching region. c It directly connects to all edge cache nodes in the edge collaborative cache area, collects node information, and provides proxy services;

[0027] S1.5 Connect all edge cache nodes in the edge collaborative cache area to each other via optical fiber, share cached content, and obtain content from cloud data center s0 via backhaul link.

[0028] Furthermore, in the content request phase:

[0029] S3.11: According to the user request model described in step S2, users within the service range of the edge collaborative caching area send several content requests through the wireless network;

[0030] S3.12: Select the edge cache node closest to the user within the edge collaborative caching region as the local edge cache node. i ;

[0031] S3.13: Local edge cache node s i The number of requests R to receive content requests from users and save all content. i ={r i，1 r i，2 ,...,r i，F}, then enter the content distribution phase of the edge area collaborative caching system.

[0032] Furthermore, in the content distribution stage:

[0033] S3.21: Local edge cache node s i Upon receiving a request, check if the requested content is cached; if so, then the local edge cache node s... i The content is transmitted wirelessly directly to the user; otherwise, proceed to step S3.22.

[0034] S 3.22: Through the regional core node s c Determine if other edge cache nodes in the edge collaborative caching area have cached the content; if so, the adjacent edge cache node that has a cache hit transmits the content to the local edge cache node via fiber optic cable. i Then local edge cache nodes s i Then send the content to the user; otherwise, proceed to step S3.23.

[0035] S3.23: When the content is not in the edge collaborative cache region, the region core node s c The request is forwarded to cloud data center s0, which then transmits it to the local edge cache node s. i Then, the local edge cache node s i Send to the user;

[0036] S3.24: After completing the content distribution work for all user requests within the time step, save the local cache node hit rate from step S3.21. And the adjacent cache node hit rate in step S3.22 Entering the cache update phase of the edge region collaborative caching system.

[0037] Furthermore, during the cache update phase:

[0038] S3.31: Through the regional core node s cCollect local information about edge cache nodes within the edge collaborative caching area, including the number of content requests for each edge cache node R = {R1, R2, ..., R...} I} and cache state C = {C1, C2, ..., C} I};

[0039] S3.32: Add edge cache node s i Abstracted into an agent i, the agent's action strategy, which is trained and learned using a multi-agent deep reinforcement learning algorithm, serves as the edge region collaborative caching strategy for edge caching nodes. This multi-agent deep reinforcement learning algorithm is a computational model composed of multiple neural networks.

[0040] S3.33: Each edge cache node within the edge collaborative caching area makes cache update decisions based on the collected information and the edge area collaborative caching strategy, proactively caching more popular content;

[0041] S3.34: Each edge cache node within the edge collaborative caching area performs a cache replacement operation based on the cache update decision, obtaining the content that needs to be cached from the cloud data center or other edge cache nodes and replacing the content that does not need to be cached with it;

[0042] S3.35: After completing the cache update of all edge cache nodes within the edge collaborative cache area, proceed to the next time step.

[0043] Furthermore, the cache update phase in step S5 will be the first phase for each time step t, and the observation space, action space, and reward will be specifically set as follows:

[0044] Step S5.1: Set the observation space

[0045] At each time step t, the regional core node will collect information from all edge cache nodes within the cooperative domain, using this as the joint observation space of the system, represented as follows:

[0046]

[0047] Among them, R t-1，i Represented as edge cache node s in the previous time step t-1 i The number of requests for all content, C t-1，i Represented as edge cache node s in the previous time step t-1 i The cache state, Let i be the observation space of the edge cache node; the observation space is the set of all observations.

[0048] Step S5.2: Set the motion space

[0049] After obtaining the observations at each time step, the edge region collaborative caching system makes a cache update decision, that is, updates the cache state of each edge cache node. Therefore, the joint action space of the system (the set of all selectable actions) is represented as follows:

[0050]

[0051] Among them, C t，i Represents the edge cache node s in the current time step after the cache update. i The cache state, Represented as edge cache node s i The action space.

[0052] Step S5.3: Set system rewards

[0053] The optimization objective in step S4 is considered as the reward for edge cache nodes in the edge collaborative caching system. After the cache update phase, the system proceeds to the content request phase and the content distribution phase. After the edge collaborative caching system completes content distribution, the edge cache nodes will share the global reward r fed back by the environment. t The reward is represented as follows:

[0054]

[0055] Where I represents the number of edge cache nodes within the edge collaborative caching area. and These represent the edge cache nodes s in the current time step. i The local cache node hit rate and the adjacent cache node hit rate.

[0056] For edge collaborative caching systems, the goal is to find the optimal strategy to maximize the global reward, that is, to maximize the edge cache hit rate within the collaboration.

[0057] Furthermore, this invention also provides an MRGQN-based multi-agent deep reinforcement learning algorithm applied in step S3.32. This algorithm framework comprises three parts: a Q-network, a target Q-network, and an experience replay buffer. The Q-network and the target Q-network are specific neural network models that perform nonlinear function approximation of the Q-value function and the target Q-value function, respectively. They have the same structure, mainly consisting of a multi-timescale recurrent RNN module, two graph convolutional layers, and a multi-layer perceptron layer, using multi-head dot product attention as the convolutional kernel of the graph convolutional neural network. The Q-value function is the basis for calculating the action policy of each agent, and the Q-network is the specific implementation of the action policy calculation model for each agent. The network parameters of the calculation model are trained and updated to optimize the action policy, i.e., to optimize the collaborative caching strategy in the edge region. During training, the network parameters are shared by each agent, and the training steps are as follows:

[0058] Step 1: Randomly initialize the Q-network parameters θ and the target Q-network parameters θ′ in the model, as well as the average return estimate. And initialize the experience replay buffer;

[0059] Step 2: For each time step t, the interaction between each agent i and the environment:

[0060] Step 2.1: Each agent i receives the current local observation value o. t，i Based on the Q-value function, the estimated Q-values ​​of all actions are calculated, and an ε-greedy policy is used to select an action 'a'. t，i That is, there is a probability of 1-ε of choosing the action that maximizes the estimated Q value. The remaining ε probabilities select random actions; where a t，i Let represent the action chosen by agent i at the current time step t, and let Q(o, a; θ) represent the Q-value function approximated using a Q-network, where θ are the parameters of the Q-network, and o t，i For the core nodes in the region at the current time step t, collect the edge cache nodes s within the collaborative domain. i Local information, including the number of content requests and cache status of edge cache nodes; Indicates the current time step t and the edge cache nodes s i The set of all selectable caching actions, that is, all combinations of selecting N content items to cache from F content items; 'a' represents a specific action in the action space.

[0061] Step 2.2: Each agent i executes action a. t，i Receive reward r t and the local observation o at the next time step t+1 t+1，i ;

[0062] Step 2.3: Transfer experience (o) t a t r t o t+1 Saved to the experience replay buffer;

[0063] Step 2.4: Once the experience replay buffer is full, the process of learning and updating parameters will begin.

[0064] Step 3: Parameter Update:

[0065] Step 3.1: Use the minibatch strategy to randomly sample S empirical data from the empirical replay buffer as training data; where S represents the number of randomly sampled empirical data.

[0066] Step 3.2: Calculate the target TD value using the target Q network. Where, r i This represents the reward value in the empirical data sample i. Let Q(o, a; θ′) represent the current average return estimate, and let Q(o, a; θ′) represent the Q-value function approximated using the target Q-network, where θ′ is the parameter of the target Q-network. i ′ represents the observation value in the next time step of the empirical data sample i, and a′ represents a certain action in the action space at the next time step;

[0067] Step 3.3: Calculate the TD error δ = y i -Q(o i a i ;θ), update Among them, y i That is, the TD target value in step 3.2, Q(o, a; θ) is the current Q value function described in step 2.1, o i and a i Let i represent the observed value and the action in the empirical data sample i, respectively. The average return estimate in step 3.1 is γ, where γ represents the update coefficient of the average return estimate.

[0068] Step 3.4: Calculate the TD loss function Gradient descent is used to update the Q-network parameters θ; where S is the number of random samples in step 3.1, I is the number of agents, δ is the TD error in step 3.3, λ represents the coefficient of regularization loss, and M represents the number of attention heads in the Q-network. This represents the distribution of attention weights of agent i in attention head m of graph convolutional layer k;

[0069] Step 3.5: After a fixed number of steps C or the current reward r t Less than a certain threshold r min When the target Q network parameter θ′ is softly updated, θ′=βθ+(1-β)θ′; where β represents the coefficient of the soft update;

[0070] Step 3.6: If ε is greater than ε min Then update ε = ε * ε decay ; where ε, ε min ε decay Here are the parameters for the ε-greedy strategy, representing the exploration rate, minimum exploration rate, and exploration decay coefficient, respectively.

[0071] Step 3.7: After completing the above steps, proceed to the interaction process of the next time step.

[0072] This invention has the following characteristics and beneficial effects:

[0073] This invention proposes a collaborative cache update method for edge regions based on multi-agent deep reinforcement learning. This method focuses on perceiving dynamically changing environments and achieving dynamic adaptation of collaborative caching strategies at the network edge where changes are instantaneous and resources are limited. It increases the influence between nodes, fully leverages the advantages of shared cached content, improves cache hit rate and cache utilization within the entire edge collaborative region, and reduces user access latency. At the same time, this method models the edge region collaborative cache update problem as a distributed partially observable Markov decision process and defines the observation space, action space, and reward to maximize the edge cache hit rate within the collaboration.

[0074] This invention proposes a multi-agent deep reinforcement learning algorithm based on multi-timescale recurrent RNNs and graph convolutional neural networks (MRGQN). This algorithm leverages the dynamic changes in content popularity in edge scenarios, introducing multi-timescale recurrent RNNs to enable MRGQN to better handle partially observable situations. It infers multi-level features of local observations based on historical information, thus analyzing dynamically changing content popularity. Simultaneously, considering the interaction and mutual influence between adjacent agents, MRGQN utilizes graph convolutional neural networks to collect more valuable information to adapt to the dynamics of the multi-agent environment. The relation kernel captures the interactions between agents through the representation of relationships between them. It learns agent collaboration by utilizing latent features generated by convolutional layers with progressively increasing receptive fields, and addresses the model scalability issue. Based on this multi-agent deep reinforcement learning algorithm based on multi-timescale recurrent RNNs and graph convolutional neural networks (MRGQN), this invention provides an edge region collaborative cache update method that enables policy collaboration during edge cache node updates. This fully leverages the advantages of shared cache content strategies within the collaborative cache region, effectively improving the edge cache hit rate. Attached Figure Description

[0075] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0076] Figure 1 This is a diagram of an edge region collaborative caching scenario in an embodiment of the present invention.

[0077] Figure 2 This is a flowchart of the edge region collaborative caching system in an embodiment of the present invention.

[0078] Figure 3 This is a diagram of the MRGQN multi-agent deep reinforcement learning algorithm architecture in an embodiment of the present invention.

[0079] Figure 4 This is a diagram showing the internal structure of an agent network based on MRGQN multi-agent deep reinforcement learning in an embodiment of the present invention. Detailed Implementation

[0080] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0081] This invention considers an edge collaborative caching scenario, which includes a remote cloud server, several requesting users, and several macro base stations and micro base stations. The edge region collaborative caching system architecture is constructed as cloud-edge-device, and four types of devices are defined: cloud data center, regional core node, edge cache node, and user devices, such as... Figure 1 As shown. Cloud data centers are deployed using remote cloud servers, regional core nodes and edge cache nodes are deployed using macro base stations and micro base stations, and user equipment refers to the various mobile network devices that users request to use.

[0082] The process of setting up content service in the edge region collaborative caching system is discretized into a series of small, identical processes, with each step requiring a unit of time, i.e., a time step. Within each time step, based on the time-varying characteristics of content popularity in the edge scenario, a shot noise model (SNM), which describes the dynamic changes in content popularity, is used to simulate the user request process and construct a user request model. Upon receiving a request, the local edge cache node checks if it has cached the requested content. If so, it directly transmits the content wirelessly to the user; otherwise, the regional core node checks if other edge cache nodes in the collaborative domain have cached the content. If so, the adjacent edge cache node with the cache hit transmits the content to the local edge cache node via fiber optic cable, and then the local edge cache node sends the content to the user; otherwise, the regional core node forwards the request to the cloud data center, which transmits it to the local edge cache node, and then the local edge cache node sends the content to the user.

[0083] For each time step, there are three phases: content request phase, content distribution phase, and cache update phase, such as... Figure 2 As shown. Specifically, during the content request phase, according to the user request model, users within the service range of the edge collaborative caching area send several content requests via the wireless network; the edge cache node closest to the user within the edge collaborative caching area is selected as the local edge cache node s. i Local edge cache nodes s i The number of requests R to receive content requests from users and save all content. i ={r i，1 r i，2, ..., r i，F}, then enter the content distribution phase of the edge area collaborative caching system.

[0084] During the content distribution phase, local edge cache nodes s i Upon receiving a request, check if the requested content is cached; if so, then the local edge cache node s... i The content is transmitted wirelessly directly to the user; otherwise, it is transmitted through the regional core node. c Determine if other edge cache nodes in the edge collaborative caching area have cached the content; if so, the adjacent edge cache node that has a cache hit transmits the content to the local edge cache node via fiber optic cable. i Then local edge cache nodes s i Then send the content to the user; otherwise, if the content is not in the edge collaborative cache area, the regional core node s c The request is forwarded to cloud data center s0, which then transmits it to the local edge cache node s. i Then, the local edge cache node s i Send to users; after completing the content distribution for all user requests within the time step, save the local cache node hit rate. and adjacent cache node hit rate Entering the cache update phase of the edge region collaborative caching system.

[0085] During the cache update phase, through the regional core node s c Collect local information about edge cache nodes within the edge collaborative caching area, including the number of content requests for each edge cache node R = {R1, R2, ..., R...} I} and cache state C = {C1, C2, ..., C I}; cache edge nodes s i Abstracted into an agent i, the action policy of the agent, trained using a multi-agent deep reinforcement learning algorithm, serves as the edge region collaborative caching strategy for edge cache nodes. This multi-agent deep reinforcement learning algorithm is a computational model composed of multi-layer neural networks. Each edge cache node within the edge collaborative caching region makes a cache update decision based on the collected information and the edge region collaborative caching strategy, proactively caching more popular content. Each edge cache node within the edge collaborative caching region executes a cache replacement operation based on the cache update decision, obtaining the content that needs to be cached from the cloud data center or other edge cache nodes and replacing the content that does not need to be cached with it. After completing the cache update of all edge cache nodes within the edge collaborative caching region, the next time step begins.

[0086] The framework based on multi-agent deep reinforcement learning algorithm is as follows: Figure 3 As shown, it comprises three parts: a Q-network, a target Q-network, and an experience replay buffer. The Q-network and the target Q-network are specific neural network models that perform nonlinear function approximations of the Q-value function and the target Q-value function, respectively, and have the same structure, as shown below. Figure 4 As shown, it mainly consists of a multi-timescale recurrent RNN module, two graph convolutional layers, and a multi-layer perceptron layer, using multi-head dot product attention as the convolutional kernel of the graph convolutional neural network. The multi-timescale RNN module processes some observable information and infers multi-level features of local observations based on historical information. The graph convolutional layers integrate the feature vectors of local regions for each agent to generate latent feature vectors. By stacking more convolutional layers, the agent's receptive field gradually increases, collecting more information, and thus increasing the scope of cooperation. The Q-value function is the basis for calculating the action policy of each agent, and the Q-network is the specific implementation of the calculation model for each agent's action policy. The network parameters of the calculation model are trained and updated to optimize the action policy, that is, to optimize the collaborative caching strategy in the edge region.

[0087] The algorithm model is deployed on the core nodes of the region for training and optimization of the action strategy of each agent, i.e., the edge region collaborative caching strategy of the edge caching nodes. During training, the network parameters of each agent are shared. The specific training steps are as follows:

[0088] Step 1: Randomly initialize the Q-network parameters θ and the target Q-network parameters θ′ in the model, as well as the average return estimate. And initialize the experience replay buffer;

[0089] Step 2: For each time step t, the interaction between each agent i and the environment;

[0090] Step 2.1: Each agent i receives the current local observation value o. t，i Based on the Q-value function, the estimated Q-values ​​of all actions are calculated, and an ε-greedy policy is used to select an action 'a'. t，i That is, there is a probability of 1-ε of choosing the action that maximizes the estimated Q value. The remaining ε probabilities select random actions; where a t，i Let represent the action chosen by agent i at the current time step t, and let Q(o, a; θ) represent the Q-value function approximated using a Q-network, where θ are the parameters of the Q-network, and o t，i For the core nodes in the region at the current time step t, collect the edge cache nodes s within the collaborative domain. i Local information, including the number of content requests and cache status of edge cache nodes; Indicates the current time step t and the edge cache nodes s iThe set of all selectable caching actions, that is, all combinations of selecting N content items to cache from F content items; 'a' represents a specific action in the action space.

[0091] Step 2.2: Each agent i executes action a. t，i Receive reward r t and the local observation o at the next time step t+1 t+1，i ;

[0092] Step 2.3: Transfer experience (o) t a t r t o t+1 Saved to the experience replay buffer;

[0093] Step 2.4: Once the experience replay buffer is full, the process of learning and updating parameters will begin.

[0094] Step 3: Parameter Update

[0095] Step 3.1: Use the minibatch strategy to randomly sample S empirical data from the empirical replay buffer as training data; where S represents the number of randomly sampled empirical data.

[0096] Step 3.2: Calculate the target TD value using the target Q network. Where, r i This represents the reward value in the empirical data sample i. Let Q(o, a; θ′) represent the current average return estimate, and let Q(o, a; θ′) represent the Q-value function approximated using the target Q-network, where θ′ is the parameter of the target Q-network. i ′ represents the observation value in the next time step of the empirical data sample i, and a′ represents a certain action in the action space at the next time step;

[0097] Step 3.3: Calculate the TD error δ = y i -Q(o i a i ;θ), update Among them, y i That is, the TD target value in step 3.2, Q(o, a; θ) is the current Q value function described in step 2.1, o i a i Let i represent the observed value and the action in the empirical data sample i, respectively. The average return estimate in step 3.1 is γ, where γ represents the update coefficient of the average return estimate.

[0098] Step 3.4: Calculate the TD loss function Gradient descent is used to update the Q-network parameters θ; where S is the number of random samples in step 3.1, I is the number of agents, δ is the TD error in step 3.3, λ represents the coefficient of regularization loss, and M represents the number of attention heads in the Q-network. This represents the distribution of attention weights of agent i in attention head m of graph convolutional layer k;

[0099] Step 3.5: After a fixed number of steps C or the current reward r t Less than a certain threshold r min When the target Q network parameter θ′ is softly updated, θ′=βθ+(1-β)θ′; where β represents the coefficient of the soft update;

[0100] Step 3.6: If ε is greater than ε min Then update ε = ε * ε decay ;

[0101] Step 3.7: After completing the above steps, proceed to the interaction process of the next time step.

[0102] In this context, the interaction between each agent and the environment during the training process constitutes the cache update phase of each edge cache node in the edge region collaborative caching scenario; the size of the experience replay cache is set to 1024; the total amount of content F stored in the cloud data center and the limited cache capacity N of each edge cache node are set to 10000 and 10, respectively; the number of random samples S is set to 32; the update coefficient γ of the average revenue estimate is set to 0.01; the coefficient λ of the regularization loss is set to 0.03; ε, ε min ε decay The parameters for the ε-greedy strategy are , representing the exploration rate, minimum exploration rate, and exploration decay coefficient, respectively, set to 0.9, 0.1, and 0.999; C, β, r min These are the parameters for soft updates, set to 200, 0.01, and 0.8 respectively.

[0103] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An edge region collaborative cache updating method based on multi-agent deep reinforcement learning, characterized in that, Comprise the following steps: S1, the edge area collaborative cache system architecture is designed as cloud edge-user, build edge area collaborative cache system model; S2, the user request process is simulated by using the shot noise model, build user request model; S3, build edge area collaborative cache system content service model: S3.1, set content request stage; S3.2, set content distribution stage; S3.3, set cache update stage; S4, with the maximum edge collaborative cache area in the edge cache hit rate as the target, build optimization objective function; S5, build distributed partially observable Markov decision process: The edge area collaborative cache update problem is modeled as Dec-POMDP, and three elements are built: observation space, action space, and reward.

2. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In the step S1, the edge area collaborative cache system exists cloud data center, regional core node, edge cache node and user equipment four kinds of equipment, wherein, the cloud data center is realized by remote cloud server deployment, the regional core node and the edge cache node are realized by the existing macro base station and micro base station deployment, and the user equipment is various mobile network equipment used by users.

3. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, The specific process of step S1 is as follows: S1.1, Let s0 represent the cloud data center, where s0 stores all content that meets user needs, D = {d1, d2, ..., d...}. F The total number of contents is F; S1.2, according to the geographical position, the edge cache node is divided into several edge collaborative cache areas, and there are I (I≥2) edge cache nodes in each edge collaborative cache area; S1.3, set S = {s1, s2,..., s I} represents edge cache nodes within the edge cooperative cache area, each edge cache node has a limited cache capacity N, and the corresponding cache state is C = {C1, C2,..., C I} i = {c i,1 ,c i,2 ,...,c i,F}, c i,f ∈ {0, 1}, c i,f = 1 represents that the edge cache node s i caches the content d f ; S1.4, deploy a regional core node s inside each edge cooperative cache region c Directly connect all edge cache nodes in the edge cooperative cache region, collect node information, and provide proxy services; S1.5, through the optical fiber, all edge cache nodes in the edge collaborative cache area are connected with each other, share cache content, and obtain content from the cloud data center s0 through the backhaul link.

4. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In the step S3, the following three stages are repeated at each time step: content request stage, content distribution stage and cache update stage.

5. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, The specific process of step S3.1 is as follows: S3.11: according to the user request model of step S2, the user in the service range of the edge collaborative cache area sends several content requests through the wireless network; S3.12: Select the edge cache node closest to the user within the edge collaboration cache area as the local edge cache node s i ; S3.13: Local edge cache node s i Upon receiving the user's content request, save the request times R of all contents i = {r i,1 , r i,2 ,..., r i,F , enter the content distribution phase of the edge area collaborative cache system.

6. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, The specific process of step S3.2 is as follows: S3.21: Local edge cache node s i Upon receiving the request, it checks whether it has cached the content requested by the user; if so, the local edge cache node s i transmits the content directly to the user wirelessly; Otherwise, go to step S3.22; S3.22: Through the regional core node s c determines whether other edge cache nodes in the edge collaborative cache region cache the content; if so, the neighboring edge cache node of the cache hit transmits the content to the local edge cache node s through the optical fiber i Then the local edge cache node s i sends the content to the user again; Otherwise, go to step S3.23; S3.23: When the content is not in the edge collaboration cache region, the regional core node s c forwards the request to the cloud data center s0, which is transmitted to the local edge cache node s i by the local edge cache node s i to the user; S3.24: save the local cache hit rate in step S3.21 when all content distribution work requested by users in the time step is completed and the adjacent cache hit rate in step S3.22 Enter the cache update phase of the edge area cooperative cache system.

7. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, The specific process of step S3.3 is as follows: S3.31: Through the regional core node s c Collect local information about edge cache nodes within the edge collaborative caching area, including the number of content requests for each edge cache node, R = {R1, R2, ..., R...}. I } and cache state C = {C1, C2, ..., C I }; S3.32: The edge cache node s i The agent i is abstracted, and the action strategy of the agent obtained by training and learning based on the multi-agent deep reinforcement learning algorithm is used as the edge area cooperative caching strategy of the edge cache node. The multi-agent deep reinforcement learning algorithm is a calculation model composed of multiple neural networks; S3.33: each edge cache node in the edge collaborative cache area makes cache update decision according to the collected information and the edge area collaborative cache strategy, and actively caches more popular content; S3.34: each edge cache node in the edge collaborative cache area executes cache replacement operation according to the cache update decision, obtains the content to be cached from the cloud data center or other edge cache nodes, and replaces the content not to be cached; S3.35: after completing the cache update of all edge cache nodes in the edge collaborative cache area, go to the next time step.

8. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In the step S4, the optimization objective function is as follows: c i，f ∈ {0, 1}; 1≤i≤I; 1≤f≤F where I denotes the number of edge cache nodes in the edge-cooperative cache region, N denotes the limited cache capacity of each edge cache node, F denotes the total number of contents stored in the cloud data center, and denote the local cache hit rate and the adjacent cache hit rate of edge cache node s i respectively, c i，f is a 0-1 variable, indicating the cache status of content d f in edge cache node s i .

9. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In the step S5, the cache update stage will be the first stage of each time step t, and the observation space, action space and reward are set as follows: S5.1, set observation space: At the beginning of each time step t, the regional core node will collect the information of all edge cache nodes in the edge collaborative cache region, taking this as the joint observation space of the system which is represented as follows: wherein R t-1，i represents the edge cache node s i the number of requests for all content, C t-1，i represents the edge cache node s i 's cache state, represents the observation space of the edge cache node i; S5.2, set action space: After obtaining the observation value of each time step, the edge area cooperative caching system makes a caching update decision to update the cache state of each edge cache node, so the joint action space of the system is represented as follows: where C t，i represents the cache state of edge cache node s i at the current time step after cache update, represents the action space of edge cache node s i ; S5.3, set system reward: The optimization target of step S4 is regarded as the reward of the edge cache node in the edge cooperative cache system, and the content request stage and the content distribution stage are entered after the cache update stage; after the edge cooperative cache system completes the content distribution stage, the edge cache node will feed back the global reward r t , the global reward r t is represented as follows: where I denotes the number of edge cache nodes in the edge cooperative cache region, and denote the local cache hit rate and the neighboring cache hit rate of the edge cache node s i in the current time step, respectively.

10. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 7, characterized in that, The multi-agent deep reinforcement learning algorithm in step S3.32 comprises three parts: a Q network, a target Q network, and an experience replay buffer, wherein the Q network and the target Q network are specific neural network models for nonlinear function approximation of a Q value function and a target Q value function, respectively, have the same structure, comprise a multi-time scale recurrent neural network (RNN) module, two graph convolution layers, and a multi-layer perceptron layer, and use multi-head dot-product attention as a convolution kernel of a graph convolutional neural network; the Q value function is a basis for calculating an action policy of each agent, the Q network is a specific implementation of a calculation model of the action policy of each agent, network parameters of the calculation model are trained and updated to optimize the action policy, i.e., to optimize the edge region collaborative caching strategy, and network parameters of each agent are shared.

11. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 10, characterized in that, The multi-agent deep reinforcement learning algorithm is obtained by the following steps: Step 1: Randomly initialize the Q-network parameters θ and target Q-network parameters θ' in the model, as well as the average reward estimate and initialize the experience replay buffer; Step 2: for each time step t, each agent i interacts with the environment; Step 3: parameter updating.

12. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 11, characterized in that, The specific process of step 2 is as follows: Step 2.1: Each agent i receives the current local observation o t，i , calculates the estimated Q values of all actions based on the Q value function, and selects an action a t，i using an ε-greedy policy, that is, there is a probability of 1-ε to select the action with the maximum estimated Q value , and the remaining probability of ε to select a random action; wherein a t，i represents the action selected by agent i at the current time step t, Q(o, a; θ) represents the Q value function approximated by the Q network, θ is the parameter of the Q network, and o t，i is the local information of the edge cache node s i collected by the regional core node at the current time step t, including the content request times and the cache state of the edge cache node; represents the set of all selectable cache actions of the edge cache node s i at the current time step t, that is, all selection combinations of selecting N contents from F contents; a represents a certain action in the action space; Step 2.2: Each agent i performs action a t，i , obtains reward r t , and the local observation o t+1，i for the next time step t+1; Step 2.3: Save the experience (o t , a t , r t , o t+1 ) into the experience replay buffer; Step 2.4: when the experience replay buffer is filled with experiences, the process of learning and updating parameters starts.

13. The edge region collaborative cache updating method based on multi-agent deep reinforcement learning according to claim 11, characterized in that, The parameter updating in step 3 is to optimize the calculation model of the action policy, i.e., to optimize the edge region collaborative caching strategy, and the specific process is as follows: Step 3.1: S experience data are randomly sampled from the experience replay buffer as training data using a minibatch strategy; wherein S represents the number of randomly sampled experience data; Step 3.2: Compute TD target value by target Q network where r i denotes the reward value in the experience data sample i, is the current mean return estimate, Q(o, a; θ') denotes the Q value function approximated by the current target Q network, θ' is the parameter of the target Q network, o i ' denotes the observation value at the next time step in the experience data sample i, a' denotes a certain action in the action space at the next time step; Step 3.3: Calculate the TD error δ = y i -Q(o i a i ;θ), update Among them, y i That is, the TD target value in step 3.2, Q(o, a; θ) is the current Q value function described in step 2.1, o i and a i Let i represent the observed value and the action in the empirical data sample i, respectively. The average return estimate in step 3.1 is γ, where γ represents the update coefficient of the average return estimate. Step 3.4: Calculate the TD loss function Update the Q network parameters θ by gradient descent; wherein S is the number of random samplings in the step 3.1, I is the number of agents, δ is the TD error in the step 3.3, λ represents the coefficient of the regularization loss, M represents the number of attention heads in the Q network, represents the attention weight distribution of the agent i in the attention head m in the graph convolution layer k; Step 3.5: Fixed number of steps C or current reward r t Less than a certain threshold r min Soft update the target Q network parameters θ', θ' = βθ + (1 - β)θ' ; where β represents the coefficient of soft update. Step 3.6: If ε is greater than ε min , then update ε = ε * ε decay ; wherein ε, ε min , ε decay are parameters of the ε-greedy policy, representing the exploration rate, the minimum exploration rate, and the exploration decay coefficient, respectively. Step 3.7: after the above steps are completed, the interaction process of the next time step starts.

Images