Edge caching method and system based on multi-agent reinforcement learning and federated learning

Abstract

The present invention provides an edge caching method and system based on multi-agent reinforcement learning and federated learning, the method comprising: constructing a MADRL framework including states, actions, rewards and strategies, and using federated deep learning to predict popular content within the coverage of an SBS; based on the MADRL framework, the SBS adjusts the popular content cached locally by the SBS according to the local state and the state of adjacent SBSs. The SBS of the present invention can predict the popularity of content within its coverage; and can effectively adjust the popular content cached locally by the SBSs, thereby realizing the effective use of distributed edge caching.

Description

Technical Field

[0001] The present invention relates to the field of data caching technology, and in particular to an edge caching method and system based on multi-agent reinforcement learning and federated learning. Background Art

[0002] In recent years, with the increasing penetration of smart devices, mobile data traffic has grown unprecedentedly, which has placed a heavy traffic burden on wireless networks. As a result, wireless networks may become very congested and it is difficult to satisfy the requests of User Equipments (UEs) with satisfactory service quality. To address this challenge, edge caching has been proposed as a promising solution for next-generation network communications. Edge caching can be achieved by assigning cache units in wireless edge nodes (e.g., small-cell base stations (SBSs)) so that UEs can pre-fetch popular content nearby. Subsequently, the cached content can be delivered to the UE without repeated transmission in the front-end and back-end links. This process significantly reduces the traffic load, alleviates network congestion, reduces latency, and thus improves system performance.

[0003] Due to the unique preferences of each UE, the popularity of content in different SBSs may show surprising differences. In this case, each SBS is required to predict the popularity of content within its coverage area. Machine Learning (ML) can effectively predict popular content by extracting hidden features through training user data. Most machine learning ML algorithms train models in a centralized manner, where data generated by multiple UEs must be sent to wireless edge nodes for analysis. These generated data may involve personal sensitive information of UEs. Not only that, as the number of UEs within the coverage area of ​​SBSs increases, the data generated by UEs will also increase. Due to the high computational and communication costs incurred, centralized machine learning ML algorithms may have difficulty processing such data. Federated Deep Learning (FDL) has become a potentially feasible solution.

[0004] In addition, when the storage capacity of SBS is limited, after each SBS predicts the popularity of the content within its coverage area, it becomes a difficult problem to decide which SBS to cache the popular content predicted by different SBSs. The dimension of the cache operation increases with the number of SBSs and content, which inevitably increases the complexity of the cache problem in the entire system, making it difficult to coordinate the SBS cache in the next generation network to effectively utilize the distributed edge cache. Summary of the invention

[0005] To this end, an embodiment of the present invention provides an edge caching method and system based on multi-agent reinforcement learning and federated learning, which is used to solve the problem in the prior art that it is difficult to coordinate SBS cache in the next generation network to effectively utilize distributed edge cache.

[0006] To this end, an embodiment of the present invention provides an edge caching method based on multi-agent reinforcement learning and federated learning, the method comprising:

[0007] Construct a MADRL framework including states, actions, rewards, and strategies, and use federated deep learning to predict popular content within the coverage area of ​​SBS;

[0008] The method of using federated deep learning to predict popular content within the coverage of SBS includes the following steps:

[0009] Iteratively train the adversarial encoder model using federated deep learning;

[0010] The SBS sends the trained adversarial encoder model to all UEs within the coverage area, sets the active users in the UE, selects the neighboring users of each active user and constructs the matrix;

[0011] The number of contents of interest to each user is counted, and the UE selects Pn contents with the largest content popularity as the predicted contents of interest to the UE, where the number of contents of interest is called the content popularity of the content;

[0012] After all UEs within the coverage of the SBS upload their predicted interesting contents, the SBS compares the predicted interesting contents uploaded by all UEs and selects Pn most interesting contents as predicted popular contents;

[0013] Based on the MADRL framework, an SBS adjusts its locally cached popular contents according to the local state and the states of neighboring SBSs.

[0014] Preferably, a MADRL framework including states, actions, rewards and strategies is constructed, specifically including:

[0015] State: The local state of SBSb at time slot t Defined as The state of the system is defined as in, is the local cache status of SBSb, p b It is the content predicted to be popular in the SBSb coverage area;

[0016] Action: In time slot t, the action of SBSb is defined as The action of the system is defined as Among them, F p =|pb |Represents the number of predicted popular contents;

[0017] Reward: At time slot t, the reward of SBSb is defined as The system rewards are Among them, α, β, χ and η are predefined constants, and It is the statistical data of the number of contents acquired by UEs within the coverage of SBSb during the time interval [t, t+1);

[0018] Strategy: Use π={π1,...,π b ,...,π B} represents the caching strategy to maximize the expected long-term discounted return π * =argmin J(π), where is the discount factor.

[0019] Preferably, iteratively training the adversarial encoder model using federated deep learning comprises the following steps:

[0020] SBS initializes its global model ω in the first round of iterative training r , in each round of iterative training, SBS updates the global model and converts the global model ω r Delivered to all UEs within its coverage area;

[0021] Each UE under the coverage of the SBS will download the global model ω r Set it as its own initial local model and update the local model through training iterations;

[0022] After the local model is updated, UE i will update the local model Upload to local SBS;

[0023] All UEs within the SBS coverage area upload their updated local models After that, the SBS calculates all the updated local models received The weighted average of r+1 , and its calculation formula is:

[0024]

[0025] Among them, η is a fixed learning rate;

[0026] Through multiple rounds of iterative training, the final global model is obtained, and the adversarial encoder model completes the training.

[0027] Preferably, during the local model update process, a flexible local update algorithm is used to set a specific weight for each UE, and a personalized local model is trained for each UE based on the difference between the global model and the local model.

[0028] Preferably, in the local model update process, a flexible local update algorithm is used to set a specific weight for each UE, and training a personalized local model for each UE based on the difference between the global model and the local model specifically includes:

[0029] First, define the weight distance formula, the formula is as follows:

[0030]

[0031] Among them, ω a and ω b Represent the weights of the two models, dis(ω a ,ω b ), the larger the difference between the two models;

[0032] Then calculate each layer of local model separately and the global model ω r The difference and introduction of α i As the elasticity parameter, the calculation formula is as follows:

[0033]

[0034] in, Is a local model The weight of the lth layer, is the global model ω r The weight of the lth layer of |L| represents the number of layers of the deep learning network, which is used to average the weight distance function of each layer, α i The larger it is, the greater the gap between the global model and the local model.

[0035] Finally, according to the elastic update formula Update the local model.

[0036] Preferably, the SBS sends the trained adversarial encoder model to all UEs within the coverage area, sets active users in the UE, selects neighboring users of each active user and constructs a matrix specifically including:

[0037] SBS sends the trained adversarial encoder model to all UEs within the coverage area and converts the rating matrix X of UE i to the content i As input to the trained adversarial encoder model, the reconstructed rating matrix is ​​output

[0038] Calculate the number of contents with non-zero scores for each user in the UE, and mark a set number of users with the maximum number of non-zero contents as active users;

[0039] UE will reconstruct the scoring matrix and its personal information matrix H i Combined into a combination matrix Hi, calculate the similarity between each active user and other users;

[0040] For each active user a, UE selects the users with the K largest similarities as the K neighboring users of user a and extracts the rating matrix X i The vectors of K neighboring users of each active user in and construct the matrix

[0041] Preferably, the method for calculating the similarity between each active user and other users is:

[0042] The similarity between each active user and other users is calculated based on the cosine similarity, and the calculation formula is:

[0043]

[0044] Among them, H i (a,:) and H i (b,:) are the vectors corresponding to the active users a and b in the combination matrix Hi, respectively. i (a,:)‖2 and ‖H i (b,:)‖2 are H i (a,:) and H i 2-norm of (b,:).

[0045] The embodiment of the present invention further provides an edge cache system based on multi-agent reinforcement learning and federated learning, the system comprising:

[0046] Popular content prediction module, which is used to build a MADRL framework including states, actions, rewards and strategies, and uses federated deep learning to predict popular content within the coverage area of ​​SBS;

[0047] The method of using federated deep learning to predict popular content within the coverage of SBS includes the following steps:

[0048] Iteratively train the adversarial encoder model using federated deep learning;

[0049] The SBS sends the trained adversarial encoder model to all UEs within the coverage area, sets the active users in the UE, selects the neighboring users of each active user and constructs the matrix;

[0050] The number of contents of interest to each user is counted, and the UE selects Pn contents with the largest content popularity as the predicted contents of interest to the UE, where the number of contents of interest is called the content popularity of the content;

[0051] After all UEs within the coverage of the SBS upload their predicted interesting contents, the SBS compares the predicted interesting contents uploaded by all UEs and selects Pn most interesting contents as predicted popular contents;

[0052] The edge cache module is used for adjusting the popular contents of the SBS local cache according to the local state and the states of the neighboring SBSs based on the MADRL framework.

[0053] An embodiment of the present invention also provides a network device, including a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement any of the above-mentioned edge caching methods based on multi-agent reinforcement learning and federated learning.

[0054] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute any one of the above-mentioned edge caching methods based on multi-agent reinforcement learning and federated learning.

[0055] It can be seen from the above technical solutions that the present invention has the following advantages:

[0056] The embodiment of the present invention provides an edge caching method and system based on multi-agent reinforcement learning and federated learning. The present invention uses FDL to predict content popularity, and jointly trains a shared global model from UEs under its coverage under the coordination of SBS, where UEs train their own models locally. A flexible local update algorithm is used in FDL to set specific weights for each UE, and a personalized model is trained based on the difference between the global model and the local model. After each SBS predicts the popularity of content within its coverage. The present invention proposes a multi-agent deep reinforcement learning collaborative caching MADRL framework, in which each SBS makes its own caching decision based on the status of local and neighboring SBSs to effectively utilize distributed edge caching. Experimental results show that the method proposed by the present invention is superior to other baseline caching methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0058] Figure 1 It is a flowchart of an edge caching method based on multi-agent reinforcement learning and federated learning provided in an embodiment;

[0059] Figure 2 A network schematic diagram provided in an embodiment;

[0060] Figure 3 (a) and (b) are schematic diagrams comparing the cost and cache hit rate of the method of the present invention and the other three algorithms in different cache sizes;

[0061] Figure 4 The loss trend of the training process of the method of the present invention;

[0062] Figure 5 (a), (b), and (c) are comparative schematic diagrams of the cost, cache hit rate, and reward of different cache sizes in each round of the method of the present invention in the test phase, respectively;

[0063] Figure 6 (a), (b), and (c) are comparative schematic diagrams of the cost, cache hit rate, and reward of the method of the present invention with different numbers of SBSs in different cache sizes;

[0064] Figure 7 (a), (b), and (c) are the weight distances of the encoder network, decoder network, and discriminator network layers in the adversarial encoder model of the method of the present invention, respectively. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] like Figure 1 As shown, an embodiment of the present invention proposes an edge caching method based on multi-agent reinforcement learning and federated learning, the method comprising:

[0067] S1: Construct a MADRL framework including states, actions, rewards, and strategies, and use federated deep learning to predict popular content within the coverage area of ​​SBS;

[0068] The method of using federated deep learning to predict popular content within the coverage of SBS includes the following steps:

[0069] Iteratively train the adversarial encoder model using federated deep learning;

[0070] The SBS sends the trained adversarial encoder model to all UEs within the coverage area, sets the active users in the UE, selects the neighboring users of each active user and constructs the matrix;

[0071] The number of contents of interest to each user is counted, and the UE selects Pn contents with the largest content popularity as the predicted contents of interest to the UE, where the number of contents of interest is called the content popularity of the content;

[0072] After all UEs within the coverage of the SBS upload their predicted interesting contents, the SBS compares the predicted interesting contents uploaded by all UEs and selects Pn most interesting contents as predicted popular contents;

[0073] S2: Based on the MADRL framework, the SBS adjusts the popular contents in its local cache according to the local state and the states of neighboring SBSs.

[0074] An edge caching method based on multi-agent reinforcement learning and federated learning is provided. The content popularity is predicted by using FDL. The shared global model is jointly trained from UEs under its coverage under the coordination of SBS, where UEs train their own models locally. A flexible local update algorithm is used in FDL to set specific weights for each UE, and a personalized model is trained based on the difference between the global model and the local model. After each SBS predicts the content popularity within its coverage area, the present invention proposes a multi-agent deep reinforcement learning collaborative caching MADRL framework, in which each SBS makes its own caching decision based on the status of local and neighboring SBSs to effectively utilize distributed edge caching. Experimental results show that the method proposed by the present invention outperforms other baseline caching methods.

[0075] like Figure 2As shown in the figure, the network includes a content server (CS), a group of SBSs and a certain number of UEs. The CS is connected to the core network via a backhaul link, and the UE can directly obtain all requested content from it. All SBSs are connected to the CS via a backhaul link of the next generation NG interface; each SBS is equipped with a cache device with a certain storage capacity, and the SBSs provide content services to UEs within its coverage through wireless links. Although the cache capacity of a single SBS is actually very limited, adjacent SBSs can communicate with each other through the Xn interface to share cached content, so that the SBS can be coordinated to make full use of the distributed edge cache. In this network, UEs can obtain the requested content through the local SBS (the SBS currently accessed by the UE) and adjacent SBSs (SBSs adjacent to the local SBS).

[0076] In order to solve the problem of distributed edge cache placement, a MADRL framework including state, action, reward and strategy is constructed, including:

[0077] State: The local state of SBSb at time slot t Defined as The state of the system is defined as in, is the local cache status of SBSb, p b It is the content predicted to be popular in the SBSb coverage area;

[0078] Action: In order to adapt to the dynamic changes in content popularity, each SBS needs to delete some unpopular content or add other popular content to actively adjust the location of the local cache. At time slot t, the action of SBSb is defined as The action of the system is defined as Among them, F p =|p b |Represents the number of predicted popular contents;

[0079] Bonus: In a collaborative caching system, UE requests can be satisfied by the CS, the local SBS, or neighboring SBSs, depending on the content placement. The three ways for UE to obtain content usually correspond to different costs.

[0080] When the UE obtains the requested content from the local SBS, the service cost of the local SBS b is expressed as Where α represents the cost of delivering a content from the local SBS b, represents the amount of content obtained by SBS b from its local cache during time slot t; when UE obtains the requested content from SBSs, the service cost of neighboring SBSs is expressed as Where β represents the cost of delivering a content from the neighboring SBSs of the local SBS b, represents the amount of content obtained by the local SBS b from its neighboring SBSs during time slot t. Since content sharing between distributed SBSs consumes backhaul resources, the cost of obtaining content from neighboring SBSs is much higher than the cost of obtaining content from the local SBS, that is, β>α, and the value of β depends on the distance between SBS b and its neighboring SBSs. When the UE obtains the requested content from the CS, the service cost of the CS is expressed as Where χ represents the cost of delivering the requested content from the CS to the UE accessing SBS b, represents the amount of content transmitted from CS to SBS b during time slot t. Due to the resource consumption of backhaul and core network, the service cost of CS is much higher than that of edge cache, i.e., χ>β. In summary, the content delivery cost during time slot t can be expressed as

[0081] When the SBS decides to adjust its local cache, it removes some less popular content and adds more popular content. (Assume that all added content is downloaded from CS) Cache replacement may impose additional backhaul requirements on the core network. Therefore, the total cost of content delivery that relies on edge caching should include not only the cost of transmitting content from the local SBS, neighboring SBSs, and CS to the UE, but also the cost of updating the cache location. During time slot t, the replaced content in SBS b can be expressed as Let η denote the cost of replacing a content in SBS b, then the cost of replacing the cache is When edge caching is used, the total cost for UE to obtain content is

[0082] When edge caching is not used, all UE requests must be processed by the CS, and the corresponding cost is By pushing content closer to the UE, edge caching can help reduce the cost of content delivery, so the cost of using edge caching is usually much lower than not using edge caching. The higher the cost savings, the more effective the edge caching is. The reward of SBS b can be defined as the cost savings of edge caching compared to not using edge caching, as follows:

[0083]

[0084] Among them, α, β, χ and η are predefined constants, and is the statistical data of the number of UEs in the coverage area of ​​SBS b that obtain content during the time interval [t, t+1). The reward cannot be calculated until the end of time slot t. Maximizing the reward also corresponds to minimizing the cost of edge cache to obtain content. The reward of SBS b in time slot t depends not only on its own cache capacity but also on the cache contents of neighboring SBSs. Therefore, the reward of the entire system in time slot t is

[0085] Strategy: In a distributed edge cache system, each SBS can be regarded as an intelligent agent that must decide its own cache content according to the system state. Let π = {π1,...,π b ,...,π B} represents a cache strategy, which maps state s to action a, ie, a = π(s). Since the behavior of the agent affects both the current reward and the long-term reward, all agents should cooperate to find the best strategy π* to maximize the long-term reward. Expected long-term discounted return where γ∈(0,1) is the discount factor. Therefore, the collaborative caching problem can be formulated as a multi-agent decision-making problem to maximize the expected long-term discounted return π * =argmin J(π).

[0086] Furthermore, due to the unique preferences of each UE, the requested content received by different SBSs may be significantly different, and each SBS should predict its own content popularity and decide which content should be stored in the local cache. The present invention uses federated deep learning to predict the popularity of content within the coverage of the SBS, including the following steps:

[0087] The adversarial encoder model is iteratively trained using FDL.

[0088] 1. Model download: SBS initializes its own global model ω in the first round of iterative training r , in each round of iterative training, SBS updates the global model and converts the global model ω r Delivered to all UEs under its coverage.

[0089] 2. Local training: Each UE under the coverage of the SBS will download the global model ω r Set as its own initial local model, and update the local model through training iterations; the updated local model will serve as feedback to the SBS.

[0090] Let D = {D1,...,D i ,...} represents the training data stored in all UEs, D i represents the local training dataset of UE i, whose size is d i =|D i |, d is the size of the entire data in all UEs within the coverage of the SBS. The goal of FL is to minimize the loss function F(ω), which is expressed as follows:

[0091]

[0092] Among them, F i (ω) is the local loss function of UE i using model ω f j (ω) is the prediction loss caused by the j-th data using model ω.

[0093] Each UE updates its local model through multiple iterations, and the formula is as follows:

[0094]

[0095] Among them, η is a fixed learning rate, is the local model of UE i in round r and Is UE i using the local model In the case of gradient.

[0096] Since the update of the local model needs to be determined by its own characteristics. In the local model update, if the global model is used directly, the characteristics of each UE will be eliminated and the time in the training process will be wasted. Therefore, in the local model update process, the present invention uses a flexible local update algorithm to set a specific weight for each UE, and trains a personalized local model for each UE based on the difference between the global model and the local model. Specifically, it includes:

[0097] First, define the weight distance formula, the formula is as follows:

[0098]

[0099] Among them, ω a and ω b Represent the weights of the two models, dis(ω a ,ω b ), the larger the difference between the two models;

[0100] Then, in the learning network composed of a multi-layer network L = {l1,l2,...}, in round r, the local model of each layer is calculated separately and the global model ω r The difference and introduction of α i As the elasticity parameter, the calculation formula is as follows:

[0101]

[0102] in, Is a local model The weight of the lth layer, is the global model ω rThe weight of the lth layer of |L| represents the number of layers of the deep learning network, which is used to average the weight distance function of each layer, α i The larger it is, the greater the gap between the global model and the local model.

[0103] Finally, according to the elastic update formula Update and train the local model. During the local model update process, increasing the proportion of the global model can effectively promote the influence of the global model on the personalized model.

[0104] 3. Upload model: After the local model is updated, UE will upload the updated local model Upload to the local SBS.

[0105] 4. Weight aggregation: All UEs within the coverage of the SBS upload their updated local models After that, the SBS calculates all the updated local models received The weighted average of r+1 , and its calculation formula is:

[0106]

[0107] Here, η is a fixed learning rate.

[0108] Through multiple rounds of iterative training, the final global model is obtained, and the adversarial encoder model completes the training.

[0109] The present invention facilitates the collaborative training of deep neural network models among all UEs by caching training data on UEs. FDL significantly reduces the privacy risk of US and greatly reduces the communication cost brought by centralized machine learning.

[0110] After the adversarial encoder model is trained, the SBS sends the trained adversarial encoder model to all UEs within the coverage area, sets the active users in the UE, selects the neighboring users of each active user and constructs a matrix, which specifically includes the following steps:

[0111] SBS sends the trained adversarial encoder model to all UEs within the coverage area and converts the rating matrix X of UE i to the content i As input to the trained adversarial encoder model, the reconstructed rating matrix is ​​output Each UE also extracts a personal information matrix, where the first dimension of the matrix is ​​the user ID and the second dimension is the user's personal information, denoted as H i . Calculate the number of non-zero content scores for each user in the UE, and mark the user with the largest number of non-zero content as an active user. UE i reconstructs the score matrix and its personal information matrix H i Combined into a combination matrix Hi, the similarity between each active user and other users is calculated.

[0112] Specifically, the similarity between each active user and other users is calculated based on the cosine similarity, and the calculation formula is:

[0113]

[0114] Among them, H i (a,:) and H i (b,:) are the vectors corresponding to the active users a and b in the combination matrix Hi, respectively. i (a,:)‖2 and ‖H i (b,:)‖2 are H i (a,:) and H i 2-norm of (b,:).

[0115] For each active user a, UE selects the users with the K greatest similarities as the K neighboring users of user a. The ratings of the K neighboring users also reflect the preferences of user a to a certain extent. Extract the rating matrix X i The vectors of K neighboring users of each active user in and construct the matrix in The first dimension is the IDs of all active users' neighbors, and The second dimension of is the ratings of the content given by neighboring users. In the , content with a non-zero score is considered to be of interest to the user.

[0116] The number of contents of interest to each user is counted, and UE i selects Pn contents with the largest content popularity as predicted contents of interest to UE i, where the number of contents of interest is called the content popularity of the content; after all UEs within the coverage of the SBS upload their predicted contents of interest, the SBS compares the predicted contents of interest uploaded by all UEs and selects Pn contents of greatest interest as predicted popular contents.

[0117] The cache capacity C of each SBS, that is, the maximum number of contents that each SBS can accommodate, is usually less than Pn. Each SBS uses the above content popularity prediction algorithm to predict the popular contents of the SBS.

[0118] Next, the invention proposes a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm for collaborative edge caching to determine where to cache predicted popular content to minimize caching costs.

[0119] The MADDPG algorithm is based on a multi-agent actor-critic framework, where the actor network is used for policy improvement and the critic network is used for policy evaluation. The MADDPG algorithm uses a deep neural network as a function approximator to effectively approximate and evaluate policies, thereby forming the corresponding actor network and critic network. Each agent has a local actor network and critic network. The actor network of agent b is used to approximate policy π b , where the approximate strategy is expressed as The actions output by the actor network are based on the policy and observed states. The critic network of agent b receives local observations and actions and estimates the local expected reward. All agents share a centralized global critic network, where all agents' local observations and actions are taken as input and estimate the reward of the global critic network for them. The goal is to maximize global and local rewards while improving and evaluating the strategy. The use of the target network can ensure the stability of the algorithm. The target network includes the target actor network and the target critic network, and its network structure is the same as the actor network and the critic network, respectively. Let θ b and φ b are the parameters of the actor network and critic network of agent b, θ′ b and φ′ b are the parameters of the target actor network and target critic network of agent b, respectively. are the parameters of the global critic network, are the parameters of the target global critic network. Due to the function approximation error, the critic function still has the problem of overestimation and underestimation of the strategy. The global critic network is replaced by the twin global critics network, that is, there are two global critic networks and two target global critic networks, and their network parameters are

[0120] Furthermore, the MADDPG training algorithm has the following specific steps:

[0121] First, randomly initialize the twin global critic network parameters The actor network parameters θ for each agent bb and critic network parameter φ b , respectively make the twin target global critic network parameters The target actor network parameters θ′ for each agent b b and the target critic network parameter φ′ b Initialize to θ b and φ b . Construct a playback buffer D with enough space to cache the transformation.

[0122] Then the algorithm executes round E. In each round, each SBS b obtains the content popularity pb within its coverage area through the content popularity prediction algorithm, and randomly selects C content from pb as the cache content of the SBS. Therefore, each SBS obtains the initial state The initial state of the entire system,

[0123] Then, the algorithm is iterated from time slot 1 to time slot T. The actor network of each SBS b inputs its initial state Output Action So as to get the action of the whole system Each SBS observes the state of the next time slot after executing the action output by its own actor network So as to get the status of the whole system Then, each SBS cooperates to cache content to respond to UE requests and obtain its own rewards. Then obtain the global reward R of the entire system 1 , so that local rewards Then the tuple Stored in the replay buffer D. When the number of tuples stored in D is less than M, each SBS b inputs its own state for the next time slot into its own actor network and starts the next iteration.

[0124] When the number of stored tuples is larger than M, CS first uniformly samples M arrays from D to form a minibatch. For simplicity, let Respectively represented as s, a, R, R L ,s′,a′. Represents the i-th tuple in the mini-batch. For tuple i, CS first converts s′ i Each local state are input into the target local actor network θ′ corresponding to SBS b b , and then output the action of each SBS b Then get the whole system action Then CS will a′ i and s′ i Input into the twin target global critic network and output two state value functions respectively After that, CS calculates the target value of the twin target global critic network, and the calculation formula is as follows:

[0125]

[0126] Then the loss function can be calculated as in In the action state, they are a i ,s i The twin global critic network is then passed through, using the corresponding To update its own parameters, that is, to minimize the loss function through gradient descent.

[0127] Then the local loss function of SBS b can be calculated as:

[0128] Then the policy gradient of SBS b can be calculated as:

[0129]

[0130] SBS b use Update the local actor network parameters and use the gradient ascent method to make maximize.

[0131] At the end of time slot t, each SBS updates the parameters θ′ of the target local actor network and the critic network b =τθ b +(1-τ)θ′ b ,φ′ b =τφ b +(1-τ)φ′ b .

[0132] In the next time slot, the iteration will continue. When the number of iterations reaches Ns, this round ends. Then each SBS will obtain the content popularity pb within its own coverage area in the current round through the content popularity prediction algorithm, and randomly select C content from pb as the initial cache content of the SBS in the current round, and the next round will begin. When the number of rounds reaches N, the algorithm will eventually terminate and the training phase ends.

[0133] The present invention proposes a MADDPG test algorithm, in which the critic network, target actor network and target critic network of each SBS in the training phase are omitted in the test phase, and the twin global critic network and twin target global critic network are also omitted. The test phase adopts the optimized parameter θ b * Optimal strategy to test performance.

[0134] The present invention provides an edge caching system based on multi-agent reinforcement learning and federated learning, the system comprising:

[0135] Popular content prediction module, which is used to build a MADRL framework including states, actions, rewards and strategies, and uses federated deep learning to predict popular content within the coverage area of ​​SBS;

[0136] The method of using federated deep learning to predict popular content within the coverage of SBS includes the following steps:

[0137] Iteratively train the adversarial encoder model using federated deep learning;

[0138] The SBS sends the trained adversarial encoder model to all UEs within the coverage area, sets the active users in the UE, selects the neighboring users of each active user and constructs the matrix;

[0139] The number of contents of interest to each user is counted, and the UE selects Pn contents with the largest content popularity as the predicted contents of interest to the UE, where the number of contents of interest is called the content popularity of the content;

[0140] After all UEs within the coverage of the SBS upload their predicted interesting contents, the SBS compares the predicted interesting contents uploaded by all UEs and selects Pn most interesting contents as predicted popular contents;

[0141] The edge cache module is used for adjusting the popular contents of the SBS local cache according to the local state and the states of the neighboring SBSs based on the MADRL framework.

[0142] The system is used to implement the above-mentioned edge caching method based on multi-agent reinforcement learning and federated learning. In order to avoid redundancy, it will not be repeated here.

[0143] In order to illustrate and verify the advantages of the method of the present invention, a simulation experiment was conducted on the caching method of the present invention, and the results are as follows.

[0144] from Figure 3It can be seen that the cost of all methods decreases with the increase of cache size, and the cache hit rate increases with the increase of cache size. This is because all SBSs have larger cache sizes to cache more content, so it is easier to obtain the content requested by UEs from the SBSs edge devices, which reduces the pressure on CS and thus the cache performance in the entire network is better. In addition, it can be seen that the Random method provides the worst cache performance (highest cost and lowest cache hit rate) because this method only randomly selects content without considering content popularity. In addition, the proposed and C-ε-greedy methods outperform the Random and Thompson Sampling methods because the Random and Thompson Sampling methods cannot predict cache content by learning, while the proposed and C-ε-greedy methods decide the cache content by observing the content of historical requests. In addition, the proposed is better than the C-ε-greedy method because the proposed method captures useful hidden features from the data to predict accurate popular content.

[0145] During the training process of the MADDPG algorithm, the number of SBSs is set to 2. Figure 4 As shown in the figure, the global loss decays rapidly until 400 slots and then gradually stabilizes; the local losses of the two SBSs decay rapidly until 750 slots and then gradually stabilize. Since the loss represents the error between the state value function and the actual reward, the state value function gradually approaches the actual reward as the training progresses. This shows that the learning method converges after training for 400 and 750 slots, respectively, and the well-trained global and local critic networks can be used to accurately estimate the value function.

[0146] During the testing of the MADDPG algorithm, the number of SBSs is set to 2, and the MADDPG algorithm will use the strategy learned in the training phase to test the performance. Figure 5 (a), (b), and (c) show the performance of the proposed method in the test phase in terms of cost, cache hit rate, and reward in different rounds. It can be seen that the performance of the strategy learned by the proposed method in the training phase tends to be stable in different test rounds.

[0147] Figure 6(a), (b), and (c) show how the number of multi-agents (i.e., SBSs) affects the proposed method. When the number of SBSs is 1, the worst performance is provided in this case. This is because the proposed method has become a single-agent decision method, which is no longer a collaborative cache. UEs can only obtain the requested content from the local SBS and CS. After that, as the number of SBSs increases, the proposed method basically gets better and better in terms of cost, cache hit rate, and reward performance. This is because more SBSs cache more content, and it also shows that the proposed algorithm can coordinate edge caches a lot.

[0148] Figure 7 (a), (b), and (c) respectively show the weight distances of the encoder network, decoder network, and discriminator network layers in the adversarial encoder model of the present invention. The curves represent the average weight distances of each layer in different networks of the adversarial encoder model. Figure 7 As can be seen from the figure, as the number of gradient descent processes increases, the distance between the local model and the global model tends to decrease. This is because during FDL training, the elastic update percentage of the global model is measured based on the distance between the global model and the local model, thereby effectively promoting the influence of the global model on the local model. After FDL training reaches a certain number of communication rounds, the weight distances of each layer in the three networks converge.

[0149] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

[0150] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0151] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. An edge caching method based on multi-agent reinforcement learning and federated learning, characterized in that: include: Construct a MADRL framework including states, actions, rewards, and strategies, and use federated deep learning to predict popular content within the coverage area of ​​SBS; The construction includes a MADRL framework of states, actions, rewards and strategies, specifically including: State: The local state of SBS b at time slot t Defined as The state of the system is defined as in, is the local cache status of SBSb, p b It is the content predicted to be popular in the SBSb coverage area; Action: In time slot t, the action of SBS b is defined as The action of the system is defined as Among them, F p =|p b |Represents the number of predicted popular contents; Reward: At time slot t, the reward of SBS b is defined as The system rewards are Among them, α, β, χ and η are predefined constants, and It is the statistical data of the number of contents acquired by UEs within the coverage of SBSb during the time interval [t, t+1); Strategy: Use π={π1,...,π b ,...,π B } represents the caching strategy to maximize the expected long-term discounted return π * =argminJ(π), where γ∈(0,1) is the discount factor; The method of using federated deep learning to predict popular content within the coverage of SBS includes the following steps: Iteratively train the adversarial encoder model using federated deep learning; The SBS sends the trained adversarial encoder model to all UEs within the coverage area, sets the active users in the UE, selects the neighboring users of each active user and constructs a matrix, including: SBS sends the trained adversarial encoder model to all UEs within the coverage area and converts the rating matrix X of UE i to the content i As input to the trained adversarial encoder model, the reconstructed rating matrix is ​​output Calculate the number of contents with non-zero scores for each user in the UE, and mark a set number of users with the maximum number of non-zero contents as active users; UE will reconstruct the scoring matrix and its personal information matrix H i Combined into a combination matrix Hi, calculate the similarity between each active user and other users; For each active user a, UE selects the users with the K largest similarities as the K neighboring users of user a and extracts the rating matrix X i The vectors of K neighboring users of each active user in and construct the matrix The method to calculate the similarity between each active user and other users is: The similarity between each active user and other users is calculated based on the cosine similarity, and the calculation formula is: Among them, H i (a,:) and H i (b,:) are the vectors corresponding to the active users a and b in the combination matrix Hi, respectively. i (a,:)‖2 and ‖H i (b,:)‖2 are H i (a,:) and H i 2-norm of (b,:); The number of contents of interest to each user is counted, and the UE selects Pn contents with the largest content popularity as the predicted contents of interest to the UE, where the number of contents of interest is called the content popularity of the content; After all UEs within the coverage of the SBS upload their predicted interesting contents, the SBS compares the predicted interesting contents uploaded by all UEs and selects Pn most interesting contents as predicted popular contents; Based on the MADRL framework, the SBS adjusts the popular contents in its local cache according to the local state and the states of neighboring SBSs; Among them, iterative training of the adversarial encoder model using federated deep learning includes the following steps: SBS initializes its global model ω in the first round of iterative training r , in each round of iterative training, SBS updates the global model and converts the global model ω r Delivered to all UEs within its coverage area; Each UE under the coverage of the SBS will download the global model ω r Set it as its own initial local model and update the local model through training iterations; The local model is updated, and UEi will update the local model Upload to local SBS; All UEs within the SBS coverage area upload their updated local models After that, the SBS calculates all the updated local models received The weighted average of r+1 , and its calculation formula is: Where η is a fixed learning rate; d is the size of the entire data in all UEs within the coverage area of ​​the SBS; d i =|D i |, where D i represents the local training dataset of UEi; Through multiple rounds of iterative training, the final global model is obtained, and the adversarial encoder model completes the training.

2. The edge caching method based on multi-agent reinforcement learning and federated learning according to claim 1 is characterized in that: During the local model update process, a flexible local update algorithm is used to set specific weights for each UE, and a personalized local model is trained for each UE based on the difference between the global model and the local model.

3. The edge caching method based on multi-agent reinforcement learning and federated learning according to claim 2 is characterized in that: During the local model update process, a flexible local update algorithm is used to set a specific weight for each UE. Based on the difference between the global model and the local model, a personalized local model is trained for each UE. Specifically, the following steps are performed: First, define the weight distance formula, the formula is as follows: Among them, ω a and ω b Represent the weights of the two models, dis(ω a ,ω b ), the larger the difference between the two models; Then calculate each layer of local model separately and the global model ω r The difference and introduction of α i As the elasticity parameter, the calculation formula is as follows: in, Is a local model The weight of the lth layer, is the global model ω r The weight of the lth layer of |L| represents the number of layers of the deep learning network, which is used to average the weight distance function of each layer, α i The larger it is, the greater the gap between the global model and the local model. Finally, according to the elastic update formula Update the local model.

4. An edge caching system based on multi-agent reinforcement learning and federated learning, characterized in that: The system is used to implement the edge caching method based on multi-agent reinforcement learning and federated learning as described in any one of claims 1 to 3, including: Popular content prediction module, which is used to build a MADRL framework including states, actions, rewards and strategies, and uses federated deep learning to predict popular content within the coverage area of ​​SBS; The method of using federated deep learning to predict popular content within the coverage of SBS includes the following steps: Iteratively train the adversarial encoder model using federated deep learning; The SBS sends the trained adversarial encoder model to all UEs within the coverage area, sets the active users in the UE, selects the neighboring users of each active user and constructs the matrix; The number of contents of interest to each user is counted, and the UE selects Pn contents with the largest content popularity as the predicted contents of interest to the UE, where the number of contents of interest is called the content popularity of the content; After all UEs within the coverage of the SBS upload their predicted interesting contents, the SBS compares the predicted interesting contents uploaded by all UEs and selects Pn most interesting contents as predicted popular contents; The edge cache module is used for adjusting the popular contents of the SBS local cache according to the local state and the states of the neighboring SBSs based on the MADRL framework.

5. A network device, characterized in that: It includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory, so as to implement the edge caching method based on multi-agent reinforcement learning and federated learning as described in any one of claims 1 to 3.

6. A computer storage medium, characterized in that: The computer storage medium stores a computer software product, which includes a number of instructions for enabling a computer device to execute the edge caching method based on multi-agent reinforcement learning and federated learning as described in any one of claims 1 to 3.

Images