Multi-agent joint edge caching method based on federated learning architecture

Abstract

The invention belongs to the technical field of mobile communication, and particularly relates to a multi-agent joint edge caching method based on a federated learning architecture, which is suitable for a macro-micro heterogeneous network environment. According to the method, the content popularity is predicted, and a cache decision is optimized by adopting a deep reinforcement learning method; according to the method, distributed prediction is carried out on content popularity through a federated learning architecture and a denoising auto-encoder long-short term memory network, and dynamic optimization of content cache is realized. And optimizing a caching strategy and resource allocation by combining a game model between the micro base station and the macro base station so as to maximize the effectiveness of the infrastructure provider. According to the method, a multi-agent deep reinforcement learning algorithm without information sharing is introduced to optimize the system efficiency. The system reduces transmission delay and improves network performance and user experience by calculating a dynamic adjustment strategy of terminal user content popularity and combining a communication resource allocation strategy. The method effectively solves the problems of high delay, low cache resource utilization rate and the like in the existing system, and is suitable for a complex network environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications and mobile edge computing, and specifically relates to a multi-agent joint edge caching method based on a federated learning architecture. Background Art

[0002] With the rapid development of mobile Internet, users' demand for low-latency and high-throughput network services continues to rise. The traditional centralized caching mechanism can no longer meet these demands due to the long data transmission distance and time. In the macro-micro network (HetNet) environment, traditional caching systems face problems such as high latency and low network resource utilization, which seriously affects the user experience. For this reason, the mobile edge computing (MEC) caching strategy has gradually become a solution. Compared with centralized caching, edge caching stores content on network nodes close to users, significantly reducing data transmission distance and latency, thereby improving the real-time nature of content delivery and resource utilization efficiency.

[0003] Although edge caching strategies have shown advantages in coping with dynamic network loads and improving user experience, traditional edge caching strategies still face several challenges in scenarios with high dynamics, multiple users, and multiple content requests. First, the spatiotemporal dynamic changes in content popularity can easily lead to delayed updates of cached content, affecting the timeliness and relevance of cached data; second, with the continuous growth of data traffic and the intensification of the tidal effect of network communications, the congestion of communication links further reduces the effectiveness of caching strategies. In addition, although existing caching strategies attempt to optimize caching by predicting content popularity, in distributed collaborative scenarios involving multiple content service providers (CSPs), traditional centralized training methods are difficult to effectively protect user data privacy.

[0004] To solve the above problems, the present invention proposes a multi-agent joint edge caching method based on a federated learning architecture. This method introduces a federated learning architecture and realizes accurate prediction of content popularity through distributed training, thereby improving prediction accuracy while protecting user data privacy. In addition, the present invention adopts a multi-agent deep reinforcement learning algorithm to coordinate multiple edge nodes to optimize caching strategies for dynamic content requests and network status changes, improve network resource utilization efficiency, reduce communication link congestion, and achieve efficient edge caching and resource allocation. This method effectively solves the bottleneck problem in the prior art and significantly improves the user experience and network performance in macro and micro network environments. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a multi-agent joint edge caching method based on a federated learning architecture, which solves the problems raised in the above background technology.

[0007] (II) Technical solution

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A multi-agent joint edge caching method based on a federated learning architecture includes a macro-micro heterogeneous network model in which a macro base station, a micro base station, and multiple content service providers participate. Federated learning and multi-agent deep reinforcement learning algorithms are used to optimize content caching and communication resource allocation. Specifically, the following steps are included:

[0010] Step 1: Build a macro-micro heterogeneous network model. Macro base stations are responsible for large-scale coverage, while micro base stations are deployed in densely populated areas to provide efficient data caching and transmission services. Multiple content service providers provide diverse content for different base stations. The caching process of each base station is dynamically adjusted according to user requests. The specific mathematical modeling is as follows:

[0011] As the basic equipment provider, mobile network operators deploy mobile edge servers or devices on macro base stations and micro base stations to effectively cache data and content at the edge of the network. , has a large cache capacity and can cache all content by default. The micro base station collection is , whose devices have limited cache space.

[0012] As the final consumer of the requested content, the set of end users can be represented as , Indicates End users in micro base stations Within the service range of the base station, wireless link communication is carried out with the base station; the server cache capacity of the macro base station and the micro base station is and .

[0013] Due to the complexity and high demand of heterogeneous network content, the content provided by each content service provider changes dynamically with the time slot update. Indicates that the content The size of express.

[0014] Furthermore, each end user sends a content request in each time slot, and the base station makes a cache decision based on the content characteristics, user request characteristics, and its own revenue. To indicate content Whether to cache in micro base station superior: Represents cache, Indicates not cached.

[0015] Furthermore, to meet the cache capacity limit, the total cache content of the macro and micro base stations must meet the following requirements:

[0016]

[0017]

[0018] Furthermore, user request patterns and preferences directly affect content popularity and demand, guiding content service providers and infrastructure providers to adjust cache placement strategies.

[0019] Step 2: Content popularity prediction and server optimization strategy under the federated learning architecture. Using the federated learning architecture, the Federated Stacked Denoising Auto Encoder (F-SDAE) is combined with the Long Short-Term Memory (LSTM) network to perform distributed training on the user's historical request data to predict the popularity of the content. The micro base station acts as a client and uses local user request data for training without sharing the original data, thereby protecting user privacy.

[0020] Furthermore, each micro base station uses the historical content request dataset of users in its coverage area to train a stacked denoising autoencoder to extract low-dimensional feature representations of content requests. The denoising autoencoder optimizes the feature representation layer by layer through a multi-layer autoencoder structure to reduce noise interference in the data.

[0021] Furthermore, the hidden feature vector generated by the denoising autoencoder is input into the long short-term memory network. LSTM captures the time series dependency through its input gate, forget gate and output gate mechanism. The specific calculation formula is:

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] in, is the unit state, is the hidden layer output variable, , are the weight matrix and bias vector respectively. is a nonlinear activation function, is the hyperbolic tangent activation function.

[0028] Furthermore, the features of each dimension are integrated through a fully connected network, and nonlinear transformation is performed through an activation function to generate a predicted value of content popularity at the next time point. .in, represents the content request pattern features extracted for each time slot, Denotes the activation function. The model updates all weight matrices and bias terms through the Adam optimization algorithm, aiming to narrow the gap between the model prediction value and the actual observation value. Among them, the loss function between the predicted value and the true value is After the micro base station local model is trained, the weight matrix , Deviation term Parameters such as are shared to the parameter server, and distributed content popularity prediction is performed through the federated learning architecture.

[0029] Furthermore, the federated learning architecture completes the global prediction and cache optimization of content popularity through the following steps: Each micro base station uses its local user request dataset Perform model training to extract correlation and time domain features. After model training is completed, the micro base station will set the parameters Upload to the global server. The global server aggregates the model parameters of all micro base stations , update the global model parameters by weighted average method:

[0030]

[0031] Global model parameters It is sent to each micro base station, which updates the local model and conducts the next round of local training until the global model converges. The optimization goal of federated learning is to minimize the global loss function:

[0032]

[0033] in, represents the size of the historical request training dataset of the micro-cell content service provider, is the loss function.

[0034] Furthermore, the content popularity in the system can be predicted in a distributed manner. Specifically, the local controller obtains The historical request information of each local content service provider is then input into the training model of the micro base station, and the encoder outputs yes Local popularity prediction at a certain moment. On the server side of the macro base station, historical request information of global content service providers is input into the trained global prediction model, and its output It's time. . Prediction of global popularity. In addition, in order to protect the privacy of the terminal, once the information is requested Once it is input into the global prediction model, it will be erased immediately.

[0035] Step 3: Step 3: Optimizing the transmission rate by allocating bandwidth and power resources between the macro base station and the micro base station. In a macro-micro heterogeneous network, in order to reduce content transmission delay and increase data transmission rate, the present invention optimizes the communication path between the macro base station and the micro base station by allocating bandwidth and power resources, especially in the scenario where the content cached by the micro base station is preferentially transmitted.

[0036] Furthermore, when the micro base station has cached the content requested by the terminal user, the content is directly transmitted to the terminal user through the micro base station. The content transmission delay at this time can be expressed as:

[0037]

[0038] in, represents the cache decision variable, and the data transmission rate from the micro base station to the end user can be expressed as:

[0039]

[0040] in, is the bandwidth, is the micro base station transmit power, represents the channel gain, Represents the power spectral density of Gaussian white noise.

[0041] If the edge cache servers on the micro base station do not cache the content requested by the end user, the content will be delivered to the end user from the macro base station. The content transmission delay of this process is:

[0042]

[0043] Among them, the transmission rate from the macro base station to the terminal user can be expressed as:

[0044]

[0045] in,

[0046]

[0047]

[0048] Therefore, the delivery delay of the content requested by the end user can be specifically expressed as:

[0049] .

[0050] Furthermore, the service provider utility consists of two parts. The first part is the gain obtained by transmitting the content cached by the macro or micro base station to the end user, which involves the content delivery delay limit, which can be specifically expressed as:

[0051]

[0052] in, represents the gain per unit of satisfaction, indicates the total number of end users who requested the content, Display content The maximum delay for a request.

[0053] The communication resources purchased from the infrastructure provider require a certain cost, which can be expressed as:

[0054]

[0055] in, is the pricing strategy of the infrastructure provider for the transmission power of macro base stations. Pricing strategy for micro base station transmission power.

[0056] The utility of the service provider is the difference between its revenue and cost, which is related to the content delivery delay and can be expressed as:

[0057]

[0058] The utility of the infrastructure provider is also the difference between its revenue and cost, involving the utility of macro base stations and micro base stations, which can be specifically expressed as:

[0059]

[0060] in, represents the unit transmission power cost of the macro base station, Represents the unit transmission power cost of micro base stations.

[0061] Furthermore, each participant in the Stackelberg game has its own interests, and the goal is to find a Stackelberg Equilibrium (SE). In this equilibrium, each master obtains maximum utility given the best response of the slave. At this point, no master or slave has any motivation to unilaterally change its strategy.

[0062] The present invention aims to explore the edge caching problem in the macro-micro network environment by optimizing the caching decision and resource allocation strategy, and improve the network performance and service quality. Therefore, an optimization problem model for maximizing the utility of the infrastructure provider under the game theory is established, which can be specifically expressed as:

[0063]

[0064] Among them, C1 in the optimization problem represents maximizing the The effect , which reflects the maximum benefit of the service provider when transmitting cached content on the base station of its choice, and is coordinated with the goal of maximizing the utility of the infrastructure provider. C2 ensures that each micro base station The total cache content popularity ratio is at least , ensuring that high-popularity content is cached to improve cache hit probability. C3 represents the cache decision variable The value range of makes the cache decision flexible. C4 represents the pricing strategy and The non-negativity of ensures the rationality of the price. C5 represents the cache capacity constraint of each micro base station, which ensures effective cache decisions under limited storage resources. C6 represents the content transmission delay constraint, which guarantees the user experience, that is, the acquisition of content does not exceed the maximum tolerable delay. These interdependent variables and constraints together constitute a complex decision-making problem. To effectively solve this problem, it is necessary to adopt a highly flexible optimization strategy that can adapt to environmental changes.

[0065] Step 4: Aiming at the optimization problem of maximizing the utility of infrastructure providers, a multi-agent deep reinforcement learning content delivery strategy is proposed to manage and optimize complex heterogeneous network systems. Content popularity analysis provides end users with a foresight of future content demand. This invention takes the utility of infrastructure providers as the final reward value and proposes a multi-agent deep reinforcement learning algorithm without information sharing to further improve the overall network performance.

[0066] Furthermore, the optimization problem proposed in step 3 can be modeled as a partially observable Markov decision process, which is a multi-agent extension of MDP. The observable state space of an agent is , the state transition probability function is In a partially observable Markov decision process, the state space is partially known and the decision is based on the global observation space The global observation space can be defined as ,in , , . Representing an Agent in the past The action history of each time slot. Representation and Agents The relevant content popularity value. Representing an Agent The cache content state. The action space is defined as , which includes the cache strategy of the content service provider and the communication resource price of the infrastructure provider, wherein the pricing strategy adopted by the present invention is differentiated pricing. The reward space is defined as ,in , . All agents at time After taking an action, the reward is calculated according to the utility function, and then the next round of the game begins. At the beginning, the agent The actions are the cache strategy and the pricing strategy. After all agents take actions, the observations are transformed into , the entire observation conversion process is a random process. The goal of multi-agent deep reinforcement learning is to optimize each agent Strategy . Agent The strategy optimization problem can be expressed as:

[0067]

[0068] By formalizing the collaborative caching under the game-theory concept into a multi-agent partially observable Markov decision process, the edge caching strategy of macro-micro networks can be optimized through multi-agent deep reinforcement learning methods in a non-cooperative environment.

[0069] Furthermore, the multi-agent deep reinforcement learning network considers multiple agents, each of which is equipped with a deep reinforcement learning controller module for formulating caching strategies. The actor network directly generates action decisions based on the current observation through a series of fully connected neural network layers. The critic network is responsible for converting observations into feature vectors and extracting relevant features from its memory. Combining multi-dimensional features, it can make a final estimate of the current observation and update the memory bank through write operations. The replay buffer is responsible for storing recent game records for updating the actor and critic networks. After the update is completed, the buffer content will be cleared to store new records.

[0070] (III) Beneficial effects

[0071] Compared with the prior art, the present invention provides a multi-agent joint edge caching method based on a federated learning architecture, which solves the problems of dynamic changes in content information and tidal effects of communication load in edge caches under heterogeneous networks, and has the following beneficial effects:

[0072] The present invention studies an efficient hybrid collaborative caching strategy to cope with the user's low latency and high data throughput requirements that are difficult to meet with traditional caching mechanisms under the rapid development of mobile Internet; constructs a macro-micro network model supported by multiple service providers, introduces a caching model for content placement and delivery, and introduces a federated learning architecture to improve the prediction accuracy of content popularity; uses a master-slave game model to propose a content delivery strategy for a multi-agent deep reinforcement learning algorithm, verifies its effectiveness in improving network resource utilization efficiency and reducing cache latency, and demonstrates its potential value and practicality in actual networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A flowchart of a multi-agent joint edge caching method based on a federated learning architecture provided in an embodiment of the present invention;

[0074] Figure 2 A schematic diagram of the structure of a mobile edge cache system for establishing a macro-micro network according to an embodiment of the present invention;

[0075] Figure 3 A schematic diagram of a content placement structure under a federated learning architecture provided in an embodiment of the present invention;

[0076] Figure 4 A schematic diagram of the Stan Kohlberg game framework structure provided for an embodiment of the present invention.

[0077] Figure 5 A schematic diagram of the multi-agent deep reinforcement learning network architecture provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] Example

[0080] like Figure 1-5 As shown, a multi-agent joint edge caching method based on a federated learning architecture proposed in one embodiment of the present invention includes the following steps:

[0081] Step 1: Build a macro-micro heterogeneous network, where macro and micro base stations deploy mobile edge servers to achieve effective caching of content data at the edge of the network. Macro base stations are responsible for large-scale coverage, while micro base stations provide efficient data transmission based on user requests and caching strategies. Multiple content service providers provide diversified content services for different base stations.

[0082] Step 2: Adopting the federated learning architecture, the content placement strategy under the federated learning architecture is formulated through the federated stacked denoising autoencoder (F-SDAE), and the user's historical request data is distributedly trained to predict the content popularity. Based on the prediction results, the content caching strategy of macro base stations and micro base stations is optimized so that the cached content can more effectively meet user needs.

[0083] Step 3: Optimize the transmission rate and reduce the content transmission delay by allocating bandwidth and power resources between the macro base station and the micro base station. If the micro base station has cached the content requested by the user, it will be transmitted directly through the micro base station. If it is not cached, it will be transmitted through the macro base station. By establishing a utility function, the cache revenue and transmission cost of each base station and service provider are described. The utility function is based on a game model, with the service provider as the slave and the infrastructure provider as the master. The two parties establish a game relationship based on resource purchase and content transmission. In this game model, the optimal utility of the service provider and the infrastructure provider is achieved through the Stackelberg Equilibrium (SE). The cache decision and the allocation of communication resources are dynamically adjusted according to the game results to maximize the overall network utility and meet the constraints of content transmission delay, cache capacity and a certain hit rate.

[0084] Step 4: Based on the multi-agent deep reinforcement learning algorithm, the partially observable Markov decision process is used to optimize the cache decision and resource allocation of each agent (including macro base stations and micro base stations). Each agent learns and executes the optimal cache and transmission strategy by observing user request patterns, content popularity and network status, thereby improving the utilization of network resources and the overall user experience.

[0085] like Figure 2 As shown in FIG. 1 , a schematic diagram of a structure of a mobile edge cache system in a macro-micro network proposed in an embodiment of the present invention includes the following contents: a mobile network operator, as an infrastructure provider, deploys mobile edge servers or devices in macro base stations and micro base stations so that data and content are effectively cached at the edge of the network. The macro base station is represented by , has a large cache capacity and can cache all content by default. The micro base station collection is , whose devices have limited cache space.

[0086] As the final consumer of the requested content, the set of end users can be represented as , Indicates End users in micro base stations Within the service range of the base station, wireless link communication is carried out with the base station; the server cache capacity of the macro base station and the micro base station is and .

[0087] Due to the complexity and high demand of heterogeneous network content, the content provided by each content service provider changes dynamically with the time slot update. Indicates that the content The size of Each terminal user sends a content request in each time slot, and the base station makes a cache decision based on the content characteristics, user request characteristics, and its own benefits. Caching decision variable To indicate content Whether to cache in micro base station superior: Represents cache, Indicates that no cache is available. To meet the cache capacity limit, the total cache content of the macro and micro base stations must meet the following requirements:

[0088]

[0089]

[0090] User request patterns and preferences directly affect content popularity and demand, guiding content service providers and infrastructure providers to adjust cache placement strategies.

[0091] like Figure 3 As shown, the content placement strategy under the federated learning architecture proposed in the embodiment of the present invention includes the following processes:

[0092] Step 1: At each micro base station, the denoising autoencoder model is trained using the historical content request data of local users to extract the low-dimensional feature representation of content requests. The denoising autoencoder extracts and compresses data features layer by layer through a multi-layer autoencoder structure. The output of each layer is used as the input of the next layer, and the feature representation is gradually optimized to reduce noise interference in the data.

[0093] Step 2: Input the hidden feature vector generated by the denoising autoencoder into the long short-term memory network to capture the time series dependency of user content requests. LSTM processes the short-term and long-term dependency characteristics of content requests through the input gate, forget gate, and output gate mechanism to enhance the ability to predict future content request trends.

[0094] Step 3: After the micro base station completes the training of the denoising autoencoder and LSTM model locally, it uploads the trained model parameters (including weights and biases) to the global server. At the global server, the model parameters uploaded by multiple micro base stations are weighted averaged and aggregated to generate a global prediction model. The original user data is not shared during this aggregation process, thereby ensuring the protection of user privacy. The aggregated global model adapts to the differences in user request patterns in each micro base station area.

[0095] Step 4: The global server sends the aggregated global model parameters to each micro base station, and the micro base station further updates the local model based on the new content request data to improve the prediction accuracy. By regularly updating the global model and local model, the system can dynamically predict the popularity of content, and then adjust the cache strategy according to the prediction results to optimize the allocation of cache resources.

[0096] Step 5: Based on the content popularity prediction results, each micro base station dynamically adjusts the placement and update strategy of cached content, giving priority to caching content with higher popularity, thereby improving the cache hit rate, reducing content transmission delay, and improving the utilization efficiency of network resources.

[0097] like Figure 4 As shown in FIG. 1 , the schematic diagram of the framework structure of the Stan Kohlberg game proposed in the embodiment of the present invention includes the following processes:

[0098] Step 1: Calculate content transmission delay, including:

[0099] If the micro base station has cached the content requested by the end user, the content will be transmitted to the end user through the cellular communication link of the micro base station. The transmission delay of the content requested by the end user It can be expressed as:

[0100]

[0101] in, represents the cache decision variable, and the data transmission rate from the micro base station to the terminal user is expressed as:

[0102]

[0103] in, is the bandwidth, is the micro base station transmit power, represents the channel gain, represents the power spectral density of Gaussian white noise;

[0104] If the micro base station does not cache the content requested by the terminal user, it is transmitted through the macro base station, and the transmission delay is:

[0105]

[0106] Among them, the transmission rate from the macro base station to the terminal user can be expressed as:

[0107]

[0108] in,

[0109]

[0110]

[0111] Finally, the total transmission delay of the end-user request content can be expressed as:

[0112]

[0113] Step 2: Establish a utility function model for the macro base station and the micro base station. The utility function is determined by the cache hit rate, content transmission delay and power cost, and specifically includes:

[0114] The utility of the service provider includes the benefits of transmitting macro or micro base station cache content to end users and the gain associated with content transmission delay, which is specifically expressed as:

[0115]

[0116] in, represents the unit satisfaction gain, Indicates the total number of end users who requested the content, Indicates the maximum tolerable delay of content request;

[0117] The cost of purchasing communication resources from the infrastructure provider is expressed as:

[0118]

[0119] in, is the macro base station transmission power pricing strategy, Micro base station transmission power pricing strategy;

[0120] The utility of a service provider is the difference between its revenue and cost, associated with the content delivery delay:

[0121]

[0122] The utility of the infrastructure provider is the difference between revenue and cost:

[0123]

[0124] in, represents the unit transmission power cost of the macro base station, Represents the unit transmission power cost of micro base stations.

[0125] Step 3: The game model is used to describe the game relationship between the service provider and the infrastructure provider, with the service provider as the slave and the infrastructure provider as the master. The infrastructure provider sets the service pricing strategy and resource allocation mechanism based on its bandwidth and power resources.

[0126] Step 4: The optimal strategies of infrastructure providers and service providers are solved through Stackelberg Equilibrium (SE). Infrastructure providers control the resource usage of micro and macro base stations and optimize transmission rates by setting resource pricing strategies. Service providers adjust their caching strategies and content transmission choices under given pricing and resource allocation schemes to maximize utility.

[0127] Step 5: Dynamically adjust the cache decision and communication resource allocation strategy based on the game results to optimize the overall network utility while meeting the system constraints of content transmission delay, cache capacity and hit rate; the system gives priority to using micro base stations for cache transmission to reduce delay; if the micro base station does not cache the requested content, it is transmitted through the macro base station to ensure timely content transmission and efficient use of network resources.

[0128] like Figure 5 As shown in FIG. 1 , a schematic diagram of a multi-agent deep reinforcement learning network architecture proposed in an embodiment of the present invention includes the following steps:

[0129] Step 1: Define agents for macro and micro base stations respectively. The agents are modeled using a partially observable Markov decision process (POMDP) ​​to process observed user requests, content popularity, cache status, network bandwidth, etc. The elements in POMDP are defined as follows:

[0130] The observable state space of an agent is , the state transition probability function is ; In POMDP, the state space is partially known and decisions are based on the global observation space The global observation space can be defined as ,in , , . Representing an Agent in the past Action history of each time slot; Representation and Agents Related content popularity value; Representing an Agent The cache content state; the action space is defined as ,in , Indicates the cache strategy of the content service provider; , represents the price of communication resources of the infrastructure provider, where the pricing strategy adopted by the algorithm is differentiated pricing; the reward space is defined as ,in , . All agents at time After taking an action, the reward is calculated based on the utility function, and then the next round of the game begins.

[0131] Step 2: Each agent independently adjusts its cache strategy based on user request patterns, content popularity, and current network bandwidth usage, including the following:

[0132] Step 2.1: Based on the predicted content popularity value, select and cache high-popularity content first;

[0133] Step 2.2: If the popularity of the content decreases or the cache capacity reaches the upper limit, remove the low-popularity content in time to release cache space;

[0134] Step 2.3: Dynamically adjust resource allocation for content transmission, including bandwidth and power scheduling, based on network congestion.

[0135] Step 3: Optimize the strategy of each agent through the Multi-Agent Deep Reinforcement Learning (MADRL) algorithm without information sharing, which includes the following process:

[0136] Step 3.1: Each agent performs local learning independently without sharing the local information it observes; the agent updates its strategy through the Actor-Critic Architecture; the ActorNetwork updates the strategy based on the agent’s current observations. , choose the best action , including cache decisions and resource allocation strategies; the Critic Network estimates the value function of the current strategy based on observations and actions, and stores historical experience through the Replay Buffer for strategy optimization.

[0137] Step 3.2: At each time slot, the agent dynamically adjusts the cache and transmission strategies based on its local observations of user requests, content popularity, and cache hit rate.

[0138] Step 3.3: The agent receives rewards from the environment after performing an action , and adjust the strategy for the next time slot by accumulating the reward value. The goal of reinforcement learning is to maximize the long-term cumulative reward, where is the discount factor.

[0139] Step 4: The strategies of each intelligent agent are federally coordinated and scheduled through a global server or a central controller to ensure balanced resource utilization between macro base stations and micro base stations. The central controller collects the model parameters or observation results uploaded by each micro base station, evaluates the overall network performance, and calculates the global optimal caching strategy. Using the federated learning architecture, the central controller periodically sends down the aggregated global model parameters, and each micro base station adjusts its local caching strategy based on these parameters, dynamically optimizing the caching and transmission efficiency of the entire network to ensure global optimization.

[0140] Step 5: Based on the strategic interaction between intelligent agents, the transmission path selection between the micro base station and the macro base station is dynamically adjusted to reduce cross-layer interference and optimize the content transmission path; the micro base station gives priority to transmitting cached content. If the micro base station does not cache the content requested by the user, it is transmitted through the macro base station with a smaller load, reducing network congestion and delay and optimizing the transmission path.

[0141] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-agent joint edge caching method based on a federated learning architecture, involving a macro base station, a micro base station and a macro-micro heterogeneous network model constructed by multiple content service providers, characterized in that: The method comprises the following steps: Step 1: Build a macro-micro heterogeneous network, where macro and micro base stations deploy mobile edge servers to achieve effective caching of content data at the edge of the network; macro base stations are responsible for large-scale coverage, while micro base stations provide efficient data transmission based on user requests and caching strategies; multiple content service providers provide diversified content services for different base stations; Step 2: Adopt a federated learning architecture and formulate a content placement strategy under the federated learning architecture through a federated stacked denoising autoencoder. Perform distributed training on the user's historical request data to predict content popularity. Based on the prediction results, optimize the content caching strategy of macro and micro base stations so that the cached content can more effectively meet user needs. Step 3: Optimize the transmission rate and reduce the content transmission delay by allocating bandwidth and power resources between the macro base station and the micro base station. If the micro base station has cached the content requested by the user, it will be directly transmitted through the micro base station. If it is not cached, it will be transmitted through the macro base station. By establishing a utility function, the cache revenue and transmission cost of each base station and service provider are described. The utility function is based on a game model. The service provider is the slave and the infrastructure provider is the master. The two parties establish a game relationship based on resource purchase and content transmission. In this game model, the optimal utility of the service provider and the infrastructure provider is achieved through the Stan-Kohlberg equilibrium. The cache decision and the allocation of communication resources are dynamically adjusted according to the game results to maximize the overall network utility and meet the constraints of content transmission delay, cache capacity and a certain hit rate. Step 4: Based on the multi-agent deep reinforcement learning algorithm, the cache decision and resource allocation of each agent are optimized using the partially observable Markov decision process. Each agent learns and executes the optimal cache and transmission strategy by observing the user request pattern, content popularity and network status, thereby improving the utilization of network resources and the overall user experience.

2. According to the multi-agent joint edge caching method based on the federated learning architecture of claim 1, it is characterized in that: The content popularity prediction in the federated learning architecture is implemented by a model combining a federated stacked denoising autoencoder and a long short-term memory network, which specifically includes the following steps: Step 1: At each micro base station, the denoising autoencoder model is trained using the historical content request data of local users to extract low-dimensional feature representations of content requests; the denoising autoencoder extracts and compresses data features layer by layer through a multi-layer autoencoder structure, and the output of each layer is used as the input of the next layer, gradually optimizing the feature representation to reduce noise interference in the data; Step 2: Input the hidden feature vector generated by the denoising autoencoder into the long short-term memory network to capture the time series dependency of user content requests; the long short-term memory network processes the short-term and long-term dependency characteristics of content requests through the input gate, forget gate, and output gate mechanism to enhance the ability to predict future content request trends; Step 3: After the micro base station completes the training of the denoising autoencoder and the long short-term memory network model locally, it uploads the trained model parameters, including weights and biases, to the global server. At the global server, the model parameters uploaded by multiple micro base stations are weighted averaged and aggregated to generate a global prediction model. The original user data is not shared during the aggregation process to ensure the protection of user privacy. The aggregated global model adapts to the differences in user request patterns in each micro base station area. Step 4: The global server sends the aggregated global model parameters to each micro base station. The micro base station further updates the local model based on the new content request data to improve the prediction accuracy. By regularly updating the global model and local model, the system can dynamically predict the popularity of content, and then adjust the cache strategy according to the prediction results to optimize the allocation of cache resources. Step 5: Based on the content popularity prediction results, each micro base station dynamically adjusts the placement and update strategy of cached content, giving priority to caching content with higher popularity, thereby improving the cache hit rate, reducing content transmission delay, and improving the utilization efficiency of network resources.

3. According to the multi-agent joint edge caching method based on the federated learning architecture of claim 1, it is characterized in that: The step of optimizing the transmission rate between the macro base station and the micro base station by allocating bandwidth and power resources includes the following: Step 1: Calculate content transmission delay, including: If the micro base station has cached the content requested by the end user, the content will be transmitted to the end user through the cellular communication link of the micro base station. The transmission delay of the content requested by the end user It can be expressed as: ； in, represents the cache decision variable, represents the cache content size, and the data transmission rate from the micro base station to the terminal user is expressed as: ； in, is the bandwidth, is the micro base station transmit power, represents the channel gain, represents the power spectral density of Gaussian white noise, represents the weight parameter, Indicates content popularity; If the micro base station does not cache the content requested by the terminal user, it is transmitted through the macro base station, and the transmission delay is: ； Among them, the transmission rate from the macro base station to the terminal user can be expressed as: ； Among them, the interference between macro base stations and micro base stations is as follows: ； ； Finally, the total transmission delay of the end-user request content can be expressed as: ； Step 2: Establish a utility function model for the macro base station and the micro base station. The utility function is determined by the cache hit rate, content transmission delay and power cost, and specifically includes: The utility of the service provider includes the benefits of transmitting macro or micro base station cache content to end users and the gain associated with content transmission delay, which is specifically expressed as: ； in, represents the unit satisfaction gain, Indicates the total number of end users who requested content, Indicates the maximum tolerable delay of content request; The cost of purchasing communication resources from the infrastructure provider is expressed as: ； in, is the macro base station transmission power pricing strategy, Micro base station transmission power pricing strategy; The utility of a service provider is the difference between its revenue and cost, which is related to the content delivery delay: ； The utility of the infrastructure provider is the difference between revenue and cost: ； in, represents the unit transmission power cost of the macro base station, represents the unit transmission power cost of micro base stations; Step 3: The game relationship between the service provider and the infrastructure provider is described through a game model. The service provider is the slave and the infrastructure provider is the master. The infrastructure provider sets the service pricing strategy and resource allocation mechanism based on its bandwidth and power resources. Step 4: Solve the optimal strategies of infrastructure providers and service providers through Stan-Kohlberg equilibrium; infrastructure providers control the resource usage of micro and macro base stations and optimize transmission rates by setting resource pricing strategies; service providers adjust their caching strategies and content transmission choices under given pricing and resource allocation schemes to maximize utility; Step 5: Dynamically adjust the cache decision and communication resource allocation strategy based on the game results to optimize the overall network utility while meeting the system constraints of content transmission delay, cache capacity and hit rate; the system gives priority to using micro base stations for cache transmission to reduce delay; if the micro base station does not cache the requested content, it is transmitted through the macro base station to ensure timely content transmission and efficient use of network resources.

4. According to the multi-agent joint edge caching method based on the federated learning architecture of claim 1, it is characterized in that: The dynamic adjustment process of cache decision and communication resource allocation is implemented through a multi-agent collaboration mechanism, which specifically includes the following steps: Step 1: Define agents for macro and micro base stations respectively. The agents are modeled through partially observable Markov decision processes to process observed user requests, content popularity, cache status, network bandwidth and other information. The elements of the partially observable Markov decision process are defined as follows: The observable state space of an agent is , the state transition probability function is ; In a partially observable Markov decision process, the state space is partially known and the decision is based on the global observation space The global observation space can be defined as ,in , , ; Representing an Agent in the past Action history of each time slot; Representation and Agents Related content popularity value; Representing an Agent The cache content state; the action space is defined as ,in , Indicates the cache strategy of the content service provider; , represents the price of communication resources of the infrastructure provider, where the pricing strategy adopted by the algorithm is differentiated pricing; the reward space is defined as ,in , ; All agents at time After taking an action, the reward is calculated according to the utility function, and then the next round of the game begins; Step 2: Each agent independently adjusts the cache strategy based on user request patterns, content popularity, and current network bandwidth usage, including the following; Step 2.1: Based on the predicted content popularity value, select and cache high-popularity content first; Step 2.2: If the popularity of the content decreases or the cache capacity reaches the upper limit, remove the low-popularity content in time to release cache space; Step 2.3: Dynamically adjust resource allocation for content transmission, including bandwidth and power scheduling, based on network congestion; Step 3: Optimize the strategy of each agent through a multi-agent deep reinforcement learning algorithm without information sharing, which specifically includes the following process; Step 3.1: Each agent performs local learning independently without sharing its observed local information; the agent updates its strategy through the actor-critic architecture; the actor network updates its strategy based on the agent’s current observations. , choose the best action , including cache decisions and resource allocation strategies; the critic network estimates the value function of the current strategy based on observations and actions, and stores historical experience in the memory buffer for strategy optimization; Step 3.2: At each time slot, the agent dynamically adjusts the cache and transmission strategies based on its local observations of user requests, content popularity, and cache hit rate; Step 3.3: The agent receives rewards from the environment after performing an action , and adjust the strategy for the next time slot by accumulating the reward value. The reinforcement learning goal is to maximize the long-term cumulative reward, where is the discount factor; Step 4: The strategies of each intelligent agent are federally coordinated and scheduled through a global server or a central controller to ensure balanced resource utilization between macro base stations and micro base stations. The central controller collects model parameters or observation results uploaded by each micro base station, evaluates the overall network performance, and calculates the global optimal caching strategy. Using the federated learning architecture, the central controller periodically sends down the aggregated global model parameters, and each micro base station adjusts the local caching strategy based on these parameters, dynamically optimizing the caching and transmission efficiency of the entire network to ensure global optimization. Step 5: Based on the strategic interaction between intelligent agents, the transmission path selection between the micro base station and the macro base station is dynamically adjusted to reduce cross-layer interference and optimize the content transmission path; the micro base station gives priority to transmitting cached content. If the micro base station does not cache the content requested by the user, it is transmitted through the macro base station with a smaller load, reducing network congestion and delay and optimizing the transmission path.

Images