A Mobile Edge Caching and Recommendation Method Based on Federated Reinforcement Learning
By employing federated reinforcement learning in mobile edge networks, content caching and recommendation strategies are optimized, solving the latency and resource waste problems of traditional network architectures and achieving low-energy, high-efficiency content transmission and privacy protection.
Patent Information
- Application Number
- CN202410016844.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-05
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-01-05
AI Technical Summary
Traditional cellular network architecture suffers from high backhaul link load, high latency, and resource waste when processing user requests. Furthermore, existing methods struggle to effectively address caching strategies, reasonable deployment of recommendation systems, and privacy protection in mobile edge networks.
We employ a federated reinforcement learning approach to model content caching and recommendation as Markov decision processes. By combining deep reinforcement learning and federated learning, and through a distributed agent collaborative sharing model, we can make decisions without uploading user data, thereby optimizing the energy consumption of content transmission and recommendation strategies.
It effectively reduces the transmission power consumption of mobile edge networks, improves cache hit rate and the accuracy of recommendation systems, and protects user privacy and location information.
Smart Images

Figure CN117857378B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of mobile edge caching and recommendation technology, specifically to a mobile edge caching and recommendation method based on federated reinforcement learning. Background Technology
[0002] With the continuous development of communication technology, the ever-increasing demand for networks, and the exponential growth of user device applications, we have entered the "Internet of Things (IoT) era." However, traditional cellular network architectures, limited by hardware development, can no longer cope with the challenges posed by servers and network bandwidth. In traditional network architectures, when all user requests are sent to cloud servers for processing, it increases the load on the backhaul links, and the long distance between the user end and the cloud server leads to large data transmission delays. When the workload is too large, it can cause system congestion, thus affecting the user experience. In addition, some users may send the same requests simultaneously; processing these duplicate tasks wastes network resources and increases the likelihood of link congestion.
[0003] Mobile edge computing (MEC) offers a solution comparable to cloud computing. MEC offloads some cloud computing functions to edge devices (EDs), enabling these devices to perform cloud computing functions such as information collection, storage, and processing. Deploying tasks to the network edge brings them closer to users, significantly improving performance, resulting in lower latency, faster response times, and quicker decision-making. In mobile edge computing, edge caching is a crucial research area for processing system resource deployment and improving quality of experience (QoE). For example, in video services, resource deployment on EDs cannot be arbitrarily limited by cost constraints. Traditional caching strategies assume fixed user request patterns, meaning recently requested or frequently requested videos are more likely to be accessed in the future. However, in mobile video networks, past request patterns often fail to accurately represent future video popularity. Some studies attempt to model popularity by selecting information such as user preferences and request history, but beyond explicit features, implicit characteristics in mobile video networks, such as user mobility and network status, can also influence video popularity, making it difficult to accurately model popularity using such strategies. Therefore, the quality of caching strategies is paramount.
[0004] Furthermore, recommender systems can reshape the probability of different users requesting content, affecting the effectiveness of caching. Recommender systems can have the following effects: (1) Increase cache hit rate: Recommender systems can guide users to request content already cached at the network edge based on their historical behavior and preferences. This can significantly improve the cache hit rate, reduce data transmission in the network, and thus reduce network latency and costs. (2) Optimize cache replacement strategy: Recommender systems can optimize cache replacement strategies based on users' historical behavior and preferences. For example, frequently requested content can be cached first to improve the cache hit rate. Therefore, combining recommender systems with content caching is a worthwhile research direction.
[0005] Several learning-based methods, such as Deep Reinforcement Learning (DRL), exist to address long-term optimization problems in edge caching and recommendation decisions. However, centralized learning methods require users to send their personal data to a central server, potentially raising privacy concerns. Even with anonymization or encryption, privacy breaches remain a risk. Federated Learning (FL) is a promising technique for addressing this issue. FL allows model training on distributed devices without centrally storing the original dataset on a central server. This means individual users' data can remain on their local devices for centralized learning, with only updated model parameters being transmitted to the central server for aggregation. This helps protect data privacy and security, reduces data transfer volume, and lowers communication costs.
[0006] However, several challenges still need to be addressed:
[0007] 1) How to make more reasonable recommendation and caching decisions in mobile edge networks?
[0008] 2) Recommendations on how to influence user requests
[0009] 3) How to deploy decision agents in a large network system. Summary of the Invention
[0010] To address the aforementioned challenges, we propose a mobile edge caching and recommendation method and system based on federated reinforcement learning. This method models content caching and recommendation in mobile networks as an optimization problem, aiming to reduce transmission energy consumption in the network. The proposed optimization problem is a nonlinear integer programming problem, which is NP-hard and difficult to find the optimal solution in polynomial time. Therefore, the content caching and recommendation process is modeled as a Markov decision process, and a reinforcement learning-based method is proposed to obtain an approximate optimal solution. This invention combines federated learning and reinforcement learning, proposing a novel jointly weighted federated reinforcement learning algorithm for the efficient joint weighted aggregation of terminal reinforcement models, thereby protecting the terminal's location information, contact information, and data privacy.
[0011] This invention provides a mobile edge caching and recommendation method based on federated reinforcement learning, comprising the following steps:
[0012] S1. Constructing a mobile network architecture;
[0013] S2. Based on the aforementioned mobile network architecture, construct a content transmission model;
[0014] S3. Based on the aforementioned mobile network architecture, construct a content popularity model;
[0015] S4. Based on the mobile network architecture, construct a node contact model;
[0016] S5. Construct a cache replacement model based on the content popularity model;
[0017] S6. Construct a node request model based on the content popularity model and the node contact model;
[0018] S7. Construct an optimization objective function with the goal of minimizing system energy consumption;
[0019] S8. Construct a Markov decision process based on the content delivery model and the node request model;
[0020] S9. Combining reinforcement learning and federated learning, we find the optimal strategy for the Markov decision process obtained in S8 to optimize content caching and content recommendation.
[0021] Preferably, the method for constructing the mobile network architecture in step S1 specifically includes the following steps:
[0022] The system is set to run for a total of T time slots, using sets express;
[0023] Set up content library for S f It is the size of the content f;
[0024] The network architecture includes a base station (BS) and several user equipment units. Each user device is considered a node, and each user device has a deep reinforcement learning agent;
[0025] Base station BS cache has content library All content in; for C u It is the cache capacity of user equipment u, defined For user equipment u in time slot The cache state;
[0026] for This indicates that user equipment u has no cached content f in time slot t. This indicates that there is cached content f in user equipment u in time slot t;
[0027] The contact between user devices changes as the user's location moves. The contacting user devices transmit content through a D2D link without occupying the cellular link; the D2D link refers to an end-to-end link.
[0028] use Let u be the contact vector of user equipment u, where This indicates the contact status between user equipment u and user equipment v in time slot t. This indicates that two user devices are in contact. This indicates that the two user devices are not in contact; Defined as 0; the group formed by all user devices in contact with a user device is the user's D2D group.
[0029] Preferably, the method for constructing the content delivery model in step S2 includes the following steps:
[0030] If the content f requested by user device u is already cached on the device itself, then... This indicates the corresponding latency for retrieving the content.
[0031] If the requested content f is not in the cache of user device u, i.e. The determination of whether content f can be obtained through the D2D link is based on the current contact state and the cache state of the contacting user equipment: when user equipment u and user equipment v are in contact in time slot t, and user equipment v has cached content f, i.e. User equipment u can then obtain content f from user equipment v via a D2D link; The delay in transmitting content is represented by the following method:
[0032]
[0033] in, This refers to the transmission rate from user equipment u to user equipment v. The system uses an orthogonal frequency division multiple access (OFDM) model to allocate non-overlapping orthogonal radio resources for D2D communication. The specific calculation method for the transmission rate is as follows:
[0034]
[0035] In this case, the original channel of user equipment i is divided into n equal parts. i Sub-channels, B i This represents the available bandwidth of user equipment i. This represents the channel gain of user equipment i and user equipment j in time slot t. ω represents the D2D transmission power of user equipment i. t σ represents the co-channel interference level from neighboring cells in time slot t. v 2 This represents the power of additive white Gaussian noise.
[0036] If content f cannot be obtained through local caching and D2D linking, i.e. and User equipment u will then redirect the request to the base station and obtain the requested content f via the cellular link; This represents the delay in which user equipment u obtains content f from the base station in time slot t, and is calculated as follows:
[0037]
[0038] Among them, data transmission rate The calculation method is as follows:
[0039]
[0040] Among them B B , These represent the corresponding channel bandwidth, transmit power, channel gain, and background noise power σ. b The ratio function;
[0041] The above three transmission delays satisfy:
[0042] As a preferred embodiment, the method for constructing the content popularity model in step S3 specifically includes the following steps:
[0043] Obtain historical rating data of users for content, construct a rating matrix of users for content based on the historical rating data, and use a collaborative filtering algorithm based on width incremental learning to predict the missing values in the rating matrix to obtain a complete rating matrix without missing values.
[0044] for User u to content library The predicted rating vector for all content in the content is defined as rate. u ={r u,1 ,r u,2 ,...,r u,F},
[0045] Where, r u,F This indicates that user u has access to the content library. The predicted score for content F in the text;
[0046] use This indicates the user device u's preference for content f. Following the Zipf distribution, the user device u's ratings of the content are sorted from highest to lowest to obtain the ranking of each piece of content. The calculation method is as follows:
[0047]
[0048] Where σ is the slope parameter of the Zipf distribution;
[0049] Finally, the user device u's preference for content f is normalized, so that...
[0050] The popularity model consists of two parts:
[0051] (1) Calculation of request probability from the user device's perspective:
[0052] Set request probability Let f be the probability that user device u requests content f in time slot t, which varies with the actual request from the user device; set Let be the initial request probability of user device u for content f, and let be...
[0053] Suppose that user equipment u in time slot t has a probability If content f is requested, then in time slot t+1, the probability of user device u requesting content f becomes... The request probability must be normalized in each time slot;
[0054] (2) Calculation of D2D group content popularity:
[0055] Setting D2D group content popularity In the D2D cluster of user equipment u, the popularity of content f in time slot t is calculated as follows:
[0056] .
[0057] Preferably, the method for constructing the node contact model in step S4 specifically includes the following steps:
[0058] set up For user equipment u, there is geographical contact with other user equipment in time slot t; for user equipment v∈U, v≠u, in, This indicates that there is a geographical connection between user equipment u and user equipment v; otherwise, set up =0;
[0059] set up Let be the preference similarity vector between user equipment u and other user equipment in time slot t; use cosine similarity to calculate the similarity between user equipment u and user equipment v.
[0060]
[0061] in, It is the preference vector of user device u.
[0062] Based on geographical and preference factors, the contact probability between user equipment u and user equipment v is obtained, and the calculation method is as follows:
[0063] .
[0064] As a preferred embodiment, the method for constructing the cache replacement model in step S5 specifically includes the following steps:
[0065] Configure when user device u needs to cache new content x that is not locally available, i.e. If the local available cache is insufficient to continue caching the new content, that is... Then perform cache replacement, where S x It is the size of the new content x;
[0066] The replacement method includes: calculating the D2D group content popularity of all content cached in user device u according to the calculation method of D2D group content popularity in step S3, clearing the cache starting from the content with the lowest group content popularity, until the remaining local available cache is greater than or equal to the size of the new content x to be cached.
[0067] As a preferred embodiment, the method for constructing the node request model in step S6 specifically includes the following steps:
[0068] In the federated learning architecture, each user device is treated as an independent intelligent agent, and content is recommended for that user device, thereby influencing the user's request behavior.
[0069] Set the probability that user device u receives recommended content in time slot t as follows: The probability that u does not accept recommended content in time slot t is: Let the probability of user equipment u receiving a recommendation in the initial time slot be set as follows:
[0070] Set to recommend content to user device u in time slot t This indicates that no content is recommended. This indicates specific content within the recommended content library; the content requested by user device u in time slot t is...
[0071] The next step involves determining the content of u's request based on the agent's recommendation and updating the probability of accepting the recommendation.
[0072] When the agent does not recommend any content to the user.
[0073] Requested content It is determined solely by the probability of u requesting the content, i.e., based on the request probability. Determine the actual content requested; the probability of the user accepting the recommendation remains unchanged.
[0074] When an agent recommends content from its content library to a user
[0075] Firstly, in time slot t, based on... The probability of requests for content other than those specified is used to pre-determine the content that user device u might request. Then with Probability of requesting recommended content by Probability request content Determine the content of the actual request.
[0076] Request content Then, update u to The request probability is adjusted to avoid users repeatedly requesting the same content; and the probability of u accepting recommendations is updated. If the requested content is the same as the recommended content, then it is further determined whether the requested content can be retrieved locally or via a D2D link; if so, the probability of accepting the recommendation is increased; otherwise, the probability is decreased; if the requested content is not the recommended content, then... Will Control it within [0.4, 0.7].
[0077] As a preferred embodiment, the method for constructing the optimization objective function in step S7 specifically includes the following steps:
[0078] In this method, we aim to optimize the energy consumption of content transmission for user devices in joint content caching and recommendation decision-making. System energy consumption is proportional to transmission latency; we define the energy consumption generated per unit transmission time as p. u , representing the transmission power of user equipment u, and the transmission delay is . The energy consumed by user equipment u to acquire content in time slot t. The calculation method is as follows:
[0079]
[0080] The system optimization objective is expressed as:
[0081]
[0082] Where γ∈[0,1) is the discount factor.
[0083] As a preferred embodiment, the method for constructing the Markov decision process in step S8 specifically includes the following steps:
[0084] The optimization problem proposed in step S7 is modeled as a Markov decision process. The state, action, and system reward of each user device are set as follows: For user device u:
[0085] (1) Status:
[0086] At time slot t, the state space of user equipment u is represented as follows: in These represent the request probability and content caching status of user device u in time slot t, respectively; This is the ratio of the remaining cache capacity of user equipment u in time slot t to the total cache capacity.
[0087] (2) Action:
[0088] Based on state The intelligent agent of user equipment u performs an action in time slot t. This indicates that content is cached. And recommend a piece of content and These represent not caching any content and not recommending any content, respectively; the content to be cached is set in time slot t. It is cached in real time;
[0089] (3) System Rewards:
[0090] Based on state and actions u's reward in time slot t The calculation method is as follows:
[0091]
[0092] Where η is a hyperparameter used to control the ratio of the two parts of the reward; It is the energy loss caused by the request. It is the expected reward generated by D2D transmission;
[0093] User equipment u sends a content request in time slot t. Content transmission latency loss This request generates the calculation method as follows:
[0094]
[0095] Where K is a positive number, it is the ratio of the upper limit of the content size range to the lower limit of the D2D link transmission rate range; when a request When this can be achieved through D2D linking or local caching. Set to a negative value; when the request can only be obtained via a cellular link. Set to a positive value.
[0096] This represents the gain in content acquisition through local caching and the D2D link; at the beginning of each time slot, for user device u, initialization... =0; iterate through all content in the content library, for If cached locally Or it may be cached by the user device it is in contact with. Then add as follows
[0097]
[0098] (4) Reconstruction and optimization objectives:
[0099] Based on the construction of the Markov decision process described above, we reconstruct the optimization objective as follows: given the system in a stochastic initial state S... rand The optimal strategy aims to reduce the energy consumption of content transmission and cellular link utilization of user equipment clusters in the system, maximize the ability of user equipment to obtain content through local caching and D2D links, and better meet user preferences.
[0100]
[0101] st:
[0102] C1:
[0103] C2:
[0104] C3:
[0105] C4:
[0106] Where γ∈[0,1) is the discount factor.
[0107] Preferably, step S9 specifically includes the following steps:
[0108] By implementing distributed joint deep reinforcement learning and federated learning agent collaborative sharing models in mobile edge networks, decision-making can be made securely without uploading user data.
[0109] Each user device trains a DQN network based on its own user data to learn the value function of different action strategies in a given state, enabling it to make optimal decisions and maximize rewards. The DQN value function is as follows:
[0110]
[0111] Where γ is the discount factor, γ∈[0,1), and α is the learning rate, α∈[0,1); DQN consists of two Q-networks with identical architectures: the evaluation network Q and the target network. θ and These represent the parameters of the two networks respectively; the empirical replay of the DQN model stores historical transformation samples in the replay buffer B. u In this context, the loss function of the DQN model is expressed as:
[0112]
[0113] in, It is B u Random data drawn in batches from the playback buffer. It is a time difference target;
[0114] The network parameters are evaluated by training to minimize the loss; we use the gradient descent (GD) method.
[0115]
[0116] The evaluation network and the target network will collaborate to update the parameters of the DQN, and the evaluation network parameters will be updated accordingly. The parameters of the evaluation network are updated after each training step; after a certain number of updates, the parameters of the evaluation network are passed to the target network and used as the target network parameters. The new parameters; the local DQN training process on the user device includes the following steps:
[0117] The agent is placed in the initial state of the environment and selects an initial action. The agent executes the action, and when executing a cached action, it needs to determine whether cache replacement is necessary. Then, it receives a reward and moves to the next state. The agent's experience is stored in an experience replay buffer, where each experience is a quadruple consisting of the state, the executed action, the reward, and the next state. Samples are randomly drawn from the experience replay buffer to train a Q-network, enabling it to more accurately predict the long-term reward of each action, thereby improving the agent's policy. Using the trained Q-network, the agent selects the optimal action based on the current state.
[0118] A central server is set up at the base station as a proxy for the federated aggregation of distributed local model parameters. The central proxy has the same Q network architecture as the proxy on the user equipment and periodically performs federated parameter aggregation; the network parameters of the central proxy are... ep is the current round number;
[0119] The system runs for several rounds, with a preset number of steps in each round. Contact between user devices changes periodically based on the mobile node model. At each step, each user device trains using its local data according to its local DQN training algorithm. At the beginning of a round, the total reward is initialized. After all agents have completed training, each agent sums its own rewards from the entire round to obtain the total reward for the current round. If federated aggregation is required in the current round, then the global parameters It is obtained by weighting and aggregating the network parameters of all user devices using the total reward of the user devices throughout the entire round as the weight:
[0120]
[0121] The aggregated global model parameters are then distributed to agents on all user devices, and the agents update the network parameters for subsequent training.
[0122] Currently, research exists in the field of edge networks on caching, recommendation, reinforcement learning, and federated learning. However, existing single or combined methods have failed to effectively solve the problems in mobile edge networks. This patent aims to optimize content caching and recommendation models in mobile edge networks into a unified whole to address the challenges in real-world scenarios, particularly reducing network transmission energy consumption.
[0123] The proposed optimization problem involves nonlinear integer programming, making it difficult to obtain the optimal solution in polynomial time. Therefore, this patent employs a Markov decision process to model the content caching and recommendation process, and obtains an approximate optimal solution based on reinforcement learning. Simultaneously, combining federated learning, a novel federated reinforcement learning algorithm is proposed, aiming to protect the location information, contact information, and data privacy of terminal devices. This method will more effectively handle content caching and recommendation problems in mobile edge networks, thereby reducing energy consumption and improving efficiency. Attached Figure Description
[0124] Figure 1 The system architecture for the proposed mobile edge caching and recommendation method based on federated reinforcement learning is presented.
[0125] Figure 2 This is the training method for DQN;
[0126] Figure 3 A joint training method for federated reinforcement learning.
[0127] Figure 4 The flowchart of this invention Detailed Implementation
[0128] This invention considers a scenario in a mobile edge network, comprising a content library, a base station, and multiple mobile users covered by that base station. A central server is located at the base station, and each mobile user is equipped with a caching device. Each user device also has a deep reinforcement learning agent. Each deep reinforcement learning agent recommends content to the user and makes caching decisions for the user device. When a user requests content, the requested content can be obtained through the user device's own cache, from a contacting user device via a D2D link, or from the base station via a cellular link, incurring corresponding energy consumption.
[0129] To reduce transmission energy consumption in networks, this invention models content caching and recommendation in mobile networks as an optimization problem. The proposed optimization problem is a nonlinear integer programming problem, which is NP-hard and difficult to find the optimal solution in polynomial time. Therefore, the content caching and recommendation process is modeled as a Markov decision process, and a reinforcement learning-based method is proposed to obtain an approximate optimal solution. This invention combines federated learning and reinforcement learning, proposing a novel jointly weighted federated reinforcement learning algorithm for the efficient joint weighted aggregation of terminal reinforcement models, thereby protecting the terminal's location information, contact information, and data privacy.
[0130] A mobile edge caching and recommendation method based on federated reinforcement learning includes the following steps:
[0131] 1. Constructing a mobile network architecture:
[0132] The system is set to run for a total of T time slots, using sets This indicates that each time slot is long enough to transmit any content. A content library is configured in the system. for S f It refers to the size of the content f. For example... Figure 1 As shown, the network architecture includes one base station (BS) and a group of user equipment. Each user device is also considered a node and has a deep reinforcement learning agent. We set the BS's cache to be infinitely large, so when the BS needs to retrieve content f, it can retrieve it instantly from the cloud, which means the BS is considered to have cached all the content in the content library. C u This is the cache capacity of the user device u, set. This refers to the buffer state of user equipment u in time slot t. This indicates that user equipment u has no cached content f in time slot t, and vice versa.
[0133] Users move within the system, and the contact between user devices changes with this movement. Contacting user devices can transmit content via end-to-end (Device-to-Device, D2D) links without consuming cellular links, thus improving transmission efficiency and reducing the transmission load on cellular links. We use... Indicates user equipment u The contact vector. Where... This indicates that in time slot t, user equipment u and user equipment Contact details. This indicates that the two user devices are in contact; otherwise... It is worth noting that, Furthermore, we define a user device as the center, and the group formed by all user devices that come into contact with it as that user's D2D group.
[0134] 2. Construct a content delivery model:
[0135] For user equipment, there are three content transmission methods, each with different transmission latency and power consumption. The system is configured with sufficiently long time slots, ensuring that any content can be transmitted within a single time slot, regardless of the transmission method used.
[0136] Method 1: If the content f requested by user u is already cached on the device itself, i.e. Then the request can be fulfilled directly. Let... This indicates the corresponding latency for retrieving the content.
[0137] Method 2: If the cache of user device u cannot satisfy the content f requested by user device, i.e. Therefore, it is necessary to determine whether content f can be obtained through the D2D link based on the current contact state and the cache state of the contacting user equipment. Assume that in time slot t, user equipment u and user equipment v are in contact, and user equipment v has cached content f, i.e. User equipment u can then obtain content f from user equipment v via a D2D link. We use... The delay in transmitting content is represented by the following method:
[0138]
[0139] in, This is the transmission rate from user equipment u to user equipment v. The system uses an orthogonal frequency division multiple access (OFDM) model to allocate non-overlapping orthogonal radio resources for D2D communication. The specific calculation method for the transmission rate is as follows:
[0140]
[0141] In this case, the original channel of user equipment i is divided into n equal parts. i Sub-channels, B i This represents the available bandwidth of user equipment i. This represents the channel gain of user equipment i and user equipment j in time slot t. ω represents the D2D transmission power of user equipment i. t σ represents the co-channel interference level from neighboring cells in time slot t. v 2 This represents the power of additive white Gaussian noise.
[0142] Method 3: If content f cannot be retrieved through local caching and D2D linking, i.e. and User equipment u can then redirect the request to the base station and obtain the requested content f via the cellular link. This patent uses... This represents the delay at which user equipment u obtains content f from the base station in time slot r. It is calculated as follows:
[0143]
[0144] Among them, data transmission rate The calculation method is as follows:
[0145]
[0146] Among them B B , These represent the corresponding channel bandwidth, transmit power, channel gain, and background noise power σ. b The ratio function.
[0147] The transmission delays of the above three methods satisfy:
[0148]
[0149] 3. Construct a content popularity model:
[0150] Based on historical user rating data for content, a user rating matrix is constructed. Then, a collaborative filtering algorithm based on width incremental learning (BILCF) is used to predict missing values, resulting in a complete rating matrix without missing values. (The last sentence appears to be incomplete and possibly refers to setting a different parameter for a given topic.) The predicted rating vector for all content is rate. u ={r u,1 ,r u,2 ,...,r u,F}
[0151] We use This represents the user device u's preference for content f, and it follows a Zipf distribution to more reasonably describe the relationship between frequency and rating level. We will consider the user device... The content is ranked from highest to lowest score, with each piece of content corresponding to a ranking. So The calculation method is as follows:
[0152]
[0153] Where σ is the slope parameter of the Zipf distribution. Finally, we normalize the user device u's preference for content f, so that...
[0154] The popularity model mainly consists of two parts: the request probability from the user device perspective and the content popularity from the D2D group perspective, both of which are calculated based on user preferences.
[0155] (1) Request probability calculation method:
[0156] Set request probability This represents the probability that user device u will request content f in time slot t, and it varies with the actual request from the user device. (Setting) Let be the initial request probability of user device u for content f, and let be... This system is designed to ensure complete transmission of any content within a single time slot. User equipment (UE) is guaranteed to receive the requested content in the current time slot, and UEs do not frequently request the same content. Therefore, the probability of requesting the same content again in the next time slot decreases. Assume that UE u in time slot t has a probability of... If content f is requested, then in time slot t+1, the probability of user device u requesting content f becomes... In addition, the request probability must be normalized in each time slot.
[0157] (2) Calculation method for D2D group content popularity:
[0158] set up In a D2D cluster for user equipment u, the popularity of content f in time slot t represents the popularity of content f within a local area. The calculation method is as follows:
[0159]
[0160] 4. Construct the mobile node model:
[0161] We set user device contact changes based on geographical and preference factors.
[0162] Geographical factors:
[0163] Because user equipment is constantly moving, contact between user equipment is affected by the user's geographical location. User equipment that is far apart in the system cannot perform D2D transmission. (Settings) For user equipment u, this refers to the geographical contact between user equipment and other user equipment in time slot t. This indicates that there is a geographical connection between user equipment u and user equipment v. Otherwise, It is worth noting that, It is always 0.
[0164] Preference factors:
[0165] We assume that users are more willing to share content with other users who have similar interests, and model preference exposure based on user preference similarity. Let v be the preference similarity vector between user equipment u and other user equipment in time slot t. Cosine similarity is used to calculate the similarity between user equipment u and user equipment v.
[0166]
[0167] in, It is the preference vector of user device u.
[0168] Based on the aforementioned geographical and preference factors, the contact probability between user equipment u and user equipment v can be calculated. The actual contact will be generated based on this probability. The calculation method is as follows:
[0169]
[0170] 5. Construct a cache replacement model:
[0171] The cache of user devices in the system is fixed, so when the cache is insufficient, its contents need to be replaced. Let's define what happens when user device u needs to cache new content x that is not currently present locally. If the local available cache is insufficient to continue caching the new content, that is... Then, cache replacement is performed. The replacement rule is to replace the least popular content in the D2D group of user device u in the cache until the remaining local available cache is greater than or equal to the size of the new content x to be cached.
[0172] 6. Build recommendation and request models:
[0173] In a federated learning architecture, each user device acts as an independent intelligent agent, recommending content for itself and thus influencing user requests. The probability that user device u receives recommended content in time slot t is set as follows: The probability that u does not accept recommended content in time slot t is: Set the initial probability of user device u accepting the recommendation as follows: Set to recommend content to user device u in time slot t This indicates that no content is recommended. This refers to specific content within the recommended content library. The content requested by user device u in time slot t is...
[0174] The next step involves determining the content requested by u based on the agent's recommendation, as well as updating the probability of accepting the recommendation.
[0175] (1)
[0176] When the agent does not recommend any content to the user, the requested content... It is determined solely by the probability of u requesting the content, i.e., based on the request probability list. Determine the actual content requested. The probability of the user accepting the recommendation remains unchanged.
[0177] (2)
[0178] When the agent recommends content from the content library to the user, it first considers u in time slot t. Predetermine the content that might be requested based on the probability of requesting other content. Then with Probability of requesting recommended content by Probability request content Determine the content of the actual request. Request content Then, update u to The request probability is adjusted to avoid users repeatedly requesting the same content. The probability of u accepting a recommendation is also updated. If the requested content is the same as the recommended content, then it is further determined whether the requested content can be retrieved locally or via a D2D link. If it can, the probability of accepting the recommendation is increased; otherwise, the probability is decreased. If the requested content is not the recommended content, then... Finally, Control it within [0.4, 0.7].
[0179] 7. Construct the optimization objective function:
[0180] In this method, we aim to optimize the energy consumption of content transmission on the user terminal during joint content caching and recommendation decisions. System energy consumption is directly proportional to transmission latency. We define the energy consumption per unit transmission time as p. u , representing the transmission power of user equipment u, and the transmission delay is . The energy consumed by user equipment u to acquire content in time slot t. The calculation method is as follows:
[0181]
[0182] The system optimization objective is expressed as:
[0183]
[0184] Where γ∈[0,1) is a discount factor.
[0185] 8. Construct a Markov decision process:
[0186] We model the optimization problem proposed in step S7 as a Markov Decision Process (MDP), setting the state, action, and system reward for each user device. Taking user device u as an example:
[0187] (1) Status:
[0188] At time slot t, the state space of u is represented as: in These represent the request probability of u in time slot t and the content caching status, respectively. It is the ratio of the remaining cache capacity of u in time slot t to the total cache capacity.
[0189] (2) Action:
[0190] Based on state The deep reinforcement learning agent of user equipment u performs an action in time slot t. Cache one piece of content respectively And recommend a piece of content set up and These represent not caching any content and not recommending any content, respectively. We also set the content to be cached in time slot t. It is cached in real time.
[0191] (3) System Rewards:
[0192] Based on state and actions u's reward in time slot t The calculation method is as follows:
[0193]
[0194] Here, η is a hyperparameter used to control the ratio of the two parts of the reward. The energy loss caused by the request is a negative impact. The expected reward generated by D2D transmission is a positive effect.
[0195] The energy consumption for transmitting content is directly proportional to the transmission delay; therefore, energy loss can be represented by delay. Suppose user equipment u sends a content request in time slot t. So what about the transmission latency loss when acquiring content? This request generates the calculation method as follows:
[0196]
[0197] Where K is a positive number, representing the ratio of the upper limit of the content size range to the lower limit of the D2D link transmission rate range. When a request... When this can be achieved through D2D linking or local caching. Setting it to a negative value has a positive effect on both content retrieval methods. This is especially true when the request can only be retrieved via a cellular link. Setting it to a positive value has a negative effect. When transmitting via a D2D link, the latency loss is implemented using an exponential function, meaning that the smaller the D2D transmission latency, the smaller the loss.
[0198] This represents the gain in content acquisition through local caching and the D2D link. At the beginning of each time slot (taking time slot t as an example), for user equipment u, initialization... The value is 0. Traverse all content in the content library, for... If cached locally Or it may be cached by the user device it is in contact with. Then add as follows
[0199]
[0200] (4) Reconstruction and optimization objectives:
[0201] Based on the construction of the Markov decision process described above, we reconstruct the optimization objective as follows: given the system in a stochastic initial state S... rand To reduce the energy consumption of content transmission and cellular link utilization of user equipment clusters in the system, we aim to maximize the ability of user equipment to retrieve content through local caching and D2D links, and better meet user preferences. Our optimal strategy can be expressed as:
[0202]
[0203]
[0204]
[0205]
[0206]
[0207] Where γ∈[0,1) is the discount factor.
[0208] 9. Methods for implementing content caching and recommendation algorithms based on federated reinforcement learning:
[0209] Content caching and recommendation algorithms based on federated reinforcement learning are used to solve the aforementioned optimization problem. This method makes decisions securely without uploading user data by implementing a distributed joint deep reinforcement learning (DRL) and federated learning (FL) agent collaborative sharing model in a mobile edge network. It consists of two parts: local deep Q-network (DQN) training and federated learning.
[0210] (1) Local DQN training
[0211] Each user device trains a DQN network based on its own user data to learn the value function of different action strategies in a given state, enabling it to make optimal decisions and maximize rewards. The DQN value function is as follows:
[0212]
[0213] Where γ is the discount factor, γ∈[0,1), and α is the learning rate, α∈[0,1). DQN consists of two Q-networks with identical architectures: the evaluation network Q and the target network. θ and These are the parameters of the two networks, respectively. The empirical replay of the DQN model stores historical transformation samples in the replay buffer B. u In this model, the network can iteratively learn from samples, addressing the issue of temporal correlation between samples and improving training stability. Using empirical replay, the loss function of the DQN model is expressed as:
[0214]
[0215] in, It is B u Random data drawn in batches from the playback buffer. It is a time difference (TD) target.
[0216] The network parameters are evaluated by training to minimize the loss; we use the gradient descent (GD) method.
[0217]
[0218] The evaluation network and the target network will collaborate to update the parameters of the DQN, and the evaluation network parameters will be updated accordingly. The parameters of the evaluation network are updated after each training step. After a certain number of updates, the parameters of the evaluation network are passed to the target network and used as the target network parameters. New parameters.
[0219] like Figure 2 As shown, the local DQN training process on the user equipment is as follows:
[0220] The agent is placed in an initial state in the environment and chooses an initial action. The agent performs the action, receives a reward, and transitions to the next state. The agent's experience is stored in an experience replay buffer, where each experience is a quadruple consisting of the state, the action performed, the reward, and the next state. Samples are randomly drawn from the experience replay buffer to train a Q-network, enabling it to more accurately predict the long-term reward of each action, thereby improving the agent's policy. Using the trained Q-network, the agent selects the optimal action based on the current state. This process is iterated until the agent learns a good policy.
[0221] (2) Federated Learning
[0222] like Figure 3 As shown, a central aggregation server is set up at the base station as a proxy for the federated aggregation of distributed local model parameters. The central aggregation proxy has the same Q network architecture as the user equipment's proxy and periodically performs federated parameter aggregation. The network parameters of the central aggregation proxy are... ep is the current round number.
[0223] The system runs in multiple rounds, with a certain number of steps in each round. The interactions between user devices change periodically based on the mobile node model. At each step, each user device trains using its local data according to its local DQN training algorithm. At the beginning of a round, the total reward is initialized. After all agents have completed training, each agent sums its own rewards from the entire round to obtain the total reward for the current round. If federated aggregation is required in the current round, then the global parameters It is obtained by weighting and aggregating the network parameters of all user devices using the total reward of the user devices throughout the entire round as the weight:
[0224]
[0225] The aggregated global model parameters are then distributed to agents on all user devices, and the agents update the network parameters for subsequent training.
[0226] like Figure 4 The diagram shows the overall process of a mobile edge caching and recommendation method based on federated reinforcement learning.
[0227] The above description is only a part of the specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the protection scope of the present invention.
Claims
1. A mobile edge caching and recommendation method based on federated reinforcement learning, characterized in that, Includes the following steps: S1. Constructing a mobile network architecture; S2. Based on the aforementioned mobile network architecture, construct a content transmission model; S3. Based on the aforementioned mobile network architecture, construct a content popularity model; S4. Based on the mobile network architecture, construct a node contact model; S5. Construct a cache replacement model based on the content popularity model; S6. Construct a node request model based on the content popularity model and the node contact model; S7. Construct an optimization objective function with the goal of minimizing system energy consumption; S8. Construct a Markov decision process based on the content delivery model and the node request model; S9. Combining reinforcement learning and federated learning, we find the optimal strategy for the Markov decision process obtained in S8 to optimize content caching and content recommendation. In step S6, the method for constructing the node request model specifically includes the following steps: In a federated learning architecture, each user device is treated as an independent intelligent agent, and content is recommended for that user device, thereby influencing the user's request behavior. Configure user equipment In the time slot The probability of accepting recommended content is ,So In the time slot The probability of not accepting recommended content is Configure user equipment The probability of accepting the recommendation is ; Set in time slot To user equipment Recommended content , This indicates that no content is recommended. This refers to specific content within the recommended content library; user device In the time slot The content of the request is ; Next, based on the agent's recommendation, determine... The process of requesting content and updating the probability of accepting recommendations; When the agent does not recommend any content to the user. Requested content Only by The probability of requesting content is determined, that is, based on the request probability. Determine the actual content requested; the probability of the user accepting the recommendation remains unchanged. ; When an agent recommends content from its content library to a user Firstly, in the time slot Based on, in addition to The probability of requests for content other than those from the user device is predetermined. Possible request content Then with Probability of requesting recommended content ,by Probability request content Sure The actual content requested ; Request content Afterwards, update right The request probability is adjusted to prevent users from repeatedly requesting the same content; and updates are performed. Probability of accepting a recommendation If the requested content is the same as the recommended content, then it is further determined whether the requested content can be retrieved locally or via a D2D link; if so, the probability of accepting the recommendation is increased; otherwise, the probability is decreased; if the requested content is not the recommended content, then... ;Will Controlled ; Step S9 specifically includes the following steps: By implementing distributed joint deep reinforcement learning and federated learning agent collaborative sharing models in mobile edge networks, decision-making can be made securely without uploading user data. Each user device trains a DQN network based on its own user data to learn the value function of different action strategies in a given state, enabling it to make optimal decisions and maximize rewards. The DQN value function is as follows: in It is a discount factor. , It's the learning rate. DQN consists of two Q-networks with identical architectures: the evaluation network and the Q-network. and target network , and These represent the parameters of the two networks, respectively; the empirical replay of the DQN model stores historical transformation samples in the replay buffer. In this context, the loss function of the DQN model is expressed as: in, yes Random data drawn in batches from the playback buffer. It is a time difference target; The network parameters are evaluated by training to minimize the loss using the gradient descent (GD) method. The evaluation network and the target network will collaborate to update the parameters of the DQN, and the evaluation network parameters will be updated accordingly. The parameters of the evaluation network are updated after each training step; after a certain number of updates, the parameters of the evaluation network are passed to the target network and used as the target network parameters. The new parameters; the local DQN training process on the user device includes the following steps: The agent is placed in the initial state of the environment and selects an initial action. The agent executes the action, and when executing a cached action, it needs to determine whether cache replacement is necessary. Then, it receives a reward and moves to the next state. The agent's experience is stored in an experience replay buffer, where each experience is a quadruple consisting of the state, the executed action, the reward, and the next state. Samples are randomly drawn from the experience replay buffer to train a Q-network, enabling it to more accurately predict the long-term reward of each action, thereby improving the agent's policy. Using the trained Q-network, the agent selects the optimal action based on the current state. A central server is set up at the base station as a proxy for the federated aggregation of distributed local model parameters. The central proxy has the same Q network architecture as the proxy on the user equipment and periodically performs federated parameter aggregation; the network parameters of the central proxy are... , This is the current round number; The system runs for several rounds, and in each round the system runs... Step count; the contact between user devices changes periodically according to the mobile node model; each user device trains using its local data according to the local DQN training algorithm at each step; at the beginning of a round, the total reward is initialized; after all agents have completed training, each agent adds up its own rewards for the entire round to obtain the total reward for the current round. If federated aggregation is required in the current round, then the global parameters... It is obtained by weighting and aggregating the network parameters of all user devices using the total reward of the user devices throughout the entire round as the weight: The aggregated global model parameters are then distributed to agents on all user devices, and the agents update the network parameters for subsequent training.
2. The mobile edge caching and recommendation method based on federated reinforcement learning according to claim 1, characterized in that, In step S1, the method for constructing the mobile network architecture specifically includes the following steps: Configure the system to run together Each time slot, using a set express; Set up content library ,for , It is content Size; The network architecture includes a base station (BS) and several user equipment units. Each user device is considered a node, and each user device has a deep reinforcement learning agent; Base station BS cache has content library All content in; for , User equipment Cache capacity, defined For user equipment In the time slot The cache state; for Indicates in time slot User equipment No cached content , Indicates in time slot User equipment There is cached content. ; The contact between user devices changes as the user's location moves. The contacting user devices transmit content through a D2D link without occupying the cellular link; the D2D link refers to an end-to-end link. use Indicates user equipment The contact vector, where Indicates in time slot User equipment and user equipment Contact situation, , This indicates that two user devices are in contact. This indicates that the two user devices are not in contact; Defined as 0; the group formed by all user devices in contact with a user device is the user's D2D group.
3. The mobile edge caching and recommendation method based on federated reinforcement learning according to claim 2, characterized in that, In step S2, the method for constructing the content delivery model specifically includes the following steps: If user equipment Requested content It is already cached on the device itself, set This indicates the corresponding latency for retrieving the content. ; If user equipment The requested content is not in the cache. ,Right now Based on the current contact status and the cache status of the user's device, it is determined whether content can be retrieved via the D2D link. When in a time slot User equipment and user equipment Contact, and user equipment Content cached ,Right now Then the user equipment From user equipment via D2D link Get content ;use The delay in transmitting content is represented by the following method: in, User equipment To user equipment The transmission rate; the system uses an orthogonal frequency division multiple access model to allocate non-overlapping orthogonal radio resources for D2D communication, and the specific calculation method for the transmission rate is as follows: Among them, user equipment The original channel was divided into equal parts. Sub-channels, Indicates user equipment Available bandwidth, Indicates user equipment and user equipment In the time slot Channel gain, Indicates user equipment D2D transmission power, Indicates time slot Co-channel interference levels from neighboring cells This represents the power of additive white Gaussian noise. If content cannot be retrieved through local caching and D2D linking ,Right now and Then the user equipment The request is redirected to the base station, and the requested content is obtained via the cellular link. ;use Indicates user equipment In the time slot Content obtained from the base station The delay is calculated as follows: Among them, data transmission rate The calculation method is as follows: in , , These are the corresponding channel bandwidth, transmit power, channel gain, and background noise power. The ratio function; The three transmission delays satisfy: .
4. The mobile edge caching and recommendation method based on federated reinforcement learning according to claim 3, characterized in that, In step S3, the method for constructing the content popularity model specifically includes the following steps: Obtain historical rating data of users for content, construct a rating matrix of users for content based on the historical rating data, and use a collaborative filtering algorithm based on width incremental learning to predict the missing values in the rating matrix to obtain a complete rating matrix without missing values. for , will users For content library The predicted rating vector for all content in is defined as follows: , in, Indicates user The predicted score for content F; use Indicates user equipment For content preferences, Follows the Zipf distribution; User equipment The content is ranked from highest to lowest score to obtain a ranking for each piece of content. , The calculation method is as follows: in, It is the slope parameter of the Zipf distribution; Finally, the user equipment For content The preferences are normalized so that ; The popularity model consists of two parts: (1) Calculation of request probability from the user device's perspective: Set request probability For user equipment In the time slot Request content The probability varies with the actual request from the user's device; setting For user equipment For content The initial request probability, and let ; User equipment In the time slot With probability Request content So in the time slot User equipment For content The request probability becomes The request probability must be normalized in each time slot. (2) Calculation of D2D group content popularity: Setting D2D group content popularity For user equipment In a D2D cluster, content In the time slot The popularity is calculated as follows: 。 5. A mobile edge caching and recommendation method based on federated reinforcement learning according to claim 4, characterized in that, In step S4, the method for constructing the node contact model specifically includes the following steps: set up For user equipment In the time slot Geographical contact with other user equipment; for user equipment , ;in, Indicates user equipment and user equipment There must be geographical contact between them; otherwise... ,set up =0; set up For user equipment In the time slot Preference similarity vectors between user devices; cosine similarity is used to calculate user device similarity. and user equipment of : in, User equipment The preference vector, ; Based on geographical and preference factors, user equipment is obtained. and user equipment The probability of contact between them is calculated as follows: 。 6. A mobile edge caching and recommendation method based on federated reinforcement learning according to claim 4, characterized in that, In step S5, the method for constructing the cache replacement model specifically includes the following steps: Configure when user device We need to cache new content that is not available locally. At that time, that is If the available local cache is insufficient to continue caching the new content, i.e. If so, then cache replacement will be performed, where This is new content. Size; The replacement method includes: calculating the content popularity of D2D groups cached on user devices according to the calculation method in step S3. The D2D group content popularity of all content in the cache is used to clear the cache, starting with the content with the lowest group content popularity, until the remaining local available cache is greater than or equal to the new content to be cached. Size.
7. A mobile edge caching and recommendation method based on federated reinforcement learning according to claim 3, characterized in that, In step S7, the method for constructing the optimization objective function specifically includes the following steps: Optimize the energy consumption of user devices during content transmission in joint content caching and recommendation decision-making; system energy consumption is proportional to transmission latency; set the energy consumption generated per unit transmission time as... , indicating user equipment The transmission power and transmission delay are User equipment In the time slot Energy consumed in acquiring content The calculation method is as follows: The system optimization objective is expressed as: in It is a discount factor.
8. A mobile edge caching and recommendation method based on federated reinforcement learning according to claim 7, characterized in that, In step S8, the method for constructing a Markov decision process specifically includes the following steps: The optimization problem proposed in step S7 is modeled as a Markov decision process. The state, actions, and system reward for each user device are set as follows: For each user device… : (1) Status: In the time slot At that time, user equipment The state space is represented as ;in They represent user equipment. In the time slot Request probability and content cache status; For user equipment In the time slot The ratio of the remaining cache capacity to the total cache capacity; (2) Action: Based on state User equipment The intelligent agent in the time slot Make an action , This indicates that content is cached. And recommend a piece of content ; and These represent not caching any content and not recommending any content, respectively; settings are in time slots. Content of caching decisions It is cached in real time; (3) System rewards: Based on state and actions , In the time slot Rewards The calculation method is as follows: in, It is a hyperparameter used to control the ratio of the two parts of the reward; It is the energy loss caused by the request. It is the expected reward generated by D2D transmission; Configure user equipment In the time slot A content request was sent. The transmission latency loss of the acquired content This request generates the calculation method as follows: in, It is a positive number, representing the ratio of the upper limit of the content size range to the lower limit of the D2D link transmission rate range; when a request... When this can be achieved through D2D linking or local caching. Set to a negative value; when the request can only be obtained via a cellular link. Set to a positive value. This represents the gain in content obtained through local caching and the D2D link; at the beginning of each time slot, for the user device ,initialization =0; iterate through all content in the content library, for If cached locally Or the user equipment that comes into contact with cache Then add as follows : (4) Reconstruction and optimization objectives: Based on the construction of the Markov decision process described above, we reconstruct the optimization objective as follows: given the system in a stochastic initial state... The optimal strategy aims to reduce the energy consumption of content transmission and cellular link utilization of user equipment clusters in the system, maximize the ability of user equipment to obtain content through local caching and D2D links, and better meet user preferences. in It is a discount factor.
Citation Information
Patent Citations
File caching method and device, edge node and computer readable storage medium
CN112118295A
Intelligent collaborative content caching method
CN115208952A
