An internet of things edge collaborative caching method based on deep reinforcement learning
By deploying deep reinforcement learning models on edge servers and central servers, collaborative caching of the distributed edge caching system is achieved, which solves the problem of high computational complexity of centralized solutions, improves cache hit rate and resource utilization efficiency, and reduces latency and backhaul load.
Patent Information
- Application Number
- CN202310296228.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-24
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2043-03-24
AI Technical Summary
Most existing edge caching systems based on deep reinforcement learning adopt a centralized approach. The computational complexity increases exponentially with the number of servers, and it is difficult to effectively utilize the storage and computing resources of edge servers, making it unable to efficiently cope with edge caching environments with high-dimensional state spaces.
A distributed content caching scheme is adopted, which deploys training models on edge servers and central servers respectively, uses deep reinforcement learning algorithms for centralized model training and distributed caching operations, realizes collaborative caching between edge servers, and uses local and global gradient parameters to optimize the model and improve the cache hit rate.
By using edge-cloud collaboration, we can reduce backhaul traffic consumption, optimize caching strategies in real time, improve system cache hit rate, reduce communication overhead, and protect user information privacy.
Smart Images
Figure CN116346837B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of edge caching technology, and particularly relates to an IoT edge collaborative caching method based on deep reinforcement learning. Background Technology
[0002] With the rapid growth of internet users and the upgrading of video quality from traditional 1080P high-definition content to 4K and 8K levels, along with the emergence of complex human-computer interaction methods such as VR and AR, backbone networks are facing severe pressure. Edge caching technology can effectively reduce latency and backhaul load by placing the most popular content on servers closer to the requesting user. However, due to the limited storage space of edge servers and the changing popularity of content over time and space, edge caching technology faces many challenges.
[0003] Reinforcement learning can adapt to environmental changes without requiring any prior knowledge of the environment's dynamics. However, traditional reinforcement learning algorithms are limited to dynamic environments with fully observable low-dimensional state spaces. Real-world edge caching environments typically have high-dimensional state spaces, making it very difficult to manually extract all useful features from the environment. Deep reinforcement learning can automatically determine the optimal policy based on the original high-dimensional environment state space, effectively solving the curse of dimensionality and providing an effective solution for realistic edge caching environments.
[0004] Currently, most edge caching systems based on deep reinforcement learning use centralized content caching. However, centralized schemes require a centralized controller to collect local parameters from all servers and then generate content caching decisions for them. The computational complexity of centralized schemes increases exponentially with the number of servers. Therefore, some studies have confirmed the effectiveness of distributed content caching schemes. Summary of the Invention
[0005] Objective: This invention provides an IoT edge collaborative caching method based on deep reinforcement learning. It applies deep reinforcement learning to content popularity perception and caching decisions in edge caching, fully leveraging the adaptive capabilities of deep reinforcement learning to sensitively perceive network status, user requests, and content popularity, and respond promptly. By continuously optimizing the caching strategy through learning, it improves the cache hit rate of the edge cache, efficiently utilizes the storage and computing resources of edge servers, and reduces latency and backhaul load.
[0006] Technical Solution: To address the aforementioned technical problems, this invention proposes an IoT edge collaborative caching method based on deep reinforcement learning, which includes the following steps:
[0007] Step 1: The edge server collects video cache information from terminal layer user devices in its region and constructs a dynamic log file dataset. The dataset elements include video user ID, timestamp, and video content ID.
[0008] Step 2: Each distributed edge server trains its dataset through the model training module. The neural network input to the model training module is a dynamic log file dataset containing video user IDs, timestamps, and video content ID tags. After training, the model training module outputs local gradient parameters containing edge server caching information. Synchronously forwarded to the central server and nearby edge servers; the neural network in the distributed edge server is used to obtain local gradient parameters. This parameter reflects the cache status of the edge server;
[0009] Step 3: The central server receives the local gradient parameters sent by each distributed edge server. Then, through the parameter aggregation module, the local gradient parameters are processed. By aggregating the data, we obtain the global gradient parameters G. τ Each distributed edge server shares its local gradient parameters through a collaborative sharing module. To enable the exchange of cached information between edge servers;
[0010] Step 4: The central server inputs the fitted global gradient parameters G through the parameter training module. τ After training the neural network, the updated global model parameters ω are output. τ The central server's neural network is used to obtain the global model parameters ω. τ This parameter can further optimize the neural network in the edge server; the central server will use the global model parameter ω τ The parameters are sent to each distributed edge server for a new round of local gradient parameter processing. Global gradient parameter G τ Global model parameters ω τ Update;
[0011] Step 5: Repeat steps one through four until the prediction model converges, obtain the prediction model for online video requests from video users, and perform video content request prediction to obtain a prediction list of online video requests from users; the prediction model will be automatically updated after training.
[0012] Step Six: Based on the predicted list of user online video requests obtained from the prediction model, multiple distributed edge servers collaborate to cache the data until each distributed edge server reaches its storage limit.
[0013] Furthermore, the specific method for step one is as follows: Distributed edge server m collects video cache information i from terminal layer user devices d within its coverage area, and each distributed edge server establishes a dynamic log file dataset X based on the video cache information. m The data in the dataset is categorized by label, resulting in three categories: video user ID, timestamp, and video content ID.
[0014] Furthermore, the specific method for step two is as follows:
[0015] Step 2.1, Dataset Splitting: Divide the labeled dynamic log file dataset X into two parts. m Divided into batches with minimum batch size A subset of the training dataset, where β represents the batch size and M represents the number of edge servers;
[0016] Step 2.2, Generate the output matrix. For a DNN neural network, the distributed edge server generates the output matrix:
[0017]
[0018] in, α is the input matrix of the l-th layer of the neural network in the distributed edge server. m It is the rectified linear unit activation function in the edge server, used to transform the input of each layer of the neural network into a non-linear pattern. It defines global model parameters ω, where ω = (W, v), and W = [W1, ..., W]. l ,…,W L ] and v = [v1, ..., v l ,…,v L ], W l It is the global weight matrix, v l It is the global bias vector, and L represents the number of layers in the neural network;
[0019] Step 2.3: Calculate the prediction loss function. In the output layer, generate... The prediction loss p used to find the minimum batch iteration number τ of edge server m m (ω τ ):
[0020]
[0021] Where τ represents the iterative process of training with a small batch of samples; t represents the time to complete the iterative process τ; ω τ This represents the global model parameters during the iteration process τ; It is the input matrix of the distributed edge server. elements, It is the output matrix of the distributed edge server. Element;
[0022] Step 2.4, calculate the local gradient parameters; by calculation:
[0023]
[0024] This leads to the local gradient parameters of the distributed edge server.
[0025] Furthermore, step three involves calculating the global gradient parameter G. τ The formula is as follows:
[0026]
[0027] Furthermore, the specific steps of step four are as follows:
[0028] Step 4.1, calculate the learning step size λ, based on the neural network deployed by the parameter training module of the central server, and the global gradient parameters, and then apply η. τ and δ τ Considered as G respectively τ and The exponential moving average is used to estimate the mean in order to predict the variance at the current sample iteration τ, η. τ+1 and δ τ+1 The update formula is as follows:
[0029]
[0030]
[0031] in, and Representing η respectively τ and δ τ The exponentially decaying step size at τ is used to update the global model parameters ω. τ Add a learning step size λ to determine how to update the global model ω in each iteration τ. τ The update formula for the learning step size λ is as follows:
[0032]
[0033] Step 4.2, calculate the global model parameters ω for the next iteration τ+1. τ+1 :
[0034]
[0035] Where, ω τ+1The dataset used by the edge server to learn the next τ+1, where ε represents a constant;
[0036] Step 4.3, predict the global model parameters ω τ+1 The parameters are sent to each distributed edge server for a new round of local gradient parameter processing. Global gradient parameter G τ and global model parameters ω τ Update.
[0037] Furthermore, the specific steps of step six are as follows:
[0038] Step 6.1: Each distributed edge server interacts with a predicted list of online video requests from terminal layer users within its coverage area;
[0039] Step 6.2: Count the number of video requests based on the frequency of different online videos appearing in different users' prediction lists;
[0040] Step 6.3: Based on the number of video requests, the distributed edge servers perform collaborative caching. If the current server has cached the video, the adjacent servers will not cache the video again until each edge server reaches its storage limit.
[0041] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:
[0042] 1. This invention deploys training models on edge servers and central servers respectively, performs centralized model training and distributed caching operations on cached data based on deep reinforcement learning algorithms, and reduces backhaul traffic consumption and optimizes caching strategies in real time through edge-cloud collaboration.
[0043] 2. This invention implements collaborative edge caching, allowing distributed edge servers to collaborate and further improve the system's cache hit rate. By transmitting parameters, it achieves data sharing while protecting user information privacy and reducing communication overhead.
[0044] 3. This invention realizes self-learning caching strategy adjustment. The adaptive capability of deep reinforcement learning can realize real-time analysis of data requests and design corresponding caching strategies. As the amount of cached data accumulates, the learning ability of the deep reinforcement learning model is also improved, further improving the cache hit rate. Attached Figure Description
[0045] Figure 1 This is a general framework diagram of the IoT edge collaborative caching system and operation method based on deep reinforcement learning, which is the subject of this invention.
[0046] Figure 2 This is a diagram illustrating the central server architecture and workflow in this invention;
[0047] Figure 3 This is a diagram illustrating the edge server architecture and workflow in this invention. Detailed implementation method:
[0048] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0049] like Figure 1 The diagram shown is the overall framework of the IoT edge collaborative caching system and operation method based on deep reinforcement learning according to this invention. It illustrates the hierarchical relationship between the central server, edge servers, and terminals in this invention. Based on this, this invention proposes an IoT edge collaborative caching method based on deep reinforcement learning, which includes the following steps:
[0050] Step 1: The edge server collects video cache information from terminal layer user devices in its region and constructs a dynamic log file dataset. The dataset elements include video user ID, timestamp, and video content ID.
[0051] Step 2: Each distributed edge server trains its dataset through the model training module. The neural network input to the model training module is a dynamic log file dataset containing video user IDs, timestamps, and video content ID tags. After training, the model training module outputs local gradient parameters containing edge server caching information. Synchronously forwarded to the central server and nearby edge servers; the neural network in the distributed edge server is used to obtain local gradient parameters. This parameter reflects the cache status of the edge server;
[0052] Step 3: The central server receives the local gradient parameters sent by each distributed edge server. Then, through the parameter aggregation module, the local gradient parameters are processed. By aggregating the data, we obtain the global gradient parameters G. τ Each distributed edge server shares its local gradient parameters through a collaborative sharing module. To enable the exchange of cached information between edge servers;
[0053] Step 4: The central server inputs the fitted global gradient parameters G through the parameter training module. τ After training the neural network, the updated global model parameters ω are output. τ The central server's neural network is used to obtain the global model parameters ω. τThis parameter can further optimize the neural network in the edge server; the central server will use the global model parameter ω τ The parameters are sent to each distributed edge server for a new round of local gradient parameter processing. Global gradient parameter G τ Global model parameters ω τ Update;
[0054] Step 5: Repeat steps one through four until the prediction model converges, obtain the prediction model for online video requests from video users, and perform video content request prediction to obtain a prediction list of online video requests from users; the prediction model will be automatically updated after training.
[0055] Step Six: Based on the predicted list of user online video requests obtained from the prediction model, multiple distributed edge servers collaborate to cache the data until each distributed edge server reaches its storage limit.
[0056] Furthermore, the specific method for step one is as follows: Distributed edge server m collects video cache information i from terminal layer user devices d within its coverage area, and each distributed edge server establishes a dynamic log file dataset X based on the video cache information. m The data in the dataset is categorized by label, resulting in three categories: video user ID, timestamp, and video content ID.
[0057] Furthermore, the specific method for step two is as follows:
[0058] Step 2.1, Dataset Splitting: Divide the labeled dynamic log file dataset X into two parts. m Divided into batches with minimum batch size A subset of the training dataset, where β represents the batch size and M represents the number of edge servers;
[0059] Step 2.2, Generate the output matrix. For a DNN neural network, the distributed edge server generates the output matrix:
[0060]
[0061] in, α is the input matrix of the l-th layer of the neural network in the distributed edge server. m It is the rectified linear unit activation function in the edge server, used to transform the input of each layer of the neural network into a non-linear pattern. It defines global model parameters ω, where ω = (W, v), and W = [W1, ..., W]. l ,…,W L ] and v = [v1, ..., v l ,…,v L ], W lIt is the global weight matrix, v l It is the global bias vector, and L represents the number of layers in the neural network;
[0062] Step 2.3: Calculate the prediction loss function. In the output layer, generate... The prediction loss p used to find the minimum batch iteration number τ of edge server m m (ω τ ):
[0063]
[0064] Where τ represents the iterative process of training with a small batch of samples; t represents the time to complete the iterative process τ; ω τ This represents the global model parameters during the iteration process τ; It is the input matrix of the distributed edge server. elements, It is the output matrix of the distributed edge server. Element;
[0065] Step 2.4, calculate the local gradient parameters; by calculation:
[0066]
[0067] This leads to the local gradient parameters of the distributed edge server.
[0068] Furthermore, step three involves calculating the global gradient parameter G. τ The formula is as follows:
[0069]
[0070] Furthermore, the specific steps of step four are as follows:
[0071] Step 4.1, calculate the learning step size λ, based on the neural network deployed by the parameter training module of the central server, and the global gradient parameters, and then apply η. τ and δ τ Considered as G respectively τ and The exponential moving average is used to estimate the mean in order to predict the variance at the current sample iteration τ, η. τ+1 and δ τ+1 The update formula is as follows:
[0072]
[0073]
[0074] in, and Representing η respectively τ and δ τ The exponentially decaying step size at τ is used to update the global model parameters ω. τ Add a learning step size λ to determine how to update the global model ω in each iteration τ. τ The update formula for the learning step size λ is as follows:
[0075]
[0076] Step 4.2, calculate the global model parameters ω for the next iteration τ+1. τ+1 :
[0077]
[0078] Where, ω τ+1 The dataset used by the edge server to learn the next τ+1, where ε represents a constant;
[0079] Step 4.3, predict the global model parameters ω τ+1 The parameters are sent to each distributed edge server for a new round of local gradient parameter processing. Global gradient parameter G τ and global model parameters ω τ Update.
[0080] Furthermore, the specific steps of step six are as follows:
[0081] Step 6.1: Each distributed edge server interacts with a predicted list of online video requests from terminal layer users within its coverage area;
[0082] Step 6.2: Count the number of video requests based on the frequency of different online videos appearing in different users' prediction lists;
[0083] Step 6.3: Based on the number of video requests, the distributed edge servers perform collaborative caching. If the current server has cached the video, the adjacent servers will not cache the video again until each edge server reaches its storage limit.
[0084] like Figure 2 As shown, the central server includes a parameter aggregation module, a parameter training module, and a cache state global view module. The parameter aggregation module aggregates the local gradient parameters sent by each distributed edge server. Obtain the global gradient parameter G τ The parameter training module is based on the global gradient parameter G. τ The central server will continue to calculate the global model parameters ω for the next iteration. τThis information is then sent to each distributed edge server. Each distributed edge server will then determine the appropriate parameters based on the global model parameter ω. τ This allows for adjustments to subsequent caching strategies; the global cache status view module provides an overview of the caching status of all edge servers.
[0085] like Figure 3 As shown, the edge server includes a model training module, a collaborative sharing module, and a reward module. Each distributed edge server trains its dataset through the model training module. The neural network input to the model training module is a dynamic log file dataset containing video user IDs, timestamps, and video content ID tags. After training, the model training module outputs local gradient parameters containing cached information from the edge servers. Synchronously forwarded to the central server and nearby edge servers; the neural network in the distributed edge server is used to obtain local gradient parameters. This parameter reflects the cache status of the edge server; the collaborative sharing module is used to receive cache information from other edge servers, and then, based on local data requests, combine it with the global model parameter ω issued by the central server. τ The model training module helps make better caching decisions; the reward module is used to calculate the reward value for caching operations. Data cached in both local and neighboring servers contributes to the system hit rate. Therefore, the reward is set as a weighted sum of the two to promote cooperation between edge servers.
Claims
1. A collaborative edge caching method for the Internet of Things based on deep reinforcement learning, characterized in that, The method includes the following steps: Step 1: The edge server collects video cache information from terminal layer user devices in its region and constructs a dynamic log file dataset. The dataset elements include video user ID, timestamp, and video content ID. Step 2: Each distributed edge server trains its dataset through the model training module. The neural network input to the model training module is a dynamic log file dataset containing video user IDs, timestamps, and video content ID tags. After training, the model training module outputs local gradient parameters containing edge server caching information. Synchronously forwarded to the central server and adjacent edge servers; the neural network in the distributed edge server is used to obtain local gradient parameters. This parameter reflects the cache status of the edge server; Step 3: The central server receives the local gradient parameters sent by each distributed edge server. Then, the local gradient parameters are processed by the parameter aggregation module. By aggregating the data, we obtain the global gradient parameters G. τ Each distributed edge server shares its local gradient parameters through a collaborative sharing module. To enable the exchange of cached information between edge servers; Step 4: The central server inputs the fitted global gradient parameters G through the parameter training module. τ After training the neural network, the updated global model parameters ω are output. τ The central server's neural network is used to obtain the global model parameters ω. τ This parameter further optimizes the neural network in the edge server; the central server will use the global model parameter ω τ The parameters are sent to each distributed edge server for a new round of local gradient parameter processing. Global gradient parameter G τ Global model parameters ω τ Update; Step 5: Repeat steps 1 to 4 until the prediction model converges, obtain the prediction model for online video requests from video users, and perform video content request prediction to obtain a prediction list of online video requests from users; the prediction model after training is automatically updated. Step 6: Based on the predicted list of user online video requests obtained from the prediction model, multiple distributed edge servers collaborate to cache the data until each distributed edge server reaches its storage limit. The specific method for step two is as follows: Step 2.1, Dataset Splitting: Divide the labeled dynamic log file dataset X into two parts. m Divided into batches with minimum batch size A subset of the training dataset, where β represents the batch size and M represents the number of edge servers; Step 2.2, Generate the output matrix. For a DNN neural network, the distributed edge server generates the output matrix: in, α is the input matrix of the l-th layer of the neural network in the distributed edge server. m It is the rectified linear unit activation function in the edge server, used to transform the input of each layer of the neural network into a non-linear pattern. It defines global model parameters ω, where ω = (W, v), and W = [W1, ..., W]. l ,…,W L ] and v = [v1, ..., v l ,…,v L ], W l It is the global weight matrix, v l It is the global bias vector, and L represents the number of layers in the neural network; Step 2.3: Calculate the prediction loss function and generate it in the output layer. The prediction loss p used to find the minimum batch iteration number τ of edge server m m (ω τ ): Where, ω τ This represents the global model parameters at the point where the minimum batch iteration count τ is completed; It is the input matrix of the distributed edge server. elements, It is the output matrix of the distributed edge server. Element; Step 2.4, calculate the local gradient parameters; by calculation: This leads to the local gradient parameters of the distributed edge server. Step 3: Calculate the global gradient parameter G τ The formula is as follows: The specific steps for step four are as follows: Step 4.1, calculate the learning step size λ, based on the neural network deployed by the parameter training module of the central server, and the global gradient parameters, and then apply η. τ and δ τ Considered as G respectively τ and The exponential moving average is used to estimate the mean in order to predict the variance at the current sample iteration τ, η. τ+1 and δ τ+1 The update formula is as follows: in, and Representing η respectively τ and δ τ The exponentially decaying step size at τ is used to update the global model parameters ω. τ Add a learning step size λ to determine the update of the global model ω in each iteration τ. τ The update formula for the learning step size λ is as follows: Step 4.2, calculate the global model parameters ω for the next iteration τ+1. τ+1 : Where, ω τ+1 The dataset used by the edge server to learn the next τ+1, where ε represents a constant; Step 4.3, predict the global model parameters ω τ+1 The parameters are sent to each distributed edge server for a new round of local gradient parameter processing. Global gradient parameter G τ and global model parameters ω τ Update.
2. The IoT edge collaborative caching method based on deep reinforcement learning according to claim 1, characterized in that, The specific method for step one is as follows: Distributed edge server m collects video cache information i from terminal layer user devices d within its coverage area, and each distributed edge server establishes a dynamic log file dataset X based on the video cache information. m The data in the dataset is categorized by label, resulting in three categories: video user ID, timestamp, and video content ID.
3. The IoT edge collaborative caching method based on deep reinforcement learning according to claim 1, characterized in that, The specific steps for step six are as follows: Step 6.1: Each distributed edge server interacts with a predicted list of online video requests from terminal layer users within its coverage area; Step 6.2: Count the number of video requests based on the frequency of different online videos appearing in different users' prediction lists; Step 6.3: Based on the number of video requests, the distributed edge servers perform collaborative caching. If the current server has cached the video, the adjacent servers will not cache the video again until each edge server reaches its storage limit.