NDN caching strategy based on offline-online deep reinforcement learning

Abstract

The invention discloses an NDN (Named Data Networking) caching strategy based on offline-online deep reinforcement learning, which relates to the field of named data networks and comprises the following steps: adding an interest request table in a router of an NDN; extracting cached log data; dynamically distributing each attribute weight by using a self-attention mechanism; performing offline training, and extracting an optimal cache strategy; adjusting and optimizing the strategy by using an IQL model online fine adjustment module; and the on-line fine tuning module is executed periodically. According to the method, large-scale simulation is carried out for the real topology TISCALI, and compared with an existing typical caching strategy, the method can obviously improve the overall caching hit rate in the network, reduce the time delay, reduce the link stretch rate and improve the user experience.

Description

Technical Field

[0001] The present invention relates to the field of named data networks, and in particular to an NDN caching strategy based on offline-online deep reinforcement learning. Background Art

[0002] With the rapid development of the Internet and the widespread use of mobile smart devices and network technologies, the focus of Internet applications has shifted from simple text information to content acquisition including rich multimedia audio and video. The distribution of video and other high-traffic content accounts for a large part of the total Internet traffic, and this trend continues to rise. It is estimated that from the end of 2023 to the end of 2029, the total mobile data traffic will increase threefold, and this growth has brought new requirements for network architecture. Obviously, the traditional Internet architecture based on the TCP / IP protocol can no longer meet the development of future Internet applications. In order to adapt to users' demand for rapid access to information and enhance the mobility, security and scalability of the Internet architecture, researchers have proposed a series of new node-based or content-based network architectures to cope with this change.

[0003] Information-centric Networking (ICN) is one of these innovations. It changes the original end-to-end communication mechanism and focuses on the content itself rather than its specific location. Named Data Networking (NDN) is one of the most representative distributed network architectures in ICN. The network built-in cache is one of the core features of NDN, which allows users to obtain content directly from router nodes without having to obtain it from the server, greatly improving user experience and transmission efficiency while reducing server load. With the evolution of future network and cache requirements, researchers have proposed a variety of innovative cache strategies based on heuristic methods. These heuristic-based methods use the experience of domain experts on environmental characteristics to formulate cache strategies. Although these methods perform well in specific situations, they often cannot adapt to dynamic changes in the network environment. In order to overcome this problem, deep reinforcement learning (DRL) methods are introduced.

[0004] As a clever combination of reinforcement learning (RL) and deep neural networks (DNN), DRL can effectively handle the large state / action space of complex network cache systems, solve sequential decision problems without expert knowledge, and continuously adapt to environmental changes. Existing work is mainly based on online DRL development, which continuously interacts with the network cache system to train the optimal cache strategy. However, although existing work has demonstrated the potential of cache strategies developed based on online DRL, this method still has some significant limitations: First, online DRL relies on continuous interaction with the actual network environment, which not only increases the computational and time costs of the learning process, but also may seriously affect the cache performance due to experimental decisions in actual deployment; second, online learning requires a large amount of data to achieve the ideal performance level, which may lead to policy instability and performance fluctuations in the initial stage; in addition, due to the continuous changes in the network environment, maintaining a continuously updated and efficient online learning model also faces significant challenges.

[0005] Therefore, it is necessary to provide an NDN caching strategy based on offline-online deep reinforcement learning to solve the above problems. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] An NDN caching strategy based on offline-online deep reinforcement learning includes the following steps:

[0008] Add an interest request table in the router of the NDN network to collect the name of the interest packet and the corresponding request quantity for subsequent popularity calculation of each content;

[0009] Extract cache log data of the NDN network, including four attribute information: node importance, edge degree, idleness, and content popularity, for offline training of subsequent models;

[0010] Use the self-attention mechanism to dynamically assign the weight of each attribute. The self-attention mechanism automatically adjusts the weight of each attribute according to the timely characteristics of the state attributes, achieves rapid response to changes in network status, and ensures the rationality of the cache location;

[0011] Use the IQL model offline training module to perform offline training on the data and extract the optimal caching strategy;

[0012] Deploy the cache strategy after offline training to the actual NDN network and use the IQL model online fine-tuning module to tune the strategy;

[0013] The online fine-tuning module is executed periodically to adapt to the dynamic changes of the network environment.

[0014] Preferably, the IQL offline training steps are:

[0015] Create a Q function neural network and randomly initialize the parameters θ;

[0016] Create a goal Function neural network and initialize parameters

[0017] Randomly sample a mini-batch of interactions from the dataset and calculate the value function V(ψ);

[0018] Update the Q function neural network;

[0019] Reset target every P steps Functional Neural Network

[0020] Repeat the above until convergence or the maximum number of training times is reached, and extract the final strategy.

[0021] Preferably, the IQL online fine-tuning module specifically tunes the strategy as follows:

[0022] Initialize experience replay pool B and environment;

[0023] Load the Q function neural network parameters θ trained offline;

[0024] Load the target Q function neural network parameters trained offline

[0025] For each time step, an action a is selected using the ∈-greedy algorithm t And execute, get reward r from the environment t and the next state s t+1 , the new online interaction (s t， a t， r t， s t+1 ) is stored in B;

[0026] Randomly sample a mini-batch of interactions from B, update the Q function neural network, and reset the target every P steps Functional Neural Network

[0027] Repeat the above until convergence or the maximum number of training times is reached; extract the final strategy.

[0028] Beneficial effects of the present invention: The present invention analyzes the cache log data of the NDN network for offline training, thereby learning the optimal cache decision strategy, which comprehensively considers multi-dimensional state attributes such as node importance, edge degree, idleness and content popularity, and uses a self-attention mechanism to dynamically allocate weights to optimize the efficiency and effectiveness of cache decisions; after offline training, the strategy is deployed in the actual NDN network and fine-tuned online to adapt to the dynamic changes of the network environment; the present invention conducts large-scale simulations on the real topology TISCALI. Compared with existing typical cache strategies, the present invention can significantly improve the overall cache hit rate in the network, reduce latency, reduce link stretching rate and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a comparison chart of cache hit rate variation trends between the present invention and other methods.

[0030] Figure 2 This is a comparison chart of user delay variation trends between the present invention and other methods.

[0031] Figure 3 This is a comparison chart of link stretching rate variation trends between the present invention and other methods. DETAILED DESCRIPTION

[0032] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0033] like Figures 1 to 3 As shown in the figure, an NDN caching strategy based on offline-online deep reinforcement learning includes the following detailed steps:

[0034] An interest request table is added to the router of the NDN network to collect the name of the interest packet and the corresponding request quantity for subsequent popularity calculation of each content. On the basis of the original content storage table, pending interest table and forwarding information table, an interest request table IRT is added to each edge node to collect the name and request quantity of each interest packet to facilitate the calculation of the popularity of each content;

[0035] As shown in IRT Table 1, the first column is the name of the content requested by the passing Interest packets, and the second column is the number of Interest packets;

[0036] IRT Table 1

[0037] Content Name Number of requests / net / news / A.mp4 30 / net / news / B.mp4 25 / net / news / C.mp4 18

[0038] Extract cache log data of the NDN network, including four attribute information: node importance, edge degree, idleness, and content popularity, for offline training of subsequent models. The calculation method of each attribute is as follows:

[0039] 1. Node importance: In the network topology, different nodes have different importance. Some nodes are in the "central position" of the network. They can connect to more other nodes and may have more requests passing through these nodes. Node importance can reflect the degree of demand for different content at different cache nodes. When different centrality metrics are used to evaluate the importance of nodes in the same network, different results are often obtained due to different centrality metrics having different mechanisms and different defects. Therefore, we integrate four different centrality metrics to measure the importance of node nB n :

[0040] B n=β1DC(n)+β2CC(n)+β3BC(n)+β4EC(n)#(1)#

[0041] Among them, DC(n) represents degree centrality, CC(n) represents closeness centrality, BC(n) represents betweenness centrality, and EC(n) represents eigenvector centrality. β1, β2, β3, and β4 are the weights of each centrality.

[0042] The normalized node importance is expressed as:

[0043]

[0044] Here, B max Indicates the maximum node importance among all nodes on the path.

[0045] 2. Node edge degree: When making caching decisions, the concept of node edge degree is introduced. The closer the node is to the user, the greater the node edge degree. If the content is cached in a node with a larger edge degree, the user will take less time to obtain the content, which can provide a better experience for the user. The edge degree of node n is defined as L n , represents the number of hops between the node and the content server. Normalize the edge degree of node n, where L max Represents the maximum edge degree among all nodes on the path.

[0046]

[0047] 3. Idle rate: We use the number of replacements in a cycle to represent the replacement frequency I of node n n , I max Indicates the maximum number of replacements in a cycle, normalized and represented by i n Indicates the replacement rate. The idle rate is inversely proportional to the replacement rate. The idle rate reflects the idleness of the node. Although some nodes are very important, the accumulation of a large amount of content may cause the node to be overloaded and the replacement rate to be too high, resulting in the loss of popular content. We use -in +1 represents the idle rate of node n, and normalizes it, where Y max Indicates the maximum idle rate of node n during the period.

[0048]

[0049] 4. Content popularity: The popularity of edge content can better reflect the real needs of users. Based on the number of user requests for content, the content popularity will be calculated on the management server. max Indicates the maximum popularity of the content in the current period. In order to avoid the contingency of a period and reduce the impact of historical period data on the popularity of current content, we use the EWMA (exponentially weighted moving average) model to calculate content popularity:

[0050]

[0051] in is the number of requests for content c in the current period, μ is the weight parameter, After normalization, the content popularity is expressed as:

[0052]

[0053] The self-attention mechanism is used to dynamically assign the weight of each attribute. The self-attention mechanism automatically adjusts the weight of each attribute according to the timely characteristics of the state attributes, achieves rapid response to changes in network status, and ensures the rationality of the cache location. In order to more accurately determine the weight of each centrality, we use the self-attention mechanism to dynamically calculate the weight values ​​β1, β2, β3, and β4.

[0054]

[0055] Among them, Q i With K i are the query vector and the key vector respectively, and the dot product between them is used to represent the similarity between the query vector and the key vector. k is the dimension of the key vector, which is scaled to prevent the gradient vanishing or exploding problem of the softmax function.

[0056] Use the IQL model offline training module to perform offline training on the data and extract the optimal caching strategy; the IQL model is an improved Q-learning method that reduces the reliance on explicit representation of the Q function by introducing implicit policy improvements. Unlike traditional Q-learning, IQL does not directly estimate the Q function, but indirectly approximates the Q value through policy optimization, thereby improving the efficiency and stability of the algorithm. In addition, due to IQL's powerful offline performance and the introduction of weighted behavioral cloning strategies, the IQL model is very suitable for online fine-tuning. In order to improve the performance of the caching strategy, the state space, action space, and rewards based on IQL are used to model and solve;

[0057] Three basic elements are introduced:

[0058] State space: The state is information describing the environment. It contains the current situation and characteristics of the environment in which the agent is currently located. In IQL, the state is used to represent the current state of the problem and is the input information that the algorithm needs to learn. In the cache decision problem, the state can include indicators such as the cache status of the node for all content, the importance of the node, the edge degree of the node, the idle rate of the node, and the popularity of the content.

[0059] Action Space: Actions are actions that the agent can perform in a certain state. In reinforcement learning, the agent influences the environment by choosing different actions and adopts different strategies in different states. In the cache decision problem, the action can be whether to cache this content on this node.

[0060] Reward: Reward is the feedback signal returned by the environment after the agent performs an action. Reward is used to evaluate the quality of the action and is the goal of agent learning. Reward signal guides the agent to learn in the direction of maximizing long-term cumulative rewards. In the selection of cache nodes, we use cache hit rate as reward.

[0061] The cache strategy after offline training is deployed to the actual NDN network, and the strategy is tuned using the IQL model online fine-tuning module; different nodes have different importance, and different content has different popularity. Popular content cached on nodes with high node importance can better meet the requests of global users, and nodes close to users can also respond to user requests faster, thereby greatly reducing latency. At the same time, we should also consider the "busyness" of the node. Therefore, our goal is to cache content with higher popularity on nodes with high node importance, node edge degree, and idle rate, and cache content with lower popularity on nodes with low node importance, node edge degree, and idle rate. In this way, we can optimize the location of the cache, and each content can be cached in their relatively optimal location. Taking the content propagation path as the optimization unit, periodic matching decisions are made for the nodes and content on the path. The cache decision problem can be expressed as:

[0062]

[0063] C1 ensures that each request can be satisfied by at least one cache unit. C2 indicates that the space size of all content cached by each routing node n will not exceed the maximum cache space size of the node. C3 Restricted to binary.

[0064] To solve the NDN network caching problem with offline-online DRL, we model it as an MDP. The state space, action space, and reward function are defined as follows:

[0065] State space: In this paper, the caching decision mainly depends on the node attributes and content popularity of the node. Therefore, we record the state of node n at time t as Among them, X n，t is a vector, indicating the cache status of all content at node n at time t; B n , L n and They represent the node importance, node edge degree and idle rate of node n at time t respectively; Represents the popularity of content c at time t.

[0066] Action space: At time t, when node n receives a content-related data packet, it will perform a cache operation on it, that is, whether to cache this content. We define this process as Indicates that at time t, content c is cached on node n. This means no caching.

[0067] Reward function: Our goal is to cache content with higher popularity on nodes with high node importance, node edge and idle rate, and cache content with lower popularity on nodes with low node importance, node edge and idle rate. Therefore, the reward function at time t is defined as:

[0068]

[0069] Afterwards, we adopt the IQL algorithm to achieve our goal.

[0070] First, we need to learn an independent value function V(ψ) that accurately estimates the expected reward in a given state, regardless of the action.

[0071]

[0072] Where D is a static data set, is the Q value in the target network. is the loss function used for expectileregression, and u is the residual;

[0073] Afterwards, we will train the obtained V ψ (s′) inserts the loss function L Q (θ), thereby updating the Q function.

[0074]

[0075] Since IQL does not clearly indicate the most appropriate strategy, the advantage weighted regression method (AWR) is used to extract the final strategy.

[0076]

[0077] The details of the offline training process are shown in Algorithm 1. Where θ is the Q function neural network parameter. The static dataset is generated by the behavior policy π based on the heuristic method. β After that, we repeatedly train according to formula (11) and formula (12) until the Q function converges and extracts the final strategy. In this process, the target The function will be updated every P steps according to the parameters of the Q function.

[0078]

[0079] The details of the online fine-tuning process in step (5) are shown in Algorithm 2. The cache strategy trained offline is deployed to the actual network and fine-tuned through interaction with the environment to adapt to the dynamically changing network environment.

[0080]

[0081]

[0082] The online fine-tuning module is executed periodically to achieve dynamic changes in the cache strategy to adapt to the dynamic changes in the network environment and maintain the long-term good performance of the cache strategy.

[0083] In order to improve the cache utilization of nodes and reduce latency and node pressure, the goal of the present invention is to cache content with higher popularity on nodes with high node importance, node marginality and idle rate, and cache content with lower popularity on nodes with low node importance, node marginality and idle rate;

[0084] Example:

[0085] In order to truly simulate the NDN network, the present invention uses the ndnSIM simulator to simulate the TISCALI network topology with 44 server nodes, 36 user nodes and 160 router nodes. In the simulation experiment, the replacement strategy of all comparative experiments adopts LRU, and the user request path is the shortest path calculated by the Dijkstra algorithm. The ratio of the cache capacity of all routing nodes to the number of content objects, and the cache capacity of the routing nodes is the same. The process of user request content obeys the Poisson distribution, and the probability of user request obeys the Zipf distribution. The default value of the Zipf parameter α is 0.8, and the value range is 0.7 to 1.1. The experimental parameters are shown in Table 1. Finally, the three evaluation criteria of cache hit rate, user delay and link stretching rate of different methods are statistically analyzed. The methods compared with others are LCE, LCD, ProbCache, CL4M, RAND and the previously proposed cache strategy FACS based on the improved firefly algorithm.

[0086] 1. Cache hit rate

[0087] Cache hit rate is one of the important indicators for evaluating NDN performance. The higher the cache hit rate, the more requested content consumers can obtain on the router node, reducing the pressure on the server. The formula is as follows:

[0088]

[0089] Among them, N hit Represents the number of cache hits in the entire network, N server Indicates the number of times the origin server served the content.

[0090] Figure 1 The effect of S and Zipf parameter α on the average cache hit rate is shown. As the cache capacity increases, the cache hit rate of each strategy also increases. COORL's cache hit rate is always better than other strategies. This is because COORL not only takes into account the node status factors and content popularity, but also COORL's continuous learning ability enables the cache strategy to be dynamically adjusted, thereby maintaining high performance in a changing network environment. Figure 1In the data, when the parameter S = 0.25, the cache hit rate of COORL is 40.3%, which is 8.9% higher than that of FACS. Although FACS takes into account the node status factors and content popularity, due to the lack of a long-term reward mechanism, its adaptability and dynamic adjustment capabilities are weak, and it is difficult to cope with dynamically changing network request patterns. Although CL4M utilizes node centrality, it also leads to a high replacement rate and the loss of popular content, so the cache hit rate of CL4M is very low. The higher the Zipf parameter, the more concentrated the popular content tends to be, and the popularity of hot content in the network is increasing. Therefore, as the Zipf parameter α increases, the average click rate is also increasing. COORL has the best performance because when the probability of users requesting highly popular content continues to increase, COORL's advantages are further amplified, and more popular content is cached on more reasonable nodes.

[0091] 2. User Delay

[0092] Latency represents the average delay required for a requesting node in the network to obtain content. The formula is as follows:

[0093]

[0094] Among them, d i is the total time from when node i sends request to when it receives the corresponding content, and N represents the total number of requests in the network.

[0095] Figure 2 The effect of S and the Zipf parameter α on the average latency is shown. As the cache capacity increases, the average latency of all strategies decreases, among which COORL always has the lowest latency. This is because COORL tries to cache popular content as close to the user as possible so that the user can get the content faster. Figure 2 In the example, when the parameter S=0.25, the average delay of the scheme is 62.16ms, which is 4.9% lower than FACS. As the Zipf parameter α increases, the delay time of each cache strategy decreases, and the performance of COORL is still the best. When the parameter α increases, the popular content of the user edge cache increases and the delay decreases.

[0096] 3. Link stretching rate

[0097] We use the business path stretching rate to represent the length of the hit path. The formula is as follows:

[0098]

[0099] Where N is the total number of requests in the network, path i，s，r is the path length of the request content i from the business node s to the request node r, path i，o，r is the path length from the original source node o to the request node r.

[0100] Figure 3 The effect of the ratio S and the Zipf parameter α on the average service path stretch ratio is shown. As the cache capacity increases, the average service path stretch ratio of each strategy decreases, and COORL can always maintain the lowest delay. Because COORL takes the distance factor into account, it greatly shortens the distance for users to obtain popular content. Figure 3 When the parameter S = 0.25, the average path extension ratio of the scheme is 0.66, which is 4.3% lower than that of FACS. When the Zipf parameter α varies between 0.7 and 1.1, the path stretching ratio performance of our scheme is always the best.

[0101] It is known from common technical knowledge that the present invention can be implemented by other embodiments that do not deviate from its spirit or essential features. Therefore, the above disclosed embodiments are only illustrative in all respects and are not exclusive. All changes within the scope of the present invention or within the scope equivalent to the present invention are included in the present invention.

Claims

1. An NDN caching strategy based on offline-online deep reinforcement learning, characterized in that: The following steps are involved: Add interest request table in routers of NDN network; Extract cache log data of NDN network; Use the self-attention mechanism to dynamically assign weights to each attribute; Use the IQL model offline training module to perform offline training on the data and extract the optimal caching strategy; Deploy the cache strategy after offline training to the actual NDN network and use the IQL model online fine-tuning module to tune the strategy; The online fine-tuning module is executed periodically to adapt to the dynamic changes of the network environment.

2. According to the NDN cache strategy based on offline-online deep reinforcement learning as described in claim 1, it is characterized by: The interest request table is added in the router of the NDN network, specifically to collect the name of the interest packet and the corresponding request quantity, which is used for the subsequent popularity calculation of each content.

3. According to the NDN cache strategy based on offline-online deep reinforcement learning as described in claim 1, it is characterized by: The extracted cache log data of the NDN network includes four attribute information: node importance, edge degree, idleness and content popularity, so as to be used for offline training of subsequent models.

4. According to the NDN cache strategy based on offline-online deep reinforcement learning as described in claim 1, it is characterized by: The self-attention mechanism is used to dynamically allocate the weights of each attribute. The self-attention mechanism automatically adjusts the weights of each attribute according to the timely characteristics of the state attributes, achieves rapid response to changes in network status, and ensures the rationality of the cache location.

5. According to the NDN cache strategy based on offline-online deep reinforcement learning as described in claim 1, it is characterized by: The IQL offline training steps are: Create a Q function neural network and randomly initialize the parameters θ; Create a goal Function neural network and initialize parameters Randomly sample a mini-batch of interactions from the dataset and calculate the value function V(ψ); Update the Q function neural network; Reset target every P steps Functional Neural Network Repeat the above until convergence or the maximum number of training times is reached, and extract the final strategy.

6. According to the NDN cache strategy based on offline-online deep reinforcement learning as described in claim 1, it is characterized by: The IQL online fine-tuning module specifically tunes the strategy as follows: Initialize experience replay pool B and environment; Load the Q function neural network parameters θ trained offline; Load the target Q function neural network parameters trained offline For each time step, an action a is selected using the ∈-greedy algorithm t And execute, get the reward r and next state s from the environment t+1 , the new online interaction (s t , a t , r t ,s t+1 ) is stored in B; Randomly sample a mini-batch of interactions from B, update the Q function neural network, and reset the target every P steps Functional Neural Network Repeat the above until convergence or the maximum number of training times is reached; extract the final strategy.

Images