A link aggregation node caching method for differentiated QoS assurance

Through deep reinforcement learning technology, dynamically adjusting the cache capacity of link aggregation nodes, the problem of failure to effectively provide differentiated QoS guarantees in the existing technology is solved, and the transmission delay, throughput and reduction are achieved while reducing hardware costs. The effect of packet loss rate.

CN118400337BActive Publication Date: 2024-11-29NANJING UNIV OF INFORMATION SCI & TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410866271.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-01
Publication Date
2024-11-29
Estimated Expiration
2044-07-01

AI Technical Summary

Technical Problem

The existing cache design method has failed to effectively solve the problem of differentiated services in network services and cannot provide differentiated QoS guarantees under the trend of network scale expansion and personalized business development.

Method used

Using deep reinforcement learning (DQN) technology, network services are divided into delay-sensitive, bandwidth-sensitive and reliability-sensitive services. The cache capacity of link aggregation nodes is dynamically adjusted based on the current queue length, dequeue rate and queuing delay, and through the main Q network and target Q network train agents, optimize cache strategies to achieve differentiated QoS guarantees.

Benefits of technology

In NS3 network topology simulation, compared with FIFO, RED, and CoDel algorithms, the DQN cache design strategy reduces the average transmission delay by 2.7%, improves throughput by 1.38%, and reduces the packet loss rate by 0.9%, providing differentiated services QoS guarantee.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118400337B_ABST
    Figure CN118400337B_ABST
Patent Text Reader

Abstract

The present invention discloses a link aggregation node caching method for differentiated QoS guarantee. First, differentiated services are classified; then a training data set is constructed, and then a DQN cache adjustment neural network is constructed. The cache space of the aggregation node is used as the environment, and the current queue length, dequeue rate, and current queuing delay are used as the state space. The action space is defined as increasing or decreasing the cache capacity, and the total reward is set. The action that maximizes the reward function in the current state is selected, and the DQN cache adjustment neural network is trained: the main Q network is used to evaluate the current strategy, calculate the loss between the Q value of the action in the current state and the target Q value, and continuously update the main Q network to obtain a link aggregation node cache model for differentiated QoS guarantee; output cache strategy. This invention can effectively improve the performance of the transmission delay, throughput, and packet loss rate of the active queue management algorithm, thereby providing technical support for the QoS guarantee of differentiated services.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of information engineering, and in particular relates to a link aggregation node caching method oriented to differentiated QoS guarantee. Background Art

[0002] Cache design refers to designing the maximum / minimum value of cache capacity under the requirements of different network performance boundaries. As a key network device of a communication network node, the performance of an Ethernet switch or router directly affects the overall performance of the network, and thus affects the user's communication experience. Cache design technology is widely used in switching devices based on store-and-forward, such as routers and switches. In the store-and-forward mechanism, when the output port flow is controlled or the received data frame fails to be dispatched to the sending port in time, the data frame needs to be stored. Therefore, the cache capacity directly affects the packet loss rate and latency. If the cache capacity is too large, the packet loss rate of the queue can meet the requirements, but the latency cannot be guaranteed; conversely, if the cache capacity is too small, the packet latency can meet the requirements, but the packet loss rate may not be guaranteed. The current cache design methods mainly include two categories: non-adaptive cache design and dynamic adaptive cache design.

[0003] Among them, non-adaptive cache design mainly includes empirical rules, tiny buffer rules, buffer rules based on packet loss rate, and cache design rules based on queuing theory. Specifically, the empirical rule sets the buffer size of the node to be slightly larger than the bandwidth-delay product (BDP) of the network, that is, B=RTT*C. The main basis of this rule is to maximize the bandwidth utilization of the network interface. The tiny buffer rule is to set the buffer size to ,in W Represents the average TCP window size. This method reduces the cache size by sacrificing some link utilization (about 10~15%), but the premise is that the aggregated traffic of the core bottleneck link follows the Poisson distribution. The cache rule based on packet loss rate designs the buffer size according to the change of packet loss rate. The goal of this method is to make full use of the bottleneck bandwidth while ensuring that the packet loss rate does not exceed a given upper limit. The cache design based on queuing theory determines the buffer size based on traffic characteristics, that is, given an input traffic model and service model, the queue length distribution is analyzed using queuing theory to obtain a suitable buffer size.

[0004] Most of the current cache design methods are based on strictly assumed mathematical models. However, due to the time-varying and complexity of network traffic characteristics, it is difficult to find an accurate mathematical model to describe the behavioral characteristics of the network. For this reason, dynamic adaptive cache design has emerged. A dynamic adaptive cache design algorithm ADT (Adaptive Drop-Tail) in the prior art measures the utilization level of the link in real time, dynamically changes the available cache threshold according to the relationship between the current utilization level and the target utilization, and minimizes the queuing delay while ensuring high utilization. ADT changes the arrival characteristics of traffic by feeding back the changes in available cache space to the source end. The key to the establishment of this method is that the arrival of traffic is elastic. However, the adjustment of queue parameters may cause queue oscillation or other instability. A dynamic adaptive cache design strategy in the prior art continuously updates and corrects the cache size according to the transmission time of the incoming event, thereby gaining appropriate reordering time for each incoming event and alleviating the problem of out-of-order transmission. Simulation results show that compared with static cache design, the dynamic cache design method can reduce transmission delay and process more out-of-order data. In addition, its dynamic characteristics can cope with sudden changes in the network environment. To address the problem of excessive residence time of low-latency services caused by static cache exhaustion in 5G scenarios, the 5G-BDP algorithm in the prior art is used. Simulation verification shows that the proposed algorithm reduces the latency to the minimum (9.12ms-9.43ms) while maximizing link utilization (0.99947-0.99949).

[0005] In summary, the non-adaptive cache design method can effectively reduce hardware costs, and the adaptive cache design method has a stronger ability to adapt to network conditions. However, with the expansion of network scale and the personalized development trend of network services, how to achieve differentiated services for network services is an issue worthy of attention. Summary of the invention

[0006] The present invention aims at the problem that neither the current non-adaptive cache design method nor the dynamic adaptive cache design method has ever considered the differentiated service problem of network services. The present invention provides a link aggregation node method for differentiated QoS guarantee, divides network services into delay-sensitive services, bandwidth-sensitive services, and reliability-sensitive services, and realizes dynamic adjustment of link aggregation node cache under three types of services based on DQN (Deep Q-Learning Network, deep reinforcement learning), and utilizes the dynamic adjustment of the cache of network traffic aggregation nodes such as routers to enhance the differentiated service capability of the aggregation node for network services.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: a link aggregation node caching method for differentiated QoS guarantee, comprising the following steps:

[0008] S1. Differentiate network services into three types: delay-sensitive services, bandwidth-sensitive services, and reliability-sensitive services.

[0009] S2. Construct a dynamic adjustment strategy for the DQN cache: For the three types of services, the current queue length, dequeue rate, and queuing delay in the aggregation node cache are used as the basis for dynamic adjustment of the DQN cache, and the corresponding dynamic adjustment results are obtained: dynamically increase or decrease the cache space.

[0010] S3, constructing an intelligent agent for the DQN cache dynamic adjustment strategy, which includes a main Q network and a target Q network;

[0011] The cache space of the sink node is used as the environment of the DQN agent, and the current queue length, dequeue rate, and current queuing delay in the cache space are used as the state space for the training of the DQN agent. The action space of the DQN agent is defined as: increasing the cache capacity and reducing the cache capacity; based on differentiated business classification, the reward function is divided into delay reward, throughput reward, and packet loss rate reward, and the total reward is obtained by weighted summation; using The strategy selects the action that maximizes the reward function in the current state;

[0012] Using DQN cache to dynamically adjust the strategy training agent: the main Q network is used to evaluate the current strategy, and the target Q network stabilizes the training process by providing a stable target Q value. In the training process, the experience replay mechanism is used to calculate the Q value of all actions in the next state using the target Q network, and the maximum Q value is selected to calculate the target Q value. The main Q network is used to calculate the Q value of the action in the current state, and the loss between the Q value of the action in the current state and the target Q value is calculated. Then the main Q network is continuously updated until the preset number of training times is reached, and a link aggregation node cache model for differentiated QoS assurance is obtained;

[0013] S4. Taking different types of network services as input, using the link aggregation node cache model for differentiated QoS guarantee, output the cache strategy of the corresponding network services.

[0014] Furthermore, the aforementioned step S2 includes the following sub-steps:

[0015] S201: Cache capacity of aggregation node B Initialize;

[0016] S202, executing action 0 or action 1 according to the current queue length, dequeue rate, and queue delay status;

[0017] S203: If action 0 is executed, the capacity of one data packet is increased based on the current cache space. In order to avoid excessive cache capacity, when the capacity is increased to the upper limit of the cache capacity, U When , stop increasing the cache space;

[0018] S204: If action 1 is executed, the capacity of one data packet is reduced based on the current cache space. In order to avoid the cache capacity being 0, when it is reduced to the lower limit of the cache capacity, L When , stop reducing the cache space;

[0019] S205: Determine whether there is a queue in the current aggregation node cache, if yes, return to step S202, otherwise end the dynamic adjustment.

[0020] Furthermore, in the aforementioned agent of the DQN cache dynamic adjustment strategy, the main Q network and the target Q network have the same structure; the main Q network and the target Q network both include an input layer, a hidden layer, a fully connected layer, and an output layer:

[0021] The input layer is used to receive input status data. The input layer includes three input units, which correspond to the three states of the current queue length, the dequeue rate, and the queuing delay in the sink node buffer area respectively;

[0022] The hidden layer is used to perform nonlinear transformation and feature extraction on the input state data, and map the input state data to a higher dimensional representation space;

[0023] The fully connected layer is used to integrate the feature information of the hidden layer and convert the information into the output of the cache adjustment strategy; the output layer is used to convert the output into action probability. The output layer includes 2 output units and uses a softmax activation function to realize the probability output of action 0 or action 1.

[0024] Furthermore, in the aforementioned step S3, the latency reward, throughput reward, packet loss rate reward, and total reward are specifically:

[0025] Delayed Rewards:

[0026] ,(1)

[0027] in, D target Indicates the target queuing delay of the queue in the cache, D current Indicates the queue delay of the current queue, obtained through the GetSeconds() function. D current The smaller the Reward delay The closer D target ;

[0028] Throughput Rewards: , (2)

[0029] in, R dequeue Indicates the dequeue rate of the queue, R phy represents the physical bandwidth of the bottleneck link, R dequeue The closer R phy Indicates that the throughput is greater;

[0030] Packet loss rate reward: , (3)

[0031] in, packetLossRate Indicates the current packet loss rate. N dropped Indicates the number of packets currently discarded. N enq Indicates the number of data packets currently successfully enqueued;

[0032] Total Rewards:

[0033] , (4)

[0034] in, w 1. w 2. w 3 corresponds to the weights of latency reward, throughput reward, and packet loss reward respectively. clip(*) represents the clipping function, which is used to limit the reward value to (-1,1).

[0035] Furthermore, in the aforementioned step S3, The decision-making method of the strategy is: , (5)

[0036] in, a Indicates the current state s The action adjustment strategy made under m is a random number in (0,1), is the exploration rate, when When , choose the action with the largest expected value; when When , a parameter adjustment action is randomly selected. , episode is the number of training rounds, s ={ L current ,R duqueue ,D current}, Lcurrent Indicates the current queue length. R dequeue Indicates the dequeue rate, D curren t Indicates the current queue delay. is the parameter of the Q network, Q(s,a;θ) represents the Q value of the agent choosing action a in state s when the neural network parameter is θ.

[0037] Furthermore, in the aforementioned step S3, the target Q value calculation formula is as follows: (6)

[0038] in, r For the current reward, is the discount factor, indicating the importance of future rewards, a’ For the next action, s’ Indicates the next state, are the parameters of the target Q network, Indicates that the neural network parameters are , the next state state s’ When , the agent chooses the next action a’ Q value.

[0039] Furthermore, in the aforementioned step S3, the loss between the Q value of the action in the current state and the target Q value is calculated as follows, and then the parameters of the main Q network are updated by the back propagation algorithm and the gradient descent method: , in order to minimize the loss

[0040] (7).

[0041] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0042] The simulation verification of NS3 network topology shows that compared with the fixed cache strategies in the three main queue management algorithms of FIFO, RED and CoDel, the present invention, based on the DQN cache design strategy, can achieve a maximum reduction of 2.7% in average transmission delay, a maximum increase of 1.38% in throughput, and a maximum reduction of 0.9% in packet loss rate under the premise of significantly reducing the cache capacity. The above results show that the link aggregation node method for differentiated QoS guarantee proposed in the present invention can effectively improve the performance of transmission delay, throughput and packet loss rate of the active queue management algorithm under the premise of reducing hardware costs, thereby providing technical support for QoS guarantee of differentiated services. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of the present invention.

[0044] Figure 2 It is a flow chart of the cache adjustment strategy of the present invention.

[0045] Figure 3 It is a structural diagram of the deep neural network designed by the present invention.

[0046] Figure 4 It is a DQN network structure diagram designed by the present invention.

[0047] Figure 5 This is the network topology diagram in NS3.

[0048] Figure 6 This is a comparison chart of the average transmission delay performance of the fixed cache and DQN cache models.

[0049] Figure 7 This is a comparison chart of the average throughput performance of the fixed cache and DQN cache models.

[0050] Figure 8 This is a comparison chart of the packet loss rate performance of the fixed cache and DQN cache models.

[0051] Fig. 9 This is a graph showing the changing relationship between the cache boundary and queue length under the DQN cache model.

[0052] Fig.10 This is a comparison chart of the average cache of the fixed cache and DQN cache models during the simulation process. DETAILED DESCRIPTION

[0053] In order to better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0054] Various aspects of the invention are described herein with reference to the accompanying drawings, in which many illustrative embodiments are shown. The embodiments of the invention are not limited to those described in the accompanying drawings. It should be understood that the invention is implemented by any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed in the invention are not limited to any implementation. In addition, some aspects disclosed in the invention may be used alone or in any appropriate combination with other aspects disclosed in the invention.

[0055] refer to Figure 1 The present invention provides a link aggregation node caching method for differentiated QoS guarantee, comprising the following steps:

[0056] S1. Differentiate network services into three types: delay-sensitive services, bandwidth-sensitive services, and reliability-sensitive services.

[0057] S2. Construct a dynamic adjustment strategy for the DQN cache: For the three types of services, the current queue length, dequeue rate, and queuing delay in the aggregation node cache are used as the basis for dynamic adjustment of the DQN cache, and the corresponding dynamic adjustment results are obtained: dynamically increase or decrease the cache space.

[0058] S3, constructing an agent for the DQN cache adjustment strategy, which includes a main Q network and a target Q network;

[0059] The cache space of the sink node is used as the environment of the DQN agent, and the current queue length, dequeue rate, and current queuing delay in the cache space are used as the state space for the training of the DQN agent. The action space of the DQN agent is defined as: increasing the cache capacity and reducing the cache capacity; based on differentiated business classification, the reward function is divided into delay reward, throughput reward, and packet loss rate reward, and the total reward is obtained by weighted summation; using The strategy selects the action that maximizes the reward function in the current state;

[0060] Using DQN cache to dynamically adjust the strategy training agent: the main Q network is used to evaluate the current strategy, and the target Q network stabilizes the training process by providing a stable target Q value. In the training process, the experience replay mechanism is used to calculate the Q value of all actions in the next state using the target Q network, and the maximum Q value is selected to calculate the target Q value. The main Q network is used to calculate the Q value of the action in the current state, and the loss between the Q value of the action in the current state and the target Q value is calculated. Then the main Q network is continuously updated until the preset number of training times is reached, and a link aggregation node cache model for differentiated QoS assurance is obtained;

[0061] S4. Taking different types of network services as input, using the link aggregation node cache model for differentiated QoS guarantee, output the cache strategy of the corresponding network services.

[0062] The present invention divides different network services into three categories, namely, delay-sensitive services, bandwidth-sensitive services and reliability-sensitive services.

[0063] (1) Delay-sensitive services: refers to services that are very sensitive to network transmission delays. Such services usually require data to be transmitted from the source node to the destination node in a very short time. Any additional delay may have a serious impact on the service.

[0064] (2) Bandwidth-sensitive services: refers to services that have very high requirements for network throughput. Such services require large amounts of data transmission and rely on high bandwidth to ensure the continuity and smoothness of data flow.

[0065] (3) Reliability-sensitive services: refers to services that are extremely sensitive to network packet loss rates. Such services have high requirements for data integrity and accuracy, and any data loss may lead to serious consequences.

[0066] As a preferred embodiment of the present invention, in step S2, the current queue length, dequeue rate, and queuing delay in the sink node cache are used as the basis for dynamic adjustment of the DQN cache. The cache dynamic adjustment strategy process is as follows: Figure 2 Specifically including the following sub-steps S201 to S205:

[0067] S201: Cache capacity of aggregation node B Initialize;

[0068] S202, executing action 0 or action 1 according to the current queue length, dequeue rate, and queue delay status;

[0069] S203: If action 0 is executed, the capacity of one data packet is increased based on the current cache space. In order to avoid excessive cache capacity, when the capacity is increased to the upper limit of the cache capacity, U When , stop increasing the cache space;

[0070] S204: If action 1 is executed, the capacity of one data packet is reduced based on the current cache space. In order to avoid the cache capacity being 0, when it is reduced to the lower limit of the cache capacity, L When , stop reducing the cache space;

[0071] S205: Determine whether there is a queue in the current aggregation node cache, if yes, return to step S202, otherwise end the dynamic adjustment.

[0072] As a preferred embodiment of the present invention, step S3 adopts a traditional deep neural network model structure to construct a main Q network and a target Q network of a DQN cache adjustment neural network. Figure 3 As shown in the figure, the deep neural network consists of four parts: input layer, hidden layer, fully connected layer and output layer.

[0073] (1) Input layer: responsible for receiving status data. The present invention uses three input units I 1, I2 and I3 correspond to the three states of the current queue length, dequeue rate, and queuing delay in the buffer area of ​​the aggregation node, respectively.

[0074] (2) Hidden layer: responsible for nonlinear transformation and feature extraction of input state data, and mapping the input state data to a higher dimensional representation space. The present invention sets the number of neurons n in the hidden layer to 64, and uses the ReLU activation function to achieve nonlinear transformation.

[0075] (3) Fully connected layer: responsible for integrating the feature information of the hidden layer and converting this information into the output of the cache adjustment strategy. The present invention sets the number of neurons m in the fully connected layer to 64, and uses the ReLU activation function to achieve nonlinear transformation.

[0076] (4) Output layer: responsible for converting the output into action probability. The present invention uses two output units and a softmax activation function to realize the probability output of action 0 and action 1.

[0077] The DQN (Deep Q-Network) algorithm is a technology that combines deep learning and reinforcement learning to solve decision-making problems with continuous state spaces. It uses deep neural networks to approximate the Q function (i.e., the action value function), so that it can handle high-dimensional inputs. The core idea of ​​the DQN algorithm to update the Q value is to use the two technologies of "experience replay" and "fixed Q target" to increase the stability and efficiency of learning.

[0078] The construction of the DQN cache adjustment neural network is as follows:

[0079] (1) DQN cache environment and state space construction

[0080] like Figure 4 As shown, the present invention uses the cache space of the aggregation node as the environment of the intelligent agent of the DQN cache adjustment strategy, and uses the current queue length in the cache space L current , Dequeue Rate R duqueue , Current queue delay D current As the state space of DQN for the training of the agent, the state space can be expressed as s ={ L current ,R duqueue ,D current}.

[0081] (2) DQN action space construction

[0082] In the DQN cache adjustment strategy, the present invention designs two actions of increasing cache capacity and decreasing cache capacity, and thus defines an action space a={0,1}, where a=0 corresponds to increasing cache capacity and a=1 corresponds to decreasing cache capacity.

[0083] (3) Design of DQN’s reward function

[0084] The present invention is based on differentiated service classification and divides the reward function into delay reward, throughput reward and packet loss rate reward.

[0085] Delayed Rewards: ; (1)

[0086] in, D target Indicates the target queuing delay of the queue in the cache, D currentt Indicates the queue delay of the current queue, obtained through the GetSeconds() function. D current The smaller the Reward delay The closer D target .

[0087] Throughput Rewards: ; (2)

[0088] in, R dequeue Indicates the dequeue rate of the queue, R phy represents the physical bandwidth (data rate) of the bottleneck link, R dequeue The closer R phy Indicates that the throughput is greater.

[0089] Packet loss rate reward: ; (3)

[0090] in, packetLossRate Indicates the current packet loss rate. N dropped Indicates the number of packets currently discarded. N enq Indicates the number of packets currently successfully enqueued.

[0091] Total reward: In order to balance the impact of the three different types of rewards and ensure that the agent can consider the consequences of short-term behavior while pursuing long-term goals, the present invention weightedly sums the latency reward, throughput reward, and packet loss reward to obtain the total reward function: ; (4)

[0092] in, w 1. w 2. w 3 respectively correspond to the weights of latency reward, throughput reward, and packet loss reward, and clip(*) represents the clipping function, which is used to limit the reward value to between (-1, 1). In addition, in order to prevent extreme values ​​from appearing during DQN training, which leads to algorithm instability, the present invention uses clipping to limit the reward function value to between -1 and 1.

[0093] (4) DQN action selection strategy design

[0094] use strategy to select the action a that maximizes the reward function r(s,a), The decision-making method of the strategy is: , (5)

[0095] in a Indicates the current state s The action adjustment strategy made under the condition. Where m is a random number in (0,1), is the exploration rate, when When , choose the action with the largest expected value; when When the neural network parameter is θ, a parameter adjustment action is randomly selected, and Q(s,a;θ) represents the Q value of the agent selecting action a in state s when the neural network parameter is θ. The exploration rate of the present invention adopts the formula Calculate, where episode is the number of training rounds. The purpose of the exploration rate gradually decreasing with the increase of the number of training rounds is to explore the environment more in the early stage of training, discover more state-action pairs, and accumulate more experience. As the training progresses, the agent gradually learns the patterns and laws of the environment from experience, gradually forms more precise strategies and maximizes the long-term cumulative rewards.

[0096] The agent training process of DQN cache adjustment strategy is as follows Figure 4 As shown, the specific steps are as follows:

[0097] 1. Model initialization: The agent of the DQN cache adjustment strategy consists of a main Q network and a target Q network, which have the same structure but independent parameters. The main Q network is used to evaluate the current strategy (i.e., estimate the Q value), and the target Q network stabilizes the training process by providing a stable target Q value. The parameters of the main Q network are , the parameters of the target Q network are .

[0098] 2. Experience replay: The experience replay pool is used to store experience samples during the interaction between the agent and the aggregation node cache environment , and randomly sample from the experience replay pool during training, where s represents the current state, a represents the current action taken, r' represents the current reward received, and s' represents the next state.

[0099] 3. Calculate the target Q value: During the agent training process, for each experience sample , use the target Q network to calculate the Q value of all possible actions in the next state, and then select the maximum Q value , the formula for calculating the target Q value is as follows: , (6)

[0100] Where r is the immediate reward, is the discount factor, indicating the importance of future rewards, a' is the next action, Indicates that the neural network parameters are , the next state state s’ When , the agent chooses the next action a’ Q value.

[0101] 4. Update the main Q network: Use the main network to calculate the Q value of taking action a in the current state s , and then calculate the loss between this Q value and the target Q value y. The loss function uses the mean square error: , (7)

[0102] Then the parameters of the main Q network are updated by back propagation algorithm and gradient descent method , in order to minimize this loss.

[0103] 5. Update the target Q network: In order to maintain the stability of the training process, the parameters of the target Q network Will be updated regularly to the parameters of the main network , to guide the learning of the main Q network.

[0104] 6. Repeat steps 2 to 5, caching the neural network to gradually make the best decision during the training process until the end of the training round.

[0105] The present invention performs simulation in NS3, and the network topology and network load design are as follows:

[0106] Network topology design

[0107] In NS3, a typical bottleneck link network topology is designed. The topology is as follows: Figure 5 As shown in the figure. The number of source nodes and destination nodes are both 50, R1 and R2 are link aggregation nodes, the active queue management algorithm is deployed on R1 and R2, the bottleneck link between R1 and R2 uses a P2P link, the link rate is 250kbps, and the link transmission delay is 2ms. The link between the source node and R1 and the link between the destination node and R2 uses a P2P link, the link rate is 10Mbps, and the transmission delay is set to 2ms.

[0108] Network load design

[0109] The present invention simulates the sudden and dynamic changes of network load by simulating the difference in the number of UDP flows in different time periods, and uses the ON / OFF traffic model to simulate the concurrent scenarios of 7 services, specifically:

[0110] 1. Business flow sending rules: such as Figure 5As shown, S1 sends data to D1, S2 sends data to D2, and so on. 50 To D 50 Send data.

[0111] 2. Network load increase rules: As shown in Table 1, Table 1 shows the network load design in NS3; the service sending time is 50s in total, of which the number of UDP flows within 0-10s is 50, and the service type sent is video service, the number of UDP flows within 10-20s is 100, and the service type sent is document service, the number of UDP flows within 20-30s is 200, and the service type sent is fax and image service, the number of UDP flows within 30-40s is 100, and the service type sent is MF (message) and VMF (variable message) service, the number of UDP flows within 40-50s is 50, and the service type sent is voice service;

[0112] Table 1:

[0113]

[0114] 3. Business flow model: As shown in Table 2, Table 2 shows the relevant parameter settings of the business model in NS3;

[0115] The present invention designs 7 different service flows based on the ON / OFF traffic model, among which the ON / OFF sending time of video, text, fax, image, MF, and VMF services adopts the Pareto distribution function model of NS3, and the ON / OFF sending time of voice services adopts the exponential distribution function model of NS3.

[0116] Table 2:

[0117]

[0118] 4. The NS3 simulation duration is set to 60s.

[0119] Replace the fixed cache in the active queue management algorithm in NS3 with the DQN cache model;

[0120] (1) Environmental parameter settings

[0121] Set the initial value of the cache capacity B =5 packets , the upper limit of cache capacity increase is U =50 packets , the lower limit of cache capacity reduction is L =1 packets, queue target queuing delay D target =0.03s.

[0122] (2) Model training parameter setting

[0123] Number of training rounds episode=50, Learning Rate , discount factor In addition, Adam optimizer is used to optimize network parameters.

[0124] (3) Selection of active queue management algorithm

[0125] The FIFO algorithm, RED algorithm, and CoDel algorithm provided in NS3 are selected for simulation verification, and their fixed cache capacity is replaced with the DQN cache model.

[0126] Run the traditional fixed cache based active queue management algorithm in NS3;

[0127] Based on the aforementioned network topology and network load design, the three active queue management algorithms of NS3, FIFO, RED, and CoDel, are run respectively.

[0128] Run the active queue management algorithm based on the DQN cache design strategy in NS3;

[0129] Based on the aforementioned network topology and network load design, three active queue management algorithms, FIFO, RED, and CoDel, based on the DQN cache design strategy are run respectively.

[0130] The performance of the active queue management algorithm under the two cache modes is compared to verify whether the present invention has improved the performance of the active queue management algorithm. In order to ensure the validity of the simulation results, the cache capacity in the fixed cache strategy is set to a fixed value of 50 packets.

[0131] Average transmission delay performance comparison Figure 6 As shown in the figure, it can be seen that the transmission delay of the queue management algorithm under the DQN cache strategy is lower than that of the queue management algorithm under the fixed cache strategy. In the FIFO algorithm, the average transmission delay under the fixed cache strategy is 3.14697s, and the reward function weights in the DQN cache strategy are set as: w 1 = 0.08 , w 2=0.01, w 3=0.01, the corresponding transmission delay is 3.06105s, and the average transmission delay is reduced by 2.7%. In the RED algorithm, the average transmission delay under the fixed cache strategy is 2.72299s, and the reward function weights in the DQN cache strategy are set as: w 1 = 0.06, w 2=0.01, w3=0.01, the corresponding average transmission delay is 2.6527, and the average transmission delay is reduced by 2.6%. In the CoDel algorithm, the average transmission delay under the fixed cache strategy is 2.2446s, and the reward function weights in the DQN cache strategy are set as: w 1 = 0.086, w 2=0.01, w 3=0.01, the corresponding average transmission delay is 2.22236s, and the average transmission delay is reduced by 0.99%.

[0132] Average throughput performance comparison Figure 7 As shown in the figure, it can be seen that the throughput of the queue management algorithm under the DQN cache strategy is higher than that of the queue management algorithm under the fixed cache strategy. In the FIFO algorithm, the average throughput under the fixed cache strategy is 227.452 kbps, and the reward function weights in the DQN cache strategy are set as: w 1 = 0.01, w 2=0.08, w 3=0.01, the corresponding average throughput is 230.588kbps, and the average throughput is increased by 1.38%. In the RED algorithm, the throughput of the fixed cache strategy is 227.671kbps, and the reward function weights in the DQN cache strategy are set as: w 1 = 0.001, w 2=1, w 3=0.001, the corresponding throughput is 227.728 kbps, which is slightly higher than the average throughput of the fixed cache strategy. In the CoDel algorithm, the average throughput of the fixed cache strategy is 228.264 kbps, and the reward function weights in the DQN cache strategy are set to w 1 = 0.01, w 2=0.06, w 3=0.01, the corresponding average throughput is 228.138 Kbps, and the average throughput is only reduced by 0.06%, which is basically the same.

[0133] Packet loss rate performance comparison Figure 8 As shown in the figure, it can be seen that the packet loss rate of the queue management algorithm under the DQN cache strategy is basically the same as the packet loss rate of the queue management algorithm under the fixed cache strategy. In the FIFO algorithm, the packet loss rate under the fixed cache strategy is 44.0342%, and the reward function weights in the DQN cache strategy are set as: w 1 = 0.001 w 2=0.001, wThe packet loss rate corresponding to 3=1 is 44.0037%, which is slightly lower than the packet loss rate under fixed cache and is basically the same. In the RED algorithm, the packet loss rate under the fixed cache strategy is 47.373%, and the reward function weights in the DQN cache strategy are set as: w 1 = 0.001, w 2=0.001, w 3=1, the corresponding packet loss rate is 46.9352%, and the packet loss rate is reduced by 0.9%. In the CoDel algorithm, the packet loss rate under the fixed cache strategy is 43.9731%, and the reward function weights in the DQN cache strategy are set as: w 1 = 0.02, w 2=0.02, w The packet loss rate corresponding to 3=2 is 43.668%, and the packet loss rate is reduced by 0.69%.

[0134] Fig. 9 The relationship between the cache boundary and the queue length in the aggregation node under the DQN cache model is shown. It can be seen that the upper limit of the cache capacity of the aggregation node is 50 packets, and the cache boundary is dynamically adjusted as the queue length changes, indicating that the proposed DQN cache model is effective.

[0135] Fig.10 is the average cache size at the aggregation node under the dynamic cache strategy during the simulation process. It can be seen that under the FIFO model, the average cache of the present invention is only 59.32% of the fixed cache; under the RED model, the average cache of the present invention is only 61.88% of the fixed cache; under the CoDel model, the average cache of the present invention is only 53.56% of the fixed cache.

[0136] The above results show that under the premise of significantly reducing the average cache capacity, the link aggregation node cache design method for differentiated QoS guarantee proposed in the present invention can effectively improve the transmission delay, throughput and packet loss rate performance of the active queue management algorithm.

[0137] Although the present invention has been described above with preferred embodiments, it is not intended to limit the present invention. A person skilled in the art of the present invention may make various modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the definition of the claims.

Claims

1. A link aggregation node caching method for differentiated QoS guarantee, characterized in that: The steps include: S1. Differentiate network services into three types: delay-sensitive services, bandwidth-sensitive services, and reliability-sensitive services. S2. Construct a dynamic adjustment strategy for the DQN cache: For the three types of services, the current queue length, dequeue rate, and queuing delay in the aggregation node cache are used as the basis for dynamic adjustment of the DQN cache, and the corresponding dynamic adjustment results are obtained: dynamically increase or decrease the cache space; including the following sub-steps: S201: Cache capacity of aggregation node B Initialize; S202, executing action 0 or action 1 according to the current queue length, dequeue rate, and queue delay status; S203: If action 0 is executed, the capacity of one data packet is increased based on the current cache space. In order to avoid excessive cache capacity, when the capacity is increased to the upper limit of the cache capacity, U When , stop increasing the cache space; S204: If action 1 is executed, the capacity of one data packet is reduced based on the current cache space. In order to avoid the cache capacity being 0, when it is reduced to the lower limit of the cache capacity, L When , stop reducing the cache space; S205, determine whether there is a queue in the current aggregation node cache, if it returns to step S202, otherwise the dynamic adjustment ends; S3, constructing an intelligent agent for the DQN cache dynamic adjustment strategy, which includes a main Q network and a target Q network; The cache space of the sink node is used as the environment of the DQN agent, and the current queue length, dequeue rate, and current queuing delay in the cache space are used as the state space for the training of the DQN agent. The action space of the DQN agent is defined as: increasing the cache capacity and reducing the cache capacity; based on differentiated business classification, the reward function is divided into delay reward, throughput reward, and packet loss rate reward, and the total reward is obtained by weighted summation; using The strategy selects the action that maximizes the reward function in the current state; In the agent of the DQN cache dynamic adjustment strategy, the main Q network and the target Q network have the same structure; the main Q network and the target Q network both include an input layer, a hidden layer, a fully connected layer, and an output layer: The input layer is used to receive input status data. The input layer includes three input units, which correspond to the three states of the current queue length, the dequeue rate, and the queuing delay in the sink node buffer area respectively; The hidden layer is used to perform nonlinear transformation and feature extraction on the input state data, and map the input state data to a higher dimensional representation space; The fully connected layer is used to integrate the feature information of the hidden layer and convert the information into the output of the cache adjustment strategy; the output layer is used to convert the output into action probability, the output layer includes 2 output units, and the softmax activation function is used to realize the probability output of action 0 or action 1; Using DQN cache to dynamically adjust the strategy training agent: the main Q network is used to evaluate the current strategy, and the target Q network stabilizes the training process by providing a stable target Q value. In the training process, the experience replay mechanism is used to calculate the Q value of all actions in the next state using the target Q network, and the maximum Q value is selected to calculate the target Q value. The main Q network is used to calculate the Q value of the action in the current state, and the loss between the Q value of the action in the current state and the target Q value is calculated. Then the main Q network is continuously updated until the preset number of training times is reached, and a link aggregation node cache model for differentiated QoS assurance is obtained; Among them, the latency reward, throughput reward, packet loss rate reward, and total reward are as follows: Delayed Rewards: (1); in, D target Indicates the target queuing delay of the queue in the cache, D current Indicates the queue delay of the current queue, obtained through the GetSeconds() function. D current The smaller the Reward delay The closer D target ; Throughput Rewards: (2); in, R dequeue Indicates the dequeue rate of the queue, R phy represents the physical bandwidth of the bottleneck link, R dequeue The closer R phy Indicates that the throughput is greater; Packet loss rate reward: (3); in, packetLossRate Indicates the current packet loss rate. N dropped Indicates the number of packets currently discarded. N enq Indicates the number of data packets currently successfully queued; Total Rewards: (4); in, w 1. w 2. w 3 respectively correspond to the weights of latency reward, throughput reward, and packet loss reward, and clip(*) represents the clipping function; The decision-making method of the strategy is: (5); in, a Indicates the current state s The action adjustment strategy made under m is a random number in (0,1), is the exploration rate, when When , choose the action with the largest expected value; when When , a parameter adjustment action is randomly selected. , episode is the number of training rounds, s ={ L current ,R duqueue ,D current }, L current Indicates the current queue length. R dequeue Indicates the dequeue rate, D curren t Indicates the current queue delay. is the parameter of the Q network; Q(s,a;θ) represents the Q value of the agent choosing action a in state s when the neural network parameter is θ; The target Q value calculation formula is as follows: (6); in, r For the current reward, is the discount factor, indicating the importance of future rewards, a’ For the next action, s’ Indicates the next state, are the parameters of the target Q network; Indicates that the neural network parameters are , the next state state s’ When , the agent chooses the next action a’ Q value; Calculate the loss between the Q value of the action in the current state and the target Q value, as shown below, and then update the parameters of the main Q network through the back propagation algorithm and gradient descent method , to minimize this loss, (7); S4. Taking different types of network services as input, using the link aggregation node cache model for differentiated QoS guarantee, output the cache strategy of the corresponding network services.

Citation Information

Patent Citations

  • Adaptive selection method of wireless sensor network routing algorithm based on deep reinforcement learning

    CN116471645A

  • WiFi network resource scheduling method and system based on MAPPO algorithm

    CN117412323A