A method for synchronizing KDN controllers based on deep reinforcement learning

Abstract

This invention discloses a KDN controller synchronization method based on deep reinforcement learning, comprising: a data plane monitoring the network in real time to obtain network state data; a management plane analyzing and processing the received network state data to extract network state features; preprocessing the state features; a knowledge plane receiving the preprocessed state features and, based on the current network, controller state, and historical state, outputting a controller synchronization decision using the KDN-CSP-L algorithm; a control plane receiving the controller synchronization decision from the knowledge plane and executing the corresponding synchronization decision operation, sending the specific instructions for the synchronization decision operation to the corresponding network device, i.e., the controller; the network device performing controller information synchronization operation on the network according to the received optimal controller synchronization decision instruction; and the network device generating new state data and initiating a new data acquisition and processing cycle.

Description

Technical Field

[0001] This invention belongs to the field of computer technology and relates to a method for synchronizing a KDN controller based on deep reinforcement learning. Background Technology

[0002] Knowledge-Defined Networking (KDN) is a knowledge-based network architecture that combines network intelligence and adaptability, enabling the network to automatically configure and manage network resources according to the needs of applications and services. The KDN architecture inherits the control plane from SDN and therefore also includes a controller component. In a distributed multi-controller network, because the controllers are distributed across the network, each controller needs to coordinate and control a subset of network resources. Therefore, it requires continuous state updates and synchronization to maintain a logically centralized network view. If the states of the controllers are out of sync, it can lead to serious anomalies such as vulnerabilities, black holes, and other problems caused by inconsistent policies, impacting network performance and reliability.

[0003] The paper "Probabilistically bounded staleness for practical partial quorums" proposes a probabilistically bounded failure model to predict the strength of eventual data consistency among distributed nodes and provide a reference for balancing consistency and network performance. While the probabilistically bounded failure model can predict the strength of eventual data consistency among distributed nodes and improve latency, it is only applicable to general distributed systems and does not provide specific optimization methods considering the characteristics of SDN.

[0004] The paper "Kandoo: A Framework for Efficient and Scalable Offloading of Control Applications" proposes a hierarchical framework to achieve consistency among multiple controllers. The upper and lower controllers form a tree structure, with the lower controller managing the local switch and the upper controller coordinating synchronization to maintain a consistent network view. While the highly configurable and scalable hierarchical control plane framework proposed above can achieve synchronization among multiple controllers and maintain a consistent network view, this strategy essentially only changes the communication method between multiple controllers and cannot alleviate the fundamental conflict between consistency and performance.

[0005] The paper "Logically Centralized State Distribution Trade-offs in Software-Defined Networks" proposes two schemes: Link Balancer Controller (LBC) and Separate State Link Balancer Controller (SSLBC). The local NIB records switch information within its domain and updates it in real time, while the remote NIB records switch information from other domains and updates it periodically. The paper proposes that the local controller in SSLBC detects changes in switch load in real time and distributes forwarding rules to the switch with the least load based on the current state of the local domain. In terms of load balancing, SSLBC and LBC have the same number of synchronizations. However, due to the inconsistent time intervals between two consecutive synchronizations, SSLBC may lead to error scenarios such as forwarding loops and black holes.

[0006] DRL (Dynamic Relationship Management) methods are employed to address the state synchronization problem between KDN (Knowledge, Network, and Data) controllers, such as adaptive policy-based methods and reinforcement learning-based methods. These methods have improved the efficiency and stability of state synchronization between KDN controllers to some extent. However, these methods still face some challenges, such as low training efficiency, slow convergence speed, and high communication overhead between controllers. Meanwhile, because logically centralized controllers have complete knowledge of the network state, they can make globally optimal decisions. However, this centralized control is also plagued by scalability and reliability issues. Summary of the Invention

[0007] To address the aforementioned technical problems, the present invention aims to provide a method for KDN controller synchronization based on deep reinforcement learning. This method is based on the KDN-CSP(L) algorithm and an action branch architecture controller synchronization strategy, which can solve the problem of multi-controller deployment and coordination while reducing the complexity of the action space.

[0008] This invention provides a method for KDN controller synchronization based on deep reinforcement learning, comprising:

[0009] Step 1: The data plane monitors the network in real time to obtain network status data;

[0010] Step 2: The management plane analyzes and processes the received network status data to extract the network status characteristics;

[0011] Step 3: Preprocess the state features;

[0012] Step 4: The knowledge plane receives the preprocessed state features, and then, based on the current network and the controller's state and historical state, uses the KDN-CSP-L algorithm to output the controller synchronization decision;

[0013] Step 5: After receiving the controller synchronization decision made by the knowledge plane, the control plane executes the corresponding synchronization decision operation and sends the specific instructions of the synchronization decision operation to the corresponding network device, i.e., the controller.

[0014] Step 6: The network device performs controller information synchronization operation on the network based on the received optimal controller synchronization decision instruction;

[0015] Step 7: The network device generates new status data and starts a new data acquisition and processing cycle.

[0016] Furthermore, step 1 specifically involves:

[0017] The data plane senses network status data through data acquisition, processing, and storage. The network status data includes: node failures, topology changes, controller synchronization, link latency updates, data size, network bandwidth, controller processing speed, and network transmission delay information.

[0018] Furthermore, step 2 specifically involves:

[0019] The latency of each link is stored in a two-dimensional array, with the first dimension being the link's originating node and the second dimension being the link's destination node.

[0020] Calculate the number of node failures, count the nodes that have failed, and store them according to their node identifiers;

[0021] Topology changes and controller synchronization are also stored in digital form based on node identifiers and synchronization information identifiers.

[0022] Furthermore, step 3 specifically involves:

[0023] Step 3.1: Standardize the network transmission delay information and node failures by dividing the delay of each link by the preset maximum delay and dividing the number of node failures by the total number of nodes.

[0024] Step 3.2: Then binarize the state features and output the preprocessed state features through data transformation.

[0025] Furthermore, step 4 specifically involves:

[0026] Step 4.1: Adopt an action branching architecture to decompose the action space into multiple subspaces. Actions include whether they are synchronized and the information owned by the synchronization controller. Each subspace controls one action dimension.

[0027] Step 4.2: Obtain the Q-value estimate for each action by combining the state value estimate with the sub-action advantage estimated by the assigned action arm. The state value estimate is shared by all possible actions for a given state.

[0028] Step 4.3: Calculate the action corresponding to the state, model the controller synchronization strategy as a Markov decision process of 3-tuple (S,A,R), and design the state, action and reward value in the Markov decision process;

[0029] Step 4.4: Construct a priority experience replay pool, and assign a priority to each experience based on the error of the experience;

[0030] Step 4.5: Introduce the Bellman equation, using the value form of network states to maximize long-term cumulative rewards. The Bellman equation is shown below:

[0031]

[0032] Among them, v π (s) represents the expected reward obtained by acting according to the optimal policy π in state s, where R t+1 This indicates that it is a random variable at time t+1, and γ is the discount factor;

[0033] Step 4.6: Find the maximum expected return, defining the return as the sum of the reward sequence, as shown in the following formula:

[0034] G t =R t+1 +γR t+2 +γ 2 R t+3 +…

[0035] Here, γ is a value in the range of 0 to 1. When γ = 0, the agent only cares about maximizing the immediate reward. When γ is close to 1, the agent considers future rewards more.

[0036] Step 4.7: Construct the Q-function state-action value function. This function represents the expected total reward that the agent can obtain in the future after taking action a in state s. With the introduced action branching architecture, let a... i If a represents a sub-action belonging to action arm i, then a i The Q value is represented as Q(s,a) i The Q function is defined as follows:

[0037] Q(s,a i )=E(G t )

[0038] Among them, (s,a) i ) is a state-action pair, G t In return, if the number of arms that output "1" is greater than the maximum number of controller synchronizations set according to the network synchronization overhead, the sub-actions with larger Q values ​​are selected first until the maximum number of controller synchronizations is reached. The selected sub-actions are then sent to the decision execution module for synchronization.

[0039] Step 4.8: Obtain the optimal policy, that is, given any state, determine the optimal action the agent should take in that state. The optimal policy is defined as follows:

[0040] π * (s)=argmax a Q(s,a)

[0041] Each action 'a' corresponds to a different reward value, max a Q(s,a) represents the expected total reward for each action a, and the action that maximizes the Q-function state-action value function is selected.

[0042] Furthermore, step 4.3 specifically involves:

[0043] 4.3.1: State Space Design: The number of controllers represents the dimension of a state, and a state corresponds to the set of time slot counts since the last broadcast of the latest value of the network view of each controller;

[0044] 4.3.2: Motion space design: Assign a separate motion arm to each controller. The motion is defined as whether to synchronize the corresponding controller. A value of 1 indicates synchronization, and a value of 0 indicates desynchronization.

[0045] 4.3.3 Reward Function Design: The immediate reward associated with the state-action pair is represented by a reward function, denoted by R(s,a), where s and a are the state vector and action vector, respectively. The reward function is shown in the following equation:

[0046] R(s,a)=Delay after_sync -Delay before_sync

[0047] Among them, |Delay after_sync Delay before_sync These represent the network service request delays before and after taking action a in state s, respectively.

[0048] Furthermore, in step 4.4, the empirical priority is calculated according to the following formula:

[0049]

[0050] Where, p i δ represents the priority of the i-th experience. i This represents the error of the experience. o is a small constant to prevent the priority from being 0. α is a hyperparameter used to control the distribution of priorities.

[0051] Furthermore, the information synchronized in step 6 includes: the current available network bandwidth owned by the controller, the number of requests initiated by mobile devices connecting to the network in the current time interval, and the current available network link information.

[0052] The method for synchronizing a KDN controller based on deep reinforcement learning, as proposed in this invention, has at least the following beneficial effects:

[0053] This invention leverages the concept of Knowledge-Defined Networks (KDNs), describing the functions of the data plane, management plane, knowledge plane, and control plane in its network architecture as state awareness, state analysis, synchronization decision-making, and decision execution, respectively. These components coordinate and communicate with each other, forming the control loop of the controller synchronization strategy. A controller synchronization strategy based on the KDN-CSP-L algorithm and action branching architecture is then designed. This strategy effectively solves the problem of multi-controller deployment and coordination, significantly reducing the complexity of the action space. Furthermore, because this mechanism employs deep reinforcement learning, it can continuously learn and adjust to optimize the controller's decision-making strategy, resulting in significant improvements in convergence speed, average request latency, and consistency level, further enhancing the network's performance and efficiency. Attached Figure Description

[0054] Figure 1 This is a flowchart of a method for synchronizing a KDN controller based on deep reinforcement learning according to the present invention;

[0055] Figure 2 This is a comparison chart of the average reward values ​​of the KDN-CSP-L algorithm of this invention with the Q-learning algorithm and the Sarsa algorithm for deep reinforcement learning training;

[0056] Figure 3 This is a box plot of the average reward value of the KDN-CSP-L algorithm, Q-learning algorithm, and Sarsa algorithm used in deep reinforcement learning training according to the present invention;

[0057] Figure 4 This is a comparison chart of the average request latency of the KDN-CSP-L algorithm of this invention with the Q-learning algorithm and the Sarsa algorithm for deep reinforcement learning training;

[0058] Figure 5The average request latency box plot of the KDN-CSP-L algorithm, Q-learning algorithm, and Sarsa algorithm of this invention for deep reinforcement learning training;

[0059] Figure 6 This is a comparison chart of the average request latency of the KDN-CSP-L algorithm of this invention and the synchronization strategies of Full Sync, No Sync, Anti-entropy, and Greedy;

[0060] Figure 7 This is a comparison chart of the synchronization overhead of the KDN-CSP-L algorithm of this invention and the synchronization strategies of Full Sync, Anti-entropy, and Greedy;

[0061] Figure 8 This is a comparison chart of the consistency levels of the invented KDN-CSP-L algorithm and the synchronization strategies Full Sync, Anti-entropy, and Greedy. Detailed Implementation

[0062] like Figure 1 As shown, the present invention provides a method for synchronizing a KDN controller based on deep reinforcement learning, comprising:

[0063] Step 1: The data plane performs real-time monitoring of the network to obtain network status data. Specifically, Step 1 involves:

[0064] The data plane senses network status data through data acquisition, processing, and storage. The network status data includes: node failures, topology changes, controller synchronization, link latency updates, data size, network bandwidth, controller processing speed, and network transmission delay information.

[0065] Step 2: The management plane analyzes and processes the received network status data to extract the network status features. Specifically, Step 2 involves:

[0066] The latency of each link is stored in a two-dimensional array, with the first dimension being the link's originating node and the second dimension being the link's destination node.

[0067] Calculate the number of node failures, count the nodes that have failed, and store them according to their node identifiers;

[0068] Topology changes and controller synchronization are also stored in digital form based on node identifiers and synchronization information identifiers.

[0069] Step 3: Preprocess the state features, specifically:

[0070] Step 3.1: Standardize the network transmission delay information and node failures by dividing the delay of each link by the preset maximum delay and dividing the number of node failures by the total number of nodes.

[0071] Step 3.2: Then binarize the state features and output the preprocessed state features through data transformation.

[0072] Step 4: The knowledge plane receives the preprocessed state features, and then, based on the current network and the controller's state and historical state, uses the KDN-CSP-L algorithm to output the controller synchronization decision. Specifically, Step 4 is as follows:

[0073] Step 4.1: Adopt an action branching architecture to decompose the action space into multiple subspaces. Actions include whether they are synchronized and the information owned by the synchronization controller. Each subspace controls one action dimension.

[0074] Step 4.2: Obtain the Q-value estimate for each action by combining the state value estimate with the sub-action advantage estimated by the assigned action arm. The state value estimate is shared by all possible actions for a given state.

[0075] Step 4.3: Calculate the action corresponding to the state, model the controller synchronization strategy as a Markov decision process of 3-tuple (S,A,R), and design the state, action and reward value in the Markov decision process. Specifically, Step 4.3 is as follows:

[0076] 4.3.1: State Space Design: The number of controllers represents the dimension of a state, and a state corresponds to the set of time slot counts since the last broadcast of the latest value of the network view of each controller;

[0077] 4.3.2: Motion space design: Assign a separate motion arm to each controller. The motion is defined as whether to synchronize the corresponding controller. A value of 1 indicates synchronization, and a value of 0 indicates desynchronization.

[0078] 4.3.3 Reward Function Design: The immediate reward associated with the state-action pair is represented by a reward function, denoted by R(s,a), where s and a are the state vector and action vector, respectively. The reward function is shown in the following equation:

[0079] R(s, a) = Delay after_sync -Delay before_sync

[0080] Among them, |Delay after_sync Delay before_sync These represent the network service request delays before and after taking action a in state s, respectively.

[0081] Step 4.4: Construct a priority experience replay pool, assigning a priority to each experience based on its error. Specifically, the priority of an experience is calculated using the following formula:

[0082]

[0083] Where, p i δ represents the priority of the i-th experience. i This represents the error of the experience. o is a small constant to prevent the priority from being 0. α is a hyperparameter used to control the distribution of priorities.

[0084] Step 4.5: Introduce the Bellman equation, using the value form of network states to maximize long-term cumulative rewards. The Bellman equation is shown below:

[0085]

[0086] Among them, v π (s) represents the expected reward obtained by acting according to the optimal policy π in state s, where R t+1 This indicates that it is a random variable at time t+1, and γ is the discount factor;

[0087] Step 4.6: Find the maximum expected return, defining the return as the sum of the reward sequence, as shown in the following formula:

[0088] G t =R t+1 +γR t+2 +γ 2 R t+3 +…

[0089] Here, γ is a value in the range of 0 to 1. When γ = 0, the agent only cares about maximizing the immediate reward. When γ is close to 1, the agent considers future rewards more.

[0090] Step 4.7: Construct the Q-function state-action value function. This function represents the expected total reward that the agent can obtain in the future after taking action a in state s. With the introduced action branching architecture, let a... i If a represents a sub-action belonging to action arm i, then a i The Q value is represented as Q(s, a) i The Q function is defined as follows:

[0091] Q(s, a) i )=E(G t )

[0092] Among them, (s, a) i ) is a state-action pair, G tIn return, if the number of arms that output "1" is greater than the maximum number of controller synchronizations set according to the network synchronization overhead, the sub-actions with larger Q values ​​are selected first until the maximum number of controller synchronizations is reached. The selected sub-actions are then sent to the decision execution module for synchronization.

[0093] Step 4.8: Obtain the optimal policy, that is, given any state, determine the optimal action the agent should take in that state. The optimal policy is defined as follows:

[0094] π * (s)=argmax a Q(s,a)

[0095] Each action 'a' corresponds to a different reward value, max a Q(s,a) represents the expected total reward for each action a, and the action that maximizes the Q-function state-action value function is selected.

[0096] Step 5: After receiving the controller synchronization decision made by the knowledge plane, the control plane executes the corresponding synchronization decision operation and sends the specific instructions of the synchronization decision operation to the corresponding network device, i.e., the controller.

[0097] Step 6: The network device performs controller information synchronization operation on the network based on the received optimal controller synchronization decision instruction;

[0098] The information synchronized during implementation includes: the current available network bandwidth owned by the controller, the number of requests initiated by mobile devices connecting to the network in the current time interval, and the current available network link information.

[0099] Step 7: The network device generates new status data and starts a new data acquisition and processing cycle.

[0100] The above steps are repeated continuously to form a closed loop. Throughout the process, the reinforcement learning agent continuously learns from the network's feedback, optimizes the network synchronization strategy, and thus achieves better network control.

[0101] The KDN-CSP-L algorithm of this invention is compared with the Q-learning algorithm and the Sarsa algorithm. It shows significant improvement in evaluation metrics such as average reward value, average service request latency, and synchronization overhead. For example, it exhibits faster convergence speed, better synchronization decision-making performance, and the ability to achieve intelligent decision-making for controller synchronization. Specifically:

[0102] Reward values: The designed KDN-CSP-L algorithm model was trained 10,000 times, and the reward values ​​generated during the training process were recorded as follows: Figure 2As shown, the average reward value of the three algorithms continuously increases with the number of training iterations. In terms of the rate of increase in average reward value, the KDN-CSP-L algorithm shows a significantly faster increase than the Sarsa and Q-Learning algorithms. Figure 3 The box plot of the average reward value shows that the KDN-CSP-L algorithm still performs best in terms of reward value, with its median, upper quartile, and lower quartile all significantly higher than those of the Q-Learning and Sarsa algorithms. Furthermore, the KDN-CSP-L algorithm has a smaller standard deviation, indicating higher stability.

[0103] Service request latency: Figure 4 This study compares the performance of the KDN-CSP-L algorithm with two classic reinforcement learning algorithms, Q-Learning and Sarsa, in terms of average request latency. As shown in the figures, all three algorithms exhibit low service request latency in the initial training phase. However, as training progresses and the network begins to simulate various network events, the service request latency increases. During the stable phase, the KDN-CSP-L algorithm gradually outperforms the other two algorithms, maintaining a lower service request latency and achieving better performance. Figure 5 The box plot shows the average request latency, with the horizontal axis representing different reinforcement learning algorithms and the vertical axis representing the average service request latency. The KDN-CSP-L algorithm has a lower median and third quartile than the Q-Learning and Sarsa algorithms, indicating that the proposed method performs better in controller synchronization and enables faster service request responses.

[0104] In addition to comparing reinforcement learning algorithms, this invention also selects four synchronization strategies—completely asynchronous (No Sync), fully synchronous (Full Sync), anti-entropy, and Greedy—as comparison algorithms for the synchronization strategy based on the KDN-CSP-L algorithm of this invention, as detailed below:

[0105] Service request latency: Figure 6 The paper compares the average service request latency of the KDN-CSP-L algorithm with other synchronization strategies over 10,000 training rounds. The results show that the KDN-CSP-L algorithm of this invention is optimal as the network size increases, while using full synchronization leads to network performance degradation due to excessive synchronization, even performing worse than the Greedy synchronization strategy and Anti-entropy.

[0106] Synchronization overhead: Figure 7The results show a comparison of synchronization overhead based on different synchronization strategies, including Full Sync, Anti-entropy, Greedy, and the KDN-CSP-L algorithm of this invention. Full Sync has the highest synchronization overhead, while the synchronization overhead of the method of this invention is significantly lower than that of Full Sync.

[0107] Consistency level: Figure 8 This paper compares the consistency levels of KDN-CSP-L with four synchronization strategies: Full Sync, Anti-entropy, and Greedy. It shows that Full Sync achieves the highest consistency level, while the invented method is second only to Full Sync, demonstrating a high level of consistency.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for synchronizing a KDN controller based on deep reinforcement learning, characterized in that, include: Step 1: The data plane monitors the network in real time to obtain network status data; Step 2: The management plane analyzes and processes the received network status data to extract the network status characteristics; Step 3: Preprocess the state features; Step 4: The knowledge plane receives the preprocessed state features, and then, based on the current network and the controller's state and historical state, uses the KDN-CSP-L algorithm to output the controller synchronization decision; Step 5: After receiving the controller synchronization decision made by the knowledge plane, the control plane executes the corresponding synchronization decision operation and sends the specific instructions of the synchronization decision operation to the corresponding network device, i.e., the controller. Step 6: The network device performs controller information synchronization operation on the network based on the received optimal controller synchronization decision instruction; Step 7: The network device generates new status data and initiates a new data acquisition and processing cycle; Step 4 specifically involves: Step 4.1: Adopt the action branch architecture to decompose the action space into multiple subspaces. Actions include whether to synchronize the controller and the information that the controller has to be synchronized. Each subspace controls one action dimension. Step 4.2: Obtain the Q-value estimate for each action by combining the state value estimate with the sub-action advantage estimated by the assigned action arm. The state value estimate is shared by all possible actions for a given state. Step 4.3: Calculate the action corresponding to the state, model the controller synchronization strategy as a Markov decision process of 3-tuple (S,A,R), and design the state, action and reward value in the Markov decision process; Step 4.4: Construct a priority experience replay pool, and assign a priority to each experience based on the error of the experience; Step 4.5: Introduce the Bellman equation, using the value form of network states to maximize long-term cumulative rewards. The Bellman equation is shown below: in, Let R represent the expected reward obtained by acting according to the optimal policy π in state s. t+1 This indicates that it is a random variable at time t+1, and γ is the discount factor; Step 4.6: Find the maximum expected return, defining the return as the sum of the reward sequence, as shown in the following formula: Here, γ is a value in the range of 0 to 1. When γ = 0, the agent only cares about maximizing the immediate reward. When γ is close to 1, the agent considers future rewards more. Step 4.7: Construct the Q-function state-action value function. This function represents the expected total reward that the agent can obtain in the future after taking action a in state s. With the introduced action branching architecture, let a... i If a represents a sub-action belonging to action arm i, then a i The Q value is represented as Q(s, a) i The Q function is defined as follows: Among them, (s, a) i ) is a state-action pair, G t In return, if the number of arms that output "1" is greater than the maximum number of controller synchronizations set according to the network synchronization overhead, the sub-actions with larger Q values ​​are selected first until the maximum number of controller synchronizations is reached. The selected sub-actions are then sent to the decision execution module for synchronization. Step 4.8: Obtain the optimal policy, that is, given any state, determine the optimal action the agent should take in that state. The optimal policy is defined as follows: Each action 'a' corresponds to a different reward value. This means calculating the expected total reward for each action 'a' and selecting the action that maximizes the Q-function state-action value function.

2. The method for KDN controller synchronization based on deep reinforcement learning as described in claim 1, characterized in that, Step 1 specifically involves: The data plane senses network status data through data acquisition, processing, and storage. The network status data includes: node failures, topology changes, controller synchronization, link latency updates, data size, network bandwidth, controller processing speed, and network transmission delay information.

3. The method for synchronizing a KDN controller based on deep reinforcement learning as described in claim 2, characterized in that, Step 2 specifically involves: The latency of each link is stored in a two-dimensional array, with the first dimension being the link's originating node and the second dimension being the link's destination node. Calculate the number of node failures, count the nodes that have failed, and store them according to their node identifiers; Topology changes and controller synchronization are also stored in digital form based on node identifiers and synchronization information identifiers.

4. The method for KDN controller synchronization based on deep reinforcement learning as described in claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Standardize the network transmission delay information and node failures by dividing the delay of each link by the preset maximum delay and dividing the number of node failures by the total number of nodes. Step 3.2: Then binarize the state features and output the preprocessed state features through data transformation.

5. The method for KDN controller synchronization based on deep reinforcement learning as described in claim 1, characterized in that, Step 4.3 specifically involves: 4.3.1: State Space Design: The number of controllers represents the dimension of a state, and a state corresponds to the set of time slot counts since the last broadcast of the latest value of the network view of each controller; 4.3.2: Motion space design: Assign a separate motion arm to each controller. The motion is defined as whether to synchronize the corresponding controller. A value of 1 indicates synchronization, and a value of 0 indicates desynchronization. 4.3.3 Reward Function Design: A reward function is used to represent the immediate reward associated with the state-action pair. Let be the state vector and action vector, respectively, and let be the reward function as shown in the following equation: Among them, Delay after_sync Delay before_sync These represent the network service request delays before and after taking action a in state s, respectively.

6. The method for synchronizing a KDN controller based on deep reinforcement learning as described in claim 1, characterized in that, In step 4.4, the empirical priority is calculated according to the following formula: Where, p i This indicates the priority of the i-th experience. This represents the error of the experience. o is a small constant to prevent the priority from being 0. α is a hyperparameter used to control the distribution of priorities.

7. The method for synchronizing a KDN controller based on deep reinforcement learning as described in claim 1, characterized in that, The information synchronized in step 6 includes: the current available network bandwidth owned by the controller, the number of requests initiated by mobile devices connecting to the network in the current time interval, and the current available network link information.

Images