Route scheduling method based on deep reinforcement learning
Through deep reinforcement learning methods, network state is monitored and state space and action space are built. Deep neural network is used to generate routing decisions, which solves the problem of insufficient response and adjustment capabilities of traditional routing scheduling methods in complex network environments, and achieves more efficient routing Scheduling strategy optimization.
Patent Information
- Application Number
- CN202411104848.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Traditional routing scheduling methods are difficult to respond quickly and adjust to adapt to complex and changeable network environments, and cannot effectively handle high-dimensional and complex network transmission paths.
Adopt the routing scheduling method based on deep reinforcement learning, monitor the network state through sensors and network management systems, design reasonable reward functions and neural network structures, build state space and action space, use deep neural networks to generate routing decisions, and optimize strategies through training Functions to adapt to network changes.
It realizes automatic learning and adjustment of routing strategies, improves the optimization ability of routing decisions, adapts to complex network environments, and improves network performance and reliability.
Smart Images

Figure CN118945100A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of routing scheduling methods, and in particular to a routing scheduling method based on deep reinforcement learning. Background Art
[0002] Routing scheduling methods refer to strategies and technologies for selecting and managing the transmission path of data packets from the source address to the destination address during network communication. This process involves the application of routing algorithms, the allocation of network resources, congestion control and other aspects to ensure that data can be transmitted efficiently and reliably in the network. However, when using routing scheduling methods, the transmission path of data packets is generally determined by a fixed routing algorithm, and the shortest path or optimal path is calculated by using the network topology and link status information. It is difficult to adapt to complex and changing network environments, and traditional routing algorithms may not be able to respond and adjust quickly. Summary of the invention
[0003] In order to overcome the problem that when using routing scheduling methods, fixed routing algorithms are generally used to determine the transmission path of data packets, and the network topology and link status information are used to calculate the shortest path or the optimal path. However, it is difficult to adapt to complex and changing network environments, and traditional routing algorithms may not be able to respond and adjust quickly.
[0004] The technical solution of the present invention is: a routing scheduling method based on deep reinforcement learning, including the following methods:
[0005] S11: First, the network status is fully perceived through monitoring tools such as sensors and network management systems;
[0006] S12: Select a suitable neural network structure;
[0007] S13: Design a reasonable reward function to evaluate the impact of the actions taken by the agent on the network performance;
[0008] S14: Collect and store data during network operation;
[0009] S15: Evaluate and deploy the router scheduling strategy;
[0010] S16: Monitor and provide feedback on the network operation status.
[0011] Preferably, when representing the network state of the router scheduling environment modeling, the following steps are included:
[0012] S21: Quantify the network state and convert it into a form that the agent can understand and process;
[0013] S22: Use the interaction between the SDN controller and network devices to obtain real-time network status information, and use network monitoring equipment to ensure the accuracy and real-time nature of the information.
[0014] Preferably, when constructing the state space for modeling the router scheduling environment, the following steps are included:
[0015] S31: Construct an appropriate state space based on the network size and complexity;
[0016] S32: Use feature selection, dimensionality reduction and other techniques to optimize the construction of the state space and improve the operating efficiency of the algorithm.
[0017] Preferably, when constructing the network architecture, the following method is included:
[0018] S41: Design a deep neural network as the core of the intelligent agent, including an input layer, a hidden layer and an output layer. The input layer receives network status information, the hidden layer performs feature extraction and transformation, and the output layer generates routing decisions.
[0019] S42: According to the specific application scenario, select different network architectures, such as CNN, RNN or their variants.
[0020] Preferably, through environmental modeling, the perceived network state can be abstracted into a form that can be understood by the deep reinforcement learning model, and the state space can contain sufficient information to support the intelligent agent to make reasonable decisions, which can facilitate the scheduling system to select the optimal route or scheduling action according to the current state.
[0021] Preferably, when adjusting the policy function, the following steps are included:
[0022] S51: Selecting an action based on the current state, representing the neural network, and optimizing it through training;
[0023] S52: The policy function selects the same action in a given state, or selects different actions with a certain probability in a given state.
[0024] Preferably, when defining the action space of the agent, the following aspects are included:
[0025] A11: The action space includes all the actions that the agent can choose. In the routing scheduling problem, the actions may include selecting the next hop node, adjusting the routing weight, optimizing resource allocation, etc.
[0026] A12: The design of the action space should take into account the actual needs and constraints of the network to ensure the feasibility and effectiveness of the actions.
[0027] Preferably, when determining the reward mechanism for the agent, the following steps are included:
[0028] S61: Design a reasonable reward mechanism to evaluate the behavior of the agent. The reward mechanism should be able to reflect the optimization goal of network performance.
[0029] S62: At the end of each time step, the reward value is calculated based on the change in network state and performance indicators, and fed back to the agent to guide its learning.
[0030] Preferably, by setting up a reward mechanism for router scheduling, it can be used to evaluate the impact of the actions taken by the intelligent agent on the network performance. The reward function can reflect the key performance indicators of the network. According to changes in the network environment and scheduling goals, the parameters of the reward function can be dynamically adjusted to optimize the performance of the router scheduling strategy.
[0031] Preferably, when training the agent, the following steps are included:
[0032] S71: Use deep reinforcement learning algorithms to train intelligent agents. The training process usually includes data collection, model training, strategy evaluation, and strategy update.
[0033] S72: During the training process, the agent optimizes its policy function through continuous trial and error and learning to maximize the long-term cumulative reward value;
[0034] S73: Use optimization algorithms such as gradient descent to update the weight parameters of the neural network.
[0035] Preferably, when making a decision on router scheduling, the following steps are included:
[0036] S81: The agent generates routing decisions in real time based on the status information of the current network environment;
[0037] S82: In some application scenarios, when multiple objectives need to be optimized simultaneously, multi-objective optimization techniques can be used to balance the conflicts and trade-offs between different objectives.
[0038] Preferably, when optimizing the router scheduling system, the following aspects are included:
[0039] A21: Improve the performance and reliability of routing scheduling by studying more efficient and stable deep reinforcement learning algorithms;
[0040] A22: Explore multi-agent collaboration mechanisms to achieve cross-domain or cross-layer routing scheduling optimization;
[0041] A13: Optimize the algorithm's computing performance and real-time performance to meet the needs of highly dynamic network environments.
[0042] Beneficial effects of the present invention:
[0043] 1. Compared with traditional routing scheduling methods, which generally use fixed routing algorithms to determine the transmission path of data packets, and use the network topology and link status information to calculate the shortest path or the optimal path, it is difficult to adapt to complex and changing network environments. Traditional routing algorithms may not be able to respond and adjust quickly. This routing scheduling method uses deep reinforcement learning. By combining the feature extraction ability of deep learning and the decision optimization ability of reinforcement learning, it can automatically learn and adjust routing strategies. It is more suitable for processing high-dimensional and complex network environments, and can extract more useful features and optimize routing decisions.
[0044] 2. By setting up a reward mechanism for router scheduling, it can be used to evaluate the impact of the actions taken by the agent on network performance. The reward function can reflect the key performance indicators of the network. According to changes in the network environment and scheduling goals, the parameters of the reward function can be dynamically adjusted to optimize the performance of the router scheduling strategy;
[0045] 3. Through environmental modeling, the perceived network state can be abstracted into a form that can be understood by the deep reinforcement learning model. The state space can contain enough information to support the intelligent agent to make reasonable decisions, which can facilitate the scheduling system to select the optimal route or scheduling action based on the current state. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 What is shown is a schematic diagram of the workflow of a routing scheduling method based on deep reinforcement learning of the present invention;
[0047] Figure 2 What is presented is a schematic diagram of the state space construction process of a routing scheduling method based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION
[0048] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0049] See also Figure 1-2 The present invention provides an embodiment: a routing scheduling method based on deep reinforcement learning, including the following methods:
[0050] S11: First, the network status is fully perceived through monitoring tools such as sensors and network management systems;
[0051] S12: Select a suitable neural network structure;
[0052] S13: Design a reasonable reward function to evaluate the impact of the actions taken by the agent on the network performance;
[0053] S14: Collect and store data during network operation;
[0054] S15: Evaluate and deploy the router scheduling strategy;
[0055] S16: Monitor and provide feedback on the network operation status.
[0056] Preferably, when representing the network state of the router scheduling environment modeling, the following steps are included:
[0057] S21: Quantify the network state and convert it into a form that the agent can understand and process;
[0058] S22: Use the interaction between the SDN controller and network devices to obtain real-time network status information, and use network monitoring equipment to ensure the accuracy and real-time nature of the information.
[0059] Preferably, when constructing the state space for modeling the router scheduling environment, the following steps are included:
[0060] S31: Construct an appropriate state space based on the network size and complexity;
[0061] S32: Use feature selection, dimensionality reduction and other techniques to optimize the construction of the state space and improve the operating efficiency of the algorithm.
[0062] Preferably, when constructing the network architecture, the following methods are included:
[0063] S41: Design a deep neural network as the core of the intelligent agent, including an input layer, a hidden layer and an output layer. The input layer receives network status information, the hidden layer performs feature extraction and transformation, and the output layer generates routing decisions.
[0064] S42: According to the specific application scenario, select different network architectures, such as CNN, RNN or their variants.
[0065] Preferably, through environmental modeling, the perceived network state can be abstracted into a form that can be understood by the deep reinforcement learning model, and the state space can contain sufficient information to support the intelligent agent to make reasonable decisions, which can facilitate the scheduling system to select the optimal route or scheduling action according to the current state.
[0066] Preferably, when adjusting the policy function, the following steps are included:
[0067] S51: Selecting an action based on the current state, representing the neural network, and optimizing it through training;
[0068] S52: The policy function selects the same action in a given state, or selects different actions with a certain probability in a given state.
[0069] Preferably, when defining the action space of the agent, the following aspects are included:
[0070] A11: The action space includes all the actions that the agent can choose. In the routing scheduling problem, the actions may include selecting the next hop node, adjusting the routing weight, optimizing resource allocation, etc.
[0071] A12: The design of the action space should take into account the actual needs and constraints of the network to ensure the feasibility and effectiveness of the actions.
[0072] Preferably, when determining the reward mechanism for the agent, the following steps are included:
[0073] S61: Design a reasonable reward mechanism to evaluate the behavior of the agent. The reward mechanism should be able to reflect the optimization goal of network performance.
[0074] S62: At the end of each time step, the reward value is calculated based on the change in network state and performance indicators, and fed back to the agent to guide its learning.
[0075] Preferably, by setting up a reward mechanism for router scheduling, it can be used to evaluate the impact of the actions taken by the intelligent agent on the network performance. The reward function can reflect the key performance indicators of the network. According to changes in the network environment and scheduling goals, the parameters of the reward function can be dynamically adjusted to optimize the performance of the router scheduling strategy.
[0076] Preferably, when training the agent, the following steps are included:
[0077] S71: Use deep reinforcement learning algorithms to train intelligent agents. The training process usually includes data collection, model training, strategy evaluation, and strategy update.
[0078] S72: During the training process, the agent optimizes its policy function through continuous trial and error and learning to maximize the long-term cumulative reward value;
[0079] S73: Use optimization algorithms such as gradient descent to update the weight parameters of the neural network.
[0080] Preferably, when making a decision on router scheduling, the following steps are included:
[0081] S81: The agent generates routing decisions in real time based on the status information of the current network environment;
[0082] S82: In some application scenarios, when multiple objectives need to be optimized simultaneously, multi-objective optimization techniques can be used to balance the conflicts and trade-offs between different objectives.
[0083] Preferably, when optimizing the router scheduling system, the following aspects are included:
[0084] A21: Improve the performance and reliability of routing scheduling by studying more efficient and stable deep reinforcement learning algorithms;
[0085] A22: Explore multi-agent collaboration mechanisms to achieve cross-domain or cross-layer routing scheduling optimization;
[0086] A13: Optimize the algorithm's computing performance and real-time performance to meet the needs of highly dynamic network environments.
[0087] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of those skilled in the art without departing from the spirit of the present invention.
Claims
1. A routing scheduling method based on deep reinforcement learning; characterized in that: The following methods are included: S11: First, the network status is fully perceived through monitoring tools such as sensors and network management systems; S12: Select a suitable neural network structure; S13: Design a reasonable reward function to evaluate the impact of the actions taken by the agent on the network performance; S14: Collect and store data during network operation; S15: Evaluate and deploy the router scheduling strategy; S16: Monitor and provide feedback on the network operation status.
2. A routing scheduling method based on deep reinforcement learning according to claim 1, characterized in that: When representing the network state of the router scheduling environment modeling, the following steps are included: S21: Quantify the network state and convert it into a form that the agent can understand and process; S22: Use the interaction between the SDN controller and network devices to obtain real-time network status information, and use network monitoring equipment to ensure the accuracy and real-time nature of the information.
3. A routing scheduling method based on deep reinforcement learning according to claim 2, characterized in that: When constructing the state space for modeling the router scheduling environment, the following steps are included: S31: Construct an appropriate state space based on the network size and complexity; S32: Use feature selection, dimensionality reduction and other techniques to optimize the construction of the state space and improve the operating efficiency of the algorithm.
4. A routing scheduling method based on deep reinforcement learning according to claim 3, characterized in that: When building the network architecture, the following methods are included: S41: Design a deep neural network as the core of the intelligent agent, including an input layer, a hidden layer and an output layer. The input layer receives network status information, the hidden layer performs feature extraction and transformation, and the output layer generates routing decisions. S42: According to the specific application scenario, select different network architectures, such as CNN, RNN or their variants.
5. A routing scheduling method based on deep reinforcement learning according to claim 4, characterized in that: When adjusting the policy function, the following steps are included: S51: Selecting an action based on the current state, representing the neural network, and optimizing it through training; S52: The policy function selects the same action in a given state, or selects different actions with a certain probability in a given state.
6. A routing scheduling method based on deep reinforcement learning according to claim 5, characterized in that: When defining the action space of an agent, the following aspects are included: A11: The action space includes all the actions that the agent can choose. In the routing scheduling problem, the actions may include selecting the next hop node, adjusting the routing weight, optimizing resource allocation, etc. A12: The design of the action space should take into account the actual needs and constraints of the network to ensure the feasibility and effectiveness of the actions.
7. The routing scheduling method based on deep reinforcement learning according to claim 6, characterized in that: When determining the reward mechanism for an agent, the following steps are included: S61: Design a reasonable reward mechanism to evaluate the behavior of the agent. The reward mechanism should be able to reflect the optimization goal of network performance. S62: At the end of each time step, the reward value is calculated based on the change in network state and performance indicators, and fed back to the agent to guide its learning.
8. The routing scheduling method based on deep reinforcement learning according to claim 7, characterized in that: When training an agent, the following steps are involved: S71: Use deep reinforcement learning algorithms to train intelligent agents. The training process usually includes data collection, model training, strategy evaluation, and strategy update. S72: During the training process, the agent optimizes its policy function through continuous trial and error and learning to maximize the long-term cumulative reward value; S73: Use optimization algorithms such as gradient descent to update the weight parameters of the neural network.
9. A routing scheduling method based on deep reinforcement learning according to claim 8, characterized in that: When making decisions about router scheduling, the following steps are included: S81: The agent generates routing decisions in real time based on the status information of the current network environment; S82: In some application scenarios, when multiple objectives need to be optimized simultaneously, multi-objective optimization techniques can be used to balance the conflicts and trade-offs between different objectives.
10. The routing scheduling method based on deep reinforcement learning according to claim 9, characterized in that: When optimizing the router scheduling system, the following aspects are included: A21: Improve the performance and reliability of routing scheduling by studying more efficient and stable deep reinforcement learning algorithms; A22: Explore multi-agent collaboration mechanisms to achieve cross-domain or cross-layer routing scheduling optimization; A13: Optimize the algorithm's computing performance and real-time performance to meet the needs of highly dynamic network environments.
Citation Information
Cited By
Routing optimization method, control device and medium
CN119544594A
Cross-group routing optimization method and device, and storage medium
CN120263716A
