Self-adaptive routing optimization method and system based on reinforcement learning multi-target path

Abstract

The invention provides a self-adaptive routing optimization method and system based on a reinforcement learning multi-target path. The method comprises the following steps: acquiring network routing related data; carrying out multi-objective path optimization modeling, and converting a network state and a service demand in the related data into an optimizable mathematical model; performing multi-target path optimization solution based on reinforcement learning, and learning an optimal routing decision strategy through training an intelligent agent by using a deep reinforcement learning DQN algorithm; and outputting the optimal routing path, and guiding the network equipment to carry out routing adjustment. According to the method, multi-target path optimization is realized, a plurality of QoS indexes such as bandwidth, time delay and packet loss rate can be considered at the same time, and differential requirements of different services are met.

Description

Technical Field

[0001] The present invention relates to the field of network communication technology, and in particular to an adaptive routing optimization method and system based on reinforcement learning multi-objective paths. Background Art

[0002] In modern network communications, with the diversification of business demands and the increasing complexity of network environments, traditional routing technologies have become unable to meet the demands of multi-objective optimization and dynamic adaptation. For example, in scenarios such as enterprise networks and data center networks, multiple Quality of Service (QoS) metrics, such as bandwidth, latency, and packet loss rate, must be considered simultaneously to ensure efficient transmission and stable operation of different services. However, existing routing technologies mostly optimize for a single objective and have limited adaptability to dynamic network changes, making it difficult to achieve multi-objective path optimization and real-time adaptive routing adjustments.

[0003] Traditional routing algorithms, such as Dijkstra and Bellman-Ford, primarily focus on calculating the shortest path, aiming to minimize path length or cost. However, they cannot simultaneously address multiple QoS metrics, such as bandwidth, latency, and packet loss rate. Furthermore, these algorithms require recalculating routes when network topology changes occur, resulting in slow convergence and an inability to respond promptly to dynamic network changes. This results in low routing efficiency and an inability to meet the demands of real-time services.

[0004] In order to solve the multi-objective routing optimization problem, a variety of routing technologies have been proposed and applied in practice. Among them, the multi-objective routing optimization method based on genetic algorithm can be considered as the closest existing technology to the technical solution of this patent because it attempts to solve similar multi-objective optimization problems and has a certain degree of self-adaptation. However, when applied to large-scale, dynamically changing modern network environments, genetic algorithms still expose many limitations, such as the easy generation of routing loops, slow convergence speed, and easy to fall into local optimal solutions, which makes it difficult to meet the requirements of modern networks for real-time, high efficiency and stability. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide an adaptive routing optimization method and system based on reinforcement learning multi-objective path.

[0006] According to the present invention, an adaptive routing optimization method based on reinforcement learning multi-objective paths is provided, comprising:

[0007] Step S1: Obtain network routing related data;

[0008] Step S2: multi-objective path optimization modeling, converting the network status and service requirements in the relevant data into an optimizable mathematical model;

[0009] Step S3: Multi-objective path optimization solution based on reinforcement learning, using the deep reinforcement learning DQN algorithm to learn the optimal routing decision strategy by training the intelligent agent;

[0010] Step S4: Output the optimal routing path and guide the network device to adjust the routing.

[0011] Preferably, the network routing related data includes network topology information, node and link status information, and service flow requirements.

[0012] Preferably, step S2 includes:

[0013] Step S2.1: Establish the underlying network model, communication traffic model, and traffic mapping model;

[0014] Step S2.2: Establish capacity constraints and formulate an optimization problem based on the capacity constraints.

[0015] Preferably, the step S2.1 includes:

[0016] Step S2.1.1: Establish an underlying network model, which is a weighted directed graph. The formula is as follows:

[0017]

[0018] Where, and ε represent the node and link sets respectively;

[0019] Step S2.1.2: Establish a communication flow model. Assume that there are M communication flows in the system. The set of flows is represented by To express, if the SFC of all flows in the network is the same, but the mapping nodes of different flows are different, it is necessary to distinguish the SFC of different flows. The SFC of flow k is as follows:

[0020]

[0021] in, is the mth VNF of flow k, that is, the virtual node of flow k, L k represents the total number of VNFs for flow k, and The flow between them is a virtual link represented by

[0022] Step S2.1.3: Establish a traffic mapping model and introduce node mapping variables Indicates whether node i provides flow k. If node i provides flow k but If node i does not provide but

[0023] Since only VNF Only nodes with The mapping target node, so there is

[0024]

[0025] As mentioned above, since VNFs of the same flow are not allowed to be mapped to different nodes, the following are the problems:

[0026]

[0027] Introducing link mapping variables Indicates the mapping relationship of virtual links, that is, Indicates a virtual link is mapped to the physical link (i, j), otherwise

[0028] for have:

[0029]

[0030] Preferably, the node set includes two types: nodes with VNF functions and ordinary forwarding nodes;

[0031] The node with VNF function is used to provide specific network functions;

[0032] The common forwarding node is only used for forwarding data packets;

[0033] Each flow can be mapped to only one physical path.

[0034] Preferably, for node i and link (i, j), the capacity constraints include:

[0035]

[0036] Among them, C i represents the computing power of node i, B ij 、D ij and A ij They represent the bandwidth capacity, maximum packet loss rate and delay tolerance of link (i, j) respectively. represents the computing resources required by node i to process flow k, represents the bandwidth resources consumed by flow k on physical link (i, j), represents the packet loss rate consumed by flow k on physical link (i, j), represents the delay consumed by flow k on physical link (i, j).

[0037] Preferably, said establishing the optimization problem includes defining a service cost, and modeling the communication resource reconfiguration problem according to said service cost;

[0038] The service cost formula is as follows:

[0039]

[0040] Where, φ c (·),φ b (·),φ d (·),φ a (·) denotes the price function of computing resources, bandwidth resources, packet loss rate, and latency, respectively;

[0041] The communication resource reconfiguration problem is modeled as follows:

[0042]

[0043] Preferably, step S3 includes:

[0044] Step S3.1: Construct a reinforcement learning environment. The environment of the reinforcement learning model corresponds to the modeling of the optimization problem and interacts with the agent. When the agent makes a decision action, the environment returns a state corresponding to the action. Based on this state, the reward value of the corresponding decision action can be obtained, thereby training the agent.

[0045] Step S3.2: Build an agent model. When the corresponding decision action is input into the environment and the corresponding environmental state is returned, the agent optimizes the parameters based on the reward value corresponding to the action and state at that time, thereby guiding the next action, until each corresponding environmental state can output the decision action with the highest reward value;

[0046] Step S3.3: Define a reward function, and use minimizing network congestion or maximizing network remaining resources as the optimization goal to reconfigure network service routing, i.e., the reward function;

[0047] Step S3.4: Train the intelligent agent, input network link traffic, business traffic demand, network spatiotemporal traffic analysis information and network status information into the intelligent agent, and the intelligent agent selects the business routing decision action with the largest corresponding reward value - Q value and inputs it into the environment. The environment reroutes according to the input decision action and updates the current environmental state.

[0048] Preferably, when training the intelligent agent, the data generated in each interaction process is stored in the memory for training the intelligent agent, and the parameters of the convolutional neural network and artificial neural network in the intelligent agent are updated using the stochastic gradient descent method until the model converges.

[0049] According to the present invention, an adaptive routing optimization system based on reinforcement learning multi-objective paths is provided, comprising:

[0050] Module M1: Obtain network routing related data;

[0051] Module M2: Multi-objective path optimization modeling, converting the network status and business requirements in the relevant data into an optimizable mathematical model;

[0052] Module M3: Multi-objective path optimization solution based on reinforcement learning, using the deep reinforcement learning DQN algorithm to learn the optimal routing decision strategy by training intelligent agents;

[0053] Module M4: Outputs the optimal routing path and guides network devices to make routing adjustments.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. The present invention realizes multi-objective path optimization, which can simultaneously consider multiple QoS indicators such as bandwidth, delay, and packet loss rate to meet the differentiated needs of different services.

[0056] 2. The present invention improves the adaptive ability of the routing algorithm, can quickly respond to changes in network topology and traffic demand, realize real-time routing adjustment, and improves the convergence speed of the routing algorithm, reduces the time of routing calculation, and meets the real-time requirements of the network.

[0057] 3. The present invention avoids the generation of routing loops, ensures the availability of routing and the stability of the network, overcomes the problem of local optimal solutions, can find the global optimal routing path, and improves the overall performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0059] Figure 1 A schematic diagram of the network routing planning process based on reinforcement learning;

[0060] Figure 2 This is the on-demand routing scenario graph in the present invention;

[0061] Figure 3 is the reinforcement learning decision graph in the present invention;

[0062] Figure 4 This is the core principle diagram of the deep reinforcement learning DQN algorithm in the present invention;

[0063] Figure 5This is a diagram of an enterprise's data center network adaptive routing strategy in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention is described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, without departing from the scope of the present invention, a number of variations and improvements may be made by those skilled in the art. These all fall within the scope of protection of the present invention.

[0065] Example 1

[0066] According to the present invention, an adaptive routing optimization method based on reinforcement learning multi-objective path is provided. Figure 1 and Figure 2 Shown, including:

[0067] Step S1: Acquire network routing related data, including network topology information, node and link status information, service flow requirements, etc.

[0068] Step S2: Multi-objective path optimization modeling, which converts network status and service requirements into an optimizable mathematical model to provide a basis for subsequent path optimization. Step S2 includes:

[0069] Step S2.1: Establishing the underlying network model, communication traffic model, and traffic mapping model. Step S2.1 includes:

[0070] Step S2.1.1: Establish an underlying network model, which is a weighted directed graph. The formula is as follows:

[0071]

[0072]

[0073] Where v and ε represent the node and link set respectively. The node set includes two types: nodes with VNF functions and ordinary forwarding nodes. The nodes with VNF functions (Virtual Network Function) are used to provide specific network functions, such as Mobility Management Entity (MME), Network Address Translation (NAT), firewall, etc. The ordinary forwarding nodes are only used for forwarding data packets. Assume that all VNF functions that can be provided in the system are obtained as the node set 0-1 variable h i (π)∈{0,1} is used to mark nodes Does it have network functionality?

[0074] Step S2.1.2: Establish a communication flow model. Assume that there are M communication flows in the system. The set of flows is represented by To express it, if the SFC of all flows in the network is the same, but the mapping nodes of different flows may be different, it is still necessary to distinguish the SFC (Service Function Chain) of different flows. The SFC of flow k is as follows:

[0075]

[0076] in, is the mth VNF of flow k, that is, the virtual node of flow k, L k Similarly, we can express the total number of VNFs for flow k by and The flow between is called a virtual link and is represented as The kth flow is recorded as (s k , d k , μ k (t)), where s k , d k , μ k (t) represents the source node and destination node of the kth flow, and the traffic demand of the kth flow at time t, respectively, where t represents time. Each flow is only allowed to be mapped to one physical path to avoid resource coordination overhead caused by flow splitting. In addition, the communication traffic model considers dynamic traffic demand.

[0077] Step S2.1.3: Establish a traffic mapping model, consider the node mapping of communication flow k, and introduce 0-1 variables Used to indicate whether node o provides flow k That is, if node i provides but If node o does not provide but

[0078] Since only Only nodes with The mapping target node, so there is

[0079]

[0080] As mentioned above, since VNFs of the same flow are not allowed to be mapped to different nodes, the following are the problems:

[0081]

[0082] Since the source node and destination node of flow k are pre-determined by s k and d k Therefore, only the mapping of other nodes needs to be considered. For the node mapping of flow k, considering that the SFC of flow k is composed of multiple VNFs in sequence, it is necessary to map each node in turn. To this end, a 0-1 variable is introduced. To express the mapping relationship of virtual links, that is Indicates a virtual link is mapped to the physical link (i, j), otherwise Among them, the variable It is also used to ensure that the flow is not split, which is guaranteed by the flow conservation relationship on each physical node. Specifically, at the source node s k and destination node d k Above, there are

[0083]

[0084] and

[0085]

[0086] in Is a dummy virtual node used to represent the source node s k . The flow conservation relationship on other physical nodes and node mapping variables It can be seen that if the virtual node is mapped to physical node i, then the virtual link It will be mapped to a physical path with i as the end point, and the virtual link It will also be mapped to a physical path starting from i. This means that have:

[0087]

[0088] Step S2.2: Establish capacity constraints and formulate an optimization problem based on the capacity constraints.

[0089] The capacity constraint is established by assuming that the computational resources and bandwidth resources consumed by each flow are proportional to the data rate of the flow. Therefore, it is assumed that one unit of data flow of VNF type π consumes α(π) units of computational resources. In addition, the forwarding function is also considered as a special VNF, and the computational resources consumed per unit rate can be recorded as α(π) f ). Similarly, the bandwidth resource consumed by a flow of unit flow rate is ω, the latency resource consumed is α, and the packet loss rate resource consumed is β. Then the computing resources required by physical node i to process flow k are:

[0090]

[0091] The bandwidth resources consumed by flow k on physical link (i, j) are:

[0092]

[0093] The packet loss rate consumed by flow k on physical link (i, j) is:

[0094]

[0095] The delay consumed by flow k on physical link (i, j) is:

[0096]

[0097] Then, the computing power of node i, the bandwidth capacity of link (i, j), the maximum packet loss rate and the delay tolerance of the service are C respectively. i , B ij , D ij , A ij Then for node i and link (i, j), there are the following four capacity constraints:

[0098]

[0099] and

[0100]

[0101] and

[0102]

[0103] and

[0104]

[0105] It is worth noting that for different communication flows, C i , B ij It is also different. High-priority communication flows regard the computing and bandwidth resources occupied by low-priority flows as available resources. Therefore, when higher-priority flows need them, the resources of low-priority flows can be released to meet the allocation needs of high-priority resources.

[0106] The optimization problem is to adapt to the dynamic nature of network traffic demand and to adaptively reconfigure the network according to business needs to minimize resource consumption and save transmission delay and packet loss tolerance. The resources consumed by the service communication task are equal to the sum of the resources consumed by all flows in the network. In order to define the service cost, the price functions of computing resources, bandwidth resources, packet loss rate and delay are designed as φ respectively.c (·),φ b (·),φ d (·),φ a (·)Then the service price is:

[0107]

[0108] Finally, the communication resource reconfiguration problem can be modeled as:

[0109]

[0110] Since this problem is essentially a sequential decision problem, the long-term cost is used as the optimization objective in this invention. Constraints (1) and (2) ensure that each VNF of each flow is served by at most one physical node; constraints (3)-(5) are flow conservation constraints; constraints (6)-(13) are the capacity constraints of nodes and links respectively; constraints (15) and (16) are the node mapping variables and link map variables The 0-1 constraint.

[0111] Since the above problem is an integer linear programming problem, it is an NP-hard problem and currently cannot be explicitly solved within polynomial time complexity. Considering that the optimization of long-term goals is very suitable for solving using reinforcement learning, the present invention ultimately uses deep reinforcement learning to solve it.

[0112] Step S3: Multi-objective path optimization solution based on reinforcement learning, using deep reinforcement learning DQN algorithm, such as Figure 4 As shown in , the optimal routing decision strategy is learned by training the intelligent agent. This step can quickly and accurately output the optimal routing path according to the changes in network status. Figure 3 As shown, step S3 includes:

[0113] Step S3.1: Construct a reinforcement learning environment. The environment of the reinforcement learning model corresponds to the modeling of the optimization problem and interacts with the agent. When the agent takes a corresponding decision action, the environment returns a state corresponding to that action. Based on this state, the reward value for the corresponding decision action can be obtained, thereby training the agent. For the intelligent reconstruction problem of communication networks, the corresponding reinforcement learning environment is the communication network, including relevant information such as network node location information, link connectivity, and resource allocation.

[0114] Step S3.2: Build the agent model. The reinforcement learning agent is the part that makes decisions. When a decision action is input into the environment and the corresponding environmental state is returned, the agent optimizes its parameters based on the reward value corresponding to the action and state at that time, thus guiding the next action. The optimal decision problem is solved until the decision action with the highest reward value is output for each corresponding environmental state.

[0115] Step S3.3: Define the reward function. The reward function is a crucial component of reinforcement learning. It represents the decision-making objective of the reinforcement learning model. When the agent's decision-making action makes the corresponding state of the environment more closely aligned with the optimization objective of the decision problem, the greater the reward function value, and the higher the probability that the agent will choose that action next time. Reconstructing network service routing with the optimization objective of minimizing network congestion and maximizing remaining network resources is performed, i.e., the reward function. After adjustment, the lower the network congestion and the greater the remaining network resources, the greater the corresponding reward value. Conversely, the greater the network congestion and the lower the remaining network resources, the smaller the corresponding reward value.

[0116] Step S3.4: Train the agent. Network link traffic, service traffic demand, network spatiotemporal traffic analysis, and network status information are input into the agent. The agent selects the service routing decision action with the highest reward value (Q-value) and inputs it into the environment. The environment reroutes based on the input decision action and updates the current state of the environment. The data generated during each interaction is stored in memory for agent training. Stochastic gradient descent is used to update the parameters of the convolutional neural network and artificial neural network in the agent until the model converges.

[0117] Step S4: Output the optimal routing path and guide the network device to adjust the routing.

[0118] The service routing optimization decision-making process of the present invention includes inputting network status information: network link traffic, service traffic demand, and network spatiotemporal traffic analysis information into the agent. The agent then outputs the optimal routing decision: based on the current network status and trained strategies, the agent selects the service routing decision action with the highest reward. The environment then reroutes and updates the state: based on the agent's decision action, the environment reroutes and updates the network state, completing the routing optimization process.

[0119] Furthermore, the adaptive routing optimization method based on reinforcement learning multi-objective paths of the present invention is described in detail as follows in combination with the application scenario:

[0120] Application Scenario 1: Network Optimization of an Enterprise Data Center

[0121] In an enterprise's data center network, service traffic is complex and diverse, including data backup, virtual machine migration, and online transaction processing. These services have different requirements for network QoS metrics such as bandwidth, latency, and packet loss rate. For example, data backup requires high bandwidth and low packet loss rate to ensure data integrity and transmission efficiency; virtual machine migration is sensitive to latency, as excessive latency can cause migration failure; and online transaction processing requires low latency and high reliability to ensure real-time and accurate transactions.

[0122] Using reinforcement learning-based multi-objective path optimization adaptive routing technology, the system first collects real-time information about network topology, node and link status, and traffic demands of various services. The multi-objective path optimization modeling submodule then transforms this information into a mathematical model, establishing the underlying network model, communication traffic model, and traffic mapping model, and formulating an optimization problem based on capacity constraints. Next, the reinforcement learning-based multi-objective path optimization solution submodule utilizes a deep reinforcement learning algorithm to train the intelligent agent to learn the optimal routing decision strategy.

[0123] like Figure 5 As shown in the figure, during actual operation, the agent dynamically adjusts routing paths based on the current network status and business needs. For example, when data backup is initiated, the agent selects a path with sufficient bandwidth and low packet loss for data transmission. During virtual machine migration, the agent prioritizes the path with the lowest latency to ensure smooth migration. For online transaction processing, the agent comprehensively considers latency and reliability to select the optimal routing path, ensuring real-time and accurate transactions. By monitoring network status and changes in business needs in real time, the agent can quickly and accurately adjust routing, ensuring the rational allocation and utilization of network resources, and improving the overall performance and service quality of the data center network.

[0124] The adaptive routing strategy based on reinforcement learning selects the optimal link suitable for the characteristics of different business data for data transmission.

[0125] Application Scenario 2: Intelligent Traffic Signal Control System Optimization

[0126] In an intelligent traffic signal control system, traffic flow at an intersection fluctuates in real time, with significant variations in vehicle, pedestrian, and non-motorized vehicle traffic at different time periods. Furthermore, special events (such as traffic accidents and road construction) can also impact traffic flow. The traffic signal control system needs to optimize signal timing based on real-time traffic flow and road conditions to improve traffic efficiency and reduce congestion.

[0127] The reinforcement learning-based multi-objective path optimization adaptive routing technology is applied to an intelligent traffic signal control system. The system first collects traffic flow data, road condition information, and special event information at each intersection. The multi-objective path optimization modeling submodule then transforms this information into a mathematical model, establishing a traffic flow model and a signal control model, and formulating an optimization problem based on road capacity constraints. The reinforcement learning-based multi-objective path optimization solution submodule uses a deep reinforcement learning algorithm to train an intelligent agent to learn the optimal signal control strategy.

[0128] During actual operation, the agent dynamically adjusts traffic light timing based on real-time traffic flow and road conditions. For example, during the morning rush hour, when traffic at a certain intersection is heavy, the agent will extend the green light duration at that intersection to improve traffic efficiency. During periods of high pedestrian crossing demand, the agent will appropriately increase the green light duration of pedestrian signals to ensure safe passage. When a traffic accident causes traffic congestion, the agent will adjust the signal timing at surrounding intersections based on the accident scene and surrounding road conditions, directing vehicles to detours and alleviating congestion. By monitoring changes in traffic flow and road conditions in real time, the agent can quickly and accurately optimize signal control, improving the adaptability and overall performance of the intelligent traffic signal control system.

[0129] The purpose of this invention is to provide a routing optimization method that can simultaneously consider multiple QoS metrics, achieving multi-objective path optimization. This method improves the routing algorithm's adaptability to dynamic network changes, enabling rapid and accurate routing adjustments. It also avoids routing loops, ensuring routing availability and network stability. It also improves the routing algorithm's convergence speed, meeting real-time network requirements. It also overcomes the problem of genetic algorithms easily falling into local optimal solutions, thereby finding the globally optimal routing path.

[0130] Example 2

[0131] The present invention also provides an adaptive routing optimization system based on reinforcement learning multi-objective paths. The adaptive routing optimization system based on reinforcement learning multi-objective paths can be implemented by executing the process steps of the adaptive routing optimization method based on reinforcement learning multi-objective paths. That is, those skilled in the art can understand the adaptive routing optimization method based on reinforcement learning multi-objective paths as a preferred implementation of the adaptive routing optimization system based on reinforcement learning multi-objective paths.

[0132] According to the present invention, an adaptive routing optimization system based on reinforcement learning multi-objective paths is provided, comprising:

[0133] Module M1: Acquire network routing related data, including network topology information, node and link status information, and service flow requirements.

[0134] Module M2: Multi-objective path optimization modeling, converting the network status and business requirements in the relevant data into an optimizable mathematical model. Module M2 includes: Module M2.1: Establishing the underlying network model, communication traffic model, and traffic mapping model. Module M2.1 includes: Module M2.1.1: Establishing the underlying network model, which is a weighted directed graph, with the following formula:

[0135]

[0136]

[0137] Where, and ε represent the node and link sets respectively. The node set includes two types: nodes with VNF functions and ordinary forwarding nodes. The nodes with VNF functions are used to provide specific network functions. The ordinary forwarding nodes are only used to forward data packets. Each flow is only allowed to be mapped to one physical path. Module M2.1.2: Establish a communication traffic model. Suppose there are M communication flows in the system. The flow set is represented by To express, if the SFC of all flows in the network is the same, but the mapping nodes of different flows are different, it is necessary to distinguish the SFC of different flows. The SFC of flow k is as follows:

[0138]

[0139] in, is the mth VNF of flow k, that is, the virtual node of flow k, L k represents the total number of VNFs for flow k, and The flow between them is a virtual link represented by Module M2.1.3: Establish a traffic mapping model and introduce node mapping variables Indicates whether node i provides flow k. If node i provides flow k but If node i does not provide but Since only Only nodes with The mapping target node, so there is

[0140]

[0141] As mentioned above, since VNFs of the same flow are not allowed to be mapped to different nodes, the following are the problems:

[0142]

[0143] Introducing link mapping variables Indicates the mapping relationship of virtual links, that is, Indicates a virtual link is mapped to the physical link (i, j), otherwise for have:

[0144]

[0145] Module M2.2: Establish capacity constraints and formulate optimization problems based on them. For node i and link (i, j), the capacity constraints include:

[0146]

[0147] Among them, C i represents the computing power of node i, B ij 、D ij and A ij They represent the bandwidth capacity, maximum packet loss rate and delay tolerance of link (i, j) respectively. represents the computing resources required by node i to process flow k, represents the bandwidth resources consumed by flow k on physical link (i, j), represents the packet loss rate consumed by flow k on physical link (i, j), represents the delay consumed by flow k on physical link (i, j). The establishment of the optimization problem includes defining a service cost and modeling the communication resource reconfiguration problem based on the service cost. The service cost formula is as follows:

[0148]

[0149] Where, φ c (·),φ b (·),φ d (·),φ a (·) represents the price function of computing resources, bandwidth resources, packet loss rate, and latency, respectively. The communication resource reconfiguration problem is modeled as follows:

[0150]

[0151] Module M3: Multi-objective path optimization solution based on reinforcement learning, using deep reinforcement learning DQN algorithm, to learn the optimal routing decision strategy by training the intelligent agent. The module M3 includes: Module M3.1: Constructing a reinforcement learning environment. The environment of the reinforcement learning model corresponds to the modeling of the optimization problem and interacts with the intelligent agent. When the intelligent agent gives a corresponding decision action, the environment will return a state corresponding to the action. According to the state at this time, the reward value of the corresponding decision action can be obtained, thereby training the intelligent agent. Module M3.2: Constructing an intelligent agent model. When the corresponding decision action is input into the environment and the corresponding environmental state is returned, the intelligent agent optimizes the parameters according to the reward value corresponding to the action and state at this time, thereby guiding the next action, until each corresponding environmental state can output the decision action with the highest reward value. Module M3.3: Define the reward function, and use minimizing network congestion / maximizing network remaining resources as the optimization goal to reconstruct the network service routing, that is, the reward function. Module M3.4: Training the Agent: Network link traffic, service traffic requirements, spatiotemporal traffic analysis, and network status information are input into the agent. The agent selects the service routing decision action with the highest reward value (Q) and inputs it into the environment. The environment then reroutes based on the input decision action and updates the current state. During agent training, the data generated by each interaction process is stored in memory for agent training. Stochastic gradient descent is used to update the parameters of the convolutional neural network and artificial neural network in the agent until the model converges.

[0152] Module M4: Outputs the optimal routing path and guides network devices to make routing adjustments.

[0153] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0154] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. An adaptive routing optimization method based on reinforcement learning multi-objective path, characterized in that: include: Step S1: Obtain network routing related data; Step S2: multi-objective path optimization modeling, converting the network status and service requirements in the relevant data into an optimizable mathematical model; Step S3: Multi-objective path optimization solution based on reinforcement learning, using the deep reinforcement learning DQN algorithm to learn the optimal routing decision strategy by training the intelligent agent; Step S4: Output the optimal routing path and guide the network device to adjust the routing.

2. The adaptive routing optimization method based on reinforcement learning multi-objective path according to claim 1, characterized in that: The network routing related data includes network topology information, node and link status information, and service flow requirements.

3. The adaptive routing optimization method based on reinforcement learning multi-objective path according to claim 1, characterized in that: The step S2 comprises: Step S2.1: Establish the underlying network model, communication traffic model, and traffic mapping model; Step S2.2: Establish capacity constraints and formulate an optimization problem based on the capacity constraints.

4. The adaptive routing optimization method based on reinforcement learning multi-objective path according to claim 3, characterized in that: The step S2.1 includes: Step S2.1.1: Establish an underlying network model, which is a weighted directed graph. The formula is as follows: Where, and ε represent the node and link sets, respectively; Step S2.1.2: Establish a communication flow model. Assume that there are M communication flows in the system. The set of flows is represented by To express, if the SFC of all flows in the network is the same, but the mapping nodes of different flows are different, it is necessary to distinguish the SFC of different flows. The SFC of flow k is as follows: in, is the mth VNF of flow k, that is, the virtual node of flow k, L k represents the total number of VNFs for flow k, and The flow between them is a virtual link represented by Step S2.1.3: Establish a traffic mapping model and introduce node mapping variables Indicates whether node i provides flow k. If node i provides flow k but If node i does not provide but Since only Only nodes with The mapping target node, so there is As mentioned above, since VNFs of the same flow are not allowed to be mapped to different nodes, the following are the problems: Introducing link mapping variables Indicates the mapping relationship of virtual links, that is, Indicates a virtual link is mapped to the physical link (i, j), otherwise for have:

5. The adaptive routing optimization method based on reinforcement learning multi-objective path according to claim 4, characterized in that: The node set includes two types: nodes with VNF functions and ordinary forwarding nodes; The node with VNF function is used to provide specific network functions; The common forwarding node is only used for forwarding data packets; Each flow can be mapped to only one physical path.

6. The adaptive routing optimization method based on reinforcement learning multi-objective paths according to claim 3, characterized in that: For node i and link (i, j), the capacity constraints include: Among them, C i represents the computing power of node i, B ij 、D ij and A ij They represent the bandwidth capacity, maximum packet loss rate and delay tolerance of link (i, j) respectively. represents the computing resources required by node i to process flow k, represents the bandwidth resources consumed by flow k on physical link (i, j), represents the packet loss rate consumed by flow k on physical link (i, j), represents the delay consumed by flow k on physical link (i, j).

7. The adaptive routing optimization method based on reinforcement learning multi-objective path according to claim 3, characterized in that: The establishing of the optimization problem includes defining a service cost and modeling the communication resource reconfiguration problem according to the service cost; The service cost formula is as follows: Where, φ c (·),φ b (·),φ d (·),φ a (·) denotes the price function of computing resources, bandwidth resources, packet loss rate, and latency, respectively; The communication resource reconfiguration problem is modeled as follows:

8. The adaptive routing optimization method based on reinforcement learning multi-objective paths according to claim 1, characterized in that: The step S3 comprises: Step S3.1: Construct a reinforcement learning environment. The environment of the reinforcement learning model corresponds to the modeling of the optimization problem and interacts with the agent. When the agent makes a decision action, the environment returns a state corresponding to the action. Based on this state, the reward value of the corresponding decision action can be obtained, thereby training the agent. Step S3.2: Build an agent model. When the corresponding decision action is input into the environment and the corresponding environmental state is returned, the agent optimizes the parameters based on the reward value corresponding to the action and state at that time, thereby guiding the next action, until each corresponding environmental state can output the decision action with the highest reward value; Step S3.3: Define a reward function, and use minimizing network congestion or maximizing network remaining resources as the optimization goal to reconfigure network service routing, i.e., the reward function; Step S3.4: Train the intelligent agent, input network link traffic, business traffic demand, network spatiotemporal traffic analysis information and network status information into the intelligent agent, and the intelligent agent selects the business routing decision action with the largest corresponding reward value - Q value and inputs it into the environment. The environment reroutes according to the input decision action and updates the current environmental state.

9. The adaptive routing optimization method based on reinforcement learning multi-objective paths according to claim 8, characterized in that: When training the intelligent agent, the data generated by each interaction process is stored in the memory for the training of the intelligent agent, and the parameters of the convolutional neural network and artificial neural network in the intelligent agent are updated using the stochastic gradient descent method until the model converges.

10. An adaptive routing optimization system based on reinforcement learning multi-objective path, characterized in that: include: Module M1: Obtain network routing related data; Module M2: Multi-objective path optimization modeling, converting the network status and business requirements in the relevant data into an optimizable mathematical model; Module M3: Multi-objective path optimization solution based on reinforcement learning, using the deep reinforcement learning DQN algorithm to learn the optimal routing decision strategy by training intelligent agents; Module M4: Outputs the optimal routing path and guides network devices to make routing adjustments.

Images