Multi-agent segment routing system based on online telemetering

Abstract

The invention provides a multi-agent segmented routing system based on online telemetering, and is applied to the technical field of network intelligent routing optimization. The system comprises network control equipment and a plurality of intelligent agents, the network control equipment is used for constructing a network topology model and deploying a plurality of intelligent agents at different network nodes; the intelligent agent is used for initializing a decision network, an experience storage unit and training parameters of the intelligent agent; acquiring real-time link state information through an online telemetering mechanism to obtain local observation data; planning a segmented routing path based on local observation data and a decision network; traffic forwarding operation is executed in a segmented routing mode, and action, reward and state information in the interaction process is automatically recorded to an experience storage unit; and the network control equipment is used for iteratively updating decision network parameters and training parameters of the intelligent agents through centralized training based on the data in the experience storage units of the intelligent agents, and optimizing decision strategies of the intelligent agents so as to realize load balancing of links of the whole network.

Description

Technical Field

[0001] This invention relates to the field of network intelligent routing optimization technology, and in particular to a multi-agent segmented routing system based on in-network telemetry. Background Technology

[0002] In scenarios such as large-scale data centers and 5G networks, networks need to handle explosively increasing traffic while meeting the requirements of low latency and high reliability. As a core network function, traffic routing requires precise planning of data packet transmission paths, balancing link loads to avoid congestion, and ensuring the efficient operation of the overall network. This places extremely high demands on the dynamic adaptability and state awareness accuracy of routing mechanisms.

[0003] In existing technologies, path planning often adopts a centralized control architecture, which collects the network topology and link status through a central controller and calculates forwarding paths in a unified manner; link status awareness relies on periodic sampling or polling of protocols such as NetFlow and SNMP to obtain information such as link throughput and latency to assist in routing decisions.

[0004] However, such solutions have significant drawbacks: On the one hand, under a centralized architecture, the computational complexity and communication overhead of the controller increase exponentially with the expansion of the network scale. When faced with concurrent multi-service flows and dynamic topology changes, decision delays and control plane congestion are likely to occur, making it impossible to meet real-time routing requirements. Furthermore, there is a risk of single point of failure, resulting in poor scalability. On the other hand, the perception methods of periodic sampling and polling are coarse-grained and have low timeliness, making it difficult to capture fine-grained state changes such as sudden congestion and microsecond-level failures. This can easily lead to a disconnect between path decisions and the actual network state, causing link overload or resource waste, and making it unsuitable for the operational needs of large-scale, highly dynamic networks. Summary of the Invention

[0005] This invention provides a multi-agent segmented routing system based on in-network telemetry to solve the problems of complex centralized routing calculations and decision delays, as well as the coarse granularity and poor timeliness of traditional link awareness. It can realize parallelization of path decision-making and real-time state awareness in large-scale, highly dynamic networks, reduce maximum link utilization, and improve link load balancing and overall network operating efficiency.

[0006] This invention provides a multi-agent segmented routing system based on in-network telemetry, comprising: a network control device and multiple agents; the network control device is used to construct a network topology model and deploy multiple agents at different network nodes, with each network node corresponding to at least one agent; each agent is used to initialize its own decision network, experience storage unit, and training parameters; obtain local observation data by collecting real-time link status information through the in-network telemetry mechanism; plan segmented routing paths based on the local observation data and the decision network; perform traffic forwarding operations through segmented routing, and automatically record the actions, rewards, and status information during the interaction process to the experience storage unit; the network control device is used to optimize the decision strategy of the agents by iteratively updating the decision network parameters and training parameters of each agent through centralized training based on the data in the experience storage unit of each agent, thereby achieving network-wide link load balancing.

[0007] According to the present invention, a multi-agent segmented routing system based on in-network telemetry is provided, wherein the network topology model is an undirected graph. ;in, Represents the set of vertices, containing all nodes in the network; This represents the set of edges, containing all links between nodes; For the set of link capacities, Indicates link Maximum load capacity; For the set of link weights, Indicates link Priority or transmission cost.

[0008] According to the present invention, a multi-agent segmented routing system based on online telemetry is provided, wherein the network control device is further configured to establish a traffic matrix, the traffic matrix being used to record the traffic demand between a source node and a destination node, wherein the source node and the destination node are both nodes in the vertex set.

[0009] According to the present invention, a multi-agent segmented routing system based on online telemetry is provided. The step of planning segmented routing paths based on the local observation data and the decision network includes: inputting the local observation data into the decision network, outputting the decision priority of each candidate segmented routing path through the decision network; filtering candidate paths by combining traffic conservation constraints and link capacity constraints, and combining path segments with link utilization rates lower than a preset threshold and satisfying the constraints to obtain the final segmented routing path.

[0010] According to the present invention, a multi-agent segmented routing system based on online telemetry is provided, wherein the flow conservation constraint is for any flow and any node ,satisfy: ; in, Represents a stream Does it flow through the link? ; Represents a stream Does it flow through the link? ; Represents a stream At the node The flow conservation indicator at the location, For flow When the source node ,exist For flow When the destination node is ,exist For flow The intermediate node is ; Represents a stream It is the set of all streams Any one of the streams; Represents a node It is a set of vertices Any node in it.

[0011] According to the present invention, a multi-agent segmented routing system based on in-network telemetry is provided, wherein the link capacity constraint is for any link ,satisfy: ; in, Indicates on the link Total bandwidth consumed by all streams Represents a stream In the link The transmission ratio on Represents a stream Its own traffic volume; It is a link Maximum load capacity; Indicates link For the set of edges Any link in it.

[0012] According to the present invention, a multi-agent segmented routing system based on online telemetry is provided, wherein the agents are further configured to use auxiliary variables. Indicates link Is it occupied? when When greater than 0, auxiliary variable =1; when When not greater than 0, auxiliary variable =0; Among them, auxiliary variables The following constraints must be met: .

[0013] According to the present invention, a multi-agent segmented routing system based on online telemetry is provided, wherein the routing decision problem of the agents is formalized as a partially observable Markov decision process. ;in, For the set of network environment states, For the agent's local observation set, For the segmented routing decision action set of the intelligent agent, For the reward function, This is the state transition function. The discount factor; the agent is based on the local observations. Execution of decision-making actions Through the state transition function The network state is updated by the reward function. Feedback on decision-making effectiveness drives iterative optimization of decision-making strategies.

[0014] According to the present invention, a multi-agent segmented routing system based on online telemetry is provided, wherein the feedback value of the reward function is negatively correlated with the link load balancing degree. When the segmented routing path planned by the agent makes the maximum link utilization of the corresponding link decrease, the reward function increases; when the maximum link utilization increases, the reward function decreases.

[0015] According to the present invention, a multi-agent segmented routing system based on in-network telemetry is provided, wherein the network control device optimizes the decision-making strategy of the agents with the objective of minimizing the maximum link utilization of the entire network. : ; ; in, Indicates a single link utilization rate Indicates all links that pass through The total flow Indicates a single link Maximum load capacity.

[0016] The multi-agent segmented routing system based on in-network telemetry provided by this invention, on the one hand, adopts a distributed deployment architecture, with at least one agent deployed on each network node. Each agent independently performs segmented routing path planning and traffic forwarding operations, without relying on a central controller for unified network-wide path calculation. This effectively distributes the computational burden of routing decisions, avoiding the exponential increase in computational complexity and communication overhead caused by network expansion in centralized architectures. It also reduces decision latency and control plane congestion risks, improving system scalability and resistance to single points of failure. On the other hand, because the agents collect real-time link status information through in-network telemetry mechanisms… Instead of traditional periodic sampling or polling methods, it can accurately capture fine-grained changes in link status, solving the shortcomings of traditional link perception in terms of coarse granularity and poor timeliness. This ensures that path decisions match the actual network status in real time, reducing link overload and resource waste. Furthermore, since the network control device optimizes decision-making strategies by iteratively updating the decision network parameters and training parameters of each agent through centralized training based on the experience storage unit data of each agent, and the agents plan segmented routing paths based on the decision network, the load on the entire network links can be balanced, significantly reducing the maximum link utilization and improving the overall operating efficiency and service quality of large-scale, highly dynamic networks. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of the multi-agent segmented routing system based on online telemetry provided by the present invention; Figure 2 This is a schematic diagram of the processing flow of agents in a multi-agent segmented routing system based on in-network telemetry provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0021] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0022] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0023] This application describes some exemplary embodiments for illustrative purposes. It should be understood that this application may be implemented in other ways not specifically shown in the accompanying drawings.

[0024] like Figure 1 As shown in the figure, this application provides a multi-agent segmented routing system based on online telemetry, which may include: a network control device and multiple agents. The network control device may include a topology modeling module, an agent deployment module, an experience receiving module, and a centralized training module.

[0025] The network control device is used to construct a network topology model through a topology modeling module. This topology modeling module can transmit network structure data to the agent deployment module to support the agent deployment module in deploying multiple agents on different network nodes, with one network node corresponding to at least one agent. For example, node 1 corresponds to agent 1, node 2 corresponds to agent 2, and node n corresponds to agent n.

[0026] The intelligent agent is used to initialize its own decision network, experience storage unit, and training parameters; to obtain local observation data by collecting real-time link status information through the network telemetry mechanism; to plan segmented routing paths based on the local observation data and the decision network; to perform traffic forwarding operations through segmented routing, and to automatically record the actions, rewards, and status information during the interaction process to the experience storage unit.

[0027] like Figure 2 As shown, the agent can first perform the operation of "initializing the decision network, experience storage unit and training parameters", and then enter the round loop to determine "whether the round has ended". If the round has not ended, it performs the operation of "collecting real-time link status information through the network telemetry mechanism to obtain local observation data", and then determines "whether the path of all traffic is fully planned". If the path is not fully planned, it performs traffic forwarding operation through segmented routing and observes the results, and records the action, status and other information into the experience storage unit. If the path has been fully planned, it calculates the reward and updates the decision strategy, and optimizes its own decision logic with the help of experience data. After completion, it returns to the judgment of "whether the round has ended" until the round ends.

[0028] The network control device is used to collect data from the experience storage units of each agent through the experience receiving module, and transmit these data to the centralized training module. The centralized training module can update the decision network parameters and training parameters of the agent through centralized training iteration based on the data in the experience storage units of each agent, optimize the decision strategy of the agent, and achieve full network link load balancing.

[0029] Optionally, the network topology model is an undirected graph. ;in, Represents the set of vertices, containing all nodes in the network; This represents the set of edges, containing all links between nodes; For the set of link capacities, Indicates link Maximum load capacity; For the set of link weights, Indicates link Priority or transmission cost.

[0030] Specifically, the network control device abstracts the network topology into an undirected graph by scanning all nodes in the network and the connections between nodes. , where the vertex set Each physical or logical node in the network is mapped with a unique identifier to ensure no node is missed; edge set The actual communication links between the corresponding nodes are arranged in an ordered pair of nodes. Identify the links, and due to the undirected nature of the graph, the links... and Indicates the same communication channel; set of link capacities In this framework, each element is determined by actual parameters such as link hardware specifications and protocol negotiation bandwidth, clearly defining the maximum data transmission rate that each link can stably carry; the link weight set... The elements in the system are dynamically configured according to network optimization needs. If transmission efficiency is the guiding principle, performance indicators such as link latency and packet loss rate can be quantified into weight values. If deployment cost is the guiding principle, weights can be set based on link construction and maintenance costs, providing a quantitative basis for priority evaluation of routing paths.

[0031] It should be noted that by abstracting the network topology into a standardized undirected graph structure, a structured representation of core resources such as network nodes, links, capacity, and priorities is achieved, making complex network environments quantifiable and computable. The explicit division of vertex and edge sets ensures a complete mapping between nodes and links, the link capacity set provides precise hardware parameters for constraint verification, and the link weight set supports flexible adjustment of optimization directions according to business needs. This provides a unified mathematical model for subsequent routing path planning and constraint verification, enhances the system's adaptability to different network architectures, and makes routing decisions more logical and interpretable.

[0032] Optionally, the network control device is further configured to establish a traffic matrix, which records the traffic demand between the source node and the destination node, wherein the source node and the destination node are nodes in the vertex set.

[0033] Specifically, the network control device can construct a two-dimensional traffic matrix based on historical traffic statistics, real-time service request information, and preset traffic demand configurations. The row index of the matrix corresponds to the source node number, the column index corresponds to the destination node number, and the matrix elements... Precise characterization from source node Send to the destination node The total traffic size, and in the matrix A value of 0 indicates that there is no need for a node to send traffic to itself.

[0034] It should be noted that by constructing a two-dimensional traffic matrix, the distribution of traffic demand between source and destination nodes across the entire network is presented comprehensively and intuitively, providing the agent with accurate traffic decision-making references. This matrix-based traffic representation facilitates network control devices in quickly identifying traffic hotspots and transmission bottlenecks, enabling the agent to proactively avoid high-traffic-load links during path planning and achieve predictive traffic distribution. Simultaneously, the data support from the traffic matrix shifts routing decisions from "passive response" to "proactive optimization," effectively improving the rationality of network-wide traffic allocation and reducing localized congestion caused by uneven traffic distribution.

[0035] Optionally, the step of planning segmented routing paths based on the local observation data and the decision network includes: inputting the local observation data into the decision network, outputting the decision priority of each candidate segmented routing path through the decision network; filtering candidate paths by combining traffic conservation constraints and link capacity constraints, and combining path segments with link utilization rates lower than a preset threshold and meeting the constraints to obtain the final segmented routing path.

[0036] Specifically, the agent first performs standardized preprocessing on the local observation data, converting indicators of different dimensions such as link utilization, latency, and weight values ​​into a unified numerical range, and then inputs them into the decision network. The policy network in the decision network extracts features and performs nonlinear mapping on the input data through a multilayer perceptron structure, outputting the probability distribution of each candidate segment routing path. The higher the probability value, the higher the decision priority of the path. The value network simultaneously calculates the expected revenue value corresponding to each path and performs secondary verification of the priority. In the candidate path selection stage, the agent first traverses all the path segment combinations with the highest priority ranking, verifying one by one whether they meet the traffic conservation constraint and link capacity constraint, and at the same time, determines whether the current utilization rate of each link in the combination is lower than a preset threshold. Finally, it selects the path segment combination that meets all constraints and makes the link load distribution more balanced, forming a complete segment routing path. The combination method of the path segments adopts a stack structure for storage, and each path segment is represented by a unique identifier to ensure that the nodes can be parsed in order during the forwarding process.

[0037] It should be noted that the collaborative work of the policy network and the value network in the decision network ensures the accuracy of candidate path priority evaluation and avoids local optimum decisions through expected benefit verification; standardized preprocessing improves the availability of observation data and ensures the stability of the decision network output. The introduction of traffic conservation constraints and link capacity constraints fundamentally guarantees the feasibility of routing paths and avoids problems such as traffic accumulation and link overload; while the screening condition of preset threshold for link utilization further strengthens the load balancing orientation, so that the final selected combination of path segments can not only meet the transmission requirements, but also achieve balanced use of link resources.

[0038] Optionally, the flow conservation constraint is for any flow and any node ,satisfy: ; in, Represents a stream Does it flow through the link? ; Represents a stream Does it flow through the link? ; Represents a stream At the node The flow conservation indicator at the location, For flow When the source node ,exist For flow When the destination node is ,exist For flow The intermediate node is ; Represents a stream It is the set of all streams Any one of the streams; Represents a node It is a set of vertices Any node in it.

[0039] Specifically, the core of the flow conservation constraint is to ensure the "balance of inflow and outflow" of flow at network nodes. For any flow k, when node u is its source node, flow k needs to initiate transmission from that node. At this time, the sum of the transmission ratios of the outflow links minus the sum of the transmission ratios of the inflow links equals 1, meaning that all flow flows out from the source node. When node u is the destination node of flow k, flow k needs to terminate transmission at that node. At this time, the sum of the transmission ratios of the inflow links is greater than the sum of the transmission ratios of the outflow links, and the difference between the two is -1, indicating that all flow is received at the destination node. When node u is an intermediate forwarding node, flow k flows in from one link and out from another link through that node. The sum of the transmission ratios of the inflow and outflow are equal, and the difference is 0, ensuring that flow will not accumulate or be lost at intermediate nodes. This constraint runs through the entire transmission process of all flows and covers every node in the network.

[0040] It's important to note that by strictly defining the "inflow and outflow rules" for traffic at source, destination, and intermediate nodes, it ensures that traffic doesn't arise out of thin air or disappear without cause during network transmission, guaranteeing the integrity and continuity of data transmission. The "inflow and outflow balance" constraint at intermediate nodes prevents traffic accumulation or loss at nodes, improving network transmission stability. Meanwhile, the directional transmission constraints between source and destination nodes clarify the start and end logic of traffic, providing a basis for closed-loop verification of routing paths. This constraint, as the core criterion for routing decisions, ensures that the transmission process of all flows conforms to the basic principles of network communication, laying the foundation for overall network load balancing.

[0041] Optionally, the link capacity constraint is for any link ,satisfy: ; in, Indicates on the link Total bandwidth consumed by all streams Represents a stream In the link The transmission ratio on Represents a stream Its own traffic volume; It is a link Maximum load capacity; Indicates link For the set of edges Any link in it.

[0042] Specifically, the link capacity constraint applies to all links in the entire network, and for any link... The agent needs to calculate the actual bandwidth consumption of all flows passing through the link, i.e., the bandwidth consumption of each flow. Transmission percentage of this link With flow size The product of the products of all flows is then summed over all flows to obtain the link. The total bandwidth usage; this total bandwidth usage must be less than or equal to the link's maximum capacity. If a certain combination of path segments causes the total bandwidth usage of a link to exceed the limit... If the combination is not specified, it will be directly excluded. This constraint can fundamentally avoid problems such as congestion and packet loss caused by link overload, thus ensuring the stability of network transmission.

[0043] It's important to note that by limiting the total bandwidth usage of each link to no more than its maximum capacity, link overload issues are avoided at the hardware resource limitation level, ensuring the stability and reliability of link transmission. This constraint directly targets the core causes of network congestion. By quantitatively calculating the actual bandwidth consumption of all flows on the link, it ensures that link resources are not over-utilized, effectively reducing the probability of transmission failures such as packet loss and latency jitter. Simultaneously, the rigidity of the constraint prompts the agent to proactively distribute traffic during path planning, preventing multiple high-traffic flows from concentrating on the same link, indirectly promoting the realization of network-wide load balancing.

[0044] Optionally, the agent is further configured to use auxiliary variables Indicates link Is it occupied? when When greater than 0, auxiliary variable =1; when When not greater than 0, auxiliary variable =0; Among them, auxiliary variables The following constraints must be met: .

[0045] Specifically, auxiliary variables It is a binary variable that describes the link occupancy state, and its value is determined by the link. The total number of flows is determined by analyzing all flows. Transmission percentage of this link The summation is obtained; when the summation result is greater than 0, it indicates that at least one flow is using the link. A value of 1 indicates that the link is in an "occupied" state; when the summation result equals 0, it means that no flow passes through the link. Setting it to 0 indicates that the link is in an "idle" state; to ensure that this logic is strictly true, constraints are set. The left boundary constraint guarantees that when the total flow is greater than 0, The value cannot be 0; the right boundary constraint guarantees that when the total flow is 0... It cannot be 1, thus enabling accurate identification of the link occupancy status.

[0046] It should be noted that the use of a binary auxiliary variable enables precise identification of link occupancy status, allowing the agent to quickly determine link availability and significantly improving path planning efficiency. The constraints ensure a strict correspondence between variable values ​​and actual link occupancy status, avoiding decision-making biases caused by misjudgments. Furthermore, this auxiliary variable provides the agent with a concise and intuitive reference to link status, enabling it to quickly eliminate saturated or high-load links when selecting candidate paths, prioritizing idle links, further optimizing traffic allocation strategies and improving the overall network link resource utilization efficiency.

[0047] Optionally, the routing decision problem of the agent is formalized as a partially observable Markov decision process. ;in, For the set of network environment states, For the agent's local observation set, For the segmented routing decision action set of the intelligent agent, For the reward function, This is the state transition function. The discount factor; the agent is based on the local observations. Execution of decision-making actions Through the state transition function The network state is updated by the reward function. Feedback on decision-making effectiveness drives iterative optimization of decision-making strategies.

[0048] Specifically, in the partially observable Markov decision-making process, the set of network environment states It includes all state information related to routing decisions, such as the operating status of all nodes in the entire network, link utilization, capacity, and weight; and the agent's local observation set. yes A subset thereof, consisting of the states of surrounding links and the state of its own nodes collected by the agent through network telemetry mechanisms, is limited by the observation range. Incomplete coverage Therefore, it exhibits partially observable characteristics; the agent's segmented routing decision action set Includes all possible path segment combinations and forwarding strategy selection; state transition function. The probability distribution of the network environment transitioning from the current state to the next state after performing a specific decision action in a given state is defined. This distribution is determined by factors such as changes in link traffic and fluctuations in node states; reward function. Used to quantify the merits of decision-making actions; discount factor The value range is (0,1), used to balance the weights of current rewards and long-term future rewards; during operation, the agent uses local observations... From action set Select the optimal action to execute through the state transition function. Triggering network state updates, then the reward function Based on the updated status feedback reward value, the agent adjusts its decision-making strategy to achieve iterative optimization.

[0049] It should be noted that standardized Markov decision process modeling provides a rigorous mathematical framework for agent decision-making and learning, making the policy optimization process quantifiable and traceable. The consideration of some observable characteristics aligns with the observational limitations of agents in real-world network environments, making the model more realistically adaptable. The clear delineation of core elements such as state transition functions and reward functions provides a clear logical guide for the agent's policy iteration. This modeling approach enables the agent to dynamically adjust its decisions based on historical experience and real-time observations, improving the policy's adaptability and robustness, allowing it to cope with complex scenarios such as network topology changes and fluctuating traffic demands.

[0050] Optionally, the feedback value of the reward function is negatively correlated with the link load balancing degree. When the segmented routing path planned by the agent makes the maximum link utilization of the corresponding link decrease, the reward function increases; when the maximum link utilization increases, the reward function decreases.

[0051] Specifically, the reward function is constructed using a weighted summation method, with core weight coefficients allocated to link load balancing related indicators. When the segmented routing path planned by the agent reduces the maximum link utilization of the corresponding link, the reward function increases; when the maximum link utilization increases, the reward function decreases. In addition, the reward function can also introduce a link weight adaptation term. If the path selection is consistent with the link weight guidance (such as prioritizing low-weight links), an additional positive reward can be added. Through this reward and punishment mechanism, clear guidance is provided for the agent's decision-making, accelerating the convergence of the strategy towards the optimal goal.

[0052] It's important to note that the reward and penalty mechanism provides clear optimization guidance for the agent's decision-making, enabling it to quickly perceive the impact of its actions on load balancing. Positive rewards guide the agent to proactively choose paths that bring link utilization closer to the network average, promoting overall network load balancing; negative penalties effectively curb decision-making behaviors that could lead to link overload, preventing localized congestion. This reward mechanism accelerates the convergence of the decision-making strategy towards the goal of minimizing maximum link utilization, while also improving the stability of the strategy, preventing the agent from getting trapped in local optima, and ensuring a long-term load-balanced state for the network.

[0053] Optionally, the network control device optimizes the agent's decision-making strategy with the goal of minimizing the maximum link utilization of the entire network. : ; ; in, Indicates a single link utilization rate Indicates all links that pass through The total flow Indicates a single link Maximum load capacity.

[0054] Specifically, the maximum link utilization rate of the entire network The calculation process is as follows: First, traverse all links in the network. For each link, calculate its total traffic. With link capacity The ratio of the utilization rates of all links is the utilization rate of a single link, reflecting the resource consumption of the link. The maximum utilization rate from all links is then selected as the [value]. ; Among them, total link traffic It is the sum of the actual traffic of all flows passing through this link, by analyzing each flow. Size With this flow in the link Transmission ratio The product is summed to obtain the result; the network control device minimizes... This core objective can drive intelligent agents to optimize routing strategies, achieving balanced distribution of network link load without increasing link hardware resources, thereby improving the overall network carrying capacity.

[0055] It's important to note that the core optimization objective is to minimize the maximum link utilization across the entire network. This directly targets key pain points in network operation, maximizing the network's traffic carrying capacity without increasing link hardware capacity. By quantitatively calculating the utilization of each link and selecting the maximum value for optimization, a balanced allocation of network link resources is ensured, preventing the scenario where "an overloaded link drags down the entire network." This goal-oriented approach ensures that the agent's routing decisions always revolve around efficient resource utilization, effectively improving the overall utilization of network links, reducing congestion risk, ensuring stable network operation under high traffic loads, and significantly improving network service quality and return on resource investment.

[0056] In this embodiment, on the one hand, due to the adoption of a distributed deployment architecture, each network node deploys at least one intelligent agent, and the intelligent agent independently performs segmented routing path planning and traffic forwarding operations without relying on a central controller for unified network-wide path calculation. This effectively distributes the computational pressure of routing decisions, avoiding the exponential increase in computational complexity and communication overhead caused by network expansion under a centralized architecture. It also reduces decision latency and control plane congestion risks, and improves system scalability and resistance to single points of failure. On the other hand, because the intelligent agent collects real-time link status information through a network telemetry mechanism, rather than the traditional periodic sampling... By employing either polling or sampling methods, the system can accurately capture fine-grained changes in link status, overcoming the shortcomings of traditional link sensing, such as coarse granularity and poor timeliness. This ensures that path decisions match the actual network status in real time, reducing link overload and resource waste. Furthermore, since the network control device optimizes decision-making strategies by iteratively updating the decision network parameters and training parameters of each agent through centralized training based on the experience storage unit data of each agent, and the agents plan segmented routing paths based on the decision network, the network link load can be balanced, significantly reducing the maximum link utilization and improving the overall operating efficiency and service quality of large-scale, highly dynamic networks.

[0057] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent segmented routing system based on online telemetry, characterized in that, include: Network control devices and multiple intelligent agents; The network control device is used to construct a network topology model and deploy multiple intelligent agents at different network nodes, with one network node corresponding to at least one intelligent agent. The intelligent agent is used to initialize its own decision network, experience storage unit, and training parameters; to obtain local observation data by collecting real-time link status information through the network telemetry mechanism; to plan segmented routing paths based on the local observation data and the decision network; to perform traffic forwarding operations through segmented routing, and to automatically record the actions, rewards, and status information during the interaction process to the experience storage unit. The network control device is used to optimize the decision-making strategy of the agents by updating the decision network parameters and training parameters of the agents through centralized training iteration based on the data in the experience storage unit of each agent, so as to achieve load balancing of the entire network link.

2. The multi-agent segmented routing system based on online telemetry according to claim 1, characterized in that, The network topology model is an undirected graph. ; in, Represents the set of vertices, containing all nodes in the network; This represents the set of edges, containing all links between nodes; For the set of link capacities, Indicates link Maximum load capacity; For the set of link weights, Indicates link Priority or transmission cost.

3. The multi-agent segmented routing system based on online telemetry according to claim 2, characterized in that, The network control device is also used to establish a traffic matrix, which is used to record the traffic demand between the source node and the destination node, wherein the source node and the destination node are nodes in the vertex set.

4. The multi-agent segmented routing system based on online telemetry according to claim 1, characterized in that, The segmented routing path planning based on the local observation data and the decision network includes: The local observation data is input into the decision network, and the decision network outputs the decision priority of each candidate segment routing path; By combining traffic conservation constraints and link capacity constraints to screen candidate paths, the path segments with link utilization below a preset threshold and meeting the constraints are combined to obtain the final segmented routing path.

5. The multi-agent segmented routing system based on online telemetry according to claim 4, characterized in that, The flow conservation constraint is for any flow and any node ,satisfy: ； in, Represents a stream Does it flow through the link? ,flow The sum of the transmission proportions on each link; Represents a stream Does it flow through the link? ; Represents a stream At the node The flow conservation indicator at the location, For flow When the source node ,exist For flow When the destination node is ,exist For flow The intermediate node is ; Represents a stream It is the set of all streams Any one of the streams; Represents a node It is a set of vertices Any node in it.

6. The multi-agent segmented routing system based on online telemetry according to claim 4, characterized in that, The link capacity constraint is for any link ,satisfy: ； in, Indicates on the link Total bandwidth consumed by all streams Represents a stream In the link The transmission ratio on Represents a stream Its own traffic volume; It is a link Maximum load capacity; Indicates link For the set of edges Any link in it.

7. The multi-agent segmented routing system based on online telemetry according to claim 6, characterized in that, The intelligent agent is also used to use auxiliary variables Indicates link Is it occupied? when When greater than 0, auxiliary variable =1; when When not greater than 0, auxiliary variable =0; Among them, auxiliary variables The following constraints must be met: .

8. The multi-agent segmented routing system based on online telemetry according to claim 1, characterized in that, The routing decision problem of the agent is formalized as a partially observable Markov decision process. ;in, For the set of network environment states, For the agent's local observation set, For the segmented routing decision action set of the intelligent agent, For the reward function, This is the state transition function. The discount factor; the agent is based on the local observations. Execution of decision-making actions Through the state transition function The network state is updated by the reward function. Feedback on decision-making effectiveness drives iterative optimization of decision-making strategies.

9. The multi-agent segmented routing system based on online telemetry according to claim 8, characterized in that, The feedback value of the reward function is negatively correlated with the link load balancing degree. When the segmented routing path planned by the agent makes the maximum link utilization of the corresponding link smaller, the reward function increases; when the maximum link utilization increases, the reward function decreases.

10. The multi-agent segmented routing system based on online telemetry according to any one of claims 1-9, characterized in that, The goal of the network control device in optimizing the agent's decision-making strategy is to minimize the maximum link utilization of the entire network. : ； ； in, Indicates a single link utilization rate Indicates all links that pass through The total flow Indicates a single link Maximum load capacity.

Images