SDN (Software Defined Network) inter-domain traffic engineering method based on reinforcement learning

Abstract

The invention provides an SDN (Software Defined Network) inter-domain traffic engineering method based on reinforcement learning, which comprises the following steps of: deploying a data traffic demand monitoring platform and a control system, and constructing a global network topological graph; calculating a short link identifier for the link in the network and distributing the short link identifier to each network device; flow judgment is carried out, upward notification is carried out according to requirements, and pre-operation of intelligent routing is cooperatively completed; deploying a reinforcement learning model in the total intelligent body, outputting an optimal cross-domain path strategy to the cooperative controller, disassembling the optimal cross-domain path strategy into flow table rules which can be executed by each domain, and issuing the flow table rules to local controllers of related domains; and each local controller pushes the flow table configuration to the domain switching equipment to complete the forwarding decision of the flow. According to the method, a complete closed-loop process of flow measurement, intelligent decision making, cross-domain control and path issuing is realized, feasible reference is provided for actual deployment of an intelligent network, and the method has good engineering popularization value and is suitable for intelligent scheduling scenes such as an operator backbone network, an industrial internet and metro edge cloud.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer networks, in particular to an SDN inter-domain traffic engineering method based on reinforcement learning. BACKGROUND

[0002] With the emergence of big data, cloud computing, Internet of Things and other technologies, the rapid development of various multimedia services and network devices, traditional IP networks have problems such as difficult to control, bloated network architecture, and difficult to guarantee high-quality services. Therefore, the industry introduces SDN controllers. In the implementation scheme of the SDN controller, two common methods, centralized and distributed, are usually used to improve scalability. The centralized model is the most basic architecture, but this approach introduces a single point of failure for the network. The distributed controller model can eliminate single point of failure and improve network scalability. In order to realize the interconnection of global SDN domains and promote inter-domain interoperability, a mechanism is needed to coordinate the cooperative work of different domain controllers to provide a more optimal configuration scheme for cross-domain services. The traditional cross-domain mechanism is completed by the Border Gateway Protocol (BGP) through the border router to realize the message transfer between adjacent autonomous systems (AS), thereby realizing inter-domain message synchronization. However, due to the cumbersome configuration of the BGP method in the SDN environment and the problem of route oscillation, a cross-domain coordination layer is introduced to solve this problem.

[0003] A basic problem that traditional traffic engineering mainly focuses on is intra-domain TE. The classic optimization goal is to minimize the maximum link load within an autonomous network domain (for example, an operator-level network). It mainly relies on static algorithms or heuristic rules, and is artificially offline planned, or relies on IGP parameter adjustment and static MPLS tunnels, with a long cycle and only capable of balancing bandwidth within a single domain. Once the service traffic needs to cross multiple ASes, cloud resource pools or edge nodes, end-to-end path optimization and unified scheduling cannot be achieved. This approach cannot perceive real-time congestion, application SLA and cross-domain topology, and is also difficult to cope with micro-bursts and east-west traffic of cloud services. Therefore, how to implement cross-domain traffic engineering has become a problem to be solved. SUMMARY

[0004] In order to meet the demand of cross-domain network interconnection, and to get rid of the inflexible data forwarding of traditional inter-domain routing algorithms and the lack of cross-domain traffic engineering scheme, the present application provides an SDN inter-domain traffic engineering method based on reinforcement learning, which realizes an SDN cross-domain intelligent path scheme based on distributed agent deep reinforcement learning to solve the problems proposed in the background technology. The technical scheme provided by the present application is as follows:

[0005] A method for inter-domain traffic engineering of an SDN network based on reinforcement learning, the network being composed of a data plane, a collaborative management plane, a control plane and a knowledge plane, comprising the following steps:

[0006] Step 1, deploying a data traffic demand monitoring platform, deploying a local controller and an agent in each network domain, deploying a collaborative controller in the control plane of the entire network, and deploying a total agent in the knowledge plane, constructing a global network topology graph G through a topology fusion algorithm global ;

[0007] Step 2, based on the global network topology graph, calculating a short link identifier Short_LID for the in-network link and distributing it to each network device, and synchronously updating the short link identifier to the forwarding table generation logic of the local controller and the collaborative controller;

[0008] Step 3, judging the traffic and performing upward notification according to the demand, and collaboratively completing the pre-operation of intelligent routing;

[0009] Step 4, deploying a reinforcement learning model in the total agent, the reinforcement learning model being an Actor-Critic network based on a PPO algorithm, and dynamically generating an optimal cross-domain path strategy meeting a minimum maximum link utilization rate target based on the current network state and historical experience;

[0010] Step 5, the optimal cross-domain path strategy being handed over to the collaborative controller by the total agent, the generated flow table rule being disassembled into flow table rules executable by each domain according to path segmentation mapping, and being respectively distributed to the local controllers of the related domains;

[0011] Step 6, each local controller pushing the received flow table configuration to the exchange equipment in the domain, installing the flow table by the exchange equipment, completing the forwarding decision of the traffic, and realizing the end-to-end cross-domain data forwarding link.

[0012] Preferably, the global network topology graph ; wherein is a network in each domain, is a set of inter-domain connection links; is a set of nodes; is a set of edges, and the algorithm process for obtaining the global network topology graph is as follows:

[0013] Initializing the global network topology graph G global to be empty; for each region i=(1,n), adding all nodes and edges in G i to G global ; for each pair of adjacent regions (i,i+1), finding a boundary node set B i in G i and a boundary node set B i+1 in G i+1For each pair of boundary nodes To G global Add an edge Set the capacity and delay of the edge; return the merged global network topology graph G. global .

[0014] Preferably, the Short Link Identifier (Short_LID) is generated based on a feature hashing algorithm, which integrates feature values. After concatenating with the current network information S, a hash value is calculated, and the first k bits are truncated as Short_LID. The feature values ​​are then integrated. The formula is:

[0015]

[0016] in, Use the Sigmoid activation function; The feature weight function; This is the feature normalization function, which maps the original features to the interval [0,1].

[0017] Preferably, the specific process of step 3 is as follows:

[0018] Step 31: Detect the generated traffic demand. The source terminal initiates a communication request to trigger a new traffic demand. The traffic measurement platform captures the traffic demand in real time through the traffic acquisition module, converts it into a standardized traffic matrix TM, and pushes it to the local controller.

[0019] Step 32: After receiving the traffic request, the local controller evaluates whether the traffic request can be routed within the local domain. It determines whether the source and destination hosts are located in the same domain and whether the path meets the bandwidth and latency requirements by querying the local domain network topology and link status.

[0020] If it is determined that it can be processed locally, the local routing and forwarding module is called to calculate the optimal path and generate flow table rules based on Short_LID and send them to the local switch.

[0021] Otherwise, if the requirement involves cross-domain transmission, the local controller will report the traffic matrix TM to the coordinating controller;

[0022] Step 33: After receiving the corresponding cross-domain traffic request, the collaborative controller determines whether the link status and global network topology map stored in the system have expired.

[0023] If it has not expired, the source domain, destination domain and other information involved in the cross-domain flow are extracted, integrated and input into the overall intelligent agent for intelligent path decision-making;

[0024] If the network has expired, the local controller will be notified immediately to collect network status and topology information, which will then be re-integrated and handed over to the main intelligent agent.

[0025] Preferably, the optimization objective of step 4 is to minimize the maximum link utilization. , where f e Indicates the total load on link e: ; It is one of the n candidate paths for each business flow; This indicates whether the path passes through link e, therefore ; Indicates traffic demand; C represents the traffic splitting ratio corresponding to a single path. e Indicates the capacity of link e; the final optimization objective is ;

[0026] The constraints include: normalization constraints: Non-negativity constraint: Link capacity constraints: .

[0027] Preferred reinforcement learning models include the following:

[0028] State space S t At time t, the network contains complete information about the current network, including the following: global network topology G. global The link e status, represented by the short link identifier Short_LID, includes the link capacity C(e) and the current remaining bandwidth B. t (e) Point link D t (e) Packet loss rate L t (e); Flow Matrix TM (s,d) An N×N matrix represents the business flow requirements for each source-destination (s,d) pair;

[0029] Policy function: uses a neural network structure, takes input state representation and business flow representation; outputs the probability distribution of candidate paths;

[0030] Action Space A t Action A at each moment t This represents the decision result of the routing strategy. For each service flow, a path P is selected. s,d It consists of the Short Link ID of the path;

[0031] Reward function R t The reward function is used to guide the policy to optimize the global objective. It compresses the negative objective function into a positive reward through an exponential function and introduces a temperature coefficient. This is used to control the sensitivity of rewards to MLU.

[0032] Preferably, step 4 includes the following specific processes:

[0033] Step 41: Select candidate paths to construct the action space, collect traffic state information to construct the state space vector, and use a heuristic algorithm to generate the optimal path selection scheme for each business flow in the current state as a training label.

[0034] Step 42: Use a feedforward neural network for supervised pre-training to output a K-dimensional vector for each traffic flow, representing the allocation of traffic to K paths;

[0035] Step 43: Use the pre-trained model as the initial weights of the policy network. The policy network is based on the current state s. t Output the path allocation probability distribution for all service flows, apply the actions to the simulated network environment, calculate the link load distribution, and obtain the global reward r. t Then, the PPO algorithm is used for policy gradient update.

[0036] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0037] This invention proposes a collaborative SDN control architecture that combines a hierarchical control model with a local controller and a collaborative controller, breaking away from the problems of control silos and locally optimal paths in traditional SDN in multi-domain environments. The local controller is responsible for local topology acquisition and status monitoring, while the collaborative controller unifies topology integration and scheduling strategies, achieving globally optimal path calculation and management for cross-domain traffic.

[0038] This invention introduces a short link identifier generated based on multi-dimensional attribute fusion, which effectively simplifies the path representation and flow table distribution process. The short link identifier not only improves the efficiency of path lookup and mapping based on traffic requirements, but also supports rapid path reconstruction and path tracing capabilities.

[0039] This invention employs intelligent algorithms such as deep reinforcement learning to learn strategies and optimize paths for cross-domain traffic. Compared with traditional static strategies or heuristic algorithms, it has stronger adaptability, autonomy, and generalization ability when facing changes in network state or sudden traffic.

[0040] This invention realizes a complete closed-loop process of traffic measurement, intelligent decision-making, cross-domain control and path distribution, providing a feasible reference for the actual deployment of intelligent networks, with good engineering promotion value, and applicable to intelligent scheduling scenarios such as operator backbone networks, industrial internet, and metropolitan area edge cloud. Attached Figure Description

[0041] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0042] Figure 1This is an abstract framework diagram of the network model proposed in this invention;

[0043] Figure 2 This is an overall flowchart of the method proposed in this invention;

[0044] Figure 3 This is a flowchart illustrating the modeling process of reinforcement learning proposed in this invention;

[0045] Figure 4 This is the network structure diagram of the strategy proposed in this invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] To make the above-mentioned objectives, features and effects of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] Example 1: A reinforcement learning-based SDN network inter-domain traffic engineering method integrates the global network topology constructed by the cooperative controller, the traffic matrix TM, and the state space of the link state Short_LID(e) to achieve cross-domain optimal path selection under the objective of minimizing maximum link utilization (MLU). This invention utilizes the PPO algorithm and the Actor-Critic network structure, supplementing routing and forwarding with an innovative short link identifier Short_LID based on feature hashing. This ensures that the identifier is short and unique across the entire network, facilitating efficient path mapping and rapid reconstruction.

[0049] This invention constructs an SDN network architecture based on cross-domain collaboration. The overall architecture abstract model is as follows: Figure 1 As shown, the entire network model consists of a data plane, a collaborative management plane, a control plane, and a knowledge plane. These planes work closely together to achieve cross-domain traffic perception, intelligent path selection, and dynamic policy distribution.

[0050] The data plane consists of multiple network autonomous systems and various forwarding devices within them. The forwarding devices in each autonomous system support status reporting and policy enforcement functions. They are mainly responsible for periodically reporting link status to the control plane, generating short link identifiers (Short_LID), receiving flow table rules from the local controller, and executing forwarding policies.

[0051] The control plane consists of a local controller and a coordinating controller. The former handles topology discovery and intra-domain policy distribution, while the latter coordinates cross-domain traffic scheduling and unifies policy distribution. The local controller is deployed in each autonomous system (AS), uses the LLDP protocol for topology discovery, periodically monitors intra-domain link status (bandwidth, latency, packet loss rate, etc.), and performs intra-domain path calculation, forwarding traffic without cross-domain requirements directly within the AS. The local controller distributes policies to forwarding devices via a southbound interface.

[0052] As the global coordination center, the collaborative controller establishes communication with each local controller through mechanisms such as Socket, summarizes the status of the entire network, can coordinate and handle cross-domain traffic requests, schedule inter-domain paths, and synchronize inter-domain routing policies to the relevant local controllers.

[0053] The collaborative management plane is responsible for integrating network topology information from various domains, as well as detected traffic demand information. Its core components include a data processing module and a data storage module. By periodically acquiring the link e's status (Short_LID(e)) from the local controller, the management plane extracts key metrics such as latency, packet loss rate, and link bandwidth to construct a joint traffic matrix, providing training data support for the knowledge plane.

[0054] Based on intelligent algorithms such as reinforcement learning, the knowledge base trains routing strategy models within and between domains to achieve adaptive and optimized path decision-making.

[0055] The overall implementation steps of the present invention are as follows: Figure 2 As shown, it includes the following steps.

[0056] Step 1: Deploy a data traffic demand monitoring platform and deploy local controllers and agents in each network domain for local control and data acquisition. Simultaneously, deploy a collaborative controller on the control plane and a general agent on the knowledge plane for global coordination and cross-domain decision-making. Each local controller is responsible for topology discovery within its domain, including automatic identification of switches and links, and continuous monitoring of link status information, such as key performance indicators like bandwidth utilization, latency, and packet loss rate. Local controllers are also responsible for distributing flow table policies to local switching devices to control traffic within their domains. Furthermore, all local controllers must aggregate and report the real-time collected network topology and status information to the collaborative controller, which performs centralized fusion and unified management of network-wide information, providing data support for subsequent path optimization and policy distribution. Finally, a topology fusion algorithm is used to integrate the network graphs from each domain to construct a global network topology graph G. global This enables visualization and unified scheduling of multi-domain heterogeneous networks.

[0057] Global network topology diagram ;in These are networks within various domains. It is a collection of inter-domain connection links; It is a set of nodes; It is a set of edges.

[0058] Because different regions have their own topological structures (Figure G1, G2, ..., G...), n The topology graphs of these independent domains need to be merged to form a global view G. global The algorithm for obtaining the global network topology graph is as follows:

[0059] Initialize the global network topology graph G global Empty; for each region i=1 to n, G i All nodes and edges are added to G global In the middle; for each pair of adjacent regions (i, i+1), find G. i B is the set of boundary nodes in i and G i+1 B is the set of boundary nodes in i+1 For each pair of boundary nodes To G global Add an edge Set the capacity of this edge. With delay Return the merged global network topology graph G. global .

[0060] Step 2: Based on the global topology, calculate and assign Short Link Identifiers (Short_LIDs) to relevant network devices for intranet links. These Short Link Identifiers are generated using an algorithm that integrates basic link characteristics. This relies on the aggregation of multi-dimensional link features, including the link's topological location (such as its domain and upstream / downstream node information) for locating the specific upstream starting device, performance attributes (bandwidth, latency, stability) for rapid optimization based on traffic demands, path accessibility, and policy preferences, as well as information for rapid filtering, identification, and reconstruction of subsequent forwarding paths. Because performance attributes are constantly changing, information is collected periodically for a numbering update. The allocation of Short Link Identifiers is synchronously updated in the forwarding table generation logic of the local controller and the cooperating controller, so that they can be directly mapped to physical path segments when generating paths.

[0061] The Short_LID identifier is generated based on a feature hashing algorithm, which requires combining multiple characteristics of the link to generate a globally unique identifier. Considering practical deployment, the identifier is designed to be as short as possible while ensuring sufficient distinguishability between different links.

[0062] The core formula for calculating Short_LID is as follows, which represents the fusion of feature values. After concatenating with S, calculate the hash value and extract the first k bits as Short_LID:

[0063]

[0064] in, Use the Sigmoid activation function; The feature weight function; This is the feature normalization function, which maps the original features to the interval [0,1].

[0065] Step 3: After the business flow is generated, the controller in the system will determine the traffic and execute upward notification as needed to complete the pre-operation of intelligent routing.

[0066] Step 31: Detect the generated traffic demand. The source terminal initiates a communication request to trigger a new traffic demand. The traffic measurement platform captures the traffic demand in real time through the traffic acquisition module, converts it into a standardized traffic matrix TM, and pushes it to the local controller.

[0067] Step 32: After receiving the traffic request, the local controller first evaluates whether the traffic request can be routed within the local domain. It determines whether the source and destination hosts are located in the same domain and whether the path meets the bandwidth and latency requirements by querying the local network topology and link status.

[0068] If it is determined that it can be processed locally, the local routing and forwarding module is immediately invoked to calculate the optimal path (e.g., using heuristic algorithms or local reinforcement learning strategies) and generate flow table rules based on Short_LID to be sent to the local switch.

[0069] Otherwise, if the requirement involves cross-domain transmission, the local controller will report the traffic matrix TM and other data to the coordinating controller.

[0070] Step 33: After receiving the corresponding cross-domain traffic request, the collaborative controller determines the Short_LID(e) link state information stored in the system and constructs a global network view G through the topology fusion algorithm. global Is the topology data outdated?

[0071] If usable, extract the source domain, destination domain, and other information involved in the cross-domain flow, integrate them, and input them into the overall intelligent agent for intelligent path decision-making;

[0072] If it is unusable, immediately notify the local controller to collect network status and topology information, and then reassemble it before handing it over to the main intelligent agent.

[0073] Step 4: Deploy reinforcement learning models or other training and optimization modules within the overall agent. Based on the PPO algorithm, the Actor-Critic network utilizes a pre-trained path planning strategy. The system can dynamically generate optimal cross-domain path strategies that satisfy objectives such as minimizing maximum link utilization, based on the current network state and historical experience, and deliver these strategies to the collaborative controller. Path information is encoded using the Short_LID short link identifier.

[0074] This invention is based on the global network topology graph G global In addition to considering business flow requirements and Short_LID(e), the goal is to select one or more optimal paths to ensure the minimization of maximum link utilization. This type of problem can be described as a multi-product network flow (MCNF) problem, and the problem modeling is as follows:

[0075] Optimization objective: such as minimizing maximum link utilization. ;

[0076] Where f e Indicates the total load on link e: ; It is one of the n candidate paths for each business flow, and the business flow will select one or more paths; This indicates whether the path passes through link e, therefore ; Indicates traffic demand; C represents the traffic splitting ratio corresponding to a single path. e This represents the capacity of link e. The final optimization objective is... .

[0077] The constraints include: normalization constraints: Non-negativity constraint: Link capacity constraints: .

[0078] The reinforcement learning process provided by this invention is as follows: Figure 3 As shown, the core design is as follows:

[0079] State space S t At time t, the state vector needs to contain complete information about the current network, mainly including the following: global network topology G global The link e state, represented by the short link identifier Short_LID(e), incorporates the following specific characteristics, including a link capacity of C(e) and a current remaining bandwidth of B. t (e) Point link D t (e) Packet loss rate L t (e); Flow Matrix TM (s,d) An N×N matrix represents the business flow requirements for each source-destination (s,d) pair.

[0080] Policy function: using a neural network structure, such as Figure 4 As shown, the input states are represented (fixed-length vectors) and the business flow (s,d); the output is the probability distribution of candidate paths.

[0081] Action Space A t Action A at each moment t This indicates the decision result of the routing strategy. For each service flow, a path P will be selected. s,d It consists of the short link identifier (Short_LID) of the path.

[0082] Reward function R t The reward function is used to guide the policy to optimize the global objective. It compresses the negative objective function into a positive reward through an exponential function and introduces a temperature coefficient. This is used to control the sensitivity of rewards to MLU.

[0083] Based on the above design, the specific process of step 4 is as follows:

[0084] Step 41: Select candidate paths to construct the action space, collect traffic state information to construct the state space vector, and use heuristic algorithms (such as shortest path + minimum load) to generate the optimal path selection scheme for each business flow in the current state, and use it as training labels.

[0085] Step 42 involves supervised pre-training using a feedforward neural network, outputting a K-dimensional vector for each traffic flow, representing the allocation of traffic to K paths. After training on a large amount of historical data, this network can quickly generate high-quality path allocation schemes based on the current state, which are used for initializing the reinforcement learning policy network, improving sample efficiency and initial policy quality.

[0086] Step 43: Use the above model as the initial weights for the policy network. The policy network is then adjusted according to the current state s. t Output the path allocation probability distribution for all service flows, apply the actions to the simulated network environment, calculate the link load distribution, and obtain the global reward r. t Then, the PPO algorithm is used for policy gradient updates. This enables the model to maintain heuristic decision-making capabilities while possessing adaptive and generalization abilities, thereby achieving efficient, online optimization of routing policies in complex network environments.

[0087] The PPO algorithm uses an Actor-Critic network structure for policy updates. The core process is as follows: In the sampling phase, the old policy is sampled in a simulated environment; in the evaluation phase, the Critic network is used to evaluate value and advantage; and finally, in the update phase, the policy network is used to update the policy. CLIPMaximize and update the Critic network to fit the value evaluation function. PPO uses the following loss function to prune the objective function:

[0088]

[0089] in, The ratio of the old to the new strategy; value assessment function. Used to calculate the advantage function ; Used to control the range of policy changes; the clip operation limits the policy update magnitude to prevent training instability.

[0090] Step 5: The path policy output by PPO is handed over to the cooperative controller by the overall agent. Based on the path segmentation mapping, the generated flow table rules are decomposed into flow table rules that can be executed in each domain and then sent to the local controllers of the relevant domains.

[0091] Step 6: Each local controller will receive the flow table configuration pushed to the local domain switching device, which will then install the flow table, complete the traffic forwarding decision, and realize the end-to-end cross-domain data forwarding link.

[0092] Example 2: The computer-readable storage medium of this example stores a computer program that, when executed by a processor, implements the steps of the reinforcement learning-based SDN network inter-domain traffic engineering method in Example 1.

[0093] The computer-readable storage medium in this embodiment can be an internal storage unit of the terminal, such as the terminal's hard disk or memory; the computer-readable storage medium in this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. equipped on the terminal; furthermore, the computer-readable storage medium can include both the terminal's internal storage unit and external storage devices.

[0094] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0095] Example 3: The computer device of this example includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the reinforcement learning-based SDN network inter-domain traffic engineering method of Example 1.

[0096] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The memory can include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0097] Those skilled in the art will clearly understand that each implementation can be achieved using software plus the necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0098] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A reinforcement learning-based inter-domain traffic engineering method for SDN networks, characterized in that, The network consists of a data plane, a collaborative management plane, a control plane, and a knowledge plane, and includes the following steps: Step 1: Deploy a data traffic demand monitoring platform. Deploy local controllers and agents in each network domain, a collaborative controller on the control plane of the entire network, and a general agent on the knowledge plane. Construct a global network topology graph G using a topology fusion algorithm. global ; Step 2: Based on the global network topology map, calculate the Short Link Identifier (Short_LID) for intra-network links and assign it to each network device. The assignment of the Short Link Identifier is synchronously updated in the forwarding table generation logic of the local controller and the cooperating controller. Step 3: Traffic is assessed and upward notifications are executed as needed to collaboratively complete the pre-operation of intelligent routing; Step 4: Deploy a reinforcement learning model inside the overall agent. The reinforcement learning model is an Actor-Critic network based on the PPO algorithm. Based on the current network state and historical experience, it dynamically generates the optimal cross-domain path strategy that satisfies the objective of minimizing the maximum link utilization. Step 5: The optimal cross-domain path strategy is handed over by the overall agent to the cooperative controller. Based on the path segmentation mapping, the generated flow table rules are decomposed into flow table rules that can be executed in each domain and then distributed to the local controllers of the relevant domains. Step 6: Each local controller will receive the flow table configuration pushed to the local domain switching device, which will then install the flow table, complete the traffic forwarding decision, and realize the end-to-end cross-domain data forwarding link.

2. The reinforcement learning-based inter-domain traffic engineering method for SDN networks according to claim 1, characterized in that, The global network topology diagram ;in These are networks within various domains. It is a collection of inter-domain connection links; It is a set of nodes; Given a set of edges, the algorithm for obtaining the global network topology graph is as follows: Initialize the global network topology graph G global Empty; for each region i=(1,n), G i Add all nodes and edges to G global In the middle; for each pair of adjacent regions (i, i+1), find G. i B is the set of boundary nodes in i and G i+1 B is the set of boundary nodes in i+1 For each pair of boundary nodes To G global Add an edge Set the capacity and delay of this edge; Return to the merged global network topology graph G global .

3. The reinforcement learning-based inter-domain traffic engineering method for SDN networks according to claim 2, characterized in that, The Short Link Identifier (Short_LID) is generated based on a feature hashing algorithm, which integrates feature values. After concatenating with the current network information S, a hash value is calculated, and the first k bits are truncated as Short_LID. The feature values ​​are then integrated. The formula is: ； in, Use the Sigmoid activation function; The feature weight function; This is the feature normalization function, which maps the original features to the interval [0,1].

4. The reinforcement learning-based inter-domain traffic engineering method for SDN networks according to claim 1, characterized in that, The specific process of step 3 is as follows: Step 31: Detect the generated traffic demand. The source terminal initiates a communication request to trigger a new traffic demand. The traffic measurement platform captures the traffic demand in real time through the traffic acquisition module, converts it into a standardized traffic matrix TM, and pushes it to the local controller. Step 32: After receiving the traffic request, the local controller evaluates whether the traffic request can be routed within the local domain. It determines whether the source and destination hosts are located in the same domain and whether the path meets the bandwidth and latency requirements by querying the local domain network topology and link status. If it is determined that it can be processed locally, the local routing and forwarding module is called to calculate the optimal path and generate flow table rules based on Short_LID and send them to the local switch. Otherwise, if the requirement involves cross-domain transmission, the local controller will report the traffic matrix TM to the coordinating controller; Step 33: After receiving the corresponding cross-domain traffic request, the collaborative controller determines whether the link status and global network topology map stored in the system have expired. If it has not expired, the source domain, destination domain and other information involved in the cross-domain flow are extracted, integrated and input into the overall intelligent agent for intelligent path decision-making; If the network has expired, the local controller will be notified immediately to collect network status and topology information, which will then be re-integrated and handed over to the main intelligent agent.

5. The reinforcement learning-based inter-domain traffic engineering method for SDN networks according to claim 1, characterized in that, The optimization objective of step 4 is to minimize the maximum link utilization. , where f e Indicates the total load on link e: ; It is one of the n candidate paths for each business flow; This indicates whether the path passes through link e, therefore ; Indicates traffic demand; C represents the traffic splitting ratio corresponding to a single path. e Indicates the capacity of link e; the final optimization objective is ; The constraints include: normalization constraints: Non-negativity constraint: Link capacity constraints: .

6. The SDN network inter-domain traffic engineering method based on reinforcement learning according to claim 5, characterized in that, Reinforcement learning models include the following: State space S t The information at time t contains complete information about the current network, including the following: Global network topology G global ; The link e state, represented by the short link identifier Short_LID, includes the link capacity C(e) and the current remaining bandwidth B. t (e) Point link D t (e) Packet loss rate L t (e); Flow Matrix TM (s,d) An N×N matrix represents the business flow requirements for each source-destination (s,d) pair; Policy function: uses a neural network structure, takes input state representation and business flow representation; outputs the probability distribution of candidate paths; Action Space A t Action A at each moment t This represents the decision result of the routing strategy. For each service flow, a path P is selected. s,d It consists of the Short Link ID of the path; Reward function R t The reward function is used to guide the policy to optimize the global objective. It compresses the negative objective function into a positive reward through an exponential function and introduces a temperature coefficient. This is used to control the sensitivity of rewards to MLU.

7. The SDN network inter-domain traffic engineering method based on reinforcement learning according to claim 6, characterized in that, Step 4 includes the following specific steps: Step 41: Select candidate paths to construct the action space, collect traffic state information to construct the state space vector, and use a heuristic algorithm to generate the optimal path selection scheme for each business flow in the current state as a training label. Step 42: Use a feedforward neural network for supervised pre-training to output a K-dimensional vector for each traffic flow, representing the allocation of traffic to K paths; Step 43: Use the pre-trained model as the initial weights of the policy network. The policy network is based on the current state s. t Output the path allocation probability distribution for all service flows, apply the actions to the simulated network environment, calculate the link load distribution, and obtain the global reward r. t Then, the PPO algorithm is used for policy gradient update.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the reinforcement learning-based SDN network inter-domain traffic engineering method as described in any one of claims 1-7.

9. A computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the reinforcement learning-based SDN network inter-domain traffic engineering method as described in any one of claims 1-7.

Images