SD-WAN and differentiated service-oriented strict priority queue scheduling method and system

Abstract

The invention provides a strict priority queue scheduling method oriented to SD-WAN and differentiated services, and belongs to the technical field of differentiated services of software defined wide area networks, and the method comprises the steps: obtaining the time delay, throughput and related queue information of each service flow in a network based on a flow INT technology; based on a graph convolution deep reinforcement learning algorithm, reasonable priorities are automatically allocated to different services according to maximum time delay and minimum throughput requirements of various services so as to guide a scheduling process of a strict priority queue; a dynamic queue mapping adaptive algorithm is adopted, more priorities are flexibly mapped to less strict priority queues, and the priority space is expanded. According to the method, the high-priority service requirement is better ensured, and the priority configuration can be automatically generated based on the reinforcement learning technology only by giving out the visual service requirement; the communication overhead of the control plane is small, the exponential level is not increased along with the increase of the network scale, and a plurality of controllers can obtain the whole network information only by exchanging information of adjacent nodes.

Description

Technical Field

[0001] The present invention relates to the technical field of differentiated services in software-defined wide area networks (SD-WAN), and particularly relates to a strict priority queue scheduling method and system for SD-WAN and differentiated services. Background Art

[0002] Software-defined network (SDN) is a new type of network architecture that separates the control plane from the data plane of network devices, allowing network administrators to dynamically program and adjust network behavior through a centralized controller. This architecture provides higher flexibility and programmability, enabling the network to respond more intelligently to changes in business requirements. Based on SDN, software-defined wide area network (SD-WAN) technology has emerged as the core technology for connecting distributed enterprise networks. SD-WAN uses the principles of SDN to manage wide area network connections, including the Internet, MPLS, and private lines, etc. It allows enterprises to flexibly transmit traffic between branches in different locations and optimize network performance.

[0003] The strict priority queue is a priority-based queue scheduling mechanism that allows high-priority data packets to be processed before low-priority data packets, which can effectively guarantee the quality of service for critical services, ensure that high-priority traffic is not affected by low-priority traffic, and thus meet the differentiated service quality requirements. However, in the SD-WAN scenario, the strict priority queue scheduling faces some challenges. First, the number of priorities is usually limited by the number of priority queues provided by the switch, and in SD-WAN, the number of services far exceeds the number of priority queues, and the limited priorities are difficult to meet the different needs of differentiating various services. Second, the performance of the strict priority queue depends severely on the specific configuration of the priorities, and in the SD-WAN scenario, the network topology is complex and the types of services are numerous, and calculating the optimal configuration is complex. In addition, in the SD-WAN scenario, to ensure scalability and fault tolerance, each branch of the network is controlled by a separate controller, and each controller usually only has information about local traffic and network conditions, and the current scheduling algorithms lack an effective cooperation mechanism between controllers. The existing SDN-based scheduling algorithms are based on a unified perspective of the entire network, and a large amount of communication across the wide area network is required to synchronize the network-wide information when applied in the SD-WAN scenario, which not only has a huge overhead, but also this overhead will increase exponentially as the network scale increases. In addition, some scheduling methods operate entirely locally and only work at the single router level, and cannot utilize the information of other parts of the network to calculate the global optimal strategy.

[0004] A scheduling method considering communication delay, which is used to construct a power communication service data resource library. This method identifies and classifies the header information of the power communication service data stream, divides the services into several categories, and adopts different scheduling algorithms. Specifically, Class I services are scheduled using the strict priority queue (PQ) algorithm, while Class II, III, and IV services use the improved weighted fair queue (WFQ) algorithm. This method also introduces a dynamic bandwidth allocation mechanism, which adjusts the bandwidth allocation in real time according to the changes in queue length and weight to optimize network performance and stability. However, this method still has some limitations. First, the service requirements are relatively simple. Class I services can be intuitively divided into a few fewer priorities, so using a strict priority queue can meet the requirements. But in many scenarios, more priorities may be needed to meet the needs of different services and solve the priority allocation problem. Second, although this method introduces dynamic bandwidth allocation, its scheduling strategy is relatively fixed and can only be selected from several fixed schemes, lacking the ability to adapt to the network state in real time.

[0005] A time-sensitive network (TSN) cross-domain traffic scheduling method aims to solve the problems of time synchronization, traffic congestion, and resource scheduling in a multi-domain environment. This method models through a Markov decision process, converts the cross-domain scheduling and routing problems into decision problems in a multi-dimensional action space, and combines the main network, target network, and Munchausen mechanism to optimize the selection of time slots and routes. Specifically, this method includes steps such as initializing the TSN state, selecting actions through the main network and greedy strategy, calculating rewards and the next state, storing transitions, sampling transitions, calculating the time difference target value, predicting Q values, calculating the mean square error and updating the main network parameters, periodically updating the target network parameters, judging the end condition, and outputting the scheduling strategy. However, this method still has some limitations. First, when dealing with multi-domain scheduling, its priority mapping mechanism is too simple to effectively handle the dynamic mapping of a large number of priorities. Although this patent considers cross-domain transmission, its implementation method is relatively complex and strongly coupled with several protocols, and may face compatibility and scalability problems in practical applications. Summary of the Invention

[0006] The purpose of the present invention is to provide a strict priority queue scheduling method and system for SD-WAN and differentiated services to solve at least one of the technical problems in the above background technology.

[0007] To achieve the above purpose, the present invention adopts the following technical solutions:

[0008] In the first aspect, the present invention provides a strict priority queue scheduling method for SD-WAN and differentiated services, including:

[0009] Based on the in-flow INT technology, obtain the delay, throughput, and related queue information of each traffic flow in the network; among them, by inserting an INT header into the data stream, the data packet can carry network status information when passing through each hop of network device, and finally collect and report it to the controller at the egress node to achieve real-time monitoring of the network status;

[0010] Based on the graph convolutional deep reinforcement learning algorithm, according to the maximum delay and minimum throughput requirements of various services, automatically assign reasonable priorities to different services to guide the scheduling process of the strict priority queue; among them, use the DGN multi-agent reinforcement learning algorithm to construct a multi-agent system. In this system, each branch is deployed with an intelligent queue agent, and these agents only need to exchange information with neighbor nodes to integrate the traffic conditions of the entire network;

[0011] Adopt an adaptive algorithm of dynamic queue mapping to flexibly map more priorities to fewer strict priority queues to expand the priority space.

[0012] As a further limitation of the first aspect of the present invention, the INT header adds fields for recording processing delay and queuing delay, as well as Qindex and Qdepth fields to obtain fine-grained queue information; among them, the Qindex field is used to identify the index of the queue, and the Qdepth field is used to record the depth of the queue.

[0013] As a further limitation of the first aspect of the present invention, in the INT collection process, the collection of the network status is achieved through the following steps: when the traffic arrives at the network entrance, insert an INT header into a part of the data packets of the flow; when the data packet is about to leave the network, remove the INT header at the egress node and collect the information therein; at the same time, when the data packet is about to leave the network, update the throughput of the flow and store the result in the switch register.

[0014] As a further limitation of the first aspect of the present invention, the model of the intelligent queue agent designed based on DGN includes three levels: an encoder, a convolutional layer, and a Q network; the encoder is used to encode observation sets of different dimensions into the same dimension, and map high-dimensional network status data into a lower-dimensional and more effective representation; the input of the convolutional layer is the output of the local encoder and the output of the neighbor encoder, and uses the attention mechanism to integrate local and neighbor network status information to achieve the perception of the network status information of the entire network when only communicating with neighbors; the Q network takes the output of the last convolutional layer as the input and uses a neural network to evaluate the value of each action to determine the best priority adjustment strategy.

[0015] As a further limitation of the first aspect of the present invention, a dynamic queue mapping mechanism is adopted to optimize the scheduling order of data packets by adjusting the queue boundary in real time; when a data packet arrives at the switch, it is judged according to the priority of the data packet and the boundary value of the current queue; if the priority of the data packet is less than the boundary value of the current queue, the data packet will be pushed up to a queue with a higher priority.

[0016] As a further limitation of the first aspect of the present invention, during the pushing-up stage, the boundary values of each queue will continuously increase until the priority of a data packet arriving at a certain moment may be lower than the boundary value of the queue with the highest priority and it cannot be normally queued; when the above situation occurs, the boundary of the queue with the highest priority is reduced to be the same as the priority of the data packet, and the queue boundaries of other queues are also reduced by the same value.

[0017] In a second aspect, the present invention provides a strict priority queue scheduling system for SD-WAN and differentiated services, including:

[0018] A network status collection module, which is used to obtain the delay, throughput and related queue information of each service flow in the network based on the in-flow INT technology; wherein, by inserting an INT header into the data flow, the data packet can carry network status information when passing through each hop of network devices, and finally the network status is collected and reported to the controller at the egress node to realize real-time monitoring of the network status;

[0019] A priority assignment module, which is used to automatically assign reasonable priorities to different services based on the graph convolutional deep reinforcement learning algorithm according to the maximum delay and minimum throughput requirements of various services to guide the scheduling process of the strict priority queue; wherein, the DGN multi-agent reinforcement learning algorithm is used to construct a multi-agent system. In this system, each branch is deployed with an intelligent queue agent, and these agents only need to exchange information with neighbor nodes to integrate the traffic conditions of the entire network;

[0020] A priority extension module, which is used to adopt an adaptive algorithm of dynamic queue mapping to flexibly map more priorities to fewer strict priority queues to expand the priority space.

[0021] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the strict priority queue scheduling method for SD-WAN and differentiated services as described in the first aspect is realized.

[0022] Fourthly, the present invention provides a computer device, comprising a memory and a processor, where the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the strict priority queue scheduling method for SD-WAN and differentiated services as described in the first aspect.

[0023] Fifthly, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device runs, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the strict priority queue scheduling method for SD-WAN and differentiated services as described in the first aspect.

[0024] Term Explanation:

[0025] ① Software Defined Network: Software Defined Network (SDN) is a new type of network innovation architecture proposed by the Clean-Slate research group at Stanford University in the United States. It is a way to achieve network virtualization. Its core technology, OpenFlow, separates the control plane and data plane of network devices, thereby realizing flexible control of network traffic, making the network more intelligent as a pipeline, and providing a good platform for the innovation of core networks and applications.

[0026] ② P4 Programmable Data Plane: Programming Protocol-independent Packet Processors (P4) is a domain-specific language for network devices, specifying how data plane devices (switches, NICs, routers, filters, etc.) process packets.

[0027] ③ INT In-band Network Telemetry: In-network Telemetry is a network information collection technology that records the forwarding and processing process and time information of a packet through a specific header structure, thereby measuring fine-grained network status information.

[0028] Advantages of the present invention: It better ensures the high-priority service requirements: Based on a strict priority queue, it solves the challenges in its application to the SD-WAN scenario, and fully utilizes the advantage of the strict priority to ensure the service quality requirements of high-priority services; Simple configuration: There is no need for users to perform complex parameter design for services and networks. Only by giving intuitive service requirements, the priority configuration can be automatically generated based on reinforcement learning technology; The communication overhead of the control plane is small and does not increase exponentially with the increase of the network scale: Based on the DGN technology, multiple controllers do not need to exchange information pairwise. Only adjacent nodes need to exchange information to obtain the network-wide information and perform cooperative scheduling with other nodes; High flexibility: Based on the characteristics of the P4 protocol independence and platform, the scheduling mechanism is not bound to the underlying transport protocol and specific devices, and can be widely applied to various transport protocols and programmable switching devices.

[0029] The advantages of the additional aspects of the present invention will be more clearly given in the following description part, or can be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0031] Figure 1 It is a functional principle framework diagram of a highly scalable intelligent scheduling system for a strict priority queue according to an embodiment of the present invention.

[0032] Figure 2 It is an INT header structure diagram according to an embodiment of the present invention.

[0033] Figure 3 It is an SD-WAN scenario diagram according to an embodiment of the present invention.

[0034] Figure 4 It is a functional principle framework diagram of a priority allocation module according to an embodiment of the present invention.

[0035] Figure 5 It is a model structure framework diagram of an intelligent queue proxy based on DGN according to an embodiment of the present invention.

[0036] Figure 6 It is a flowchart of a dynamic queue mapping mechanism according to an embodiment of the present invention.

[0037] Figure 7 It is a structural diagram of a typical partially meshed SD-WAN network according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

[0039] Those skilled in the art of the present technology can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the art to which the present invention pertains.

[0040] It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with their meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless defined as herein.

[0041] Those skilled in the art of the present technology can understand that, unless specifically stated, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0042] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. Without conflict, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0043] To facilitate the understanding of the present invention, the present invention will be further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0044] Those skilled in the art should understand that the drawings are only schematic diagrams of the embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0045] The present invention relates to the differentiated service problem of software-defined wide area network (SD-WAN), and mainly proposes a strict priority queue scheduling method and corresponding system based on multi-agent reinforcement learning. The present invention uses the flow INT (in-band network telemetry) technology to measure latency and throughput, and uses the DGN (graph convolution deep reinforcement learning) algorithm to automatically calculate the priority configuration according to the network status and business needs. At the same time, the present invention also proposes a queuing mechanism based on the P4 programmable data plane, which can expand the number of priorities. The method described in the present invention can effectively meet the differentiated service quality requirements of different businesses in the SD-WAN scenario, and has strong scalability and is suitable for large-scale networks.

[0046] The low communication overhead SD-WAN queue scheduling method and system for differentiated services proposed in the present invention utilizes the unique communication mechanism of DGN to allow the controller to obtain the service status information of the entire network by exchanging information only with neighbors. While avoiding the large amount of overhead caused by synchronization information in the SD-WAN scenario, the controllers cooperate with each other to calculate the optimal priority configuration. In addition, the present invention proposes a dynamic queue mapping mechanism that can provide more priority numbers when the number of queues provided by the switch hardware is small, so as to better adapt to the large number of services in the SD-WAN scenario. The priority configuration and mapping mechanism cooperate with each other to achieve intelligent, flexible, and easy-to-configure strict priority queue scheduling under the conditions of low communication overhead and hardware limitations, so as to meet the differentiated service quality requirements of different services. From the above perspective, it is difficult to find alternative solutions in this scenario. However, for other scenarios, such as switches providing queues other than strict priority queues, or scenarios with simpler services and smaller network scales, there should be some other solutions proposed.

[0047] Example 1

[0048] like Figure 1As shown in the figure, in the first embodiment, a strict priority queue scheduling system for SD-WAN and differentiated services is first provided. The system includes: a network state collection module, which is used to obtain the delay, throughput, and related queue information of each service flow in the network based on the in-flow INT technology. Specifically, by inserting an INT header into the data stream, the data packet can carry network state information when passing through each hop of the network device, and finally collect and report it to the controller at the egress node to achieve real-time monitoring of the network state. A priority assignment module, which is used to automatically assign reasonable priorities to different services based on the graph convolutional deep reinforcement learning algorithm according to the maximum delay and minimum throughput requirements of various services to guide the scheduling process of the strict priority queue. Specifically, the DGN multi-agent reinforcement learning algorithm is used to construct a multi-agent system. In this system, intelligent queue agents are deployed in each branch, and these agents only need to exchange information with neighboring nodes to integrate the traffic conditions of the entire network. A priority extension module, which is used to adopt an adaptive algorithm of dynamic queue mapping to flexibly map more priorities to fewer strict priority queues to expand the priority space.

[0049] Specifically, the network state collection module: Based on the in-flow INT technology, the accurate measurement of the delay, throughput, and related queue information of each flow in the network serves as an important basis for subsequent queue scheduling decisions. This module is optimized for the specific requirements of SD-WAN based on the traditional INT technology. It significantly reduces the header overhead, simplifies the acquisition process, and can measure fine-grained network information without time synchronization.

[0050] Specifically, the priority assignment module: Based on the graph convolutional deep reinforcement learning (DGN) algorithm, it automatically assigns reasonable priorities to different services according to the maximum delay and minimum throughput requirements of various services to guide the scheduling process of the strict priority queue. Considering the actual situation that multiple controllers manage local scheduling policies in the SD-WAN scenario, this solution uses the DGN multi-agent reinforcement learning algorithm to construct an efficient collaborative multi-agent system. In this system, intelligent queue agents are deployed in each branch, and these agents only need to exchange information with neighboring nodes to integrate the traffic conditions of the entire network, including key metrics such as the average delay and throughput of each flow. In this way, the agent can automatically adjust the priority settings, calculate the optimal priority configuration scheme that meets the service differentiated quality of service requirements while maintaining good scalability of the wide area network, thereby realizing the dynamic optimal allocation of network resources and improving the overall network service quality.

[0051] Specifically, the priority expansion module: By adopting an adaptive algorithm running on the P4 programmable data plane, it flexibly maps more priorities to fewer strict priority queues, thereby effectively expanding the priority space. This algorithm fully considers the processing requirements of high-priority data packets in its design. By dynamically adjusting the mapping strategy, it ensures that high-priority data packets are processed first as much as possible, thus maximizing the service quality of high-priority services under limited hardware resources and enhancing the support ability of the entire network for diversified services.

[0052] In the SD-WAN scenario, in order to accurately collect network status for subsequent differential service quality configuration, this embodiment designs a network status collection scheme based on in-situ INT (In-situ Measurement). As Figure 2 shown, this scheme inserts an INT header into the data stream, enabling data packets to carry network status information when passing through each hop of network devices, and finally collecting and reporting it to the controller at the egress node to achieve real-time monitoring of network status. The following introduces this module from two aspects: header design and collection process.

[0053] Based on the official INT specification, corresponding modifications and upgrades are made according to the SD-WAN scenario and scheduling requirements, and a new packet structure is designed. Regarding the measurement object, delay is one of the important indicators of network status. Delay can be divided into four parts: processing delay, queuing delay, transmission delay, and propagation delay. However, in the SD-WAN scenario, processing delay and queuing delay are mainly concerned. This is because the measurement of transmission delay requires time synchronization, and in the wide area network scenario, time synchronization between different nodes is relatively complex and costly. In addition, the wide area network scenario usually has a large bandwidth, and the transmission delay is relatively small and can be ignored. The propagation delay is relatively fixed and only needs to be measured in advance and then added later, without the need for real-time measurement at each node. Based on the above considerations, the original INT header is optimized. Redundant fields are deleted to reduce the size of the header and transmission overhead. At the same time, fields for recording processing delay and queuing delay are added, as well as Qindex and Qdepth fields to obtain fine-grained queue information. Specifically, the Qindex field is used to identify the index of the queue, and the Qdepth field is used to record the depth of the queue. In this way, the processing situation of data packets in the queue can be more accurately understood at each node, providing richer information for subsequent scheduling and priority configuration.

[0054] In the INT collection process design, the collection of network status is achieved through the following steps: When traffic arrives at the network entrance, INT headers are inserted into a part of the packets of the flow. These packets will serve as the "messengers" of the network status, carrying network status information at each hop. When the packets pass through each hop of the network device, the device will insert the processing delay and queuing delay of passing through this node. This information is recorded in the INT header and accumulates continuously as the packets are transmitted. When the packets are about to leave the network, the INT headers are removed at the egress node and the information therein is collected. By analyzing this information, the average delay of the flow in the network can be calculated and the result is stored in the switch register. At the same time, when the packets are about to leave the network, the throughput of the flow is updated and the result is stored in the switch register. Throughput is another important metric for measuring network performance, which reflects the amount of data that the network can transmit per unit time. The controller retrieves this stored data from the switch at regular intervals. By regularly obtaining this data, the controller can understand the changes in the network status in real time and make corresponding scheduling and configuration adjustments based on this information. It should be noted that each node can only directly obtain the delay and throughput of the services with this node as the egress node.

[0055] Through the above design, it is possible to achieve real-time and accurate collection of network status in an SD-WAN scenario as Figure 3 shown. This provides reliable data support for subsequent scheduling strategy calculations and can better meet the complex and differentiated quality of service requirements of different services.

[0056] The priority allocation module uses the DGN algorithm to construct a multi-agent system. Based on the network status information collected by the above network status collection module and the information obtained by communicating with the intelligent queue agents of adjacent nodes, it calculates the queue scheduling strategy, that is, the priorities of each service.

[0057] As Figure 4 shown, an intelligent queue agent runs on the controller of each node. Based on the network information collected by the above network status collection scheme and the information exchanged with the agents of adjacent nodes, it assigns priorities to each service and distributes them to the switch, so as to guide the packets of the corresponding services to enter different strict priority queues when passing through this node, thus meeting the differentiated quality of service requirements of each service.

[0058] Each intelligent queue agent has different observation sets, as well as the same action set and reward function. The specific design is as follows:

[0059] Action Set: A compact action space is adopted, that is, increasing or decreasing the priority of a certain service by 1. Correspondingly, let the observation set contain the priorities of current services, so as to ensure reasonable adjustment actions can be made. The main purpose of this design is that the DGN algorithm used by the intelligent queue agent is based on the Deep Q-learning (DQN) algorithm. If the agent is allowed to adjust the priorities of multiple services at once or allow the priority to be adjusted to any value instead of increasing or decreasing by 1, the action space will become too large to train an effective policy.

[0060] Observation Set: As mentioned before, the observation set contains the priorities of current services. In addition, the observation set also includes the information collected by the network status collection module, that is, the average delay and throughput of services with the current node as the egress node. These data reflect the performance metrics of services under the current network status and are important bases for the agent to make decisions. By collecting these data, the agent can understand the load situation of the current network and the priority requirements of services.

[0061] Reward Function: The intelligent queue agent adopts a simple weighted reward function to punish the situations that do not meet the service requirements. Specifically, if the delay or throughput of a certain service does not reach the preset threshold, the agent will be punished. In this way, the agent is motivated to adopt a priority policy that can meet the service requirements, thereby improving the overall performance of the network.

[0062] The model structure of the intelligent queue agent designed based on DGN is as Figure 5 shown, consisting of three levels: an encoder, a convolutional layer, and a Q-network. The following will elaborate on these three levels in detail.

[0063] Encoder: First, use the encoder to process the input of the model. The encoder consists of only a linear layer and a Rectified Linear Unit (ReLU). It has two functions. First, as mentioned before, each node can only directly obtain the average delay and throughput data of services with this node as the egress node. This is the main part of the observation set of each agent. Since the number of services with each node as the destination node is different, the dimension number of the observation set of each node is different. Using an autoencoder, the observation sets with different dimensions can be encoded into the same dimension, so that the same model can be used for processing subsequently. This is the guarantee for sharing model parameters between agents and reducing training costs. In addition, it maps the high-dimensional network state data into a lower-dimensional and more effective representation, extracts the important information in the network state, and reduces the overhead of subsequent calculations and inter-agent communication.

[0064] Convolutional layer: The input to the convolutional layer is the output of the local encoder and the output of the neighbor encoder. They are placed on the adjacency matrix representing the network topology. For other non-neighbor nodes in the network, since they cannot communicate with them to obtain information, they are set to 0. The convolutional layer uses an attention mechanism to integrate the local and neighbor network state information. This process can also be regarded as a dynamic convolutional process, similar to integrating the information of a pixel point and its surroundings through convolution in image processing. By stacking convolutional layers, the receptive field range of the agent can be expanded, so as to perceive the network state information of the whole network when only communicating with neighbors.

[0065] Value network: The last layer is the value network, similar to the classic DQN algorithm. It receives the output of the last convolutional layer as input and uses a neural network to evaluate the value of each action, so as to determine the best priority adjustment strategy. Through the value network, the agent learns which actions to take in different environmental states to maximize the reward.

[0066] In the above levels, other levels except the encoder share weights among different agents. After training is completed, all agents use the same model. Considering that the encoder has only a single-layer linear layer, the training cost of the above network is low, ensuring the scalability of the network.

[0067] Through the above design, agents can cooperate with each other, obtain the information of the whole network when only communicating with neighbors, and adjust the scheduling strategy. This design makes full use of the advantages of DGN, enabling each node to dynamically adjust its own priority queue configuration according to the network topology and neighbor information, so as to achieve efficient cooperative scheduling of multiple queues.

[0068] In this embodiment, the following measures are taken to optimize the model training process:

[0069] State initialization: When initializing the priorities of each service, we simply sort their delay and throughput requirements and assign corresponding priorities according to the degree of strictness of their requirements. Experiments show that our algorithm does not depend on a specific priority configuration and can always converge to the optimal strategy, so a simple state initialization algorithm can meet the requirements.

[0070] ε-greedy exploration strategy: Use the ε-greedy exploration strategy, that is, the intelligent queuing agent does not always choose the action with the optimal value, but randomly selects an action with a certain probability. This helps prevent the agent from falling into a local optimum and ensures a comprehensive exploration of the action space. During the training process, the probability of randomly selecting an action will gradually decrease, and finally a stable optimal strategy will be obtained.

[0071] Replay Buffer: The replay buffer is used to store the records of the intelligent queue agent making decisions and obtaining observations. During the training process, these data are randomly sampled to update the model, improving data utilization and stabilizing the training process.

[0072] Target Network Mechanism: The DGN algorithm used in this invention is improved from DQN, and algorithms based on DQN face instability problems, that is, the frequent change of values leads to the model overestimating or underestimating the value of the current action during the training process and fluctuating repeatedly for a long time. To solve this problem, a target network mechanism is introduced, using two separate networks for action selection and Q-value update. This helps to maintain the stability of training and prevent the overestimation of Q-values. After calculating the priority adjustment strategy, the priority policy distribution module is responsible for distributing the priority policy generated by the intelligent queuing module to the switch. This module communicates with the switch through the P4Runtime API and configures the priority policy into the queue scheduler of the switch.

[0073] As Figure 6 shown, the priority extension module is responsible for solving the problem of insufficient priority caused by the limited number of switch hardware queues. The priority extension module effectively expands the priority space by running a set of adaptive algorithms on the P4 programmable data plane to flexibly map more priorities to fewer strict priority queues.

[0074] The core idea of the dynamic queue mapping mechanism is that when mapping more priorities to fewer priority queues, different mapping methods will lead to different degrees of priority inversion. Priority inversion means that low-priority packets are scheduled before high-priority packets, thus affecting the performance of high-priority services. To minimize priority inversion as much as possible, this embodiment adopts a dynamic queue mapping mechanism to optimize the packet scheduling order by adjusting the queue boundaries in real time.

[0075] The calculation process of the optimal mapping consists of two stages: pushing up and pulling down.

[0076] When a packet arrives at the switch, the system judges according to the priority of the packet and the boundary value of the current queue. If the priority of the packet is less than the boundary value of the current queue, the packet will be pushed up to a higher-priority queue. Specifically, the system starts from queue 7 and checks the boundary value of each queue one by one until it finds a queue whose boundary value is greater than or equal to the priority of the packet. At this time, the packet is inserted into this queue, and the boundary value of this queue is updated to the priority of the packet. This process is called pushing up, which ensures that high-priority packets can be scheduled first, thus reducing the occurrence of priority inversion.

[0077] In the upward push stage, the boundary values of each queue will continuously increase until, at a certain moment, the priority of the arriving data packet may be lower than the boundary value of the queue with the highest priority (queue 0), and it cannot be enqueued normally. When the above situation occurs, the data packet is made to enter queue 0, the boundary of queue 0 is reduced to the same as the priority of the data packet, and the queue boundaries of other queues are also reduced by the same value. After repeatedly going through the two stages, the queue boundaries of each queue will converge to a certain value according to the current traffic.

[0078] In addition, an early pull-down mechanism is introduced to better ensure that the high-priority service requirements are met. In practical applications, if the boundary value of queue 1 is too large, it may cause low-priority data packets to be pushed up to queue 0 subsequently, resulting in serious priority inversion and having a greater impact on high-priority services. To solve this problem, this embodiment sets a warning boundary value. After the upward push, when the boundary value of queue 1 is greater than the warning boundary value, the system will perform a pull-down operation in advance. Specifically, the system will reduce the boundary value of queue 1 to the warning boundary value and make the boundary values of other queues also decrease by the same value. This operation can effectively control the degree of inversion in queue 0 and avoid the performance of high-priority services being severely affected.

[0079] Embodiment 2

[0080] In this embodiment, a strict priority queue scheduling method for SD-WAN and differentiated services is implemented using the system shown in Embodiment 1, including:

[0081] Based on the in-flow INT technology, obtain the delay, throughput, and relevant queue information of each service flow in the network; among them, by inserting an INT header into the data stream, the data packet can carry network state information when passing through each hop of the network device, and finally collect and report it to the controller at the egress node to achieve real-time monitoring of the network state;

[0082] Based on the graph convolutional deep reinforcement learning algorithm, automatically assign reasonable priorities to different services according to the maximum delay and minimum throughput requirements of various services to guide the scheduling process of the strict priority queue; among them, the DGN multi-agent reinforcement learning algorithm is used to construct a multi-agent system. In this system, intelligent queue agents are deployed in each division, and these agents only need to exchange information with neighbor nodes to integrate the traffic conditions of the entire network;

[0083] Adopt an adaptive algorithm of dynamic queue mapping to flexibly map more priorities to fewer strict priority queues to expand the priority space.

[0084] Such as Figure 7As shown, it is a typical partially meshed SD-WAN network for communication between the company headquarters and each branch. Intelligent queuing agents can be run separately at each branch for configuring service priorities. First, QoS metrics are formulated for various different services of the company. For example, for video conferencing, big data, industrial control, and regular office work, each has different latency and throughput requirements. The intelligent queuing agent will only communicate with adjacent nodes. For example, the intelligent queuing agent of Branch 1 only needs to exchange information with the agents of the headquarters and Branch 2 to calculate the optimal priority configuration. Usually, the number of priorities is the same as the number of strict priority queues provided by the switch. Commonly, it is 8 or 16 queues, that is, corresponding to 8 or 16 priorities respectively. By deploying the above-mentioned strict priority queue scheduling method for SD-WAN and differentiated services described in this embodiment on the switch, the number of priorities can be expanded to assign 64 different priorities to services. This embodiment focuses on queue scheduling and does not require the use of a specific transport protocol. Enterprises can use different transport protocols such as MPLS and IP according to their needs.

[0085] In non-SD-WAN network scenarios, such as in data center networks, sometimes there is also a situation where the number of priority queues provided by the switch is too small to meet the diverse service requirements. For example, the switch only provides 8 priorities, but there are 32 different services in the data center with their own different quality of service requirements. At this time, the dynamic queue mapping module of the present invention can be used alone to expand the number of priority queues.

[0086] Embodiment 3

[0087] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the strict priority queue scheduling method for SD-WAN and differentiated services as described above is implemented. The method includes:

[0088] Based on the in-flow INT technology, obtain the latency, throughput, and relevant queue information of each service flow in the network; wherein, by inserting an INT header into the data stream, the data packet can carry network state information when passing through each hop of the network device, and finally collect and report it to the controller at the egress node to realize real-time monitoring of the network state;

[0089] Based on the graph convolutional deep reinforcement learning algorithm, according to the maximum latency and minimum throughput requirements of various services, reasonable priorities are automatically assigned to different services to guide the scheduling process of the strict priority queue; among them, the DGN multi-agent reinforcement learning algorithm is used to construct a multi-agent system. In this system, intelligent queue agents are deployed in each branch. These agents only need to exchange information with neighboring nodes to integrate the traffic conditions of the entire network;

[0090] An adaptive algorithm of dynamic queue mapping is adopted to flexibly map more priorities to fewer strict priority queues, expanding the priority space.

[0091] Example 4

[0092] This Example 4 provides a computer device, including a memory and a processor. The processor and the memory communicate with each other. The memory stores program instructions executable by the processor. The processor calls the program instructions to execute the strict priority queue scheduling method for SD-WAN and differentiated services as described above. This method includes:

[0093] Based on the in-flow INT technology, the latency, throughput, and related queue information of each service flow in the network are obtained; among them, by inserting an INT header into the data stream, the data packet can carry network status information when passing through each hop of network devices, and finally, the network status is collected and reported to the controller at the egress node to achieve real-time monitoring of the network status;

[0094] Based on the graph convolutional deep reinforcement learning algorithm, according to the maximum latency and minimum throughput requirements of various services, reasonable priorities are automatically assigned to different services to guide the scheduling process of the strict priority queue; among them, the DGN multi-agent reinforcement learning algorithm is used to construct a multi-agent system. In this system, intelligent queue agents are deployed in each branch. These agents only need to exchange information with neighboring nodes to integrate the traffic conditions of the entire network;

[0095] An adaptive algorithm of dynamic queue mapping is adopted to flexibly map more priorities to fewer strict priority queues, expanding the priority space.

[0096] Example 5

[0097] This Example 5 provides an electronic device, including: a processor, a memory, and a computer program; among them, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device runs, the processor executes the computer program stored in the memory so that the electronic device executes the instructions for implementing the strict priority queue scheduling method for SD-WAN and differentiated services as described above. This method includes:

[0098] Based on the in-flow INT technology, obtain the delay, throughput, and related queue information of each service flow in the network; among them, by inserting INT headers into the data stream, the data packets can carry network status information when passing through each hop of network devices, and finally collect and report to the controller at the egress node to achieve real-time monitoring of the network status;

[0099] Based on the graph convolutional deep reinforcement learning algorithm, according to the maximum delay and minimum throughput requirements of various services, automatically assign reasonable priorities to different services to guide the scheduling process of the strict priority queue; among them, the DGN multi-agent reinforcement learning algorithm is used to construct a multi-agent system. In this system, each branch is deployed with an intelligent queue agent, and these agents only need to exchange information with neighbor nodes to integrate the traffic conditions of the entire network;

[0100] Adopt an adaptive algorithm of dynamic queue mapping to flexibly map more priorities to fewer strict priority queues to expand the priority space.

[0101] In summary, the strict priority queue scheduling method and system for SD-WAN and differentiated services according to the embodiments of the present invention include three algorithmic strategies: First, a network status collection algorithm based on in-flow INT is proposed to achieve accurate measurement of end-to-end network status information in-band and without time synchronization; Second, a priority assignment algorithm based on DGN is proposed to effectively integrate network-wide information and calculate the globally optimal scheduling strategy while ensuring high scalability in distributed computing, meeting the differentiated quality of service requirements of different services; Third, a priority extension algorithm based on the P4 programmable data plane is proposed to ensure the flexibility of priority scheduling in the case of a large number of service types and limited hardware resources. The system is jointly coordinated by a network status collection module, a priority assignment module, and a priority extension module to complete scheduling. Its working process is as follows: First, the network status collection module collects service information with the current node as the egress node; Then, the priority assignment module calculates the priority based on the above service data and calculates the priority configuration while only communicating with the neighbor node controller; Finally, during the forwarding process, the data packet will be dynamically mapped to a limited number of strict priority queues by the priority extension module according to the priority calculated above. A network status collection algorithm based on in-flow INT is proposed. Specifically, by inserting lightweight INT headers into some service data packets, the queuing delay and processing delay of each hop are collected in real time to achieve accurate network status perception to support dynamic scheduling decisions. A priority assignment algorithm based on DGN is proposed. Specifically, through the network structure of DGN that combines graph convolutional ideas and attention mechanisms, agents can cooperate efficiently. Each agent only needs to communicate with agents in adjacent divisions to obtain the service status information of the entire network, realizing that the agents of the entire network cooperate to configure priorities for each service under the conditions of low communication overhead and high scalability, meeting the quality of service requirements of each service. A dynamic queue mapping method based on the P4 programmable data plane is proposed. Specifically, through three actions of pushing up, pulling down, and pulling down in advance, the mapping strategy is dynamically adjusted according to the currently arriving data packets, mapping more priorities to fewer strict priority queues while ensuring that high-priority data packets are preferably processed as much as possible, thereby achieving efficient priority extension under limited hardware resources and providing more flexible differentiated service capabilities.

[0102] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0104] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device to perform a series of operation steps on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0106] Although the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, it is not a limitation on the protection scope of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or deformations that can be made by those skilled in the art without creative efforts should be covered within the protection scope of the present invention.

Claims

1. A strict priority queue scheduling method for SD-WAN and differentiated services, characterized in that Including: Based on the in-flow INT technology, obtain the delay, throughput, and related queue information of each traffic flow in the network; among them, by inserting an INT header into the data stream, the data packet can carry network status information when passing through each hop of network devices, and finally collect and report it to the controller at the egress node to achieve real-time monitoring of the network status; Based on the graph convolutional deep reinforcement learning algorithm, automatically assign reasonable priorities to different services according to the maximum delay and minimum throughput requirements of various services to guide the scheduling process of the strict priority queue; among them, the DGN multi-agent reinforcement learning algorithm is used to construct a multi-agent system. In this system, each branch is deployed with an intelligent queue agent, and these agents only need to exchange information with neighbor nodes to integrate the traffic conditions of the entire network; Adopt an adaptive algorithm for dynamic queue mapping to flexibly map more priorities to fewer strict priority queues and expand the priority space.

2. The strict priority queue scheduling method for SD-WAN and differentiated services according to claim 1, characterized in that The INT header adds fields for recording processing delay and queuing delay, as well as Qindex and Qdepth fields to obtain fine-grained queue information; among them, the Qindex field is used to identify the index of the queue, and the Qdepth field is used to record the depth of the queue.

3. The strict priority queue scheduling method for SD-WAN and differentiated services according to claim 2, wherein In the INT collection process, the collection of network status is achieved through the following steps: when traffic arrives at the network entrance, insert an INT header into a part of the data packets of the flow; When the data packet is about to leave the network, remove the INT header at the egress node and collect the information therein; at the same time, when the data packet is about to leave the network, update the throughput of the flow and store the result in the switch register.

4. The strict priority queue scheduling method for SD-WAN and differentiated services according to claim 1, characterized in that Based on DGN, the model of the intelligent queue agent includes three levels: an encoder, a convolutional layer, and a Q network; the encoder is used to encode observation sets of different dimensions into the same dimension, mapping high-dimensional network status data into a lower-dimensional and more effective representation; the input of the convolutional layer is the output of the local encoder and the output of the neighbor encoder, and the attention mechanism is used to integrate local and neighbor network status information to achieve the perception of the network status information of the entire network when only communicating with neighbors; the Q network takes the output of the last convolutional layer as the input and uses a neural network to evaluate the value of each action to determine the best priority adjustment strategy.

5. The strict priority queue scheduling method for SD-WAN and differentiated services according to claim 1, characterized in that, Adopt a dynamic queue mapping mechanism to optimize the scheduling order of data packets by adjusting the queue boundary in real time; when a data packet arrives at the switch, judge according to the priority of the data packet and the boundary value of the current queue; if the priority of the data packet is less than the boundary value of the current queue, the data packet will be pushed up to a higher-priority queue.

6. The strict priority queue scheduling method for SD-WAN and differentiated services according to claim 5, characterized in that In the push-up stage, the boundary values of each queue will continue to rise until the priority of a data packet arriving at a certain moment may be lower than the boundary value of the highest-priority queue and cannot be normally enqueued; when the above situation occurs, reduce the boundary of the highest-priority queue to the same as the priority of the data packet, and reduce the queue boundaries of other queues by the same value.

7. A strict priority queue scheduling system for SD-WAN and differentiated services, characterized in that, Including: A network status collection module, which is used to obtain the delay, throughput, and related queue information of each service flow in the network based on the in-flow INT technology; among them, by inserting an INT header into the data stream, the data packet can carry network status information when passing through each hop of network devices, and finally collect and report it to the controller at the egress node to achieve real-time monitoring of the network status; A priority assignment module, which is used to automatically assign reasonable priorities to different services based on the graph convolutional deep reinforcement learning algorithm according to the maximum delay and minimum throughput requirements of various services to guide the scheduling process of the strict priority queue; among them, the DGN multi-agent reinforcement learning algorithm is used to construct a multi-agent system. In this system, each branch is deployed with an intelligent queue agent, and these agents only need to exchange information with neighbor nodes to integrate the traffic conditions of the entire network; A priority extension module, which is used to adopt an adaptive algorithm of dynamic queue mapping to flexibly map more priorities to fewer strict priority queues to expand the priority space.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the strict priority queue scheduling method for SD-WAN and differentiated services as described in any one of claims 1-6 is implemented.

9. A computer device, characterized in that, It includes a memory and a processor. The processor and the memory communicate with each other. The memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the strict priority queue scheduling method for SD-WAN and differentiated services as described in any one of claims 1-6.

10. An electronic device, characterized in that, It includes: A processor, a memory, and a computer program; among them, the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device runs, the processor executes the computer program stored in the memory so that the electronic device executes the instructions for implementing the strict priority queue scheduling method for SD-WAN and differentiated services as described in any one of claims 1-6.

Images