Switch dynamic ECN configuration method and system based on reinforcement learning, and switch

Abstract

A switch dynamic ECN configuration method and system based on reinforcement learning and a switch belong to the field of network congestion control and AI optimization, and the method comprises the following steps: training a reinforcement learning model by using historical data, deploying the model, collecting multi-dimensional state data S of a switch port in real time, and outputting an ECN configuration action value A; and calculating a reward value R, updating an experience pool in combination with state transition data {S, A, R, S '}, and optimizing a model by adopting a double-inference mechanism, a gradual change epsilon-greedy strategy and historical optimal action backtracking. By dynamically optimizing the ECN threshold, balancing the throughput rate and the queue stability, reducing the packet loss rate and delay jitter and improving the network performance and the model reliability, the method is suitable for a real-time congestion control scene of a large-scale data center switch.

Description

Technical Field

[0001] This application relates to the field of network congestion control and AI optimization, specifically to a method, system, and switch for dynamic ECN configuration of a switch based on reinforcement learning. Background Technology

[0002] With the rapid development of technologies such as high-performance computing, big data analytics, artificial intelligence, and the Internet of Things, and the widespread adoption of centralized storage, distributed storage, and cloud databases, business applications are increasingly retrieving data from the network. This places higher and higher demands on the exchange speed and performance of data center networks, requiring lossless networks with high bandwidth and low latency.

[0003] Currently, the main technologies for implementing network congestion management are PFC (Priority-based Flow Control) and ECN (Explicit Congestion Notification). PFC is a congestion control mechanism initiated when there is congestion at the inlet of a switch, while ECN is a congestion control mechanism initiated at the outlet of a switch.

[0004] The working principle of ECN is as follows: When the switch's egress buffer reaches a set threshold, the switch modifies the ECN bit in the packet header to tag the data with an ECN label. When the data with the ECN label arrives at the receiving end, the receiving end generates a CNP (Congestion Notification Packet) and sends it to the sending end. The CNP contains information about the flow (data flow) or QP (Queue Pair) that caused the congestion. Upon receiving the CNP, the receiving end takes measures to reduce the transmission rate. Therefore, ECN is a congestion control mechanism based on TCP flow or RDMA QP; it only affects the flow or QP that caused the congestion and does not affect other applications.

[0005] ECN (Enhanced Congestion Control) is an important technology for achieving lossless networks, enabling end-to-end congestion management and mitigating the spread and deterioration of congestion. Most mainstream switches currently support ECN functionality, using ECN-aware rate control on end hosts to control the injection rate during congestion. Although ECN-based congestion control schemes have been widely adopted in data center networks, with increasing network speeds and stringent requirements for low-latency operation, data center networks still face many challenges. Determining the appropriate ECN configuration is challenging. For large numbers of switches and heterogeneous network environments, network operators typically need weeks or months to fine-tune ECN settings. ECN tuning becomes even more difficult when data center switches are supplied by different vendors.

[0006] In multi-tenant networks, pre-set static ECN values ​​are often difficult to adapt to tenants with different traffic categories, and data center network operators need to make trade-offs between network utilization and tenant performance.

[0007] The nature of tenant networks results in significant variations in traffic patterns across space and time. Large volumes of data and bursts of traffic coexist within the network. The dynamic nature of various traffic workloads necessitates parameter tuning at different time scales. However, manually updating the ECN tag thresholds of thousands of running switches is both time-consuming and error-prone.

[0008] This invention mainly targets the congestion control mechanism of ECN. Through online learning and inference using reinforcement learning, ECN is dynamically configured, achieving a lossless network with "zero congestion", "zero packet loss" and high port utilization (port utilization is close to 100%). Summary of the Invention

[0009] This application provides a method and system for dynamic ECN configuration of switches based on reinforcement learning, which can solve the technical problem that existing technologies, which configure ECN-related parameters through manual experience, cannot meet the needs of complex and ever-changing networks.

[0010] In a first aspect, embodiments of this application provide a method for dynamic ECN configuration of a switch based on reinforcement learning, the method comprising: Use historical data to train reinforcement learning models offline and generate pre-trained models; After deploying the pre-trained model to the switch, network status data S of the switch port is collected online in real time. Based on the S, the reinforcement learning model outputs the ECN configuration action value A, which includes the low threshold kmin, the high threshold kmat, and the maximum label probability pmax. Collect the next state S', and combine it with A and S' to calculate the reward value R, where R is the actual calculated network performance feedback value; The R and state transition data {S, A, R, S'} are stored in the experience pool, and the reinforcement learning model is updated using the deep Q-network algorithm. The Deep Q-Network algorithm uses a dual inference mechanism to dynamically adjust the ECN configuration. The dual inference mechanism includes selecting A through a gradual ε-greedy strategy, where the ε value decays with the number of training rounds, and when the ε value approaches a stable value, if the current R value is lower than a preset threshold, then A with the highest R value is selected from the historical N rounds, where N is the preset number of training rounds.

[0011] Secondly, embodiments of this application provide a dynamic ECN configuration system for switches based on reinforcement learning, the system comprising: The model pre-training module uses historical data to train reinforcement learning models offline and generate pre-trained models. The model deployment module is used to deploy the pre-trained model to the switch and collect the network status data S of the switch port online in real time. The model update module is used to output an ECN configuration action value A based on the S through the reinforcement learning model, where A includes a low threshold kmin, a high threshold kmat, and a maximum label probability pmax; collect the next state S', combine A and S' to calculate a reward value R, where R is the actual calculated network performance feedback value; store R and the state transition data {S, A, R, S'} into an experience pool, and update the reinforcement learning model through a deep Q-network algorithm; The Deep Q-Network algorithm uses a dual inference mechanism to dynamically adjust the ECN configuration. The dual inference mechanism includes selecting A through a gradual ε-greedy strategy, where the ε value decays with the number of training rounds, and when the ε value approaches a stable value, if the current R value is lower than a preset threshold, then A with the highest R value is selected from the historical N rounds, where N is the preset number of training rounds.

[0012] In conjunction with the second aspect, in one embodiment, the system further includes: The experience pool filtering module compares R with a preset good_reward threshold and only stores experience data where R ≥ good_reward in the experience pool. The model storage module is used to automatically store the updated final model according to preset conditions, including triggering model saving after completing N rounds of training or M rounds of operation.

[0013] Thirdly, embodiments of this application provide a switch, characterized in that it includes: Memory is used to store reinforcement learning model and network state data; The processor, connected to the memory, is used to run reinforcement learning algorithms, execute dynamic ECN configuration, and interact with external AI programs via the Socket communication protocol. The driver configuration module is used to receive ECN configuration action values ​​and update the switch's congestion control parameters.

[0014] In conjunction with the third aspect, in one implementation, the processor employs a virtual trajectory design when running reinforcement learning algorithms, defining training rounds with discretized action space lengths to improve model convergence efficiency.

[0015] The beneficial effects of the technical solutions provided in this application include: By using offline pre-trained models to learn congestion control patterns from historical data in advance, only fine-tuning of model parameters is needed during the online training phase. This can reduce online training time by more than 50% compared to traditional methods, significantly shortening the online training time.

[0016] Real-time collection of multi-dimensional state data (queue length, output rate, ECN tagging and grouping rate), combined with a dual inference mechanism (gradual ε-greedy strategy and historical best action backtracking), reduces exploration costs while avoiding congestion exacerbation caused by misselected actions when reward values ​​decrease, thus achieving dynamic optimization of ECN threshold configuration.

[0017] Dynamic adjustment of ECN parameters can be completed without manual intervention, making it suitable for real-time congestion management of large-scale data center switches, significantly improving the intelligence level of network resource scheduling, and realizing automated and intelligent control. Attached Figure Description

[0018] Figure 1 This is a flowchart of an embodiment of the reinforcement learning-based dynamic ECN configuration method for switches according to this application; Figure 2 This is a schematic diagram of the overall process of an embodiment of the reinforcement learning-based dynamic ECN configuration method for switches in this application. Figure 3 This is a flowchart illustrating the separation line training phase in one embodiment of this application; Figure 4 This is a flowchart illustrating the online training + inference phase in one embodiment of this application; Figure 5 This is a schematic diagram of the label probability curve in one embodiment of this application; Figure 6 This is a schematic diagram of the discretized action space in one embodiment of this application; Figure 7 This is a schematic diagram of action dimensionality reduction in one embodiment of this application; Figure 8 This is a schematic diagram of the principle of a deep Q-network in one embodiment of this application; Figure 9 This is a schematic diagram of the deep Q-network training process in one embodiment of this application; Figure 10 This document describes the convergence of the offline training loss of a deep Q-network in one embodiment of this application. Figure 11This is a schematic diagram of the protocol format for AI-side program receiving data in one embodiment of this application; Figure 12 This is a schematic diagram of the protocol format for sending data by the AI-side program in one embodiment of this application; Figure 13 This is a functional module diagram of an embodiment of the reinforcement learning-based dynamic ECN configuration system for switches according to this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0020] Firstly, embodiments of this application provide a method for dynamic ECN configuration of a switch based on reinforcement learning. In one embodiment, referring to... Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating an embodiment of the reinforcement learning-based dynamic ECN configuration method for switches according to this application. Figure 2 This is a schematic diagram illustrating the overall flow of an embodiment of the reinforcement learning-based dynamic ECN configuration method for switches according to this application. Figure 1 and Figure 2 As shown, the reinforcement learning-based dynamic ECN configuration method for switches includes: Step S1: Offline training of a reinforcement learning model using historical data to generate a pre-trained model. Step S2: After deploying the pre-trained model to the switch, online real-time collection of network status data S of the switch ports. Step S3: Based on S, outputting ECN configuration action value A through the reinforcement learning model, where A includes a low threshold kmin, a high threshold kmat, and a maximum label probability pmax. Step S4: Collecting the next state S', combining A and S', calculating the reward value R, where R is the actual calculated network performance feedback value. Step S5: Storing R and the state transition data {S, A, R, S'} into an experience pool, and updating the reinforcement learning model through a deep Q-network algorithm. The deep Q-network algorithm uses a dual-inference mechanism to dynamically adjust the ECN configuration. The dual-inference mechanism includes selecting A through a gradual ε-greedy strategy, where the ε value decays with the training rounds, and when the ε value approaches a stable value, if the current R value is lower than a preset threshold, then selecting A with the highest R value from the historical N rounds, where N is the preset number of training rounds.

[0021] In this embodiment, ECN technology can accurately label congested traffic, thereby enabling dynamic speed reduction at the sending end. However, network environments are highly complex and dynamically changing. Traditional ECN parameters (such as low threshold kmin, high threshold kmat, and maximum labeling probability pmax) that rely on manual experience cannot adapt to the changing network conditions in real time, resulting in inefficient congestion control or even misjudgments.

[0022] The limitation of traditional Q-Learning lies in the fact that while it maintains a Q-table (state-action value table) to store the reward value Q(s, a) for taking action a in each state s, this method is only applicable to discrete state spaces. In real-world network scenarios, the state space (such as queue length, output rate, etc.) is often continuous, and the number of states is infinite. Traditional Q-tables cannot effectively store and update the value function, limiting their application in complex networks.

[0023] This invention employs Deep Q-network (DQN) reinforcement learning technology to address the following innovative issues: **Value Function Approximation:** A neural network replaces the Q-table, constructing a function Q(s, a; w) that approximates the state-action value function, where w represents the neural network parameters. **Continuous State Space Adaptation:** The neural network can handle continuous state inputs (such as queue length and rate), eliminating the need for a discrete table and significantly improving algorithm flexibility. **Online Dynamic Optimization:** By real-time acquisition of multi-dimensional state data S from switch ports (including queue length, output rate, and ECN tagging rate), the optimal ECN configuration action A is dynamically output, enabling adaptive parameter adjustment. **Efficient Convergence Mechanism:** Combining a gradual ε-greedy strategy with historical best action backtracking balances exploration and utilization, accelerating model convergence and avoiding local optima. Deep Q-networks approximate the value function through neural networks, overcoming the dependence of traditional Q-learning on a discrete state space. This allows for accurate capture of network dynamics, dynamic optimization of ECN parameter configuration, and significantly improved real-time performance, stability, and scalability of congestion control, making it suitable for complex network scenarios such as large-scale data centers.

[0024] Furthermore, in one embodiment, the core feature of the reinforcement learning-based dynamic ECN configuration method for switches is the implementation of a separate training phase and an online training + inference phase. (Refer to...) Figure 3 In the offline training phase, historical ECN configuration data from the existing network (such as low threshold kmin, high threshold kmat, and maximum label probability pmax) is used to train the deep neural network offline. This process incorporates existing network congestion control experience into the model, forming a pre-trained model. The pre-trained model is used for initializing the neural network parameters when the switch is first powered on, shortening the online training convergence time. (Refer to...) Figure 4 In the online training and inference phases, the inference phase outputs the optimal ECN configuration action values ​​A (kmin, kmax, and pmax) based on real-time collected network state data S (queue length, port output rate, and ECN tagging rate). Online training dynamically updates the neural network parameters based on the real-time network state S', the current configuration A, and the evaluation score R (reflecting the effectiveness of the ECN configuration), embedding newly learned knowledge into the model for continuous optimization.

[0025] In one specific embodiment, during the offline training and online training + inference phases, this invention collects the following three types of core data based on each data queue: State data S includes three parameters: current queue length (reflecting the instantaneous congestion level), port output rate (the rate of data packets forwarded per unit time or bit rate), and ECN marking packet output rate (the number or proportion of data packets marked as congested per unit time). ECN configuration data A includes three parameters: low threshold kmin (a threshold to avoid premature marking), high threshold kmat (a threshold to trigger the maximum marking probability), and maximum marking probability pmax (controlling the strength of congestion notification). When the queue length is less than the low threshold, no packets are marked, and the marking probability is 0%. When the queue length exceeds the high threshold, all newly arrived packets are marked. When the queue length is between the low and high thresholds, newly arrived packets are randomly marked, and a maximum marking probability is set; the longer the queue, the higher the marking probability. If the packet length is plotted on the x-axis and the marking probability on the y-axis, the marking probability curve is as follows: Figure 5 As shown. The evaluation score R = (W1×TR+W2×DL)×10000, where TR represents the ratio of port output rate to bandwidth, quantifying bandwidth utilization. DL represents the queue length distribution coefficient (range 0≤DL<1), reflecting queue uniformity. W1 and W2 represent the weighting coefficients of TR and DL, satisfying W1+W2=1. The normalization coefficient of R (×10000) avoids numerical overflow and improves calculation accuracy. W1 and W2 are hyperparameters that can be set according to different network environments and needs. For example, the trained model aims to maximize port utilization, such as W1=0.7 and W2=0.3. Through testing, (W1=0.6, W2=0.4) is a pair of values ​​that can guarantee port utilization and quickly learn the optimal ECN configuration with minimal packet loss when facing unseen network conditions.

[0026] In another specific embodiment, to support trajectory sorting and data correlation analysis, this invention further expands the data content in the data acquisition during offline and online training phases, in addition to the current state S and configured action A: historical action configuration value a, recording the ECN configured action at the previous moment for action sequence modeling; and a timestamp, marking the data acquisition time to support time-dimensional analysis (such as congestion dynamic changes). This extended design enhances the model's ability to learn the temporal dependencies between actions and states, optimizing the synergistic effect between offline and online phases.

[0027] Furthermore, in one embodiment, during the offline training and online training + inference phases, the input features of the deep Q network adopt a K-step state sequence or a state-action combination sequence to enhance the model's ability to perceive dynamic changes in the network.

[0028] Input structure scheme 1: Input only the K-step state value sequence S={s0, s... 1, ..., s0 K+1}, capturing historical state trends (such as changes in queue length and output rate). Input structure scheme two: Input K-step state-action combination sequence S={(s0, a0), (s1, a1), ..., (s0} K+1, a K+1)}, combining action history and state changes. Input structure scheme three: Input K-step state values ​​and the current action combination S={s0, s 1, ..., s0 K+1, a0}, strengthen the influence of the current action on the state. The value of K (≥1) and the state category (such as queue length, output rate, etc.) can be dynamically adjusted according to the training and inference effects to adapt to the needs of different network scenarios. When K=1, and the state only takes the value s, then S is the same as the state data s mentioned above.

[0029] DQN training relies on historical experience data sets {S, A, R, S'}, where S is a generalized state, which can be a K-step state sequence (Scheme 1), a K-step state-action sequence (Scheme 2), or a combination of a K-step state and the current action (Scheme 3). A is the currently configured action value (kmin, kmax, pmax). R is the evaluation score, calculated using the formula R=(W1×TR+W2×DL)×10000. S' is the next true state value (non-generalized state) after executing action A. To support trajectory sorting and analysis, the data collection format is {a, s, r, timestamp, a'}, as shown in Table 1 below.

[0030] Table 1 Data Acquisition Format Table

[0031] The reason for collecting data in this format is that it's a format that can be completed in a single interaction between a human and a machine. This ensures that the collected data can be sorted according to the trajectory. Data in this format facilitates data analysis. For example, even if the issued action configuration doesn't change the state, manually changing the action configuration value makes it easier to analyze and find the optimal action configuration value for that state.

[0032] Furthermore, in one embodiment, before model training, the collected raw data needs to be sorted, optimized, and restructured to ensure the continuity of the trajectory and the standardization of the model training data.

[0033] In this embodiment, when the original data is sorted by timestamp, the trajectory order may become disordered due to the counter resetting to zero (e.g., re-counting after the timestamp reaches its maximum value), failing to accurately reflect the dynamic changes in the network state. Therefore, based on the timestamp sorting, the sorting logic is further corrected by checking the consistency of actions at adjacent time points. The check rule is that if the next action a' at time n-1 is inconsistent with action a at time n, the sorting priority of the data at time n is adjusted to ensure the continuity of the action sequence. For example, if a' of Step_0 is inconsistent with a of Step_1 after the timestamp is reset to zero, the trajectory starting point needs to be repositioned through manual annotation or algorithm verification. The goal of data structure reorganization is to convert the sorted data into a standardized empirical data set {s, a, r, s'} required for training a deep Q-network, where s represents the current state (e.g., queue length and output rate), and a represents the current configured action (kmin, kmax, and pmax). r represents the evaluation score (using the formula R=(W1×TR+W2×DL)×10000). s' represents the next true state value (non-generalized state) after performing action a. Taking Step_0 to Step_2 in Table 1 as an example, they are reorganized into the format shown in Table 2.

[0034] Table 2 Data Acquisition Format after Recombination

[0035] Furthermore, in one embodiment, during model training, the empirical data set {S, A, R, S'} is the core input for model learning. This invention stores the reorganized data into an empirical pool (Replay Buffer) through the following steps. S is a generalized state, defined according to a K-step input strategy (e.g., S = {s0, s...}). 1, ..., s0 For example, when K=2, S=[s1, s0, a1]. A: The ECN configuration action at the current moment (e.g., A=[a1]). R: The reward value obtained after executing action A (e.g., R=[r1]). S': The next real state after executing action A (e.g., S'=[s2]). The reorganized {S, A, R, S'} are stored in the experience pool in sequence, forming the following format:

[0036] Here, Sn represents the generalized state at step n, An represents the corresponding action, Rn represents the reward value, and S'n represents the next true state. The experience pool storing {S, A, R, S'} ensures the integrity of state transitions, avoiding trajectory breaks caused by timestamps returning to zero and guaranteeing trajectory continuity. S is defined by combining K steps of states and actions, supporting flexible adjustment of the K value under different network scenarios (e.g., combining historical states and current actions when K=2), ensuring dynamic K value adaptation. The experience pool reduces the dependence of online training on real-time data through offline storage and random sampling, accelerating model convergence and improving training efficiency. Combining the reward value R with a preset threshold (e.g., good_reward), only high-value experience data is stored to optimize model learning quality.

[0037] Furthermore, in one embodiment, model pre-training with ECN configuration is achieved through a deep Q-network during the offline training phase, which requires addressing the computational complexity problem caused by action space explosion in traditional Q-learning.

[0038] In this embodiment, Deep Q-Network (DQN) is used as a deep reinforcement learning model. It combines neural networks with the traditional Q-Learning algorithm in reinforcement learning, transforming the Q-table of Q-Learning into a Q-Network. The state is used as the neural input, and the Q-value is calculated as the output using the Monte Carlo method (MC / temporal difference method). Then, the function parameters w and weights are trained until the model converges. The Q-value here can be understood as a score for all actions. Action selection is performed by choosing the action with the highest Q-value. The basic principle is as follows: Figure 8 As shown, its training process is as follows: Figure 9 As shown.

[0039] The loss function of the deep Q-network model is calculated using the mean squared error between the true Q-value and the simulated Q-value, as described below: ,in .

[0040] The input to a deep Q-network is S from {S, A, R, S'}, and the output is the Q-value for each action. The action value configuration A consists of three configuration items: [low threshold kmin, high threshold kmax, and maximum label probability pmax]. Here, kmin ∈ {kmin | kmin ≤ 10240, x ∈ N+}, kmax ∈ {kmax | kmax ≤ 10240, x ∈ N+}, and pmax ∈ {pmax | pmax ≤ 100, x ∈ N+}.

[0041] DQN outputs the Q-value of each action through a neural network, but ECN configures action A with three independent parameters (kmin, kmax, pmax), resulting in a total of 10240×10240×100 action combinations. This leads to an output dimension as high as (10240×10240×100,1), consuming huge computational resources and resulting in low training efficiency. Therefore, the discretization of the action space needs to strike a balance between configuration effectiveness and computational complexity to avoid affecting the model's practicality due to the explosion of action combinations. The optimization strategy is to adopt a discretization rule of dense for small values ​​and sparse for large values, segmenting kmin, kmax, and pmax: kmin: small value intervals (e.g., 1-100) with small step size (e.g., step size 1), and large value intervals (e.g., 100-10240) with large step size (e.g., step size 100), reducing invalid low-threshold combinations. kmax: Similar to kmin, ensuring high threshold coverage of critical congestion trigger points. pmax: Proportional discretization (e.g., 0‰, 10‰, 50‰, 100‰) avoids overly fine-grained probability classification. Reduces the total number of action combinations from the full 10240×10240×100 to within 980 after discretization (e.g., ...). Figure 6 As shown in the figure, while maintaining the effectiveness of ECN configuration, the model complexity is significantly reduced.

[0042] Furthermore, in one embodiment, DQN needs to output an independent Q-value for each action combination, but the three-dimensional action parameters (kmin, kmax, pmax) cannot be directly mapped to the Q-value. The solution is to reduce the three-dimensional actions to one-dimensional index values ​​by fixing the action space order, thus achieving a one-to-one mapping between Q-values ​​and actions. Index value generation rule: Assign unique index values ​​according to the order of the discretized action combinations (e.g., ... Figure 7 As shown), for example, the action combination (1,128,1) is mapped to index 0, (2,128,1) is mapped to index 1, and so on. Action values ​​of size 3 are converted to index values ​​of size 1, thus establishing a correspondence between data of different dimensions, achieving one action element corresponding to one Q value. Neural network output: DQN takes the index value as input and outputs the Q value of the corresponding action (e.g., Output0 corresponds to the Q value Q(0) of index 0), avoiding redundant calculations. Through manipulation, the Q value calculation logic is maliciously simplified, reducing the model dimension and improving training efficiency.

[0043] Furthermore, in one embodiment, in network scenarios without definite trajectories, a virtual trajectory is defined using the length of the discretized action space. The rationale is that if the discretized action space length is 980, then 980 training iterations constitute one episode, ensuring that every action combination is explored. Through the design of the virtual trajectory length, it is probabilistically guaranteed that each action value is trained at least once in a single episode, avoiding the omission of critical configurations.

[0044] In this embodiment, in reinforcement learning, an episode typically constitutes a complete trajectory, with a clearly defined start and end. However, this scenario lacks the concept of a trajectory; therefore, the length of the discretized action space is proposed as the length of a "virtual trajectory." For example, if the length of the discretized action space is 980, 980 training iterations would constitute one episode. The reason for using the length of the discretized action space as the length of the virtual trajectory is that, probabilistically, with such an episode's training length, on average, each action value can be trained and explored.

[0045] Furthermore, in one embodiment, during the offline training phase, when the model loss function J(θ) approaches a stable value (e.g. Figure 10 The convergence curve shown indicates that the model has converged. At this point, the trained model parameters are stored as a pre-trained model file for initialization when the switch is powered on for the first time, thus shortening the online training time.

[0046] Due to the complexity of data center network environments and data traffic (such as sudden congestion and topology changes), a single pre-trained model is insufficient to adapt to all scenarios, requiring continuous supplementation of new knowledge through real-time online training. This invention proposes a layered online training architecture combining online training for switches and online training for servers.

[0047] During online training of switches, some switches cannot run AI programs independently due to performance limitations and need to rely on servers for online training. The AI ​​program (ECN_RL agent) and the switch's ECN driver configuration module exchange data and control information through the Socket communication protocol to achieve dynamic configuration.

[0048] When training online, the server provides stronger computing power to handle complex training tasks and adapt to high-load scenarios.

[0049] When the AI ​​program runs on a switch or server, cross-platform interaction can be achieved simply by changing the IP address. The communication data includes state S, action A, reward R, and timestamp, supporting real-time training and inference.

[0050] When the management terminal issues the "AI_ECN disable" command, the switch stops sending data, and the AI ​​program enters a pause state after detecting no data. When the management terminal issues the "AI_ECN enable" command, the switch resumes data transmission, and the AI ​​program restarts training and inference.

[0051] By supplementing new scenario data in real time through online training, the model's responsiveness to sudden congestion is improved. The hierarchical training architecture adapts to switches with varying performance levels, reducing deployment costs. The Socket communication mechanism simplifies cross-platform deployment and enhances system flexibility.

[0052] In one specific embodiment, a "data interaction communication protocol" is proposed between AI and the "driver configuration module" in the switch. By configuring different host IPs in the AI ​​program, data is exchanged in binary encoding and decoding mode, realizing online training and inference of a set of AI code adapted to and compatible with switches and servers.

[0053] The data interaction communication protocol defines the total number of queues that the AI ​​program needs to process as the frame header of the interaction data frame to locate the interaction data and the parsing length of the interaction data. Its format is as follows: .

[0054] The AI ​​receives "que_data_n" data from the switch, characterized by the following features: .

[0055] Where port is the port number, queue is the queue number, reward (evaluation score), throughput_rate is the output data rate of the port, len is the current queue length, ECN pps is the output rate of ECN labeled packets, kmin is the low threshold, kmat is the high threshold, pmax is the maximum labeling probability, and timestamp is the timestamp.

[0056] Furthermore, in one embodiment, online training and inference are divided into online training and inference for the switch and online training and inference for the server. To achieve optimal code compatibility, this invention proposes using a socket communication mechanism to decouple the AI ​​code from other programs on the switch. By establishing a data communication protocol between the AI ​​and the driver configuration module in the switch, and by configuring different host IPs in the AI ​​program, data is exchanged using binary encoding and decoding, thus achieving a single set of AI code that is compatible with both the switch and the server for online training and inference. The data communication protocol is defined as follows.

[0057] Because the data processed in a switch changes very frequently due to different networks, application environments, scenarios, and time periods, this invention proposes and designs the first byte of the data protocol message as the data length of that data interaction. This data length refers to the total number of queues that need to be inferred and computed using deep Q-network reinforcement learning, as follows: Figure 11 and Figure 12 The first byte, num, is shown in the figure.

[0058] The protocol format for AI-side programs to receive data is as follows: Figure 11 As shown, the data received by the AI ​​program from the switch consists of five parts: port number and sequence number. The total number of all combinations of port number and queue number is the total number of data queues (num) contained in the data frame. Evaluation score (Reward): The evaluation score (througphput_rate) of the queue's state after the action value (i.e., ECN configuration value) issued by the AI ​​program in the previous round. State value: The characteristic state of the current queue, specifically including {output data rate per port (througphput_rate), current queue length (len), output rate of ECN tag packets per port (ECN pps)}. Action value: The action value (i.e., ECN configuration value) issued by the AI ​​program in the previous round, specifically including {low threshold kmin, high threshold kmat, maximum tag probability pmax}. Timestamp: The timestamp is used to identify the data collection time of this queue. The protocol format for the AI ​​program to send data is as follows: Figure 12 As shown, the data sent by the AI ​​program to the switch consists of three parts: port number and sequence number. The port number and sequence number of the sending end correspond one-to-one with the port number and sequence number of the receiving end. The action value (i.e., the ECN configuration value) is calculated through deep Q-network inference, and its specific values ​​include {low threshold kmin, high threshold kmat, and maximum label probability pmax}. The error code is used by the AI ​​program to notify the "driver configuration module" of information such as inference timeout.

[0059] Furthermore, in one embodiment, to optimize the inference effect during the online inference stage, this invention proposes two inference mechanisms, which run in parallel. These two mechanisms are: an epsilon-greedy strategy and an optimal reward inference mechanism. Both mechanisms process data at the que_data level, meaning that different que_data sets within the same dataset may employ different inference methods and mechanisms.

[0060] The gradual ε-greedy policy inference mechanism explores an action randomly with probability ε, or selects the action with the highest predicted value from the Q-network. The ε probability value is relatively high at program startup and varies according to the proposed gradual function, eventually stabilizing at a stable value where most action values ​​are selected from the deep Q-network. The characteristics of the ε probability value are: epsilon=lambda frame_idx: cfg.epsilon_end + \(cfg.epsilon_start -cfg.epsilon_end) * \math.exp(-1. * frame_idx / cfg.epsilon_decay), Where `frame_idx` is the epsilon decay count, `epsilon_start` is the initial epsilon probability value, `epsilon_end` is the final epsilon probability value, and `epsilon_decay` is the epsilon decay rate. A larger `epsilon_end` value is used to encourage more random exploration during program startup, allowing for rapid learning of good practices within the network. A smaller `epsilon_end` value is used to maintain "occasional" random exploration of new actions or to discover a more optimized effect even after the program has stabilized. In the gradual ε-greedy policy inference mechanism, the optimal reward inference mechanism starts when the ε probability value approaches the set `epsilon_end` value. A good-reward threshold is set in the inference mechanism. When the received state's reward value is higher than the set good-reward threshold, ε-greedy policy inference continues; when it is lower than the set good-reward threshold, the optimal reward inference mechanism is activated. The optimal reward inference mechanism selects the action value with the highest reward value from the previous N rounds as the action value for the current inference state. The purpose of this mechanism is to enable the current deep Q-network model to quickly learn and adjust when calculating the action values ​​of crossovers if it has not seen the network state before.

[0061] Compared to offline training, online training differs in that it incorporates a data filtering mechanism to select high-quality data for the deep Q-network to learn from. Since the deep Q-network has already encountered both good and bad data during offline training, online training only feeds it good data, allowing it to quickly learn from successful experiences. This is achieved by setting a `good_reward` threshold; data exceeding this threshold is added to the replay buffer for further learning by the deep Q-network.

[0062] Based on practical experience, the time required for one online training session is 10 to 30 times that of one online inference session. Therefore, in order to quickly achieve dynamic configuration of the switch's ECN, this invention proposes an N-round inference-1-round training mechanism. The AI ​​agent program performs N rounds of inference and then performs one round of online training. In practice, N=16 is a good choice.

[0063] This invention proposes an automatic online training model storage mechanism. The AI ​​program stores the trained model after N episodes of training, or after M episodes of training. The purpose of this step is to ensure that when the switch loses power and then gains power again, a model with the latest knowledge is available, allowing for excellent inference performance within a short time after power-on.

[0064] Secondly, embodiments of this application also provide a dynamic ECN configuration system for switches based on reinforcement learning. In one embodiment, referring to... Figure 13 , Figure 13 This is a functional module diagram of an embodiment of the reinforcement learning-based dynamic ECN configuration system for switches according to this application. Figure 13 As shown, the reinforcement learning-based dynamic ECN configuration system for switches includes: a model pre-training module 1, which offline trains a reinforcement learning model using historical data to generate a pre-trained model; a model deployment module 2, which deploys the pre-trained model to the switch and collects network status data S of the switch ports online in real time; and a model update module 3, which, based on S, outputs ECN configuration action values ​​A through the reinforcement learning model, where A includes a low threshold kmin, a high threshold kmat, and a maximum labeling probability pmax. The next state S' is collected, and combined with A and S', a reward value R is calculated, where R is the actual calculated network performance feedback value. R and the state transition data {S, A, R, S'} are stored in an experience pool, and the reinforcement learning model is updated using a deep Q-network algorithm.

[0065] The Deep Q-Network algorithm uses a dual inference mechanism to dynamically adjust the ECN configuration. The dual inference mechanism includes selecting A through a gradual ε-greedy strategy, where the ε value decays with the number of training rounds, and when the ε value approaches a stable value, if the current R value is lower than a preset threshold, then A with the highest R value is selected from the historical N rounds, where N is the preset number of training rounds.

[0066] Furthermore, in one embodiment, the system further includes: The experience pool filtering module compares R with a preset good_reward threshold and only stores experience data where R ≥ good_reward in the experience pool.

[0067] The model storage module is used to automatically store the updated final model according to preset conditions, including triggering model saving after completing N rounds of training or M rounds of operation.

[0068] The functional implementation of each module in the reinforcement learning-based dynamic ECN configuration system for switches corresponds to the steps in the embodiment of the reinforcement learning-based dynamic ECN configuration method for switches, and their functions and implementation processes will not be described in detail here.

[0069] Thirdly, embodiments of this application provide a switch, including: a memory for storing reinforcement learning models and network state data; a processor connected to the memory for running reinforcement learning algorithms, executing dynamic ECN configuration, and interacting with external AI programs via a Socket communication protocol; and a driver configuration module for receiving ECN configuration action values ​​and updating the switch's congestion control parameters.

[0070] Furthermore, in one embodiment, when the processor runs the reinforcement learning algorithm, it employs a virtual trajectory design to define training rounds with a discretized action space length, thereby improving model convergence efficiency.

[0071] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for dynamic ECN configuration of a switch based on reinforcement learning, characterized in that, The reinforcement learning-based dynamic ECN configuration method for switches includes: Use historical data to train reinforcement learning models offline and generate pre-trained models; After deploying the pre-trained model to the switch, network status data S of the switch port is collected online in real time. Based on the S, the reinforcement learning model outputs the ECN configuration action value A, which includes the low threshold kmin, the high threshold kmat, and the maximum label probability pmax. Collect the next state S', and combine it with A and S' to calculate the reward value R, where R is the actual calculated network performance feedback value; The R and state transition data {S, A, R, S'} are stored in the experience pool, and the reinforcement learning model is updated using the deep Q-network algorithm. The Deep Q-Network algorithm uses a dual inference mechanism to dynamically adjust the ECN configuration. The dual inference mechanism includes selecting A through a gradual ε-greedy strategy, where the ε value decays with the number of training rounds, and when the ε value approaches a stable value, if the current R value is lower than a preset threshold, then A with the highest R value is selected from the historical N rounds, where N is the preset number of training rounds.

2. The reinforcement learning-based dynamic ECN configuration method for switches as described in claim 1, characterized in that, The S includes queue length, port output rate, and ECN tag packet output rate; Queue length refers to the number of data packets waiting to be forwarded in the current buffer of a switch port or the amount of memory occupied, reflecting the instantaneous degree of congestion; Port output rate refers to the actual rate or bit rate of data packets forwarded by a switch port per unit time, used to measure bandwidth utilization. ECN tagging packet output rate refers to the number or proportion of packets marked as congested on a switch port per unit time, reflecting the frequency of congestion notification triggering.

3. The reinforcement learning-based dynamic ECN configuration method for switches as described in claim 1, characterized in that, The kmin refers to the lower bound threshold of the queue length. When the queue length is less than kmin, the ECN flag is not triggered to avoid premature congestion notification. The kmax refers to the upper bound threshold of the queue length. When the queue length exceeds kmax, the data packet is marked with the maximum probability pmax, triggering the sending end to slow down. pmax refers to the maximum probability value of marking a data packet when the queue length exceeds kmax, and is used to control the strength of congestion notification.

4. The reinforcement learning-based dynamic ECN configuration method for switches as described in claim 1, characterized in that, The formula for calculating R is: R=(W1×TR+W2×DL)×10000R=(W1×TR+W2×DL)×10000 Where TR is the ratio of port output rate to port bandwidth; DL is the queue length distribution coefficient, which refers to the uniformity of the distribution of switch port queue length in time or space, and its value is less than 1; W1 and W2 are weighting coefficients and W1 + W2 = 1.

5. The method for dynamic ECN configuration of a switch based on reinforcement learning as described in claim 1, characterized in that, When processing A, the deep Q-network algorithm solidifies the multi-dimensional action space composed of kmin, kmat, and pmax into a preset order. It achieves a one-to-one mapping between Q-values ​​and action combinations by using the position index of each action combination in the action space as the mapping value for that action combination. The Q-value is the expected network performance feedback value of the pre-trained model for the selected A under S. The solidification order includes the following steps: Discretize kmin, kmat, and pmax in the action space to form a finite number of action combinations; The action combinations are arranged in order according to a preset rule, and each action combination corresponds to a unique index value. The preset rule is an arithmetic or geometric discretization with denser small values ​​and sparser large values. Deep Q-networks take index values ​​as input and output the Q-value corresponding to the index, realizing a one-to-one mapping between Q-values ​​and action combinations.

6. The method for dynamic ECN configuration of a switch based on reinforcement learning as described in claim 1, characterized in that, The method further includes: The updated final model is automatically stored according to preset conditions, including model saving triggered after completing N rounds of training or M rounds of execution.

7. A dynamic ECN configuration system for switches based on reinforcement learning, characterized in that, The system includes: The model pre-training module uses historical data to train reinforcement learning models offline and generate pre-trained models. The model deployment module is used to deploy the pre-trained model to the switch and collect the network status data S of the switch port online in real time. The model update module is used to output an ECN configuration action value A based on the S through the reinforcement learning model, where A includes a low threshold kmin, a high threshold kmat, and a maximum label probability pmax; collect the next state S', combine A and S' to calculate a reward value R, where R is the actual calculated network performance feedback value; store R and the state transition data {S, A, R, S'} into an experience pool, and update the reinforcement learning model through a deep Q-network algorithm; The Deep Q-Network algorithm uses a dual inference mechanism to dynamically adjust the ECN configuration. The dual inference mechanism includes selecting A through a gradual ε-greedy strategy, where the ε value decays with the number of training rounds, and when the ε value approaches a stable value, if the current R value is lower than a preset threshold, then A with the highest R value is selected from the historical N rounds, where N is the preset number of training rounds.

8. The reinforcement learning-based dynamic ECN configuration system for switches as described in claim 7, characterized in that, The system also includes: The experience pool filtering module compares R with a preset good_reward threshold and only stores experience data where R ≥ good_reward in the experience pool. The model storage module is used to automatically store the updated final model according to preset conditions, including triggering model saving after completing N rounds of training or M rounds of operation.

9. A switch, characterized in that, include: Memory is used to store reinforcement learning model and network state data; The processor, connected to the memory, is used to run reinforcement learning algorithms, execute dynamic ECN configuration, and interact with external AI programs via the Socket communication protocol. The driver configuration module is used to receive ECN configuration action values ​​and update the switch's congestion control parameters.

10. The switch as described in claim 9, characterized in that, When running reinforcement learning algorithms, the processor employs a virtual trajectory design, defining training rounds with the length of the discretized action space to improve model convergence efficiency.

Images