Congestion control parameter adaptive adjustment method based on collaborative machine learning

Abstract

The invention belongs to the technical field of data center network congestion, and discloses a congestion control parameter adaptive adjustment method based on collaborative machine learning. According to the invention, training of a machine learning performance prediction model and a reinforcement learning agent is carried out through historical traffic data, prediction of a port-level global ECN threshold matrix is carried out through the machine learning performance prediction model of an integrated controller, and reasoning of the reinforcement learning agent of a switch port is carried out to obtain a local ECN threshold matrix. A parameter decision module of a switch port performs weighted summation on the global ECN threshold matrix and the local ECN threshold matrix to obtain an optimal ECN threshold and configures the optimal ECN threshold, so that the method can adapt to flow characteristics changing along with time and user requirements, and provides optimal transmission performance for data center network services; the method can effectively deal with the heterogeneity of the data center cluster network traffic of a multi-tenant architecture, and reduces the damage of traffic burst to the network performance.

Description

Technical Field

[0001] This invention relates to the field of data center network congestion control technology, and in particular to an adaptive adjustment method for congestion control parameters based on cooperative machine learning. Background Technology

[0002] Data center networks support various applications such as distributed storage, big data analytics, and distributed machine learning training and inference. The high throughput and low latency of these applications directly impact Quality of Service (QoS). Efficient traffic optimization techniques, such as congestion control, are crucial for improving network transmission performance. However, with the widespread deployment of virtualization, multi-tenant architectures, and hybrid loads in data centers, the traffic characteristics in real-world data center network environments are becoming increasingly complex. This poses a greater challenge to accurate and rapid congestion control mechanisms: on the one hand, the business and global traffic characteristics of data centers change over time and with user demands; on the other hand, the hybrid deployment of services leads to heterogeneity in network traffic characteristics between different clusters within the same data center. Explicit Congestion Notification (ECN), as the mainstream congestion control technology in current data center networks, is widely deployed in commercial switches. ECN-based congestion control algorithms, such as DCTCP and DCQCN, generally have a large number of adjustable parameters. Among these, the ECN threshold is a key parameter that directly determines the effectiveness of congestion control.

[0003] Common ECN threshold setting strategies mainly fall into three categories: static methods, dynamic methods, and adaptive methods. However, when facing highly dynamic network environments, static threshold settings often require conservative settings to cope with different tenant traffic categories, which to some extent sacrifices network performance; dynamic threshold settings, although adjustable, rely on manually set rules and cannot adapt to dynamic changes in traffic in real time; adaptive threshold settings have been extensively studied, and their performance is superior to static and dynamic methods.

[0004] Current mainstream congestion control parameter adaptive adjustment mechanisms can be broadly categorized into two types: switch-based methods and centralized controller-based methods. In switch-based methods, reinforcement learning, due to its unique ability to interact with the environment, has proven to be an effective threshold tuning scheme. Existing work utilizes different reinforcement learning algorithms, such as DDQN and PPO, for rapid congestion state detection and parameter adjustment. However, congestion control is an end-to-end process involving the state of all ports on the entire link, while switches can only observe changes in their local ports and can only collect and process information within a short period. This makes switch-based methods inherently unstable in their decision-making, as decisions based on local information may only benefit the switch locally and fail to achieve overall performance optimization across multiple switches. Furthermore, switches, whether in the data plane or control plane, have limited computational resources, making it difficult to implement more complex and fine-grained mechanisms, which limits the overall accuracy of parameter adjustment. In centralized controller-based methods, given their smaller hardware resource constraints and the advantage of collecting network traffic information through the control plane, previous studies have utilized more complex algorithms such as relational graph convolutional networks for network modeling and simulated annealing search algorithms for adjusting congestion control parameters. However, centralized controller-based methods also have potential drawbacks. The communication latency between the centralized controller and the network device control plane is typically in the millisecond range, and the runtime of the model on the centralized controller is also significant. This makes it difficult for centralized controller-based methods to effectively cope with rapidly changing network environments. Summary of the Invention

[0005] To address the dynamic changes in data center network traffic over time and the heterogeneous characteristics in space, this invention provides an adaptive adjustment method for congestion control parameters based on collaborative machine learning. This method aims to coordinate the optimization of local networks by distributed switches and the prediction of overall network performance by a centralized controller. It fully considers the overall network status and information independently monitored by each switch to adaptively adjust congestion control parameters, effectively mitigating the performance losses caused by either a single centralized or distributed approach.

[0006] To achieve the above objectives, this invention provides an adaptive adjustment method for congestion control parameters based on cooperative machine learning, comprising the following steps: Step 1: Train the machine learning performance prediction model and the reinforcement learning agent using historical traffic data; Step 2: During the online inference phase, the traffic size distribution and topology link information collected by each port of the switch are periodically reported to the global network collector module of the central controller. The machine learning performance prediction model then predicts the global ECN threshold matrix at the port level. Step 3: Collect queue length, output rate, ECN marking rate, and current ECN threshold from each port of the switch as state information of the reinforcement learning agent. The reinforcement learning agent then infers the local ECN threshold matrix. Step 4: The parameter decision module of the switch port selects the optimal ECN threshold after weighted summation of the global ECN threshold matrix and the local ECN threshold matrix, and then configures it.

[0007] Furthermore, in step 1, the training of the machine learning performance prediction model includes the following steps: Step 1.1: Constructing the training dataset: The input to the machine learning performance prediction model is the traffic size distribution, network topology, traffic workload, congestion control parameters, and flow completion time. The flow completion time is obtained through simulation using the network simulator NS3. Step 1.2: Min-max simulation: Decompose the network topology into several independent paths. For each path, run a fast flow-level simulation based on the max-min fair allocation algorithm to simulate a coarse-grained feature map of the completion time of the foreground traffic and the completion time of the background traffic on the path. The foreground traffic represents the flow that has completely passed through the path, and the background traffic represents the flow that has partially passed through the path. Step 1.3: The machine learning performance prediction model further trains a Transformer-based sequence model, corrects the interaction between background flow completion time and foreground flow completion time through an attention mechanism, and combines different ECN threshold parameter combinations as input to the MLP-based prediction model, finally outputting a prediction of the flow completion time distribution on the path.

[0008] Furthermore, step 2 involves the prediction of the global ECN threshold matrix at the port level using a machine learning performance prediction model, including the following steps: Step 2.1: The traffic volume distribution and topology link information of the switch ports are integrated by the network monitor and then synchronized to the global network collector of the centralized controller through the control plane of the data center network. Step 2.2: Perform path weighted sampling according to the number of flows in different paths in the network, decompose the network into independent paths, and run Min-max simulations in parallel for each path; Step 2.3: Iterate through the ECN threshold parameters and use the performance model to make predictions to obtain the performance index matrix of FCT slowdown under different ECN threshold combinations; Step 2.4: Each switch port performs a weighted summation of the matrices of the different paths it has traversed to obtain a prediction of the global ECN threshold matrix at the port level.

[0009] Furthermore, in step 1, the training of the reinforcement learning agent includes the following steps: Step 1.4: Associate each switch port with an independent reinforcement learning agent and model it as a deep reinforcement learning problem using Markov decision processes; Step 1.5: For each reinforcement learning agent, based on the network state information obtained from the port, combined with the reward function and action space, the deep double Q network algorithm DDQN of deep reinforcement learning is used for training, thereby obtaining the local ECN threshold matrix representing the value of action selection based on the transient network state decision.

[0010] Furthermore, in step 3, the reinforcement learning agent infers the local ECN threshold matrix, including the following steps: Step 3.1: For each reinforcement learning agent, adopt a hybrid offline and online training mode and set up a global experience replay pool.

[0011] Furthermore, the Markov decision process in step 1.4 consists of a quintuple. Specifically, it means: The state space is represented by basic parameters that characterize network performance and congestion control features, including the current queue length. Output data rate of each link ECN tag packet output rate for each link and the current ECN threshold ,Right now ; To evaluate changes in queue length and throughput over consecutive time slices, the queue state over the past k monitoring time slices is used as the state information for each tuning inference. ,Right now ; This represents the action space, which includes the low tagging threshold set by the ECN parameters. High labeling threshold The action space uses discrete templates and is matched with the centralized ECN threshold dimension, with time intervals set. Used to collect status information and make and execute ECN configuration decisions; The reward function is represented by link utilization and queue length, which are both representative of network performance and can be obtained from switch ports. ,in Indicates link utilization. Indicates the output rate of the link. Indicates the total bandwidth of the link; The delay is represented by the reciprocal of the current queue length. , As weight, satisfying ; In the state Execute action After transitioning to state The probability, which is obtained by training the deep double Q network algorithm DDQN of deep reinforcement learning; This is a discount factor that controls preferences for immediate and future rewards.

[0012] Furthermore, in step 4, the optimal ECN threshold is selected and configured after weighted summation of the global ECN threshold matrix and the local ECN threshold matrix, including the following steps: Step 4.1: Each switch port collects status information and inputs it into the reinforcement learning agent to obtain the local ECN threshold matrix representing the action selection. At the same time, it combines the global ECN threshold matrix of the central controller to obtain the Q value corresponding to the final action execution. Step 4.2: Select the optimal ECN threshold obtained by weighted summation of the global ECN threshold matrix and the local ECN threshold matrix, and configure it into the network, denoted as: ,in, This indicates the Q value selected after fusion. As a weighting factor, This represents the normalized local ECN threshold matrix obtained by the local agent during inference. This represents the global ECN threshold matrix after normalization of switch ports. and Ensure that the dimensions are the same.

[0013] Beneficial effects: This invention provides an adaptive adjustment method for congestion control parameters based on cooperative machine learning, which has the following technical effects: (1) By running a reinforcement learning algorithm on each switch port, it realizes the collection of state information such as queue length, output rate, ECN marking rate, and current ECN threshold within a certain time interval (nanosecond level), and outputs an ECN threshold suitable for the current port network condition through the DNN model obtained by offline training; (2) The centralized controller collects the traffic size distribution of each port within a certain time interval (second level), decomposes the network into independent paths, and uses min-max The fair algorithm and machine learning algorithm predict the network performance indicators of different switch ports under different ECN threshold configurations, and obtain the global ECN threshold matrix to improve the shortcomings of local network adjustment in the case of heterogeneous network traffic space; (3) After weighted summation of the global ECN threshold matrix and the local ECN threshold matrix, the ECN threshold parameters are selected as the final network configuration parameters and sent to the data plane for execution, which can take into account the network traffic characteristics under local transient and global steady state; (4) It can adapt to the traffic characteristics that change with time and user needs, and provide the best transmission performance for data center network services; It can effectively cope with the heterogeneity of network traffic in multi-tenant architecture data center clusters and reduce the damage of traffic bursts to network performance. Attached Figure Description

[0015] Figure 1 This is an overall structural diagram of the congestion control parameter adaptive adjustment method based on cooperative machine learning involved in the embodiments of the present invention. Detailed Implementation

[0016] like Figure 1 As shown, this invention provides an adaptive adjustment method for congestion control parameters based on cooperative machine learning.

[0017] Example 1: This invention proposes a congestion control parameter adaptive adjustment method (CoPTER) based on cooperative machine learning. It utilizes the advantages of fast response of distributed switches and global information observation of centralized controllers to obtain optimal congestion control parameters suitable for network environments with dynamic changes in time and space.

[0018] The overall framework of the Cooperative Machine Learning-Based Adaptive Adjustment Method for Congestion Control Parameters (CoPTER) is as follows: Figure 1 As shown, it mainly includes two core mechanisms: a congestion control parameter prediction mechanism based on machine learning (also known as a "machine learning performance prediction model") and a congestion control parameter tuning mechanism based on distributed reinforcement learning (also known as a "reinforcement learning agent"). It is worth noting that the architecture described in this invention is applicable to congestion control parameters of various algorithms; this embodiment uses the ECN threshold as a key parameter as an example.

[0019] First, the performance model and reinforcement learning agent are pre-trained using historical traffic data. During the online inference phase, each port of the switch collects traffic size distribution and topology link information, which is periodically reported to the network collector module of the central controller. Then, the performance prediction model predicts the global ECN threshold matrix. At the same time, the switch ports collect queue length, output rate, ECN marking rate, and current ECN threshold as state information of the reinforcement learning agent, and infer the local ECN threshold matrix (also known as the "Q-value matrix"). The parameter decision module of the switch port performs a weighted summation of the global ECN threshold matrix and the local ECN threshold matrix to obtain the optimal ECN threshold and configures it.

[0020] I. Machine Learning-Based Congestion Control Parameter Prediction Mechanism (Machine Learning Performance Prediction Model) After collecting global network information, path-level decomposition is performed. For each decomposed path, a performance model is used to predict the impact of ECN thresholds on network performance at the port level, resulting in a global ECN threshold prediction matrix for each port. Detailed implementation steps are as follows: Step 1: Training the Machine Learning Performance Prediction Model The machine learning performance prediction model consists of three parts: a Min-max simulation model, a Transformer-based sequence model, and an MLP-based prediction model.

[0021] Constructing the training dataset: The model input consists of traffic size distribution, network topology, traffic workload, congestion control parameters, and flow complete time (FCT). The training set parameters are shown in Table 1 below. The flow complete time is obtained using the network simulator NS3. .

[0022] (2) Min-max simulation: For each path, run a fast flow-level simulation based on the max-min fair allocation algorithm to simulate the foreground flow (flow that has completely passed through the path) and background flow (flow that has partially passed through the path) on the path, thereby generating a coarse-grained feature map of the flow completion time.

[0023] (3) The Min-max simulation does not consider the dynamic characteristics of transmission, so there is a large error. The performance model then trains a sequence model based on Transformer, corrects the interaction between background flow and front-end flow through attention mechanism, and combines different ECN threshold parameter combinations as input to the MLP-based prediction model, and finally outputs the prediction of the flow completion time distribution on the path.

[0024] Step 2: Generate a port-level global ECN threshold matrix The traffic distribution and topology information of the switch ports are integrated by the network monitor and synchronized to the global network collector of the centralized controller through the control plane of the data center network, thereby enabling the controller to perceive the overall network traffic information. Path-weighted sampling is performed according to the flow quantity of different paths in the network, decomposing the network into independent paths (e.g., 500 paths). Min-max simulations are run in parallel on each path, traversing the ECN threshold parameters and using a machine learning performance prediction model to make predictions, obtaining the performance index matrix of FCT slowdown under different ECN threshold combinations. For example, under the parameter combination Kmin={20KB, 30KB, 40KB, 50KB}, Kmax={50KB, 60KB, 70KB, 80KB, 90KB, 100KB}, the global ECN threshold matrix is ​​obtained as follows: ; Each switch port performs a weighted summation of the matrices of the different paths it has traversed to obtain the globally predicted ECN threshold matrix (i.e., the "global ECN threshold matrix") for that port.

[0025] II. Congestion Control Parameter Tuning Mechanism Based on Distributed Reinforcement Learning (Also known as "Reinforcement Learning Agent") Step 1: Associate each switch port with an independent reinforcement learning agent and model it as a deep reinforcement learning problem using Markov decision processes; Step 2: For each reinforcement learning agent, based on the network state information obtained from the switch port, combined with the reward function and action space, the Deep Double Q Network (DDQN) algorithm of deep reinforcement learning is used for training, thereby obtaining the local ECN threshold matrix representing the value of action selection based on the transient network state decision. Step 3: For each reinforcement learning agent, a hybrid offline and online training mode is adopted, and a global experience replay pool is set up to accelerate the training process; Step 4: The local ECN threshold matrix and the global ECN threshold matrix are weighted and summed to obtain the ECN threshold matrix, and the ECN threshold corresponding to the maximum value of the matrix is ​​selected and configured to the switch port.

[0026] Step 1 refers to associating each switch port with an independent reinforcement learning agent, modeling it as a deep reinforcement learning problem using a Markov decision process. Specifically, the Markov decision process consists of a quintuple. express; The state space is represented by basic parameters that characterize network performance and congestion control features, including the current queue length ( ), output data rate of each link ( ), ECN tag packet output rate for each link ( ) and the current ECN threshold ( ),Right now To help the agent generalize to different network environments, the collected statistical data is normalized. Furthermore, to evaluate changes in queue length and throughput over consecutive time slices, the queue states of the past k monitoring time slices are used as the state information for each tuning inference step. ,Right now ; This represents the action space, which includes the low tagging threshold set by the ECN parameters. High labeling threshold To reduce the action space and integrate it with centralized guidance information, the action space uses a discrete template and is matched with the centralized ECN threshold dimension. Time intervals are set. Used to collect status information and make and execute ECN configuration decisions; The reward function measures the effectiveness of an action, and the cumulative change in reward value is used to optimize the agent's learning strategy. Link utilization and queue length, which represent both network performance and can be obtained from switch ports, are chosen to characterize the reward function. ,in Indicates link utilization. Indicates the output rate of the link. Indicates the total bandwidth of the link; The latency is represented by the reciprocal of the current queue length. The reward function essentially balances high link utilization and low queue accumulation for each switch. , For weights ( Operation and maintenance personnel can configure it according to the needs of the running application; In the state Execute action After transitioning to state The probability, which is obtained by training a reinforcement learning algorithm; This is a discount factor that controls preferences for immediate and future rewards.

[0027] Step 2 refers to using a deep double-Q network to obtain a local ECN threshold matrix (also known as a "Q-value matrix") representing the value of action selection based on transient network state decisions. The local ECN threshold refers to the low-label threshold. High labeling threshold The selection of different ECN threshold combinations (i.e., different low-labeling thresholds) High labeling threshold The combination of actions will correspond to different Q values, which form the Q-value matrix, also known as the local ECN threshold matrix. DDQN is a value-based deep reinforcement learning algorithm, improved to address the overestimation problem of the target Q-value in Deep Q-Networks (DQNs). It selects actions based on the cumulative discount reward, i.e., the Q-value. Specifically, the DDQN algorithm is deployed and learned independently on each switch port. DDQN consists of two deep neural networks (DNNs): an evaluation network and a target network. The evaluation network determines the action, and the target network determines the value of the action. The target network is updated by copying from the evaluation network every n steps. Target value The calculation method is defined as follows: ,in, This represents the reward value at time t. Indicates the discount factor. This represents the state information at time t+1. This indicates the evaluation of network parameters. Let Q represent the target network parameters, Q() represent the action value function (Q-value), and argmax represent the operation where the parameters reach their maximum value. Formally, argmax... x f(x) represents "finding the value of the independent variable x that makes the function f(x) reach its maximum value", which in the formula represents the value of the function when Q is maximized. , Indicates an action.

[0028] The evaluation network is updated using the following loss function: Where N represents the sample size. Indicates the target value. This represents the state information at time t. This represents the action at time t. This indicates the evaluation of network parameters.

[0029] Step 3 refers to the use of a hybrid offline and online training mode for each agent, along with the establishment of a global experience replay pool to accelerate the training process. Specifically, the model is pre-trained based on historical traffic data. To ensure the generalization of offline training, the traffic data includes features such as various traffic patterns, message sizes, and continuously changing traffic load. After offline training, the same pre-trained model is deployed on all switch ports. During online training, each port optimizes the model using real traffic, and the probability of exploring actions decays exponentially, leading to rapid model convergence. To enhance the model's stability and generalization, a global experience replay pool is established, centrally storing the agent's experience history, specifically in the form of four-tuple information. ,in, This represents the state information at time t. This represents the action at time t. This represents the reward value at time t. This represents the state information at time t+1, and the switch port agents share experience at regular time intervals.

[0030] Step 4 involves obtaining the ECN threshold matrix by weighted summation of the local and global ECN threshold matrices, and then configuring the ECN threshold corresponding to the maximum value of the matrix to the switch port. Specifically, each port collects state information and inputs it into the reinforcement learning agent to obtain the local ECN threshold matrix representing action selection. This matrix is ​​then combined with the ECN prediction matrix from the central controller to obtain the Q-value corresponding to the final action execution, and the corresponding ECN threshold is selected and configured to the network. The specific combination method is as follows: ,in This indicates the Q value selected after fusion. As a weighting factor, This represents the normalized local ECN threshold matrix obtained by the local agent during inference. This represents the normalized global ECN threshold matrix for this port. Clearly, here... and Ensure that the dimensions are the same.

[0031] This invention provides a method for adaptive adjustment of congestion control parameters based on cooperative machine learning. It can coordinate the local network optimization by distributed switches and the overall network performance prediction by centralized controllers. It fully considers the overall network status and the information monitored independently by each switch to adaptively adjust the congestion control parameters, effectively improving the network performance loss caused by separate centralized and distributed approaches.

[0032] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. However, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for adaptive adjustment of congestion control parameters based on cooperative machine learning, characterized in that, The method comprises the following steps: Step 1: training of a machine learning performance prediction model and a reinforcement learning agent through historical traffic data; Step 2: in the online inference stage, the switch ports collect traffic size distribution and topology link information and periodically report to the global network collector module of the centralized controller, and the machine learning performance prediction model performs prediction of the global ECN threshold matrix at the port level; Step 3: the switch ports collect queue length, output rate, ECN marking rate and current ECN threshold as state information of the reinforcement learning agent, and the reinforcement learning agent infers to obtain a local ECN threshold matrix; Step 4: the parameter decision module of the switch port performs weighted summation on the global ECN threshold matrix and the local ECN threshold matrix, selects the optimal ECN threshold and configures the same. 2.The method of claim 1, wherein, In step 1, the training of the machine learning performance prediction model comprises the following steps: Step 1.1: constructing a training data set: the input of the machine learning performance prediction model is traffic size distribution, network topology, traffic workload, congestion control parameters and flow completion time, and the flow completion time is obtained through network simulator NS3 simulation; Step 1.2: Min-max simulation: the network topology structure is decomposed into a plurality of independent paths, and a fast flow level simulation based on the maximum minimum fair allocation algorithm is run for each path to simulate a coarse-grained feature map of foreground traffic flow completion time and background traffic flow completion time on the path, wherein the foreground traffic represents a flow that completely passes through the path, and the background traffic represents a flow that partially passes through the path; Step 1.3: the machine learning performance prediction model further trains a sequence model based on a Transformer, corrects the interaction between the background traffic flow completion time and the foreground traffic flow completion time through an attention mechanism, and combines different ECN threshold parameter combinations as the input of the MLP prediction model to finally output the prediction of the flow completion time distribution on the path.

3. The method of claim 2, wherein, In step 2, the prediction of the global ECN threshold matrix at the port level by the machine learning performance prediction model comprises the following steps: Step 2.1: the traffic size distribution and topology link information of the switch port are integrated through the network monitor and then synchronized to the global network collector of the centralized controller through the control plane of the data center network; Step 2.2: the network is decomposed into independent paths according to the number of flows in different paths in the network, and the Min-max simulation is run in parallel for each path; Step 2.3: ECN threshold parameters are traversed and the performance model is used for prediction to obtain a performance index matrix of FCTslowdown under different ECN threshold combinations; Step 2.4: each switch port performs weighted summation on the matrices of different paths to obtain the prediction of the global ECN threshold matrix at the port level.

4. The method of claim 1, wherein, In step 1, the training of the reinforcement learning agent comprises the following steps: Step 1.4: each switch port is associated with an independent reinforcement learning agent, and a Markov decision process is modeled as a deep reinforcement learning problem; Step 1.5: For each reinforcement learning agent, according to the network state information obtained by the port, combined with the reward function and the action space, the deep double Q network algorithm DDQN of deep reinforcement learning is used for training, so as to obtain the local ECN threshold matrix representing the action selection value according to the transient network state decision.

5. The method of claim 4, wherein, In step 3, the reinforcement learning agent infers the local ECN threshold matrix, including the following steps: Step 3.1: For each reinforcement learning agent, an offline and online hybrid training mode is adopted, and a global experience replay pool is set.

6. The method of claim 4, wherein, The Markov decision process in step 1.4 is represented by a five tuple , in particular: represents a state space, the state space is selected to represent basic parameters characterizing network performance and congestion control features, including current queue length , output data rate of each link , ECN marked packet output rate of each link and current ECN threshold , i.e. ; To evaluate the change of queue length and throughput in consecutive time slices, the queue status of the past k monitoring time slices is taken as the state information of each tuning inference i.e. ; representing an action space, the action space comprising a low mark threshold value for ECN parameter setting , a high mark threshold value , the action space employing a discrete template and matching a centralized ECN threshold dimension, a setting time interval for collecting state information, making and executing ECN configuration decisions; represents a reward function, selecting link utilization and queue length that can represent network performance and be obtained from switch ports to characterize the reward function wherein represents link utilization, represents the output rate of the link, represents the total bandwidth of the link; represents delay, characterized by the inverse of the current queue length, , is a weight, satisfying ; To move to state Perform action To move to state with probability, this parameter is obtained by the deep double Q-network algorithm DDQN of deep reinforcement learning. is a discount factor that controls the preference for immediate versus future rewards.

7. The method of claim 1, wherein, In step 4, the optimal ECN threshold obtained by weighted summation of the global ECN threshold matrix and the local ECN threshold matrix is selected and configured, including the following steps: Step 4.1: Each switch port collects state information and inputs the reinforcement learning agent to obtain the local ECN threshold matrix representing the action selection, and combines the global ECN threshold matrix of the centralized controller to obtain the final action execution corresponding Q value; Step 4.2: and select the optimal ECN threshold configuration selected after the weighted sum of the global ECN threshold matrix and the local ECN threshold matrix to the network, denoted as wherein, denotes the selected Q value after fusion, is a weight factor, denotes the normalized local ECN threshold matrix obtained by reasoning of the local agent, denotes the normalized global ECN threshold matrix of the switch port, and to ensure the same dimension.

Images