A method and apparatus for dynamically setting ECN threshold values

Abstract

This application discloses a method and apparatus for dynamically setting ECN threshold values. The method includes: constructing an experience replay pool based on each ECN threshold value set within a preset time period, along with corresponding network status data and network status evaluation values; performing result-feedback training on an initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges, obtaining the real-time model, and determining the network status and its optimal ECN threshold value. This invention solves the technical problem in the prior art where static ECN threshold values ​​cannot be dynamically adjusted according to the actual network situation, and the congestion status of the data center network cannot be accurately obtained, resulting in poor network congestion control.

Description

Technical Field

[0001] This application relates to the field of data center technology, and in particular to a method and apparatus for dynamically setting ECN threshold values. Background Technology

[0002] Currently, data center network congestion mainly manifests at the egress ports of network devices (such as switches and routers). When the traffic received at the ingress port is much greater than the traffic sent through the egress port, a large number of packets will remain in the corresponding queue at the egress port, causing data center network congestion. This leads to severe degradation of network performance, such as packet transmission latency and throughput, and even packet loss.

[0003] In related technologies, static ECN (Explicit Congestion Notification) configuration is commonly used in general data center scenarios to control network congestion. Specifically, based on the configured static ECN (Explicit Congestion Notification) threshold, it is determined whether the queue corresponding to the outgoing port is congested. If congestion is detected, an ECN identifier is added to the queue corresponding to the outgoing port and sent to the destination device. After receiving the packet with the ECN identifier, the destination device sends a Congestion Notification Packet (CNP) packet to the source device, notifying the source device to reduce the traffic transmission rate, thereby achieving the purpose of controlling network congestion.

[0004] However, network congestion changes with time and load, and static ECN thresholds cannot be dynamically adjusted according to the actual network situation, thus failing to accurately obtain the data center network congestion status, resulting in poor network congestion control. Summary of the Invention

[0005] This application provides a method and apparatus for dynamically setting ECN threshold values, which can solve the technical problem of poor network congestion control in the prior art.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] In a first aspect, embodiments of this application provide a method for dynamically setting an ECN threshold value, the method comprising:

[0008] An experience replay pool is built based on the ECN threshold value set each time within a preset time period, as well as the corresponding network status data and network status evaluation value.

[0009] Based on the data in the experience replay pool, the initial real-time model is trained using a result feedback method until the loss function of the initial real-time model converges, thus obtaining the real-time model and determining the network state and its optimal ECN threshold value.

[0010] Secondly, embodiments of this application provide an apparatus for dynamically setting an ECN threshold value, the apparatus comprising:

[0011] The data processing module is configured to build an experience replay pool based on the ECN threshold value set each time within a preset time period, as well as the corresponding network status data and network status evaluation value.

[0012] The model training module is configured to perform result feedback training on the initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges, thereby obtaining the real-time model and determining the network state and its optimal ECN threshold value.

[0013] Thirdly, embodiments of this application also provide an electronic device, including: a memory and a processor; the processor is configured to read and execute a computer program stored in the memory to implement the steps of the aforementioned method for dynamically setting an ECN threshold value.

[0014] Fourthly, embodiments of this application also provide a computer storage medium storing computer-executable instructions, which, when executed, implement the steps of the aforementioned method for dynamically setting an ECN threshold value.

[0015] Fifthly, embodiments of this application also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned method for dynamically setting an ECN threshold value.

[0016] The beneficial effects of the technical solutions provided in this application include:

[0017] An experience replay pool is constructed based on the ECN threshold value set for each time within a preset time period, along with the corresponding network state data and network state evaluation value. The initial real-time model is then trained using a result feedback method based on the data in the experience replay pool until the loss function of the initial real-time model converges, thus obtaining the real-time model and determining the network state and its optimal ECN threshold value. This invention trains an initial real-time model using feedback methods, based on each set ECN threshold value, the corresponding length of the current switch egress queue, the output data rate of each link, the output rate of ECN tag packets on each link, the currently set ECN threshold value, and the network state assessment value. It learns the relationship between the network state and the set ECN threshold value and dynamically adjusts the mapping between them to construct a mapping relationship between the network state and its optimal ECN threshold configuration. In other words, the ECN threshold value can be dynamically adjusted according to the actual network situation. Even in the event of sudden traffic bursts in the network, the ECN threshold value can be dynamically adjusted to accurately obtain the congestion status of the data center network, thereby reducing the traffic transmission rate and achieving the goal of controlling network congestion. This solves the technical problem in existing technologies where static ECN threshold values ​​cannot be dynamically adjusted according to the actual network situation, resulting in inaccurate acquisition of data center network congestion status and poor network congestion control. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the first embodiment of the method for dynamically setting the ECN threshold value for this application;

[0020] Figure 2 A flowchart illustrating a second embodiment of the method for dynamically setting ECN threshold values ​​for this application;

[0021] Figure 3 A flowchart illustrating the third embodiment of the method for dynamically setting ECN threshold values ​​in this application;

[0022] Figure 4 for Figure 3 A detailed flowchart of step S50;

[0023] Figure 5 A schematic diagram of the functional modules of an embodiment of the apparatus for dynamically setting the ECN threshold value for this application;

[0024] Figure 6This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0027] In a first aspect, embodiments of this application provide a method for dynamically setting ECN threshold values.

[0028] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the method for dynamically setting the ECN threshold value according to this application. Figure 1 As shown, methods for dynamically setting ECN threshold values ​​include:

[0029] Step S10: Construct an experience replay pool based on the ECN threshold value set each time within a preset time period, as well as the corresponding network status data and network status evaluation value.

[0030] In this embodiment, the action space (action_space) is constructed based on the ECN threshold values ​​set at each step within a preset time period T from system startup to time t.

[0031]

[0032] Where, a=(k min ,k max ,p max ), k min Indicates the low labeling threshold, k max p represents the high labeling threshold. max This represents the label probability value. Where k... max >k min And this is the maximum number of congestion notification signals that the sender can receive when there is network congestion; 0 <p max <100. T is the total duration of data collected within the preset duration T from system startup to time t.

[0033] Each time an ECN threshold is set, the network status data and network status assessment value will change accordingly. Therefore, each set ECN threshold value has its corresponding network status data and network status assessment value.

[0034] A data unit (S) consists of the ECN threshold value set at time t, the network state information at time t after setting the ECN threshold value, the network state evaluation value at time t, and the network state information at time t+1. t ,a t ,r t, S t+1 This is called an empirical QS. t The collection of stored experiences is called the experience replay pool, denoted as: replay_pod =

[0035] {(S1,a1,r 1, S2),(S2,a2,r2,S3),…,(S t ,a t ,r t ,S t+1 ),…,(S i ,a i ,r i ,S i+1 |i≥1 and is an integer}.

[0036] Optionally, in one embodiment, the network status data includes: the current length of the switch egress queue, the output data rate of each link, the output rate of ECN tag packets for each link, and the currently set ECN threshold value.

[0037] In this embodiment, the network (queue) status information S can be represented by a quadruple of the current switch egress queue length (qlen), the output data rate of each link (txRate), the output rate of each link ECN tag packet (txRate_mark), and the currently set ECN threshold value (ECN_cur), i.e., S = (qlen, txRate, txRate_mark, ECN_cur).

[0038] During the construction of the action space, each ECN threshold value is set to the network state S collected after the switch, and the accumulated state space is state_space = (S1, S2, S3, ..., S...). v ), where v is a positive integer.

[0039] Optionally, in one embodiment, the network state assessment value is calculated based on the output data rate of each link, the fixed bandwidth of the link, and the length of the switch queue, including:

[0040] The output data rate of each link after each ECN threshold setting is divided by the fixed bandwidth of the link, and the quotient is used as the throughput of the switch queue after each ECN threshold setting.

[0041] The throughput is multiplied by the corresponding weight, and the length of the switch queue after each ECN threshold setting is multiplied by the corresponding weight. The sum is used as the network status evaluation value corresponding to each ECN threshold setting.

[0042] In this embodiment, the output data rate of each link after each ECN threshold setting is divided by the fixed bandwidth of the link, and the quotient is used as the throughput of the switch queue after each ECN threshold setting. Specifically, T i The throughput of the switch queue after setting the ECN threshold value at all times In the formula, T i (R) represents T i The throughput of the switch queue after setting the ECN threshold value, txRate i T represents i The output data rate of each link after setting the ECN threshold value at all times; BW represents the fixed bandwidth of the link.

[0043] Based on T i After setting the ECN threshold value, the throughput of the switch queue and T i The length of the switch queue after setting the ECN threshold value is calculated as T. i The network status evaluation value is set after the ECN threshold is set at all times to avoid instability in the calculated value caused by frequent changes in link bandwidth utilization and instantaneous queue length. Specifically, T i The network status evaluation value r after setting the ECN threshold at any time i =μ1*T i (R)+μ2*L i (Q), where r i T represents i The network status evaluation value after setting the ECN threshold value at any time, where μ1 is the proportional weight of the switch queue throughput and μ2 is the proportional weight of the switch queue length. In the formula, a t′ This represents the ECN threshold value set at time t′. s represents time t t′ power, s t′ This represents the network state evaluation value after setting the ECN threshold at time t′, where t′ can be either time t+1 or t+n. i (Q) represents T i The length of the switch queue after setting the ECN threshold value at any time.

[0044] By following this logic, the network status assessment value corresponding to each set ECN threshold value can be calculated.

[0045] Step S20: Based on the data in the experience replay pool, perform result feedback training on the initial real-time model until the loss function of the initial real-time model converges to obtain the real-time model, and determine the network state and its optimal ECN threshold value.

[0046] In this embodiment, the data in the experience replay pool includes the ECN threshold value set each time and its corresponding current switch egress queue length, the output data rate of each link, the output rate of ECN tag packets of each link, the currently set ECN threshold value, and the network status evaluation value.

[0047] The initial real-time model is trained using feedback-based training on data from the experience replay pool. This means that each time an ECN threshold is set to the switch, network state data (current switch egress queue length, output data rate of each link, output rate of ECN tag packets for each link, and the currently set ECN threshold) and a network state evaluation value are obtained. Based on the ECN threshold set at time t, the network state information at time t after setting the ECN threshold, the network state evaluation value at time t, and the network state information at the next time step (t+1), the initial real-time model is trained using feedback-based training to learn the relationship between the network state and the set ECN threshold, and dynamically adjust the mapping between the two to construct a mapping relationship between the network state and its optimal ECN configuration. Traditional congestion control mechanisms rely on packet loss as a congestion signal, but packet loss can trigger rapid retransmission and recovery operations by the sender, leading to a degraded network performance. By using the method in this embodiment, ECNs can be dynamically marked according to the congestion status of the switch's egress queue, and an ECN threshold value can be set to notify the sender to reduce the packet sending rate in order to reduce packet loss or even avoid packet loss, thereby improving network performance and better adapting to the needs of high-speed networks.

[0048] The training process calculates the loss function of the initial real-time model to determine the degree of convergence of the initial real-time model, which characterizes the training degree of the initial real-time model. If the change of the loss function is less than the set threshold, that is, the loss function converges, the initial real-time model is considered to be usable for inference work, and the training ends at this time, and the real-time model is obtained.

[0049] The loss function is as follows:

[0050]

[0051] In the formula, y j Let r represent the loss function of the initial real-time model. j Indicates t jECN threshold value a at time 1 j The corresponding network state evaluation value, γ∈[0,1], is a discount factor. S represents a deep neural network with parameters θ′. j+1 This represents the network state evaluation value at time t(j+1). This represents mapping the network state evaluation value to the set ECN threshold value, and maximizing the discount to accumulate the network state evaluation value to find the optimal ECN threshold value for the network state evaluation value. 'a' represents the set ECN threshold value, 'θ' represents the parameters of the evaluated network, and 'θ′' represents the parameters of the target network. The target network is obtained by saving the evaluated network every N steps (with periodic updates to the target).

[0052] Optionally, in one embodiment, the step of performing result-feedback training on the initial real-time model based on the data in the experience replay pool includes:

[0053] The ECN threshold value set at time t, the network status information at time t corresponding to the ECN threshold value set at time t, the network status evaluation value at time t, and the network status information at time t+1 are used as a data unit.

[0054] When the number of data units is greater than the preset number, P consecutive data units are randomly selected as training datasets to perform result feedback training on the initial real-time model.

[0055] Among them, the training dataset

[0056] E(p,q)={((S q ,a q ,r q ,S q+1 ),(S q+1 ,a q+1 ,r q+1 ,S q+2 ),…,(S q+p-1 ,a q+p-1 ,r q+p-1 ,S q+p 0)}, q takes values ​​in the range [1, n]}, S q Let a represent the q-th network state information. q r represents the q-th set ECN threshold value. q S represents the evaluation value of the q-th network state. q+1 This represents the (q+1)th network state information, where n is a positive integer.

[0057] In this embodiment, an action cycle one order of magnitude longer than the Round-Trip Time (RTT) is used to adjust the ECN threshold value. This is to prevent the congestion control rate from dynamically adjusting the ECN threshold value at the same frequency, which could cause control interference and thus affect congestion control performance. The ECN threshold value a set at time t in the action_space is... t The network state information St at time t after setting the ECN threshold, and the network state evaluation value r at time t. t And the network state information S at the next time step t+1. t+1 Constitutes a data unit (S) t ,a t ,r t ,S t+1 This is called an empirical QS. t The collection of stored experiences is called the experience replay pool, denoted as: replay_pod = {(S1, a1, r1, S2), (S2, a2, r2, S3), ..., (S... t ,a t ,r t ,S t+1 ), ..., (S i ,a i ,r i ,S i+1 |i>=1 and is an integer}.

[0058] With a time interval of Δt, The iteration count is defined as follows: when the number of data units in the replay_pod exceeds a preset number, P consecutive data units are randomly selected as the training dataset to perform feedback training on the initial real-time model. The training dataset E(p,q) = {((S...} q ,a q ,r q ,S q+1 ),(S q+1 ,a q+1 ,r q+1 ,S q+2 ),…,(S q+p-1 ,a q+p-1 ,r q+p-1 ,S q+p ))}, where q takes values ​​in the range [1, n]. Where S q Let a represent the q-th network state information. q r represents the q-th set ECN threshold value. q S represents the evaluation value of the q-th network state. q+1Let P represent the (q+1)th network state information, where n is a positive integer and P is less than or equal to a preset number. It's easy to see that the data in the training dataset will differ depending on when the data is acquired.

[0059] In this embodiment, an experience replay pool is constructed based on the ECN threshold value set each time within a preset time period, as well as the corresponding network state data and network state evaluation value. The initial real-time model is trained using a result feedback method based on the data in the experience replay pool until the loss function of the initial real-time model converges, thereby obtaining the real-time model and determining the network state and its optimal ECN threshold value. In this embodiment, the initial real-time model is trained using a feedback method based on the ECN threshold value set each time, the corresponding length of the current switch egress queue, the output data rate of each link, the output rate of ECN tag packets of each link, the currently set ECN threshold value, and the network state evaluation value. This learns the relationship between the network state and the set ECN threshold value, and dynamically adjusts the mapping between the two to construct a mapping relationship between the network state and its optimal ECN threshold configuration. In other words, the ECN threshold value can be dynamically adjusted according to the actual network situation. Even in the event of sudden traffic bursts in the network, the ECN threshold value can be dynamically adjusted to accurately obtain the congestion status of the data center network, thereby reducing the traffic transmission rate and achieving the goal of controlling network congestion. This solves the technical problem in the prior art where static ECN threshold values ​​cannot be dynamically adjusted according to the actual network situation, making it impossible to accurately obtain the congestion status of the data center network, resulting in poor network congestion control.

[0060] Optionally, in one embodiment, reference is made to Figure 2 , Figure 2 This is a flowchart illustrating a second embodiment of the method for dynamically setting the ECN threshold value in this application. Figure 2 As shown, prior to the step of performing result-feedback training on the initial real-time model based on the data in the experience replay pool, the following steps are included:

[0061] Step S30: When the model meets the pre-training conditions, pre-train the model until the model's loss function converges to obtain the initial real-time model.

[0062] Pre-training conditions include the model not being trained, changes in the network environment, or the model being applied to a new network environment.

[0063] In this embodiment, the model needs to be pre-trained in three situations: when the model has not been trained, when the network environment changes, or when it is applied to a new network environment. This training process is called the initial exploration phase. During the initial exploration phase, after constructing an experience replay pool based on the ECN threshold value set each time within a preset time period, along with the corresponding network state data and network state evaluation values, the model is pre-trained. This is called result-feedback training, which continues until the loss function of the model converges. At this point, the initial exploration phase ends, and the initial real-time model is obtained.

[0064] Optionally, in one embodiment, the method includes:

[0065] The parameters of the initial real-time model are updated once each time the initial real-time model is trained.

[0066] The parameters of the real-time model are updated once after each preset number of training iterations.

[0067] In this embodiment, since the initial real-time model is trained using a feedback method, the parameters of the initial real-time model need to be updated in each iteration. The process of updating the parameters θ of the initial real-time model is as follows:

[0068]

[0069] In the formula, L(θ) represents the target network, N represents the number of iterations, and y j Let Q(S) represent the loss function of the initial real-time model. j ,a j S(θ) represents a deep neural network with parameter θ. j Indicates t j The network state evaluation value at time a j Indicates t j The ECN threshold value set at each time step, where θ represents the parameters of the network being evaluated.

[0070] In three scenarios—no model training, a changed network environment, or application to a new network environment—after pre-training the model to obtain an initial real-time model, the model parameters are updated once every preset number of training iterations. This regular parameter update ensures pre-inference applications in different scenarios. This approach smooths out the challenges to model robustness and adaptability brought about by changes in scenario and network state, guaranteeing network quality and resource utilization.

[0071] The process of updating the model's parameters θ is as follows:

[0072]

[0073] In the formula, Let y represent the gradient, N represent the number of iterations, and y represent the number of iterations.j denotes the loss function of the initial real-time model, Q(S j , a j , θ) denotes a deep neural network with parameter θ, S j denotes t j the network state evaluation value at time, a j denotes t j the ECN threshold value set at time, and θ represents the parameters of the evaluation network.

[0074] Optionally, in one embodiment, refer to Figure 3 , Figure 3 which is a schematic flowchart of the third embodiment of the method for dynamically setting the ECN threshold value in this application. As Figure 3 shown, after the step of performing result feedback training on the initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges to obtain the real-time model, it includes:

[0075] Step S40, obtaining the evaluation value of the network state for continuous M rounds and / or the throughput rate of the switch queue based on the real-time model;

[0076] Step S50, when the evaluation value of the network state for continuous M rounds and / or the throughput rate of the switch queue are both less than their corresponding thresholds, updating the experience replay pool to obtain an updated experience replay pool;

[0077] Step S60, performing result feedback training on the real-time model based on the updated experience replay pool until the loss function of the real-time model converges, and then returning to execute the step of obtaining the evaluation value of the network state for continuous M rounds and / or the throughput rate of the switch queue based on the real-time model.

[0078] In this embodiment, the evaluation value of the network state output by the real-time model for continuous M rounds and the throughput rate of the switch queue are obtained. When the evaluation value of the network state for continuous M rounds is less than its corresponding threshold, and / or the throughput rate of the switch queue for continuous M rounds is less than its corresponding threshold, it is determined at this time that the adaptability of the real-time model is poor, that is, the configured ECN threshold value deduced has a poor effect on network congestion and throughput management. Specifically, the evaluation value r<score of the network state for continuous m rounds and / or the throughput rate T(R)<rate_high of the switch queue, where score is the threshold corresponding to the evaluation value of the network state, and rate_high is the threshold corresponding to the throughput rate of the switch queue, specifically when the bandwidth is close to 100% utilization, such as 0.95.

[0079] At this point, the experience exploration phase begins, updating the experience replay pool to obtain an updated experience replay pool. Then, based on the data in the updated experience replay pool, the real-time model undergoes result-feedback training until the loss function of the real-time model converges. Finally, the process returns to the step of obtaining the evaluation values ​​of the network state for M consecutive rounds and / or the throughput of the switch queues based on the real-time model.

[0080] In another embodiment, when the network state evaluation value and the throughput of the switch queue output by the real-time model for M consecutive rounds are both greater than or equal to their corresponding thresholds, it is determined that the real-time model has good adaptability, that is, the ECN threshold configuration inferred has a good effect on network congestion and throughput management.

[0081] Furthermore, the network status includes normal communication status and congestion status. After obtaining the throughput of the switch queue and the current network status assessment value, based on the current network status assessment value or the switch queue throughput, it is determined whether the queue corresponding to the outgoing port is congested. If congestion occurs, the current ECN threshold value is added to the queue corresponding to the outgoing port and sent to the destination device. After receiving the message with the ECN threshold value identifier, the destination device sends a Congestion Notification Packet (ECN packet) message to the source device, notifying the source device to reduce the traffic transmission rate, thereby achieving the purpose of controlling network congestion.

[0082] Optionally, in one embodiment, reference is made to Figure 4 , Figure 4 for Figure 3 A detailed flowchart of step S50. (See attached diagram.) Figure 4 As shown, the step of updating the experience replay pool to obtain the updated experience replay pool includes:

[0083] Step S501: Compare the network state evaluation value corresponding to setting the ECN threshold value at time t-2 in the experience replay pool with the network state evaluation value corresponding to setting the ECN threshold value at time t-1, where time t-2 is the time before time t-1.

[0084] Step S502: If the network state evaluation value after setting the ECN threshold value at time t-1 is greater than or equal to the network state evaluation value after setting the ECN threshold value at time t-2, then the updated ECN threshold value set at time t is obtained based on the ECN threshold value set at time t-1, where time t-1 is the time before time t.

[0085] Step S503: Using time t-1 as time t-2, return to the step of comparing the network state evaluation value corresponding to the setting of the ECN threshold value at time t-2 in the experience replay pool with the network state evaluation value corresponding to the setting of the ECN threshold value at time t-1.

[0086] Step S504: The network state evaluation value corresponding to the ECN threshold value set at each time moment in the experience replay pool has been compared to obtain the updated experience replay pool.

[0087] In this embodiment, time t-2 is the time preceding time t-1. Taking time t-2 as time t1 and time t-1 as time t2 as an example, the network state evaluation value corresponding to the ECN threshold value at time t1 in the experience replay pool is compared with the network state evaluation value corresponding to the ECN threshold value at time t2, that is, a1=(k min1 k max1 p max1 The network state evaluation values ​​r1 and a2 = (k) are respectively. min2 k max2 p max2 The network state evaluation value r2 is compared with the network state evaluation value r2.

[0088] If the network state evaluation value r2 corresponding to the ECN threshold value at time t2 is greater than or equal to the network state evaluation value r1 corresponding to the ECN threshold value at time t1, that is, when the difference between r2 and r1 is greater than or equal to zero, then the updated ECN threshold value a3′ at time t3 is obtained based on the ECN threshold value at time t2. Here, time t3 is the time after time t2.

[0089] If the network state evaluation value r2 corresponding to the ECN threshold value at time t2 is less than the network state evaluation value r1 corresponding to the ECN threshold value at time t1, that is, when the difference between r2 and r1 is less than zero, then the ECN threshold value at time t3 remains unchanged.

[0090] Using time t2 as time t1, the process returns to the step of comparing the network state evaluation value r1 corresponding to the ECN threshold value at time t1 in the action space with the network state evaluation value r2 corresponding to the ECN threshold value at time t2. It's easy to see that since time t2 is used as time t1, if r2 is greater than or equal to r2, the step of updating the ECN threshold value at time t3 essentially updates the ECN threshold value a4 at time t4, resulting in the updated ECN threshold value a4′ at time t4.

[0091] The updated experience replay pool can be obtained by comparing the network state evaluation values ​​corresponding to the ECN threshold values ​​set at each time point in the experience replay pool.

[0092] Optionally, in one embodiment, the step of obtaining the updated ECN threshold value set at time t based on the ECN threshold value set at time t-1 includes:

[0093] Select any value from the high labeling threshold, low labeling threshold, and labeling probability value included in the ECN threshold set at time t-1 as the first target value;

[0094] Select a second target value that is greater than the first target value from the range of the first target value;

[0095] The first target value is replaced with the minimum value among the second target values, and the ECN threshold value after replacing the first target value is used as the updated ECN threshold value set at time t.

[0096] In this embodiment, time t-1 is the time preceding time t, and the ECN threshold includes the high labeling threshold kma, the low labeling threshold kmin, and the labeling probability value pmax. Taking time t-1 as time t2 and time t as time t3, a2 = (k min2 ,k max2 ,p max2 Taking [10, 128, 50] as an example, from the ECN threshold value a2 = (k) at time t2... min2 ,k max2 ,p max2 Select any value from the list as the first target value.

[0097] If the selected first target value is the low labeling threshold k min2 Then from the low labeling threshold k min Select values ​​greater than k from the range of values. min2 =10 is the second target value. The low-labeling threshold k is used. min Taking the value range of [1, 2, 5, 10, 20, 40, 80, 160, 320, 640, 1280, 2560, 5120, 10240] as an example, the second target value includes 20, 40, 80, 160, 320, 640, 1280, 2560, 5120, and 10240. The minimum value of the second target value, 20, is used to replace the first target value k. min2 The ECN threshold value [20, 128, 50] after replacing the first target value is obtained. [20, 128, 50] is used as the updated ECN threshold value a3′ at time t3, that is, a3′=[20, 128, 50]. Here, time t2 is the time before time t3.

[0098] In another embodiment, if the selected first target value is a high-marking threshold k max2 Then from the high labeling threshold kmax Select values ​​greater than k from the range of values. max2 =128 is the second target value. High-labeling threshold k max Taking the value range of [128, 256, 512, 1024, 2048, 5120, 10240] as an example, the second target value includes 256, 512, 1024, 2048, 5120, and 10240. The first target value k is replaced with the minimum value of the second target value, 256. max2 The ECN threshold value [10, 256, 50] after replacing the first target value is obtained. [10, 256, 50] is used as the updated ECN threshold value a3′ at time t3, that is, a3′=[10, 256, 50].

[0099] In another embodiment, if the selected first target value is the labeling probability threshold p max2 Then from the label probability threshold p max Select values ​​greater than p from the range of possible values. max2 The second target value is 50. The probability threshold p is used as the marker. max Taking the value range of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 25, 50, 75, 100] as an example, the second target value includes 75 and 100. The first target value p is replaced with the minimum value of the second target value, 75. max2 The ECN threshold value [10, 128, 75] after replacing the first target value is obtained. [10, 128, 75] is used as the updated ECN threshold value a3′ at time t3, that is, a3′=[10, 128, 75].

[0100] It should be noted that if there is no second target value greater than the first target value within the range of the first target value, then the first target value remains unchanged.

[0101] Secondly, embodiments of this application also provide a device for dynamically setting ECN threshold values.

[0102] In one embodiment, reference is made to Figure 5 , Figure 5 A schematic diagram of the functional modules of an embodiment of the apparatus for dynamically setting the ECN threshold value according to this application. Figure 5 As shown, the device for dynamically setting the ECN threshold value includes:

[0103] The data processing module 10 is configured to build an experience replay pool based on the ECN threshold value set each time within a preset time period, as well as the corresponding network status data and network status evaluation value.

[0104] The model training module 20 is configured to perform result feedback training on the initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges, thereby obtaining the real-time model and determining the network state and its optimal ECN threshold value.

[0105] Optionally, in one embodiment, the network status data includes: the current length of the switch egress queue, the output data rate of each link, the output rate of ECN tag packets of each link, and the currently set ECN threshold value;

[0106] The network status assessment value is calculated based on the output data rate of each link, the fixed bandwidth of the link, and the length of the switch queue.

[0107] Optionally, in one embodiment, the apparatus for dynamically setting the ECN threshold value further includes a calculation module configured to:

[0108] The output data rate of each link after each ECN threshold setting is divided by the fixed bandwidth of the link, and the quotient is used as the throughput of the switch queue after each ECN threshold setting.

[0109] The throughput is multiplied by the corresponding weight, and the length of the switch queue after each ECN threshold setting is multiplied by the corresponding weight. The sum is used as the network status evaluation value corresponding to each ECN threshold setting.

[0110] Optionally, in one embodiment, the model training module 20 is specifically configured to:

[0111] The ECN threshold value set at time t, the network status information at time t corresponding to the ECN threshold value set at time t, the network status evaluation value at time t, and the network status information at time t+1 are used as a data unit.

[0112] When the number of data units is greater than the preset number, P consecutive data units are randomly selected as training datasets to perform result feedback training on the initial real-time model.

[0113] Among them, the training dataset

[0114] E(p,q)=

[0115] {((S q ,a q ,r q ,S q+1 ),(S q+1 ,a q+1 ,r q+1 ,S q+2 ),…,(S q+p-1 ,a q+p-1 ,rq+p-1 ,S q+p ))}, q takes values ​​in the range [1, n], S q Let a represent the q-th network state information. q r represents the q-th set ECN threshold value. q S represents the evaluation value of the q-th network state. q+1 This represents the (q+1)th network state information, where n is a positive integer.

[0116] Optionally, in one embodiment, the model training module 20 is further configured to:

[0117] When the model meets the pre-training conditions, the model is pre-trained until the model's loss function converges, and the initial real-time model is obtained.

[0118] Pre-training conditions include the model not being trained, changes in the network environment, or the model being applied to a new network environment.

[0119] Optionally, in one embodiment, the model training module 20 is further configured to:

[0120] The parameters of the initial real-time model are updated once each time the initial real-time model is trained.

[0121] The parameters of the real-time model are updated once after each preset number of training iterations.

[0122] Optionally, in one embodiment, the model training module 20 is further configured to:

[0123] Based on the real-time model, obtain the evaluation values ​​of the network status for M consecutive rounds and / or the throughput of the switch queue;

[0124] When the evaluation value of the network state and / or the throughput of the switch queue are both less than their corresponding thresholds for M consecutive rounds, the experience replay pool is updated to obtain the updated experience replay pool.

[0125] The real-time model is trained using a feedback mechanism based on the updated experience replay pool until the loss function of the real-time model converges. Then, the process returns to the step of obtaining the evaluation values ​​of the network state for M consecutive rounds and / or the throughput of the switch queue based on the real-time model.

[0126] Optionally, in one embodiment, the model training module 20 is further configured to:

[0127] The network state evaluation value corresponding to setting the ECN threshold value at time t-2 in the experience replay pool is compared with the network state evaluation value corresponding to setting the ECN threshold value at time t-1, where time t-2 is the time before time t-1.

[0128] If the network state evaluation value after setting the ECN threshold value at time t-1 is greater than or equal to the network state evaluation value after setting the ECN threshold value at time t-2, then the updated ECN threshold value set at time t is obtained based on the ECN threshold value set at time t-1, where time t-1 is the time before time t.

[0129] Using time t-1 as time t-2, return to the step of comparing the network state evaluation value corresponding to the setting of the ECN threshold value at time t-2 in the experience replay pool with the network state evaluation value corresponding to the setting of the ECN threshold value at time t-1.

[0130] The updated experience replay pool is obtained by comparing the network state evaluation values ​​corresponding to the ECN threshold values ​​set at each time point in the experience replay pool.

[0131] Optionally, in one embodiment, the model training module 20 is further configured to:

[0132] Select any value from the high labeling threshold, low labeling threshold, and labeling probability value included in the ECN threshold set at time t-1 as the first target value;

[0133] Select a second target value that is greater than the first target value from the range of the first target value;

[0134] The first target value is replaced with the minimum value among the second target values, and the ECN threshold value after replacing the first target value is used as the updated ECN threshold value set at time t.

[0135] The functions of each module in the above-mentioned device for dynamically setting the ECN threshold value correspond to the steps in the above-mentioned method embodiment for dynamically setting the ECN threshold value, and their functions and implementation processes will not be described in detail here.

[0136] Thirdly, embodiments of this application also provide an electronic device, the structure of which is as follows: Figure 6 As shown, it includes: a memory and a processor, wherein the processor is used to read and execute a computer program stored in the memory to implement the aforementioned method for dynamically setting an ECN threshold value.

[0137] Fourthly, embodiments of this application also provide a computer storage medium storing computer-executable instructions, which, when executed, implement the aforementioned method for dynamically setting an ECN threshold value.

[0138] Fifthly, embodiments of this application provide a computer program product that is stored in a storage medium and is executed by at least one processor to implement the various processes of the above-described method embodiment for dynamically setting the ECN threshold value, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0139] Finally, it should be noted that while some processes described in the embodiments of this application include multiple operations or steps that appear in a specific order, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0140] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for dynamically setting an ECN threshold value, characterized in that, The method includes: An experience replay pool is built based on the ECN threshold value set each time within a preset time period, as well as the corresponding network status data and network status evaluation value. Based on the data in the experience replay pool, the initial real-time model is trained using a result feedback method until the loss function of the initial real-time model converges, thus obtaining the real-time model and determining the network state and its optimal ECN threshold value. After the step of training the initial real-time model using feedback based on data from the experience replay pool until the loss function of the initial real-time model converges to obtain the real-time model, the process includes: Based on the real-time model, obtain the evaluation values ​​of the network status for M consecutive rounds and / or the throughput of the switch queue; When the evaluation value of the network state and / or the throughput of the switch queue are both less than their corresponding thresholds for M consecutive rounds, the experience replay pool is updated to obtain the updated experience replay pool. The real-time model is trained using a feedback-based method based on the updated experience replay pool until the loss function of the real-time model converges. Then, the process returns to the step of obtaining the evaluation values ​​of the network state for M consecutive rounds and / or the throughput of the switch queue based on the real-time model. The step of updating the experience replay pool to obtain the updated experience replay pool includes: The network state evaluation value corresponding to setting the ECN threshold value at time t-2 in the experience replay pool is compared with the network state evaluation value corresponding to setting the ECN threshold value at time t-1, where time t-2 is the time before time t-1. If the network state evaluation value after setting the ECN threshold value at time t-1 is greater than or equal to the network state evaluation value after setting the ECN threshold value at time t-2, then the updated ECN threshold value set at time t is obtained based on the ECN threshold value set at time t-1, where time t-1 is the time before time t. Using time t-1 as time t-2, return to the step of comparing the network state evaluation value corresponding to the setting of the ECN threshold value at time t-2 in the experience replay pool with the network state evaluation value corresponding to the setting of the ECN threshold value at time t-1. The updated experience replay pool is obtained by comparing the network state evaluation values ​​corresponding to the ECN threshold values ​​set at each time point in the experience replay pool.

2. The method for dynamically setting the ECN threshold value according to claim 1, characterized in that, The network status data includes: the current length of the switch egress queue, the output data rate of each link, the output rate of ECN tag packets for each link, and the currently set ECN threshold value; The network status assessment value is calculated based on the output data rate of each link, the fixed bandwidth of the link, and the length of the switch queue.

3. The method for dynamically setting the ECN threshold value according to claim 2, characterized in that, The network state assessment value is calculated based on the output data rate of each link, the fixed bandwidth of the link, and the length of the switch queue, including: The output data rate of each link after each ECN threshold setting is divided by the fixed bandwidth of the link, and the quotient is used as the throughput of the switch queue after each ECN threshold setting. The throughput is multiplied by the corresponding weight, and the length of the switch queue after each ECN threshold setting is multiplied by the corresponding weight. The sum is used as the network status evaluation value corresponding to each ECN threshold setting.

4. The method for dynamically setting the ECN threshold value according to claim 3, characterized in that, The step of performing result-feedback training on the initial real-time model based on the data in the experience replay pool includes: The ECN threshold value set at time t, the network status information at time t corresponding to the ECN threshold value set at time t, the network status evaluation value at time t, and the network status information at time t+1 are used as a data unit. When the number of data units is greater than the preset number, P consecutive data units are randomly selected as training datasets to perform result feedback training on the initial real-time model. Among them, the training dataset The value of q is in the range [1, n]. This represents the q-th network state information. This represents the q-th set ECN threshold value. This represents the evaluation value of the q-th network state. This represents the (q+1)th network state information, where n is a positive integer.

5. The method for dynamically setting the ECN threshold value according to claim 1, characterized in that, Prior to the step of performing result-feedback training on the initial real-time model based on the data in the experience replay pool, the following steps are included: When the model meets the pre-training conditions, the model is pre-trained until the model's loss function converges, and the initial real-time model is obtained. Pre-training conditions include the model not being trained, changes in the network environment, or the model being applied to a new network environment.

6. The method for dynamically setting the ECN threshold value according to claim 5, characterized in that, The method includes: The parameters of the initial real-time model are updated once each time the initial real-time model is trained. The parameters of the real-time model are updated once after each preset number of training iterations.

7. The method for dynamically setting the ECN threshold value according to claim 1, characterized in that, The step of obtaining the updated ECN threshold value set at time t based on the ECN threshold value set at time t-1 includes: Select any value from the high labeling threshold, low labeling threshold, and labeling probability value included in the ECN threshold set at time t-1 as the first target value; Select a second target value that is greater than the first target value from the range of the first target value; The first target value is replaced with the minimum value among the second target values, and the ECN threshold value after replacing the first target value is used as the updated ECN threshold value set at time t.

8. A device for dynamically setting ECN threshold values, characterized in that, The device includes: The data processing module is configured to build an experience replay pool based on the ECN threshold value set each time within a preset time period, as well as the corresponding network status data and network status evaluation value. The model training module is configured to perform result feedback training on the initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges, thereby obtaining the real-time model and determining the network state and its optimal ECN threshold value. The model training module is also configured to: Based on the real-time model, obtain the evaluation values ​​of the network status for M consecutive rounds and / or the throughput of the switch queue; When the evaluation value of the network state and / or the throughput of the switch queue are both less than their corresponding thresholds for M consecutive rounds, the experience replay pool is updated to obtain the updated experience replay pool. The real-time model is trained using a feedback-based method based on the updated experience replay pool until the loss function of the real-time model converges. Then, the process returns to the step of obtaining the evaluation values ​​of the network state for M consecutive rounds and / or the throughput of the switch queue based on the real-time model. The model training module is also configured to: The network state evaluation value corresponding to setting the ECN threshold value at time t-2 in the experience replay pool is compared with the network state evaluation value corresponding to setting the ECN threshold value at time t-1, where time t-2 is the time before time t-1. If the network state evaluation value after setting the ECN threshold value at time t-1 is greater than or equal to the network state evaluation value after setting the ECN threshold value at time t-2, then the updated ECN threshold value set at time t is obtained based on the ECN threshold value set at time t-1, where time t-1 is the time before time t. Using time t-1 as time t-2, return to the step of comparing the network state evaluation value corresponding to the setting of the ECN threshold value at time t-2 in the experience replay pool with the network state evaluation value corresponding to the setting of the ECN threshold value at time t-1. The updated experience replay pool is obtained by comparing the network state evaluation values ​​corresponding to the ECN threshold values ​​set at each time point in the experience replay pool.

Images