Method and device for dynamically setting ECN threshold value

Abstract

The invention discloses a method and a device for dynamically setting an ECN threshold value. The method comprises the following steps: constructing an experience playback pool based on the ECN threshold value set every time in a preset duration and corresponding network state data and network state evaluation values; and performing result feedback training on the initial real-time model based on the data in the experience playback pool until a loss function of the initial real-time model converges to obtain a real-time model, and determining a network state and an optimal ECN threshold value thereof. According to the invention, the technical problem that the network congestion control effect is poor because the static ECN threshold value cannot be dynamically adjusted according to the actual condition of the network and the congestion condition of the data center network cannot be accurately obtained in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of data centers, and in particular to a method and device for dynamically setting an ECN threshold value. Background Art

[0002] At present, data center network congestion is mainly manifested in the egress ports of network devices (such as switches and routers). When the traffic received by the ingress port is much larger than the traffic sent through the egress port, a large number of messages will be stranded in the corresponding queue of the egress port, which will cause data center network congestion, resulting in serious degradation of network performance such as message transmission delay and throughput, and even packet loss.

[0003] In the related art, static ECN (Explicit Congestion Notification) configuration is generally used in common scenarios of data centers to control network congestion. Specifically, according to the configured static ECN (Explicit Congestion Notification) threshold, it is determined whether the queue corresponding to the outbound port is congested. If congested, an ECN identifier is added to the queue corresponding to the outbound port and sent to the destination device. After the destination device receives the message with the ECN identifier, it sends a congestion CNP (Congestion Notification Packets) message to the source device to notify the source device to reduce the traffic sending rate, so as to achieve the purpose of controlling network congestion.

[0004] However, the network congestion situation will change with time and load, and the static ECN threshold value cannot be dynamically adjusted according to the actual situation of the network, so the congestion situation of the data center network cannot be accurately obtained, resulting in poor network congestion control effect. Summary of the invention

[0005] The present application provides a method and device for dynamically setting an ECN threshold value, which can solve the technical problem of poor network congestion control effect existing in the prior art.

[0006] To achieve the above objectives, this application provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a method for dynamically setting an ECN threshold value, the method comprising:

[0008] Build an experience replay pool based on the ECN threshold value set each time within a preset time period and the corresponding network status data and network status evaluation value;

[0009] The initial real-time model is trained in a result feedback manner based on the data in the experience replay pool until the loss function of the initial real-time model converges, thereby obtaining a real-time model and determining the network state and its optimal ECN threshold value.

[0010] In a second aspect, an embodiment of the present application provides a device for dynamically setting an ECN threshold value, the device comprising:

[0011] A data processing module is configured to construct an experience replay pool based on the ECN threshold value set each time within a preset time period and the corresponding network status data and network status evaluation value;

[0012] The model training module is configured to perform result feedback training on the initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges, obtain the real-time model, and determine the network state and its optimal ECN threshold value.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor; the processor is used to read and execute a computer program stored in the memory to implement the steps of the aforementioned method for dynamically setting an ECN threshold value.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and when the computer executable instructions are executed, the steps of the aforementioned method for dynamically setting an ECN threshold value are implemented.

[0015] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned method for dynamically setting an ECN threshold value.

[0016] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0017] An experience replay pool is constructed based on the ECN threshold value set each time within a preset time period and the corresponding network status data and network status evaluation value; the initial real-time model is trained in a result feedback manner based on the data in the experience replay pool until the loss function of the initial real-time model converges, a real-time model is obtained, and the network status and its optimal ECN threshold value are determined. Through the present invention, the initial real-time model is trained in a result feedback manner based on the ECN threshold value set each time and its corresponding length of the current switch outlet queue, the output data rate of each link, the output rate of the ECN marked packet of each link, the currently set ECN threshold value and the network state evaluation value, the relationship between the network state and the set ECN threshold value is learned, and the mapping between the two is dynamically adjusted to build a mapping relationship between the network state and its optimal ECN threshold value configuration, that is, the ECN threshold value can be dynamically adjusted according to the actual situation of the network, even if there is burst traffic in the network, the ECN threshold value can be dynamically adjusted, so as to accurately obtain the congestion situation of the data center network, and then reduce the traffic sending rate to achieve the purpose of controlling network congestion, thereby solving the technical problem that the static ECN threshold value in the prior art cannot be dynamically adjusted according to the actual situation of the network, the congestion situation of the data center network cannot be accurately obtained, and the network congestion control effect is poor. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a first embodiment of a method for dynamically setting an ECN threshold value of the present application;

[0020] Figure 2 A flowchart of a second embodiment of a method for dynamically setting an ECN threshold value of the present application;

[0021] Figure 3 A flowchart of a third embodiment of a method for dynamically setting an ECN threshold value of the present application;

[0022] Figure 4 for Figure 3 A detailed flow chart of step S50;

[0023] Figure 5 A schematic diagram of functional modules of an embodiment of a device for dynamically setting an ECN threshold value of the present application;

[0024] Figure 6This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be described in detail below with reference to the accompanying drawings.

[0027] In a first aspect, an embodiment of the present application provides a method for dynamically setting an ECN threshold value.

[0028] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for dynamically setting the ECN threshold value of this application. Figure 1 As shown, the method for dynamically setting the ECN threshold value includes:

[0029] Step S10, constructing an experience replay pool based on the ECN threshold value set each time within a preset time period and the corresponding network status data and network status evaluation value;

[0030] In this embodiment, the action space action_space is constructed based on each set ECN threshold value collected within the preset time period T from system startup to time t.

[0031]

[0032] Where a=(k min ,k max ,p max ), k min represents the low labeling threshold, k max represents the high labeling threshold, p max represents the label probability value. max >k min , and is the value of the maximum number of congestion notification signals that the sender can receive when the network is congested; 0 <p max <100. T is the total duration of data collected during the preset time period T from system startup to time t.

[0033] After each setting of the ECN threshold value, the network status data and the network status evaluation value will change accordingly. Therefore, each setting of the ECN threshold value has its corresponding network status data and network status evaluation value.

[0034] The ECN threshold value set at time t, the network status information at time t corresponding to the setting of the ECN threshold value at time t, the evaluation value of the network status at time t, and the network status information at time t+1 constitute a data unit (S t ,a t ,r t, S t+1 ) is called an experience QS t , the collection of stored experience is called the experience replay pool, expressed as: replay_pod =

[0035] {(S1,a1,r 1, S2),(S2,a2,r2,S3),…,(S t ,a t ,r t ,S t+1 ),…,(S i ,a i ,r i ,S i+1 )|i≥1 and is an integer}.

[0036] Optionally, in one embodiment, the network status data includes: the length of the current switch egress queue, the output data rate of each link, the output rate of ECN marked packets of each link, and the currently set ECN threshold value.

[0037] In this embodiment, the network (queue) status information S can be represented by a four-tuple of the length of the current switch egress queue (qlen), the output data rate of each link (txRate), the output rate of each link ECN marked packet (txRate_mark), and the currently set ECN threshold value (ECN_cur), that is, S = (qlen, txRate, txRate_mark, ECN_cur).

[0038] In the process of constructing the action space, each ECN threshold is set to the network state S collected after the switch is accumulated to form a state space state_space = (S1, S2, S3, ..., S v ), where v is a positive integer.

[0039] Optionally, in one embodiment, the network status evaluation value is calculated based on the output data rate of each link, the fixed bandwidth of the link, and the length of the switch queue, including:

[0040] The output data rate of each link after each ECN threshold value is set is divided by the fixed bandwidth of the link, and the quotient obtained is used as the throughput rate of the switch queue after each ECN threshold value is set;

[0041] The throughput is multiplied by the corresponding weight, and the sum is added to the length of the switch queue after each setting of the ECN threshold value multiplied by the corresponding weight to obtain a network status evaluation value corresponding to each set ECN threshold value.

[0042] In this embodiment, the output data rate of each link after each ECN threshold value is set is divided by the fixed bandwidth of the link, and the quotient obtained is used as the throughput rate of the switch queue after each ECN threshold value is set. Specifically, T i The throughput rate of the switch queue after setting the ECN threshold at all times Where, T i (R) indicates T i The throughput rate of the switch queue after setting the ECN threshold at all times, txRate i Indicates T i The output data rate of each link after the ECN threshold is set at all times. BW represents the fixed bandwidth of the link.

[0043] Based on T i The throughput rate of the switch queue after setting the ECN threshold value at all times and T i The length of the switch queue after setting the ECN threshold at all times is calculated to obtain T i The network status evaluation value after setting the ECN threshold value at all times is used to avoid the instability of the calculated value caused by frequent changes in link bandwidth utilization and instantaneous queue length. Specifically, T i The evaluation value r of the network status after setting the ECN threshold value at all times i =μ1*T i (R)+μ2*L i (Q), where r i Indicates T i The evaluation value of the network status after setting the ECN threshold value at all times, μ1 is the proportional weight of the throughput rate of the switch queue, and μ2 is the proportional weight of the length of the switch queue. In the formula, a t′ Indicates the ECN threshold value set at time t′ represents s at time t t′ Power, s t′ It indicates the evaluation value of the network status after the ECN threshold is set at time t′. t′ can be time t+1 or time t+n. L i (Q) means T i The length of the switch queue after setting the ECN threshold at all times.

[0044] By analogy, the network status evaluation value corresponding to each set ECN threshold value can be calculated.

[0045] Step S20, performing result feedback training on the initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges, obtaining the real-time model, and determining the network state and its optimal ECN threshold value.

[0046] In this embodiment, the data in the experience replay pool includes the ECN threshold value set each time and its corresponding length of the current switch egress queue, the output data rate of each link, the output rate of the ECN marked packet of each link, the currently set ECN threshold value and the network status evaluation value.

[0047] The initial real-time model is trained by feedback based on the data in the experience replay pool, that is, each time the ECN threshold is set to the switch, the network status data (the length of the current switch export queue, the output data rate of each link, the output rate of the ECN marked packet of each link and the currently set ECN threshold) and the network status evaluation value are obtained. The initial real-time model is trained by feedback based on the ECN threshold set at time t, the network status information at time t corresponding to the setting of the ECN threshold at time t, the evaluation value of the network status at time t and the network status information at the next time, that is, time t+1, to learn the relationship between the network status and the set ECN threshold, and dynamically adjust the mapping between the two to construct a mapping relationship between the network status and its optimal ECN configuration. The traditional congestion control mechanism relies on packet loss as a congestion signal, and packet loss may cause fast retransmission and fast recovery operations on the sender, which will lead to network performance degradation. By using the method in this embodiment, ECN can be dynamically marked according to the congestion of the switch egress queue, the ECN threshold value can be set, and the sender can be notified to reduce the message sending rate to reduce the packet loss rate or even avoid packet loss, thereby improving network performance and better adapting to the needs of high-speed networks.

[0048] The training process calculates the loss function of the initial real-time model and determines the convergence degree of the initial real-time model to characterize the training degree of the initial real-time model. If the change of the loss function is less than the set threshold, that is, the loss function converges, it is considered that the initial real-time model can be used for reasoning. At this time, the training ends and the real-time model is obtained.

[0049] Among them, the loss function is as follows:

[0050]

[0051] In the formula, y j represents the loss function of the initial real-time model, r j Indicates t jThe ECN threshold value a at the time j The corresponding network state evaluation value, γ∈[0,1], is a discount factor, Represents a deep neural network with a parameter θ′, S j+1 represents the network status evaluation value at time t(j+1), It indicates mapping from the evaluation value of the network state to setting the ECN threshold value, and maximizing the discounted cumulative network state evaluation value to find the ECN threshold value of the optimal network state evaluation value, a indicates setting the ECN threshold value, θ indicates the parameters of the evaluation network, and θ′ indicates the parameters of the target network. Among them, the target network is obtained by saving the evaluation network every N steps (regularly updating the target).

[0052] Optionally, in one embodiment, the step of performing result feedback training on the initial real-time model based on the data in the experience replay pool includes:

[0053] The ECN threshold value set at time t, the network status information at time t corresponding to the setting of the ECN threshold value at time t, the evaluation value of the network status at time t, and the network status information at time t+1 are taken as a data unit;

[0054] When the number of the data units is greater than a preset number, randomly selecting P consecutive data units as training data sets to perform result feedback training on the initial real-time model;

[0055] Among them, the training data set

[0056] E(p,q)={((S q ,a q ,r q ,S q+1 ),(S q+1 ,a q+1 ,r q+1 ,S q+2 ),…,(S q+p-1 ,a q+p-1 ,r q+p-1 ,S q+p 0)}, q range is [1, n]}, S q Indicates the qth network status information, a q represents the qth set ECN threshold, r q represents the evaluation value of the qth network state, S q+1 Represents the q+1th network status information, where n is a positive integer.

[0057] In this embodiment, an action cycle that is one order of magnitude longer than the RTT (Round-Trip Time) is used to adjust the ECN threshold value to prevent the congestion control rate from being in the same frequency as the dynamic adjustment of the ECN threshold value, causing control interference and thus affecting the congestion control performance. t , the network status information St at time t corresponding to the setting of the ECN threshold at time t, and the evaluation value r of the network status at time t t And the network status information S at the next time t+1 t+1 Constitute a data unit (S t ,a t ,r t ,S t+1 ) is called an experience QS t , the collection of stored experiences is called the experience replay pool, which is expressed as: replay_pod = {(S1, a1, r1, S2), (S2, a2, r2, S3), …, (S t ,a t ,r t ,S t+1 ),…,(S i ,a i ,r i ,S i+1 )|i>=1 and is an integer}.

[0058] At △t time interval, is the number of iterations. When the number of data units in the experience replay pool replay_pod is greater than the preset number, P consecutive data units are randomly selected as the training data set to perform result feedback training on the initial real-time model. Training data set E(p,q)={((S q ,a q ,r q ,S q+1 ),(S q+1 ,a q+1 ,r q+1 ,S q+2 ),…,(S q+p-1 ,a q+p-1 ,r q+p-1 ,S q+p ))}, q range is [1, n]. Among them, S q Indicates the qth network status information, a q represents the qth set ECN threshold, r q represents the evaluation value of the qth network state, S q+1represents the q+1th network status information, n is a positive integer, and P is less than or equal to a preset number. It is easy to imagine that the data in the obtained training data set will be different when the data is obtained at different times.

[0059] In this embodiment, an experience replay pool is constructed based on the ECN threshold value set each time within a preset time period and the corresponding network status data and network status evaluation value; the initial real-time model is trained in a result feedback manner based on the data in the experience replay pool until the loss function of the initial real-time model converges, a real-time model is obtained, and the network status and its optimal ECN threshold value are determined. In this embodiment, the initial real-time model is trained in a result feedback manner based on the ECN threshold value set each time and its corresponding length of the current switch egress queue, the output data rate of each link, the output rate of the ECN marked packet of each link, the currently set ECN threshold value and the network status evaluation value, the relationship between the network status and the set ECN threshold value is learned, and the mapping between the two is dynamically adjusted to construct a mapping relationship between the network status and its optimal ECN threshold value configuration, that is, the ECN threshold value can be dynamically adjusted according to the actual situation of the network, even if there is burst traffic in the network, the ECN threshold value can be dynamically adjusted, so as to accurately obtain the congestion situation of the data center network, and then reduce the traffic sending rate to achieve the purpose of controlling network congestion, thereby solving the technical problem that the static ECN threshold value in the prior art cannot be dynamically adjusted according to the actual situation of the network, the congestion situation of the data center network cannot be accurately obtained, and the network congestion control effect is poor.

[0060] Optionally, in one embodiment, referring to Figure 2 , Figure 2 This is a flow chart of the second embodiment of the method for dynamically setting the ECN threshold value of this application. Figure 2 As shown, before the step of performing result feedback training on the initial real-time model based on the data in the experience replay pool, it includes:

[0061] Step S30, when the model meets the pre-training conditions, pre-train the model until the loss function of the model converges to obtain an initial real-time model;

[0062] Among them, the pre-training conditions include that the model has not been trained, the network environment has changed, or the model is applied to a new network environment.

[0063] In this embodiment, in the three cases where the model has not been trained, the network environment has changed, or it is applied to a new network environment, the model needs to be pre-trained. The training process at this time is called the initial exploration stage. In the initial exploration stage, after building an experience replay pool based on the ECN threshold value set each time within a preset time length and the corresponding network status data and network status evaluation value, the model is pre-trained, that is, the model is trained in a result feedback manner until the loss function of the model in the initial exploration stage converges. At this time, the initial exploration stage ends and the initial real-time model is obtained.

[0064] Optionally, in one embodiment, the method includes:

[0065] Each time the initial real-time model is trained, the parameters of the initial real-time model are updated;

[0066] The real-time model is trained a preset number of times and the parameters of the model are updated once.

[0067] In this embodiment, since the initial real-time model is subjected to feedback training, the parameters of the initial real-time model need to be updated in each iteration. The process of updating the parameters θ of the initial real-time model is as follows:

[0068]

[0069] In the formula, L(θ) represents the target network, N represents the number of iterations, and y j represents the loss function of the initial real-time model, Q(S j ,a j ,θ) represents a deep neural network with a parameter θ, S j Indicates t j The network status evaluation value at time a j Indicates t j The ECN threshold value set at the moment, θ represents the parameters of the evaluation network.

[0070] In the three cases where the model is not trained, the network environment changes, or it is applied to a new network environment, the model is pre-trained to obtain the initial real-time model, and the model parameters are updated every time the real-time model is trained for a preset number of times. The model parameters are updated regularly to facilitate pre-inference applications in different application scenarios. This method smooths the model robustness and adaptability challenges brought about by changes in scenarios and network status, and ensures network quality and resource utilization.

[0071] The process of updating the model's parameters θ is as follows:

[0072]

[0073] In the formula, represents the gradient, N represents the number of iterations, yj represents the loss function of the initial real-time model, Q(S j , a j , θ) represents a deep neural network with parameter θ, and S j represents the network state evaluation value at time t j , and a j represents the ECN threshold set at time t j , and θ represents the parameters of the evaluation network.

[0074] Optionally, in one embodiment, referring to Figure 3 , Figure 3 , which is a schematic flowchart of the third embodiment of the method for dynamically setting the ECN threshold in this application. As Figure 3 shown, after the step of performing result feedback training on the initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges to obtain the real-time model, the following steps are included:

[0075] Step S40: Obtain the evaluation values of the network state for continuous M rounds and / or the throughput rate of the switch queue based on the real-time model;

[0076] Step S50: When the evaluation values of the network state for continuous M rounds and / or the throughput rate of the switch queue are both less than their corresponding thresholds, update the experience replay pool to obtain an updated experience replay pool;

[0077] Step S60: Perform result feedback training on the real-time model based on the updated experience replay pool until the loss function of the real-time model converges, and then return to execute the step of obtaining the evaluation values of the network state for continuous M rounds and / or the throughput rate of the switch queue based on the real-time model.

[0078] In this embodiment, the evaluation values of the network state output by the real-time model for continuous M rounds and the throughput rate of the switch queue are obtained. When the evaluation values of the network state for continuous M rounds are both less than their corresponding thresholds, and / or the throughput rates of the switch queue for continuous M rounds are both less than their corresponding thresholds, it is determined at this time that the adaptability of the real-time model is poor, that is, the configured ECN threshold deduced has a poor effect on network congestion and throughput management. Specifically, the evaluation values r<score of the network state for continuous m rounds and / or the throughput rate T(R)<rate_high of the switch queue, where score is the threshold corresponding to the evaluation value of the network state, and rate_high is the threshold corresponding to the throughput rate of the switch queue, specifically when the bandwidth is close to 100% utilization, such as 0.95.

[0079] At this point, the experience exploration phase will be entered to update the experience replay pool to obtain an updated experience replay pool. Then, the real-time model will be trained in a result feedback manner based on the data in the updated experience replay pool until the loss function of the real-time model converges, and the step of obtaining the evaluation value of the network state for M consecutive rounds and / or the throughput rate of the switch queue based on the real-time model is returned.

[0080] In another embodiment, when the evaluation value of the network status and the throughput rate of the switch queue output by the real-time model for M consecutive rounds are both greater than or equal to their corresponding thresholds, it is determined that the real-time model has good adaptability, that is, the inferred ECN threshold value configuration has a good effect on network congestion and throughput management.

[0081] Furthermore, the network state includes a normal communication state and a congestion state. After obtaining the throughput rate of the switch queue and the evaluation value of the network state at this time, it is determined whether the queue corresponding to the port is congested based on the evaluation value of the network state at this time or the throughput rate of the switch queue. If congested, the ECN threshold value at this time is added to the queue corresponding to the port and sent to the destination device. After receiving the message with the ECN threshold value identifier, the destination device sends a congestion CNP (Congestion Notification Packets, ECN data packet) message to the source device to notify the source device to reduce the traffic sending rate, thereby achieving the purpose of controlling network congestion.

[0082] Optionally, in one embodiment, referring to Figure 4 , Figure 4 for Figure 3 Detailed flow chart of step S50 in FIG. Figure 4 As shown, the step of updating the experience replay pool to obtain an updated experience replay pool includes:

[0083] Step S501, comparing the network status evaluation value corresponding to the ECN threshold value set at time t-2 in the experience replay pool with the network status evaluation value corresponding to the ECN threshold value set at time t-1, wherein time t-2 is the previous time of time t-1;

[0084] Step S502, if the network status evaluation value corresponding to the setting of the ECN threshold value at time t-1 is greater than or equal to the network status evaluation value corresponding to the setting of the ECN threshold value at time t-2, then the updated ECN threshold value set at time t is obtained based on the ECN threshold value set at time t-1, wherein time t-1 is a time before time t;

[0085] Step S503, taking time t-1 as time t-2, returning to the step of comparing the network status evaluation value corresponding to the setting of the ECN threshold value at time t-2 in the experience replay pool with the network status evaluation value corresponding to the setting of the ECN threshold value at time t-1;

[0086] Step S504, until the network status evaluation values ​​corresponding to the ECN threshold values ​​set at each moment in the experience replay pool are compared, an updated experience replay pool is obtained.

[0087] In this embodiment, the time t-2 is the time before the time t-1. Taking the time t-2 as the time t1 and the time t-1 as the time t2 as an example, the evaluation value of the network state corresponding to the ECN threshold value at the time t1 in the experience replay pool is compared with the evaluation value of the network state corresponding to the ECN threshold value at the time t2, that is, a1=(k min1 , k max1 、p max1 ) The evaluation value of the network state corresponding to r1 and a2 = (k min2 , k max2 、p max2 ) is compared with the evaluation value r2 of the corresponding network status.

[0088] If the evaluation value r2 of the network state corresponding to the ECN threshold value at time t2 is greater than or equal to the evaluation value r1 of the network state corresponding to the ECN threshold value at time t1, that is, when the difference r2-r1 is greater than or equal to zero, then the updated ECN threshold value a3′ at time t3 is obtained based on the ECN threshold value at time t2, where time t3 is the next time after time t2.

[0089] If the evaluation value r2 of the network state corresponding to the ECN threshold value at time t2 is less than the evaluation value r1 of the network state corresponding to the ECN threshold value at time t1, that is, when the difference r2-r1 is less than zero, the ECN threshold value at time t3 remains unchanged.

[0090] Taking time t2 as time t1, return to the step of comparing the evaluation value r1 of the network state corresponding to the ECN threshold value at time t1 in the action space with the evaluation value r2 of the network state corresponding to the ECN threshold value at time t2. It is easy to think that since time t2 is taken as time t1, at this time, if r2 is greater than or equal to r2, when executing the step of updating the ECN threshold value at time t3, it is actually to update the ECN threshold value a4 at time t4, and obtain the updated ECN threshold value a4′ at time t4.

[0091] Until the evaluation values ​​of the network status corresponding to the ECN threshold values ​​set at each moment in the experience replay pool are compared, an updated experience replay pool can be obtained.

[0092] Optionally, in one embodiment, the step of obtaining the updated ECN threshold value set at time t based on the ECN threshold value set at time t-1 includes:

[0093] Select any value from the high marking threshold, the low marking threshold, and the marking probability value included in the ECN threshold value set at time t-1 as the first target value;

[0094] Selecting a second target value greater than the first target value from the value range of the first target value;

[0095] The first target value is replaced by the minimum value of the second target values, and the ECN threshold value after replacing the first target value is used as the updated ECN threshold value set at time t.

[0096] In this embodiment, time t-1 is the moment before time t, and the ECN threshold value includes the high marking threshold kmax, the low marking threshold kmin, and the marking probability value pmax. Time t-1 is time t2, time t is time t3, a2=(k min2 ,k max2 ,p max2 )=[10,128,50] as an example, from the ECN threshold value a2=(k min2 ,k max2 ,p max2 ) as the first target value.

[0097] If the first target value selected is the low marking threshold k min2 , then from the low marking threshold k min Select a value greater than k from the range of min2 = the second target value of 10. With a low marking threshold k min The value range of k is [1, 2, 5, 10, 20, 40, 80, 160, 320, 640, 1280, 2560, 5120, 10240], for example, the second target value includes 20, 40, 80, 160, 320, 640, 1280, 2560, 5120, 10240. Replace the first target value k with the minimum value 20 in the second target value min2 , the ECN threshold value [20, 128, 50] after replacing the first target value is obtained, and [20, 128, 50] is used as the updated ECN threshold value a3′ at time t3, that is, a3′=[20, 128, 50]. Among them, time t2 is the time before time t3.

[0098] In another embodiment, if the selected first target value is the high marking threshold k max2 , then from the high marking threshold kmax Select a value greater than k from the range of max2 = The second target value of 128. The high marking threshold k max For example, the value range of k is [128, 256, 512, 1024, 2048, 5120, 10240], then the second target value includes 256, 512, 1024, 2048, 5120, 10240. Replace the first target value k with the minimum value 256 in the second target value max2 , the ECN threshold value [10, 256, 50] after replacing the first target value is obtained, and [10, 256, 50] is used as the updated ECN threshold value a3′ at time t3, that is, a3′=[10, 256, 50].

[0099] In another embodiment, if the selected first target value is the marking probability threshold p max2 , then from the label probability threshold p max Select a value greater than p from the range of max2 = the second target value of 50. Using the marking probability threshold p max For example, the value range of is [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 25, 50, 75, 100], then the second target value includes 75 and 100. Replace the first target value p with the minimum value 75 in the second target value max2 , and obtain the ECN threshold value [10, 128, 75] after replacing the first target value. [10, 128, 75] is used as the updated ECN threshold value a3′ at time t3, that is, a3′=[10, 128, 75].

[0100] It should be noted that if there is no second target value greater than the first target value in the value range of the first target value, the first target value remains unchanged.

[0101] In a second aspect, an embodiment of the present application also provides a device for dynamically setting an ECN threshold value.

[0102] In one embodiment, referring to Figure 5 , Figure 5 This is a functional module diagram of an embodiment of a device for dynamically setting an ECN threshold value of the present application. Figure 5 As shown, the device for dynamically setting the ECN threshold value includes:

[0103] The data processing module 10 is configured to construct an experience replay pool based on the ECN threshold value set each time within a preset time period and the corresponding network status data and network status evaluation value;

[0104] The model training module 20 is configured to perform result feedback training on the initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges, obtain the real-time model, and determine the network state and its optimal ECN threshold value.

[0105] Optionally, in one embodiment, the network status data includes: the length of the current switch egress queue, the output data rate of each link, the output rate of ECN marked packets of each link, and the currently set ECN threshold value;

[0106] The network status evaluation value is calculated based on the output data rate of each link, the fixed bandwidth of the link, and the length of the switch queue.

[0107] Optionally, in one embodiment, the apparatus for dynamically setting the ECN threshold value further includes a calculation module configured to:

[0108] The output data rate of each link after each ECN threshold value is set is divided by the fixed bandwidth of the link, and the quotient obtained is used as the throughput rate of the switch queue after each ECN threshold value is set;

[0109] The throughput is multiplied by the corresponding weight, and the sum is added to the length of the switch queue after each setting of the ECN threshold value multiplied by the corresponding weight to obtain a network status evaluation value corresponding to each set ECN threshold value.

[0110] Optionally, in one embodiment, the model training module 20 is specifically configured to:

[0111] The ECN threshold value set at time t, the network status information at time t corresponding to the setting of the ECN threshold value at time t, the evaluation value of the network status at time t, and the network status information at time t+1 are taken as a data unit;

[0112] When the number of the data units is greater than a preset number, randomly selecting P consecutive data units as training data sets to perform result feedback training on the initial real-time model;

[0113] Among them, the training data set

[0114] E(p,q)=

[0115] {((S q ,a q ,r q ,S q+1 ),(S q+1 ,a q+1 ,r q+1 ,S q+2 ),…,(S q+p-1 ,a q+p-1 ,rq+p-1 ,S q+p ))}, q ranges from [1, n], S q Indicates the qth network status information, a q represents the qth set ECN threshold, r q represents the evaluation value of the qth network state, S q+1 Represents the q+1th network status information, where n is a positive integer.

[0116] Optionally, in one embodiment, the model training module 20 is further configured to:

[0117] When the model meets the pre-training conditions, the model is pre-trained until the loss function of the model converges to obtain the initial real-time model;

[0118] Among them, the pre-training conditions include that the model has not been trained, the network environment has changed, or the model is applied to a new network environment.

[0119] Optionally, in one embodiment, the model training module 20 is further configured to:

[0120] Each time the initial real-time model is trained, the parameters of the initial real-time model are updated;

[0121] The real-time model is trained a preset number of times and the parameters of the model are updated once.

[0122] Optionally, in one embodiment, the model training module 20 is further configured to:

[0123] Acquire evaluation values ​​of M consecutive rounds of network states and / or throughput rates of switch queues based on the real-time model;

[0124] When the evaluation values ​​of the network status in M ​​consecutive rounds and / or the throughput of the switch queue are both less than their corresponding thresholds, the experience replay pool is updated to obtain an updated experience replay pool;

[0125] The real-time model is trained in a result-feedback manner based on the updated experience replay pool until the loss function of the real-time model converges, and the step of obtaining evaluation values ​​of M consecutive rounds of network states and / or the throughput rate of the switch queue based on the real-time model is returned to be executed.

[0126] Optionally, in one embodiment, the model training module 20 is further configured to:

[0127] Compare the network status evaluation value corresponding to the ECN threshold value set at time t-2 in the experience replay pool with the network status evaluation value corresponding to the ECN threshold value set at time t-1, where time t-2 is the previous time of time t-1;

[0128] If the network status evaluation value corresponding to the setting of the ECN threshold value at time t-1 is greater than or equal to the network status evaluation value corresponding to the setting of the ECN threshold value at time t-2, then the updated ECN threshold value set at time t is obtained based on the ECN threshold value set at time t-1, where time t-1 is the moment before time t;

[0129] Taking time t-1 as time t-2, returning to the step of comparing the network status evaluation value corresponding to the setting of the ECN threshold value at time t-2 in the experience replay pool with the network status evaluation value corresponding to the setting of the ECN threshold value at time t-1;

[0130] Until all network status evaluation values ​​corresponding to the ECN threshold values ​​set at each moment in the experience replay pool are compared, an updated experience replay pool is obtained.

[0131] Optionally, in one embodiment, the model training module 20 is further configured to:

[0132] Select any value from the high marking threshold, the low marking threshold, and the marking probability value included in the ECN threshold value set at time t-1 as the first target value;

[0133] Selecting a second target value greater than the first target value from the value range of the first target value;

[0134] The first target value is replaced by the minimum value of the second target values, and the ECN threshold value after replacing the first target value is used as the updated ECN threshold value set at time t.

[0135] The functional implementation of each module in the apparatus for dynamically setting the ECN threshold value corresponds to each step in the method embodiment for dynamically setting the ECN threshold value, and its functions and implementation processes are not described one by one here.

[0136] In a third aspect, the present application also provides an electronic device, the structure of which is as follows: Figure 6 As shown, it includes: a memory and a processor, wherein the processor is used to read and execute the computer program stored in the memory to implement the aforementioned method for dynamically setting the ECN threshold value.

[0137] In a fourth aspect, an embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and when the computer executable instructions are executed, the aforementioned method for dynamically setting an ECN threshold value is implemented.

[0138] In a fifth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the various processes of the method embodiment for dynamically setting the ECN threshold value as described above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0139] Finally, it should be noted that: in some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0140] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for dynamically setting an ECN threshold value, characterized in that: The method comprises: Build an experience replay pool based on the ECN threshold value set each time within a preset time period and the corresponding network status data and network status evaluation value; The initial real-time model is trained in a result feedback manner based on the data in the experience replay pool until the loss function of the initial real-time model converges, thereby obtaining a real-time model and determining the network state and its optimal ECN threshold value.

2. The method for dynamically setting an ECN threshold value according to claim 1, characterized in that: The network status data includes: the length of the current switch egress queue, the output data rate of each link, the output rate of the ECN marked packets of each link, and the currently set ECN threshold value; The network status evaluation value is calculated based on the output data rate of each link, the fixed bandwidth of the link, and the length of the switch queue.

3. The method for dynamically setting the ECN threshold value according to claim 2, characterized in that: The network status evaluation value is calculated based on the output data rate of each link, the fixed bandwidth of the link, and the length of the switch queue, including: The output data rate of each link after each ECN threshold value is set is divided by the fixed bandwidth of the link, and the quotient obtained is used as the throughput rate of the switch queue after each ECN threshold value is set; The throughput is multiplied by the corresponding weight, and the sum is added to the length of the switch queue after each setting of the ECN threshold value multiplied by the corresponding weight to obtain a network status evaluation value corresponding to each set ECN threshold value.

4. The method for dynamically setting the ECN threshold value according to claim 3, characterized in that: The step of performing result feedback training on the initial real-time model based on the data in the experience replay pool includes: The ECN threshold value set at time t, the network status information at time t corresponding to the setting of the ECN threshold value at time t, the evaluation value of the network status at time t, and the network status information at time t+1 are taken as a data unit; When the number of the data units is greater than a preset number, randomly selecting P consecutive data units as training data sets to perform result feedback training on the initial real-time model; Among them, the training data set E(p,q)= {((S q ,a q ,r q ,S q+1 ),(S q+1 ,a q+1 ,r q+1 ,S q+2 ),…,(S q+p-1 ,a q+p-1 ,r q+p-1 ,S q+p ))}, q ranges from [1, n], S q Indicates the qth network status information, a q represents the qth set ECN threshold, r q represents the evaluation value of the qth network state, S q+1 Represents the q+1th network status information, where n is a positive integer.

5. The method for dynamically setting an ECN threshold value according to claim 1, characterized in that: Before the step of performing result feedback training on the initial real-time model based on the data in the experience replay pool, the method includes: When the model meets the pre-training conditions, the model is pre-trained until the loss function of the model converges to obtain the initial real-time model; Among them, the pre-training conditions include that the model has not been trained, the network environment has changed, or the model is applied to a new network environment.

6. The method for dynamically setting the ECN threshold value according to claim 5, characterized in that: The method comprises: Each time the initial real-time model is trained, the parameters of the initial real-time model are updated; The real-time model is trained a preset number of times and the parameters of the model are updated once.

7. The method for dynamically setting an ECN threshold value according to claim 1, characterized in that: After the step of performing result feedback training on the initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges to obtain the real-time model, the method includes: Acquire evaluation values ​​of M consecutive rounds of network states and / or throughput rates of switch queues based on the real-time model; When the evaluation values ​​of the network status in M ​​consecutive rounds and / or the throughput of the switch queue are both less than their corresponding thresholds, the experience replay pool is updated to obtain an updated experience replay pool; The real-time model is trained in a result-feedback manner based on the updated experience replay pool until the loss function of the real-time model converges, and the step of obtaining evaluation values ​​of M consecutive rounds of network states and / or the throughput rate of the switch queue based on the real-time model is returned to be executed.

8. The method for dynamically setting the ECN threshold value according to claim 7, characterized in that: The step of updating the experience replay pool to obtain an updated experience replay pool comprises: Compare the network status evaluation value corresponding to the ECN threshold value set at time t-2 in the experience replay pool with the network status evaluation value corresponding to the ECN threshold value set at time t-1, where time t-2 is the previous time of time t-1; If the network status evaluation value corresponding to the setting of the ECN threshold value at time t-1 is greater than or equal to the network status evaluation value corresponding to the setting of the ECN threshold value at time t-2, then the updated ECN threshold value set at time t is obtained based on the ECN threshold value set at time t-1, where time t-1 is the moment before time t; Taking time t-1 as time t-2, returning to the step of comparing the network status evaluation value corresponding to the setting of the ECN threshold value at time t-2 in the experience replay pool with the network status evaluation value corresponding to the setting of the ECN threshold value at time t-1; Until all network status evaluation values ​​corresponding to the ECN threshold values ​​set at each moment in the experience replay pool are compared, an updated experience replay pool is obtained.

9. The method for dynamically setting an ECN threshold value according to claim 8, characterized in that: The step of obtaining the updated ECN threshold value set at time t based on the ECN threshold value set at time t-1 includes: Select any value from the high marking threshold, the low marking threshold, and the marking probability value included in the ECN threshold value set at time t-1 as the first target value; Selecting a second target value greater than the first target value from the value range of the first target value; The first target value is replaced by the minimum value of the second target values, and the ECN threshold value after replacing the first target value is used as the updated ECN threshold value set at time t.

10. A device for dynamically setting an ECN threshold value, characterized in that: The device comprises: A data processing module is configured to construct an experience replay pool based on the ECN threshold value set each time within a preset time period and the corresponding network status data and network status evaluation value; The model training module is configured to perform result feedback training on the initial real-time model based on the data in the experience replay pool until the loss function of the initial real-time model converges, obtain the real-time model, and determine the network state and its optimal ECN threshold value.

Images