Congestion control method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a congestion control method, device, electronic device and storage medium, the method comprising: when the preset collection period is reached, obtaining the network status data of the current collection period as the first network status data; based on the first network status data, calculating the reward value of the current collection period as the first reward value; inputting the first network status data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy; adjusting the current congestion window according to the target adjustment strategy. For different network environments, there is no need to distinguish whether the congestion signal is caused by network congestion, and the reward value can be calculated based on the first network status data, thereby achieving the adjustment of the congestion window. Therefore, the congestion control method of the present application can be applied to complex network environments and can improve the effectiveness of congestion control.

Description

Technical Field

[0001] The present application relates to the field of computer network technology, and in particular to a congestion control method, device, electronic device and storage medium. Background Art

[0002] With the development of network technology, the current network environment is becoming more and more complex, and there are many factors that affect network transmission efficiency. In order to avoid network congestion, ensure network stability and efficient data transmission, the data transmission rate of the data sender can be adjusted through congestion control.

[0003] For example, in related technologies, heuristic congestion control methods use packet loss rate and delay as congestion signals, and dynamically control the sending rate or congestion window to avoid network congestion. However, as the network environment becomes increasingly complex, this method cannot effectively distinguish whether the congestion signal is caused by network congestion, and it is difficult to adapt to the current complex network environment, resulting in low effectiveness of congestion control. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a congestion control method, device, electronic device and storage medium, which are applicable to complex network environments and improve the effectiveness of congestion control. The specific technical solutions are as follows:

[0005] According to a first aspect of an embodiment of the present application, a congestion control method is provided, the method comprising:

[0006] When the preset collection period is reached, the network status data of the current collection period is obtained as the first network status data;

[0007] Wherein, the first network status data includes the sending rate and receiving rate of the current collection cycle; the sending rate of the current collection cycle is determined based on the sending rate collected at each specified time in the current collection cycle; the receiving rate of the current collection cycle is determined based on the receiving rate collected at each specified time in the current collection cycle; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last received ACK message and the sending time of the data packet corresponding to the first time;

[0008] Based on the first network status data, calculating a reward value of a current collection cycle as a first reward value;

[0009] Inputting the first network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy; wherein the adjustment strategy prediction network model is trained based on a reinforcement learning algorithm;

[0010] According to the target adjustment strategy, the current congestion window is adjusted.

[0011] Optionally, the first reward value is positively correlated with the receiving rate of the current acquisition cycle, and negatively correlated with the rate difference of the current acquisition cycle; the rate difference of the current acquisition cycle represents the difference between the sending rate of the current acquisition cycle and the receiving rate of the current acquisition cycle.

[0012] Optionally, the first network status data further includes at least one of the following:

[0013] The minimum round-trip time of the current collection cycle means: in the current collection cycle, when each specified time is reached, the minimum value of the round-trip time of each specified time that has been collected;

[0014] The average round-trip time of the current collection cycle means: the average round-trip time collected at each specified time in the current collection cycle;

[0015] The average latency of the current collection period is: the average value of the latency collected at each specified time in the current collection period;

[0016] The average congestion window size of the current collection period is: the average value of the congestion window sizes collected at each specified time in the current collection period;

[0017] The average in-flight data size of the current collection period indicates: the average value of the in-flight data size collected at each specified time in the current collection period; the in-flight data size collected at a specified time indicates: the size of the data packet that has been sent at the specified time and has not received the corresponding ACK message;

[0018] The size of data sent during the current collection cycle;

[0019] The size of the data packet in response to the ACK message received during the current collection cycle;

[0020] The size of the data packets lost in the current collection cycle;

[0021] The number of ACK messages received during the current collection period that indicate congestion signals.

[0022] Optionally, the specified time also includes a second time at which a packet loss event is detected;

[0023] The round trip time collected at each second moment represents: the round trip time collected when the last ACK message was received before the second moment;

[0024] The delay collected at each second moment represents: the delay collected when the last ACK message was received before the second moment;

[0025] The sending rate collected at each second moment represents: the sending rate collected when the last ACK message was received before the second moment;

[0026] The receiving rate collected at each second moment represents: the receiving rate collected when the last ACK message was received before the second moment.

[0027] Optionally, the first reward value is negatively correlated with the average delay of the current acquisition cycle.

[0028] Optionally, the calculating, based on the first network status data, a reward value of the current collection cycle as a first reward value includes:

[0029] Determine whether the average delay of the current collection period is less than a first threshold;

[0030] If yes, then determining the first reward value of the current acquisition period as the receiving rate of the current acquisition period;

[0031] If not, the reward value of the current collection cycle is calculated as the first reward value based on the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle.

[0032] Optionally, the first threshold is positively correlated with the minimum round-trip time of the current acquisition cycle.

[0033] Optionally, the first threshold is calculated based on a first formula;

[0034] The first formula is:

[0035] S=εMinRtt+ρ

[0036] Wherein, S represents the first threshold, MinRtt represents the minimum round trip time of the current acquisition cycle, ε represents the first preset parameter, and ρ represents the second preset parameter;

[0037] The reward value of the current collection cycle is calculated as the first reward value based on the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle, including:

[0038] Based on the sending rate of the current collection cycle, the sending rate of the current collection cycle of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle, the reward value of the current collection cycle is calculated according to the second formula as the first reward value; wherein the second formula is:

[0039]

[0040] Among them, reward represents the first reward value, AR represents the receiving rate of the current collection cycle, D represents the average delay of the current collection cycle, MinRtt represents the minimum round-trip time of the current collection cycle, SR represents the sending rate of the current collection cycle, and δ represents the third preset parameter.

[0041] Optionally, the sending rate of the current collection cycle is the sending rate collected at the last specified time in the current collection cycle; the receiving rate of the current collection cycle is the receiving rate collected at the last specified time in the current collection cycle.

[0042] Optionally, the target adjustment strategy includes two or more first specified adjustment multiples greater than 1, two or more second specified adjustment multiples that are reciprocals of the two or more first specified adjustment multiples, and a probability corresponding to each specified adjustment multiple;

[0043] The adjusting the current congestion window according to the target adjustment strategy includes:

[0044] The current congestion window is adjusted according to the specified adjustment multiple with the highest corresponding probability.

[0045] Optionally, the method further includes:

[0046] When the preset adjustment period is reached, data packets are sent at a rate lower than the current receiving rate within a first time period.

[0047] Optionally, before inputting the first network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy, the method further includes:

[0048] Acquire network status data of a preset number of historical cycles before the current collection cycle as second network status data;

[0049] The step of inputting the first network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy includes:

[0050] The first network state data, the second network state data and the first reward value are input into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy.

[0051] Optionally, the training process of the adjustment strategy prediction network model includes the following steps:

[0052] Obtaining network status data of a sample period as sample network status data;

[0053] Wherein, the sample network status data includes: the sending rate and receiving rate of the sample period; the sending rate of the sample period is determined based on the sending rate collected at each specified time in the sample period; the receiving rate of the sample period is determined based on the receiving rate collected at each specified time in the sample period; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last received ACK message and the sending time of the data packet corresponding to the first time;

[0054] Based on the sample network status data, a second reward value of the sample period is calculated; wherein the second reward value is positively correlated with the receiving rate of the sample period, and negatively correlated with the rate difference of the sample period; the rate difference represents the difference between the sending rate of the sample period and the receiving rate of the sample period;

[0055] Input the sample network state data and the second reward value into the adjustment strategy prediction network model of the initial parameters to obtain a sample adjustment strategy and a strategy score value;

[0056] Adjust the current congestion window according to the sample adjustment strategy;

[0057] Based on the strategy score value and the second reward value, the model parameters of the initial parameter adjustment strategy prediction network model are adjusted until a convergence condition is reached.

[0058] According to a second aspect of an embodiment of the present application, a congestion control device is provided, the device comprising:

[0059] A first network status collection module, used for acquiring network status data of a current collection period as first network status data when a preset collection period is reached;

[0060] Wherein, the first network status data includes the sending rate and receiving rate of the current collection cycle; the sending rate of the current collection cycle is determined based on the sending rate collected at each specified time in the current collection cycle; the receiving rate of the current collection cycle is determined based on the receiving rate collected at each specified time in the current collection cycle; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last received ACK message and the sending time of the data packet corresponding to the first time;

[0061] A first reward value calculation module, used to calculate the reward value of the current collection cycle based on the first network status data as a first reward value;

[0062] A target adjustment strategy acquisition module, used to input the first network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy; wherein the adjustment strategy prediction network model is obtained by training based on a reinforcement learning algorithm;

[0063] The congestion window adjustment module is used to adjust the current congestion window according to the target adjustment strategy.

[0064] Optionally, the first reward value is positively correlated with the receiving rate of the current acquisition cycle, and negatively correlated with the rate difference of the current acquisition cycle; the rate difference of the current acquisition cycle represents the difference between the sending rate of the current acquisition cycle and the receiving rate of the current acquisition cycle.

[0065] Optionally, the first network status data further includes at least one of the following:

[0066] The minimum round-trip time of the current collection cycle means: in the current collection cycle, when each specified time is reached, the minimum value of the round-trip time of each specified time that has been collected;

[0067] The average round-trip time of the current collection cycle means: the average round-trip time collected at each specified time in the current collection cycle;

[0068] The average latency of the current collection period is: the average value of the latency collected at each specified time in the current collection period;

[0069] The average congestion window size of the current collection period is: the average value of the congestion window sizes collected at each specified time in the current collection period;

[0070] The average in-flight data size of the current collection period indicates: the average value of the in-flight data size collected at each specified time in the current collection period; the in-flight data size collected at a specified time indicates: the size of the data packet that has been sent at the specified time and has not received the corresponding ACK message;

[0071] The size of data sent during the current collection cycle;

[0072] The size of the data packet in response to the ACK message received during the current collection cycle;

[0073] The size of the data packets lost in the current collection cycle;

[0074] The number of ACK messages received during the current collection period that indicate congestion signals.

[0075] Optionally, the specified time also includes a second time at which a packet loss event is detected;

[0076] The round trip time collected at each second moment represents: the round trip time collected when the last ACK message was received before the second moment;

[0077] The delay collected at each second moment represents: the delay collected when the last ACK message was received before the second moment;

[0078] The sending rate collected at each second moment represents: the sending rate collected when the last ACK message was received before the second moment;

[0079] The receiving rate collected at each second moment represents: the receiving rate collected when the last ACK message was received before the second moment.

[0080] Optionally, the first reward value is negatively correlated with the average delay of the current acquisition cycle.

[0081] Optionally, the first reward value calculation module includes:

[0082] A first threshold judgment submodule, used to judge whether the average delay of the current acquisition cycle is less than a first threshold, and if so, trigger the first reward value calculation submodule; if not, trigger the second reward value calculation submodule;

[0083] A first reward value calculation submodule, used to determine the first reward value of the current acquisition cycle as the receiving rate of the current acquisition cycle;

[0084] The second reward value calculation submodule is used to calculate the reward value of the current collection cycle as the first reward value based on the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle.

[0085] Optionally, the first threshold is positively correlated with the minimum round-trip time of the current acquisition cycle.

[0086] Optionally, the first threshold is calculated based on a first formula;

[0087] The first formula is:

[0088] S=εMinRtt+ρ

[0089] Wherein, S represents the first threshold, MinRtt represents the minimum round trip time of the current acquisition cycle, ε represents the first preset parameter, and ρ represents the second preset parameter;

[0090] The second reward value calculation submodule is used to calculate the reward value of the current collection cycle as the first reward value according to the second formula based on the sending rate of the current collection cycle, the sending rate of the current collection cycle of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle; wherein the second formula is:

[0091]

[0092] Among them, reward represents the first reward value, AR represents the receiving rate of the current collection cycle, D represents the average delay of the current collection cycle, MinRtt represents the minimum round-trip time of the current collection cycle, SR represents the sending rate of the current collection cycle, and δ represents the third preset parameter.

[0093] Optionally, the sending rate of the current collection cycle is the sending rate collected at the last specified time in the current collection cycle; the receiving rate of the current collection cycle is the receiving rate collected at the last specified time in the current collection cycle.

[0094] Optionally, the target adjustment strategy includes two or more first specified adjustment multiples greater than 1, two or more second specified adjustment multiples that are reciprocals of the two or more first specified adjustment multiples, and a probability corresponding to each specified adjustment multiple;

[0095] The congestion window adjustment module is specifically used to adjust the current congestion window according to the designated adjustment multiple with the maximum corresponding probability.

[0096] Optionally, the device further comprises:

[0097] The sending rate adjustment module is used to send data packets at a rate lower than the current receiving rate within a first time period when a preset adjustment period is reached.

[0098] Optionally, the device further comprises:

[0099] A second network status acquisition module is used to obtain network status data of a preset number of historical cycles before the current collection cycle as second network status data before inputting the first network status data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy;

[0100] The target adjustment strategy acquisition module is used to input the first network status data, the second network status data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy.

[0101] Optionally, the device further comprises:

[0102] A training module, used for obtaining network status data of a sample period as sample network status data;

[0103] Wherein, the sample network status data includes: the sending rate and receiving rate of the sample period; the sending rate of the sample period is determined based on the sending rate collected at each specified time in the sample period; the receiving rate of the sample period is determined based on the receiving rate collected at each specified time in the sample period; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last received ACK message and the sending time of the data packet corresponding to the first time;

[0104] Based on the sample network status data, a second reward value of the sample period is calculated; wherein the second reward value is positively correlated with the receiving rate of the sample period, and negatively correlated with the rate difference of the sample period; the rate difference represents the difference between the sending rate of the sample period and the receiving rate of the sample period;

[0105] Input the sample network state data and the second reward value into the adjustment strategy prediction network model of the initial parameters to obtain a sample adjustment strategy and a strategy score value;

[0106] Adjust the current congestion window according to the sample adjustment strategy;

[0107] Based on the strategy score value and the second reward value, the model parameters of the initial parameter adjustment strategy prediction network model are adjusted until a convergence condition is reached.

[0108] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0109] Memory, used to store computer programs;

[0110] The processor is used to implement any of the above-mentioned method steps when executing the program stored in the memory.

[0111] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the method steps described above is implemented.

[0112] A congestion control method provided by an embodiment of the present application can obtain the sending rate and receiving rate of the current acquisition cycle for different network environments, and the sending rate and receiving rate can effectively reflect the current network status. Therefore, congestion control can be effectively implemented based on the information obtained. Since there is no need to distinguish whether the congestion signal is caused by network congestion, as long as the sending rate and receiving rate of the current acquisition cycle are obtained, the congestion window can be adjusted. Therefore, the congestion control method of the present application can be applied to complex network environments and can improve the effectiveness of congestion control. In addition, since the reward value is related to the sending rate and receiving rate of the current acquisition cycle, the receiving rate is made equal to the sending rate as much as possible through the reward mechanism. Therefore, adjusting the congestion window according to the target adjustment strategy obtained based on the above-mentioned reward value can also obtain a larger bandwidth utilization rate, and can avoid the over-issuance problem caused by the sending rate being greater than the receiving rate.

[0113] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0115] Figure 1 A flow chart of a congestion control method provided in an embodiment of the present application;

[0116] Figure 2 A schematic diagram of calculating the sending rate and receiving rate at a specified time provided in an embodiment of the present application;

[0117] Figure 3 A schematic diagram of a congestion control method provided in an embodiment of the present application;

[0118] Figure 4 A training flow chart of an adjustment strategy prediction network model in a congestion control method provided in an embodiment of the present application;

[0119] Figure 5 A schematic diagram of generating an adjustment strategy based on an adjustment strategy prediction network model provided in an embodiment of the present application;

[0120] Figure 6 A schematic diagram of the structure of a congestion control device provided in an embodiment of the present application;

[0121] Figure 7 Another structural schematic diagram of the congestion control device provided in an embodiment of the present application;

[0122] Figure 8 Another structural schematic diagram of the congestion control device provided in an embodiment of the present application;

[0123] Fig. 9 Another structural schematic diagram of the congestion control device provided in an embodiment of the present application;

[0124] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0125] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0126] In order to be applicable to complex network environments and improve the effectiveness of congestion control, the present application embodiment provides a congestion control method, see Figure 1 , Figure 1 A flow chart of a congestion control method provided in an embodiment of the present application, the method may include the following steps:

[0127] Step S101: when a preset collection period is reached, network status data of the current collection period is obtained as first network status data.

[0128] Among them, the first network status data includes the sending rate and receiving rate of the current collection cycle; the sending rate of the current collection cycle is determined based on the sending rate collected at each specified time in the current collection cycle; the receiving rate of the current collection cycle is determined based on the receiving rate collected at each specified time in the current collection cycle; the specified time includes the first time when an ACK (Acknowledgement) message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded to by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded to by the last ACK message received before the data packet corresponding to the first time is sent and the sending time of the data packet corresponding to the first time.

[0129] Step S102: Based on the first network status data, a reward value of the current collection period is calculated as a first reward value.

[0130] Step S103: input the first network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy.

[0131] Among them, the adjustment strategy prediction network model is obtained by training based on the reinforcement learning algorithm.

[0132] Step S104: adjusting the current congestion window according to the target adjustment strategy.

[0133] By applying the congestion control method provided by the embodiment of the present application, the sending rate and receiving rate of the current acquisition cycle can be obtained for different network environments, and the sending rate and receiving rate can effectively reflect the current network state. Therefore, congestion control can be effectively implemented based on the information obtained. Since there is no need to distinguish whether the congestion signal is caused by network congestion, as long as the sending rate and receiving rate of the current acquisition cycle are obtained, the congestion window can be adjusted. Therefore, the congestion control method of the present application can be applied to complex network environments and can improve the effectiveness of congestion control. In addition, since the reward value is related to the sending rate and receiving rate of the current acquisition cycle, the receiving rate is equal to the sending rate as much as possible through the reward mechanism. Therefore, according to the target adjustment strategy obtained based on the above-mentioned reward value, the congestion window is adjusted, a larger bandwidth utilization rate can be obtained, and the over-issuance problem caused by the sending rate being greater than the receiving rate can be avoided.

[0134] For step S101, the cycle duration of the preset collection cycle can be a fixed duration. Alternatively, the cycle duration of the preset collection cycle can also be determined based on the RTT (Round Trip Time) of the current data packet. For example, at the end of a collection cycle, the RTT of the data packet at the current time can be determined, and the cycle duration of the next collection cycle is determined as the RTT.

[0135] In this application, an event occurring at a specified time may be referred to as a specified event. For example, a specified event includes an event of receiving an ACK message. In each acquisition cycle, when a specified event occurs, the current network status data may be obtained, including the current sending rate and receiving rate, that is, the sending rate and receiving rate collected at the current specified time. The network status data of a collection cycle is determined based on the network status data collected at each specified time in the collection cycle.

[0136] The sending rate of the current collection cycle is determined based on the sending rates collected at each specified time in the current collection cycle. For example, the sending rate of the current collection cycle can be the sending rate collected at the last specified time in the current collection cycle, or the sending rate collected at other specified times in the current collection cycle, or the average of the sending rates collected at each specified time in the current collection cycle.

[0137] The receiving rate of the current collection period is determined based on the receiving rates collected at each specified time in the current collection period. For example, the receiving rate of the current collection period can be the receiving rate collected at the last specified time in the current collection period, or the receiving rate collected at other specified times in the current collection period, or the average value of the sending rates collected at each specified time in the current collection period.

[0138] In network communication, the sender can send a data packet to the receiver, and then, after receiving the data packet, the receiver can send an ACK message to the sender in response to the received data packet. When the sender receives the ACK message, it can also determine the data packet to which the ACK message responds. A data packet corresponding to a first moment indicates the data packet to which the ACK message received at the first moment responds, that is, an ACK message received at the first moment is used to respond to the data packet corresponding to the first moment.

[0139] When the transmitting end sends a data packet, the transmitting time of the data packet can be recorded. When the transmitting end receives an ACK message sent by the receiving end, the corresponding receiving time can also be recorded. The receiving rate at the first moment can represent the rate of receiving data in a historical time period before the first moment (which can be called the first time period). Correspondingly, the sending rate at the first moment can represent the rate of sending data in another historical time period (which can be called the second time period). The ACK message received in the first time period corresponds to the data packet sent in the second time period.

[0140] See also Figure 2 , Figure 2 A schematic diagram of calculating the sending rate and receiving rate at a specified time provided by an embodiment of the present application. Wherein, the horizontal axis represents time, the black rectangular box represents the sent data packet, and the white rectangular box represents the received ACK message. The time when data packet A is sent is T1, the time when ACK message A1 is received is T2, the time when data packet B is sent is T3, and the time when ACK message B1 is received is T4. The data packet responded by ACK message B1 is data packet B, the last ACK message received before sending data packet B is ACK message A1, and the data packet responded by ACK message A1 is data packet A.

[0141] When the first moment is T4, it can be seen that the sending rate at T4 is the ratio of the size of the data packet sent between T3 and T1 to the time interval between T3 and T1. The receiving rate at T4 is the ratio of the size of the data packet received in response to the ACK message between T4 and T2 to the time interval between T4 and T2.

[0142] The sending rate at time T4 can be calculated based on formula (1):

[0143]

[0144] Among them, send_rate represents the sending rate at time T4, send represents the size of the data packet sent between time T3 and time T1, and T3-T1 represents the time interval between time T3 and time T1.

[0145] The receiving rate at time T4 can be calculated based on formula (2):

[0146]

[0147] Among them, acked_rate represents the receiving rate at time T4, acked represents the size of the data packet responded to the ACK message received between time T4 and time T2, and T4-T2 represents the time interval between time T4 and time T2.

[0148] With respect to step S102, in one embodiment, the first reward value is positively correlated with the receiving rate of the current acquisition cycle, and negatively correlated with the rate difference of the current acquisition cycle; the rate difference of the current acquisition cycle represents the difference between the sending rate of the current acquisition cycle and the receiving rate of the current acquisition cycle. Since the first reward value is positively correlated with the receiving rate of the current acquisition cycle, that is, the higher the receiving rate of the current acquisition cycle, the larger the first reward value, therefore, in order to obtain a larger reward value, the target adjustment strategy formed by the adjustment strategy prediction network model will promote the increase of the receiving rate. The first reward value is negatively correlated with the rate difference of the current acquisition cycle, and the rate difference of the current acquisition cycle represents the difference between the sending rate of the current acquisition cycle and the receiving rate of the current acquisition cycle. That is, the smaller the difference between the sending rate of the current acquisition cycle and the receiving rate of the current acquisition cycle, the larger the first reward value, therefore, in order to obtain a larger reward value, the target adjustment strategy formed by the adjustment strategy prediction network model will reduce the rate difference of the current acquisition cycle, try to make the receiving rate equal to the sending rate, and then obtain a larger bandwidth utilization without the problem of over-issuance.

[0149] For step S103 and step S104, the first network state data includes network state data characterizing the current network state, such as the sending rate of the current collection cycle and the receiving rate of the current collection cycle. The adjustment strategy prediction network model can perceive the network state of the current collection cycle based on the first network state data, and then obtain the target adjustment strategy in combination with the reward value. The target adjustment strategy includes an adjustment action for adjusting the current congestion window, and the probability corresponding to each adjustment action. For example, the adjustment action can represent increasing the current congestion window, such as increasing the preset multiple or increasing the fixed value to increase the current congestion window. The adjustment action can also represent reducing the current congestion window, such as reducing the preset multiple or reducing the fixed value to reduce the current congestion window. Alternatively, the adjustment action can also represent keeping the size of the congestion window unchanged. Furthermore, according to the probability corresponding to each adjustment action, the adjustment action with a large probability can be selected to adjust the current congestion window size, and accordingly, data packets can be sent according to the adjusted congestion window to achieve congestion control.

[0150] In one embodiment, the sending rate of the current collection cycle is the sending rate collected at the last specified time in the current collection cycle; the receiving rate of the current collection cycle is the receiving rate collected at the last specified time in the current collection cycle. The sending rate and receiving rate collected at the last specified time in the current collection cycle can effectively reflect the current network status. Therefore, the sending rate and receiving rate collected at the last specified time in the current collection cycle are used as the sending rate and receiving rate of the current collection cycle to adjust the congestion window, which can improve the effectiveness of the adjustment.

[0151] In one embodiment, the first network status data may further include at least one of the following:

[0152] The minimum round-trip time of the current collection cycle means: in the current collection cycle, when each specified time is reached, the minimum value of the round-trip time of each specified time that has been collected.

[0153] The average round-trip time of the current collection cycle means: the average round-trip time collected at each specified time in the current collection cycle.

[0154] The average latency of the current collection period is the average value of the latency collected at each specified time in the current collection period. The latency collected at a specified time can be the difference between the round-trip time collected at the specified time and the minimum round-trip time collected at the specified time.

[0155] The average congestion window size in the current collection period is the average value of the congestion window sizes collected at each specified time in the current collection period.

[0156] The average in-flight data size of the current collection cycle represents the average value of the in-flight data size collected at each specified time in the current collection cycle; the in-flight data size collected at a specified time represents: the size of the data packet that has been sent at the specified time and has not yet received the corresponding ACK message.

[0157] The size of the data sent during the current collection cycle.

[0158] The size of the data packet responded to the ACK message received during the current collection period.

[0159] The size of the data packets lost in the current collection period.

[0160] The number of ACK messages received during the current collection period that indicate congestion signals.

[0161] In one example, the minimum round trip time in each collection cycle can be recorded, and the minimum round trip time can be updated in the collection cycle. In the collection cycle, when each specified moment is reached, the round trip time at the specified moment is collected. If the round trip time collected at the specified moment is less than the minimum round trip time of the current collection cycle currently recorded, the minimum round trip time of the current collection cycle is updated to the round trip time collected at the specified moment, otherwise, the minimum round trip time of the current collection cycle is not updated. That is, the minimum round trip time of the current collection cycle represents the minimum value of the round trip time of each specified moment collected when each specified moment is reached in the current collection cycle.

[0162] When the sender sends a data packet, the sending time of the data packet can be recorded. When the sender receives an ACK message corresponding to the data packet sent by the receiver, the corresponding receiving time can also be recorded. Based on the receiving time of the ACK message and the sending time of the corresponding data packet, the round trip time can be calculated.

[0163] For each designated moment in the current acquisition cycle, the size of the currently sent data packet and the size of the data packet to which the currently received ACK message responds can be recorded at the designated moment. Further, the size of the in-flight data at the designated moment can be calculated based on the recorded data. For example, the difference between the size of the currently sent data packet recorded at the designated moment and the size of the data packet to which the currently received ACK message responds can be calculated as the in-flight data size.

[0164] For each designated moment in the current collection cycle, the size of the currently sent data packet can be recorded at the designated moment. Then, the size of the data packet sent at the last designated moment of the collection cycle and the size of the data packet sent at the last designated moment of the previous collection cycle can be obtained, and then the difference between the two values ​​obtained can be calculated to obtain the size of the data sent by the connection in the collection cycle.

[0165] For each designated moment in the current collection cycle, the size of the data packet in response to the currently received ACK message can be recorded at the designated moment. Further, the size of the data packet in response to the received ACK message at the last designated moment of the collection cycle and the size of the data packet in response to the received ACK message at the last designated moment of the previous collection cycle can be obtained, and then the difference between the two values ​​obtained is calculated to obtain the size of the data packet in response to the ACK message received during the collection cycle.

[0166] Due to network congestion and other reasons, data packet loss may occur. Therefore, the size of the lost data packet can also reflect the network status during the collection cycle. For each specified time in the current collection cycle, the size of the currently lost data packet can be recorded at the specified time. Then, the size of the data packet lost at the last specified time of the collection cycle and the size of the data packet lost at the last specified time of the previous collection cycle can be obtained. Then, the difference between the two values ​​obtained is calculated to obtain the size of the data packet lost during the collection cycle.

[0167] When congestion occurs in the network, routers that support ECN (Explicit Congestion Notification) can set a mark in the data packet, that is, the explicit congestion signal. After the receiving end receives the data packet with the explicit congestion signal added, it sends the corresponding ACK message carrying the explicit congestion signal to the sending end. Therefore, the number of explicit congestion signals in the ACK message received within a collection cycle can also reflect the network status within the collection cycle.

[0168] The first network status data may include at least one of the above-mentioned data that can describe the network status of the current collection cycle. These data have clear physical meanings, making the first network status data more meaningful for describing the network status. Based on the first network status data, the network status of the current collection cycle can be better perceived, and the network congestion situation in the current collection cycle can be clearly known, thereby improving the adaptability of the congestion control method provided in the embodiment of the present application to the network environment and obtaining better control effects.

[0169] In one embodiment, the specified time also includes a second time at which a packet loss event is detected.

[0170] The round trip time collected at each second moment represents: the round trip time collected when the last ACK message was received before the second moment.

[0171] The delay collected at each second moment represents: the delay collected when the last ACK message was received before the second moment.

[0172] The sending rate collected at each second moment represents: the sending rate collected when the last ACK message was received before the second moment.

[0173] The receiving rate collected at each second moment represents: the receiving rate collected when the last ACK message was received before the second moment.

[0174] Since the sender does not receive an ACK message when a packet loss event occurs, and the round-trip time, latency, sending rate, and receiving rate need to be obtained based on the ACK message. Therefore, these network status data cannot be collected when a packet loss event is detected. In order to describe the network status when a packet loss event occurs, the network status data corresponding to the packet loss event that cannot be collected can be determined based on the relevant data collected when the last ACK message received before the packet loss event is detected.

[0175] The minimum round-trip time, round-trip time, delay, congestion window size, in-flight data size, and sent data size collected at the second moment are similar to the way these data are collected at the first moment.

[0176] If an ACK message corresponding to a data packet is not received for a certain period of time, it is considered that a packet loss event has occurred. When a packet loss event occurs, it is very likely that network congestion has occurred. In other words, the network status data when the packet loss event occurs can also effectively reflect the state of the network. Therefore, congestion control can also be performed based on the network status data when the packet loss event occurs.

[0177] In one embodiment, the first reward value is negatively correlated with the average delay of the current collection cycle. The first reward value decreases as the average delay of the current collection cycle increases, that is, the average delay of the current collection cycle can be penalized by the first reward value to avoid the increase of delay, thereby avoiding the occurrence of bufferbloat problems and achieving better congestion control effects.

[0178] In one implementation, the first reward value may be calculated according to formula (3):

[0179] reward=α×acked_rate-β×(send_rate-acked_rate)-γ×delay (3)

[0180] Among them, reward represents the first reward value, acked_rate represents the receiving rate of the current acquisition cycle, send_rate represents the sending rate of the current acquisition cycle, delay represents the average delay of the current acquisition cycle, and α, β, and γ represent preset coefficients. For example, α and β can be 1 or 2, and γ can be 0.05 or 0.04.

[0181] In one embodiment, based on the first network status data, calculating the reward value of the current collection cycle as the first reward value (S102) includes:

[0182] Step 1: Determine whether the average delay of the current collection cycle is less than the first threshold; if yes, execute step 2; if no, execute step 3.

[0183] Step 2: Determine the reward value of the current collection period as the receiving rate of the current collection period.

[0184] Step 3: Calculate the reward value of the current collection cycle based on the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle.

[0185] If the average delay of the current collection period is less than the first threshold, the reward value of the current collection period is determined as the receiving rate of the current collection period, and the greater the receiving rate of the current collection period, the greater the reward value. Thus, through this reward mechanism, the receiving rate and the sending rate are made equal as much as possible to obtain a greater bandwidth utilization.

[0186] If the average delay of the current collection cycle is not less than the first threshold, the reward value of the current collection cycle is calculated based on the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle. Since the reward value is related to the sending rate and receiving rate of the current collection cycle, the receiving rate is made equal to the sending rate as much as possible through the reward mechanism to obtain a higher bandwidth utilization.

[0187] Based on the above processing, if the first threshold is set to a smaller value, the average delay of the current collection cycle can be controlled lower, thereby reducing data queuing; if the first threshold is set to a larger value, in order to obtain a larger reward value, the average delay of the current collection cycle can still be controlled within a larger range, thereby increasing the amount of data sent, thereby improving the ability to occupy the cache.

[0188] For example, the first threshold may be a preset fixed value.

[0189] Alternatively, the first threshold value may also be positively correlated with the minimum round trip time of the current collection cycle. Accordingly, when the minimum round trip time of the current collection cycle is large, the first threshold value is also large. At this time, rewards can still be given for larger average delays, which promotes an increase in the sending rate and the size of in-flight data, ensuring that more data packets are in the queue and improving the ability to occupy the cache. When the minimum round trip time of the current collection cycle is small, the first threshold value is also small, and the average delay of the current collection cycle can be controlled lower, thereby reducing data queuing.

[0190] For example, the first threshold is calculated based on formula (4):

[0191] S=εMinRtt+ρ (4)

[0192] Wherein, S represents the first threshold, MinRtt represents the minimum round trip time of the current acquisition cycle, ε represents the first preset parameter, and ρ represents the second preset parameter.

[0193] Step three includes:

[0194] Based on the sending rate of the current collection cycle, the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle, the reward value of the current collection cycle is calculated according to formula (5) as the first reward value:

[0195]

[0196] Among them, reward represents the first reward value, AR represents the receiving rate of the current collection cycle, D represents the average delay of the current collection cycle, MinRtt represents the minimum round-trip time of the current collection cycle, SR represents the sending rate of the current collection cycle, and δ represents the third preset parameter.

[0197] When ε and ρ are large, that is, the first threshold is set to a large value, then rewards can still be given for larger average delays, which promotes an increase in the sending rate and the size of in-flight data, ensuring that more data packets are in the queue and improving the ability to occupy the cache. When ε and ρ are small, that is, the first threshold is set to a small value, the average delay of the current collection cycle can be controlled to be lower, thereby reducing data queuing.

[0198] In the prior art, the reward value of congestion control based on reinforcement learning is calculated based on a linear combination of throughput, average delay, and size of lost data packets over a period of time, as shown in formula (6):

[0199] reward=α×T+β×D+γ×L (6)

[0200] Among them, reward represents the reward value of the period, T represents the throughput of the period, D represents the average delay of the period, L represents the size of the data packet lost during the period, and α, β, and γ represent the preset coefficients. However, in the above method of calculating the reward value, there is only a simple linear relationship between the reward value and the throughput, average delay, and size of the lost data packet within a period of time. Therefore, the reward value calculation method cannot be adjusted by coefficients to make it applicable to all network environments, resulting in a relatively limited network environment for which this method is applicable.

[0201] In the reward value calculation method provided in the embodiment of the present application, the relationship between the reward value and the network status data is nonlinear. Therefore, by adjusting the preset parameters, the calculated reward value can be applied to different network environments.

[0202] In one embodiment, the target adjustment strategy includes two or more first specified adjustment multiples greater than 1, two or more second specified adjustment multiples that are reciprocal of the two or more first specified adjustment multiples, and a probability corresponding to each specified adjustment multiple. Accordingly, according to the target adjustment strategy, the current congestion window is adjusted (S104), including:

[0203] Step S1041: adjusting the current congestion window according to the designated adjustment multiple with the highest corresponding probability.

[0204] The two or more specified adjustment multiples in the target adjustment strategy correspond to adjustment actions of different magnitudes to adjust the size of the congestion window. For example, the first specified adjustment multiple may include 2.89, 1.25, and 1.05, which respectively indicate that the congestion window size is adjusted to 2.89 times, 1.25 times, and 1.05 times of the current congestion window. Correspondingly, the second specified adjustment multiple includes: 1 / 2.89, 1 / 1.25, and 1 / 1.05, which respectively indicate that the congestion window size is adjusted to 1 / 2.89, 1 / 1.25, and 1 / 1.05 of the current congestion window. Each specified adjustment multiple has its corresponding probability, and the congestion window is adjusted according to the specified adjustment multiple with the largest corresponding probability.

[0205] In addition, in the prior art, the adjustment actions include: keeping the congestion window size unchanged, adjusting the congestion window size to 1 / 2 of the current congestion window, adjusting the congestion window size to the current congestion window minus 10, adjusting the congestion window size to the current congestion window plus 10, and adjusting the congestion window size to twice the current congestion window. For two networks with different bandwidths, the determined adjustment actions may also be the same, and the same adjustment action has different effects on the two different networks. In addition, in this method, based on different adjustment actions, the same adjusted congestion window size may be obtained. For example, the congestion window size before adjustment is 10, the congestion window size after adding 10 to the current congestion window is 20, and the congestion window size after doubling the current congestion window is also 20, which are the same. In other words, even if different adjustment actions are determined based on different network states, the same adjustment result will be obtained.

[0206] In the embodiments provided in the present application, the current congestion window size can be adjusted according to the specified adjustment multiple, which can avoid the problem that different adjustment actions obtain the same adjustment results. In addition, if the specified adjustment multiple is not 1, the congestion window size can be continuously kept in a changing state, which can also promote the change of the data transmission rate, and then, the impact of the rate change on the network state can be perceived, thereby improving the effectiveness of congestion control. Selecting a larger first specified adjustment multiple (for example, 2.89) can make the data transmission rate rise rapidly, and correspondingly, selecting a second specified adjustment multiple (for example, 1 / 2.89) corresponding to the first specified adjustment multiple can make the data transmission rate drop rapidly, so that the link is quickly emptied. By adjusting the size of the congestion window by a variety of specified adjustment multiples, the changing network environment can be detected, so that the congestion control method provided in the embodiment of the present application has a rapid adaptability to changes in the network environment.

[0207] In one embodiment, the method may further include: when a preset adjustment period is reached, sending data packets at a rate lower than a current receiving rate within a first time period.

[0208] The preset adjustment period may also be referred to as the RTT detection phase. For example, when the preset adjustment period is reached, data packets are sent at a rate represented by a preset multiple of the current receiving rate within the first duration, and the preset multiple is less than 1. For example, the period duration of the preset adjustment period may be 10S or 11S, the first duration may be 190ms or 200ms, and the preset multiple may be 0.5 times or 0.6 times. The settings of the preset adjustment period, the first duration, and the preset multiple may be adjusted according to the actual application situation without specific limitation.

[0209] The current receiving rate may be the receiving rate collected when the ACK message was last received before the current moment.

[0210] Since the network status may change, the current actual minimum round trip time may be much greater than the minimum round trip time recorded in the current collection period. Therefore, in order to make the recorded minimum round trip time more accurate, when the preset adjustment period is reached, the data packet is directly sent at a rate less than the current receiving rate, and the first duration is maintained to reduce the size of the in-flight data and drain the link, which can reduce the actual minimum round trip time of the network and make the currently recorded minimum round trip time more accurate, so that the congestion control method provided in the embodiment of the present application can adapt to the dynamic delay link.

[0211] In addition, when the first duration is reached, data can continue to be sent according to the size of the congestion window before adjustment.

[0212] In one embodiment, see Figure 3 , Figure 3 It is a principle schematic diagram of the congestion control method provided in an embodiment of the present application. The sending end and the receiving end send and receive data, and the sending end implements asynchronous congestion control through the agent decision module and the data sending module. The agent decision module is RL Agent (Reinforcement Learning Agent). The data sending module includes Event State Sampler and Round Trip State Sampler. The event state sampler collects network state data when an ACK message and a packet loss event are detected. When it is determined through the timing module that the preset collection period has been reached, the round-trip state sampler can integrate the network state data collected by the event state sampler at each specified time in the current collection period to obtain the first network state data of the current collection period.

[0213] The data sending module calculates a first reward value based on the first network state data, and transmits the first network state data and the first reward value to the agent decision module. For example, the data sending module may further include a calculation submodule, to which the round-trip state sampler may send the first network state data, and correspondingly, the calculation submodule may calculate a first reward value based on the first network state data, and transmit the first network state data and the first reward value to the agent decision module.

[0214] The agent decision module obtains a target adjustment strategy based on the first network status data and the first reward value, selects a specified adjustment multiple (i.e., adjustment action) from the target adjustment strategy, and sends the adjustment action to the data sending module. The data sending module adjusts the size of the congestion window according to the adjustment action. For example, the agent decision module can be based on Figure 5 The adjustment strategy prediction network model shown processes the first network state data and the first reward value to obtain a target adjustment strategy.

[0215] When the RTT detection phase is reached, that is, when the preset adjustment period is reached, the data sending module sends the data packet at a rate lower than the current receiving rate within the first duration. For example, the data sending module may also include a sending adjustment submodule, that is, when the preset adjustment period is reached, the sending adjustment submodule may send the data packet at a rate lower than the current receiving rate within the first duration.

[0216] Since the first reward value is calculated based on the sending rate, receiving rate and delay, this method makes the reward value calculation method clear and can effectively guide the agent decision module (i.e., the reinforcement learning agent) to converge and make good control decisions.

[0217] In one embodiment, before inputting the first network status data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy (S103), it also includes: obtaining network status data of a preset number of historical cycles before the current collection cycle as the second network status data.

[0218] Accordingly, the first network state data and the first reward value are input into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy (S103), including:

[0219] Step S1031: input the first network state data, the second network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy.

[0220] The network status data of a preset number of historical cycles can reflect the network status at a historical moment. Then, combining the network status data of a preset number of historical cycles with the network status data of the current collection cycle can also reflect the changes in the network status within a certain period of time. Then, based on the above data, the adjustment strategy prediction network model can make a more reasonable adjustment strategy selection, determine an adjustment strategy that is more suitable for the current network status, and thus obtain a better congestion control effect.

[0221] Figure 4 This is a training flow chart of the adjustment strategy prediction network model in the congestion control method provided in the embodiment of the present application, see Figure 4 ,The training process of adjusting the strategy prediction network model includes the following steps:

[0222] Step S401: Acquire network status data of a sample period as sample network status data.

[0223] Among them, the sample network status data includes: the sending rate and receiving rate of the sample period; the sending rate of the sample period is determined based on the sending rate collected at each specified time within the sample period; the receiving rate of the sample period is determined based on the receiving rate collected at each specified time within the sample period; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last received ACK message and the sending time of the data packet corresponding to the first time.

[0224] Step S402: Calculate a second reward value of a sample period based on the sample network status data.

[0225] The second reward value is positively correlated with the receiving rate of the sample period, and negatively correlated with the rate difference; the rate difference represents the difference between the sending rate of the sample period and the receiving rate of the sample period.

[0226] Step S403: Input the sample network state data and the second reward value into the adjustment strategy prediction network model of the initial parameters to obtain the sample adjustment strategy and strategy score value.

[0227] Step S404: adjusting the current congestion window according to the sample adjustment strategy.

[0228] Step S405: Based on the strategy score value and the second reward value, the model parameters of the initial parameter adjustment strategy prediction network model are adjusted until the convergence condition is reached.

[0229] Among them, the adjustment strategy prediction network model can be implemented based on the deep reinforcement learning network model.

[0230] See also Figure 5 , Figure 5 A schematic diagram of generating an adjustment strategy based on an adjustment strategy prediction network model provided in an embodiment of the present application. The network model includes two fully connected layers (fully connected layer 1 and fully connected layer 2), two activation layers (activation layer 1 and activation layer 2) and a target network layer. Each fully connected layer can include 512 neuron nodes. The network state data is input into the network model, and the feature data output by activation layer 2 can be obtained. Then, the obtained feature data is input into the target network layer together with the reward value to obtain the strategy and the evaluation value. The strategy includes the adjustment action for adjusting the congestion window and the probability corresponding to each adjustment action, and the evaluation value represents the score of the strategy. The target network layer can be an LSTM (Long Short-Term Memory, long short-term memory network) layer or a fully connected layer. During the training process, the congestion window is adjusted according to the probability corresponding to the adjustment action in the obtained strategy, and the loss value is calculated based on the strategy and the evaluation value, and the model parameters of the adjustment strategy prediction network model are adjusted based on the loss value. Correspondingly, in the deployment stage, the congestion window can be adjusted directly according to the strategy output by the target network layer.

[0231] In contrast to the above method embodiment, the present application embodiment further provides a congestion control device, see Figure 6 , Figure 6 A schematic diagram of a congestion control device provided in an embodiment of the present application, the device may include:

[0232] The first network status collection module 601 is used to obtain the network status data of the current collection period as the first network status data when the preset collection period is reached;

[0233] Wherein, the first network status data includes the sending rate and receiving rate of the current collection cycle; the sending rate of the current collection cycle is determined based on the sending rate collected at each specified time in the current collection cycle; the receiving rate of the current collection cycle is determined based on the receiving rate collected at each specified time in the current collection cycle; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last received ACK message and the sending time of the data packet corresponding to the first time;

[0234] A first reward value calculation module 602, configured to calculate a reward value of a current collection cycle based on the first network status data as a first reward value;

[0235] The target adjustment strategy acquisition module 603 is used to input the first network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy; wherein the adjustment strategy prediction network model is obtained by training based on a reinforcement learning algorithm;

[0236] The congestion window adjustment module 604 is used to adjust the current congestion window according to the target adjustment strategy.

[0237] By using the congestion control device provided by the embodiment of the present application, the sending rate and receiving rate of the current acquisition cycle can be obtained for different network environments, and the sending rate and receiving rate can effectively reflect the current network state. Therefore, congestion control can be effectively implemented based on the information obtained. Since there is no need to distinguish whether the congestion signal is caused by network congestion, as long as the sending rate and receiving rate of the current acquisition cycle are obtained, the congestion window can be adjusted. Therefore, the congestion control device of the present application can be applied to complex network environments and can improve the effectiveness of congestion control. In addition, since the reward value is related to the sending rate and receiving rate of the current acquisition cycle, the receiving rate is made equal to the sending rate as much as possible through the reward mechanism. Therefore, by adjusting the congestion window according to the target adjustment strategy obtained based on the above-mentioned reward value, a larger bandwidth utilization rate can be obtained, and the over-issuance problem caused by the sending rate being greater than the receiving rate can be avoided.

[0238] In one embodiment, the first reward value is positively correlated with the receiving rate of the current acquisition cycle and negatively correlated with the rate difference of the current acquisition cycle; the rate difference of the current acquisition cycle represents the difference between the sending rate of the current acquisition cycle and the receiving rate of the current acquisition cycle.

[0239] In one embodiment, the first network status data further includes at least one of the following:

[0240] The minimum round-trip time of the current collection cycle means: in the current collection cycle, when each specified time is reached, the minimum value of the round-trip time of each specified time that has been collected;

[0241] The average round-trip time of the current collection cycle means: the average round-trip time collected at each specified time in the current collection cycle;

[0242] The average delay of the current collection cycle means: the average value of the delays collected at each specified time in the current collection cycle; the delay collected at a specified time means the difference between the round-trip time collected at the specified time and the minimum value of the round-trip time collected at the specified time;

[0243] The average congestion window size of the current collection period is: the average value of the congestion window sizes collected at each specified time in the current collection period;

[0244] The average in-flight data size of the current collection period indicates: the average value of the in-flight data size collected at each specified time in the current collection period; the in-flight data size collected at a specified time indicates: the size of the data packet that has been sent at the specified time and has not received the corresponding ACK message;

[0245] The size of data sent during the current collection cycle;

[0246] The size of the data packet in response to the ACK message received during the current collection cycle;

[0247] The size of the data packets lost in the current collection cycle;

[0248] The number of ACK messages received during the current collection period that indicate congestion signals.

[0249] In one embodiment, the specified time also includes a second time when a packet loss event is detected;

[0250] The round trip time collected at each second moment represents: the round trip time collected when the last ACK message was received before the second moment;

[0251] The delay collected at each second moment represents: the delay collected when the last ACK message was received before the second moment;

[0252] The sending rate collected at each second moment represents: the sending rate collected when the last ACK message was received before the second moment;

[0253] The receiving rate collected at each second moment represents: the receiving rate collected when the last ACK message was received before the second moment.

[0254] In one embodiment, the first reward value is negatively correlated with the average delay of the current acquisition cycle.

[0255] In one embodiment, see Figure 7 , Figure 7 Another structural diagram of the congestion control device provided in the embodiment of the present application, the first reward value calculation module 602 includes:

[0256] A first threshold determination submodule 6021 is used to determine whether the average delay of the current acquisition cycle is less than a first threshold, and if so, trigger a first reward value calculation submodule 6022, and if not, trigger a second reward value calculation submodule 6023;

[0257] A first reward value calculation submodule 6022, configured to determine the first reward value of the current acquisition cycle as the receiving rate of the current acquisition cycle;

[0258] The second reward value calculation submodule 6023 is used to calculate the reward value of the current collection cycle as the first reward value based on the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle.

[0259] In one embodiment, the first threshold is positively correlated with the minimum round trip time of the current acquisition cycle.

[0260] In one embodiment, the first threshold is calculated based on the above formula (4);

[0261] The second reward value calculation submodule 6023 is used to calculate the reward value of the current collection cycle as the first reward value based on the sending rate of the current collection cycle, the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle according to the above formula (5).

[0262] In one embodiment, the sending rate of the current collection cycle is the sending rate collected at the last specified time in the current collection cycle; the receiving rate of the current collection cycle is the receiving rate collected at the last specified time in the current collection cycle.

[0263] In one embodiment, the target adjustment strategy includes two or more first specified adjustment multiples greater than 1, two or more second specified adjustment multiples that are reciprocals of the two or more first specified adjustment multiples, and a probability corresponding to each specified adjustment multiple;

[0264] The congestion window adjustment module 604 is specifically configured to adjust the current congestion window according to the designated adjustment multiple with the highest corresponding probability.

[0265] In one embodiment, see Figure 8 , Figure 8 Another structural schematic diagram of a congestion control device provided in an embodiment of the present application, the device further includes:

[0266] The sending rate adjustment module 605 is used to send data packets at a rate lower than the current receiving rate within a first time period when a preset adjustment period is reached.

[0267] In the embodiment of the present application, the sending rate adjustment module 605 and other modules in the congestion control device can be processed asynchronously.

[0268] In one embodiment, the apparatus further comprises:

[0269] The second network status acquisition module is used to obtain the network status data of a preset number of historical cycles before the current collection cycle as the second network status data before inputting the first network status data and the first reward value into the pre-trained adjustment strategy prediction network model to obtain the target adjustment strategy.

[0270] The target adjustment strategy acquisition module is used to input the first network state data, the second network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy.

[0271] In one embodiment, see Fig. 9 , Fig. 9 Another structural schematic diagram of a congestion control device provided in an embodiment of the present application, the congestion control device further includes:

[0272] A training module 606 is used to obtain network status data of a sample period as sample network status data;

[0273] Wherein, the sample network status data includes: the sending rate and receiving rate of the sample period; the sending rate of the sample period is determined based on the sending rate collected at each specified time in the sample period; the receiving rate of the sample period is determined based on the receiving rate collected at each specified time in the sample period; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last ACK message received before the data packet corresponding to the first time is sent and the sending time of the data packet corresponding to the first time;

[0274] Based on the sample network status data, a second reward value of the sample period is calculated; wherein the second reward value is positively correlated with the receiving rate of the sample period, and negatively correlated with the rate difference of the sample period; the rate difference represents the difference between the sending rate of the sample period and the receiving rate of the sample period;

[0275] Input the sample network state data and the second reward value into the adjustment strategy prediction network model of the initial parameters to obtain a sample adjustment strategy and a strategy score value;

[0276] Adjust the current congestion window according to the sample adjustment strategy;

[0277] Based on the strategy score value and the second reward value, the model parameters of the initial parameter adjustment strategy prediction network model are adjusted until a convergence condition is reached.

[0278] The present application also provides an electronic device, such as Fig.10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0279] Memory 1003, used for storing computer programs;

[0280] The processor 1001 is used to execute the program stored in the memory 1003, and implements the following steps:

[0281] When the first preset period is reached, the network status data of the current collection period is obtained as the first network status data;

[0282] Wherein, the first network status data includes: the sending rate and receiving rate of the current collection cycle; the sending rate of the current collection cycle is determined based on the sending rate collected at each specified time in the current collection cycle; the receiving rate of the current collection cycle is determined based on the receiving rate collected at each specified time in the current collection cycle; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last ACK message received before the data packet corresponding to the first time is sent and the sending time of the data packet corresponding to the first time;

[0283] Based on the first network status data, calculating a reward value of a current collection cycle as a first reward value;

[0284] Inputting the first network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy; wherein the adjustment strategy prediction network model is trained based on a reinforcement learning algorithm;

[0285] Adjust the current congestion window according to the target adjustment strategy.

[0286] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0287] The communication interface is used for communication between the above electronic device and other devices.

[0288] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0289] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0290] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above congestion control methods are implemented.

[0291] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any congestion control method in the above embodiments.

[0292] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0293] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0294] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0295] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A congestion control method, characterized in that: The method comprises: When the preset collection period is reached, the network status data of the current collection period is obtained as the first network status data; Wherein, the first network status data includes the sending rate and receiving rate of the current collection cycle; the sending rate of the current collection cycle is determined based on the sending rate collected at each specified time in the current collection cycle; the receiving rate of the current collection cycle is determined based on the receiving rate collected at each specified time in the current collection cycle; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last received ACK message and the sending time of the data packet corresponding to the first time; Based on the first network status data, calculating a reward value of a current collection cycle as a first reward value; Inputting the first network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy; wherein the adjustment strategy prediction network model is trained based on a reinforcement learning algorithm; According to the target adjustment strategy, the current congestion window is adjusted; The first network status data also includes: The minimum round-trip time of the current collection cycle means: in the current collection cycle, when each specified time is reached, the minimum value of the round-trip time of each specified time that has been collected; The average delay of the current collection period is: the average value of the delay collected at each specified time in the current collection period; The step of calculating the reward value of the current collection cycle based on the first network status data as the first reward value includes: Determine whether the average delay of the current collection period is less than a first threshold; If yes, then determining the first reward value of the current acquisition period as the receiving rate of the current acquisition period; If not, the reward value of the current collection cycle is calculated as the first reward value based on the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle.

2. The method according to claim 1, characterized in that: The first reward value is positively correlated with the receiving rate of the current acquisition cycle, and negatively correlated with the rate difference of the current acquisition cycle; the rate difference of the current acquisition cycle represents the difference between the sending rate of the current acquisition cycle and the receiving rate of the current acquisition cycle.

3. The method according to claim 1, characterized in that The first network status data also includes at least one of the following: The average round-trip time of the current collection cycle means: the average round-trip time collected at each specified time in the current collection cycle; The average congestion window size of the current collection period is: the average value of the congestion window sizes collected at each specified time in the current collection period; The average in-flight data size of the current collection period indicates: the average value of the in-flight data size collected at each specified time in the current collection period; the in-flight data size collected at a specified time indicates: the size of the data packet that has been sent at the specified time and has not received the corresponding ACK message; The size of the data sent during the current collection cycle; The size of the data packet in response to the ACK message received during the current collection cycle; The size of the data packets lost in the current collection cycle; The number of ACK messages received during the current collection period that indicate congestion signals.

4. The method according to claim 3, characterized in that The specified time also includes a second time when a packet loss event is detected; The round trip time collected at each second moment represents: the round trip time collected when the last ACK message was received before the second moment; The delay collected at each second moment represents: the delay collected when the last ACK message was received before the second moment; The sending rate collected at each second moment represents: the sending rate collected when the last ACK message was received before the second moment; The receiving rate collected at each second moment represents: the receiving rate collected when the last ACK message was received before the second moment.

5. The method according to claim 3, characterized in that: The first reward value is negatively correlated with the average delay of the current acquisition cycle.

6. The method according to claim 1, characterized in that The first threshold is positively correlated with the minimum round trip time of the current acquisition cycle.

7. The method according to claim 6, characterized in that The first threshold is calculated based on a first formula; The first formula is: S=εMinRtt+ρ Wherein, S represents the first threshold, MinRtt represents the minimum round trip time of the current acquisition cycle, ε represents the first preset parameter, and ρ represents the second preset parameter; The reward value of the current collection cycle is calculated as the first reward value based on the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle, including: Based on the sending rate of the current collection cycle, the sending rate of the current collection cycle of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle, the reward value of the current collection cycle is calculated according to the second formula as the first reward value; wherein the second formula is: Among them, reward represents the first reward value, AR represents the receiving rate of the current collection cycle, D represents the average delay of the current collection cycle, MinRtt represents the minimum round-trip time of the current collection cycle, SR represents the sending rate of the current collection cycle, and δ represents the third preset parameter.

8. The method according to claim 1, characterized in that The target adjustment strategy includes two or more first designated adjustment multiples greater than 1, two or more second designated adjustment multiples that are reciprocals of the two or more first designated adjustment multiples, and a probability corresponding to each designated adjustment multiple; The adjusting the current congestion window according to the target adjustment strategy includes: The current congestion window is adjusted according to the specified adjustment multiple with the highest corresponding probability.

9. The method according to claim 1, characterized in that: The method further comprises: When the preset adjustment period is reached, data packets are sent at a rate lower than the current receiving rate within a first time period.

10. According to the method of claim 1, the training process of the adjustment strategy prediction network model comprises the following steps: Obtaining network status data of a sample period as sample network status data; Wherein, the sample network status data includes: the sending rate and receiving rate of the sample period; the sending rate of the sample period is determined based on the sending rate collected at each specified time in the sample period; the receiving rate of the sample period is determined based on the receiving rate collected at each specified time in the sample period; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last received ACK message and the sending time of the data packet corresponding to the first time; Based on the sample network status data, a second reward value of the sample period is calculated; wherein the second reward value is positively correlated with the receiving rate of the sample period, and negatively correlated with the rate difference of the sample period; the rate difference represents the difference between the sending rate of the sample period and the receiving rate of the sample period; Input the sample network state data and the second reward value into the adjustment strategy prediction network model of the initial parameters to obtain a sample adjustment strategy and a strategy score value; Adjust the current congestion window according to the sample adjustment strategy; Based on the strategy score value and the second reward value, the model parameters of the initial parameter adjustment strategy prediction network model are adjusted until a convergence condition is reached.

11. A congestion control device, characterized in that: The device comprises: A first network status collection module, used for acquiring network status data of a current collection period as first network status data when a preset collection period is reached; Wherein, the first network status data includes the sending rate and receiving rate of the current collection cycle; the sending rate of the current collection cycle is determined based on the sending rate collected at each specified time in the current collection cycle; the receiving rate of the current collection cycle is determined based on the receiving rate collected at each specified time in the current collection cycle; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last received ACK message and the sending time of the data packet corresponding to the first time; A first reward value calculation module, used to calculate the reward value of the current collection cycle based on the first network status data as a first reward value; A target adjustment strategy acquisition module, used to input the first network state data and the first reward value into a pre-trained adjustment strategy prediction network model to obtain a target adjustment strategy; wherein the adjustment strategy prediction network model is obtained by training based on a reinforcement learning algorithm; A congestion window adjustment module, used to adjust the current congestion window according to the target adjustment strategy; The first network status data also includes: The minimum round-trip time of the current collection cycle means: in the current collection cycle, when each specified time is reached, the minimum value of the round-trip time of each specified time that has been collected; The average delay of the current collection period is: the average value of the delay collected at each specified time in the current collection period; The first reward value calculation module includes: A first threshold judgment submodule, used to judge whether the average delay of the current acquisition cycle is less than a first threshold, and if so, trigger the first reward value calculation submodule; if not, trigger the second reward value calculation submodule; A first reward value calculation submodule, used to determine the first reward value of the current acquisition cycle as the receiving rate of the current acquisition cycle; The second reward value calculation submodule is used to calculate the reward value of the current collection cycle as the first reward value based on the sending rate of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle.

12. The device according to claim 11, characterized in that The first reward value is positively correlated with the receiving rate of the current acquisition cycle, and negatively correlated with the rate difference of the current acquisition cycle; the rate difference of the current acquisition cycle represents the difference between the sending rate of the current acquisition cycle and the receiving rate of the current acquisition cycle.

13. The device according to claim 11, characterized in that The first network status data also includes at least one of the following: The average round-trip time of the current collection cycle means: the average round-trip time collected at each specified time in the current collection cycle; The average congestion window size of the current collection period is: the average value of the congestion window sizes collected at each specified time in the current collection period; The average in-flight data size of the current collection period indicates: the average value of the in-flight data size collected at each specified time in the current collection period; the in-flight data size collected at a specified time indicates: the size of the data packet that has been sent at the specified time and has not received the corresponding ACK message; The size of the data sent during the current collection cycle; The size of the data packet in response to the ACK message received during the current collection cycle; The size of the data packets lost in the current collection cycle; The number of ACK messages received during the current collection period that indicate congestion signals.

14. The device according to claim 11, characterized in that The specified time also includes a second time when a packet loss event is detected; The round trip time collected at each second moment represents: the round trip time collected when the last ACK message was received before the second moment; The delay collected at each second moment represents: the delay collected when the last ACK message was received before the second moment; The sending rate collected at each second moment represents: the sending rate collected when the last ACK message was received before the second moment; The receiving rate collected at each second moment represents: the receiving rate collected when the last ACK message was received before the second moment.

15. The device according to claim 11, characterized in that The first reward value is negatively correlated with the average delay of the current acquisition cycle.

16. The device according to claim 11, characterized in that The first threshold is positively correlated with the minimum round trip time of the current acquisition cycle.

17. The device according to claim 16, characterized in that The first threshold is calculated based on a first formula; The first formula is: S=εMinRtt+ρ Wherein, S represents the first threshold, MinRtt represents the minimum round trip time of the current acquisition cycle, ε represents the first preset parameter, and ρ represents the second preset parameter; The second reward value calculation submodule is used to calculate the reward value of the current collection cycle as the first reward value according to the second formula based on the sending rate of the current collection cycle, the sending rate of the current collection cycle of the current collection cycle, the receiving rate of the current collection cycle, the average delay of the current collection cycle, and the minimum round-trip time of the current collection cycle; wherein the second formula is: Among them, reward represents the first reward value, AR represents the receiving rate of the current collection cycle, D represents the average delay of the current collection cycle, MinRtt represents the minimum round-trip time of the current collection cycle, SR represents the sending rate of the current collection cycle, and δ represents the third preset parameter.

18. The device according to claim 11, characterized in that The target adjustment strategy includes two or more first designated adjustment multiples greater than 1, two or more second designated adjustment multiples that are reciprocals of the two or more first designated adjustment multiples, and a probability corresponding to each designated adjustment multiple; The congestion window adjustment module is specifically used to adjust the current congestion window according to the corresponding maximum probability specified adjustment multiple.

19. The device according to claim 11, characterized in that The device also includes: The sending rate adjustment module is used to send data packets at a rate lower than the current receiving rate within a first time period when a preset adjustment period is reached.

20. The device according to claim 11, characterized in that The device also includes: A training module, used for obtaining network status data of a sample period as sample network status data; Wherein, the sample network status data includes: the sending rate and receiving rate of the sample period; the sending rate of the sample period is determined based on the sending rate collected at each specified time in the sample period; the receiving rate of the sample period is determined based on the receiving rate collected at each specified time in the sample period; the specified time includes the first time when the ACK message is received; the receiving rate of each first time represents: the rate of receiving data between the time when the last ACK message is received before the data packet corresponding to the first time is sent and the first time; the data packet corresponding to the first time represents the data packet responded by the ACK message received at the first time; the sending rate of each first time represents: the rate of sending data between the sending time of the data packet responded by the last received ACK message and the sending time of the data packet corresponding to the first time; Based on the sample network status data, a second reward value of the sample period is calculated; wherein the second reward value is positively correlated with the receiving rate of the sample period, and negatively correlated with the rate difference of the sample period; the rate difference represents the difference between the sending rate of the sample period and the receiving rate of the sample period; Input the sample network state data and the second reward value into the adjustment strategy prediction network model of the initial parameters to obtain a sample adjustment strategy and a strategy score value; Adjust the current congestion window according to the sample adjustment strategy; Based on the strategy score value and the second reward value, the model parameters of the initial parameter adjustment strategy prediction network model are adjusted until a convergence condition is reached.

21. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 10 when executing a program stored in a memory.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 10 are implemented.

Images