Congestion window length adjusting method and device and nonvolatile storage medium

Abstract

The invention discloses a congestion window length adjusting method and device and a nonvolatile storage medium. The method comprises the following steps: acquiring a network state parameter of a target computing power network at a current moment; inputting the network state parameters at the current moment into the congestion control strategy model, and determining the current action; calculating a reward value corresponding to the current action according to a reward function in the congestion control strategy model; based on the reward value corresponding to the current action, parameters of the congestion control strategy model are optimized, and an optimized congestion control strategy model is determined; repeating the steps to obtain reward values corresponding to the current actions; and determining an optimal current action according to the reward values corresponding to the plurality of current actions, and obtaining a new congestion window length. According to the invention, the technical problem that the real network state cannot be reflected and the transmission efficiency is influenced because only single network state information is used as an index in a congestion control method in the related technology is solved.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a congestion window length adjustment method, device and non-volatile storage medium. Background Art

[0002] The computing network is a new paradigm for computing-network integration, which aims to interconnect and coordinate distributed computing nodes, and optimize and efficiently utilize network and computing resources through improved design of network architecture and protocols. Congestion is inevitable due to the randomness of traffic and diversity of paths in the computing network. When the network is congested, it will increase queuing delays, reduce network utilization, and lead to degraded application performance. Congestion control of the computing network is an important means to ensure the normal operation of the network, improve performance and user experience, maintain security, and effectively utilize resources. By adopting appropriate congestion control strategies, we can better adapt to the growing network traffic and complex application scenarios.

[0003] At present, there are many congestion control methods that continuously adjust the transmission rate on the end side to ultimately make the access capacity as close to the network's carrying capacity as possible, thereby solving the congestion problem in the network. However, the current traditional mainstream congestion control methods have the disadvantages of insufficient control accuracy and inability to adaptively adjust. The network status information used by the existing network congestion control algorithm generally selects one or two items as indicators to measure the network status information. It only considers a single indicator, has limited rules and conditions, and falls into the problem of local optimal solutions, and is difficult to adapt to the requirements of computing network traffic transmission.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a congestion window length adjustment method, device and non-volatile storage medium to at least solve the technical problem that the congestion control method in the related art only uses a single network status information as an indicator, resulting in failure to reflect the actual network status and affecting the transmission efficiency.

[0006] According to one aspect of an embodiment of the present invention, a congestion window length adjustment method is provided, comprising: obtaining network status parameters of a target computing power network at a current moment, wherein the network status parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay, and service flow completion time; inputting the network status parameters at the current moment into a congestion control policy model to determine a current action, wherein the current action represents a change in the length of the congestion window at the current moment; calculating a reward value corresponding to the current action according to a reward function in the congestion control policy model; optimizing the parameters of the congestion control policy model based on the reward value corresponding to the current action to determine an optimized congestion control policy model, wherein the optimized congestion control policy model is used for determining the current action processing next time; repeating the above steps to obtain reward values ​​corresponding to each of a plurality of current actions; determining an optimal current action according to the reward values ​​corresponding to each of the plurality of current actions; adjusting the length of the congestion window based on the optimal current action to obtain a new congestion window length.

[0007] Optionally, before inputting the network state parameters at the current moment into the congestion control strategy model, it includes: determining the congestion control strategy model based on a preset state space, action space and reward function, wherein the state space is determined according to the network state parameters, and the action space is used to define the current action.

[0008] Optionally, a weight factor is obtained, wherein the weight factor represents a weight ratio of throughput, packet loss rate, link delay and service flow completion time to the reward value; and a reward function is determined based on the weight factor.

[0009] Optionally, the reward function is expressed as follows:

[0010]

[0011] Among them, O is the throughput of the receiving end of the target computing network at the current moment, O max is the maximum throughput in the historical moment, LP is the packet loss rate at the current moment, LP min is the minimum packet loss rate in the historical moment; l is the average link delay at the current moment, l min is the minimum link delay in the history; FCT is the average completion time of the service flow at the current moment, FCT min is the minimum business flow completion time in the historical moment; a is the weight factor.

[0012] Optionally, based on the reward value corresponding to the current action, the parameters of the congestion control policy model are optimized to determine the optimized congestion control policy model, wherein the optimized congestion control policy model is used to determine the processing of the current action next time, including: based on the reward value corresponding to the current action and the objective function corresponding to the proximal policy optimization algorithm, the parameters of the congestion control policy model are optimized until the objective function value corresponding to the proximal policy optimization algorithm is maximized, and the parameters of the current congestion control policy model are determined to be the parameters of the optimized congestion control policy model; based on the parameters of the optimized congestion control policy model, the optimized congestion control policy model is determined.

[0013] Optionally, the optimal current action is determined based on the reward values ​​corresponding to each of the multiple current actions, including: successively calculating the average value of the reward values ​​corresponding to each of the multiple current actions; when the average value of the reward values ​​corresponding to each of the multiple current actions reaches a preset threshold, selecting the current action determined last as the optimal current action.

[0014] According to another aspect of an embodiment of the present invention, a congestion window length adjustment device is also provided, comprising: an acquisition module, used to acquire network state parameters of a target computing power network at a current moment, wherein the network state parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay, and service flow completion time; a first determination module, used to input the network state parameters at a current moment into a congestion control policy model to determine a current action, wherein the current action represents a change in the length of the congestion window at the current moment; a calculation module, used to calculate a reward value corresponding to the current action according to a reward function in the congestion control policy model; a second determination module, used to optimize the parameters of the congestion control policy model based on the reward value corresponding to the current action, and determine an optimized congestion control policy model, wherein the optimized congestion control policy model is used to determine the processing of the current action next time; a repetition module, used to repeat the above steps to obtain reward values ​​corresponding to each of a plurality of current actions; a third determination module, used to determine an optimal current action according to the reward values ​​corresponding to each of the plurality of current actions; an adjustment module, used to adjust the length of the congestion window based on the optimal current action to obtain a new congestion window length.

[0015] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, the non-volatile storage medium including a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned congestion window length adjustment methods.

[0016] According to another aspect of the embodiments of the present invention, a computer device is provided. The computer device includes a processor, and the processor is used to run a program. When the program is run, any one of the above congestion window length adjustment methods is executed.

[0017] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the above congestion window length adjustment methods is implemented.

[0018] In an embodiment of the present invention, a congestion window length adjustment method is adopted, by obtaining the network state parameters of the target computing power network at the current moment, wherein the network state parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay and service flow completion time; the network state parameters at the current moment are input into the congestion control strategy model to determine the current action, wherein the current action represents the change in the length of the congestion window at the current moment; according to the reward function in the congestion control strategy model, the reward value corresponding to the current action is calculated; based on the reward value corresponding to the current action, the parameters of the congestion control strategy model are optimized to determine the optimized congestion control strategy model, wherein, The optimized congestion control strategy model is used to determine the processing of the current action next time; repeat the above steps to obtain the reward values ​​corresponding to multiple current actions; determine the optimal current action according to the reward values ​​corresponding to the multiple current actions; based on the optimal current action, adjust the length of the congestion window to obtain a new congestion window length, thereby achieving the purpose of fully considering the network status information and adjusting the congestion window length in real time, thereby achieving the technical effect of accurately adjusting the congestion window length and improving the traffic transmission efficiency, and further solving the technical problem that the congestion control method in the related technology only uses a single network status information as an indicator, resulting in the inability to reflect the real network status and affecting the transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 A hardware structure block diagram of a computer terminal for implementing a congestion window length adjustment method is shown;

[0021] Figure 2 is a flow chart of a congestion window length adjustment method provided according to an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of a network congestion control method framework based on a PPO algorithm provided according to an optional embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of a computing power network traffic congestion control topology provided according to an optional embodiment of the present invention;

[0024] Figure 5 It is a structural block diagram of a congestion window length adjustment device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] According to an embodiment of the present invention, an embodiment of a congestion window length adjustment method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a congestion window length adjustment method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0030] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the congestion window length adjustment method in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the congestion window length adjustment method of the above-mentioned application program is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0031] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0032] Figure 2 is a flow chart of a method for adjusting the congestion window length according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0033] Step S201, obtain the network status parameters of the target computing power network at the current moment, wherein the network status parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay and service flow completion time.

[0034] In this step, the network status information of the computing network refers to a collection of real-time or historical data about each node, link and overall network performance in the computing network. Typical congestion control algorithms generally select one or two network status information as indicators. For example, CUBIC (Cubic BIC-TCP) detects network congestion by detecting network packet loss. It uses an online detection method to gradually increase the congestion window when there is no packet loss and reduce the congestion window when a large number of packet loss occurs; TCP Vegas uses the delay indicator to judge congestion, increases the congestion window when the delay increases, and reduces the congestion window when the delay decreases; BBR (Bottleneck Bandwidth and Round-trip propagation time) is a congestion control algorithm based on the available capacity of the link. When the available capacity of the link increases, the congestion window increases, and when the available capacity of the link decreases, the congestion window decreases. However, these algorithms only consider a single indicator, are limited by rules and conditions, fall into local optimal solutions, and are difficult to adapt to the requirements of computing network traffic transmission.

[0035] Due to the complexity and overhead of the algorithm, each additional network status information requires more computing resources and bandwidth to obtain and transmit this information, and more information requires more processing power and storage space, and the real-time and efficiency of the network itself may be affected. Therefore, the present invention proposes a congestion window length adjustment method, which introduces the current value and historical observation maximum value of multiple indicators that can reflect the network status information in the computing power network, and does not bring additional economic burden or reduce efficiency, so that the congestion control algorithm can achieve comprehensive optimization in terms of indicators such as fairness, stability and transmission efficiency.

[0036] For example, the network status information collected by the present invention may include the following parameters: RTT (Round-Trip Time), which refers to the time taken for a data packet to be sent, received, and confirmed; packet sending time slot, which refers to the frequency at which the sender sends data packets. ACK (Acknowledge character) time slot, which refers to the frequency at which the sender receives ACK data packets; throughput, which refers to the number of data bytes determined by the receiver per second; packet loss rate, which refers to the number of lost packets, which can effectively illustrate the congestion state of the network; current congestion window length, the congestion control algorithm needs to adjust the new window value according to the current congestion window length. If the current congestion window length is small, the window length can be increased at a faster rate. If the window is large, the window length is stopped or increased more slowly; link average delay, which reflects the overall congestion state of the link; service flow completion time, which reflects the degree of impact on network services.

[0037] Step S202: input the network state parameters at the current moment into the congestion control strategy model to determine the current action, wherein the current action represents changing the length of the congestion window at the current moment.

[0038] In this step, the congestion window length adjustment method provided by the present invention uses the network state information (RTT, link throughput, TCP connection relative time, ACK round-trip time, packet loss rate, sending rate, RTT deviation, etc.) as the state space, adjusts the current congestion window as the dynamic space, and uses greater throughput, low latency, low packet loss rate, low service flow completion time, etc. as reward values, which can be abstracted as a partially observable Markov decision process. It is defined as a five-tuple {S, A, R, P, γ}. Among them, S is the set of network environment states, s t ∈S represents the state observed at time t; A is the set of executable actions for adjusting the congestion control window, a t ∈A represents the action taken at time t; R is the reward value function, defined as R(s t ,a t )=E[R t+1 |s t ,a t ], indicating that the state observed at time t is s t , select action a t After that, at time t+1, the reward R is received t+1 ; P is the transition probability matrix, according to the state transition probability P(s t+1 |s t ,a t ) arrives at a new state s t+1 ; γ∈[0,1] is the discount factor, which is the penalty ratio for future reward values, that is, the reward values ​​that can be obtained immediately are given priority, and the reward values ​​obtained in the future are attenuated at a certain ratio.

[0039] Among them, the congestion control strategy model can be t ,s t )Select the current action a t That is, according to the network status parameters at the current moment, a corresponding current action can be selected to determine the adjustment method for the congestion window length at the current moment.

[0040] Step S203, calculating the reward value corresponding to the current action according to the reward function in the congestion control strategy model.

[0041] In this step, the reward function is used to evaluate the quality of performing a certain action under a given state. The design of the reward function directly affects the learning direction of the congestion control policy model. By adjusting the setting of the reward function, the congestion control policy model can be guided to learn in the direction of the most appropriate congestion window length, thereby achieving better decision-making results. The reward value can be calculated by the reward function. The network state at each moment corresponds to a current action and a reward value. The reward value is the instant feedback obtained by the congestion control policy model from the network environment after executing the action. It tells the congestion control policy model how the current action works, whether it is positive or negative, and then guides the congestion control policy model to optimize in a positive direction.

[0042] Step S204, based on the reward value corresponding to the current action, the parameters of the congestion control strategy model are optimized to determine the optimized congestion control strategy model, wherein the optimized congestion control strategy model is used for determining the processing of the current action next time.

[0043] In this step, the effect of the current action can be determined based on the reward value corresponding to the current action. Therefore, the parameters of the congestion control strategy model can be optimized according to the size of the reward value, so that the adjusted congestion control strategy model can select better effects based on network status information, that is, actions with higher reward values. The optimized congestion control strategy model can be used for the next step of determining the action.

[0044] Step S205, repeat the above steps to obtain the reward values ​​corresponding to multiple current actions.

[0045] In this step, the above steps are repeated. After a current action is obtained, the congestion control strategy model is optimized, and a new current action is determined again based on the optimized congestion control strategy model. Therefore, by continuously optimizing the model and determining the action, a current action with better and better effects can be gradually obtained, and the reward values ​​of multiple current actions calculated by the reward function are used to select the optimal current action.

[0046] Step S206, determining the optimal current action according to the reward values ​​corresponding to the multiple current actions.

[0047] In this step, by optimizing the congestion control strategy model multiple times, the optimized congestion control strategy model can select a current action each time, wherein the reward value corresponding to each current action is different. Generally speaking, since the congestion control strategy model is continuously optimized, the current action obtained can be more and more effective, and the optimal current action can be determined based on the size of the reward value. For example, the current action corresponding to the largest reward value can be directly selected as the optimal current action, or the average value of the reward values ​​corresponding to all current actions can be calculated, and the optimal current action can be determined by the size of the average value.

[0048] Step S207: Based on the optimal current action, the length of the congestion window is adjusted to obtain a new congestion window length.

[0049] In this step, the optimal current action represents the best way to adjust the congestion window length based on the network status information at the current moment. Therefore, based on the optimal current action, the congestion window length at the current moment can be adjusted. The congestion window length can be increased or decreased, and the rate of increase or decrease can also be different.

[0050] Through the above steps, the purpose of fully considering the network status information and adjusting the congestion window length in real time is achieved, thereby achieving the technical effect of accurately adjusting the congestion window length and improving traffic transmission efficiency, and further solving the technical problem that the congestion control method in the related technology only uses a single network status information as an indicator, resulting in the inability to reflect the real network status and affecting the transmission efficiency.

[0051] As an optional embodiment, before inputting the network state parameters at the current moment into the congestion control strategy model, it includes: determining the congestion control strategy model based on a preset state space, action space and reward function, wherein the state space is determined according to the network state parameters, and the action space is used to define the current action.

[0052] Optionally, the state space can contain all possible states that the computing network may be in. The state is the basis for decision-making. At each time step, the corresponding action is selected according to the current state. The states in the state space are connected to each other through the state transition probability, forming a dynamic change process. The design of the state space is the key to establishing a congestion control strategy model. For example, the state space of the congestion control strategy model can include enough network state information, mainly including the following parameters: RTT, packet sending time slot, ACK time slot, throughput, packet loss rate, current congestion window length, link average delay and service flow completion time, etc.

[0053] Closely related to the state space is the action space, which contains all possible actions that can be performed. In each state, an action can be selected to affect the state transition of the system. For example, the action of the congestion control policy model is defined as changing the current congestion window length c and increasing the length w' of n segments. The expression is as follows:

[0054] c=c old +nw'

[0055] Among them, c old Indicates the unmodified congestion window length. Based on the observed state parameter information, the congestion window length growth rate or reduction rate can be determined.

[0056] As an optional embodiment, a weight factor is obtained, wherein the weight factor represents the weight ratio of throughput, packet loss rate, link delay and service flow completion time to the reward value; and a reward function is determined based on the weight factor.

[0057] Optionally, the weight factor is an important parameter in the reward function. Generally speaking, the size of the weight factor can be preset, for example, it can be preset to 0.5. By continuously optimizing the congestion control strategy model, the weight factor can also be changed, so that the reward function of the congestion control strategy model can calculate the reward value more accurately, thereby better representing the optimization direction of the congestion control strategy model.

[0058] As an optional embodiment, the reward function is expressed as follows:

[0059]

[0060] Among them, O is the throughput of the receiving end of the target computing network at the current moment, O max is the maximum throughput in the historical moment, LP is the packet loss rate at the current moment, LP min is the minimum packet loss rate in the historical moment; l is the average link delay at the current moment, l min is the minimum link delay in the history; FCT is the average completion time of the service flow at the current moment, FCT min is the minimum business flow completion time in the historical moment; a is the weight factor.

[0061] Optionally, in the present invention, the reward function is improved in a targeted manner according to the characteristics of the computing network traffic and the transmission requirements, and the current values ​​and historical observation values ​​of multiple indicators that can reflect the network status information in the computing network are introduced, without bringing additional economic burden or reducing efficiency, so that the congestion control strategy model can achieve comprehensive optimization in terms of fairness, stability and transmission efficiency. In addition, when the following situations are observed: the current throughput is less than the minimum throughput, the current packet loss rate is greater than the maximum packet loss rate, the current delay is greater than the maximum delay, and the current service flow completion time is greater than the maximum service flow completion time, the reward value can be set to -20 to prevent the congestion control strategy model from falling into overfitting.

[0062] As an optional embodiment, based on the reward value corresponding to the current action, the parameters of the congestion control policy model are optimized to determine the optimized congestion control policy model, wherein the optimized congestion control policy model is used to determine the processing of the current action next time, including: based on the reward value corresponding to the current action and the objective function corresponding to the proximal policy optimization algorithm, the parameters of the congestion control policy model are optimized until the objective function value corresponding to the proximal policy optimization algorithm is maximized, and the parameters of the current congestion control policy model are determined to be the parameters of the optimized congestion control policy model; based on the parameters of the optimized congestion control policy model, the optimized congestion control policy model is determined.

[0063] Optionally, the optimization of the parameters of the congestion control policy model can be achieved through a proximal policy optimization algorithm (PPO). By adjusting the parameters of the congestion control policy model, the objective function value corresponding to the proximal policy optimization algorithm is maximized. The proximal policy optimization algorithm is a reinforcement learning algorithm. In the prior art, there is no specific congestion signal in the congestion control algorithm based on reinforcement learning. It is generally based on training data and action-value function, and the congestion control policy model is generated by machine learning. The network state and reward function used by these algorithms are not reasonable enough. The congestion control policy model proposed in the present invention optimizes the model through the proximal policy optimization algorithm, and intelligently adjusts the window size by monitoring the network environment online to avoid congestion. It does not require manually preset window rules or pre-time-consuming offline training. The PPO algorithm can adjust the behavior of the congestion control policy model according to real-time feedback from the environment, and a reward value will be fed back for each action as an incentive, so as to adjust the window size to the optimal through continuous interaction.

[0064] In addition, compared with other congestion control methods based on reinforcement learning algorithms, the congestion control algorithm based on the PPO algorithm can improve the training efficiency of data through importance sampling, that is, it can be trained multiple times after interacting with the environment once through the network; at the same time, clip clipping is introduced to limit the amplitude of policy gradient updates, so as to prevent the difference between the new and old policy networks caused by multiple training from being too large, resulting in large differences in policy distribution, and then leading to unstable training.

[0065] As an optional embodiment, the optimal current action is determined based on the reward values ​​corresponding to each of the multiple current actions, including: successively calculating the average value of the reward values ​​corresponding to each of the multiple current actions; when the average value of the reward values ​​corresponding to each of the multiple current actions reaches a preset threshold, selecting the current action determined last as the optimal current action.

[0066] Optionally, the optimal current action can be determined based on the reward values ​​corresponding to the multiple current actions, which can be obtained by successively calculating the average of the reward values ​​corresponding to the multiple current actions. Each time a new current action is obtained, the average of the reward values ​​corresponding to the multiple current actions that have been obtained can be calculated. The goal of the congestion control strategy model is to maximize the expected value of the reward value, that is, the average value. The expected value of the reward value is defined as:

[0067]

[0068] Where T represents the end time, γ represents the discount factor, and r represents the received reward value. When the average value of the reward value reaches the maximum, the last determined current action can be selected as the optimal current action.

[0069] As an optional embodiment, Figure 3 Schematic diagram of a network congestion control method framework based on a PPO algorithm provided according to an optional embodiment of the present invention. Figure 3 As shown, the present invention uses the virtual data space network as the environment, constructs the intelligent agent model, that is, the policy function used by the congestion control strategy model, and generates the optimal control action by observing the state information in the environment. The policy function is fitted by an artificial neural network. The intelligent agent model adjusts the congestion window length according to the action output by the policy function to optimize the performance of the virtual data space network. After generating the action and interacting with the environment, the intelligent agent model will harvest the reward value from the environment. The intelligent agent model judges the quality of the selected action according to the reward value, and updates the artificial neural network parameters according to the reward value, so that the policy function can generate actions that harvest more reward values.

[0070] It can be seen from the process of the above-mentioned congestion control method that a network congestion control method based on the PPO algorithm provided by an optional embodiment of the present invention may have multiple advantages. For example, compared with the mainstream TCP congestion control algorithm, this method has better adaptability to the network environment and significantly improves the throughput. For another example, by introducing a reinforcement learning algorithm to support congestion control, making full use of historical experience and considering the current network information status, more precise measures are taken to adjust the congestion control window size, thereby preventing the instantaneous large-scale increase or decrease of the window and ensuring the smooth operation of the network. Similarly, this method does not require preset rules, but monitors the network environment online to intelligently adjust the window size to avoid congestion. While maintaining the input of multiple network status information indicators, it does not bring additional economic burdens or reduce efficiency.

[0071] As an optional embodiment, Figure 4 Schematic diagram of a computing network traffic congestion control topology provided according to an optional embodiment of the present invention. Figure 4 As shown, the network topology of communication between two computing devices in the data center is shown. The sender sends the network information to the intelligent model, and the intelligent model sends the action back to the network device of the front-end communication node after making a decision, and adjusts the size of the network congestion window to achieve the effect of regulating network congestion. Therefore, the congestion window length adjustment method proposed in the present invention can be applied to the computing network scenario, through more comprehensive network status information collection, more efficient network information interaction, and more accurate congestion window adjustment, the network congestion control effect is improved, the performance of the computing network is guaranteed, and the bandwidth utilization of the computing network is improved.

[0072] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that the congestion window length adjustment method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, a magnetic disk, an optical disk), including a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0074] According to an embodiment of the present invention, a congestion window length adjustment device for implementing the above congestion window length adjustment method is also provided. Figure 5 is a structural block diagram of a congestion window length adjustment device provided according to an embodiment of the present invention, such as Figure 5 As shown, the device includes: an acquisition module 51, a first determination module 52, a calculation module 53, a second determination module 54, a repetition module 55, a third determination module 56 and an adjustment module 57. The congestion window length adjustment device is described below.

[0075] The acquisition module 51 is used to obtain the network status parameters of the target computing power network at the current moment, wherein the network status parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay and service flow completion time.

[0076] The first determination module 52 is connected to the acquisition module 51 and is used to input the network state parameters at the current moment into the congestion control strategy model to determine the current action, wherein the current action represents the change of the length of the congestion window at the current moment.

[0077] The calculation module 53 is connected to the first determination module 52 and is used to calculate the reward value corresponding to the current action according to the reward function in the congestion control strategy model.

[0078] The second determination module 54 is connected to the calculation module 53, and is used to optimize the parameters of the congestion control strategy model based on the reward value corresponding to the current action, and determine the optimized congestion control strategy model, wherein the optimized congestion control strategy model is used to determine the processing of the current action next time.

[0079] The repeating module 55 is connected to the second determining module 54 and is used to repeat the above steps to obtain reward values ​​corresponding to multiple current actions.

[0080] The third determination module 56 is connected to the repetition module 55 and is used to determine the optimal current action according to the reward values ​​corresponding to the multiple current actions.

[0081] The adjustment module 57 is connected to the third determination module 56 and is used to adjust the length of the congestion window based on the optimal current action to obtain a new congestion window length.

[0082] It should be noted that the acquisition module 51, the first determination module 52, the calculation module 53, the second determination module 54, the repetition module 55, the third determination module 56 and the adjustment module 57 correspond to steps S201 to S207 in the embodiment, and the examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0083] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one network device among multiple network devices of a computer network. The computer device includes a memory and a processor.

[0084] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the congestion window length adjustment method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the above-mentioned congestion window length adjustment method is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0085] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the network status parameters of the target computing power network at the current moment, wherein the network status parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay and service flow completion time; input the network status parameters at the current moment into the congestion control strategy model to determine the current action, wherein the current action represents the change in the length of the congestion window at the current moment; calculate the reward value corresponding to the current action according to the reward function in the congestion control strategy model; based on the reward value corresponding to the current action, optimize the parameters of the congestion control strategy model to determine the optimized congestion control strategy model, wherein the optimized congestion control strategy model is used for the next determination of the current action; repeat the above steps to obtain the reward values ​​corresponding to each of the multiple current actions; determine the optimal current action according to the reward values ​​corresponding to each of the multiple current actions; based on the optimal current action, adjust the length of the congestion window to obtain a new congestion window length.

[0086] Optionally, the processor may also execute program code of the following steps: before inputting the network state parameters at the current moment into the congestion control strategy model, including: determining the congestion control strategy model based on a preset state space, action space and reward function, wherein the state space is determined according to the network state parameters, and the action space is used to define the current action.

[0087] Optionally, the processor may also execute the program code of the following steps: obtaining a weight factor, wherein the weight factor represents the weight ratio of throughput, packet loss rate, link delay and service flow completion time to the reward value; and determining a reward function based on the weight factor.

[0088] Optionally, the processor may further execute program code of the following steps: The expression of the reward function is as follows:

[0089]

[0090] Among them, O is the throughput of the receiving end of the target computing network at the current moment, O max is the maximum throughput in the historical moment, LP is the packet loss rate at the current moment, LP min is the minimum packet loss rate in the historical moment; l is the average link delay at the current moment, l min is the minimum link delay in the history; FCT is the average completion time of the service flow at the current moment, FCT min is the minimum business flow completion time in the historical moment; a is the weight factor.

[0091] Optionally, the processor may also execute the program code of the following steps: based on the reward value corresponding to the current action, optimizing the parameters of the congestion control policy model, and determining the optimized congestion control policy model, wherein the optimized congestion control policy model is used to determine the processing of the current action next time, including: based on the reward value corresponding to the current action and the objective function corresponding to the proximal policy optimization algorithm, optimizing the parameters of the congestion control policy model until the objective function value corresponding to the proximal policy optimization algorithm is maximized, determining the parameters of the current congestion control policy model as the parameters of the optimized congestion control policy model; and determining the optimized congestion control policy model based on the parameters of the optimized congestion control policy model.

[0092] Optionally, the processor may also execute program code for the following steps: determining the optimal current action based on the reward values ​​corresponding to each of the multiple current actions, including: successively calculating the average value of the reward values ​​corresponding to each of the multiple current actions; when the average value of the reward values ​​corresponding to each of the multiple current actions reaches a preset threshold, selecting the current action determined last as the optimal current action.

[0093] According to an embodiment of the present invention, a congestion window length adjustment method is provided. By obtaining the network status parameters of the target computing power network at the current moment, wherein the network status parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay and service flow completion time; inputting the network status parameters at the current moment into the congestion control strategy model to determine the current action, wherein the current action represents the change in the length of the congestion window at the current moment; calculating the reward value corresponding to the current action according to the reward function in the congestion control strategy model; optimizing the parameters of the congestion control strategy model based on the reward value corresponding to the current action, and determining the optimized congestion control strategy model, wherein the optimized congestion control strategy model The model is used to determine the processing of the current action next time; repeat the above steps to obtain the reward values ​​corresponding to multiple current actions; determine the optimal current action according to the reward values ​​corresponding to the multiple current actions; based on the optimal current action, adjust the length of the congestion window to obtain a new congestion window length, so as to fully consider the network status information and adjust the congestion window length in real time, thereby achieving the technical effect of accurately adjusting the congestion window length and improving the traffic transmission efficiency, and further solving the technical problem that the congestion control method in the related technology only uses a single network status information as an indicator, which makes it impossible to reflect the real network status and affects the transmission efficiency.

[0094] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0095] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the congestion window length adjustment method provided in the above embodiment.

[0096] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0097] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining network status parameters of the target computing power network at the current moment, wherein the network status parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay and service flow completion time; inputting the network status parameters at the current moment into the congestion control policy model to determine the current action, wherein the current action represents the change in the length of the congestion window at the current moment; calculating the reward value corresponding to the current action according to the reward function in the congestion control policy model; optimizing the parameters of the congestion control policy model based on the reward value corresponding to the current action, and determining the optimized congestion control policy model, wherein the optimized congestion control policy model is used for the next determination of the current action; repeating the above steps to obtain the reward values ​​corresponding to each of the multiple current actions; determining the optimal current action according to the reward values ​​corresponding to each of the multiple current actions; adjusting the length of the congestion window based on the optimal current action to obtain a new congestion window length.

[0098] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: before inputting the network state parameters at the current moment into the congestion control strategy model, including: determining the congestion control strategy model based on a preset state space, action space and reward function, wherein the state space is determined according to the network state parameters, and the action space is used to define the current action.

[0099] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: obtaining a weight factor, wherein the weight factor represents the weight ratio of throughput, packet loss rate, link delay and service flow completion time to the reward value; and determining a reward function based on the weight factor.

[0100] Optionally, in this embodiment, the non-volatile storage medium is configured to store program codes for executing the following steps: The expression of the reward function is as follows:

[0101]

[0102] Among them, O is the throughput of the receiving end of the target computing network at the current moment, O max is the maximum throughput in the historical moment, LP is the packet loss rate at the current moment, LP min is the minimum packet loss rate in the historical moment; l is the average link delay at the current moment, l min is the minimum link delay in the history; FCT is the average completion time of the service flow at the current moment, FCT min is the minimum business flow completion time in the historical moment; a is the weight factor.

[0103] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: based on the reward value corresponding to the current action, optimizing the parameters of the congestion control policy model, and determining the optimized congestion control policy model, wherein the optimized congestion control policy model is used to determine the processing of the current action next time, including: based on the reward value corresponding to the current action and the objective function corresponding to the proximal policy optimization algorithm, optimizing the parameters of the congestion control policy model until the objective function value corresponding to the proximal policy optimization algorithm is maximized, determining the parameters of the current congestion control policy model as the parameters of the optimized congestion control policy model; and determining the optimized congestion control policy model based on the parameters of the optimized congestion control policy model.

[0104] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: determining the optimal current action based on the reward values ​​corresponding to each of the multiple current actions, including: successively calculating the average value of the reward values ​​corresponding to each of the multiple current actions; when the average value of the reward values ​​corresponding to each of the multiple current actions reaches a preset threshold, selecting the current action determined last as the optimal current action.

[0105] An embodiment of the present invention also provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can achieve: obtaining network status parameters of the target computing power network at the current moment, wherein the network status parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay and service flow completion time; inputting the network status parameters at the current moment into the congestion control policy model to determine the current action, wherein the current action represents the change in the length of the congestion window at the current moment; according to the reward function in the congestion control policy model, calculating the reward value corresponding to the current action; based on the reward value corresponding to the current action, optimizing the parameters of the congestion control policy model to determine the optimized congestion control policy model, wherein the optimized congestion control policy model is used for the next determination of the current action; repeating the above steps to obtain the reward values ​​corresponding to each of the multiple current actions; determining the optimal current action according to the reward values ​​corresponding to each of the multiple current actions; adjusting the length of the congestion window based on the optimal current action to obtain a new congestion window length.

[0106] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0107] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0110] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0112] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A congestion window length adjustment method, characterized in that: include: Obtaining network status parameters of the target computing power network at the current moment, wherein the network status parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay, and service flow completion time; Inputting the network state parameters at the current moment into a congestion control strategy model to determine a current action, wherein the current action represents changing the length of the congestion window at the current moment; Calculating a reward value corresponding to the current action according to a reward function in the congestion control strategy model; Based on the reward value corresponding to the current action, the parameters of the congestion control strategy model are optimized to determine the optimized congestion control strategy model, wherein the optimized congestion control strategy model is used for determining the processing of the current action next time; Repeat the above steps to obtain the reward values ​​corresponding to multiple current actions; Determining an optimal current action according to the reward values ​​corresponding to the multiple current actions; Based on the optimal current action, the length of the congestion window is adjusted to obtain a new congestion window length.

2. The method according to claim 1, characterized in that Before inputting the network status parameters at the current moment into the congestion control strategy model, the method includes: The congestion control strategy model is determined based on a preset state space, an action space and the reward function, wherein the state space is determined according to network state parameters, and the action space is used to define a current action.

3. The method according to claim 2, characterized in that include: Obtaining a weight factor, wherein the weight factor represents a weight ratio of the throughput, the packet loss rate, the link delay, and the service flow completion time to the reward value; Based on the weight factors, the reward function is determined.

4. The method according to claim 3, characterized in that The expression of the reward function is as follows: Wherein, O is the throughput of the receiving end of the target computing network at the current moment, O max is the maximum throughput in the historical moment, LP is the packet loss rate at the current moment, LP min is the minimum packet loss rate in the historical moment; l is the average link delay at the current moment, l min is the minimum link delay in the historical moment; FCT is the average completion time of the service flow at the current moment, FCT min is the minimum business flow completion time within the historical moment; a is the weight factor.

5. The method according to claim 1, characterized in that The step of optimizing the parameters of the congestion control strategy model based on the reward value corresponding to the current action to determine the optimized congestion control strategy model, wherein the optimized congestion control strategy model is used for determining the current action next time, includes: Based on the reward value corresponding to the current action and the objective function corresponding to the proximal policy optimization algorithm, the parameters of the congestion control strategy model are optimized until the objective function value corresponding to the proximal policy optimization algorithm is maximized, and the current parameters of the congestion control strategy model are determined as the parameters of the optimized congestion control strategy model; Based on the parameters of the optimized congestion control strategy model, the optimized congestion control strategy model is determined.

6. The method according to claim 1, characterized in that The determining the optimal current action according to the reward values ​​corresponding to the multiple current actions respectively includes: Calculating the average value of the reward values ​​corresponding to each of the multiple current actions one by one; When the average value of the reward values ​​corresponding to the multiple current actions reaches a preset threshold, the current action determined last time is selected as the optimal current action.

7. A congestion window length adjustment device, characterized in that: include: An acquisition module, used to acquire network status parameters of the target computing power network at the current moment, wherein the network status parameters include at least one of the following: congestion window length, throughput, packet loss rate, link delay, and service flow completion time; A first determination module, configured to input the network state parameter at the current moment into a congestion control strategy model to determine a current action, wherein the current action represents a change in the length of the congestion window at the current moment; A calculation module, used to calculate the reward value corresponding to the current action according to the reward function in the congestion control strategy model; A second determination module is used to optimize the parameters of the congestion control strategy model based on the reward value corresponding to the current action, and determine the optimized congestion control strategy model, wherein the optimized congestion control strategy model is used for determining the processing of the current action next time; A repeating module is used to repeat the above steps to obtain the reward values ​​corresponding to multiple current actions; A third determination module, configured to determine the optimal current action according to the reward values ​​corresponding to the multiple current actions; An adjustment module is used to adjust the length of the congestion window based on the optimal current action to obtain a new congestion window length.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the congestion window length adjustment method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the congestion window length adjustment method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the congestion window length adjustment method described in any one of claims 1 to 6 is implemented.

Images