Method and device for adjusting data transmission rate and training congestion control model

Abstract

The embodiments of the present specification provide a method and apparatus for adjusting the data transmission rate. In the method for adjusting the data transmission rate, a network parameter corresponding to at least one target data packet is extracted from multiple data packets received within a monitoring time interval; based on a target congestion control algorithm, network overload status information and a first candidate data transmission rate are obtained according to the extracted network parameters corresponding to each target data packet; the extracted network parameters corresponding to each target data packet and the network overload status information are provided to a congestion control model trained based on a reinforcement learning algorithm to obtain a second candidate data transmission rate; and a new data transmission rate is determined based on the first candidate data transmission rate and the second candidate data transmission rate.

Description

Technical Field

[0001] The embodiments of the present specification generally relate to the field of computer technology, and more particularly to methods and devices for adjusting data transmission rate and training congestion control models. Background Art

[0002] With the rapid development of Internet technology, new business areas such as cloud games, cloud desktops, VR (Virtual Reality) broadcasting, surgical robots, and new scenarios in live broadcasting such as live broadcasting and flash sales are constantly emerging, which puts forward new requirements for the real-time transmission of data such as video, audio, text, and images. Summary of the invention

[0003] In view of the above, the embodiments of this specification provide a method and device for adjusting the data transmission rate and training the congestion control model. By using this method and device, the data transmission rate can be adjusted more accurately to adapt to the current network conditions.

[0004] According to one aspect of an embodiment of the present specification, a method for adjusting a data transmission rate is provided, comprising: extracting a network parameter corresponding to at least one target data packet from a plurality of data packets received within a monitoring time interval; based on a target congestion control algorithm, obtaining network overload status information and a first candidate data transmission rate according to the network parameters corresponding to each extracted target data packet; providing the extracted network parameters corresponding to each target data packet and the network overload status information to a congestion control model trained based on a reinforcement learning algorithm to obtain a second candidate data transmission rate; and determining a new data transmission rate based on the first candidate data transmission rate and the second candidate data transmission rate.

[0005] According to another aspect of an embodiment of the present specification, a method for training a congestion control model is provided, comprising: looping through the following model training process using a training sample set until a training end condition is met, wherein each training sample in the training sample set comprises a network parameter sequence corresponding to a monitoring time interval and a corresponding data transmission rate: determining network overload status information corresponding to the network parameter sequence of a current training sample based on a target congestion control algorithm; determining a rate adjustment direction of the current training sample based on the network overload status information; generating state feature sequence data of the current training sample based on the network parameter sequence and the corresponding rate adjustment direction of the current training sample; and converting the state feature sequence of the current training sample into state feature sequence data of the current training sample. The data is provided to the current congestion control model, and the rate change parameters of the current training sample are determined from the candidate rate change parameter set; the data sending rate of the current training sample is updated according to the determined rate change parameters to obtain the next data sending rate; network data transmission is performed according to the next data sending rate to obtain the next network parameters corresponding to the current training sample; a reward value is determined according to the next network parameters corresponding to the current training sample using a reward function; and in response to not meeting the training end condition, the parameters of the current congestion control model are adjusted based on the determined reward value based on the reinforcement learning algorithm, wherein the congestion control model after the model parameter adjustment serves as the current congestion control model for the next model training process.

[0006] According to another aspect of an embodiment of the present specification, there is provided an apparatus for adjusting a data transmission rate, comprising: a monitoring extraction unit, configured to extract a network parameter corresponding to at least one target data packet from a plurality of data packets received within a monitoring time interval; a first candidate unit, configured to obtain network overload status information and a first candidate data transmission rate based on a target congestion control algorithm and the network parameters corresponding to each extracted target data packet; a second candidate unit, configured to provide the extracted network parameters corresponding to each target data packet and the network overload status information to a congestion control model trained based on a reinforcement learning algorithm to obtain a second candidate data transmission rate; and a rate determination unit, configured to determine a new data transmission rate based on the first candidate data transmission rate and the second candidate data transmission rate.

[0007] According to another aspect of an embodiment of the present specification, there is provided an apparatus for training a congestion control model, the apparatus being configured to cyclically execute the following model training process via a training unit using a training sample set until a training end condition is met, wherein each training sample in the training sample set includes a network parameter sequence corresponding to a monitoring time interval and a corresponding data transmission rate, and the training unit includes: a state feature generation module, configured to determine network overload state information corresponding to the network parameter sequence of a current training sample based on a target congestion control algorithm; determine a rate adjustment direction of the current training sample according to the network overload state information; generate state feature sequence data of the current training sample according to the network parameter sequence of the current training sample and the corresponding rate adjustment direction; a rate update module, configured to update the current training sample The state feature sequence data is provided to the current congestion control model, and the rate change parameters of the current training sample are determined from the candidate rate change parameter set; the data sending rate of the current training sample is updated according to the determined rate change parameters to obtain the next data sending rate; a reward determination module is configured to perform network data transmission according to the next data sending rate to obtain the next network parameters corresponding to the current training sample; a reward value is determined according to the next network parameters corresponding to the current training sample using a reward function; and the device also includes: a parameter adjustment unit, configured to adjust the parameters of the current congestion control model based on a reinforcement learning algorithm according to the determined reward value in response to failure to meet the training end condition, wherein the congestion control model after the model parameter adjustment serves as the current congestion control model for the next model training process.

[0008] According to another aspect of an embodiment of the present specification, there is provided an apparatus for adjusting a data transmission rate, comprising: at least one processor, and a memory coupled to the at least one processor, the memory storing instructions, which, when executed by the at least one processor, causes the at least one processor to execute the method for adjusting the data transmission rate as described above.

[0009] According to another aspect of an embodiment of the present specification, there is provided an apparatus for training a congestion control model, comprising: at least one processor, and a memory coupled to the at least one processor, the memory storing instructions, which, when executed by the at least one processor, causes the at least one processor to execute the method for training a congestion control model as described above.

[0010] According to another aspect of an embodiment of the present specification, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the method for adjusting the data transmission rate and / or the method for training a congestion control model as described above is implemented.

[0011] According to another aspect of an embodiment of the present specification, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to implement the method for adjusting a data transmission rate and / or the method for training a congestion control model as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] A further understanding of the nature and advantages of the content of this specification may be achieved by referring to the following drawings. In the drawings, similar components or features may have the same reference numerals.

[0013] Figure 1 An exemplary architecture of a method and apparatus for adjusting a data transmission rate and a method and apparatus for training a congestion control model according to an embodiment of the present specification is shown.

[0014] Figure 2 A flowchart showing an example of a method for adjusting a data transmission rate according to an embodiment of the present specification is shown.

[0015] Figure 3 A flowchart showing an example of a process of determining a second candidate data transmission rate according to an embodiment of the present specification.

[0016] Figure 4 A flowchart showing yet another example of a process for determining a second candidate data transmission rate according to an embodiment of the present specification.

[0017] Figure 5 A flowchart of another example of a method for adjusting a data transmission rate according to an embodiment of the present specification is shown.

[0018] Figure 6 A flowchart of another example of a method for adjusting a data transmission rate according to an embodiment of the present specification is shown.

[0019] Figure 7 A flowchart showing an example of a process for determining a new data transmission rate according to an embodiment of the present specification.

[0020] Figure 8 A flowchart showing another example of a process for determining a new data transmission rate according to an embodiment of the present specification.

[0021] Fig. 9 A timing diagram showing an example of an application scenario of a method for adjusting a data transmission rate according to an embodiment of this specification.

[0022] Fig.10 A flowchart of an example of a method for training a congestion control model according to an embodiment of the present specification is shown.

[0023] Fig.11 A block diagram showing an example of an apparatus for adjusting a data transmission rate according to an embodiment of the present specification.

[0024] Fig.12 A block diagram showing an example of a second candidate unit in an apparatus for adjusting a data transmission rate according to an embodiment of this specification.

[0025] Fig.13 A block diagram showing an example of a rate determination unit in an apparatus for adjusting a data transmission rate according to an embodiment of this specification.

[0026] Fig.14 A block diagram showing another example of an apparatus for adjusting a data transmission rate according to an embodiment of the present specification.

[0027] Fig.15 A block diagram showing an example of an apparatus for training a congestion control model according to an embodiment of this specification.

[0028] Fig.16 A block diagram showing an example of an apparatus for adjusting a data transmission rate according to an embodiment of the present specification.

[0029] Fig.17 A block diagram showing an example of an apparatus for training a congestion control model according to an embodiment of this specification. DETAILED DESCRIPTION

[0030] The subject matter described herein will be discussed below with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the embodiments of this specification. Various processes or components may be omitted, substituted or added to each example as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0031] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0032] In this specification, the term "congestion control" may refer to adjusting the number of data sent at a time by a certain strategy to alleviate network congestion. "Congestion control algorithm" may refer to various algorithms that can achieve the above functions.

[0033] In this specification, the term "WebRTC" may refer to a real-time communication technology that allows network applications or sites to establish peer-to-peer connections between browsers without the aid of an intermediary to achieve the transmission of video streams and / or audio streams or any other data.

[0034] In this specification, the term "Reinforcement Learning (RL)" is one of the paradigms and methodologies of machine learning, which is used to describe and solve the problem of maximizing rewards or achieving specific goals through learning strategies in the process of interaction between an agent and the environment. The term "reward function", as one of the elements of "reinforcement learning", can reflect the probability expectation value of the cumulative sum of scalar reward signals received by the agent (called "return"), and can also be regarded as the target function that the agent is constantly approaching.

[0035] The method and apparatus for adjusting the data transmission rate and the method and apparatus for training the congestion control model according to the embodiments of this specification will be described in detail below in conjunction with the accompanying drawings.

[0036] Figure 1 An exemplary architecture 100 for a method and apparatus for adjusting a data transmission rate and for training a congestion control model according to an embodiment of the present specification is shown.

[0037] exist Figure 1 In the embodiment, the network 110 is used to interconnect terminal devices 120 and 130 and an application server 140 .

[0038] The network 110 may be any type of network capable of interconnecting network entities. The network 110 may be a single network or a combination of various networks. In terms of coverage, the network 110 may be a local area network (LAN), a wide area network (WAN), etc. In terms of carrier media, the network 110 may be a wired network, a wireless network, etc. In terms of data exchange technology, the network 110 may be a circuit switching network, a packet switching network, etc.

[0039] The terminal devices 120 and 130 may be any type of electronic computing device that can connect to the network 110, access a server or website on the network 110, process data or signals, etc. For example, the terminal devices 120 and 130 may be desktop computers, laptop computers, tablet computers, smart phones, etc. Figure 1Only two terminal devices are shown in FIG. 1 , but it should be understood that a different number of terminal devices may be connected to the network 110 .

[0040] In one embodiment, the terminal devices 120 and 130 may be used by users. The terminal devices 120 and 130 may include application clients (e.g., application clients 121 and 131) that may provide various services to users. In some cases, the application clients 121 and 131 may interact with the application server 140. For example, the application clients 121 and 131 may transmit messages input by users to the application server 140, and receive responses associated with the messages from the application server 140. In this document, "message" may refer to any input information, such as streaming media data 1211 input by users, etc.

[0041] The application server 140 may be connected to the model training server 150. The model training server 150 may use the training samples to perform model training to obtain a congestion control model. It is understood that the application server 140 may also perform model training locally, in which case the model training server 150 may not be needed.

[0042] It should be understood that Figure 1 All network entities shown in the figure are exemplary. Any other network entities may be involved in the architecture 100 according to specific application requirements.

[0043] Figure 2 A flow chart of a method 200 for adjusting a data transmission rate according to an embodiment of the present specification is shown.

[0044] like Figure 2 As shown, at 210, a network parameter corresponding to at least one target data packet is extracted from a plurality of data packets received within a monitoring time interval.

[0045] In this embodiment, multiple data packets can be received within a monitoring time interval. In one example, the monitoring time interval can be a preset fixed interval, or an integer multiple of the round-trip time (RTT). It should be noted that since the round-trip time may fluctuate with changes in network conditions, in actual applications, for example, the average value of multiple round-trip time delays collected over a period of time can be used as the round-trip time delay based on the above monitoring time interval.

[0046] The network parameters corresponding to at least one target data packet can be extracted from the above multiple data packets. In one example, the target data packet can be a data packet randomly selected from all the above data packets, or a data packet obtained by selecting one every several data packets, or a number of data packets closest to the current time (for example, the 10 most recently received data packets). In one example, the network parameters can include various data used to reflect the network congestion status. In one example, the network parameters can include but are not limited to at least one of the following: throughput, delay, and packet loss rate.

[0047] At 220 , based on the target congestion control algorithm, network overload status information and a first candidate data transmission rate are obtained according to the extracted network parameters corresponding to each target data packet.

[0048] In this embodiment, the network overload status information and the first candidate data transmission rate can be obtained based on the target congestion control algorithm and the network parameters corresponding to each target data packet extracted in the above step 210. The network overload status information can be used to indicate the current congestion status of the network. In an example, -1, 0, and 1 can be used to indicate that the network is currently in an idle state, a normal traffic state, and an overload state, respectively. In an example, the above target congestion control algorithm can be selected according to the corresponding network transmission protocol (such as TCP, UDP, etc.). The first candidate data transmission rate can be used to indicate the recommended value of the data transmission rate calculated according to the target congestion control algorithm.

[0049] In one example, the network overload status information and the first candidate data transmission rate may correspond to a network parameter sequence formed by network parameters corresponding to a plurality of target data packets. For example, the extracted network parameters corresponding to each target data packet may be represented as {W1, W2, W3, W4, W5, W6, W7, W8}, where W i (i=1,2,…,8) can be used to represent the network parameter (set) of the i-th target data packet. Thus, W4, W5, W6, W7, and W8 can be used to obtain the network overload status information corresponding to target data packets 4-8 (for example, it can be represented by st8) and the first candidate data transmission rate. Similarly, W3, W4, W5, W6, and W7 can be used to obtain the network overload status information corresponding to target data packets 3-7 (for example, it can be represented by st7) and the first candidate data transmission rate, and so on.

[0050] At 230 , the extracted network parameters and network overload status information corresponding to each target data packet are provided to a congestion control model trained based on a reinforcement learning algorithm to obtain a second candidate data transmission rate.

[0051] In this embodiment, a congestion control model trained based on a reinforcement learning algorithm can be used to predict a second candidate data transmission rate as a recommended value of the data transmission rate based on network parameters and network overload status information corresponding to each extracted target data packet.

[0052] Optionally, refer to Figure 3 , Figure 3 FIG. 1 is a flowchart showing an example of a process 300 for determining a second candidate data transmission rate according to an embodiment of the present specification.

[0053] like Figure 3 As shown, at 310, a state sequence corresponding to the monitoring time interval is generated according to the obtained network overload state information and corresponding network parameters among the extracted network parameters corresponding to each target data packet.

[0054] In this embodiment, each state in the state sequence corresponding to the monitoring time interval can generate a corresponding state according to the obtained network overload state information and the network parameters corresponding to the target data packet generating the network overload state information. In one example, as described above, a corresponding state (for example, s8) can be generated according to the network overload state information (for example, st8) and the network parameters corresponding to the target data packet generating the network overload state information (for example, W4, W5, W6, W7, W8). Similarly, a corresponding state (for example, s7) can be generated according to the network overload state information (for example, st7) and the network parameters corresponding to the target data packet generating the network overload state information (for example, W3, W4, W5, W6, W7), and so on. The generated state sequence S corresponding to the monitoring time interval can be represented by {s5, s6, s7, s8}, for example.

[0055] At 320, the state sequence is provided to a congestion control model trained based on a reinforcement learning algorithm to obtain a rate change parameter.

[0056] In this embodiment, the congestion control model trained based on the reinforcement learning algorithm can be used to predict the rate change parameter according to the above state sequence. In one example, the above congestion control model can limit the value of the rate change parameter to the interval [-1,1] with the help of an activation function. It can be understood that different value ranges of the rate change parameter can be obtained according to different settings of the output layer of the above congestion control model, which is not limited here.

[0057] At 330, the data transmission rate corresponding to the at least one target data packet is updated according to the rate change parameter to obtain a second candidate data transmission rate.

[0058] In this embodiment, the data transmission rate corresponding to at least one target data packet can be updated according to the rate change parameter in various ways to obtain a second candidate data transmission rate. The data transmission rate corresponding to at least one target data packet can refer to the data transmission rate used when sending the target data packet. In one example, the rate change parameter can be multiplied by the data transmission rate corresponding to at least one target data packet to obtain the second candidate data transmission rate. In one example, the range of the second candidate data transmission rate can be pre-defined, such as [0.1Mbps, 6Mbps]. When the updated rate value exceeds the above range, the value of the second candidate data transmission rate can be determined as the corresponding endpoint value.

[0059] Based on this, this solution provides a solution that uses a congestion control model to obtain a rate change parameter based on a state sequence, and then determines the second candidate data sending rate based on the rate change parameter and the corresponding basic data sending rate. This allows the state sequence to be used to characterize the historical state as effectively as possible, and provides a more flexible rate adjustment method.

[0060] Optionally, continue to refer to Figure 4 , Figure 4 A flowchart showing another example of a process 400 for determining a second candidate data transmission rate according to an embodiment of the present specification.

[0061] like Figure 4 As shown, at 410, a parameter change value corresponding to the rate change parameter is determined according to a preset exponential function.

[0062] In one example, the preset exponential function may be y=2 x When the rate change parameter is a, the corresponding parameter change value can be 2 a It can be understood that the exponential function can be flexibly set according to actual application requirements and is not limited here.

[0063] At 420 , a product of the parameter change value and the data transmission rate corresponding to at least one target data packet is determined as a second candidate data transmission rate.

[0064] In this embodiment, the data transmission rate corresponding to at least one target data packet may refer to the data transmission rate used when the target data packet is transmitted. In one example, the range of the second candidate data transmission rate may be predefined, such as [0.1Mbps, 6Mbps]. When the updated rate value exceeds the above range, the value of the second candidate data transmission rate may be determined as the corresponding endpoint value within the above range.

[0065] Based on this, this solution can provide a more flexible rate adjustment space through the combination of exponential function and rate change parameter (for example, the rate can be reduced to half of the original rate at the lowest and doubled at the highest). Compared with the control strategy of the prior art, when encountering network congestion, the data transmission rate is reduced exponentially, resulting in a sudden drop in the amount of received data, which can easily cause the quality of the played video to deteriorate; and when the bandwidth is restored, the data transmission rate is slowly increased, which makes it difficult to quickly occupy the new available bandwidth and causes a waste of resources. This solution can adjust the data transmission rate more timely, thereby helping to improve the quality of real-time data transmission.

[0066] Back to Figure 2 , at 240, a new data transmission rate is determined based on the first candidate data transmission rate and the second candidate data transmission rate.

[0067] In this embodiment, based on the first candidate data transmission rate and the second candidate data transmission rate, the new data transmission rate can be determined in various ways. In one example, the smaller value of the first candidate data transmission rate and the second candidate data transmission rate can be determined as the new data transmission rate. In one example, the average value of the first candidate data transmission rate and the second candidate data transmission rate can be determined as the new data transmission rate.

[0068] Optionally, the network parameter corresponding to at least one target data packet extracted from the multiple data packets received within the monitoring time interval may include a round trip delay. According to the obtained round trip delay, the following queue accumulation detection and emptying process may also be performed:

[0069] In step 1, it is determined whether there is currently queue message accumulation according to whether the round-trip delay is greater than a preset multiple of the minimum delay in a second time window corresponding to the monitoring time interval.

[0070] In this embodiment, a corresponding second time window may be set for the monitoring time interval. In one example, the monitoring time interval may be one RTT, and the corresponding second time window may be a time window corresponding to the network parameter value corresponding to the most recently received (e.g., 100, 200) data packets retained up to the current monitoring time interval. Thus, the delay corresponding to the most recently received (e.g., 100, 200) data packets retained up to the current monitoring time interval may be retained and the minimum value thereof may be determined as the minimum delay. In one example, the above-mentioned preset multiple may be, for example, 1.5, 1.6, 1.8, etc.

[0071] In this embodiment, when it is determined that the round-trip delay is greater than a preset multiple of the minimum delay in the second time window corresponding to the monitoring time interval, it can be determined that there is currently queue message accumulation.

[0072] In step 2, in response to queue message accumulation and a data transmission rate corresponding to at least one target data packet being greater than a data reception rate, a new data transmission rate is determined to be a value less than the data reception rate to alleviate queue message accumulation.

[0073] In one example, the data transmission rate corresponding to at least one target data packet may refer to the data transmission rate used when the target data packet is transmitted. In one example, the above-mentioned data reception rate may be the currently adopted data reception rate, or the data reception rate adopted when the target data packet is received. In one example, the new data transmission rate may be determined as a value less than the data reception rate in various ways, for example, the new data transmission rate may be determined as the product of the data reception rate and a coefficient less than 1 (for example, 0.85, 0.9, etc.).

[0074] It should be noted that the timing of the triggering of the above queue accumulation detection and emptying process can be designed according to the actual application scenario. In one example, the above queue accumulation detection and emptying process can be performed every monitoring time interval. On the premise that it is determined that there is no queue message accumulation, a new data transmission rate is determined based on the first candidate data transmission rate and the second candidate data transmission rate. On the premise that it is determined that there is queue message accumulation, the new data transmission rate is determined according to the above step 2.

[0075] Based on this, we can determine whether there is queuing in the bottleneck link queue by comparing the delay of the current link with a certain multiple of the lowest delay of the link in the corresponding time window, and then provide a solution for reducing the sending rate to quickly drain the queue and reduce the queuing delay.

[0076] Reference below Figure 5 , Figure 5 A flowchart of another example of a method 500 for adjusting a data transmission rate according to an embodiment of the present specification is shown.

[0077] like Figure 5 As shown, at 510, a network parameter corresponding to at least one target data packet is extracted from a plurality of data packets received within a monitoring time interval.

[0078] In this embodiment, the network parameters may include latency, packet loss rate and throughput, wherein the throughput may include the amount of data successfully sent and received by the network in a unit time.

[0079] At 520 , based on the target congestion control algorithm, network overload status information and a first candidate data transmission rate are obtained according to the extracted network parameters corresponding to each target data packet.

[0080] In this embodiment, the target congestion control algorithm may include a congestion control algorithm applied to WebRTC. In one example, the target congestion control algorithm may include a GCC (Google Live Stream Congestion Control) algorithm. The congestion control algorithm applicable to WebRTC may obtain network overload status information and a first candidate data transmission rate based on network parameters corresponding to each extracted target data packet.

[0081] At 530, a maximum throughput value in a first time window and a maximum delay value in a second time window corresponding to the monitoring time interval are obtained.

[0082] In this embodiment, a corresponding first time window and a second time window may be set for the monitoring time interval. In one example, the monitoring time interval may be one RTT, and the corresponding first time window and the second time window may be time windows corresponding to the most recent (e.g., 100, 200) network parameter values ​​retained up to the current monitoring time interval. Thus, the most recent (e.g., 10) throughputs retained up to the current monitoring time interval may be retained and the maximum value thereof may be determined as the throughput maximum value. The throughput maximum value may include the maximum throughput or the minimum throughput. The most recent (e.g., 200) delays retained up to the current monitoring time interval may be retained and the maximum value thereof may be determined as the delay maximum value. The delay maximum value may include the minimum delay or the maximum delay.

[0083] At 540, for each obtained network overload status information, the relative throughput and relative delay corresponding to the network overload status information are determined according to the network overload information and the network parameters corresponding to the target data packet based on which the network overload status information is generated, as well as the corresponding maximum throughput and maximum delay.

[0084] In one example, each network overload status information may correspond to a target data packet based on which the network overload status information is generated. For example, as mentioned above, the network overload status information (e.g., can be represented by st8) can be generated based on the network parameters corresponding to target data packets 4 to 8 (e.g., can be represented by W4, W5, W6, W7, W8). Accordingly, the relative throughput and relative delay corresponding to the network overload status information can be determined based on the network parameters corresponding to the target data packet 8 (e.g., can be represented by W8) and the corresponding maximum throughput value (e.g., the maximum value of the throughput corresponding to the 10 data packets received before the target data packet 8) and the maximum delay value (e.g., the maximum value of the delay corresponding to the 200 data packets received before the target data packet 8).

[0085] In one example, the relative throughput corresponding to the network overload status information (for example, it can be represented by st8) can be the ratio between the throughput corresponding to the corresponding target data packet (for example, target data packet 8) and the corresponding maximum throughput, or the ratio between the minimum throughput and the throughput corresponding to the corresponding target data packet (for example, target data packet 8). In one example, the relative delay corresponding to the network overload status information (for example, it can be represented by st8) can be the ratio between the minimum delay and the delay corresponding to the corresponding target data packet (for example, target data packet 8), or the ratio between the delay corresponding to the corresponding target data packet (for example, target data packet 8) and the corresponding maximum delay.

[0086] Similarly, it can be understood that the network overload status information (for example, it can be represented by st7) can be generated according to the network parameters corresponding to the target data packets 3 to 7 (for example, it can be represented by W3, W4, W5, W6, W7). Accordingly, the relative throughput and relative delay corresponding to the network overload status information (for example, it can be represented by st7) can be determined according to the network parameters corresponding to the target data packet 7 (for example, it can be represented by W7) and the corresponding maximum throughput value (for example, the maximum value of the throughput corresponding to the 10 data packets received before the target data packet 7) and the maximum delay value (for example, the maximum value of the delay corresponding to the 200 data packets received before the target data packet 7). The relative throughput and relative delay corresponding to the network overload status information st6 and st5 can also be obtained.

[0087] At 550, various states are generated according to the corresponding relative throughput, relative delay, packet loss rate, and network overload state information.

[0088] In this embodiment, the relative throughput, relative delay, packet loss rate corresponding to each network overload state information and the network overload state information can be combined to obtain each state. Optionally, the relative throughput, relative delay, packet loss rate, delay gradient corresponding to each network overload state information and the network overload state information can also be combined to obtain each state. The meaning of the delay gradient can refer to the corresponding description in the GCC algorithm.

[0089] At 560, the generated states are arranged in the order of receiving the corresponding target data packets to obtain a state sequence corresponding to the monitoring time interval.

[0090] In one example, the state sequence S corresponding to the monitoring time interval can be represented by {s1, s2, s3, s4, s5, s6, s7, s8}. n ={thr rela ,delay rela ,loss rate,st}. Where n=1,2,…,8. thr rela ,delay rela ,loss rate ,st can be used to represent the relative throughput, relative delay, packet loss rate and network overload status information corresponding to target data packets 1 to 8. In an example, s n ={thr rela ,delay rela ,loss rate ,trendline,st}. Among them, trendline can be used to represent the delay gradient corresponding to target data packets 1 to 8 respectively.

[0091] At 570, the state sequence is provided to a congestion control model trained based on a reinforcement learning algorithm to obtain a rate change parameter.

[0092] At 580, the data transmission rate corresponding to the at least one target data packet is updated according to the rate change parameter to obtain a second candidate data transmission rate.

[0093] At 590, a new data transmission rate is determined based on the first candidate data transmission rate and the second candidate data transmission rate.

[0094] It should be noted that the above steps 510, 520, 570, 580, and 590 can refer to the corresponding descriptions of steps 210, 220, 320, 330, and 240 in the aforementioned embodiments, and will not be repeated here.

[0095] Based on this, this solution can process the network parameters to obtain a state sequence composed of state values ​​that can more accurately reflect the network status. It can fully reflect the characteristics of different network conditions and reduce the noise caused by irrelevant variables, thereby helping to improve the accuracy of the final determined data transmission rate.

[0096] Continue to refer Figure 6 , Figure 6 A flowchart of another example of a method 600 for adjusting a data transmission rate according to an embodiment of the present specification is shown.

[0097] like Figure 6 As shown, at 610, a network parameter corresponding to at least one target data packet is extracted from a plurality of data packets received within a monitoring time interval.

[0098] At 620 , based on the target congestion control algorithm, network overload status information and a first candidate data transmission rate are obtained according to the extracted network parameters corresponding to each target data packet.

[0099] At 630, the extracted network parameters and network overload status information corresponding to each target data packet are provided to a congestion control model trained based on a reinforcement learning algorithm to obtain a second candidate data transmission rate.

[0100] At 640, a difference between the first candidate data transmission rate and the second candidate data transmission rate is determined.

[0101] In an example, the difference value may be used to indicate an absolute value of a difference between the first candidate data transmission rate and the second candidate data transmission rate.

[0102] At 650 , according to whether the difference is greater than a preset threshold, a new data transmission rate is determined based on the first candidate data transmission rate and the second candidate data transmission rate.

[0103] In one example, in response to the difference being not greater than a preset threshold, the smaller value of the first candidate data transmission rate and the second candidate data transmission rate is determined as the new data transmission rate, so that the new data transmission rate can be quickly determined under the premise that the two algorithm decisions are similar.

[0104] Optionally, continue to refer to Figure 7 , Figure 7 FIG. 1 is a flowchart showing an example of a process 700 for determining a new data transmission rate according to an embodiment of the present specification.

[0105] At 710 , it is determined whether the difference is greater than a preset threshold.

[0106] If the above determination is yes, at 720, network data transmission is performed according to the first candidate data transmission rate and the second candidate data transmission rate, respectively, to obtain network utility parameters corresponding to the first candidate data transmission rate and the second candidate data transmission rate, respectively.

[0107] In this embodiment, the network utility parameter may include various parameters required for evaluating the performance of the data transmission rate. The data transmitting end may send a data packet to the data receiving end, and the data receiving end may obtain the network utility parameter corresponding to the data packet when receiving the data packet. It can be understood that when the data transmission rate when sending the data packet is different, the network utility parameter corresponding to the data packet may also be different. In one example, the network utility parameter may include but is not limited to at least one of the following: delay gradient, packet loss rate.

[0108] Optionally, the smaller value of the first candidate data transmission rate and the second candidate data transmission rate may be used as the data transmission rate to perform network data transmission, and obtain the corresponding network utility parameter. After the verification time interval, the larger value of the first candidate data transmission rate and the second candidate data transmission rate may be used as the data transmission rate to perform network data transmission, and obtain the corresponding network utility parameter. The verification time interval may be half of the monitoring time interval.

[0109] Based on this, by setting the data transmission rate of the previous verification time interval to the smaller of the two, and setting the data transmission rate of the next verification time interval to the larger of the two, the two rates can be prevented from interfering with each other. For example, if the data transmission rate of the previous verification time interval is too high, resulting in the accumulation of queues, it may affect the accuracy of the utility evaluation of the data transmission rate of the next verification time interval.

[0110] At 730 , utility values ​​corresponding to the first candidate data transmission rate and the second candidate data transmission rate are determined according to the utility function and the obtained network utility parameter.

[0111] In this embodiment, the utility value can be used to evaluate the performance of the data transmission rate. In one example, the utility function can be Wherein, x can be used to represent a data transmission rate (such as a first candidate data transmission rate or a second candidate data transmission rate), It can be used to represent the delay gradient, and L can be used to represent the packet loss rate. The values ​​of t, α, β, and γ can be set according to the actual scenario, for example, t = 0.9, α = 1, β = 900, and γ = 11.35.

[0112] At 740, according to the determined utility values, a candidate data transmission rate corresponding to the maximum utility value is determined as a new data transmission rate.

[0113] In one example, if the utility value determined to correspond to the first candidate data transmission rate is greater than the utility value corresponding to the second candidate data transmission rate, the first candidate data transmission rate may be determined as the new data transmission rate. In one example, if the utility value determined to correspond to the first candidate data transmission rate is less than the utility value corresponding to the second candidate data transmission rate, the second candidate data transmission rate may be determined as the new data transmission rate.

[0114] Continue to refer Figure 8 , Figure 8 A flowchart showing another example of a process 800 for determining a new data transmission rate according to an embodiment of the present specification is shown.

[0115] At 810 , it is determined whether the difference is greater than a preset threshold.

[0116] If the above determination is yes, at 820, network data transmission is performed according to the first candidate data transmission rate and the second candidate data transmission rate, respectively, to obtain network utility parameters corresponding to the first candidate data transmission rate and the second candidate data transmission rate, respectively.

[0117] At 830 , a network utility parameter corresponding to the third candidate data transmission rate is obtained according to the extracted network parameters corresponding to each target data packet.

[0118] In this embodiment, the third candidate data transmission rate is a data transmission rate corresponding to the at least one target data packet, for example, a data transmission rate when the target data packet is sent. In one example, multiple target data packets are sent at the third candidate data transmission rate within the monitoring time interval, and the delay gradient can be determined according to the delays respectively corresponding to the multiple target data packets, thereby obtaining a network utility parameter corresponding to the third candidate data transmission rate.

[0119] At 840 , utility values ​​corresponding to the first candidate data transmission rate, the second candidate data transmission rate, and the third candidate data transmission rate are determined according to the utility function and the obtained network utility parameter.

[0120] In this embodiment, reference may be made to the related description of the aforementioned step 730 .

[0121] At 850, a candidate data transmission rate corresponding to a maximum utility value among the first candidate data transmission rate, the second candidate data transmission rate, and the third candidate data transmission rate is determined as a new data transmission rate.

[0122] Based on this, the network performance corresponding to different data transmission rates can be calculated according to the designed utility function, which helps to select a more reasonable data transmission rate.

[0123] Further references Fig. 9 , Fig. 9 A timing diagram showing an example of an application scenario 900 of a method for adjusting a data transmission rate according to an embodiment of this specification.

[0124] At 910, the sender sends data at a rate x prev Send a data packet to the receiving end. prev Can refer to the current data transmission rate. In one example, x prevIt can be the data transmission rate determined by the method for adjusting the data transmission rate in the previous round. At 920, after receiving the data packet, the receiving end can extract the required network parameters. Afterwards, the receiving end can obtain the first candidate data transmission rate x based on the target congestion control algorithm and the congestion control model trained based on the reinforcement learning algorithm. c and the second candidate data transmission rate x r And x c and x r As the rate to be verified, it is fed back to the sender. At 930, assuming |x c -x r |>θ (θ is the preset threshold) and x c <x r , the sender first sends the first candidate data at rate x c At 940, after half a round trip delay, the receiving end can receive the first candidate data transmission rate x. c The data packets sent can be obtained with the first candidate data sending rate x c The corresponding network utility parameter. According to the obtained network utility parameter, the receiving end can determine the first candidate data transmission rate x c The corresponding utility value u c At 950, at the first candidate data transmission rate x c After sending data packets to the receiver for half the round trip delay, the sender can send data at the second candidate data rate x r At 960, similarly, after half a round trip delay, the receiving end can receive the data packet at the second candidate data transmission rate x. r The data packets sent can be obtained with the second candidate data sending rate x r The corresponding network utility parameter. According to the obtained network utility parameter, the receiving end can determine the second candidate data transmission rate x r The corresponding utility value u r Optionally, the receiving end may also determine the current data transmission rate x prev The corresponding utility value u prev At 970, the receiving end may determine the rate corresponding to the maximum value among the determined utility values ​​as a new data transmission rate (eg, x r ). The receiving end may send the determined new data transmission rate to the transmitting end. At 980, the transmitting end may send the determined new data transmission rate (eg, x r ) sends a data packet to the receiving end to complete the adjustment of the data transmission rate. Optionally, at the second candidate data transmission rate x rThe sender can still send data packets at the current data rate x before sending data packets to the receiver for half a round trip delay and before 980 seconds. prev Send a data packet to the receiving end.

[0125] It can be understood that the adjusted new data transmission rate (eg, c r ) can be used as the current data transmission rate x in the new round of rate adjustment process prev , thereby achieving dynamic adjustment of the online data transmission rate according to the current network conditions.

[0126] use Figure 1-Figure 9 The method for adjusting the data transmission rate disclosed in the method can effectively combine the advantages of the classic congestion control algorithm (such as the GCC algorithm designed by Google and applied to the WebRTC framework) and the congestion control algorithm based on reinforcement learning. Compared with the existing technology that only uses the packet loss rate and delay gradient as the basis for judging network congestion, which is easily interfered in weak network environments such as those with high random packet loss rate and delay jitter and cannot accurately judge the network congestion situation, it can effectively improve the effect of congestion control. Specifically, the classic congestion control algorithm of this scheme not only provides a reference for the final decision, but also generates a new state quantity that is different from that directly obtained from the data packet. At the same time, the congestion control algorithm based on reinforcement learning can not only use the data directly obtained from the data packet involved in the existing technology, but also combine the above-mentioned new state quantity obtained by the classic congestion control algorithm, thereby helping to generate a more reasonable new data transmission rate. Especially in real-time audio and video services, it can achieve effective improvement of the average transmission bit rate while controlling low latency.

[0127] Reference below Fig.10 , Fig.10 A flowchart of an example of a method 1000 for training a congestion control model according to an embodiment of the present specification is shown.

[0128] The following model training process 1010-1070 is cyclically executed using the training sample set until the training end condition is met.

[0129] In this embodiment, each training sample in the above-mentioned training sample set may include a network parameter sequence corresponding to the monitoring time interval and a corresponding data transmission rate. In one example, within a certain monitoring time interval, the transmitter may send a data packet to the receiver at a certain data transmission rate. The training sample may be generated according to the network parameter sequence corresponding to the multiple data packets received within the above-mentioned monitoring time interval and the above-mentioned data transmission rate. It can be understood that the training sample set can be generated by the data packets received within multiple monitoring time intervals. In one example, the network parameters may include various data for reflecting the network congestion status. In one example, the network parameters may include but are not limited to at least one of the following: throughput, delay, packet loss rate.

[0130] In one example, a batch of training samples can be selected from the above training sample set as the current training sample set to perform the following model training process (iteration).

[0131] At 1010, based on the target congestion control algorithm, network overload status information corresponding to the network parameter sequence of the current training sample is determined.

[0132] It should be noted that the above step 1010 can refer to the above Figure 2 The corresponding description of step 220 in the embodiment will not be repeated here.

[0133] At 1020, a rate adjustment direction of the current training sample is determined according to the network overload status information.

[0134] In one example, when the network overload status information indicates that the network is currently in an idle state, the rate adjustment direction of the current training sample can be determined to be rate increase. In one example, when the network overload status information indicates that the network is currently in an overload state, the rate adjustment direction of the current training sample can be determined to be rate decrease.

[0135] At 1030, state feature sequence data of the current training sample is generated according to the network parameter sequence of the current training sample and the corresponding rate adjustment direction.

[0136] It should be noted that the above step 1030 can refer to the above Figure 3 The corresponding description of step 310 in the embodiment will not be repeated here.

[0137] Optionally, each training sample in the training sample set may further include a maximum throughput value in a first time window and a maximum delay value in a second time window corresponding to the monitoring time interval. Figure 5 The corresponding descriptions of steps 540, 550, and 560 in the embodiment are not repeated here.

[0138] At 1040, the state feature sequence data of the current training sample is provided to the current congestion control model, and the rate change parameter of the current training sample is determined from the candidate rate change parameter set.

[0139] It should be noted that the above step 1040 can refer to the above Figure 3 The corresponding description of step 320 in the embodiment will not be repeated here.

[0140] Optionally, the above-mentioned candidate rate change parameter set may include a candidate speed-up change parameter set and a candidate speed-down change parameter set. In one example, the candidate rate change parameter set may be [-1, 1], the candidate speed-up change parameter set may be [0, 1], and the candidate speed-down change parameter set may be [-1, 0). The state feature sequence data of the current training sample may be provided to the current congestion control model, and the rate change parameters of the current training sample may be determined from the candidate speed-up change parameter set or the candidate speed-down change parameter set consistent with the rate adjustment direction. In one example, when the rate adjustment direction is rate increase, the rate change parameters of the current training sample are determined from the candidate speed-up change parameter set (e.g., [0, 1]). In one example, when the rate adjustment direction is rate reduction, the rate change parameters of the current training sample are determined from the candidate speed-down change parameter set (e.g., [-1, 0)).

[0141] Optionally, in the early stage of the training phase, the candidate speed-up change parameter set or the candidate speed-down change parameter set mapped according to the speed adjustment direction can be used to allow the model to directly learn the value of the speed adjustment amplitude, thereby accelerating the learning efficiency of reinforcement learning in the early stage of training, and effectively avoiding the problem of falling into the local optimum after learning the speed-up and speed-down, and only being able to slowly learn more fine-grained amplitude modulation knowledge, which is often caused by the use of the full set of action space training. After a period of training, the model can also be trained using the full set of action space to learn a more complete rate control mode (i.e., learning the speed-up or speed-down and the specific speed-up and speed-down amplitude at the same time).

[0142] At 1050, the data transmission rate of the current training sample is updated according to the determined rate change parameter to obtain the next data transmission rate.

[0143] At 1060, network data transmission is performed according to the next data transmission rate to obtain the next network parameter corresponding to the current training sample.

[0144] It should be noted that the above steps 1050 and 1060 can refer to the above steps respectively. Figure 3 In the embodiment, steps 330 and Figure 7Corresponding description of step 720 in the embodiment. It can be understood that the acquisition of the "next data transmission rate" and the "corresponding next network parameter" can refer to the acquisition method of the aforementioned "second candidate data transmission rate" and the "corresponding network utility parameter".

[0145] At 1070, a reward value is determined according to the next network parameter corresponding to the current training sample using a reward function.

[0146] In one example, the independent variables involved in the reward function may include the next network parameter corresponding to the current training sample.

[0147] Optionally, the independent variables involved in the reward function may also include a relative throughput determined based on the next network parameter and the throughput maximum value and a relative delay determined based on the next network parameter and the delay maximum value. In one example, the reward function may be:

[0148] reward=delay_metric+bw_metric-hitwall

[0149]

[0150]

[0151]

[0152] Among them, MIN_BAND and MAX_BAND can be used to represent the preset minimum and maximum bandwidths, for example, 0.1Mbps and 6Mbps respectively. send_rate and recv_rate represent the data sending rate and data receiving rate corresponding to the next network parameters corresponding to the current training sample respectively. loss_rate can be used to represent the packet loss rate, and delay can be used to represent the delay. max_recv_rate can be used to represent the maximum value of several (e.g., 10, 20, etc.) data receiving rates in the historical records that are closest to the acquisition time of the next network parameters corresponding to the current training sample. Similarly, min_dalay can be used to represent the maximum value of several (e.g., 100, 200, etc.) delays in the historical records that are closest to the acquisition time of the next network parameters corresponding to the current training sample. The values ​​of λ, μ, and η can be set according to the actual scenario, for example, λ=7, μ=0.7, and η=1.3.

[0153] At 1080, it is determined whether a training end condition is met.

[0154] In one example, whether the training end condition is met can be determined by judging whether the number of iterations reaches a preset number, whether the training duration reaches a preset duration, whether the reward value converges, etc.

[0155] At 1090 , in response to the training end condition not being satisfied, adjusting parameters of the current congestion control model based on a reinforcement learning algorithm according to the determined reward value.

[0156] In this embodiment, in response to not meeting the training end condition, in one example, a gradient can be determined based on a reinforcement learning algorithm according to the determined reward value, and then an adjustment value can be determined in combination with a learning rate, thereby adjusting the network parameters of the current congestion control model. The congestion control model after the model parameter adjustment can serve as the current congestion control model for the next model training process. Afterwards, the current training sample can be re-determined using the above training sample set, and the model training process 1010-1070 can be continued until the training end condition is met.

[0157] use Fig.10 The method for training a congestion control model disclosed in the present invention can combine the obtained network parameter sequence with the network overload status information obtained by other congestion control algorithms, generate state feature sequence data with better representation effect and no significant increase in training difficulty by selecting state quantities as the input vector of the congestion control model, and use the reward function to determine the reward value of each action, and implement the training of the congestion control model based on the reinforcement learning algorithm, thereby optimizing the training process of the model and improving the training efficiency.

[0158] Continue to refer Fig.11 , Fig.11 FIG. 1 is a block diagram showing an example of an apparatus 1100 for adjusting a data transmission rate according to an embodiment of the present specification. The apparatus embodiment can be used with Figure 2-Figure 9 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0159] like Fig.11 As shown, the apparatus 1100 for adjusting a data transmission rate may include a monitoring extraction unit 1110 , a first candidate unit 1120 , a second candidate unit 1130 and a rate determination unit 1140 .

[0160] The monitoring extraction unit 1110 is configured to extract network parameters corresponding to at least one target data packet from multiple data packets received within the monitoring time interval. The operation of the monitoring extraction unit 1110 can refer to the above Figure 2 The operation of 210 is described.

[0161] The first candidate unit 1120 is configured to obtain network overload status information and a first candidate data transmission rate based on the target congestion control algorithm and the network parameters corresponding to each target data packet extracted. The operation of the first candidate unit 1120 can refer to the above Figure 2 The operation of 220 is described.

[0162] The second candidate unit 1130 is configured to provide the network parameters corresponding to each extracted target data packet and the network overload status information to the congestion control model trained based on the reinforcement learning algorithm to obtain the second candidate data transmission rate. The operation of the second candidate unit 1130 can refer to the above Figure 2 The operation of 230 is described.

[0163] In one example, optionally, see Fig.12 , Fig.12 A block diagram showing an example of a second candidate unit 1200 in an apparatus for adjusting a data transmission rate according to an embodiment of this specification.

[0164] like Fig.12 As shown, the second candidate unit 1200 may include a state generating module 1210 , a rate changing module 1220 and a candidate rate module 1230 .

[0165] The state generation module 1210 is configured to generate a state sequence corresponding to the monitoring time interval according to the obtained network overload state information and the corresponding network parameters in the network parameters corresponding to each target data packet extracted. The operation of the state generation module 1210 can refer to the above Figure 3 The operation of 310 is described.

[0166] In one example, optionally, the target congestion control algorithm includes a congestion control algorithm applied to WebRTC, and the network parameters include latency, packet loss rate, and throughput. The device 1200 for adjusting the data transmission rate may also include: an acquisition unit ( Fig.12 ), configured to obtain the maximum throughput value in the first time window and the maximum delay value in the second time window corresponding to the monitoring time interval. The state generation module 1210 can be further configured to: for each obtained network overload state information, according to the network overload information and the network parameters corresponding to the target data packet based on which the network overload state information is generated, and the corresponding maximum throughput value and maximum delay value, determine the relative throughput and relative delay corresponding to the network overload state information; generate each state according to the corresponding relative throughput, relative delay, packet loss rate and network overload state information; and arrange the generated states in the order of receiving the corresponding target data packets to obtain a state sequence corresponding to the monitoring time interval.

[0167] The rate change module 1220 is configured to provide the state sequence to the congestion control model trained based on the reinforcement learning algorithm to obtain a rate change parameter. The operation of the rate change module 1220 can refer to the above Figure 3 The operation of 320 is described.

[0168] The candidate rate module 1230 is configured to update the data transmission rate corresponding to the at least one target data packet according to the rate change parameter to obtain the second candidate data transmission rate. The operation of the candidate rate module 1230 can refer to the above Figure 3 The operation of 330 is described.

[0169] In one example, the candidate rate module 1230 may be further configured to determine a parameter change value corresponding to the rate change parameter according to a preset exponential function; and determine the product of the parameter change value and the data transmission rate corresponding to the at least one target data packet as the second candidate data transmission rate. The operation of the candidate rate module 1230 may refer to the above Figure 4 The operations of 410 and 420 are described.

[0170] Back to Fig.11 The rate determination unit 1140 is configured to determine a new data transmission rate based on the first candidate data transmission rate and the second candidate data transmission rate. The operation of the rate determination unit 1140 can refer to the above Figure 2 The operation of 240 is described.

[0171] In one example, optionally, see Fig.13 , Fig.13 A block diagram showing an example of a rate determining unit 1300 in an apparatus for adjusting a data transmission rate according to an embodiment of this specification.

[0172] like Fig.13 As shown, the rate determination unit 1300 may include a difference determination module 1310 and a rate determination module 1320 .

[0173] The difference determination module 1310 is configured to determine the difference between the first candidate data transmission rate and the second candidate data transmission rate. The operation of the difference determination module 1310 can refer to the above Figure 6 The operation of 640 is described.

[0174] The rate determination module 1320 is configured to determine a new data transmission rate based on the first candidate data transmission rate and the second candidate data transmission rate according to whether the difference is greater than a preset threshold. The operation of the rate determination module 1320 can refer to the above Figure 6 The operation of 650 is described.

[0175] In one example, the rate determination module 1320 is further configured to: in response to the difference being greater than a preset threshold, perform network data transmission according to the first candidate data transmission rate and the second candidate data transmission rate, respectively, to obtain network utility parameters corresponding to the first candidate data transmission rate and the second candidate data transmission rate, respectively; determine utility values ​​corresponding to the first candidate data transmission rate and the second candidate data transmission rate, respectively, according to the utility function and the obtained network utility parameters, wherein the utility value is used to evaluate the performance of the data transmission rate; and determine the candidate data transmission rate corresponding to the maximum utility value as the new data transmission rate according to the determined utility value. The operation of the rate determination module 1320 can refer to the above. Figure 7 The operations of 720-740 are described.

[0176] In one example, the rate determination module 1320 is further configured to: perform network data transmission using a first verification rate as the data transmission rate to obtain corresponding network utility parameters; and perform network data transmission using a second verification rate as the data transmission rate after a verification time interval to obtain corresponding network utility parameters, wherein the first verification rate and the second verification rate are respectively the smaller and larger values ​​of the first candidate data transmission rate and the second candidate data transmission rate, and the verification time interval is half of the monitoring time interval.

[0177] In one example, the rate determination module 1320 is further configured to: obtain a network utility parameter corresponding to a third candidate data transmission rate based on the network parameters corresponding to each extracted target data packet, wherein the third candidate data transmission rate is a data transmission rate corresponding to the at least one target data packet; determine a utility value corresponding to the third candidate data transmission rate based on the utility function and the obtained network utility parameter; and determine the candidate data transmission rate corresponding to the maximum utility value among the first candidate data transmission rate, the second candidate data transmission rate and the third candidate data transmission rate as the new data transmission rate. The operation of the rate determination module 1320 can refer to the above. Figure 8 The operation of 830-850 is described.

[0178] Further references Fig.14 , Fig.14 A block diagram showing yet another example of an apparatus 1400 for adjusting a data transmission rate according to an embodiment of this specification.

[0179] like Fig.14As shown, the apparatus 1400 for adjusting the data transmission rate may include a monitoring extraction unit 1410 , a first candidate unit 1420 , a second candidate unit 1430 , a rate determination unit 1440 , a queue accumulation determination unit 1450 , and a queue emptying unit 1460 .

[0180] The description of the monitoring extraction unit 1410, the first candidate unit 1420, the second candidate unit 1430, and the rate determination unit 1440 can refer to the aforementioned Fig.11 Description of the monitoring extraction unit 1110, the first candidate unit 1120, the second candidate unit 1130 and the rate determination unit 1140 in the embodiment.

[0181] The queue accumulation determination unit 1450 is configured to determine whether there is currently queue message accumulation according to whether the round-trip delay is greater than a preset multiple of the minimum delay in the second time window corresponding to the monitoring time interval.

[0182] The queue emptying unit 1460 is configured to, in response to the existence of queue message accumulation and the data transmission rate corresponding to the at least one target data packet being greater than the data reception rate, determine the new data transmission rate to be a value less than the data reception rate, so as to reduce the queue message accumulation. The operations of the queue accumulation determination unit 1450 and the queue emptying unit 1460 can refer to the above Figure 2 Describes the queue accumulation detection and draining process.

[0183] Reference below Fig.15 , Fig.15 A block diagram showing an example of a device 1500 for training a congestion control model according to an embodiment of this specification.

[0184] like Fig.15 As shown, the apparatus 1500 for training a congestion control model can be configured to loop through a training unit 1510 using a training sample set to perform the following model training process until a training end condition is met. Each training sample in the training sample set includes a network parameter sequence corresponding to a monitoring time interval and a corresponding data transmission rate. The training unit 1510 includes: a state feature generation module 1511, a rate update module 1512, and a reward determination module 1513.

[0185] The state feature generation module 1511 is configured to determine the network overload state information corresponding to the network parameter sequence of the current training sample based on the target congestion control algorithm; determine the rate adjustment direction of the current training sample according to the network overload state information; and generate the state feature sequence data of the current training sample according to the network parameter sequence of the current training sample and the corresponding rate adjustment direction. The operation of the state feature generation module 1511 can refer to the above Fig.10The operations of 1010-1030 are described.

[0186] The rate update module 1512 is configured to provide the state feature sequence data of the current training sample to the current congestion control model, determine the rate change parameter of the current training sample from the candidate rate change parameter set; update the data transmission rate of the current training sample according to the determined rate change parameter to obtain the next data transmission rate. The operation of the rate update module 1512 can refer to the above Fig.10 The operations of 1040-1050 are described.

[0187] In one example, the candidate rate change parameter set includes a candidate speed-up change parameter set and a candidate speed-down change parameter set. The rate update module 1512 is further configured to: provide the state feature sequence data of the current training sample to the current congestion control model, and determine the rate change parameter of the current training sample from the candidate speed-up change parameter set or the candidate speed-down change parameter set that is consistent with the rate adjustment direction.

[0188] The reward determination module 1513 is configured to perform network data transmission according to the next data transmission rate to obtain the next network parameters corresponding to the current training sample; and determine the reward value according to the next network parameters corresponding to the current training sample using the reward function. The operation of the reward determination module 1513 can refer to the above Fig.10 The operations of 1060-1070 are described.

[0189] In one example, each training sample in the training sample set also includes a maximum throughput value in a first time window and a maximum delay value in a second time window corresponding to a monitoring time interval, and the determined reward value is obtained based on the next network parameter and the relative throughput determined by the maximum throughput value and the relative delay determined based on the next network parameter and the maximum delay value.

[0190] The apparatus 1500 for training a congestion control model may further include: a parameter adjustment unit 1520, configured to adjust the parameters of the current congestion control model based on the determined reward value based on the reinforcement learning algorithm in response to the failure to meet the training end condition, wherein the congestion control model after the model parameter adjustment serves as the current congestion control model for the next model training process. The operation of the parameter adjustment unit 1520 may refer to the above Fig.10 The operation of 1080-1090 is described.

[0191] Reference above Figures 1 to 15 , describes embodiments of a method and apparatus for adjusting a data transmission rate, and a method and apparatus for training a congestion control model according to embodiments of this specification.

[0192] The device for adjusting the data transmission rate and the device for training the congestion control model in the embodiments of this specification can be implemented by hardware, or by software or a combination of hardware and software. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the device in which it is located reading the corresponding computer program instructions in the memory into the memory and running them. In the embodiments of this specification, the device for adjusting the data transmission rate and the device for training the congestion control model can be implemented, for example, using electronic devices.

[0193] Fig.16 A schematic diagram of an apparatus 1600 for adjusting a data transmission rate according to an embodiment of the present specification is shown.

[0194] like Fig.16 As shown, the apparatus 1600 for adjusting the data transmission rate may include at least one processor 1610, a memory (e.g., a non-volatile memory) 1620, a memory 1630, and a communication interface 1640, and the at least one processor 1610, the memory 1620, the memory 1630, and the communication interface 1640 are connected together via a bus 1650. At least one processor 1610 executes at least one computer-readable instruction stored or encoded in the memory (i.e., the above-mentioned element implemented in the form of software).

[0195] In one embodiment, computer executable instructions are stored in a memory, which when executed cause at least one processor 1610 to: extract network parameters corresponding to at least one target data packet from multiple data packets received within a monitoring time interval; based on a target congestion control algorithm, obtain network overload status information and a first candidate data sending rate according to the extracted network parameters corresponding to each target data packet; provide the extracted network parameters corresponding to each target data packet and the network overload status information to a congestion control model trained based on a reinforcement learning algorithm to obtain a second candidate data sending rate; and determine a new data sending rate based on the first candidate data sending rate and the second candidate data sending rate.

[0196] It should be understood that the computer executable instructions stored in the memory, when executed, cause at least one processor 1610 to perform the above combined operations in various embodiments of the present specification. Figure 1-9 Describes the various operations and functions.

[0197] Fig.17 A schematic diagram of an apparatus 1700 for training a congestion control model according to an embodiment of the present specification is shown.

[0198] like Fig.17As shown, the apparatus 1700 for training a congestion control model may include at least one processor 1710, a memory (e.g., a non-volatile memory) 1720, a memory 1730, and a communication interface 1740, and the at least one processor 1710, the memory 1720, the memory 1730, and the communication interface 1740 are connected together via a bus 1750. At least one processor 1710 executes at least one computer-readable instruction stored or encoded in the memory (i.e., the above-mentioned element implemented in the form of software).

[0199] In one embodiment, a computer executable instruction is stored in a memory, which, when executed, causes at least one processor 1710 to: loop through the following model training process using a training sample set until a training end condition is met, wherein each training sample in the training sample set includes a network parameter sequence corresponding to a monitoring time interval and a corresponding data transmission rate: determine network overload state information corresponding to the network parameter sequence of a current training sample based on a target congestion control algorithm; determine a rate adjustment direction of the current training sample based on the network overload state information; generate state feature sequence data of the current training sample based on the network parameter sequence and the corresponding rate adjustment direction of the current training sample; and convert the state feature sequence data of the current training sample into state feature sequence data of the current training sample; The sequence data is provided to the current congestion control model, and the rate change parameters of the current training sample are determined from the candidate rate change parameter set; the data sending rate of the current training sample is updated according to the determined rate change parameters to obtain the next data sending rate; network data transmission is performed according to the next data sending rate to obtain the next network parameters corresponding to the current training sample; a reward value is determined according to the next network parameters corresponding to the current training sample using a reward function; and in response to failure to meet a training end condition, the parameters of the current congestion control model are adjusted based on the determined reward value based on a reinforcement learning algorithm, wherein the congestion control model after the model parameters are adjusted serves as the current congestion control model for the next model training process.

[0200] It should be understood that the computer executable instructions stored in the memory, when executed, cause at least one processor 1710 to perform the above combined operations in various embodiments of the present specification. Fig.10 Describes the various operations and functions.

[0201] According to one embodiment, a program product such as a computer-readable medium is provided. The computer-readable medium may have instructions (i.e., the above-mentioned elements implemented in the form of software), which, when executed by a computer, causes the computer to perform the above-mentioned combination of various embodiments of this specification. Figure 1-10 Describes the various operations and functions.

[0202] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0203] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above-mentioned embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute a part of the present invention.

[0204] The computer program code required for the operation of each part of this specification can be written in any one or more programming languages, including object-oriented programming languages, such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB, NET and Python, conventional procedural programming languages ​​such as C language, Visual Basic 2003, Perl, COBOL2002, PHP and ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code can be run on the user's computer, or run on the user's computer as an independent software package, or run partly on the user's computer and another part on the remote computer, or all on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service, such as software as a service (SaaS).

[0205] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0206] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0207] Not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units can be ignored according to actual needs. The execution order of each step is not fixed and can be determined according to needs. The device structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0208] The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply "preferred" or "advantageous" over other embodiments. The detailed description includes specific details for the purpose of providing an understanding of the described techniques. However, the techniques may be implemented without these specific details. In some instances, in order to avoid obscuring the concepts of the described embodiments, well-known structures and devices are shown in block diagram form.

[0209] The optional implementation modes of the embodiments of the present specification are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present specification are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present specification, various simple modifications can be made to the technical solutions of the embodiments of the present specification, and these simple modifications all belong to the protection scope of the embodiments of the present specification.

[0210] The above description of the contents of this specification is provided to enable any person of ordinary skill in the art to implement or use the contents of this specification. Various modifications to the contents of this specification will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the contents of this specification. Therefore, the contents of this specification are not limited to the examples and designs described herein, but are consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A method for adjusting a data transmission rate, comprising: extracting a network parameter corresponding to at least one target data packet from a plurality of data packets received within a monitoring time interval; Based on the target congestion control algorithm, network overload status information and a first candidate data transmission rate are obtained according to the network parameters corresponding to each extracted target data packet, wherein the target congestion control algorithm is a congestion control algorithm not based on reinforcement learning and selected according to the corresponding network transmission protocol; Providing the extracted network parameters corresponding to each target data packet and the network overload status information to a congestion control model trained based on a reinforcement learning algorithm to obtain a second candidate data transmission rate; and determining a new data transmission rate based on the first candidate data transmission rate and the second candidate data transmission rate; The determining a new data transmission rate based on the first candidate data transmission rate and the second candidate data transmission rate comprises: determining a smaller value between the first candidate data transmission rate and the second candidate data transmission rate as the new data transmission rate, or determining an average value of the first candidate data transmission rate and the second candidate data transmission rate as the new data transmission rate, or Determine a difference between the first candidate data transmission rate and the second candidate data transmission rate; and determine a new data transmission rate based on the first candidate data transmission rate and the second candidate data transmission rate according to whether the difference is greater than a preset threshold.

2. The method of claim 1, wherein: Providing the extracted network parameters corresponding to each target data packet and the network overload status information to a congestion control model trained based on a reinforcement learning algorithm to obtain a second candidate data transmission rate includes: Generate a state sequence corresponding to the monitoring time interval according to the obtained network overload state information and corresponding network parameters among the extracted network parameters corresponding to each target data packet; Providing the state sequence to a congestion control model trained based on a reinforcement learning algorithm to obtain a rate change parameter; and The data transmission rate corresponding to the at least one target data packet is updated according to the rate change parameter to obtain the second candidate data transmission rate.

3. The method of claim 2, wherein: The updating of the data transmission rate corresponding to the at least one target data packet according to the rate change parameter to obtain the second candidate data transmission rate comprises: Determining a parameter change value corresponding to the rate change parameter according to a preset exponential function; and The product of the parameter change value and the data transmission rate corresponding to the at least one target data packet is determined as the second candidate data transmission rate.

4. The method of claim 2, wherein: The target congestion control algorithm includes a congestion control algorithm applied to WebRTC, and the network parameters include delay, packet loss rate and throughput. Before generating a state sequence corresponding to the monitoring time interval according to the obtained network overload state information and corresponding network parameters among the extracted network parameters corresponding to each target data packet, the method further includes: Obtaining a maximum throughput value in a first time window and a maximum delay value in a second time window corresponding to the monitoring time interval, The step of generating a state sequence corresponding to the monitoring time interval according to the obtained network overload state information and corresponding network parameters among the extracted network parameters corresponding to each target data packet comprises: For each obtained network overload status information, determine the relative throughput and relative delay corresponding to the network overload status information according to the network overload status information and the network parameters corresponding to the target data packet based on which the network overload status information is generated, as well as the corresponding maximum throughput and maximum delay values; Generate each state according to the corresponding relative throughput, relative delay, packet loss rate and network overload state information; and The generated states are arranged in the order of receiving the corresponding target data packets to obtain a state sequence corresponding to the monitoring time interval.

5. The method of claim 1, wherein: Determining a new data transmission rate based on the first candidate data transmission rate and the second candidate data transmission rate includes determining a difference between the first candidate data transmission rate and the second candidate data transmission rate; and determining a new data transmission rate based on the first candidate data transmission rate and the second candidate data transmission rate according to whether the difference is greater than a preset threshold; The determining, according to whether the difference is greater than a preset threshold, a new data transmission rate based on the first candidate data transmission rate and the second candidate data transmission rate comprises: In response to the difference being greater than a preset threshold, Performing network data transmission according to the first candidate data transmission rate and the second candidate data transmission rate, respectively, to obtain network utility parameters corresponding to the first candidate data transmission rate and the second candidate data transmission rate, respectively; Determining utility values ​​corresponding to the first candidate data transmission rate and the second candidate data transmission rate, respectively, according to the utility function and the obtained network utility parameter, wherein the utility values ​​are used to evaluate the performance of the data transmission rate; and According to the determined utility values, the candidate data transmission rate corresponding to the maximum utility value is determined as the new data transmission rate.

6. The method of claim 5, wherein: The performing network data transmission according to the first candidate data transmission rate and the second candidate data transmission rate respectively to obtain network utility parameters corresponding to the first candidate data transmission rate and the second candidate data transmission rate respectively comprises: Performing network data transmission using the first verification rate as the data transmission rate to obtain a corresponding network utility parameter; and After the verification time interval, the second verification rate is used as the data transmission rate to transmit network data, and the corresponding network utility parameter is obtained. The first verification rate and the second verification rate are respectively the smaller value and the larger value of the first candidate data sending rate and the second candidate data sending rate, and the verification time interval is half of the monitoring time interval.

7. The method of claim 5, wherein: Before determining the candidate data transmission rate corresponding to the maximum utility value as the new data transmission rate according to the determined utility value, in response to the difference being greater than a preset threshold, the method further includes: Obtaining a network utility parameter corresponding to a third candidate data transmission rate according to the extracted network parameters corresponding to each target data packet, wherein the third candidate data transmission rate is a data transmission rate corresponding to the at least one target data packet; determining a utility value corresponding to the third candidate data transmission rate according to the utility function and the obtained network utility parameter, The step of determining, according to the determined utility value, the candidate data transmission rate corresponding to the maximum utility value as the new data transmission rate comprises: The candidate data transmission rate corresponding to the maximum utility value among the first candidate data transmission rate, the second candidate data transmission rate and the third candidate data transmission rate is determined as the new data transmission rate.

8. The method according to any one of claims 1 to 7, wherein: The network parameters include a round trip delay, and the method further includes: Determining whether there is currently queue message accumulation according to whether the round-trip delay is greater than a preset multiple of a minimum delay in a second time window corresponding to the monitoring time interval; and In response to queue message accumulation and a data transmission rate corresponding to the at least one target data packet being greater than a data reception rate, a new data transmission rate is determined to be a value less than the data reception rate to alleviate queue message accumulation.

9. A device for adjusting a data transmission rate, comprising: A monitoring extraction unit, configured to extract a network parameter corresponding to at least one target data packet from a plurality of data packets received within a monitoring time interval; The first candidate unit is configured to obtain network overload status information and a first candidate data transmission rate according to the extracted network parameters corresponding to each target data packet based on a target congestion control algorithm, wherein the target congestion control algorithm is a congestion control algorithm not based on reinforcement learning selected according to a corresponding network transmission protocol; A second candidate unit is configured to provide the extracted network parameters corresponding to each target data packet and the network overload status information to a congestion control model trained based on a reinforcement learning algorithm to obtain a second candidate data transmission rate; and a rate determining unit configured to determine a smaller value of the first candidate data transmission rate and the second candidate data transmission rate as a new data transmission rate, or is configured to determine an average of the first candidate data transmission rate and the second candidate data transmission rate as a new data transmission rate, or It includes a difference determination module, which is configured to determine the difference between the first candidate data sending rate and the second candidate data sending rate; and a rate determination module, which is configured to determine a new data sending rate based on the first candidate data sending rate and the second candidate data sending rate according to whether the difference is greater than a preset threshold.

10. The device of claim 9, wherein: The second candidate unit includes: A state generation module is configured to generate a state sequence corresponding to the monitoring time interval according to the obtained network overload state information and corresponding network parameters among the extracted network parameters corresponding to each target data packet; A rate change module is configured to provide the state sequence to a congestion control model trained based on a reinforcement learning algorithm to obtain a rate change parameter; and The candidate rate module is configured to update the data transmission rate corresponding to the at least one target data packet according to the rate change parameter to obtain the second candidate data transmission rate.

11. The device of claim 10, wherein: The candidate rate module is further configured to: Determining a parameter change value corresponding to the rate change parameter according to a preset exponential function; and The product of the parameter change value and the data transmission rate corresponding to the at least one target data packet is determined as the second candidate data transmission rate.

12. The device of claim 10, wherein: The target congestion control algorithm includes a congestion control algorithm applied to WebRTC, the network parameters include delay, packet loss rate and throughput, and the device also includes: an acquisition unit, configured to acquire a maximum throughput value in a first time window and a maximum latency value in a second time window corresponding to the monitoring time interval, The state generation module is further configured to: For each obtained network overload status information, determine the relative throughput and relative delay corresponding to the network overload status information according to the network overload status information and the network parameters corresponding to the target data packet based on which the network overload status information is generated, as well as the corresponding maximum throughput and maximum delay values; Generate each state according to the corresponding relative throughput, relative delay, packet loss rate and network overload state information; and The generated states are arranged in the order of receiving the corresponding target data packets to obtain a state sequence corresponding to the monitoring time interval.

13. The device of claim 9, wherein: The rate determination unit includes the difference determination module and the rate determination module, and the rate determination module is further configured to: In response to the difference being greater than a preset threshold, Performing network data transmission according to the first candidate data transmission rate and the second candidate data transmission rate, respectively, to obtain network utility parameters corresponding to the first candidate data transmission rate and the second candidate data transmission rate, respectively; Determine, according to the utility function and the obtained network utility parameter, utility values ​​corresponding to the first candidate data transmission rate and the second candidate data transmission rate, respectively, wherein the utility values ​​are used to evaluate the performance of the data transmission rate; as well as According to the determined utility values, the candidate data transmission rate corresponding to the maximum utility value is determined as the new data transmission rate.

14. The device of claim 13, wherein: The rate determination module is further configured to: Obtaining a network utility parameter corresponding to a third candidate data transmission rate according to the extracted network parameters corresponding to each target data packet, wherein the third candidate data transmission rate is a data transmission rate corresponding to the at least one target data packet; Determining a utility value corresponding to the third candidate data transmission rate according to the utility function and the obtained network utility parameter; The candidate data transmission rate corresponding to the maximum utility value among the first candidate data transmission rate, the second candidate data transmission rate and the third candidate data transmission rate is determined as the new data transmission rate.

15. The device as claimed in any one of claims 9 to 14, wherein: The network parameters include a round trip delay, and the device further includes: a queue accumulation determination unit configured to determine whether there is currently queue message accumulation according to whether the round-trip delay is greater than a preset multiple of the minimum delay in a second time window corresponding to the monitoring time interval; and The queue emptying unit is configured to determine a new data sending rate as a value less than the data receiving rate in response to queue message accumulation and the data sending rate corresponding to the at least one target data packet being greater than the data receiving rate, so as to alleviate queue message accumulation.

16. A device for adjusting a data transmission rate, comprising: At least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the method for adjusting the data transmission rate as claimed in any one of claims 1 to 8.

17. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method for adjusting a data transmission rate according to any one of claims 1 to 8.

18. A computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method for adjusting a data transmission rate according to any one of claims 1 to 8.

Images