Multipath TCP congestion control method and device based on reinforcement learning

Abstract

The present application provides a multi-path TCP congestion control method and device based on reinforcement learning, the method comprising: if a change is detected in the current network environment of the target network, the target network status data at the current moment is obtained; the target network status data is added to a data set containing prior data, and a preset intelligent agent is fine-tuned based on the updated data set with a deep reinforcement learning algorithm to obtain a new intelligent agent suitable for the current network environment of the target network, so as to perform congestion control on the target network based on the new intelligent agent; the intelligent agent is pre-trained offline based on a data set containing prior data with a deep reinforcement learning algorithm. The present application can timely and automatically fine-tune the intelligent agent online when the network environment changes, and can improve the efficiency and effectiveness of the intelligent agent update, ensure the continuous high performance of the intelligent agent, so as to improve the effectiveness and reliability of multi-path TCP congestion control using the intelligent agent.

Description

Technical Field

[0001] The present application relates to the technical field of congestion control, and in particular to a multi-path TCP congestion control method and device based on reinforcement learning. Background Art

[0002] Many network devices today are equipped with multiple network interfaces, which can improve the throughput of the device and make the device's network more stable. For example, mobile phones are configured with 4G / 5G and WiFi at the same time, and hosts are configured with wired and WiFi at the same time. Multipath TCP (MPTCP) is a multipath protocol that allows a single TCP stream to be split into multiple sub-streams across different paths, attracting a lot of attention from industry and academia. Since network conditions change rapidly, the basic goal of congestion control (CC) is to perform well under different network conditions, but when the network scenario changes (such as frequent satellite switching, burst traffic, network failures, etc.), the performance of existing congestion control schemes will drop sharply.

[0003] Currently, many MPTCP congestion control algorithms have been proposed, which can be divided into classic algorithms and learning-based algorithms. Most classic algorithms are extensions of single-path TCP algorithms on multiple paths, so they also inherit the AIMD shortcomings of TCP, including the inability to adapt to constantly changing network scenarios, incorrect responses to random packet loss, etc. Although learning-based algorithms can provide higher performance than classic algorithms, they have problems with slow convergence and poor generalization, which makes them unable to adapt to rapidly changing network conditions.

[0004] Therefore, it is necessary to design an MPTCP congestion control scheme that can quickly adapt to new network conditions. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a multi-path TCP congestion control method and device based on reinforcement learning to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present application provides a multi-path TCP congestion control method based on reinforcement learning, comprising:

[0007] If a change in the current network environment of the target network is detected, the target network state data of the target network at the current moment is obtained;

[0008] The target network status data is added to a data set containing prior data, and the preset agent is fine-tuned based on the updated data set using a deep reinforcement learning algorithm to obtain a new agent suitable for the current network environment of the target network, so as to perform multi-path TCP congestion control on the target network based on the new agent; wherein the agent is pre-trained offline based on a data set containing prior data using a deep reinforcement learning algorithm.

[0009] In some embodiments of the present application, before acquiring the target network state data of the target network at the current moment, the method further includes:

[0010] Acquire prior data of the target network, wherein the prior data includes: expert data and / or preset suboptimal data;

[0011] A data set is set according to the prior data of the target network, where the data set is used to store the corresponding relationship between the state parameter, action parameter, next state parameter and reward parameter of the target network;

[0012] Based on the deep reinforcement learning algorithm, the preset deep reinforcement learning model is pre-trained using the data set to obtain an intelligent agent that outputs action parameters from the current moment to the next moment according to the state information of the target network at the current moment.

[0013] In some embodiments of the present application, it also includes:

[0014] Inputting the target network state data of the target network at the current moment into the new agent, so that the new agent outputs the action parameters of the target network from the current moment to the next moment;

[0015] Multipath TCP congestion control is performed on the network traffic of the target network at the current moment according to the action parameters.

[0016] In some embodiments of the present application, if a change in the current network environment of the target network is detected, obtaining the target network state data of the target network at the current moment includes:

[0017] Using a preset monitoring block to collect environmental status information from the environment of the target network, and generating statistical information corresponding to the environmental status information through an acknowledgment character ACK and a timer;

[0018] If the current network environment of the target network is known to change according to the statistical information, the target network state data of the target network at the current moment is sent to the preset reward calculation module and the intelligent agent respectively;

[0019] The reward calculation module is used to calculate the target reward corresponding to the target network state data of the target network at the current moment, and the target reward is sent to the intelligent agent.

[0020] In some embodiments of the present application, the target network state data is added to a data set containing prior data, and a preset agent is fine-tuned using a deep reinforcement learning algorithm based on the updated data set to obtain a new agent suitable for the current network environment of the target network, including:

[0021] Adding the target network status data to a data set containing prior data to form a new data set;

[0022] Based on a deep reinforcement learning algorithm, the new data set and the target reward are used to perform fine-tuning training on the intelligent agent to obtain a new intelligent agent suitable for the current network environment of the target network.

[0023] In some embodiments of the present application, the target network status data includes:

[0024] The throughput, congestion window, round trip time, and round trip time average deviation of the target network;

[0025] Wherein, the congestion window and the action satisfy a preset relationship constraint condition.

[0026] In some embodiments of the present application, the deep reinforcement learning model includes: an AWAC-LSTM model for alternating execution between an online fine-tuning phase and an offline pre-training phase;

[0027] The AWAC-LSTM model is a model obtained by replacing the MLP network in the AWAC model with the LSTM network in advance.

[0028] Another aspect of the present application provides a multi-path TCP congestion control device based on reinforcement learning, comprising:

[0029] A network monitoring module is used to obtain target network status data of the target network at the current moment if a change in the current network environment of the target network is detected;

[0030] An agent fine-tuning module is used to add the target network status data to a data set containing prior data, and fine-tune the preset agent based on the updated data set using a deep reinforcement learning algorithm to obtain a new agent suitable for the current network environment of the target network, so as to perform multi-path TCP congestion control on the target network based on the new agent; wherein the agent is pre-trained offline based on a data set containing prior data using a deep reinforcement learning algorithm.

[0031] Another aspect of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the reinforcement learning-based multi-path TCP congestion control method when executing the computer program.

[0032] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the multi-path TCP congestion control method based on reinforcement learning is implemented.

[0033] The multi-path TCP congestion control method based on reinforcement learning provided by the present application, if the current network environment of the target network is detected to send a change, the target network state data of the target network at the current moment is obtained; the target network state data is added to the data set containing prior data, and the preset agent is fine-tuned based on the updated data set with a deep reinforcement learning algorithm to obtain a new agent suitable for the current network environment of the target network, so as to perform multi-path TCP congestion control on the target network based on the new agent; wherein the agent is pre-trained offline based on the data set containing prior data with a deep reinforcement learning algorithm. The present application can use a suboptimal data set to train the reinforcement learning strategy offline, and then quickly improve the strategy during online deployment to make it reach a convergence state, achieve continuous high performance, and avoid long-term retraining when encountering a new network environment. In other words, the present application can timely and automatically fine-tune the agent online when the network environment changes, and can improve the efficiency and effectiveness of the agent update, ensure the continuous high performance of the agent, so as to improve the effectiveness and reliability of using the agent for multi-path TCP congestion control.

[0034] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0035] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0037] Figure 1 The figure is a schematic diagram of the overall process of a multi-path TCP congestion control method based on reinforcement learning in one embodiment of the present application.

[0038] Figure 2 A specific flow chart of a multi-path TCP congestion control method based on reinforcement learning in one embodiment of the present application.

[0039] Figure 3 A structural diagram of a multi-path TCP congestion control device based on reinforcement learning in another embodiment of the present application.

[0040] Figure 4 Another structural schematic diagram of a multi-path TCP congestion control device based on reinforcement learning in another embodiment of the present application.

[0041] Figure 5 This is a schematic diagram of the overall framework of the Leopard solution provided in the application example of this application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0043] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0044] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0045] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0046] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0047] In one or more embodiments of the present application, CC refers to congestion control; MPTCP refers to Multipath TCP; DC refers to data center; RTT refers to round-trip time; DRL refers to Deep Reinforcement Learning; Agent refers to intelligent agent; ACK refers to Acknowledge character; Cwnd refers to congestion window.

[0048] Classic algorithms for multipath TCP congestion control algorithms are: LIA, OLIA, and BALIA. The most advanced classical algorithms always perform fixed rate adjustment based on AIMD. Despite providing good friendliness and practicality, they are basically difficult to achieve consistent high throughput and adaptability.

[0049] The learning-based algorithms include: MPCC, DRL-CC, SmartCC, and DeepCC. The practicality of learning-based algorithms is poor: 1) MPCC based on online learning converges slowly: MPCC has no prior knowledge and requires fine-grained calculations, which leads to slow convergence, unsatisfactory performance, and inability to fully utilize the bandwidth of time-varying networks; 2) DRL-CC, SmartCC, and DeepCC based on deep reinforcement learning (DRL) have poor generalization capabilities: they cannot quickly adapt to new network conditions. Algorithms based on reinforcement learning are first trained offline, and may encounter serious performance issues when deployed online because their model mapping is fixed and cannot be adjusted according to new network conditions.

[0050] Therefore, in order to improve the efficiency and effectiveness of agent updates, ensure the continuous high performance of agents, and improve the effectiveness and reliability of multipath TCP congestion control using agents, the embodiment of the present application designs a multipath TCP congestion control method based on reinforcement learning, which can be implemented based on a practical MPTCP adaptive CC framework, which can be called Leopard, to solve the problem that existing learning-based solutions cannot quickly adapt to new network conditions. The multipath TCP congestion control method based on reinforcement learning provided by the present application can use a suboptimal data set to train the reinforcement learning strategy offline, and then quickly improve the strategy during online deployment to achieve convergence, achieve continuous high performance, and avoid long-term retraining when encountering a new network environment.

[0051] The main advantages of Leopard and its reinforcement learning-based multipath TCP congestion control method include:

[0052] 1) Fast convergence: Use prior data (whether expert data or suboptimal data) for pre-training to avoid a large amount of fine-grained calculations;

[0053] 2) Good generalization: effectively combining offline training and online fine-tuning to avoid hard connections between actions and states;

[0054] 3) It can quickly adapt to changing network conditions and provide consistent high performance and fast convergence.

[0055] The details are described in detail through the following examples.

[0056] Based on this, the embodiment of the present application provides a multi-path TCP congestion control method based on reinforcement learning that can be performed by a multi-path TCP congestion control device based on reinforcement learning, see Figure 1 The multi-path TCP congestion control method based on reinforcement learning specifically includes the following contents:

[0057] Step 100: If a change in the current network environment of the target network is detected, the target network state data of the target network at the current moment is obtained.

[0058] In one or more embodiments of the present application, the target network status data refers to data used to represent the current status of the target network, and may at least include data such as the throughput, congestion window, round-trip time, and round-trip time average deviation of the target network.

[0059] Step 200: Add the target network status data to a data set containing prior data, and fine-tune the preset agent based on the updated data set using a deep reinforcement learning algorithm to obtain a new agent suitable for the current network environment of the target network, so as to perform multi-path TCP congestion control on the target network based on the new agent; wherein the agent is pre-trained offline based on a data set containing prior data using a deep reinforcement learning algorithm.

[0060] In step 200, the data set containing prior data refers to a data set that at least contains prior data. If step 100 is executed for the first time after the offline pre-training of the intelligent agent, then the data set to be added to the target network status data in step 200 only contains the prior data used in the offline pre-training; if step 100 is not executed for the first time after the offline pre-training of the intelligent agent, then the data set to be added to the target network status data in step 200, in addition to the prior data used in the offline pre-training, also contains the historical target network status data previously used for fine-tuning the intelligent agent.

[0061] From the above description, it can be seen that the multi-path TCP congestion control method based on reinforcement learning provided in the embodiment of the present application can timely and automatically fine-tune the intelligent agent online when the network environment changes, and can improve the efficiency and effectiveness of the intelligent agent update, ensure the continuous high performance of the intelligent agent, so as to improve the effectiveness and reliability of using the intelligent agent for multi-path TCP congestion control.

[0062] In order to further improve the generalization of the multipath TCP congestion control algorithm and improve its convergence efficiency during offline training, in a multipath TCP congestion control method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 , the multi-path TCP congestion control method based on reinforcement learning further includes the following contents before step 100:

[0063] Step 010: Acquire prior data of the target network, wherein the prior data includes: expert data and / or preset suboptimal data.

[0064] In step 010, prior data refers to the source of data, which can be derived from expert experience or suboptimal data. It can be understood that the suboptimal data refers to data other than expert data in a unified format. If the agent is trained using suboptimal data, the action returned by the agent is not optimal. For example, suboptimal data can be random data; collect the A data set of agent 1 and the B data set of agent 2. If the agent 2 trained with the B data set performs better than the agent 1 trained with the A data set, then A can be identified as suboptimal data. The prior data at least includes the state parameters, action parameters, next state parameters and reward parameters of the target network.

[0065] Step 020: A data set is set according to the prior data of the target network, where the data set is used to store the correspondence between the state parameters, action parameters, next state parameters and reward parameters of the target network.

[0066] Step 030: Based on the deep reinforcement learning algorithm, the preset deep reinforcement learning model is pre-trained using the data set to obtain an intelligent agent that outputs action parameters from the current moment to the next moment according to the state information of the target network at the current moment.

[0067] Specifically, in the offline stage, Leopard uses the target network’s dataset β to train the agent, and the dataset β is in the form of D = {(s, a, s′, r) j}, where s, a, s' and r represent state, action, next state and reward respectively. The dataset β is prior data, which can be expert data or suboptimal data.

[0068] In order to further improve the application effectiveness and reliability of multipath TCP congestion control, in a multipath TCP congestion control method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 , the step 200 in the multi-path TCP congestion control method based on reinforcement learning further specifically includes the following contents:

[0069] Step 300: Input the target network state data of the target network at the current moment into the new agent, so that the new agent outputs the action parameters of the target network from the current moment to the next moment.

[0070] Step 400: Perform multipath TCP congestion control on the network traffic of the target network at the current moment according to the action parameters.

[0071] For example, the action output by the agent is [a1, a2]. According to the following formula (1), [c1, c2] is obtained, where c1 corresponds to the Cwnd of path 1 and c2 corresponds to the Cwnd of path 2. The packet sender sends packets to the two sub-paths according to c1 and c2 respectively.

[0072] In order to further improve the efficiency and effectiveness of the online fine-tuning agent, in a multi-path TCP congestion control method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 , step 100 in the multi-path TCP congestion control method based on reinforcement learning specifically includes the following contents:

[0073] Step 110: using a preset monitoring block to collect environmental status information from the environment of the target network, and generating statistical information corresponding to the environmental status information through an acknowledgment character ACK and a timer.

[0074] Step 120: If it is known from the statistical information that the current network environment of the target network has changed, the target network state data of the target network at the current moment is sent to the preset reward calculation module and the agent respectively.

[0075] Step 130: Use the reward calculation module to calculate the target reward corresponding to the target network state data of the target network at the current moment, and send the target reward to the agent.

[0076] Specifically, the monitoring block collects information from the environment, continuously generates various statistical information through confirmation characters ACK and timers, and passes the required status information to the agent and reward calculation module. The reward calculation formula is is the throughput of subflow i at time t, which can take into account the throughput of multiple paths of the Multi-Path Transport Control Protocol MPTCP and the fairness of TCP traffic.

[0077] In order to further improve the monitoring efficiency and effectiveness of target network environment changes, in a multi-path TCP congestion control method based on reinforcement learning provided in an embodiment of the present application, see Figure 2 , step 200 in the multi-path TCP congestion control method based on reinforcement learning specifically includes the following contents:

[0078] Step 210: Add the target network status data to a data set containing prior data to form a new data set.

[0079] Step 220: Based on a deep reinforcement learning algorithm, the new data set and the target reward are used to fine-tune the agent to obtain a new agent suitable for the current network environment of the target network.

[0080] Specifically, in the online stage, the offline trained agent interacts with the environment, adds the collected new data (s, a, s', r) to the data set β, and uses the new data set to update the agent. In this way, the agent has both prior knowledge to shorten the learning time and can adjust itself in real time according to the real-time status of the network to obtain better results.

[0081] In order to further improve the efficiency of monitoring changes in the target network environment and the efficiency and effectiveness of TCP congestion control, in a multi-path TCP congestion control method based on reinforcement learning provided in an embodiment of the present application, the throughput, congestion window, round-trip time and average deviation of the round-trip time of the target network;

[0082] Wherein, the congestion window and the action satisfy a preset relationship constraint condition.

[0083] Specifically, the relationship between the congestion window c and action a is defined as:

[0084]

[0085] Among them, c t is the congestion window at time t, a t is the action at time t.

[0086] In order to further improve the efficiency of monitoring changes in the target network environment and the efficiency and effectiveness of TCP congestion control, in a multi-path TCP congestion control method based on reinforcement learning provided in an embodiment of the present application, the deep reinforcement learning model includes: an AWAC-LSTM model for alternating execution between an online fine-tuning phase and an offline pre-training phase;

[0087] The AWAC-LSTM model is a model obtained by replacing the MLP network in the AWAC model with the LSTM network in advance.

[0088] Specifically, the present application first pre-trains the agent with prior data, and then makes online adjustments in different environments, so that the agent can obtain the optimal action in different environments. The present application designs a new reinforcement learning algorithm AWAC-LSTM, which changes the MLP network in the original AWAC (Accelerating Online Reinforcement Learning with Offline Datasets) algorithm to a long short-term memory LSTM (Long short-term memory) network to be more suitable for the network field. LSTM has a certain memory effect and is very suitable for sequence data. Since the congestion control algorithm acts on time series data, the AWAC-LSTM algorithm is more suitable for congestion control schemes than the AWAC algorithm.

[0089] AWAC-LSTM alternates between the policy evaluation phase and the policy improvement phase to maximize the benefits. AWAC-LSTM trains an off-policy critic and an implicit policy-constrained value (actor). The off-policy refers to the fact that the agent to be learned is not the same as the agent that interacts with the environment; the actor-critic algorithm is a combination of value-based and policy-based algorithms. Actor is the policy function. It is implemented using the policy gradient algorithm. The input is the current state and the output is an action. The training goal is to maximize the expected cumulative return. Critic is the value function. It is implemented using the temporal difference method TD algorithm. Estimate the value function of the current policy, that is, evaluate the quality of the actor and guide the actor's next behavior. The actor selects the behavior based on probability, the critic judges the score of the behavior based on the actor's behavior, and the actor modifies the probability of the selected behavior according to the critic's score.

[0090] From the software level, the present application also provides a multi-path TCP congestion control device based on reinforcement learning for executing all or part of the multi-path TCP congestion control method based on reinforcement learning, see Figure 3 The multi-path TCP congestion control device based on reinforcement learning specifically includes the following contents:

[0091] The network monitoring module 10 is used to obtain the target network status data of the target network at the current moment if a change in the current network environment of the target network is detected;

[0092] The agent fine-tuning module 20 is used to add the target network status data to a data set containing prior data, and fine-tune the preset agent based on the updated data set using a deep reinforcement learning algorithm to obtain a new agent suitable for the current network environment of the target network, so as to perform multi-path TCP congestion control on the target network based on the new agent; wherein the agent is pre-trained offline based on a data set containing prior data using a deep reinforcement learning algorithm.

[0093] The embodiment of the reinforcement learning-based multipath TCP congestion control device provided in the present application can be specifically used to execute the processing flow of the embodiment of the reinforcement learning-based multipath TCP congestion control method in the above-mentioned embodiment. Its functions are not repeated here, and reference can be made to the detailed description of the above-mentioned reinforcement learning-based multipath TCP congestion control method embodiment.

[0094] The part of the multi-path TCP congestion control based on reinforcement learning performed by the multi-path TCP congestion control device based on reinforcement learning can be executed in a server, such as a central cloud server, and in another practical application scenario, all operations can also be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for specific processing of the multi-path TCP congestion control based on reinforcement learning.

[0095] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0096] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0097] From the above description, it can be seen that the multi-path TCP congestion control device based on reinforcement learning provided in the embodiment of the present application can timely and automatically perform online fine-tuning on the intelligent agent when the network environment changes, and can improve the efficiency and effectiveness of the intelligent agent update, ensure the continuous high performance of the intelligent agent, so as to improve the effectiveness and reliability of using the intelligent agent for multi-path TCP congestion control.

[0098] In order to further improve the generalization of the multipath TCP congestion control algorithm and improve its convergence efficiency during offline training, in a multipath TCP congestion control device based on reinforcement learning provided in an embodiment of the present application, see Figure 4 The multi-path TCP congestion control device based on reinforcement learning also specifically includes the following contents: an offline pre-training module 01; and the offline pre-training module 01 is specifically used to execute the following steps 010 to 030:

[0099] Step 010: Acquire prior data of the target network, wherein the prior data includes: expert data and / or preset suboptimal data.

[0100] Step 020: A data set is set according to the prior data of the target network, where the data set is used to store the correspondence between the state parameters, action parameters, next state parameters and reward parameters of the target network.

[0101] Step 030: Based on the deep reinforcement learning algorithm, the preset deep reinforcement learning model is pre-trained using the data set to obtain an intelligent agent that outputs action parameters from the current moment to the next moment according to the state information of the target network at the current moment.

[0102] To further illustrate the present solution, the present application also provides a specific application example for implementing a multi-path TCP congestion control method based on reinforcement learning. The multi-path TCP congestion control method is implemented through the aforementioned Leopard framework. Figure 5 ,The reinforcement learning based multi-path TCP congestion control method provided in the application example of ,this application mainly includes two parts: offline learning and online ,fine-tuning.

[0103] In the offline phase, Leopard uses the target network’s dataset β to train the agent. The dataset β is in the form of D = {(s, a, s′, r) j}, where s, a, s', r represent the current state, action, next state and reward respectively. The dataset β is the prior data, which can be expert data or suboptimal data.

[0104] In the online stage, the offline trained agent interacts with the environment, adds the collected new data (s, a, s', r) to the data set β, and uses the new data set to update the agent. In this way, the agent has both prior knowledge to shorten the learning time and can adjust itself in real time according to the real-time status of the network to obtain better results.

[0105] The monitoring block collects information from the environment, continuously generates various statistical information through confirmation characters ACK and timers, and passes the required status information to the agent and reward calculation module. The reward calculation formula is is the throughput of subflow i at time t, which can take into account the throughput of multiple paths of the Multi-Path Transport Control Protocol MPTCP and the fairness of TCP traffic.

[0106] The status information is [thr,c,r',m], where thr, c, r' and m represent throughput, congestion window Cwnd (congestion window), round-trip time RTT and mean deviation of round-trip time mdev_us (mean deviation of RTT), respectively. These data can effectively represent the current network status.

[0107] Since the congestion window c needs to be changed as flexibly as possible, a continuous action space is selected; however, considering that unlimited exploration of the congestion window c will make the agent training difficult, we define the relationship between the congestion window c and the action a as:

[0108]

[0109] where c t is the congestion window at time t, a t is the action at time t.

[0110] The current mainstream reinforcement learning algorithms use the same environment in both training and testing phases, which causes the following problems for the intelligent agents:

[0111] 1) Ignoring situations that have never been encountered during the training phase;

[0112] 2) In different environments, the optimal strategy corresponding to the same state is different.

[0113] Based on this, the present application first pre-trains the agent with prior data, and then makes online adjustments in different environments, so that the agent can obtain the optimal action in different environments. The present application designs a new reinforcement learning algorithm AWAC-LSTM, which changes the MLP network in the original AWAC (Accelerating Online Reinforcement Learning with Offline Datasets) algorithm to a long short-term memory LSTM (Long short-term memory) network to be more suitable for the network field. LSTM has a certain memory effect and is very suitable for sequence data. Since the congestion control algorithm acts on time series data, the AWAC-LSTM algorithm is more suitable for congestion control schemes than the AWAC algorithm.

[0114] AWAC-LSTM alternates between the policy evaluation phase and the policy improvement phase to maximize the reward. AWAC-LSTM trains an off-policy critic and an implicit policy-constrained value actor.

[0115] In the strategy evaluation phase, the Bellman operator is used to estimate the critical Q of the current strategy π π (s,a). In the case of function approximation, the Q function parameter φ k To minimize the error:

[0116]

[0117] Among them, φ k is the Q function parameter for k iterations; is the mathematical expectation; Q φ (s,a) is the Q function value of φ; y has no specific meaning for the convenience of definition; r(s,a) is the reward; The expectation when s′, a′; is the Q function value when φ is iterated k-1 times; γ is the discount factor.

[0118] In the policy evaluation phase, the policy is optimized to obtain the maximum Q function

[0119]

[0120] Among them, θ k+1 For strategy π θ Parameters; is the mathematical expectation; π θ is the actor behavior strategy; π θ (a|s) is the strategy function; Z(s) is the normalized partition function; is the advantage function; λ is the Lagrangian parameter.

[0121] The AWAC-LSTM algorithm is shown in Table 1.

[0122] Table 1

[0123]

[0124]

[0125] As shown in Table 1, the AWAC-LSTM algorithm specifically includes the following contents:

[0126] Initial: Parameter Q φ ,π θ and dataset β are initialized (rows 1-3 in Table 1).

[0127] If in the offline phase: Iteration: The agent samples {(s,a,s′,r) from the dataset β j}, and then update Q using Equation 2 φ , update π using Equation 3 θ (rows 6-8 in Table 1).

[0128] If in the online fine-tuning phase: Iteration: At each time step, the agent interacts with the environment, adds (s, a, s′, r) to β (row 13 in Table 1), and samples {(s, a, s′, r) from the dataset β j} (row 14 in Table 1), and then update Q using Equation 2 φ , update π using Equation 3 θ (rows 15-16 in Table 1).

[0129] Leopard can deploy the reinforcement learning part in user space and the rest in kernel space. Leopard's reinforcement learning part uses the AWAC-LSTM RL algorithm. Leopard can overcome two challenges:

[0130] 1) Previous suboptimal data: Leopard can be trained offline using only suboptimal data and still achieve the best results through online fine-tuning.

[0131] 2) Adjust strategy: When encountering a new environment, Leopard can achieve the optimal result within 10 time steps (about 5 seconds).

[0132] To summarize, the congestion control algorithm provided by the application example of the present application specifically involves a fast convergence mechanism, a strong generalization mechanism, and a mechanism for quickly adapting to new environments and providing consistent high performance, which can achieve fast convergence of the sending rate; can increase the generalization of the CC algorithm and perform better than other algorithms in different environments; quickly adapt to new environments and provide consistent high performance, and perform particularly well in time-varying networks.

[0133] The embodiment of the present application also provides an electronic device (ie, an electronic device), such as a central server, which may include a processor, a memory, a receiver, and a transmitter, wherein the processor is used to execute the multi-path TCP congestion control method based on reinforcement learning mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, such as by bus connection. The receiver may be connected to the processor and the memory via a wired or wireless manner.

[0134] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0135] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the multi-path TCP congestion control method based on reinforcement learning in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the multi-path TCP congestion control method based on reinforcement learning in the above method embodiment is implemented.

[0136] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The one or more modules are stored in the memory, and when executed by the processor, perform the multi-path TCP congestion control method based on reinforcement learning in the embodiment.

[0138] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0139] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0140] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0141] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the multi-path TCP congestion control method based on reinforcement learning are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0142] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0143] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0144] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0145] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-path TCP congestion control method based on reinforcement learning, characterized in that: include: If a change in the current network environment of the target network is detected, the target network state data of the target network at the current moment is obtained; The target network state data is added to a data set containing prior data, and a preset agent is fine-tuned based on the updated data set using a deep reinforcement learning algorithm to obtain a new agent suitable for the current network environment of the target network, so as to perform multi-path TCP congestion control on the target network based on the new agent; wherein the agent is pre-trained offline based on a data set containing prior data using a deep reinforcement learning algorithm; Before acquiring the target network status data of the target network at the current moment, the method further includes: Acquire prior data of the target network, wherein the prior data includes: expert data and / or preset suboptimal data; A data set is set according to the prior data of the target network, where the data set is used to store the corresponding relationship between the state parameter, action parameter, next state parameter and reward parameter of the target network; Based on the deep reinforcement learning algorithm, the preset deep reinforcement learning model is pre-trained using the data set to obtain an intelligent agent that outputs action parameters from the current moment to the next moment according to the state information of the target network at the current moment; The deep reinforcement learning model includes: an AWAC-LSTM model for alternating execution between an online fine-tuning phase and an offline pre-training phase; The AWAC-LSTM model is a model obtained by replacing the MLP network in the AWAC model with the LSTM network in advance.

2. The multi-path TCP congestion control method based on reinforcement learning according to claim 1 is characterized in that: Also includes: Inputting the target network state data of the target network at the current moment into the new agent, so that the new agent outputs the action parameters of the target network from the current moment to the next moment; Multipath TCP congestion control is performed on the network traffic of the target network at the current moment according to the action parameters.

3. The multi-path TCP congestion control method based on reinforcement learning according to claim 1 is characterized in that: If a change in the current network environment of the target network is detected, the target network state data of the target network at the current moment is obtained, including: Using a preset monitoring block to collect environmental status information from the environment of the target network, and generating statistical information corresponding to the environmental status information through an acknowledgment character ACK and a timer; If the current network environment of the target network is known to change according to the statistical information, the target network state data of the target network at the current moment is sent to the preset reward calculation module and the intelligent agent respectively; The reward calculation module is used to calculate the target reward corresponding to the target network state data of the target network at the current moment, and the target reward is sent to the intelligent agent.

4. The multi-path TCP congestion control method based on reinforcement learning according to claim 3 is characterized in that: The step of adding the target network state data to a data set containing prior data, and fine-tuning the preset agent using a deep reinforcement learning algorithm based on the updated data set to obtain a new agent suitable for the current network environment of the target network includes: Adding the target network status data to a data set containing prior data to form a new data set; Based on a deep reinforcement learning algorithm, the new data set and the target reward are used to perform fine-tuning training on the intelligent agent to obtain a new intelligent agent suitable for the current network environment of the target network.

5. The multipath TCP congestion control method based on reinforcement learning according to claim 1, characterized in that: The target network status data includes: The throughput, congestion window, round trip time, and round trip time average deviation of the target network; Wherein, the congestion window and the action satisfy a preset relationship constraint condition.

6. A multi-path TCP congestion control device based on reinforcement learning, characterized in that: include: A network monitoring module is used to obtain target network status data of the target network at the current moment if a change in the current network environment of the target network is detected; An agent fine-tuning module, used for adding the target network state data to a data set containing prior data, and fine-tuning a preset agent based on the updated data set using a deep reinforcement learning algorithm to obtain a new agent suitable for the current network environment of the target network, so as to perform multi-path TCP congestion control on the target network based on the new agent; wherein the agent is pre-trained offline based on a data set containing prior data using a deep reinforcement learning algorithm; Before obtaining the target network status data of the target network at the current moment, the multipath TCP congestion control device based on reinforcement learning further performs the following: Acquire prior data of the target network, wherein the prior data includes: expert data and / or preset suboptimal data; A data set is set according to the prior data of the target network, where the data set is used to store the corresponding relationship between the state parameter, action parameter, next state parameter and reward parameter of the target network; Based on the deep reinforcement learning algorithm, the preset deep reinforcement learning model is pre-trained using the data set to obtain an intelligent agent that outputs action parameters from the current moment to the next moment according to the state information of the target network at the current moment; The deep reinforcement learning model includes: an AWAC-LSTM model for alternating execution between an online fine-tuning phase and an offline pre-training phase; The AWAC-LSTM model is a model obtained by replacing the MLP network in the AWAC model with the LSTM network in advance.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multi-path TCP congestion control method based on reinforcement learning as described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-path TCP congestion control method based on reinforcement learning as described in any one of claims 1 to 6 is implemented.

Images