Communication network rate control system and method based on multi-agent reinforcement learning

Abstract

This invention discloses a communication network rate control system and method based on multi-agent reinforcement learning. The system includes: a source node for sending data packets at a first transmission rate sent by a central server; multiple intermediate nodes for obtaining their own second transmission rate through a policy network based on received observations; and forwarding data packets according to the obtained second transmission rate; a central server for collecting observations and action values ​​from all nodes, learning a function network and a policy network based on these observations and action values, sending the learned policy network parameters to the intermediate nodes, and sending the first transmission rate to the source node. This invention centrally trains network parameters and distributes the learned parameters to the nodes, enabling inter-node collaboration and cooperation, improving network throughput and reducing network latency, thereby enhancing network performance. This invention can be widely applied in the field of wireless network congestion control.

Description

Technical Field

[0001] This invention relates to the technical field of wireless network congestion control, and in particular to a communication network rate control system and method based on multi-agent reinforcement learning. Background Technology

[0002] The existing network architecture adopts a typical IP / TCP five-layer architecture, which consists of the physical layer, data link layer, network layer, transport layer, and application layer from bottom to top. The transport layer uses the connection-oriented TCP (Transmission Control Protect) protocol. ACK can guarantee reliable data transmission, and the congestion window (cwdm) uses slow start and AIDM (additive increase, multiplicative decrease) algorithms for effective flow control.

[0003] TCP exhibits excellent performance in wired networks, but its performance suffers significantly in multi-hop wireless networks. This is because TCP relies on the ordered arrival of ACKs as a basis for assessing network congestion. Therefore, if packet loss frequently occurs in the network, but is not caused by network congestion (e.g., high error rate, channel collisions, routing failures), the TCP layer's sending window will frequently shrink, leading to a substantial decrease in network throughput. Semi-TCP addresses this problem by decoupling congestion control from the TCP layer to a lower layer (link layer or network layer). This transforms end-to-end flow control into hop-by-hop control, providing a more accurate and rapid understanding of network congestion than the transport layer, and avoiding misjudgments.

[0004] Currently, Semi-TCP implementations mainly employ two mechanisms: 1) Source-side network rate control using a binary search algorithm to find the optimal sending rate at the MAC layer. 2) A congestion feedback mechanism that sends congestion feedback to upstream nodes after reaching a congestion threshold, adjusting the sending rate to avoid congestion. Both methods only control the rate at a single node. Upstream nodes often send data too aggressively, leading to congestion at downstream nodes. While this hop-by-hop adjustment mechanism can improve network performance to some extent, it is too passive, often only responding and adjusting after congestion occurs, lacking coordination and cooperation between nodes. Summary of the Invention

[0005] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide a communication network rate control system and method based on multi-agent reinforcement learning.

[0006] The technical solution adopted in this invention is:

[0007] A communication network rate control system based on multi-agent reinforcement learning, comprising:

[0008] The source node is used to send data packets according to the first sending rate sent by the central server;

[0009] Multiple intermediate nodes, each acting as an agent, are used to obtain their own second transmission rate through a policy network based on received observations; and to forward data packets according to the obtained second transmission rate.

[0010] The destination node is used to receive data packets;

[0011] A central server is used to collect observations and action values ​​from all nodes within a preset time slot interval, learn a function network and a policy network based on the observations and action values ​​from all nodes, send the learned policy network parameters to intermediate nodes, and send a first transmission rate to the source node; the parameters of the policy network include the observations.

[0012] In the initial state, both the source node and the intermediate node randomly generate an action value as their own transmission rate.

[0013] Furthermore, the agent includes a value function network, a policy network, and a target value function network and a target policy network required for learning; wherein, the value function network (Critic) is used to evaluate value; the policy network (Actor) is used to output policy; the target value function network is used to estimate the value of the next state, and the target policy network is used to estimate the next action; the target value function network and the target policy network are obtained using a soft update method.

[0014] Furthermore, the source node and intermediate node are provided with storage space for storing observation values ​​and action values;

[0015] The observed values ​​include: node queue length, data packet reception rate, transmission rate, and average data packet dwell time; the action value is the transmission rate.

[0016] Furthermore, the source node and intermediate nodes are updated periodically using a weighted moving average to avoid drastic fluctuations.

[0017] Furthermore, the node controls the transmission rate in the following ways:

[0018] Set a sending timer, the timeout period of which is the reciprocal of the sending rate;

[0019] When the set sending timer expires, a data packet is sent and the sending timer is restarted; wherein the timeout period of the sending timer is continuously updated according to the sending rate derived from the policy network.

[0020] Furthermore, if the transport layer is UDP, the rate at which data packets are sent downwards is controlled by a sending timer, and the timeout period of the sending timer is the reciprocal of the sending rate reported by the central server; if the transport layer is TCP, the sending window control mechanism is removed, and the sending rate is controlled by a sending timer.

[0021] At intermediate nodes, the network layer uses a tail-removal method to receive data packets. When the queue is full, data packets are lost. The sending rate of data packets is controlled by setting a sending timer.

[0022] Furthermore, the central server updates the function network (Critic) by minimizing the error between the evaluated value and the target value, updates the policy network (Actor) by maximizing the cumulative expected return, and updates the target network of both the function network and the policy network using a soft update method.

[0023] Another technical solution adopted in this invention is:

[0024] A rate control method for communication networks based on multi-agent reinforcement learning includes the following steps:

[0025] Initialization: Randomly generate an action value in the source node as its own transmission rate, and randomly generate an action value in the intermediate node as its own transmission rate;

[0026] The central server collects the observation and action values ​​of all nodes within a preset time slot interval, learns the value function and policy network parameters based on the observation and action values ​​of all nodes, sends the learned policy network parameters to the intermediate nodes, and sends the first transmission rate to the source nodes.

[0027] The source node sends data packets according to the first transmission rate received;

[0028] Each intermediate node is treated as an agent. Based on the received observations, the intermediate node obtains its own second transmission rate through the policy network and forwards data packets according to the second transmission rate.

[0029] The destination node receives data packets, thus enabling data transmission.

[0030] Furthermore, the central server updates the function network (Critic) by minimizing the error between the evaluated value and the target value, updates the policy network (Actor) by maximizing the cumulative expected return, and updates the target network of both the function network and the policy network using a soft update method.

[0031] Furthermore, the node controls the transmission rate in the following ways:

[0032] Set a sending timer, the timeout period of which is the reciprocal of the sending rate;

[0033] When the set sending timer expires, a data packet is sent and the sending timer is restarted; wherein the timeout period of the sending timer is continuously updated according to the sending rate derived from the policy network.

[0034] The beneficial effects of this invention are: This invention uses a multi-agent reinforcement learning framework to centrally train network parameters and distribute the learned parameters to nodes, thereby realizing collaboration and cooperation among nodes, improving network throughput and reducing network latency, and thus improving network performance. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of a communication network rate control system based on multi-agent reinforcement learning in an embodiment of the present invention;

[0037] Figure 2 This is a network model block diagram of a communication network rate control system based on multi-agent reinforcement learning in an embodiment of the present invention.

[0038] Figure 3 This is a flowchart of the internal structure of a node in a communication network rate control system based on multi-agent reinforcement learning, as described in an embodiment of the present invention.

[0039] Figure 4 This is a flowchart of the internal workings of the central server of a communication network rate control system based on multi-agent reinforcement learning, as described in an embodiment of the present invention. Detailed Implementation

[0040] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0041] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0042] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0043] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0044] Multi-agent reinforcement learning integrates reinforcement learning and deep learning methods. Reinforcement learning is used for decision-making, while deep learning is used for perception. Communication and interaction among multiple agents can form a consistent understanding of the environment. Utilizing this understanding for decision-making allows for stable adaptation to environmental changes and maximizes benefits. In wireless networks, each node is not an isolated entity; there is competition and cooperation among them, but their ultimate goal is the same: to optimize network performance, maximizing throughput and minimizing latency. Therefore, a cooperative control mechanism is needed to enable Semi-TCP to exhibit superior performance and better adaptability to environmental changes. Based on this, this invention proposes a communication network rate control scheme based on multi-agent reinforcement learning.

[0045] like Figure 1 and Figure 2 As shown, this embodiment provides a communication network rate control system based on multi-agent reinforcement learning, which solves the problem of lack of coordination and cooperation in Semi-TCP technology. The system specifically includes:

[0046] The source node is used to send data packets according to the first sending rate sent by the central server;

[0047] Multiple intermediate nodes, each acting as an agent, are used to obtain their own second transmission rate through a policy network based on received observations; and to forward data packets according to the obtained second transmission rate.

[0048] The destination node is used to receive data packets;

[0049] A central server is used to collect observations and action values ​​from all nodes within a preset time slot interval, learn a function network and a policy network based on the observations and action values ​​from all nodes, send the learned policy network parameters to intermediate nodes, and send a first transmission rate to the source node; the parameters of the policy network include the observations.

[0050] In the initial state, both the source node and the intermediate node randomly generate an action value as their own transmission rate.

[0051] In this embodiment, see Figure 4 This system employs a centralized training and distributed execution approach. A central server acquires the observations and actions of each agent to learn a value function network (Critic) and a policy network (Actor). The parameters of the policy network are then distributed to each agent node. Each node determines its own transmission rate through its policy network, without interfering with others. The transmission rate of the source node is obtained from the central server, while the destination node directly receives data packets and does not participate in the multi-agent reinforcement learning. After a period of learning and exploration, the multi-agent system can acquire effective decision-making strategies, thereby improving network performance.

[0052] As an alternative implementation, multi-agent cooperative control employs a multi-agent deep deterministic policy gradient algorithm, which combines value-based and policy-based approaches. The value-based network (Critic) is used to evaluate values, while the policy network (Actor) directly outputs the policy.

[0053] Specifically, each agent has two networks, plus a target network for learning, meaning each agent has four networks: a value function network (Critic), a target value function network (Target Critic), a policy network (Actor), and a target policy network (Target Actor). The value function network evaluates the value, the policy network outputs the policy, the target value function network estimates the value of the next state, and the target policy network estimates the next action. The target value function network and the target policy network are obtained using a soft update method.

[0054] See Figure 2 The central server updates the function network (Critic) by minimizing the error between the evaluated value and the target value, updates the policy network (Actor) by maximizing the cumulative expected return, and updates the target network of both the function network and the policy network using a soft update method.

[0055] As an optional implementation, the communication network rate control method operates at the network layer. Intermediate nodes select the node's queue length, packet reception rate, transmission rate, and average packet dwell time as observation values, with the transmission rate as the action value. The reward value is related to the amount of data flowing through the node, packet dwell time, link throughput, and latency, as shown in Table 1 below. Specifically, the source and destination nodes perform rate control at the transport layer.

[0056] Table 1

[0057]

[0058] See Figure 3 As an optional implementation, the transmission rate is controlled by setting a transmission timer at the network layer, with the timeout period set to the reciprocal of the transmission rate. When the transmission timer expires, the network layer sends a data packet down the transmission layer, then restarts the transmission timer. The timeout period of the transmission timer is continuously updated based on the transmission rate derived from the policy network. Specifically, the transmission timer of the source node is set at the transport layer.

[0059] As an optional implementation, the observations for each node are obtained by weighted moving average to avoid drastic fluctuations.

[0060] This embodiment also provides a communication network rate control method based on multi-agent reinforcement learning, including the following steps:

[0061] S1. Initialization: Randomly generate an action value in the source node as its own transmission rate, and randomly generate an action value in the intermediate node as its own transmission rate.

[0062] S2. The central server collects the observation and action values ​​of all nodes within a preset time slot interval, learns the value function and policy network parameters based on the observation and action values ​​of all nodes, sends the learned policy network parameters to the intermediate nodes, and sends the first transmission rate to the source nodes.

[0063] S3. The source node sends data packets according to the received first transmission rate;

[0064] S4. Treat each intermediate node as an agent. The intermediate node obtains its own second transmission rate through the policy network based on the received observation value, and forwards data packets according to the second transmission rate.

[0065] S5. The destination node receives the data packet, thus realizing the data transmission.

[0066] As an optional implementation, for each intermediate node, each node sets up storage space to save observations, including: the node's queue length, data packet reception rate, transmission rate, and average data packet dwell time. The observations are updated periodically using a weighted moving average method. A transmission timer is set, and the reciprocal of the transmission rate obtained from the policy network is set as the timeout time, which is continuously updated. When the transmission timer expires, data packets are sent downwards, and the timer is restarted.

[0067] A central server is set up to train the network parameters. The central server periodically collects the observations and action values ​​of each node and stores them in a list. The central server updates the Critic network by minimizing the error between the evaluated value and the target value, updates the Actor network by maximizing the cumulative expected return, and then updates the target networks of both networks using a soft update method. The central server sends the new policy network parameters to each node for updates, and each node determines its own sending rate based on the latest policy network. For the source node, the central server directly determines the optimal sending rate based on the observations of all nodes and feeds it back to the source node.

[0068] The above system and method will be explained in detail below with reference to the accompanying drawings and specific embodiments.

[0069] To avoid network congestion caused by sudden data spikes or excessive loads, and to maximize network throughput and minimize network latency, the transmission rate is controlled at the network layer. A decision network is trained centrally through a multi-agent network, and then the decision network obtains an appropriate transmission rate, thereby stably sending data packets, achieving inter-node collaboration, and improving network performance.

[0070] The specific solution of this embodiment is as follows:

[0071] Each intermediate node in the network is considered as an agent. In this multi-agent system, each agent determines its own transmission rate based on its own observations through a policy network.

[0072] The source node determines the sending rate through the policy network in the central server, and the destination node directly receives the data packets.

[0073] The central server collects the observations and action parameters of each node at each time slot interval. Based on the observations and action values ​​of all nodes, the central server learns the parameters of the value network and the policy network, and sends the parameters of the policy network to each node for updating.

[0074] In the initial state, each node randomly generates an action value, i.e., the sending rate. After a period of learning and exploration, the multi-agent system can obtain effective decision-making strategies, thereby improving the network performance.

[0075] The detailed operation process of each agent node and the central server is as follows:

[0076] Except for the destination node, each node is configured with storage space to save observations, including: the node's queue length, data packet reception rate, transmission rate, and average data packet dwell time. The observations are updated periodically using a weighted moving average method. A transmission timer is set, with the reciprocal of the transmission rate derived from the policy network set as the timeout period, and this timer is continuously updated. When the transmission timer expires, data packets are sent downwards, and the timer is restarted.

[0077] A central server is set up to train the network parameters. The central server periodically collects the observations and action values ​​of each node and stores them in a list. The central server updates the Critic network by minimizing the error between the evaluation value and the target value, updates the Actor network by maximizing the cumulative expected return, and then updates the target network of both networks using a soft update method.

[0078] The central server sends the new policy network parameters to each node for updates, and each node determines its own sending rate based on the latest policy network.

[0079] To meet specific needs, the following parameters can be customized: The learning rate parameter α can be customized, with a default value of 0.005. The reward discount rate can be customized, with a default value of 0.8. The action value distribution can be customized, typically using a Gaussian distribution by default.

[0080] The relevant modifications to this embodiment for a specific protocol stack instance are as follows:

[0081] (1) At the source node, if the transport layer is UDP, the rate at which data packets are sent downwards is controlled by a sending timer. The timeout of the sending timer is the reciprocal of the sending rate reported by the central server. If the transport layer is TCP, the sending window control mechanism is removed, and the sending rate is controlled by a sending timer.

[0082] (2) At intermediate nodes, the network layer uses the tail-removal method to receive data packets. When the queue is full, data packets are lost. A sending timer is set to control the sending rate of data packets.

[0083] (3) Both types of nodes maintain an observation table, record status information and feed it back to the central server.

[0084] (4) At the destination node, data packets are received directly without any control.

[0085] (5) Set up a central server to collect the observations and training network parameters of each node, update the parameters of the policy network to each node, and directly control the sending rate of the source end.

[0086] In summary, this embodiment has at least the following advantages and beneficial effects compared to the prior art:

[0087] (1) This embodiment uses a multi-agent reinforcement learning framework, which is easy to install and run, highly flexible, and applicable to various network scenarios. Slight modifications are made to the transport layer protocol and network layer. In the transport layer, the congestion window control mechanism is removed, while retaining the original reliability control function; in the network layer, a transmission timer is added to control the transmission rate.

[0088] (2) This embodiment has good robustness and effectiveness. The communication network rate control method based on multi-agent reinforcement learning is more adaptable to environmental changes through cooperation between nodes, and the policy network is constantly updated, which can show stable and good performance in different network topologies.

[0089] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0090] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0091] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A multi-agent reinforcement learning based communication network rate control system, characterized in that, include: The source node is used to send data packets according to the first sending rate sent by the central server; Multiple intermediate nodes, each acting as an agent, are used to obtain their own second transmission rate through a policy network based on received observations; and to forward data packets according to the obtained second transmission rate. The destination node is used to receive data packets; A central server is used to collect observations and action values ​​from all nodes within a preset time slot interval, learn a value function network and a policy network based on the observations and action values ​​from all nodes, send the learned policy network parameters to intermediate nodes, and send a first transmission rate to the source nodes; the parameters of the policy network include the observations. In the initial state, both the source node and the intermediate node randomly generate an action value as their own transmission rate.

2. The communication network rate control system based on multi-agent reinforcement learning according to claim 1, characterized in that, The agent includes a value function network, a policy network, and a target value function network and a target policy network for learning; wherein, the value function network is used to evaluate value; the policy network is used to output policy; the target value function network is used to estimate the value of the next state; and the target policy network is used to estimate the next action.

3. The communication network rate control system based on multi-agent reinforcement learning according to claim 1, characterized in that, The source node and intermediate node are equipped with storage space to save observation values ​​and action values; The observed values ​​include: node queue length, data packet reception rate, transmission rate, and average data packet dwell time; the action value is the transmission rate.

4. In the communication network rate control system based on multi-agent reinforcement learning according to claim 3, the source node and intermediate node periodically update the observations using a weighted moving average method.

5. A communication network rate control system based on multi-agent reinforcement learning according to claim 1, characterized in that, The node controls the sending rate in the following ways: Set a sending timer, the timeout period of which is the reciprocal of the sending rate; When the set sending timer expires, a data packet is sent and the sending timer is restarted; wherein the timeout period of the sending timer is continuously updated according to the sending rate derived from the policy network.

6. A communication network rate control system based on multi-agent reinforcement learning according to claim 5, characterized in that, At the source node, if the transport layer is UDP, the rate at which data packets are sent downwards is controlled by a sending timer, and the timeout of the sending timer is the reciprocal of the sending rate reported by the central server; if the transport layer is TCP, the sending window control mechanism is removed, and the sending rate is controlled by a sending timer. At intermediate nodes, the network layer uses a tail-removal method to receive data packets. When the queue is full, data packets are lost. The sending rate of data packets is controlled by setting a sending timer.

7. A communication network rate control system based on multi-agent reinforcement learning according to claim 1, characterized in that, The central server updates the value function network by minimizing the error between the evaluated value and the target value, updates the policy network by maximizing the cumulative expected return, and updates the target network of both the value function network and the policy network using a soft update method.

8. A rate control method for communication networks based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Initialization: Randomly generate an action value in the source node as its own transmission rate, and randomly generate an action value in the intermediate node as its own transmission rate; The central server collects the observation and action values ​​of all nodes within a preset time slot interval, learns the value function and policy network parameters based on the observation and action values ​​of all nodes, sends the learned policy network parameters to the intermediate nodes, and sends the first transmission rate to the source nodes. The source node sends data packets according to the received first transmission rate; Each intermediate node is treated as an agent. Based on the received observations, the intermediate node obtains its own second transmission rate through the policy network and forwards data packets according to the second transmission rate. The destination node receives data packets, thus enabling data transmission.

9. A communication network rate control method based on multi-agent reinforcement learning according to claim 8, characterized in that, The central server updates the value function network by minimizing the error between the evaluated value and the target value, updates the policy network by maximizing the cumulative expected return, and updates the target network of both the value function network and the policy network using a soft update method.

10. The communication network rate control method based on multi-agent reinforcement learning according to claim 8, characterized in that, The node controls the sending rate in the following ways: Set a sending timer, the timeout period of which is the reciprocal of the sending rate; When the set sending timer expires, a data packet is sent and the sending timer is restarted; wherein the timeout period of the sending timer is continuously updated according to the sending rate derived from the policy network.

Images