Network congestion control method and device, storage medium and program product

Abstract

The invention provides a network congestion control method and device, a storage medium and a program product, and relates to a network communication technology. The method is applied to intelligent agents corresponding to traffic, each traffic in a network corresponds to an independent intelligent agent, the intelligent agents share a congestion control model, and the congestion control model is obtained by a centralized multi-intelligent agent through joint optimization of multi-traffic interaction in a multi-stream environment reflecting a real network based on a global network state. The multi-flow environment can simulate competition behaviors of multiple flows under different network conditions. An intelligent agent acquires a congestion window for network congestion control of the intelligent agent in a next MTP by acquiring a local network state for collecting corresponding traffic in a current MTP and based on a congestion control model according to a congestion window of the current MTP and the local network state sampled in the current MTP, and the congestion window is used for network congestion control of the intelligent agent in the next MTP. Fair bandwidth distribution of data during transmission in a real network environment is effectively guaranteed, and efficient transmission of the data in a network is achieved.

Description

Technical Field

[0001] This application relates to network communication technology, and more particularly to a network congestion control method, apparatus, storage medium, and program product. Background Technology

[0002] In network communication, congestion occurs when the traffic carried by the network exceeds its processing capacity. Network congestion not only increases network latency but can also cause data loss, thus affecting network transmission performance and efficiency. Therefore, network congestion control technologies are needed to avoid network congestion and maintain efficient data transmission within the network.

[0003] Traditional network congestion control algorithms, such as the Transmission Control Protocol (TCP) Reno algorithm, the cube function-based network congestion control algorithm (CUBIC), and the bottleneck bandwidth and round-trip time (RTT)-based network congestion control algorithm (BBR), cannot effectively adapt to the complex network environments of reality, resulting in unfair bandwidth allocation under high traffic contention. Therefore, there is an urgent need to provide a network congestion control scheme that can effectively adapt to the complex network environments of reality, so as to ensure fair bandwidth allocation during data transmission in the network and achieve high efficiency in data network transmission. Summary of the Invention

[0004] This application provides a network congestion control method, apparatus, storage medium, and program product to effectively ensure fair bandwidth allocation when data is transmitted in a real network environment, and to achieve efficient data transmission in the network.

[0005] Firstly, this application provides a network congestion control method applied to agents corresponding to traffic flows. Each traffic flow corresponds to an independent agent, and the agents of each traffic flow in the network share a congestion control model. The congestion control model is obtained by centralized multi-agent joint optimization of multi-traffic interactions based on the global network state in a multi-flow environment that reflects the real network. The multi-flow environment can simulate the competitive behavior of multiple traffic flows under different network conditions.

[0006] The network congestion control method includes:

[0007] The agent acquires the local network status of the corresponding traffic collected during the current Monitoring Time Period (MTP);

[0008] The agent, based on the congestion control model, obtains the congestion window of the next MTP according to the congestion window of the current MTP and the local network state sampled within the current MTP. The congestion window of the next MTP is used by the agent for network congestion control in the next MTP.

[0009] In one possible implementation, the multi-flow environment can configure network parameters to simulate different network scenarios. The network parameters include at least bandwidth, latency, packet loss rate, and buffer size. Furthermore, the multi-flow environment can configure the start time, runtime, and congestion control algorithm type of traffic, and simulate network dynamics by introducing randomness.

[0010] In one possible implementation, the multi-stream environment is built based on Mahimahi and Pantheon-tunnel, and the build process includes:

[0011] Based on Mahimahi, a network topology is constructed, which includes the sending end, the receiving end, intermediate routing nodes, and bottleneck links.

[0012] Configure a multi-flow environment in the network topology;

[0013] In a multi-flow environment, multiple flows are simulated and different flow behaviors are simulated by adjusting the flow parameters. Each flow in the multi-flow is encapsulated by an independent Pantheon-tunnel instance.

[0014] A configuration disturbance mechanism is introduced for multi-flow environments, and a status monitoring module is integrated. The status monitoring module is used to collect the local network status of each flow in real time.

[0015] In one possible implementation, the multi-flow environment is encapsulated in a modular manner and connected to a reinforcement learning training platform via a standardized interface. The reinforcement learning training platform is used to train a congestion control model based on the multi-flow environment.

[0016] In one possible implementation, the congestion control model is trained in the following manner:

[0017] In multi-stream environments, deploy centralized multi-agent systems and a central learner;

[0018] In the centralized multi-agent system, each agent collects the local network state within the current MTP and uploads it to the central learner. It also sends the local network state perceived within the current MTP to the shared congestion control model. The congestion control model obtains the congestion window for the next MTP based on the local network state.

[0019] The central learner integrates the local network states within the current MTP uploaded by each agent into a global network state. Then, it constructs a reward function by linearly combining the multi-dimensional global network states, and comprehensively calculates the reward value of each dimension of the global network state. The global network state includes global bandwidth utilization, global throughput, global packet loss rate, global latency, and network link characteristics. The network link characteristics include bandwidth, latency, and buffer size. After each agent makes a congestion control decision based on the congestion window of the next MTP, the model parameters of the congestion control model are updated by jointly using the global reward function and the multi-agent deep deterministic policy gradient algorithm. The policy optimization objective is to maximize the cumulative global reward.

[0020] In one possible implementation, the multi-agent deep deterministic policy gradient algorithm constructs a multi-dimensional joint state embedding structure in Critic, splices together the local network state, and integrates network topology, network link features, and queue information; during the experience replay process, interaction constraints are introduced to eliminate potential perturbation samples; and a stability correction factor is added during Critic network training, which is used to guide Critic updates to consider multi-agent interference trends.

[0021] Secondly, this application provides a network congestion control device applied to agents corresponding to traffic flows. Each traffic flow corresponds to an independent agent, and the agents of each traffic flow in the network share a congestion control model. The congestion control model is obtained by centralized multi-agent joint optimization of multi-traffic interaction based on the global network state in a multi-flow environment that reflects the real network. The multi-flow environment can simulate the competitive behavior of multiple traffic flows under different network conditions.

[0022] Network congestion control devices include:

[0023] The collection module is used to obtain the local network status of the corresponding traffic collected in the current MTP through the intelligent agent;

[0024] The processing module is used to obtain the congestion window of the next MTP by the agent based on the congestion control model, according to the congestion window of the current MTP and the local network state sampled within the current MTP. The congestion window of the next MTP is used by the agent for network congestion control in the next MTP.

[0025] In one possible implementation, the multi-flow environment can configure network parameters to simulate different network scenarios. The network parameters include at least bandwidth, latency, packet loss rate, and buffer size. Furthermore, the multi-flow environment can configure the start time, runtime, and congestion control algorithm type of traffic, and simulate network dynamics by introducing randomness.

[0026] In one possible implementation, the multi-stream environment is built based on Mahimahi and Pantheon-tunnel, and the build process includes:

[0027] Based on Mahimahi, a network topology is constructed, which includes the sending end, the receiving end, intermediate routing nodes, and bottleneck links.

[0028] Configure a multi-flow environment in the network topology;

[0029] In a multi-flow environment, multiple flows are simulated and different flow behaviors are simulated by adjusting the flow parameters. Each flow in the multi-flow is encapsulated by an independent Pantheon-tunnel instance.

[0030] A configuration disturbance mechanism is introduced for multi-flow environments, and a status monitoring module is integrated. The status monitoring module is used to collect the local network status of each flow in real time.

[0031] In one possible implementation, the multi-flow environment is encapsulated in a modular manner and connected to a reinforcement learning training platform via a standardized interface. The reinforcement learning training platform is used to train a congestion control model based on the multi-flow environment.

[0032] In one possible implementation, the congestion control model is trained in the following manner:

[0033] In multi-stream environments, deploy centralized multi-agent systems and a central learner;

[0034] In the centralized multi-agent system, each agent collects the local network state within the current MTP and uploads it to the central learner. It also sends the local network state perceived within the current MTP to the shared congestion control model. The congestion control model obtains the congestion window for the next MTP based on the local network state.

[0035] The central learner integrates the local network states within the current MTP uploaded by each agent into a global network state. Then, it constructs a reward function by linearly combining the multi-dimensional global network states, and comprehensively calculates the reward value of each dimension of the global network state. The global network state includes global bandwidth utilization, global throughput, global packet loss rate, global latency, and network link characteristics. The network link characteristics include bandwidth, latency, and buffer size. After each agent makes a congestion control decision based on the congestion window of the next MTP, the model parameters of the congestion control model are updated by jointly using the global reward function and the multi-agent deep deterministic policy gradient algorithm. The policy optimization objective is to maximize the cumulative global reward.

[0036] In one possible implementation, the multi-agent deep deterministic policy gradient algorithm constructs a multi-dimensional joint state embedding structure in Critic, splices together the local network state, and integrates network topology, network link features, and queue information; during the experience replay process, interaction constraints are introduced to eliminate potential perturbation samples; and a stability correction factor is added during Critic network training, which is used to guide Critic updates to consider multi-agent interference trends.

[0037] Thirdly, this application provides a network congestion control device, including: a memory and a processor;

[0038] The memory stores the instructions that the computer executes;

[0039] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0040] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0041] Fifthly, this application provides a computer program product, including a computer program that, when executed, implements the first aspect and / or various possible implementations of the first aspect.

[0042] The network congestion control method, apparatus, storage medium, and program products provided in this application are applied to agents corresponding to traffic flows. Each traffic flow corresponds to an independent agent, and the agents for each flow in the network share a congestion control model. In this centralized multi-agent system, each agent can obtain information about the global network state, enabling better coordination of relationships between different flows and achieving fair bandwidth allocation. Furthermore, through centralized control, the interaction of multiple flows can be jointly optimized under the guidance of the global network state, thereby comprehensively considering the needs of different flows and the network's carrying capacity to achieve more reasonable resource allocation, such as bandwidth. The congestion control model is obtained by centralized multi-agent systems jointly optimizing the interaction of multiple flows based on the global network state in a multi-flow environment reflecting a real network. This multi-flow environment can simulate the competitive behavior of multiple flows under different network conditions, allowing the congestion control model to simulate the competitive behavior of multiple flows under different network conditions. This enables a better understanding and prediction of the mutual influence between flows, allowing for more accurate congestion control in real-world network environments. Furthermore, by acquiring the local network state of the corresponding traffic collected within the current MTP through an intelligent agent, and based on the congestion control model, obtaining the congestion window of the next MTP according to the congestion window of the current MTP and the local network state sampled within the current MTP, the congestion window of the next MTP is used by the intelligent agent for network congestion control in the next MTP. This dynamic adjustment mechanism can quickly adapt to changes in network state, providing fair and reasonable bandwidth allocation for data transmission in the network, thereby providing a network congestion control scheme that can effectively adapt to complex network environments in reality and achieve high efficiency in data network transmission. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1 A schematic diagram of a scenario for the network congestion control method provided in the embodiments of this application;

[0045] Figure 2 A flowchart illustrating a network congestion control method provided in an embodiment of this application;

[0046] Figure 3 A schematic diagram of the network congestion control device provided in the embodiments of this application;

[0047] Figure 4 A schematic diagram of the network congestion control device provided in an embodiment of this application.

[0048] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] First, let me explain the terms used in this application:

[0051] An intelligent agent is an entity capable of perceiving its environment and autonomously taking actions to achieve a specific goal. It can be software, hardware, or a system, possessing autonomy, adaptability, and interactivity. Intelligent agents perceive changes in the environment, such as through sensors or data input, and make judgments and decisions based on their learned knowledge and algorithms, thereby executing actions to influence the environment or achieve the desired goal.

[0052] Congestion control aims to prevent or eliminate network congestion when multiple data streams, or multiple traffic flows, share limited network resources. This maintains the network's characteristics of high throughput, low latency, and low packet loss rate, and ensures fairness among different traffic flows, thereby ensuring efficient data transmission in complex network environments.

[0053] Traditional network congestion control algorithms are usually based on heuristic rules, such as packet loss detection and latency measurement, to adjust the congestion window and assume that bandwidth and other network conditions are stable. However, real networks have dynamic characteristics such as wireless network fluctuations and bursts of traffic. Although these traditional algorithms have solved the network congestion problem to varying degrees, they cannot effectively adapt to the complex network environment in reality, resulting in unfair bandwidth allocation under multiple traffic competition.

[0054] To overcome the limitations of existing network congestion control algorithms, this application provides a network congestion control technique based on multi-agent deep reinforcement learning. In a multi-traffic contention environment, this technique can quickly converge to a fair point, achieving fair bandwidth sharing among multiple traffic flows while maintaining high throughput and low latency network performance. By employing a centralized multi-agent training framework, combined with global network state and multi-dimensional reward functions, the global objective of congestion control is optimized, ensuring a comprehensive improvement in convergence speed, fairness, and system stability.

[0055] Specifically, each flow in the network corresponds to an independent agent. These agents share a congestion control model, which is obtained by centralized multi-agent joint optimization of multi-flow interactions in a multi-flow environment that reflects the real network, based on the global network state. The multi-flow environment can simulate the competitive behavior of multiple flows under different network conditions. The agent acquires the local network state of the corresponding flow within the current MTP and, based on the congestion control model, obtains the congestion window for the next MTP according to the congestion window of the current MTP and the sampled local network state within the current MTP. The congestion window of the next MTP is used by the agent for network congestion control in the next MTP.

[0056] Figure 1 This is a schematic diagram of a scenario for the network congestion control method provided in an embodiment of this application. Figure 1 As shown, in the specific network of this application, multiple sending ends and multiple receiving ends transmit data through multiple intermediate routing nodes, forming multiple traffic flows. Each of these traffic flows corresponds to its own independent intelligent agent. The multiple sending ends include sending end 1, sending end 2, ... and sending end N; the multiple receiving ends include receiving end 1, receiving end 2, ... and receiving end M; and the multiple intermediate routing nodes include intermediate routing node 1, intermediate routing node 2, ... and intermediate routing node L. These multiple traffic flows may cause network congestion. In this case, the network congestion control method provided in the embodiments of this application can be used to control network congestion.

[0057] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0058] Figure 2 This is a flowchart illustrating a network congestion control method provided in an embodiment of this application. The network congestion control method provided in this application is applied to agents corresponding to traffic flows. Each traffic flow corresponds to an independent agent, and the agents of each traffic flow in the network share a congestion control model. The congestion control model is obtained by centralized multi-agent joint optimization of multi-traffic interactions based on the global network state in a multi-flow environment that reflects the real network. The multi-flow environment can simulate the competitive behavior of multiple traffic flows under different network conditions.

[0059] like Figure 2 As shown, the network congestion control method includes:

[0060] S201, The agent acquires the local network state of the corresponding traffic collected within the current MTP.

[0061] Within the current MTP, the agent continuously collects or collects the local network status of each traffic corresponding to the agent at a certain sampling frequency, such as throughput, latency, packet loss rate, bandwidth estimation, etc.

[0062] In other words, for agents corresponding to different types of traffic, the methods they use to collect the local network state of that traffic can be the same or different. The collection methods can be continuous collection or collection at a sampling frequency, where the sampling frequency can also vary. For example, for an agent dealing with video traffic, the sampling frequency can be set to 10 samples per second. This is because video streams have high real-time requirements, necessitating more frequent monitoring of the network state to ensure smooth video playback. For an agent dealing with file traffic, the sampling frequency can be set to 5 samples per second, as file transfers have relatively lower real-time requirements but still require ensuring transmission stability and integrity. The sampling frequency for agents dealing with real-time communication traffic is adjusted according to the specific application scenario of real-time communication (such as voice calls or video conferencing). For example, in a voice call scenario, the sampling frequency can be set to 8 samples per second.

[0063] Each agent samples the local network state according to a predefined collection method during its corresponding traffic transmission process. The specific local network states collected by different agents can be the same or different. Taking the agent corresponding to file-type traffic as an example, it collects the following network state characteristic data each time it collects data: throughput, latency, packet loss rate, and bandwidth estimation.

[0064] S202. Based on the congestion control model, the agent obtains the congestion window of the next MTP according to the congestion window of the current MTP and the local network state sampled within the current MTP. The congestion window of the next MTP is used by the agent for network congestion control in the next MTP.

[0065] This congestion control model can be a deep reinforcement learning-based model, obtained by a centralized multi-agent system jointly optimizing the interaction of multiple flows in a multi-flow environment that reflects the real network, based on the global network state. This multi-flow environment simulates the competitive behavior of multiple flows under different network conditions, enabling the congestion control model to better understand and predict the mutual influence between flows, thus achieving more accurate congestion control in real-world network environments.

[0066] The agent feeds the collected local network state of the current MTP into a shared congestion control model. In this model, the global network state is obtained based on the local network states input by each agent. This global network state then coordinates the relationships between different traffic flows, such as bandwidth allocation. The interaction of multiple traffic flows can be jointly optimized under the guidance of the global network state, comprehensively considering the demands of different traffic flows and the network's carrying capacity. This generates a dynamic adjustment amount for the congestion window corresponding to each agent. This dynamic adjustment amount is then logically processed with the congestion window of the current MTP for the agent corresponding to each traffic flow to obtain the congestion window for the next MTP, achieving a more rational allocation of resources such as bandwidth. Specifically, the agent's action output is mapped to the congestion window adjustment amount, achieving a balance between low latency and high throughput through optimization strategies.

[0067] In the process of integrating local network states into global network states, taking global bandwidth utilization as an example, global bandwidth utilization can be calculated by weighted averaging of the local bandwidth utilization of each agent. In this example, the local network state corresponding to global bandwidth utilization is local bandwidth utilization.

[0068] The above logical operation can be understood as follows: when the dynamic adjustment amount is negative, it means increasing the congestion window; when the dynamic adjustment amount is positive, it means decreasing the congestion window. This allows the congestion window of the next MTP to be used to control the traffic of the next MTP. The congestion window represents the maximum amount of traffic that the sender can send.

[0069] Within the next MTP, each agent performs network congestion control on the corresponding traffic based on the calculated congestion window of the next MTP. By employing a centralized multi-agent deep reinforcement learning framework, each agent controls the corresponding traffic through a shared congestion control model. Agents make decisions based on global state information (including real-time local network states and network link characteristics of multiple traffic flows) to optimize their congestion control strategies.

[0070] The network congestion control method provided in this application is applied to agents corresponding to traffic flows. Each traffic flow corresponds to an independent agent, and the agents for each flow in the network share a congestion control model. In this centralized multi-agent system, each agent can obtain information about the global network state, enabling better coordination of relationships between different flows and achieving fair bandwidth allocation. Furthermore, through centralized control, the interaction of multiple flows can be jointly optimized under the guidance of the global network state, thereby comprehensively considering the needs of different flows and the network's carrying capacity to achieve more reasonable resource allocation, such as bandwidth. The congestion control model is obtained by centralized multi-agent systems jointly optimizing the interaction of multiple flows based on the global network state in a multi-flow environment that reflects a real network. This multi-flow environment can simulate the competitive behavior of multiple flows under different network conditions, allowing the congestion control model to simulate the competitive behavior of multiple flows under different network conditions. This enables a better understanding and prediction of the mutual influence between flows, allowing for more accurate congestion control in real-world network environments. Furthermore, by acquiring the local network state of the corresponding traffic within the current MTP through an intelligent agent, and based on the congestion control model, obtaining the congestion window of the next MTP according to the congestion window of the current MTP and the local network state sampled within the current MTP, the congestion window of the next MTP is used by the intelligent agent for network congestion control in the next MTP. This dynamic adjustment mechanism can quickly adapt to changes in network state and achieve stable and efficient congestion control under different network conditions, providing fair and reasonable bandwidth allocation for data transmission in the network, and stable flow control. Thus, it provides a network congestion control scheme that can effectively adapt to complex network environments in reality, achieving high efficiency in data network transmission.

[0071] Based on the above embodiments, the multi-stream environment can configure network parameters to simulate different network scenarios. These network parameters include at least bandwidth, latency, packet loss rate, and buffer size. Furthermore, the multi-stream environment can configure the traffic start time, runtime, and congestion control algorithm type, and simulate network dynamic characteristics by introducing randomness.

[0072] This multi-stream environment is a high-fidelity multi-stream environment capable of simulating the competitive behavior of multiple traffic streams in a network. By flexibly setting network parameters such as bandwidth, latency, packet loss rate, and buffer size, it simulates different network scenarios, ensuring that the training environment can realistically reflect the complex conditions in real-world networks. Furthermore, this multi-stream environment allows for flexible configuration of parameters such as traffic start time, runtime, and congestion control algorithm type, and enhances the robustness and adaptability of training by introducing randomness to simulate dynamic network characteristics.

[0073] Optionally, the multi-flow environment is built based on Mahimahi and Pantheon-tunnel. By constructing a multi-flow environment based on Mahimahi and Pantheon-tunnel, the concurrent behavior of multiple traffic flows under different network conditions is simulated. The construction process includes:

[0074] (1) Based on Mahimahi, construct a network topology structure, which includes a transmitter, a receiver, intermediate routing nodes, and bottleneck links.

[0075] As an example, the system console or user terminal can construct a network topology based on Mahimahi according to the actual needs of the simulation task. This network topology may include multiple transmitters, receivers, intermediate routing nodes, and bottleneck links.

[0076] Optionally, the network topology supports various forms, including but not limited to dumbbell topology, star topology, tree topology, and mesh topology. The network topology can be defined through a topology description file or generated by a graphical interface, and user-defined network topologies are also supported. Each node in the network topology can be configured as a sender, receiver, or intermediate routing node.

[0077] For example, the system console or users can define the network topology by writing a topology description file. The topology description file uses JSON format and defines the node types, connectivity relationships, and link attributes (such as bandwidth, latency, packet loss rate, etc.). The system provides a graphical interface where users can build the network topology by dragging and dropping nodes and connecting links. Users can select the node type (sender, receiver, or intermediate routing node) and set the link attributes in the graphical interface.

[0078] (2) Configure a multi-flow environment in the network topology.

[0079] In the completed network topology, each link can be configured with the following parameters through the link configuration module: (1) link bandwidth, used to limit the maximum transmission rate; (2) propagation delay, used to simulate physical transmission delay; (3) buffer capacity, defining the queue depth of intermediate nodes; and (4) random packet loss probability, simulating packet loss caused by non-congestion (such as interference in wireless links). In addition, the link parameters support dynamic adjustment to simulate the volatility of link states in real networks.

[0080] For example, suppose we construct a dumbbell topology containing transmitter A, transmitter B, receiver E, receiver F, intermediate routing nodes C and D, and a bottleneck link C to D. Link bandwidth is used to limit the maximum transmission rate; the bandwidth of the bottleneck link is set to 10 Mbps, and the bandwidth of other links is set to 100 Mbps. Propagation delay is used to simulate physical transmission delay; the delay of the bottleneck link is set to 50 ms, and the delay of other links is set to 10 ms. Buffer capacity defines the queue depth of intermediate nodes; the buffer capacity of the bottleneck link is set to 50 KB, and the buffer capacity of other links is set to 100 KB. Random packet loss probability is used to simulate packet loss caused by non-congestion, such as interference in a wireless link; the random packet loss probability of the bottleneck link is set to 1%, and the random packet loss probability of other links is set to 0.1%. Link parameters support dynamic adjustment to simulate the volatility of link states in real-world networks. For example, users can dynamically adjust the bandwidth and delay of the bottleneck link through the system console or a graphical interface. Assume that during the simulation, the user dynamically adjusts the bandwidth of the bottleneck link from 10Mbps to 5Mbps and the latency from 50ms to 100ms. The system will update the link parameters in real time to simulate the changes in link status.

[0081] In a dumbbell topology, a multi-stream environment is configured to simulate senders A and B simultaneously transmitting data streams to receivers E and F. The specific configuration is as follows: sender A sends a video stream to receiver E at a data transmission rate of 5 Mbps; sender B sends a file transfer stream to receiver F at a data transmission rate of 3 Mbps. The link configuration module allows for dynamic adjustment of link parameters, enabling observation of the impact of different link states on multi-stream transmission performance. For example, when the bottleneck link bandwidth is dynamically adjusted from 10 Mbps to 5 Mbps, the changes in throughput, latency, and packet loss rate of the two data streams are observed.

[0082] (3) In a multi-flow environment, multiple flows are simulated and different flow behaviors are simulated by adjusting the flow parameters. Each flow in the multi-flow is encapsulated by an independent Pantheon-tunnel instance.

[0083] In a multi-stream environment, the Pantheon-tunnel was modified as follows: (1) Support for parallel operation of multiple streams / multiple agents: (2) Each stream (or agent) runs in an independent Pantheon-tunnel instance and sends traffic through a shared Mahimahi network interface to achieve parallel competition. This design supports the parallel use of multiple different congestion control algorithms, such as agent A using algorithm A1 and agent B using algorithm A2, comparing their performance under a unified bottleneck. (3) Transparent encapsulation and isolation: Each stream is encapsulated through its independent Pantheon-tunnel instance to ensure the independence of control, but shares the same Mahimahi network environment.

[0084] In a multi-stream environment, several transport streams are deployed through a traffic generation module. Each stream continuously sends data packets from the sender to the receiver. The traffic can be configured with the following attributes: (a) start time, duration, and end time; (b) transport protocol type (e.g., TCP, UDP); (c) initial congestion control strategy (for comparison with the learning strategy); (d) data packet size and generation mode (e.g., constant rate, Poisson process, burst mode); and (e) the traffic generator supports asynchronous activation and destruction of streams, thereby enabling high-density, dynamic, and diverse training scenarios.

[0085] For example, the traffic deployment process is executed by the traffic generation module on the server side. Users can configure parameters through the console, specifically including: (a) Determining the number of transport flows and corresponding nodes: Based on the constructed network topology, several pairs of sending and receiving nodes are selected as the start and end points of the transport flows. These nodes have all been defined as source hosts (senders) or destination hosts (receivers) during the topology construction phase. (b) Configuring flow attributes (users issue commands through the system console, and the server writes them to a configuration file or calls an API). Start time, duration, and end time: These allow different flows to partially overlap within the same time period, creating a competitive relationship; Transport protocol type: TCP, UDP, or other protocols can be selected; Initial congestion control strategy: Can be set to a traditional algorithm (e.g., CUBIC, BBR) for comparison with agent strategies; Packet size: Can be precisely set in bytes; Data generation mode: Supports constant rate, Poisson process, burst mode, etc. (c) Start the traffic generator: The server-side traffic generation module instantiates the corresponding flow objects according to the above configuration, and each flow corresponds to an independent traffic generation process; in the modified Pantheon-tunnel environment, each flow is bound to an independent tunnel instance to ensure control isolation. (d) Operation and dynamic management: Flows can be asynchronously activated or destroyed during operation, that is, users or the system can add / remove flows at any time to dynamically adjust the scene density; it supports adjusting the traffic mode during operation, such as temporarily switching the data generation rate to simulate traffic bursts or sudden drops. (e) Integration with the perturbation mechanism: During the flow operation, the perturbation mechanism module can inject events such as bandwidth drops and sudden packet loss according to a set probability or schedule to change the transmission state of the flow, thereby increasing the complexity and diversity of the training environment.

[0086] (4) A configuration disturbance mechanism is introduced for multi-flow environments, and a status monitoring module is integrated. The status monitoring module is used to collect the local network status of each flow in real time.

[0087] To enhance the robustness and generalization ability of the training strategy, a configurable perturbation mechanism module is introduced into the multi-stream environment. This includes, but is not limited to: sudden packet loss events on links; sudden drops / recoveries in link bandwidth; sudden injection or termination of traffic; intermediate node processing latency jitter; and simulated network rerouting or short-term link outages. Perturbation events can be triggered based on a schedule or dynamically based on a probabilistic model, enhancing the agent's adaptability to unexpected situations during training.

[0088] In a multi-flow environment, a status monitoring and data acquisition module is integrated to collect the following performance metrics for each flow in real time: current throughput, average round-trip time, congestion window size, packet loss rate, queue length, and effective bandwidth utilization. The monitoring data is used as part of the agent's input state for training the congestion control model.

[0089] Deploy status monitoring modules on each node to collect performance metrics for each traffic flow in real time. For example, the status monitoring module collects data once per second and sends it to the central monitoring system. The status monitoring module collects performance metrics for each traffic flow in real time and stores the data in the monitoring database. The collected monitoring data is used as part of the agent's input state for training the congestion control model.

[0090] It's important to note that the multi-stream environment enables the parallel operation and independent control of multiple agents within a shared bottleneck link environment. By modifying the Pantheon-tunnel, independent encapsulation and control of concurrent traffic from multiple agents are achieved, allowing them to compete within a shared Mahimahi network environment, overcoming the limitations of existing platforms that only support single streams or single agents. The shared Mahimahi network interface (typically a virtual veth or tap device) serves as the network exit or entry point shared by multiple traffic streams. When multiple traffic streams send or receive data through this shared interface, the link bandwidth corresponding to that interface becomes their common bottleneck. In other words, the shared bottleneck link is constructed based on the shared Mahimahi network interface. Because these traffic streams share the bandwidth resources of the same virtual network device, their transmission performance is limited by the same bottleneck bandwidth, thus forming a "shared bottleneck" and enabling multi-stream parallel competition.

[0091] The steps for constructing a shared bottleneck link are as follows: Bind multiple traffic flows to the same network interface to create bandwidth sharing and a bottleneck. Before starting the virtual environment, create a virtual network interface (such as a veth or tap device) using Mahimahi or a similar network simulation tool. This interface serves as the exit or entry point for multiple traffic flows, and parameters such as bandwidth, latency, and packet loss rate are set to establish the network characteristics of a shared bottleneck link. During virtual environment configuration, bind the exit points of multiple traffic flows to the shared virtual network interface so that all traffic flows share the same link resource. Confirm that the shared bottleneck link for traffic in the virtual environment has been constructed and is functioning correctly. The training platform interacts with the virtual environment through a standard interface, and the agent perceives the status of the shared bottleneck link (bandwidth contention, latency changes, etc.).

[0092] By combining Mahimahi and Pantheon-tunnel, multiple Pantheon-tunnels (each corresponding to a stream or agent) are formed. Encapsulated traffic is sent into a shared bottleneck link (simulating a real network) created based on Mahimahi. The traffic competes with each other, simulating multi-stream interaction in a real environment. Mahimahi provides a unified and controllable network environment (such as bandwidth limitation and adjustable latency). The modified Pantheon-tunnel realizes concurrent multi-stream transmission and independent control of agents. The combination of the two forms a scalable and repeatable multi-stream simulation platform.

[0093] Based on the above embodiments, the multi-flow environment is encapsulated in a modular manner and connected to the reinforcement learning training platform through a standardized interface. The reinforcement learning training platform is used to train and obtain a congestion control model based on the multi-flow environment.

[0094] Each traffic flow is encapsulated through its own independent Pantheon-tunnel instance, ensuring independent control while sharing the same Mahimahi network environment. Standardized interfaces (such as APIs or Sockets) support the following functions: (a) State extraction function: obtaining local or global state vectors; (b) Action receiving function: mapping policy network output actions to actual network behaviors (such as adjusting congestion windows); (c) Reward function calculation: calculating instantaneous reward values ​​based on the current state and action; (d) Environment reset and replay mechanism: supporting multi-round simulation and breakpoint recovery training. This integration method enables centralized training and policy optimization of multiple agents, which is then transformed into distributed policy execution during final deployment.

[0095] For example, create a separate Pantheon-tunnel instance for each traffic flow to ensure independent control of each flow. For instance, create one Pantheon-tunnel instance for traffic from sender A to receiver E, and another Pantheon-tunnel instance for traffic from sender B to receiver F.

[0096] Implement a standardized interface supporting the following functions: state extraction function, action receiving function, reward function calculation, and environment reset and replay mechanism. Define a state extraction function to obtain the local network state of each traffic flow, for example, obtaining the traffic state from sender A to receiver E via API interface; define an action receiving function to map the actions output by the policy network to actual network behavior, for example, adjusting the congestion window via API interface; define a reward function to calculate the instantaneous reward value based on the current state and action, for example, calculating the reward value via API interface; define an environment reset and replay mechanism to support multi-round simulation and breakpoint recovery training, for example, resetting the environment via API interface.

[0097] The encapsulated multi-stream environment is connected to the reinforcement learning training platform through a standardized interface.

[0098] Furthermore, the congestion control model is trained in the following way:

[0099] (1) Deploy centralized multi-agent and central learner in multi-stream environments.

[0100] Each traffic flow corresponds to a reinforcement learning agent, and these agents share the same policy. After connecting the virtual environment and the reinforcement learning training platform, each traffic flow is deployed as an agent by creating an agent instance for each traffic flow in the virtual environment.

[0101] (2) In the centralized multi-agent system, each agent collects the local network state within the current MTP and uploads it to the central learner. The agent also sends the local network state perceived within the current MTP to the shared congestion control model. The congestion control model obtains the congestion window of the next MTP based on the local network state.

[0102] For example, agents of each flow collect local network states within the current MTP, such as throughput, latency, and packet loss rate. The congestion control system feeds the local network states perceived within the current MTP into a shared congestion control model—a neural network based on deep reinforcement learning. The congestion control model generates a corresponding congestion window adjustment based on the perceived local network states. After performing logical operations on this adjustment with the congestion window value of the current MTP for each agent of each flow, the congestion window value of the next MTP is obtained. The congestion window value of the next MTP is then used to perform congestion control on the flow of the corresponding agent in the next MTP.

[0103] (3) The central learner integrates the local network states within the current MTP uploaded by each agent into a global network state. Then, it calculates the reward value of each dimension of the global network state by linearly combining the reward function constructed from the multi-dimensional global network states. The global network state includes global bandwidth utilization, global throughput, global packet loss rate, global latency, and network link characteristics. The network link characteristics include bandwidth, latency, and buffer size. After each agent makes a congestion control decision based on the congestion window of the next MTP, the model parameters of the congestion control model are updated by combining the global reward function and the multi-agent deep deterministic policy gradient algorithm. The policy optimization objective is to maximize the cumulative global reward.

[0104] For example, since the raw observations uploaded by each agent may have different dimensions or be difficult to compare directly, the central learner uses a neural network (e.g., a multilayer perceptron) to map each local state to a high-dimensional feature space, resulting in a unified, dense feature vector to extract effective information from the local states and convert it into a format that is easy to process in subsequent steps.

[0105] Multiple local feature vectors are fused into a single global state representation. The aggregation function can be chosen from: simple aggregation, such as summation, averaging, or maximization (element-wise maximum value); attention-based aggregation, where the central learner dynamically assigns an "importance weight" to the feature vectors of each agent; and graph neural network aggregation, where, if the network topology is known (e.g., a data center network), the network is modeled as a graph, where nodes are agents and edges are physical links. The feature vectors of each agent serve as the initial features for the corresponding node. The central learner uses GNNs (such as graph convolutional networks or graph attention networks) for multi-round message passing. In each round, nodes aggregate information from their neighboring nodes to update their own features; the aggregated vector already contains information from multiple agents. Global metrics (such as network link characteristics like bandwidth, latency, and buffer size) are concatenated with the aggregated vector to form the final global state representation.

[0106] It should be noted that the generated global state is input into the Actor network or Critic network of the central learner. The policy network outputs a global action or an action specific to each agent (such as window resizing), while the value network determines the quality of the current global state to guide policy updates. The updated policy parameters are periodically distributed to each agent to guide their actions in the next time period.

[0107] Based on the global network state, a linear combination reward function is designed. This function constructs a global reward system by linearly combining multi-dimensional global network states, including metrics such as throughput, latency, packet loss rate, fairness, and stability. This system comprehensively calculates the reward value for each dimension of the global network state. The weights of each dimension can be adjusted according to actual needs, and all agents share the same reward function, thereby comprehensively optimizing the congestion control strategy. Agents learn the optimal interaction strategy by maximizing cumulative rewards, achieving fast convergence and fairness.

[0108] For example, the reward function linearly combines fairness metrics with metrics such as throughput, latency, and packet loss rate, as shown in the formula:

[0109] R global = w1·f1 (throughput) + w2·(latency) + w3·f3 (packet loss rate) + w4·f4 (fairness)

[0110] Among them, R globalHere, f1 is the global reward function; f2, f3, f4… are standardization functions. Since the dimensions and numerical ranges of different metrics vary greatly (e.g., throughput is in the Mbps range, latency is in the ms range), direct addition is meaningless. These functions map the original metrics to similar numerical ranges (e.g., [-1,1] or [0,1]) and perform appropriate shape adjustments (e.g., taking the reciprocal or negative value for latency, as lower latency is better); w1, w2, w3, w4… are weighting coefficients that determine the relative importance of each optimization objective in the global reward.

[0111] Fairness is quantified using the Jain index to ensure balanced bandwidth distribution across multiple flows. The Jain index for bandwidth allocation across multiple flows is calculated every MTP cycle:

[0112]

[0113] Where, x i Let J(x1,…,x) be the throughput of the i-th stream. n (x1, ..., x) represents multiple streams. n The Jain index is given by n, where n is the total number of streams and i is the summation index.

[0114] Stability is constrained by the magnitude of congestion window changes. Specifically, a key stability feedback factor is introduced, which reflects the changing trend of the agent's policy output (e.g., policy gradient variance). Its purpose is to maximize system throughput and latency while minimizing training perturbations and policy convergence fluctuations, forming the "third dimension" of the optimization objective.

[0115] For example, a dynamic weighting method is used to adjust the weights of the global reward. For instance:

[0116] The first adjustment method is rule-based state-aware adjustment: Pre-defined rules are used to switch weight sets based on certain global state indicators. If the global queue length exceeds a high threshold, the weight of latency items is increased, and the weight of throughput items is decreased. If link utilization is below a low threshold, the weight of throughput items is increased.

[0117] The second adjustment method involves learning through metanetworks or supernetworks: the central learner has a weight generation network that takes the global state as input and directly outputs the weights w1, w2, w3, w4... of multiple reward items at the current time.

[0118] The third adjustment method is the attention mechanism: different performance metrics are treated as different "information sources," and the attention mechanism is used to calculate the weight of each metric. The model learns to "focus" on the metric that most needs to be optimized at the moment. For example, when the network is about to become congested (the queue grows rapidly), the attention mechanism will automatically assign high attention weights (i.e., importance) to the "latency" and "packet loss" metrics, so that the reward function strongly punishes behaviors that increase latency and packet loss;

[0119] The fourth adjustment method is multi-objective reinforcement learning: multiple objectives are merged into a single value vector, and the learning objective is to find a set of Pareto optimal policies. Weights can be understood as user preferences for different points on the "Pareto front." Weights can be dynamically adjusted to switch between different optimal policies.

[0120] In the multi-agent deep deterministic policy gradient algorithm, a Critic network is designed for each agent. During training, the network uses global information (including the states and actions of multiple agents), but during execution, each agent only uses its own Actor network to make decisions (i.e., only local observations are needed).

[0121] Actor Network (Policy Network): The input is the local observation o of agent i. i (e.g., sending rate, queue latency, packet loss, etc.), the output is action a. i (such as adjusting the sending rate), among which These are the network parameters of the Actor network.

[0122] Critic Network (Value Network): The inputs are the global state s (such as network link utilization, queue latency, etc.) and the agent's actions (a1, a2, ..., a...). N The output is a Q-value estimate of the agent, where These are the network parameters of the Critic network.

[0123] Experience replay: storing the agent's experience (s, a1, a2, ..., a N ,r1,…r N Let ,s′), where s′ is the global state of the next MTP. All agents are assigned the same global reward using the global reward function, i.e., r. i =R global .

[0124] During training, the Actor and Critic networks for each agent are updated.

[0125] For example, the Critic network update minimizes the temporal difference error. For each agent i, the Critic loss function is:

[0126]

[0127] Among them, Q i (s,a1,…,a N y is the predicted value from the Critic network. i Let s' be the target value of the Critic network, s′ be the global state of the next MTP, D represent the experience replay buffer, and E represent the target value of the Critic network. (s,a,s′)～D It expresses expectations about experience.

[0128] The target value y of the Critic network i The calculation formula is:

[0129] y i =r i +γQ′ i (s′,a′1,…,a′ N )

[0130] Where, r i Q′ represents the immediate reward obtained by the i-th agent, γ is the decay coefficient, and Q′ is the instant reward. i For the target Critic network, μ′ j For the target Actor network. Note that when calculating the target Q-value, action a′ j It is determined by the target Actor network based on the observation o′ of the next MTP. j Generated.

[0131] During Critic updates, a global reward function is used to set the reward for each agent as the global reward, i.e., r. i =R global This allows Critic to learn and determine the contribution of each agent's actions to the global reward.

[0132] The Actor network updates Actor parameters using policy gradient ascent:

[0133]

[0134] in, The expected cumulative reward with respect to the Actor network parameters of the i-th agent. The gradient, E s,a～D For the expectation operator, The Actor network outputs the gradient with respect to its parameters. Let be the gradient of the Critic network for the action of the i-th agent.

[0135] Furthermore, the multi-agent deep deterministic policy gradient algorithm constructs a multi-dimensional joint state embedding structure in Critic, splicing together local network states and fusing network topology, network link features, and queue information. During experience replay, interaction constraints are introduced to eliminate potential perturbation samples. A stability correction factor is added during Critic network training to guide updates that consider multi-agent interference trends. This multi-agent deep deterministic policy gradient algorithm effectively reduces variance during training by introducing global state information, thereby accelerating the training process and improving policy stability.

[0136] The differences between this multi-agent deep deterministic policy gradient algorithm and the traditional multi-agent deep deterministic policy gradient algorithm include:

[0137] The first difference is the enhanced structure of the joint state. Constructing a multi-dimensional joint state embedding structure not only splices together local states but also integrates network topology, link characteristics, and queue information, strengthening Critic's ability to model resource competition relationships.

[0138] For example, network topology information (such as node connectivity) is embedded into the state representation, link characteristics (such as bandwidth, latency, etc.) are incorporated into the state representation, and queue information (such as queue length, queue waiting time, etc.) is embedded into the state representation.

[0139] The second difference lies in the collaborative screening mechanism for experience samples. Interactive constraints are introduced during the experience replay process to eliminate potentially perturbed samples and reduce gradient oscillations caused by policy conflicts.

[0140] For example, during the sampling process, samples are screened to remove those that may lead to policy conflicts. For instance, if a sample's policies are highly conflicting with those of other agents under the current policy (i.e., the policy gradient directions are opposite), it is marked as a potential perturbation sample and removed. By analyzing the interactions between samples, samples more beneficial to policy updates are selected. This can be achieved by calculating the policy gradient similarity between samples, retaining samples with consistent policy gradient directions.

[0141] The third difference is the addition of a stability correction factor during Critic network training. By calculating the rate of change of the covariance matrix of the policy output, the Critic update is guided to consider cross-agent perturbation trends, thereby accelerating convergence and improving robustness.

[0142] For example, the rate of change of the covariance matrix of the policy output is calculated as a stability correction factor, which reflects the changing trend of the agent's policy output, such as the variance of the policy gradient. Incorporating the stability correction factor into the optimization objective forms a "third dimension." The optimization objective includes not only maximizing system throughput and minimizing latency, but also minimizing training perturbations and policy convergence variability. During the update process of the Critic network, the stability correction factor guides the Critic to consider cross-agent perturbation trends, enabling the Critic to more accurately determine the value of joint actions, thereby more effectively guiding the update of the Actor network.

[0143] In summary, the network congestion control method provided in this application has at least the following advantages:

[0144] By designing a high-fidelity multi-stream environment training congestion control model, a complex network environment and dynamic characteristics are effectively simulated, enabling congestion control to better adapt to complex network environments. Furthermore, by dynamically sensing changes in the local network state through real-time monitoring of the local network state, the local network states of each agent are adaptively adjusted using a deep reinforcement learning-based policy model. This allows congestion control to better cope with the frequent fluctuations in network state in complex real-world network environments. Especially when network congestion or uneven load occurs, it can automatically reduce the data transmission rate to avoid network overload and ensure data transmission stability. Conversely, when the network load decreases, it can accelerate the data transmission rate to improve bandwidth utilization efficiency.

[0145] A Multi-Agent Deep Reinforcement Learning (MADRL) framework is adopted, where each traffic flow corresponds to a reinforcement learning agent. These agents share a congestion control model and are jointly optimized through a central learner. During training, the traffic agents optimize their congestion control strategies based on global information, such as global bandwidth utilization, global packet loss rate, and global latency, thereby achieving fair and efficient bandwidth allocation.

[0146] During the training phase, policy optimization is performed using a global reward function based on global state information and a joint multi-agent deep deterministic policy gradient algorithm. By introducing global state information, the estimation variance during training is reduced, enabling the policy model to converge quickly to the fairness point, accelerating training and improving policy stability. Furthermore, the reward function achieves fair bandwidth allocation by explicitly optimizing the Jain fairness index.

[0147] By dynamically adjusting the congestion window and combining it with a throughput and latency-weighted reward function, a balance between high throughput and low latency is achieved. In addition, the multi-agent deep deterministic policy gradient algorithm improves the stability of the system by introducing a global state to reduce policy variance.

[0148] In modern banking, especially online and mobile banking services, the stability and efficiency of network communication are crucial. Bank customers use the network to transfer value from user A to user B, and to check their balances; these operations need to be completed quickly and accurately. Network congestion can lead to delays, failures, and even data loss in banking transactions, impacting customer experience and the bank's operational efficiency. Congestion control technology can be applied to a bank's network communication system to optimize network resource utilization, reduce network congestion, and improve the speed and reliability of banking transactions. Specifically, congestion control can: reduce banking transaction delays and improve customer satisfaction; enhance network stability and reliability, reducing the failure rate of banking transactions; and optimize the use of network resources, lowering operating costs.

[0149] For example, suppose a bank's network environment includes multiple data centers, branches, and customer terminals. These nodes are connected via a high-speed network to handle various banking transactions. For instance, the bank's network topology could be a tree topology, with data centers as the root node, branches as intermediate nodes, and customer terminals as leaf nodes. A congestion control model based on deep reinforcement learning is deployed in the bank's network environment to dynamically adjust the congestion windows of network links to optimize network performance. Each link corresponds to a reinforcement learning agent, and these agents share the same congestion control policy. Each agent collects the local network state within the current MTP and uploads it to a central learner. The central learner feeds the collected local network state into the congestion control model, which generates a corresponding congestion window adjustment based on these states. The agent updates the congestion window according to the adjustment and applies it to the flow control of the next MTP. The training steps of the reinforcement learning training platform have been described in detail in the above embodiments and will not be repeated here.

[0150] By deploying congestion control models, banks' network systems can dynamically adjust the congestion window of network links, optimize the use of network resources, reduce latency and failure rates in banking operations, significantly improve the efficiency and stability of network communication, optimize customer experience, and enhance the bank's competitiveness. Through reinforcement learning and deep learning techniques, congestion control models can adaptively adjust network parameters to cope with complex network environments, providing reliable network support for banking operations.

[0151] Figure 3This is a schematic diagram of a network congestion control device provided in an embodiment of this application. This application provides a network congestion control device applied to agents corresponding to traffic flows. Each traffic flow corresponds to an independent agent, and the agents of each traffic flow in the network share a congestion control model. The congestion control model is obtained by centralized multi-agent joint optimization of multi-traffic interactions based on the global network state in a multi-flow environment reflecting the real network. The multi-flow environment can simulate the competitive behavior of multiple traffic flows under different network conditions.

[0152] like Figure 3 As shown, the network congestion control device 30 provided in this application embodiment includes:

[0153] The collection module 301 is used by the agent to obtain the local network status of the corresponding traffic collected in the current MTP;

[0154] The processing module 302 is used by the agent to obtain the congestion window of the next MTP based on the congestion control model, according to the congestion window of the current MTP and the local network state sampled within the current MTP. The congestion window of the next MTP is used by the agent for network congestion control in the next MTP.

[0155] In one possible implementation, the multi-stream environment can configure network parameters to simulate different network scenarios. These network parameters include at least bandwidth, latency, packet loss rate, and buffer size. Furthermore, the multi-stream environment can configure the start time, runtime, and congestion control algorithm type of traffic, and simulate network dynamics by introducing randomness.

[0156] In one possible implementation, the multi-stream environment is built based on Mahimahi and Pantheon-tunnel, and the construction process includes:

[0157] Based on Mahimahi, a network topology is constructed, which includes the sending end, the receiving end, intermediate routing nodes, and bottleneck links.

[0158] Configure a multi-flow environment in the network topology;

[0159] In a multi-flow environment, multiple flows are simulated and different flow behaviors are simulated by adjusting the flow parameters. Each flow in the multi-flow is encapsulated by an independent Pantheon-tunnel instance.

[0160] A configuration disturbance mechanism is introduced for multi-flow environments, and a status monitoring module is integrated. The status monitoring module is used to collect the local network status of each flow in real time.

[0161] In one possible implementation, the multi-flow environment is encapsulated in a modular manner and connected to a reinforcement learning training platform through a standardized interface. The reinforcement learning training platform is used to train a congestion control model based on the multi-flow environment.

[0162] In one possible implementation, the congestion control model is trained in the following way:

[0163] In multi-stream environments, deploy centralized multi-agent systems and a central learner;

[0164] In the centralized multi-agent system, each agent collects the local network state within the current MTP and uploads it to the central learner. It also sends the local network state perceived within the current MTP to the shared congestion control model. The congestion control model obtains the congestion window for the next MTP based on the local network state.

[0165] The central learner integrates the local network states within the current MTP uploaded by each agent into a global network state. Then, it constructs a reward function by linearly combining the multi-dimensional global network states, and comprehensively calculates the reward value of each dimension of the global network state. The global network state includes global bandwidth utilization, global throughput, global packet loss rate, global latency, and network link characteristics. The network link characteristics include bandwidth, latency, and buffer size. After each agent makes a congestion control decision based on the congestion window of the next MTP, the model parameters of the congestion control model are updated by jointly using the global reward function and the multi-agent deep deterministic policy gradient algorithm. The policy optimization objective is to maximize the cumulative global reward.

[0166] In one possible implementation, the multi-agent deep deterministic policy gradient algorithm constructs a multi-dimensional joint state embedding structure in Critic, splices together the local network state, and integrates network topology, network link features, and queue information; during the experience replay process, interaction constraints are introduced to eliminate potential perturbation samples; and a stability correction factor is added during Critic network training, which is used to guide Critic updates to consider multi-agent interference trends.

[0167] The network congestion control device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0168] Figure 4 This is a schematic diagram of the network congestion control device provided in an embodiment of this application. Figure 4As shown, the network congestion control device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the network congestion control device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus.

[0169] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0170] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0171] In the above embodiments, it should be understood that the processor 401 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0172] The memory 402 may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0173] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0174] This application also provides a computer program product, including a computer program that, when executed, implements the above-described method.

[0175] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the above-described method.

[0176] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0177] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0178] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0179] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0181] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0183] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A network congestion control method, characterized in that, It is applied to the intelligent agents corresponding to traffic, wherein each traffic corresponds to an independent intelligent agent, and the intelligent agents of each traffic in the network share a congestion control model. The congestion control model is obtained by centralized multi-agent joint optimization of multi-traffic interaction based on the global network state in a multi-flow environment that reflects the real network. The multi-flow environment can simulate the competitive behavior of multiple traffic under different network conditions. The network congestion control method includes: The agent acquires the local network status of the corresponding traffic collected within the current monitoring period MTP; Based on the congestion control model, the agent obtains the congestion window of the next MTP according to the congestion window of the current MTP and the local network state sampled within the current MTP. The congestion window of the next MTP is used by the agent for network congestion control in the next MTP.

2. The network congestion control method according to claim 1, characterized in that, The multi-stream environment can be configured with network parameters to simulate different network scenarios. These network parameters include at least bandwidth, latency, packet loss rate, and buffer size. Furthermore, the multi-stream environment can be configured with traffic start time, runtime, and congestion control algorithm type, and can simulate network dynamic characteristics by introducing randomness.

3. The network congestion control method according to claim 1 or 2, characterized in that, The multi-stream environment is built based on Mahimahi and Pantheon-tunnel, and the construction process includes: Based on Mahimahi, a network topology is constructed, which includes a transmitter, a receiver, intermediate routing nodes, and a bottleneck link. In the network topology, a multi-flow environment is configured; In the multi-flow environment, multiple flows are simulated and generated, and different flow behaviors are simulated by adjusting the flow parameters. Each flow in the multi-flow is encapsulated by an independent Pantheon-tunnel instance. A configuration disturbance mechanism is introduced for the multi-flow environment, and a status monitoring module is integrated, which is used to collect the local network status of each flow in real time.

4. The network congestion control method according to claim 3, characterized in that, The multi-flow environment is encapsulated in a modular manner and connected to a reinforcement learning training platform through a standardized interface. The reinforcement learning training platform is used to train the congestion control model based on the multi-flow environment.

5. The network congestion control method according to claim 3, characterized in that, The congestion control model was trained in the following way: In the multi-stream environment, a centralized multi-agent system and a central learner are deployed; In the centralized multi-agent system, each agent collects the local network state within the current MTP and uploads it to the central learner. The agent also sends the local network state perceived within the current MTP to the shared congestion control model, which obtains the congestion window for the next MTP based on the local network state. The central learner integrates the local network states within the current MTP uploaded by each agent into a global network state. Then, it calculates the reward value of each dimension of the global network state by linearly combining the reward function constructed from the multi-dimensional global network states. The global network state includes global bandwidth utilization, global throughput, global packet loss rate, global latency, and network link characteristics. The network link characteristics include bandwidth, latency, and buffer size. After each agent makes a congestion control decision based on the congestion window of the next MTP, the model parameters of the congestion control model are updated by jointly using the global reward function and the multi-agent deep deterministic policy gradient algorithm. The policy optimization objective is to maximize the cumulative global reward.

6. The network congestion control method according to claim 5, characterized in that, The multi-agent deep deterministic policy gradient algorithm constructs a multi-dimensional joint state embedding structure in Critic, splices together the local network state, and integrates network topology, network link features, and queue information. During the experience replay process, interaction constraints are introduced to eliminate potential perturbation samples. A stability correction factor is added during Critic network training, which is used to guide Critic updates to consider agent interference trends.

7. A network congestion control device, characterized in that, It is applied to the intelligent agents corresponding to traffic, wherein each traffic corresponds to an independent intelligent agent, and the intelligent agents of each traffic in the network share a congestion control model. The congestion control model is obtained by centralized multi-agent joint optimization of multi-traffic interaction based on the global network state in a multi-flow environment that reflects the real network. The multi-flow environment can simulate the competitive behavior of multiple traffic under different network conditions. The network congestion control device includes: The collection module is used to obtain the local network status of the corresponding traffic collected within the current monitoring period MTP through the intelligent agent; The processing module is used by the agent to obtain the congestion window of the next MTP based on the congestion control model, according to the congestion window of the current MTP and the local network state sampled within the current MTP. The congestion window of the next MTP is used by the agent for network congestion control in the next MTP.

8. A network congestion control device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program that, when executed, implements the method of any one of claims 1 to 6.

Images