A network congestion control method, system, device and medium

Abstract

The present application discloses a network congestion control method, system, device and medium, which includes: inputting a state vector used to characterize the current network operation state of a network cluster and a historical action vector used to guide bandwidth resource allocation into a target model based on semantic understanding and reinforcement learning to obtain a target action vector. The target action vector is sent to each end-side network card in the network cluster to control the end-side network card to transmit data according to the bandwidth allocation strategy corresponding to the target action vector. Thus, according to different network states and historical action decisions, the traffic scheduling strategy is dynamically adjusted to improve resource utilization. In addition, the semantic information in the state vector and the historical action vector is extracted through the semantic understanding model, and the extracted information is used to guide the intelligent decision-making process of the reinforcement learning model, that is, the target action vector generated by the target model guides the bandwidth resource allocation to achieve active prediction and avoidance of network congestion risks.

Description

Technical Field

[0001] The present application relates to the field of network control technology, and in particular to a network congestion control method, system, device and medium. Background Art

[0002] With the rapid development of the Internet, the demand for computing resources is increasing day by day, network traffic is showing explosive growth, and network congestion control has become the key to ensuring data transmission efficiency and quality. Especially in a distributed training environment, the communication requirements between computing nodes have a crucial impact on the overall training efficiency.

[0003] The Transmission Control Protocol (TCP) is one of the core protocols for reliable data transmission in the Internet. It plays an important role in solving network congestion. TCP achieves effective management of network congestion through a series of mechanisms. Although TCP has made remarkable achievements in network congestion control, it also faces some challenges and limitations.

[0004] Specifically, TCP's traditional congestion control algorithm cannot effectively utilize network bandwidth and has a certain lag in quickly adjusting the sending rate. In addition, TCP mainly relies on reactive mechanisms to adjust traffic and cannot actively predict and avoid congestion, which often leads to network resource waste and reduced training efficiency when facing complex network environments and dynamically changing traffic patterns.

[0005] It can be seen that how to solve the problem of network congestion and improve resource utilization is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0006] In view of this, one aspect of the present application provides a network congestion control method, the method comprising: Obtaining a state vector for characterizing the current network operation state of the network cluster and a historical action vector for guiding bandwidth resource allocation; Inputting the state vector and the historical action vector into a target model based on semantic understanding and reinforcement learning to obtain a target action vector; The target action vector is sent to each end-side network card in the network cluster to control the end-side network card to perform data transmission according to the bandwidth allocation strategy corresponding to the target action vector.

[0007] Optionally, inputting the state vector and the historical action vector into a target model based on semantic understanding and reinforcement learning to obtain a target action vector includes: Get the current timestamp and the historical feedback reward value of the network cluster; Combining the current timestamp, the historical feedback reward value, the state vector, and the historical action vector into a sample quadruple; The sample quadruple is used as an input of the target model to perform calculations to obtain the target action vector.

[0008] Optionally, the step of calculating the target action vector by using the sample quadruple as an input of the target model includes: Performing an embedding operation on the sample quadruple, and performing a position embedding operation on the historical feedback reward value, the state vector, and the historical action vector to obtain an embedding vector; After concatenating the embedding vectors, transformer processing and fully connected layer processing are performed in sequence to obtain the target action vector.

[0009] Optionally, after controlling the terminal-side network card to perform data transmission according to the bandwidth allocation strategy corresponding to the target action vector, the method further includes: Determining whether the target model reaches a preset convergence condition; If the preset convergence condition is not met, perform the following steps: Obtaining the bandwidth utilization percentage of the end-side network card and the queue length ratio of the switch port; Determining a current feedback reward value of the network cluster according to the bandwidth utilization percentage and the queue length ratio; Optimizing the parameters of the target model according to the current feedback reward value.

[0010] Optionally, obtain a state vector used to characterize the current network operation status of the network cluster, including: Obtaining switch information and link information in the network cluster; the switch information at least includes buffer occupancy, port bandwidth utilization and port queue length; the link information at least includes packet loss rate, transmission delay and throughput; The switch information and the link information are integrated to obtain the state vector.

[0011] Another aspect of the present application provides a network congestion control system, which includes an intelligent agent that implements the network congestion control method and a simulation platform for building the network cluster.

[0012] Optionally, the system further comprises: a network topology management module, a traffic generator and a network monitoring module; The network topology management module is used to store network topology information of a plurality of network clusters; The traffic generator is used to generate data traffic of the network cluster; The network monitoring module is used to collect the switch information and link information in the network cluster in real time; and transmit the collected switch information and link information to the intelligent agent.

[0013] Another aspect of the present application provides a network congestion control device, the device comprising: A vector acquisition module is used to acquire a state vector for representing the current network operation state of the network cluster, and a historical action vector for guiding bandwidth resource allocation; A target vector determination module, used for inputting the state vector and the historical action vector into a target model based on semantic understanding and reinforcement learning to obtain a target action vector; The bandwidth allocation module is used to send the target action vector to each end-side network card in the network cluster to control the end-side network card to perform data transmission according to the bandwidth allocation strategy corresponding to the target action vector.

[0014] Another aspect of the present application provides a network congestion control device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and the processor implements the steps of the network congestion control method when executing the program.

[0015] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of the network congestion control method are implemented.

[0016] The present application provides a network congestion control method, system, device and medium, and the beneficial effects are as follows: thus, according to different network states and historical action decisions, the traffic scheduling strategy is dynamically adjusted to improve resource utilization while improving network throughput and response speed. In addition, through the integration of semantic understanding and reinforcement learning, the semantic information in the state vector and the historical action vector is extracted through the semantic understanding model, and the extracted information is used to guide the intelligent decision-making process of the reinforcement learning model, that is, the target action vector generated by the target model guides the bandwidth resource allocation, so as to actively predict and avoid network congestion risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a network congestion control method provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a network cluster architecture provided in an embodiment of the present application; Figure 3 A flowchart of a network congestion control method provided by another embodiment of the present application; Figure 4A schematic diagram of the architecture of a target model based on semantic understanding and reinforcement learning provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a network congestion control system provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a network congestion control device provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a network congestion control device provided in another embodiment of the present application.

[0018] The figure numbers are as follows: 50 is an intelligent agent, 51 is a simulation platform, 52 is a network topology management module, 53 is a traffic generator, 54 is a network monitoring module, 70 is a memory, 71 is a processor, 72 is a display screen, 73 is an input and output interface, 74 is a communication interface, 75 is a power supply, 76 is a communication bus, 701 is a computer program, 702 is an operating system, and 703 is data. DETAILED DESCRIPTION

[0019] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0020] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0021] Figure 1 A flow chart of a network congestion control method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes: S10: Obtain a state vector for representing the current network operation state of the network cluster and a historical action vector for guiding bandwidth resource allocation; In a specific embodiment, the index data representing the network operation state in the network cluster is collected, and the collected data is integrated into a state vector. Thus, the current network motion state in the network cluster can be reflected by the state vector. Among them, the index data can be at least one data used to represent the current operation state of the network. In an optional embodiment, the index data can include but is not limited to the switch buffer occupancy rate, switch port bandwidth utilization rate, link packet loss rate and transmission delay.

[0022] In addition, it is also necessary to obtain historical action vectors for guiding bandwidth resource allocation. In an optional embodiment, the historical action vectors may be stored in a designated database and directly retrieved from the database when needed.

[0023] It is worth noting that when acquiring the state vector and the historical action vector, the acquisition can be performed in real time. Of course, it is understandable that real-time acquisition will increase the pressure on computing resources. Therefore, in a preferred embodiment, the acquisition can be performed once every preset period, for example, once every 10 seconds, that is, the dynamic guidance of the bandwidth allocation strategy is performed once every preset period.

[0024] The historical action vector and the target action vector refer to vectors that can be used to guide the bandwidth resource allocation of each end-side network card in the network cluster. Therefore, the dimension of the action vector is the same as the number of end-side network cards, and the value of each dimension represents the bandwidth allocation percentage of the corresponding end-side network card.

[0025] Figure 2 This is a schematic diagram of a network cluster architecture provided in an embodiment of the present application. For ease of understanding, the following will be combined with Figure 2 Let's give an example. Figure 2 As shown, in the current network cluster architecture, multiple servers, a first switch and a second switch are included, and each server includes 4 end-side network cards.

[0026] For example, Figure 2 One of the servers shown includes four end-side network cards (NICs), including NIC 0, NIC 1, NIC 2, and NIC 3, and the total bandwidth allocated to each end-side NIC based on the target action vector is 30% for NIC 0, 35% for NIC 1, 20% for NIC 2, and 15% for NIC 3. The dimension in the target action vector is the same as the number of end-side NICs, that is, it includes three dimensions, so the target action vector can be expressed as [30%, 35%, 20%, 15%].

[0027] It should be noted that the historical action vector can be the action vector sent last time, or it can be the average of multiple historical action vectors, which is not limited in this application. In addition, it should be noted that this application does not limit the number of servers, the number of end-side network cards, and the number of switches in the network cluster. In an optional embodiment, Figure 2 The first switch in the network may be a Spine switch, and the second switch may be a Leaf switch.

[0028] S11: Input the state vector and the historical action vector into the target model based on semantic understanding and reinforcement learning to obtain the target action vector; Further, after obtaining the current state vector and historical action vector of the network cluster, the state vector and historical action vector are used as inputs of the target model for calculation, thereby obtaining the target action vector at the current moment. It can be understood that the target model is used to optimize the action vector according to the historical action vector and the state vector representing the current network operation state, thereby realizing dynamic adjustment of the network cluster traffic.

[0029] The target model refers to a model that combines the semantic understanding model with the reinforcement learning module. The introduction of the semantic understanding model can provide the reinforcement learning model with richer context information and more powerful representation capabilities, thereby guiding the reinforcement learning model to learn the optimal bandwidth allocation strategy. In this way, the best action guidance strategy under the current network state, that is, the target action vector, can be obtained.

[0030] It is understandable that in a specific embodiment, the network cluster environment is complex and changeable, and the traditional learning reinforcement algorithm is prone to overfitting and is difficult to adapt to various scenarios. The embodiment of the present application integrates the semantic understanding large model with the reinforcement learning model. The introduction of the semantic understanding large model provides rich contextual information for the reinforcement learning model, enabling it to deeply understand and flexibly respond to complex network environments, ensuring stable performance in diversified network scenarios. Thus, through joint modeling and collaborative optimization, the performance and stability of the congestion control algorithm are improved, providing a new solution for the efficient operation of network clusters.

[0031] S12: Send the target action vector to each end-side network card in the network cluster to control the end-side network card to perform data transmission according to the bandwidth allocation strategy corresponding to the target action vector.

[0032] Furthermore, the target action vector output by the target model is sent to each end-side network card in the network cluster to instruct the end-side network card to send data packets according to the specified bandwidth percentage, thereby effectively controlling network congestion. That is, the reasonable allocation of bandwidth resources is achieved based on the target action vector.

[0033] In an optional embodiment, sending the target action vector may be implemented through a network management protocol or a dedicated control interface, thereby ensuring accurate communication and execution of the instruction.

[0034] Therefore, the network congestion control method provided in the embodiment of the present application dynamically adjusts the traffic scheduling strategy according to different network states and historical action decisions, thereby improving resource utilization while increasing network throughput and response speed. In addition, through the integration of semantic understanding and reinforcement learning, the semantic information in the state vector and the historical action vector is extracted through the semantic understanding model, and the extracted information is used to guide the intelligent decision-making process of the reinforcement learning model, that is, the target action vector generated by the target model guides the allocation of bandwidth resources, thereby realizing active prediction and avoidance of network congestion risks.

[0035] Figure 3 A flowchart of a network congestion control method provided by another embodiment of the present application is shown in FIG. 1 . In an optional embodiment, as shown in FIG. 1 , Figure 3 As shown, the state vector and the historical action vector are input into the target model based on semantic understanding and reinforcement learning to obtain the target action vector, including: S30: Obtain the current timestamp and the historical feedback reward value of the network cluster; S31: Combine the current timestamp, historical feedback reward value, state vector and historical action vector into a sample quadruple; S32: The sample quadruple is used as the input of the target model for calculation to obtain a target action vector.

[0036] In a specific embodiment, when calculating the target action vector through the target model, in addition to the state vector and the historical action vector, it is also necessary to obtain the current timestamp of the current moment and the historical feedback reward value of the network cluster.

[0037] The feedback reward value refers to the indicator data used to characterize the current network performance, such as delay, packet loss rate, throughput, resource utilization, and fairness indicator data. The historical feedback reward value can be used to evaluate the impact of historical action vectors on the network status of the network cluster, and the strategy can be adjusted accordingly to optimize the bandwidth resource allocation in the network cluster.

[0038] Furthermore, the current timestamp, the historical feedback reward value, the state vector and the historical action vector are combined to obtain the input sample of the target model, that is, the sample quadruple. In an optional embodiment, the sample quadruple can be expressed as: ,in, is the current timestamp, is the historical action vector, is the state vector obtained at the current moment, is the historical feedback reward value. It should be noted that during the initial network congestion control, a historical action vector can be randomly generated as the input of the target model.

[0039] Furthermore, the sample quadruple is input into the target model for calculation to obtain a target action vector that can guide bandwidth allocation in the network cluster at the next moment.

[0040] Based on the above embodiments, Figure 3 As shown, the sample quadruple is used as the input of the target model to calculate the target action vector, including: S320: performing an embedding operation on the sample quadruple, and performing a position embedding operation on the historical feedback reward value, the state vector, and the historical action vector to obtain an embedding vector; Figure 4 This is a schematic diagram of the architecture of a target model based on semantic understanding and reinforcement learning provided in an embodiment of the present application. In a specific embodiment, the sample quadruple Input to Figure 4 The calculation is performed in the target model architecture shown.

[0041] Specifically, Figure 4 As shown, the target model is respectively , historical motion vector , state vector and historical feedback reward value Perform embedding operations separately and at the same time, perform embedding operations on historical action vectors. , state vector and historical feedback reward value Perform position embedding operation to obtain the embedding vector.

[0042] Among them, Embedding refers to the technology used to convert high-dimensional data (such as text, images, speech, etc.) into low-dimensional, dense vector representation. Position Embedding is a special kind of embedding used to capture the position information of each element in sequence data. When processing sequence data (such as text, time series, etc.), position information is very important because the same content may have different semantics at different positions.

[0043] S321: After concatenating the embedded vectors, transformer processing and fully connected layer processing are performed in sequence to obtain a target action vector.

[0044] Further, the embedding vectors obtained after the embedding operation are concatenated as follows Figure 4The input of the transformer is processed as shown in the figure. The output of the transformer is used as the output of the fully connected layer. After the fully connected layer is processed, the target action vector is obtained. Among them, Figure 4 As shown in the figure, Transformer processing includes multi-head attention mechanism (Multi-head Attention), weighting and normalization (Add&Norm), and feed forward neural network (Feed Forward Neural Network).

[0045] Therefore, by constructing a target model framework based on semantic understanding and reinforcement learning, the state vector and historical action vector representing the network state are input into the target model to generate accurate network state representation and action sequence prediction, thereby achieving efficient congestion control of the network.

[0046] As an optional embodiment, in order to further improve the reliability of congestion control, after the network card on the control end transmits data according to the bandwidth allocation strategy corresponding to the target action vector, the method further includes: Determine whether the target model has reached the preset convergence condition; If the preset convergence condition is not met, perform the following steps: Get the bandwidth utilization percentage of the end-side network card and the queue length ratio of the switch port; Determine the current feedback reward value of the network cluster based on the bandwidth utilization percentage and queue length ratio; According to the current feedback reward value, the parameters of the target model are optimized.

[0047] In a specific embodiment, the target model can be continuously optimized to improve the network congestion control performance. Specifically, after the control end side network card performs data transmission according to the bandwidth allocation strategy corresponding to the target action vector, it is first determined whether the current target model reaches the preset convergence condition, that is, whether the target model has reached the optimal state.

[0048] If the convergence condition is not met, the current feedback reward value of the network cluster is obtained so that the parameters of the target model can be optimized through the current feedback reward value. Specifically, the bandwidth utilization percentage of the end-side network card and the queue length ratio of the switch port are obtained, and the current feedback reward value is calculated based on the bandwidth utilization percentage and the queue length ratio. The specific calculation formula is formula (1): (1) in, is the number of network cards on the end side, For the The bandwidth allocated to each end-side network card, is the maximum bandwidth of the end-side network card, is the penalty weight, which is used to adjust the penalty for congestion in the reward formula. For the The queue length of the corresponding switch port on the end side, is the maximum queue length of the switch port, is the bandwidth utilization percentage, is the queue length ratio.

[0049] In a specific embodiment, the calculation of the current feedback reward value is intended to encourage improving bandwidth utilization and reducing queue length. When the bandwidth utilization percentage is higher, the current feedback reward value is larger and the queue is longer, indicating that network congestion is currently occurring. According to the change of the current feedback reward value, the parameters of the target model can be iteratively optimized. Through continuous iterative training, the target model can gradually improve the congestion control performance in different network scenarios.

[0050] It should be noted that the preset convergence condition can be that the loss curve of the target model tends to be stable. In another optional embodiment, it can also be that the loss value calculated twice does not exceed a threshold. When the target model reaches the preset convergence condition, the target model is not updated or iterated. In addition, it should be noted that when the target model is updated and iterated, in an optional embodiment, the target model can be iteratively updated by a temporal difference algorithm.

[0051] In an optional embodiment, switch information and link information in the network cluster are obtained; the switch information includes at least buffer occupancy, port bandwidth utilization and port queue length; the link information includes at least packet loss rate, transmission delay and throughput; The switch information and link information are integrated to obtain the state vector.

[0052] In an optional embodiment, it is recommended to collect and obtain switch information and link information through a network monitoring tool or device management interface to provide basic information support for subsequent congestion control. The switch information includes but is not limited to the switch buffer occupancy rate, port bandwidth utilization rate and port queue length. The link information includes but is not limited to key indicators such as link packet loss rate, transmission delay and throughput.

[0053] The switch buffer occupancy rate refers to the ratio of the number of packets stored in the switch queue to the total capacity of the queue. The port bandwidth utilization rate refers to the bandwidth usage of the port at the current moment, which can be expressed as a percentage.

[0054] Furthermore, in an optional embodiment, after the switch information and link information initially collected by the network monitoring tool or device management interface are obtained, data cleaning is performed to eliminate noise and abnormal values ​​in the data. In addition, standardization is performed to convert the data into a data format suitable for processing by the target model, which may include but is not limited to converting percentages into decimal form and normalizing discrete values ​​such as queue length. In this way, the quality and consistency of the obtained information are ensured to facilitate the construction of the subsequent state vector.

[0055] For example, in Figure 2 In the network cluster shown, the switch information and link information include the number of first switches and second switches, port connection status, bandwidth of uplinks and downlinks, switch buffer queue length and other key information that can characterize the network operation status.

[0056] Furthermore, the switch information and link information after preprocessing (including but not limited to data cleaning and standardization) are integrated into a state vector, which fully reflects the current operation status of the network. The construction of the state vector takes into account multiple dimensions such as network congestion, bandwidth usage, and data transmission efficiency, providing accurate input for the target model.

[0057] In the above embodiments, the network congestion control method is described in detail. The present application also provides a corresponding embodiment of a network congestion control system.

[0058] Figure 5 A schematic diagram of the structure of a network congestion control system provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the system includes an intelligent agent 50 that can implement the network congestion control method of any of the above embodiments, and a simulation platform 51 for building a network cluster.

[0059] In an optional embodiment, the simulation platform 51 may be an OpenAI Gym simulation platform. The construction of the simulation platform 51 is used to construct a simulation environment similar to an actual intelligent computing network cluster to implement training and verification of the intelligent agent 50. Various network states and traffic scenarios are simulated in the simulation environment, so that the target model in the intelligent agent 50 can be learned and optimized under safe and controllable conditions. Through comparative testing and result analysis, the effectiveness and stability of the target model are evaluated, and the target model is adjusted and optimized based on feedback to ensure that it can stably and efficiently control network congestion in actual applications.

[0060] In an optional embodiment, a simulation environment similar to an actual intelligent computing network cluster is constructed, including building virtual switches, links, network cards and other network components. The simulation environment should be able to simulate real network traffic, congestion, bandwidth allocation and other behaviors, and provide a reliable platform for training and verification of the target model.

[0061] The target model in the agent 50 is trained in a simulation environment. By simulating different network states and traffic scenarios, the target model learns how to make optimal bandwidth allocation decisions under various circumstances. During the training process, the performance indicators of the target model are monitored in real time, such as changes in feedback reward values, improvements in bandwidth utilization, and reductions in queue length, to evaluate the effectiveness and stability of the model. After the training is completed, the target model is verified by comparing it with traditional congestion control algorithms or other reinforcement learning models to verify the advantages and performance of the model in practical applications.

[0062] In an optional embodiment, the simulation results are analyzed in detail to find out the advantages and disadvantages of the target model, such as poor performance in certain specific scenarios, sensitivity to certain network indicators, etc. Based on the analysis results, the target model is further optimized and improved, for example, the structure of the target model, parameter settings, or calculation functions of feedback reward values ​​are adjusted to improve the overall performance and adaptability of the model.

[0063] Therefore, the network congestion control system provided by the embodiment of the present application provides strong support for the verification and optimization of the target model based on semantic understanding and reinforcement learning. Specifically, the simulation platform 51 can simulate the real data center network environment, making up for the lack of a unified evaluation platform in the past. Therefore, the model can be fully tested and adjusted, and its performance indicators can be evaluated, providing data support for the improvement and optimization of the model.

[0064] In an optional embodiment, if Figure 5 As shown, the network congestion control system further includes: a network topology management module 52 , a traffic generator 53 and a network monitoring module 54 .

[0065] The network topology management module 53 is used to store the network topology information of multiple network clusters. Specifically, the network topology management module 53 is responsible for managing and maintaining the topology information of the network clusters. The network topology management module 53 can identify and record the nodes, links and the connection relationships between them in the network, and provide accurate network topology data for other modules.

[0066] During the simulation process, the network topology management module 53 can flexibly adjust and switch different network topologies according to actual business needs, such as tree topology, ring topology, mesh topology, etc. That is, the network congestion control provided by the present application can support a variety of network topologies and traffic patterns. It can flexibly handle various network environments. Therefore, the network congestion control system provided by the present application can be widely used in various data center network scenarios, providing effective congestion control for network clusters of different sizes and types.

[0067] The traffic generator 53 is used to generate data traffic of the network cluster. Specifically, the traffic generator 53 can simulate data traffic in the network cluster. According to the set traffic mode and parameters, various types of network traffic are generated, such as batch data transmission, real-time communication traffic, random traffic, etc.

[0068] In an optional embodiment, the traffic generator 53 supports multiple traffic distribution models, such as Poisson distribution, normal distribution, etc., to meet the needs of different application scenarios. The generated traffic data is injected into the simulation platform 51 to provide a real traffic environment for the training and verification of the congestion control algorithm.

[0069] Therefore, in a specific embodiment, by properly configuring the traffic generator 53, various complex network traffic scenarios can be simulated to fully test and evaluate the performance of the congestion control algorithm.

[0070] The network monitoring module 54 is used to collect switch information and link information in the network cluster in real time, and transmit the collected switch information and link information to the intelligent agent 50. Specifically, the network monitoring module 54 is an important component responsible for real-time monitoring of network status and flow information in the network congestion control system. Through the interface connection with network devices such as switches and routers, key indicator data in the network are collected in real time, such as the buffer occupancy rate of the switch, the port bandwidth utilization rate, the port queue length, and the packet loss rate, transmission delay and throughput of the link.

[0071] In a specific embodiment, the agent 50 is a key part of implementing congestion control decisions, that is, the network congestion control method in any of the above embodiments can be implemented. Based on the construction of the target model of semantic understanding and reinforcement learning, the agent 50 uses the real-time network status information and traffic data collected by the network monitoring module 54 as input, generates a global network status representation, combines historical action vectors, calculates the optimal target action vector, and then implements efficient scheduling and congestion control of network traffic based on the target action vector.

[0072] In the above embodiments, the network congestion control method is described in detail. The present application also provides a corresponding embodiment of a network congestion control device.

[0073] Figure 6 A schematic diagram of the structure of a network congestion control device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device comprises: A vector acquisition module 60, used to acquire a state vector for characterizing the current network operation state of the network cluster, and a historical action vector for guiding bandwidth resource allocation; A target vector determination module 61 is used to input the state vector and the historical action vector into a target model based on semantic understanding and reinforcement learning to obtain a target action vector; The bandwidth allocation module 62 is used to send the target action vector to each end-side network card in the network cluster to control the end-side network card to perform data transmission according to the bandwidth allocation strategy corresponding to the target action vector.

[0074] In addition, the network congestion control device provided in the embodiment of the present application also includes: The first information acquisition module is used to obtain the current timestamp and the historical feedback reward value of the network cluster; The sample quadruple composition module is used to combine the current timestamp, historical feedback reward value, state vector and historical action vector into a sample quadruple; The calculation module is used to calculate the sample quadruple as the input of the target model to obtain the target action vector.

[0075] The embedding operation module is used to embed the sample quadruple and perform position embedding operation on the historical feedback reward value, state vector and historical action vector to obtain the embedding vector; The embedding vector processing module is used to concatenate the embedded vectors, and then perform transformer processing and fully connected layer processing in sequence to obtain the target action vector.

[0076] A convergence determination module is used to determine whether the target model has reached a preset convergence condition; if the preset convergence condition has not been reached, the second information acquisition module, the current feedback reward value determination module and the optimization module are called; The second information acquisition module is used to obtain the bandwidth utilization percentage of the end-side network card and the queue length ratio of the switch port; A current feedback reward value determination module is used to determine the current feedback reward value of the network cluster according to the bandwidth utilization percentage and the queue length ratio; The optimization module is used to optimize the parameters of the target model according to the current feedback reward value.

[0077] The third information acquisition module is used to obtain switch information and link information in the network cluster; the switch information at least includes buffer occupancy, port bandwidth utilization and port queue length; the link information at least includes packet loss rate, transmission delay and throughput; The integration module is used to integrate the switch information and the link information to obtain a state vector.

[0078] Figure 7 A schematic diagram of the structure of a network congestion control device provided in another embodiment of the present application is shown in FIG. Figure 7 As shown, the network congestion control device includes: a memory 70, for storing a computer program; The processor 71 is used to implement the steps of the network congestion control method mentioned in the above embodiment when executing the computer program.

[0079] The network congestion control device provided in this embodiment may include but is not limited to a laptop computer or a desktop computer.

[0080] Among them, the processor 71 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 71 can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processor, referred to as DSP), a field programmable gate array (Field-Programmable Gate Array, referred to as FPGA), and a programmable logic array (Programmable Logic Array, referred to as PLA). The processor 71 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (Central Processing Unit, referred to as CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 71 may be integrated with a graphics processing unit (Graphics Processing Unit, referred to as GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 71 may also include an artificial intelligence (Artificial Intelligence, referred to as AI) processor, which is used to process computing operations related to machine learning.

[0081] The memory 70 may include one or more computer-readable storage media, which may be non-transitory. The memory 70 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 70 is at least used to store the following computer program 701, wherein, after the computer program is loaded and executed by the processor 71, it can implement the relevant steps of the network congestion control method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 70 may also include an operating system 702 and data 703, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 702 may include Windows, Unix, Linux, etc. Data 703 may include, but is not limited to, relevant data involved in the network congestion control method, etc.

[0082] In some embodiments, the network congestion control device may also include a display screen 72 , an input / output interface 73 , a communication interface 74 , a power supply 75 , and a communication bus 76 .

[0083] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation to the network congestion control device, and may include more or less components than those shown in the figure.

[0084] The network congestion control device provided in the embodiment of the present application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the network congestion control method in the above embodiment.

[0085] It should be noted that, although the operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring these operations to be performed in the specific order shown or to be performed sequentially, or requiring all illustrated operations to be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

Claims

1. A network congestion control method, characterized in that: The method comprises: Obtaining a state vector for characterizing the current network operation state of the network cluster and a historical action vector for guiding bandwidth resource allocation; Inputting the state vector and the historical action vector into a target model based on semantic understanding and reinforcement learning to obtain a target action vector; The target action vector is sent to each end-side network card in the network cluster to control the end-side network card to perform data transmission according to the bandwidth allocation strategy corresponding to the target action vector.

2. The network congestion control method according to claim 1, characterized in that: The step of inputting the state vector and the historical action vector into a target model based on semantic understanding and reinforcement learning to obtain a target action vector includes: Get the current timestamp and the historical feedback reward value of the network cluster; Combining the current timestamp, the historical feedback reward value, the state vector, and the historical action vector into a sample quadruple; The sample quadruple is used as an input of the target model to perform calculations to obtain the target action vector.

3. The network congestion control method according to claim 2, characterized in that: The step of calculating the target action vector by using the sample quadruple as an input of the target model includes: Performing an embedding operation on the sample quadruple, and performing a position embedding operation on the historical feedback reward value, the state vector, and the historical action vector to obtain an embedding vector; After concatenating the embedding vectors, transformer processing and fully connected layer processing are performed in sequence to obtain the target action vector.

4. The network congestion control method according to claim 1, characterized in that: After controlling the end-side network card to perform data transmission according to the bandwidth allocation strategy corresponding to the target action vector, the method further includes: Determining whether the target model reaches a preset convergence condition; If the preset convergence condition is not met, perform the following steps: Obtaining the bandwidth utilization percentage of the end-side network card and the queue length ratio of the switch port; Determining a current feedback reward value of the network cluster according to the bandwidth utilization percentage and the queue length ratio; Optimizing the parameters of the target model according to the current feedback reward value.

5. The network congestion control method according to claim 1, characterized in that: Get the state vector used to characterize the current network operation status of the network cluster, including: Obtaining switch information and link information in the network cluster; the switch information at least includes buffer occupancy, port bandwidth utilization and port queue length; the link information at least includes packet loss rate, transmission delay and throughput; The switch information and the link information are integrated to obtain the state vector.

6. A network congestion control system, characterized in that: The system includes an intelligent agent that implements the network congestion control method described in any one of claims 1 to 5, and a simulation platform for building the network cluster.

7. The network congestion control system according to claim 6, characterized in that: The system also includes: a network topology management module, a traffic generator and a network monitoring module; The network topology management module is used to store network topology information of a plurality of network clusters; The traffic generator is used to generate data traffic of the network cluster; The network monitoring module is used to collect the switch information and link information in the network cluster in real time; and transmit the collected switch information and link information to the intelligent agent.

8. A network congestion control device, characterized in that: The device comprises: A vector acquisition module is used to acquire a state vector for representing the current network operation state of the network cluster, and a historical action vector for guiding bandwidth resource allocation; A target vector determination module, used for inputting the state vector and the historical action vector into a target model based on semantic understanding and reinforcement learning to obtain a target action vector; The bandwidth allocation module is used to send the target action vector to each end-side network card in the network cluster to control the end-side network card to perform data transmission according to the bandwidth allocation strategy corresponding to the target action vector.

9. A network congestion control device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the network congestion control method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the network congestion control method according to any one of claims 1 to 5 are implemented.

Images