A congestion control parameter self-adaptive adjustment method based on semantic level collaborative decision

Abstract

This invention discloses an adaptive adjustment method for congestion control parameters based on semantic-level collaborative decision-making. The method includes: a centralized controller collecting global traffic and topology data, encoding it using a graph attention network to generate a congestion semantic embedding vector, and mapping it to a semantic adjustment vector which is then sent to a switch; after receiving the semantic adjustment vector, the switch port reparameterizes the reward calculation function of the local reinforcement learning agent, allowing global preferences to shape local decision logic through semantic injection; the transient congestion state of the collected port is input into the agent, and congestion control parameters are output and configured using a dynamic reward generator. This invention replaces numerical-level weighted fusion with semantic-level preference guidance, achieving time-scale decoupling between slowly changing upper-layer policies and rapidly changing lower-layer execution, thus improving congestion response consistency and network transmission performance in multi-path scenarios.

Description

Technical Field

[0001] This invention relates to the field of data center network communication technology, and in particular to an adaptive adjustment method for congestion control parameters based on semantic-level collaborative decision-making. Background Technology

[0002] As the underlying infrastructure for critical applications such as cloud computing, big data analytics, and artificial intelligence training, the network transmission performance of data center networks directly determines the service quality and user experience of upper-layer applications. With the widespread adoption of virtualization technology, multi-tenant architecture, and hybrid load deployment models, traffic within data centers exhibits highly dynamic and heterogeneous characteristics: on the one hand, global traffic patterns continuously evolve with changes in time and business needs; on the other hand, traffic characteristics among different clusters within the same data center show significant spatial differences. This complex traffic environment places extremely high demands on congestion control mechanisms, requiring both rapid response capabilities to transient congestion and accurate perception of the overall network status.

[0003] Explicit congestion notification, as the mainstream mechanism for congestion control in current data center networks, has been widely deployed in commercial switches. ECN-based congestion control algorithms, such as the Data Center Transmission Control Protocol and the Data Center Quantized Congestion Notification Protocol, typically include several adjustable key parameters. Among these, the ECN flag threshold directly determines the triggering timing and intensity of congestion feedback, and is a core variable affecting network throughput and end-to-end latency. Traditional ECN threshold setting strategies can be divided into three categories: static configuration, rule-based dynamic adjustment, and adaptive adjustment. Static configuration often adopts conservative settings to accommodate worst-case scenarios when facing dynamically changing traffic, resulting in limited network resource utilization. While rule-based dynamic adjustment can change the threshold according to environmental changes, it relies on manually preset heuristic rules, making it difficult to adapt to unknown changes in traffic patterns.

[0004] In recent years, machine learning-based adaptive congestion control parameter tuning methods have received widespread attention from academia and industry, and can be mainly classified into two categories: distributed methods based on switches and global methods based on centralized controllers. Among distributed methods, reinforcement learning, due to its natural adaptability to dynamic environment interaction and decision-making, has proven to be an effective parameter tuning tool. Existing research has utilized deep reinforcement learning algorithms such as deep double-Q networks and near-end policy optimization to perform congestion state awareness and real-time adjustment of ECN thresholds locally at switch ports. However, congestion control is essentially an end-to-end, full-link collaborative problem. Switches can only observe local state information of their local ports, and independent decisions made based on local observations are unlikely to achieve globally optimal performance across multiple switches. Furthermore, the non-stationary environment caused by multi-agent parallel decision-making further exacerbates the instability of policy convergence. In addition, the computational resources of both the switch control plane and data plane are extremely limited, making it difficult to deploy complex and sophisticated decision models.

[0005] In centralized controller-based methods, the controller collects network-wide traffic distribution and topology information through the control plane, and uses complex algorithms such as graph neural networks and simulated annealing search to globally model the network state and generate parameter configuration suggestions. This type of method can overcome the limitations of local observation and achieve a better global performance trade-off. However, the communication latency between the centralized controller and network devices is typically on the order of milliseconds, and coupled with the computational time required for model inference, this makes it difficult for such methods to respond promptly to transient congestion that evolves at the microsecond level.

[0006] To balance the rapid response advantages of distributed methods with the global perspective advantages of centralized methods, researchers have proposed a centralized-distributed collaborative architecture, such as the CoPTER scheme. The core idea is that a centralized controller generates a global parameter guidance matrix, which is then linearly weighted and fused with the Q-value matrix generated by local reinforcement learning inference on the switch to generate the final configuration parameters. However, in-depth analysis reveals a deep-seated, underappreciated flaw in this collaborative scheme: the semantics of the local reinforcement learning Q-value matrix is ​​"the expected cumulative reward of choosing an action under the current local observations," while the semantics of the global guidance matrix is ​​"the predictive impact of configuring a parameter on the overall network performance under the current global traffic distribution." These two are incommensurable at the decision semantic level. When local congestion is severe and global load is low, the two matrices point in opposite directions to parameter configuration. Simple linear weighting will lead to structural distortion of the decision plane, causing continuous oscillations in parameter configuration within adjacent periods, severely impairing the stability of network transmission performance. Therefore, a new mechanism that can effectively coordinate local and global decisions at the semantic level is urgently needed. Summary of the Invention

[0007] The purpose of this invention is to provide an adaptive adjustment method for congestion control parameters based on semantic-level collaborative decision-making, in order to solve at least some of the problems existing in the prior art.

[0008] The technical solution, a method for adaptive adjustment of congestion control parameters based on semantic-level collaborative decision-making, includes the following steps: Obtain global traffic statistics and topology link status data of the data center network. The global traffic statistics include flow distribution and queue status information at the switch port level. Based on global traffic statistics and topology link status data, graph attention encoding is performed to generate congestion semantic embedding vectors that represent the semantic role of ports in global network congestion propagation. These congestion semantic embedding vectors are then mapped to semantic adjustment vectors used to regulate distributed decision preferences. The semantic adjustment vector is sent to the switch port, and the reward calculation function of the reinforcement learning agent is reparameterized according to the semantic adjustment vector to obtain the dynamic reward generator after semantic injection. The transient congestion observation data of the switch port is collected, input into the reinforcement learning agent, and combined with the dynamic reward generator for decision-making and policy update processing to generate congestion control configuration parameters. Configure the queue management register of the switch port according to the congestion control configuration parameters to complete the adaptive adjustment of the congestion control parameters.

[0009] Beneficial effects: This invention reparameterizes the distributed reinforcement learning reward function by adjusting the semantic vector, replacing the numerical layer weighted fusion with semantic layer preference guidance, eliminating parameter configuration oscillations caused by semantic discrepancies between local and global decisions, achieving time-scale decoupling between the slow changes in the upper-layer global policy and the fast changes in the lower-layer local execution, and improving the consistency of congestion response and network transmission performance in multi-path scenarios. Attached Figure Description

[0010] Figure 1 This is a flowchart of the overall solution of the present invention.

[0011] Figure 2 This is a flowchart of the process for generating congestion semantic embedding vectors according to the present invention.

[0012] Figure 3 This is a flowchart of the present invention mapping congestion semantic embedding vectors to semantic adjustment vectors.

[0013] Figure 4 This is a flowchart illustrating the reparameterization process of the reward calculation function for a reinforcement learning agent according to the present invention.

[0014] Figure 5 This is a flowchart of the present invention for collecting transient congestion observation data of switch ports and inputting it into a reinforcement learning agent.

[0015] Figure 6 This is a flowchart of the decision-making and strategy update process of the present invention, which combines a dynamic reward generator.

[0016] Figure 7 This is a flowchart illustrating how the present invention configures the queue management register of a switch port based on congestion control configuration parameters. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] like Figure 1 As shown, this embodiment describes in detail the data processing flow of a congestion control parameter adaptive adjustment method based on semantic-level collaborative decision-making, specifically including: Step S1: Obtain global traffic statistics and topology link status data of the data center network. The global traffic statistics include flow distribution and queue status information at the switch port level.

[0019] According to one aspect of this application, the data acquisition is performed by a global network acquisition module deployed on a centralized controller in a data center. This is achieved through a bidirectional streaming gRPC connection established with the control plane agents of all switches across the network, at a preset second-level acquisition cycle (exemplarily set to T). s Every 5 seconds, a status subscription request is initiated to all active physical ports of the switch. In response to this request, the hardware monitoring unit of each switch port packages and reports the accumulated local statistical summary data for the current period, forming the original source of global traffic statistical feature data.

[0020] Specifically, for each switch port i, the data packets it reports must contain at least the following two types of information: First, there is the flow distribution data at the switch port level. This data is used to characterize the macroscopic distribution of traffic flowing through the port in terms of both rate and quantity. For example, the reported flow distribution is encoded in the format of a log-interval histogram. For instance, an integer array containing 10 buckets [125, 48, 21, 9, 3, 1, 0, 0, 0, 0], where: The value of the 0th bucket (index 0) is 125, indicating that there are 125 active data streams with a transmission rate in the range [0, 100) Mbps in the current period. The value of the first bucket (index 1) is 48, which means that there are 48 streams with a rate falling within the (100, 500) Mbps range. The value of the second bucket (index 2) is 21, corresponding to a rate range of (500, 1000) Mbps; And so on, the bucket level boundaries increase exponentially, with the highest bucket level covering the (10, +∞) Gbps range.

[0021] This histogram data provides a fine-grained view of the port load composition for centralized controllers, enabling them to distinguish whether the ports primarily carry a large amount of low-speed background flow or a small amount of high-speed elephant flow.

[0022] Secondly, the queue status information at the switch port level. This information reflects the instantaneous congestion level of the port. In this embodiment, the queue status information includes at least the following two key indicators: The normalized value of the current queue length, qlen norm Specifically, this is a 32-bit floating-point number, such as 0.23. The calculation process is as follows: First, the instantaneous queue depth register value qlen is read from the queue management module of the switching chip. raw (For example, 16384 bytes), then divide that value by the total buffer capacity Buf configured for the port. total (For example, a typical configuration for a 100G port is 12MB, or 12,582,912 bytes), thus obtaining the normalized ratio. The larger this value, the more severe the backlog in the port buffer.

[0023] Average queue length change rate: Specifically a 32-bit floating-point number, such as 0.05 or -0.02. This value is obtained by calculating the difference between the current exponentially weighted moving average queue length and the average queue length of the previous period, and is used to characterize the evolution trend of congestion: a positive value indicates that congestion is worsening, and a negative value indicates that congestion is easing.

[0024] Furthermore, in addition to the aforementioned port-level traffic statistics, the centralized controller also synchronously acquires the topology link status data of the data center network. In a preferred embodiment, this data is acquired through two methods: first, by reading static physical link connection relationships from the network topology management database to construct the graph structure basis of nodes and edges; second, by collecting the real-time bandwidth utilization U of each physical link through the periodic interaction of in-band network telemetry or link layer discovery protocol messages. ij (e.g., 0.78) and link propagation delay D ij(e.g., 250.0 nanoseconds). This dynamic link state data will be used in subsequent steps to assign weights to the edges of the graph neural network, enabling the network topology model to accurately reflect the current congestion propagation path.

[0025] According to a further improvement in this embodiment, after receiving heterogeneous and multi-source raw data, the centralized controller first performs data cleaning and consistency verification operations to remove incomplete data records caused by network packet loss or device response timeouts. Subsequently, Z-Score normalization is performed on all numerical features to eliminate dimensional deviations caused by differences in physical bandwidth (e.g., 25G, 100G, 400G) and buffer capacity between different ports, ensuring the numerical stability of subsequent graph neural network processing. Through the above acquisition, splicing, and preprocessing processes, the centralized controller obtains a complete data foundation for describing the overall macroscopic operating status of the network, laying the foundation for subsequent semantic encoding and decision guidance.

[0026] Step S2: Based on global traffic statistics feature data and topology link status data, perform graph attention encoding to generate congestion semantic embedding vectors that represent the semantic role of ports in global network congestion propagation, and map the congestion semantic embedding vectors into semantic adjustment vectors for adjusting distributed decision preferences.

[0027] This embodiment transforms the acquired flattened statistics into low-dimensional, interpretable preference parameters that have clear guiding value for distributed decision-making. The detailed process of graph attention encoding in this embodiment is as follows.

[0028] First, the input tensor of the graph attention network is constructed based on global traffic statistics and topology link status data. Specifically, for N active switch ports in the entire network, the normalized flow distribution histogram data, the normalized value of the current queue length, the average queue length change rate, and the switch device model identifier of the port (converted to a One-Hot encoded vector) are concatenated along the feature dimension to generate a D-dimensional tensor for each port i. in (Exemplary setting D) in The initial feature vectors of all ports (=32) are obtained. The initial feature vectors of all ports are sorted according to their topological identifiers to form the initial feature matrix X of the nodes.

[0029] Simultaneously, a weighted adjacency matrix A is constructed using the topology link state data. In one optional implementation, for any physical link connecting port i and port j, the corresponding adjacency matrix edge weight A is... ij It is not a simple binary connection identifier, but rather an edge weight scalar calculated by weighted fusion of link bandwidth utilization and propagation delay. The calculated A... ijNumerically, it represents the current congestion cost of the link: the more congested or delayed the link is, the higher its edge weight, and the greater the propagation strength in the subsequent attention aggregation process.

[0030] Next, the initial feature matrix X of the nodes and the weighted adjacency matrix A are input into a pre-trained offline multi-head graph attention network. For example, this network contains two GAT convolutional layers, each using K=4 attention heads. Further, in the first GAT convolutional layer, for any center node i, the network first performs a linear transformation on its features using a learnable shared weight matrix W, and then calculates the original attention score between it and its neighboring node j. In this embodiment, the edge weights A in the weighted adjacency matrix are used... ij Introduced as a scaling factor into the calculation of the attention coefficient, this improvement allows neighboring nodes of links with higher congestion costs to receive higher attention weights during the feature aggregation process of the current node, thereby strengthening the modeling of congestion signal propagation along the actual physical topology.

[0031] After feature aggregation via a multi-head attention mechanism, the outputs of each head are concatenated or averaged, and then processed similarly by a second-layer GAT. The final network output is a real-valued vector of dimension d (exemplarily set to d=128) corresponding to each port. This vector is the congestion semantic embedding vector, representing the semantic role of port i in the global congestion propagation of the network. For example, ports located at the incast convergence root node exhibit significantly high activation values ​​in a specific dimension, forming a spatial "bottleneck semantic cluster"; while ports located at lightly loaded leaf nodes have relatively flat and isotropic vector distributions.

[0032] After generating the congestion semantic embedding vector, a mapping process is performed to convert it into a semantic adjustment vector for regulating distributed decision preferences.

[0033] According to one aspect of this application, the centralized controller invokes a semantic mapping network f pre-trained through meta-learning. phi The network's structure, exemplarily, consists of two fully connected hidden layers (64 and 16 dimensions respectively) and a linear output layer, using ReLU as the activation function. Its network parameters phi are obtained through meta-learning optimization on a historical traffic dataset, with the optimization objective being: for any given semantic embedding h... i , by f phi (h i The generated semantic adjustment vector can guide the distributed agent to reach the optimal policy in the network state with the fastest convergence speed.

[0034] Specifically, during the online inference phase, the congestion semantic embedding vector h generated in the previous steps is... iThe pre-trained semantic mapping network is directly input. After feature compression and nonlinear transformation in the hidden layers, the network outputs a low-dimensional semantic adjustment vector v. i For example, if we set the output dimension k=4, then v i It contains four specific floating-point values, such as [0.82, -0.15, 0.61, 0.93]. Each dimension of these values ​​has a clearly interpretable physical meaning, directly corresponding to a preference tendency in distributed decision-making. The first dimension (value 0.82): represents the throughput preference coefficient. The closer this value is to 1, the more important it is that from a global perspective, this port should prioritize high link utilization, and queue backlog can be tolerated to a certain extent.

[0035] The second dimension (value -0.15): characterizes the latency sensitivity coefficient. This value is negative and close to 0, indicating that the current global network has a high tolerance for queuing latency and there is no need to aggressively empty the queue.

[0036] The third dimension (value 0.61): characterizes the ECN labeling aggression coefficient. The larger this value, the more likely it is to recommend setting a lower ECN labeling threshold to feed back congestion signals to the source earlier.

[0037] The fourth dimension (value 0.93): represents the policy stability coefficient. The larger this value, the more likely the local agent is to limit frequent switching of ECN configurations in order to maintain the continuity of decision-making.

[0038] Through two consecutive processing stages—graph attention encoding and semantic mapping—the centralized controller completes the semantic extraction and transformation process from raw statistical data to executable decision preferences.

[0039] Step S3: Send the semantic adjustment vector to the switch port, and reparameterize the reward calculation function of the reinforcement learning agent according to the semantic adjustment vector to obtain the dynamic reward generator after semantic injection.

[0040] This step enables the physical transfer of global semantic preferences from the centralized controller to the distributed switch ports, and completes the deep embedding of decision logic locally.

[0041] First, the semantic adjustment vector is distributed. In this embodiment, the centralized controller distributes the generated semantic adjustment vector v corresponding to port i. iThe four floating-point values ​​in [0.82, -0.15, 0.61, 0.93] are converted to a single-precision floating-point binary representation according to the IEEE 754 standard, totaling 16 bytes. Subsequently, the controller adds a gRPC protocol header containing port identifiers and configuration type fields to this payload data, encapsulating it into a semantic configuration delivery message. In a preferred embodiment, this message is sent to the control plane agent process of the target switch via a dedicated out-of-band management network channel in the data center. The agent process listens for the message on a specific port, confirms data integrity through cyclic redundancy check, and then deserializes the payload to restore the four floating-point values. The agent process further writes these values ​​into the switch's operating system's shared-memory database or Redis key-value store, indexing them with specific keys. At this point, the semantic conditioning vector has been successfully deployed to the target switch's local storage area.

[0042] Secondly, semantic reparameterization of the reward calculation function is performed. This process is completed by the semantic injection module deployed in the reinforcement learning agent at this port. Specifically, in the decision loop code of the reinforcement learning agent, there is a dedicated function module for calculating immediate rewards. In traditional reinforcement learning settings, this reward function typically has a fixed static form.

[0043] Further improvements to this application involve dynamically reconstructing the reward calculation function. The semantic injection module first reads the issued and cached semantic adjustment vector v through a shared memory access mechanism. i The current value. For example, the specific values ​​for the four dimensions were read and obtained [v]. i,1 , v i,2 , v i,3 , v i,4 Subsequently, the module reparameterizes the internal structure of the original reward function, modifying it into a differentiable function with the semantic adjustment vector as the independent variable.

[0044] Furthermore, through dynamic replacement of code logic and parameter binding, the reward calculation logic of the reinforcement learning agent has undergone a fundamental change. The originally fixed reward weights have been replaced with dynamic coefficients that are continuously parameterized by the upper-layer semantic vector. For example, for the semantic adjustment vector v... i = [0.82, -0.15, ...]. After Sigmoid mapping, the weight coefficient of the throughput term is approximately 0.69, while the weight coefficient of the latency term is approximately 0.46. Under this configuration, the agent will place relatively more emphasis on achieving the throughput metric when evaluating value. Conversely, if the centralized controller detects an incast burst precursor, it will issue an updated semantic adjustment vector of v. iIf the weight is [0.35, 0.88, ...], then the throughput weight drops to approximately 0.58, while the latency weight rises to approximately 0.70. At this point, even when faced with the exact same local observation state, because the inherent preference structure of the reward calculation function has been restructured, the agent will instinctively tend to choose actions that can quickly reduce queue length and queuing latency.

[0045] After the reparameterization process is completed, the reward calculation module used by the reinforcement learning agent on this port is transformed into a dynamic reward generator with semantic injection. In each subsequent microsecond-level decision cycle, this generator will continuously calculate the reward based on the current observation, but its calculation logic has been profoundly shaped by the global semantic preferences issued from the upper layer.

[0046] Step S4: Collect transient congestion observation data of the switch port, input it into the reinforcement learning agent, and combine it with the dynamic reward generator for decision-making and policy update processing to generate congestion control configuration parameters.

[0047] First, the transient congestion observation data of the switch port is collected and preprocessed. According to one aspect of this application, in order to capture the micro-burst characteristics of data center traffic and achieve rapid response, a high-precision hardware timer is set to periodically trigger a state snapshot acquisition at extremely short time intervals Δt (exemplarily set to 50 microseconds). Specifically, each time it is triggered, the state awareness module obtains the following raw observation values ​​from the queue management unit of the switching chip through direct memory access or reading hardware registers: Instantaneous queue depth qlen raw : A 24-bit unsigned integer read from the TM_QUEUE_DEPTH register. For example, if the current snapshot reading is 20480, it means that the number of bytes of data backed up in the buffer is 20KB.

[0048] Outgoing port transmit rate txRate raw The value is calculated by the difference between two adjacent snapshots of the EGR_TX_BYTE_CNT counter. For example, the count value of the current period has increased by 625,000 bytes compared to the previous period. Since Δt = 50 microseconds, the converted instantaneous transmission rate is 100Gbps, which is exactly equal to the nominal line speed of the port.

[0049] ECN tag packet count increment txRate raw m : The difference between two snapshots, read from the ECN_MARKED_CNT register as a 32-bit integer. For example, this difference is 320 for the current period, indicating that 320 packets had their CE congestion flag set in the past 50 microseconds.

[0050] Furthermore, since micro-burst traffic causes the raw observations to fluctuate drastically between adjacent periods, directly using the raw values ​​as neural network input will result in high variance and instability in the policy network output. In a preferred embodiment, the state-aware module performs exponentially weighted moving average (EWMA) filtering on the raw observations. After EWMA processing, high-frequency noise in the observation sequence is effectively suppressed, resulting in smoothed observation data.

[0051] Subsequently, the smoothed data is normalized to ensure its numerical range matches the saturation range of the activation function in the neural network's input layer. For example, dividing the smoothed instantaneous queue depth by the total port buffer capacity yields qlen. norm The link utilization rate (txRate) is obtained by dividing the smoothed transmission rate by the nominal port bandwidth. norm According to a further improvement of this embodiment, in order to provide the reinforcement learning agent with temporal dynamic information on congestion evolution, the state vector not only includes the normalized observation value at the current moment, but also concatenates the historical states of the past K consecutive time steps (exemplarily setting K=4). Finally, the reinforcement learning state vector s at the current time step t is constructed. t It is a tensor containing rich local transient and short-term historical information.

[0052] Next, the state vector is input into the reinforcement learning agent for decision-making inference. In this embodiment, the reinforcement learning agent adopts a soft actor-critic algorithm framework, which includes a policy network and two Q networks. To meet the 50-microsecond decision latency constraint, the policy network is pre-compiled and deployed into the programmable matching-action pipeline of the switch, or inference is accelerated by a dedicated on-chip NPU.

[0053] Specifically, the constructed state vector s t The input is fed into the policy network, and after several layers of fully connected feedforward operations and nonlinear activation, it outputs a vector representing the probability distribution of action selection within the discretized congestion control parameter action space. For example, if the action space is defined as the ECN low-label threshold K... min ={10KB, 20KB, 30KB, 40KB} with high labeling threshold K max The Cartesian product of {50KB, 60KB, 70KB, 80KB} gives an action space size of |A| = 16. The 5th element in the vector output by the policy network (corresponding to action index 5, representing K) min =20KB, K max The value of 0.15 for (70KB) indicates that the probability of selecting this configuration combination is 15%.

[0054] Based on this probability distribution, the decision module performs a random sampling operation to determine the target action index a to be performed in the current period. t This probability-based random sampling, rather than greedy selection, allows the agent to maintain necessary exploration of the action space under the guidance of semantic vectors, avoiding premature convergence of the strategy to local optima.

[0055] Next, the dynamic reward generator calculates feedback from the computational environment and performs policy update processing. This is done during the execution of action a. t After writing the corresponding ECN threshold into the hardware register, the agent waits for the next Δt period to arrive before the observation environment transitions to the new state s. t+1 At this point, the decision module invokes the dynamically generated reward generator after semantic injection, substituting the link utilization and queue length during the state transition process into the aforementioned reparameterized reward formula to calculate the reward for action a. t Instant rewards r t For example, if the current semantic adjustment vector emphasizes low latency, a small increase in queue depth will result in r t The significant decrease in the value creates a strong negative feedback loop.

[0056] The agent will contain experience tuples (s) containing complete transfer information. t , a t ,r t , s t+1 The data is stored in a local priority experience replay pool. Every preset number of decision steps (e.g., every 100 actions), the agent randomly samples a batch of data from the replay pool and calculates the temporal difference error and policy gradient using the loss function of the SAC algorithm. Incremental updates are performed on the policy network parameters and Q network parameters using the backpropagation algorithm. In an optional implementation, to avoid affecting the real-time decision latency of the data plane, the model weight update operation is asynchronously executed by the CPU core of the switch's backend control plane. The updated weight parameters are then atomically refreshed to the weight storage area of ​​the data plane inference engine.

[0057] Finally, through the above decision-making and reasoning process, at the end of each Δt cycle, the agent generates a definite congestion control configuration parameter that is applicable to the current local state and conforms to global semantic preferences, such as determining the K parameter to be configured in the next cycle. min and K max The specific value.

[0058] Step S5: Configure the queue management register of the switch port according to the congestion control configuration parameters to complete the adaptive adjustment of the congestion control parameters.

[0059] Specifically, after determining the target action index a through policy network reasoning and sampling... t Then, the hardware configuration process is initiated immediately. First, the action decoding mapping table pre-stored in the switch's read-only memory or TCAM is queried. For example, assuming the selected action index is 5, the corresponding entry in the mapping table is {K}. min 20480, K max The threshold for low ECN tagging is 20480 bytes (20KB), and the threshold for high ECN tagging is 71680 bytes (70KB).

[0060] Furthermore, these byte-level values ​​cannot be directly written to hardware registers; instead, they need to be formatted according to the specific switching chip's programming specifications. In a preferred embodiment, for commercially available switching chips widely deployed in current data centers, their ECN threshold register receives unsigned integer values ​​at the "buffer unit" granularity. For example, assuming the chip's buffer unit size is 254 bytes, then K... min The register value converted to a 16-bit hexadecimal number is 0x0050; K max The register value is converted to a 16-bit hexadecimal number, which is 0x011A.

[0061] In this embodiment, the calculated register value is written to the corresponding hardware register address in the queue management module through the PCIe BAR space mapping mechanism or the Switchdev kernel driver interface. For the PCIe mapping method, a 32-bit word write operation is performed directly on the specific offset address mapped by the switch chip in the host-side memory; for the kernel driver method, the final bus write timing is completed by writing to the sysfs virtual file node.

[0062] According to one aspect of this application, write operations to hardware registers have the characteristic of immediate effect. Once the write operation is complete, the queue management hardware logic of the switching chip will automatically load the new register value in the next clock cycle. Specifically, for each data packet subsequently entering the port queue, the hardware will execute the following judgment logic: Get the instantaneous depth Q of the current queue current (Unit: buffer unit).

[0063] If Q current <K min_reg If not, the data packet will not be marked and will be forwarded normally.

[0064] If K min_reg ≤Q_{current} <K max_reg Then the hardware is based on a Q current - Kmin_reg The probability is calculated proportionally, and the CE flag is set in the ECN field of the IP header of the data packet.

[0065] If Q current ≥K max_reg Then all incoming data packets will be unconditionally set to the CE flag.

[0066] At this point, the congestion control parameters, driven by the global semantic encoding of the centralized controller, guided by semantic mapping and reward function reparameterization, and generated by the specific decisions of the distributed reinforcement learning agent, have been fully deployed into the packet forwarding pipeline of the switch's data plane. Subsequently, the port will perform congestion marking according to the newly configured threshold until the agent makes a new configuration decision in the next Δt cycle. This invention achieves semantic-level collaborative adaptive adjustment of congestion control parameters, following macroscopic changes in network traffic patterns at the second level and responding to microscopic fluctuations in port queues at the microsecond level.

[0067] like Figure 2 As shown, according to another aspect of this application, the steps for generating a congestion semantic embedding vector include: Step S2.1: For each switch port, perform feature splicing and normalization on the histogram data of the flow quantity distribution, average packet length statistics, traffic burst index and the normalized value of the current queue length reported by it to construct the initial feature matrix of the node.

[0068] Specifically, the centralized controller's data preprocessing pipeline first performs field-by-field parsing and quality verification on the raw data reported by each switch port i. For example, for port i, its reported data includes the following four characteristic fields: First, the flow distribution histogram data. This data is reported as a fixed-length integer array. In this embodiment, the array length is set to 10, for example, [125, 48, 21, 9, 3, 1, 0, 0, 0, 0]. Each bucket in this array corresponds to a specific rate range. To ensure comparability between different ports, the histogram data is subjected to sum-normalization processing, that is, each element in the array is divided by the total number of active flows in the current period of that port (exemplarily 125+48+21+9+3+1=207). After processing, a normalized flow distribution vector is obtained, for example, [0.604, 0.232, 0.101, 0.043, 0.014, 0.005, 0, 0, 0, 0], which represents the proportion of flow quantities in different rate ranges.

[0069] Secondly, the average packet length statistic. This value is read from the port statistics register as a 32-bit floating-point number, for example, 1450.5 bytes. Since the average packet length varies significantly across different service types (e.g., storage services are approximately 1500 bytes, while control plane services are approximately 64 bytes), this characteristic has a significant impact on congestion behavior. Furthermore, this value is subjected to Min-Max normalization, with the normalization interval set to the typical packet length range [64, 9000] bytes. For example, 1450.5 bytes, after normalization, is approximately 0.155.

[0070] Third, the traffic burstiness index. This index is calculated by dividing the variance of the transmission rate of micro-burst time slices (e.g., 100-microsecond granularity) within the current period by the mean. It is a 32-bit floating-point number, for example, 1.35. The larger this value, the more intermittent and bursty the traffic. Logarithmic compression is performed on this value to suppress the impact of extreme outliers. For example, 1.35 is processed to approximately 0.85.

[0071] Fourth, the normalized value of the current queue length. This value has been calculated in step S1 and is a floating-point number in the range [0,1], for example, 0.23, which can be used directly without additional processing.

[0072] In a preferred embodiment, to enhance feature representation capabilities, a port device type identifier is also introduced as an auxiliary feature. For example, the switch device model is converted into a One-Hot encoded vector, such as [1,0,0] for model A, [0,1,0] for model B, and [0,0,1] for model C. This feature helps the model learn differentiated congestion behavior patterns under different hardware platforms (such as different buffer capacities and different forwarding architectures).

[0073] Furthermore, all the processed feature components are concatenated along the feature dimensions. For example, the concatenated single-port feature vector includes: a 10-dimensional normalized flow distribution vector, a 1-dimensional normalized average packet length, a 1-dimensional logarithmic compression burst index, a 1-dimensional normalized queue length, and a 3-dimensional device type One-Hot code, totaling D. in =16 dimensions. If a richer feature set is used, D... in It can be scaled to 32 dimensions or higher. The feature vectors of all N active ports are stacked row-wise according to the device ID order in the topology management database, ultimately generating a dimension N-row D. in The initial feature matrix X of the column nodes raw Each row of the matrix corresponds to a complete feature profile of a port, and each column corresponds to a specific traffic or status attribute dimension.

[0074] Step S2.2: Obtain the network physical link connection relationship and the corresponding link bandwidth utilization and link propagation delay, perform weighted fusion and normalization processing to generate a weighted adjacency matrix, wherein the edge weight in the weighted adjacency matrix is ​​used to characterize the congestion cost of the corresponding link.

[0075] Specifically, the centralized controller reads the current physical link configuration from the network topology management database. For example, the data center network adopts a leaf-spine topology architecture, containing M leaf switches and S spine switches. Each leaf switch is connected to each spine switch by a physical link. An undirected graph G=(V, E) is constructed, with all switch ports as graph nodes and physical links as graph edges, where the number of nodes |V|=N.

[0076] Furthermore, for each edge e ij The controller collects two types of dynamic link status data in parallel: First, link bandwidth utilization U ij This value is obtained by dividing the cumulative number of bytes sent by the link within the current period by the product of the link's nominal bandwidth and the period duration, and is a floating-point number in the range [0,1]. For example, for a 100Gbps link, if the cumulative number of bytes sent within 5 seconds of the current period is 3.125 × 10^{10} bytes, then the bandwidth utilization rate is 0.50, or 50%. A higher utilization rate indicates a busier link and a greater probability of congestion.

[0077] Secondly, the link propagation delay D ij This value is the sum of the fiber optic propagation delay and the device forwarding delay of the physical link. For example, it can be obtained by inserting timestamps into data packets through in-band network telemetry, calculating the end-to-end delay at the receiving end, and subtracting the queuing delay; a typical value is between 100 nanoseconds and 500 nanoseconds. Links with longer propagation delays experience a more significant delay effect on congestion feedback signals.

[0078] In a preferred embodiment, to fuse the two heterogeneous indices into a single computable edge weight, the controller performs weighted fusion and normalization processing. For example, the formula for calculating the edge weight W_{ij} is defined as follows: W ij = λ1· U ij + λ2· (D ij / D max ); where λ1 and λ2 are preset hyperparameters. For example, if the operation and maintenance strategy focuses more on the degree of congestion, λ1 can be set to 0.7 and λ2 can be set to 0.3. D min and D maxThese are the minimum and maximum values ​​of the propagation delay across the entire network link, used to normalize the propagation delay to the [0,1] interval.

[0079] Based on a further improvement to this embodiment, the calculated edge weight W ij Numerically, W directly represents the congestion cost of the link: the higher the link utilization, the greater the propagation delay. ij The closer the value is to 1, the less resistance the link experiences in the congestion propagation network, and the easier it is for congestion signals to propagate along this link. Organize the edge weights between all N×N node pairs in matrix form. If there is no direct physical connection between node i and node j, the corresponding element is set to 0. The final generated matrix is ​​the weighted adjacency matrix A, where the non-zero elements Ai are... ij = W ij This matrix will serve as an edge weight factor in subsequent graph attention encoding, directly affecting the calculation of attention coefficients between nodes.

[0080] Step S2.3: Input the initial feature matrix and weighted adjacency matrix of the nodes into the multi-head graph attention network, perform attention coefficient calculation and multi-head feature aggregation processing, and output the congestion semantic embedding vector corresponding to each port.

[0081] Specifically, the generated initial feature matrix X of the nodes raw The weighted adjacency matrix A is used as input and fed into a multi-head graph attention network model pre-deployed on a centralized controller for forward inference computation.

[0082] In this embodiment, the multi-head graph attention network comprises two GAT convolutional layers, each configured with K=4 parallel attention heads. The network was pre-trained offline before deployment, and the training process is as follows: historical traffic data was collected, covering various traffic patterns (such as web search, data backup, and distributed training), various load intensities, and various network topology configurations. Network simulators such as NS3 were used to simulate real congestion behavior under each configuration, and each port was manually labeled with its congestion propagation role in that scenario, such as "bottleneck link ingress," "incast convergence root node," "lightly loaded leaf node," and "balanced load path node." These role labels were used as supervision signals, and the GAT network parameters were optimized end-to-end using a multi-task classification loss function. After training, the network learned the non-linear mapping ability from raw statistical features to congestion semantic roles, and this mapping relationship was internalized into the network weights.

[0083] Furthermore, during the online inference phase, the network performs the following layer-by-layer computations: The first layer of GAT convolution—attention coefficient calculation and processing—is performed. For any center node i in the graph, its initial features are first linearly transformed using a learnable shared weight matrix W to obtain an intermediate representation. Subsequently, for each neighbor node j of i, the original attention score e between them is calculated. ij .

[0084] This invention will use the edge weights A in the weighted adjacency matrix ij The attention coefficient is introduced as a scaling factor in the calculation. For example, the corrected attention score is e. ij ' = e ij A ij This means that the higher the link congestion cost between nodes i and j (i.e., A), the greater the cost. ij The closer the value is to 1, the greater the weight of the influence of the state of node j on the semantic representation of node i. Subsequently, the attention coefficient α is obtained by normalizing the corrected scores of all neighbors using the Softmax function. ij .

[0085] The first layer, GAT convolution, performs multi-head feature aggregation. After obtaining the attention coefficients, each attention head independently performs weighted aggregation of its neighboring features to generate the corresponding output feature: h. i (k). The output vectors of K=4 heads are concatenated along the feature dimension to obtain the final output h of the first layer GAT. i 1 .

[0086] The second GAT convolution layer performs semantic compression. The output h of the first layer is then processed. i 1 The second layer of GAT serves as the input feature for the second layer. The structure of the second layer is similar to the first layer, but it uses different weight matrices and attention vectors. In an optional implementation, the second layer uses average aggregation instead of concatenation aggregation, i.e., taking the arithmetic mean of the outputs of the K heads to control the dimensionality of the output features. Finally, the network outputs a real-valued vector of dimension d (exemplarily set to d=128) for each port i. This vector is the congestion semantic embedding vector h. i .

[0087] In this embodiment, the output congestion semantic embedding vector h iIt possesses the following characteristics: In a continuous high-dimensional vector space, ports with similar congestion propagation roles have embedding vectors that are close to each other in terms of cosine or Euclidean distance, forming natural clusters. For example, all ports located at incast convergence bottlenecks exhibit high activation patterns in their embedding vectors within a specific dimensional subspace; while ports located on lightly loaded paths show a more uniform and dispersed vector distribution. This semantic encoding method successfully embeds the originally discrete and difficult-to-quantify port role concept into a computable and comparable continuous vector representation, laying the mathematical foundation for subsequent semantic mapping and decision guidance.

[0088] like Figure 3 As shown, according to another aspect of this application, the step of mapping a congestion semantic embedding vector to a semantic conditioning vector includes: Step S3.1: Input the congestion semantic embedding vector into the semantic mapping network for forward inference processing. The semantic mapping network contains multiple fully connected layers and non-linear activation functions. Its network parameters are obtained through meta-learning training based on historical traffic data.

[0089] Specifically, the centralized controller invokes the semantic mapping network deployed on it. In this embodiment, the network is designed as follows: the input layer has a dimension of 128 (the dimension of the matching congestion semantic embedding vector), followed by two fully connected hidden layers. The first hidden layer contains 64 neurons and uses the ReLU activation function; the second hidden layer contains 16 neurons and also uses the ReLU activation function. The output layer is a linear fully connected layer containing k neurons (k=4 for example), without using an activation function or using an identity mapping to ensure that the output value range covers the real number range. The total number of parameters in this network structure is approximately 10,000, making it lightweight and convenient for high-frequency parallel inference on the centralized controller.

[0090] Furthermore, the parameters of the semantic mapping network are not trained through conventional supervised learning or reinforcement learning methods, but rather through a specially designed meta-learning training method based on historical traffic data. For example, the specific process of meta-learning training is as follows: Training data construction phase. Historical operational datasets covering various typical data center traffic scenarios (such as steady-state background traffic, large-scale incast bursts, and fully reduced communication modes) are collected. For each traffic scenario, hyperparameter tuning methods such as grid search or Bayesian optimization are used to find the optimal reward function weight configuration that enables the distributed reinforcement learning agent to achieve optimal network performance (e.g., minimizing average flow completion time) in that scenario. Simultaneously, the overall network state corresponding to that traffic scenario is input into a pre-trained graph attention network to generate congestion semantic embedding vector samples for each port in that scenario. This results in tens of thousands of sample pairs, serving as the training set for meta-learning.

[0091] Meta-learning training phase. An optimization-based meta-learning algorithm, such as model-independent meta-learning or the Reptile algorithm, is used to train the semantic mapping network. The inner loop of training simulates an online deployment scenario: for a sampled batch traffic scenario, the semantic mapping network in its current state first generates a semantic adjustment vector v for each port. i Subsequently, a distributed intelligent agent was simulated in v i The process involves T-step policy optimization under guidance, calculating the divergence loss between the agent's final converged policy and the optimal policy. The outer loop accumulates the losses from multiple scenarios and updates the network parameters using gradient descent. After tens of thousands of iterations, the network parameters converge to a set of optimal parameters.

[0092] In this embodiment, the semantic mapping network, after convergence through meta-learning training, possesses cross-scenario generalization capability: for any unseen traffic scenario, the generated semantic adjustment vector can guide the distributed agent to converge to the near-optimal strategy of that scenario with the fewest online exploration steps.

[0093] During the online inference phase, for any port i, the generated congestion semantic embedding vector h is... i The trained semantic mapping network is directly input. The network performs forward inference layer by layer, ultimately obtaining the original semantic adjustment vector v of dimension k. i raw .

[0094] Step S3.2: The semantic mapping network performs feature compression and nonlinear transformation on the congestion semantic embedding vector, and outputs a low-dimensional original semantic adjustment vector.

[0095] Specifically, the forward inference process described above involves performing feature compression and nonlinear transformation on the high-dimensional semantic embedding vector. The congestion semantic embedding vector h... i The dimension d=128 of the input is much higher than the dimension k=4 of the output target. This bottleneck-type network structure design (128→64→16→4) forces the network to extract a few independent directions of change that play the most decisive role in regulating distributed decision preferences from the high-dimensional embedding during the layer-by-layer transmission process; these are the semantic principal components. The introduction of a nonlinear activation function enables this compression transformation to capture the complex nonlinear relationship between input features and output preferences: for example, when the strength of the "incast feature component" in the semantic embedding exceeds a certain threshold, the output preference may undergo a phase transition-like abrupt switch.

[0096] In this embodiment, the semantic mapping network ultimately outputs a low-dimensional original semantic adjustment vector v. i rawFor example, it may contain k=4 floating-point values, such as [0.82, -0.15, 0.61, 0.93] for a certain port. The key point is that, during the training phase of the semantic mapping network, this invention assigns clear and interpretable physical meaning to these k dimensions by constraining the explicit binding relationship between the output dimensions and specific reward function terms. First dimension: Throughput preference coefficient. The value of this dimension (0.82 in the example) represents the degree of importance that port i is expected to attach to the link utilization metric in local congestion control decisions from a global network perspective. The higher the value, the more the centralized controller determines that the current network is in a "throughput-sensitive" phase (e.g., with a large number of long-running transmissions), and expects the port to prioritize maintaining a high transmission rate, tolerating a moderate increase in queue depth as a result.

[0097] The second dimension: latency sensitivity coefficient. The value of this dimension (-0.15 in the example) characterizes the sensitivity to queuing latency metrics. A negative value close to 0 indicates that the current global network has a high tolerance for latency; if the value becomes positive and large (e.g., 0.88), it means that there is a latency-sensitive short-flow or incast risk in the network, and ports are expected to respond more strongly to queue backlog.

[0098] The third dimension: ECN (Enhanced Congestion Notification) aggression coefficient. The value of this dimension (0.61 in the example) directly influences the agent's tendency to choose the ECN threshold. A larger value indicates that the centralized controller expects the port to adopt a more aggressive congestion notification strategy: that is, it tends to choose a lower K threshold. min and K max A threshold is set so that congestion feedback can be sent to the source in advance when the network queue is still in a shallow backlog, thus achieving preventive control of congestion.

[0099] Fourth dimension: Policy stability coefficient. The value of this dimension (0.93 in the example) is used to adjust the smoothness of the agent's decision-making. The larger the value, the more the centralized controller expects the port to maintain relatively stable parameter configurations over consecutive decision cycles, avoiding frequent threshold jumps caused by transient noise, thereby reducing unnecessary disturbances to the end-to-end transport layer congestion window.

[0100] Further improvements to this embodiment demonstrate the significant engineering value of this explicit semantic binding design. Network operators can intuitively understand the centralized controller's current assessment of network congestion and the intended policy guidance by monitoring the time-series changes in the values ​​of each dimension of the semantic adjustment vector for each port. For example, if the first dimension value of a port continuously decreases while the second dimension value rapidly increases, operators can immediately infer that the path of that port is experiencing a temporary latency-sensitive service burst, and the controller is guiding it to switch from a "throughput-first" mode to a "low-latency-first" mode. This high interpretability is something that traditional black-box weighted fusion solutions cannot provide.

[0101] According to another aspect of this application, before sending the semantic conditioning vector to the switch port, it also includes: Step S4.1: Obtain the set of end-to-end forwarding paths in the data center network and determine the sequence of switch ports traversed by each path.

[0102] Specifically, the centralized controller runs a topology path resolution algorithm, generating a set of end-to-end forwarding paths for the entire network based on link connectivity obtained from the network topology management database and the currently effective routing protocol configuration. In this embodiment, data center networks widely employ equal-cost multipath routing protocols or source-based routing strategies, where traffic between the same source and destination pairs may be load-balanced via multiple different physical paths.

[0103] For example, the path resolution process is as follows: The controller first constructs a directed graph G of the entire network based on the link state database. dir = (V, E dir ), where nodes are switch ports and directed edges represent the forwarding direction of the data plane. For any pair of leaf switches (or more fine-grained source and destination port pairs) in the network, the controller runs L shortest path algorithms (such as Yen's algorithm) or uses explicit path records in the segmented routing policy database to extract all active forwarding paths.

[0104] Furthermore, for each extracted path p, the controller records the sequence of switch ports it passes through along the way. For example, in a typical leaf-spine topology, a path starting from port 1 of leaf switch A, forwarded by spine switch B, and reaching port 2 of leaf switch C can be represented by the port sequence {port}. A1 port B_up}, port B_down port C2 If there are 4 concurrent paths in the network, the above source-destination pairs will correspond to 4 different port sequences, each of which constitutes an independent semantic alignment constraint unit.

[0105] In a preferred embodiment, to control computational complexity, path resolution is not performed exhaustively on all possible port pairs, but rather employs a traffic-based weighted sampling strategy. Based on the collected traffic matrix data, the controller identifies hotspot paths carrying the main service traffic and performs precise resolution and constraint calculations only on paths whose traffic share exceeds a preset threshold (e.g., 1% of total traffic), thereby achieving a balance between coverage and computational overhead. Finally, the controller generates a path-port mapping table that records each valid path p and its associated set of ports P. p The correspondence between them.

[0106] Step S4.2: For each path, calculate the arithmetic mean of the original semantic adjustment vectors corresponding to all ports on the path to obtain the path semantic adjustment center vector.

[0107] According to one aspect of this application, this embodiment statistically aggregates the independent semantic preferences of each port on each path based on the obtained path-port mapping relationship, and generates a baseline vector that can represent the overall congestion response strategy tendency of the path.

[0108] Specifically, for any path p, assume its port sequence contains m ports. The centralized controller has generated a corresponding original semantic adjustment vector v for each port i. i raw For example, if k=4, the original semantic adjustment vectors of the three ports on path p are as follows: port i1: [0.82, -0.15, 0.61, 0.93]; port i2: [0.78, -0.10, 0.58, 0.88]; port i3: [0.85, -0.20, 0.65, 0.95].

[0109] Furthermore, the controller performs a dimension-wise arithmetic mean calculation on these vectors to generate the path semantic adjustment center vector v. p For example, the calculation process is as follows: First dimension (throughput preference): (0.82 + 0.78 + 0.85) / 3 = 0.817; Second dimension (time delay sensitivity): (-0.15 - 0.10 - 0.20) / 3 = -0.15; Third dimension (ECN radicality): (0.61 + 0.58 + 0.65) / 3 = 0.613; Fourth dimension (strategy stability): (0.93 + 0.88 + 0.95) / 3 = 0.92; The calculated center vector is v p= [0.817, -0.15, 0.613, 0.92].

[0110] In this embodiment, the physical meaning of the center vector is: it represents the average semantic preference of all ports on that path under the current global traffic view. In an ideal cooperative scenario, all ports on the same path should share consistent or similar congestion response strategies, such as prioritizing throughput or prioritizing latency, to ensure semantic consistency in end-to-end congestion control behavior.

[0111] In one alternative implementation, if port i belongs to multiple different paths simultaneously (e.g., a spine switch port serves traffic between multiple source-destination pairs), then the port will participate in the calculation of the center vector of multiple paths, that is, for each path p to which it belongs, there exists a corresponding v. p .

[0112] Step S4.3: For each port belonging to the path, calculate the deviation between its original semantic adjustment vector and the path semantic adjustment center vector. If the deviation exceeds a preset constraint threshold, perform a numerical correction process to move closer to the path semantic adjustment center vector to obtain the corrected semantic adjustment vector; use the corrected semantic adjustment vector as the semantic adjustment vector to be issued.

[0113] According to one aspect of this application, this embodiment implements a path semantic alignment constraint mechanism, which eliminates end-to-end congestion response distortion caused by policy conflicts between ports by forcing the semantic adjustment vectors of each port on the same path to remain within a reasonable deviation range.

[0114] Specifically, for each port i on path p, the controller computes its original semantic adjustment vector v. i raw The calculated path semantic adjustment center vector v p The deviation between them. For example, using Euclidean distance as the deviation metric, the deviation between port i1 and the center vector is approximately 0.012, calculated using the example values ​​above.

[0115] Further, the controller compares this deviation value with a preset constraint threshold epsilon. In this embodiment, the value of epsilon is set according to the network operation and maintenance strategy, and is exemplarily set to epsilon = 0.15. If v i raw With v p If the Euclidean distance between them is ≤ epsilon, it indicates that the semantic preference of port i is sufficiently consistent with the overall path preference, and no correction is needed; v can be directly set. i raw As the final semantic adjustment vector.

[0116] Conversely, if the deviation exceeds the constraint threshold, it indicates a significant discrepancy between the local semantic preferences of port i and those of other ports on the path. This situation may occur when port i, due to its unique local traffic composition (e.g., it also carries a large amount of background traffic from outside the path), is assigned a semantic role by the graph attention network that differs significantly from other ports on the same path. Without constraints, this semantic discrepancy will lead to conflicting congestion responses from different ports on the path to the same incast event; for example, upstream ports may aggressively label the ECN while downstream ports may conservatively label it, distorting the congestion feedback signal received by the source.

[0117] In a preferred embodiment, when an excessive deviation is detected, the controller performs a numerical correction process to approximate the path semantic adjustment center vector. The correction formula exemplarily employs linear interpolation: vi = v i raw +η·(v p -v i raw ); where η is between (0,1) and is the correction intensity coefficient, exemplarily set to η = 0.5. This correction means moving the semantic adjustment vector of port i half a step in the direction pointing to the center vector, so that its preference tends to move closer to the overall consensus of the path, but without completely losing its local personalized characteristics.

[0118] According to a further improvement of this embodiment, if port i simultaneously belongs to multiple paths {p1, p2, ...s, p...} L If the required correction is not calculated, its deviation from the center vector of each path needs to be considered. For example, the weighted average of the required correction for each path is taken as the final correction, with the weights proportional to the traffic volume on each path. The vector obtained after the above correction process is the corrected semantic adjustment vector v. i The controller will v i Replace the original v i raw This serves as the semantic adjustment vector formally distributed to the switch port in subsequent steps. The introduction of this constraint mechanism ensures that the distributed intelligent agent, while maintaining the flexibility of local autonomous decision-making, is subject to the necessary constraints of global topological coordination consistency in the spatial dimension.

[0119] like Figure 4 As shown, according to another aspect of this application, the reward calculation function of the reinforcement learning agent is reparameterized, including: Step S5.1: In the data plane or control plane of the switch port, read the semantic adjustment vector that has been issued and cached, wherein the semantic adjustment vector is represented as a multi-dimensional floating-point array.

[0120] Specifically, the centralized controller has used an out-of-band management network to transfer the semantic adjustment vector v corresponding to port i. i The message is sent to the target switch. In this embodiment, after receiving the gRPC configuration message, the daemon process on the switch control plane performs parsing and verification operations, confirms the data integrity, and then sends the v... i The four floating-point values ​​are written to the shared memory area of ​​the operating system.

[0121] For example, v i The memory is stored in contiguous 16-byte blocks (each single-precision floating-point number occupies 4 bytes), with hexadecimal representations of 0x3F51EB85 0xBE19999A 0x3F1C28F6 0x3F6E147B, which decode to 0.82, -0.15, 0.61, and 0.93 respectively. This memory block is mapped to a physical address space allocated from mass pages and readable by the data plane.

[0122] Furthermore, the reinforcement learning agent deployed on this port first performs a semantic adjustment vector read operation at the start of each decision cycle. In an alternative implementation, if the agent's core inference logic runs on the CPU core of the switch control plane, it appends the memory block to its own virtual address space through standard inter-process communication mechanisms, such as system calls to shared memory keys, and then reads four floating-point values ​​through simple pointer dereferencing or memcpy operations.

[0123] In another alternative implementation, if the agent's policy network is hardware-deployed in a programmable matching-action pipeline or on-chip NPU in the data plane to meet microsecond-level latency requirements, the method of reading the semantic adjustment vector differs. For example, the NPU reads the 16-byte data block from the aforementioned shared memory physical address at a fixed frequency via a direct memory access channel and loads it into a scalar register or vector register within the NPU as a constant parameter input to the reward calculation module.

[0124] In this embodiment, regardless of the read path used, the reinforcement learning agent retrieves the currently effective semantic adjustment vector v in multidimensional floating-point array format from this cache location during each decision or reward calculation. i = [v i,1 ,v i,2 , v i,3 , v i,4 It should be noted that the update frequency of this vector (on the order of seconds) is much lower than the decision frequency of the agent (on the order of microseconds). Therefore, in most decision cycles, the read v... i The values ​​remain constant, and atomic changes only occur when a new configuration is issued by the centralized controller.

[0125] Step S5.2: Replace the weight coefficients in the original static reward function of the reinforcement learning agent with activation function values ​​that take the corresponding dimension values ​​of the semantic adjustment vector as independent variables.

[0126] According to one aspect of this application, this embodiment reshapes the agent's behavioral tendencies from the bottom layer of the decision logic by structurally reconstructing the reward function, based on the global preferences issued from the upper layer.

[0127] Specifically, in conventional reinforcement learning congestion control applications, the agent's reward function typically has a fixed static form. For example, a typical static reward function is defined as: r t =α·T t +(1-α)·L t ; where T t For link utilization, L t Let be the latency factor, α be between (0,1), and be a preset fixed weight coefficient, for example, α = 0.6. This static design means that the agent's trade-off preference for throughput versus latency remains constant across all times and network states.

[0128] In this embodiment, the present invention performs dynamic reparameterization on the static reward function through a semantic injection module. First, the semantic injection module reads the semantic adjustment vector v. i The current value. Then, the static weight coefficients in the original reward function are replaced with values ​​expressed in v. i The corresponding dimension values ​​are the activation function values ​​of the independent variables.

[0129] In a preferred embodiment, the reparameterized reward function takes the form: r t =σ(v i ,1)·T t +σ(v i ,2)·L t ; where σ() is the Sigmoid activation function, which smoothly maps the real-domain values ​​of the semantic adjustment vector to the (0,1) interval, and converts them into coefficients that can be directly used as reward weights.

[0130] For example, for v i = [0.82, -0.15, 0.61, 0.93], the weight coefficients after Sigmoid mapping are: throughput weight w T = 0.694; Delay weight w L = 0.463.

[0131] In this configuration, when the agent evaluates the value of an action, the throughput term contributes a relatively higher percentage to the reward, while the penalty for latency is relatively mild. Conversely, if the centralized controller detects an incast burst precursor, it issues an updated semantic adjustment vector v. i If the weights are [0.35, 0.88, 0.92, 0.77], then the mapped weights become w. T =0.587, w L =0.707. At this point, the structure of the reward function has undergone a semantic shift, with the delay term becoming dominant. Even if the link utilization observed by the agent remains completely unchanged, due to the change in the inherent bias of the reward calculation logic, its value assessment results will instinctively favor conservative actions that can quickly reduce queue length.

[0132] Furthermore, using the sigmoid function as the mapping function has key technical advantages. First, its continuous differentiability ensures the existence and smoothness of the gradient of the reward function relative to the semantic adjustment vector, which is crucial for training semantic mapping networks using gradient-based meta-learning. Second, saturated nonlinearity makes the weight changes more gradual when the absolute value of the semantic adjustment vector is large, avoiding policy collapse caused by extreme preference settings. Third, the bounded output (always within the (0,1) interval) guarantees the stability of the reward value, preventing reward value overflow due to drastic adjustments to the semantic vector.

[0133] In this embodiment, through dynamic replacement and function binding at the code logic level, the module inside the reinforcement learning agent used to calculate immediate rewards is transformed into a semantically injected dynamic reward generator. In the reward calculation of each microsecond-level decision cycle, this generator continuously generates reward signals based on the current observation and the selected action. However, its inherent preference structure has been profoundly and smoothly reshaped by the global semantic preferences distributed from the upper layer, thus achieving a paradigm shift from numerical fusion to semantic guidance without changing the agent's network structure.

[0134] like Figure 5 As shown, according to another aspect of this application, the steps of collecting transient congestion observation data of a switch port and inputting it into a reinforcement learning agent include: Step S6.1: Collect raw register values ​​from the switch hardware queue management module at preset microsecond intervals. The raw register values ​​include at least: instantaneous queue depth value, outgoing port transmission rate count value, and ECN tag packet count value.

[0135] Specifically, to capture the micro-burst traffic characteristics of the data center and achieve rapid congestion response, a high-precision hardware timer is set to periodically trigger state snapshot collection at extremely short time intervals Δt. In this embodiment, Δt is exemplarily set to 50 microseconds. This choice of time granularity is a trade-off design, short enough to capture micro-bursts (typically on the order of 100 microseconds) and long enough to allow the agent to complete a full inference and configuration operation.

[0136] Furthermore, each time the timer is triggered, the state awareness module reads the current values ​​of the following three types of key registers from the queue management unit of the switching chip through the hardware abstraction layer interface: first, the instantaneous queue depth value qlen. raw The data is read from the TM_QUEUE_DEPTH or TM_WRED_QUEUE_DEPTH register. For example, in a snapshot, a 24-bit unsigned integer value of 20480 is read, indicating that the number of bytes backed up in the current buffer is 20KB. For a typical 100G port, its total buffer capacity is usually around 12MB, so a queue depth of 20KB accounts for approximately 0.16% of the total buffer, which is considered a slight backlog. Secondly, the outgoing port transmit rate count value txByteCnt... raw The data is read from the EGR_TX_BYTE_CNT or a similar port statistics counter. For example, the 64-bit cumulative transmitted bytes read in the current snapshot are 0x000012A05F200000. Since this is a monotonically increasing cumulative counter, the actual transmitted amount in this period needs to be obtained by differing from the previous snapshot value. If the previous snapshot value was 0x000012A05F1E0000, the difference is approximately 131072 bytes, then the average transmission rate over the past 50 microseconds is 20.97 Gbps. Thirdly, the ECN tag packet count value ecnMarkCnt. raw Read from the ECN_MARKED_CNT or WRED_DROP_ECN_CNT register. For example, the 32-bit cumulative ECN tag count read in the current snapshot is 0x0000A120 (decimal 41248), and the previous snapshot value was 0x0000A0E0 (decimal 41184). The difference between the two is 64, indicating that 64 packets have had the CE congestion flag set in the past 50 microseconds.

[0137] In an alternative implementation, if the switch supports in-band network telemetry, the aforementioned register values ​​can also be indirectly obtained by parsing the INT report messages generated by the data plane, thereby avoiding the overhead caused by the control plane frequently accessing hardware registers. Regardless of the acquisition method used, in this embodiment, the three sets of raw register values ​​acquired in each Δt cycle constitute the most basic and real-time data source for the agent to perceive the local congestion state.

[0138] Step S6.2: Perform an exponentially weighted moving average on the collected raw register values ​​to filter out micro-burst noise and obtain smoothed observation data.

[0139] According to one aspect of this application, the acquired raw hardware register values ​​are subjected to time-series filtering to suppress observation noise caused by micro-burst traffic and improve the numerical stability of the state space in subsequent reinforcement learning.

[0140] Specifically, data center traffic exhibits typical ON / OFF burst characteristics. Within a short time window of 50 microseconds, queue depth and transmission rate can oscillate wildly between zero and line speed. If the raw values ​​are directly used as input to a neural network, this high-frequency noise will cause the action probability distribution output by the policy network to fluctuate significantly within adjacent periods, leading to frequent oscillations in parameter configuration.

[0141] In a preferred embodiment, the state-aware module performs an exponentially weighted moving average on each type of raw observation. Taking the instantaneous queue depth value as an example, the filtering formula is: qlen ema (t)=γ·qlen raw (t)+(1-γ)·qlen ema (t-1); where γ is the smoothing factor. For example, γ is set to 0.2, which means that the original observation at the current time contributes 20% to the filtered result, and the historical smoothed value contributes 80%.

[0142] Furthermore, the filtering effect is illustrated with specific numerical examples. Assume the raw queue depth readings for three consecutive cycles are 20480, 51200, and 10240 bytes respectively: t=1 Initial time: qlen ema (1) = 20480; At time t=2: qlen ema (2) = 0.2×51200 + 0.8×20480 = 10240 + 16384 = 26624; At time t=3: qlen ema (3)= 0.2×10240 + 0.8×26624 = 2048 + 21299.2 = 23347.2; As can be seen, although the original observations fluctuated wildly between 20480, 51200, and 10240, the numerical sequence after EWMA filtering was 20480, 26624, and 23347, with significantly smoother fluctuations. The difference results between the outgoing port transmission rate count and the ECN tag packet count were also processed by the aforementioned EWMA.

[0143] In this embodiment, the choice of the EWMA smoothing factor is a configurable hyperparameter. A larger γ (e.g., 0.5) makes the filtering results respond to real changes more quickly, but weakens the ability to suppress noise; a smaller γ (e.g., 0.1) makes the sequence smoother, but may lag behind the rapid evolution of the actual congestion state. Operations personnel can perform targeted optimization based on network traffic characteristics. The data processed by EWMA is the smoothed observation data, which retains the macroscopic trend of congestion state evolution while eliminating micro-burst high-frequency components that may interfere with decision-making.

[0144] Step S6.3: Normalize the smoothed observation data and concatenate it with the congestion control parameter configuration value of the previous decision cycle to construct the reinforcement learning state vector of the current time step; input the reinforcement learning state vector into the policy network and value network of the reinforcement learning agent.

[0145] According to one aspect of this application, this embodiment completes the final transformation from smoothed observation data to standardized neural network input, and introduces historical decision information to enhance the temporal representation capability of the state.

[0146] Specifically, the generated smoothed observation data is first normalized to make its numerical range match the typical activation range of the neural network input layer.

[0147] First, normalize the instantaneous queue depth after smoothing. This involves setting qlen... ema Divide by the total physical buffer capacity (Buf) of that port total For example, for a 100G port, the Buf... total A typical value is 12MB (12,582,912 bytes). If qlen... ema =23347, then the normalized queue length qlen norm = 23347 / 12582912 = 0.00186. This value represents the current buffer occupancy rate.

[0148] Secondly, the smoothed transmission rate is normalized. The number of bytes transmitted periodically after EWMA processing is converted into a bit rate, and then divided by the nominal bandwidth of the port. For example, if the smoothed periodic transmission volume is 150,000 bytes, the transmission rate is 24Gbps. For a 100G port, the normalized link utilization rate txRate is... norm = 24 / 100 = 0.24.

[0149] Third, the smoothed ECN labeling rate is normalized. The number of periodically labeled ECN packets after EWMA processing is divided by the total number of packets sent in the period (set to 0 if the total number of packets is 0). For example, if the smoothed number of ECN labeled packets is 80 and the smoothed total number of packets sent is 2000, then the normalized ECN labeling rate is ecnRate. norm = 80 / 2000 = 0.04.

[0150] Furthermore, the three normalized indices mentioned above are concatenated to form the basic observation vector o at the current moment. t =[qlen norm txRate norm ecnRate norm For example, o t = [0.00186, 0.24, 0.04].

[0151] According to a further improvement of this embodiment, in order to enable the reinforcement learning agent to perceive the temporal evolution trend of the congestion state and know its current behavior configuration, the state vector is extended and spliced ​​on the basis of the basic observation.

[0152] First, the historical observation sequence is concatenated. The basic observation vectors from the past K consecutive time steps are concatenated in chronological order. For example, if K=4, the historical observation part is [o t-3 , o t-2 , o t-1 , o t ], when flattened, is a 3×4 = 12-dimensional vector.

[0153] Secondly, the congestion control parameter configuration values ​​from the previous decision-making cycle are concatenated. The action 'a' selected by the agent in the previous cycle is introduced. t-1 The corresponding parameter configuration value is used as part of the current state. For example, the K value that was effective in the previous cycle is used... min and K max The normalized values ​​(divided by the total buffer capacity) are appended to the end of the state vector, forming two additional dimensions.

[0154] Finally, the reinforcement learning state vector s at the current time step t tThe dimension is 12 + 2 = 14. In this embodiment, s t The input is simultaneously fed into the policy network and the value network (or the two Q-networks in the SAC algorithm). The policy network calculates the output action probability distribution in a feedforward manner, while the value network evaluates the value of the current state or state-action pairs. Through the state construction process designed above, the agent gains a complete perception of the local congestion state, including "current instantaneous value + recent evolution trend + its own historical behavior".

[0155] like Figure 6 As shown, according to another aspect of this application, the steps for decision-making and policy update processing in conjunction with a dynamic reward generator include: Step S7.1: The policy network outputs the selection probability distribution of each available action in the discretized congestion control parameter action space based on the input reinforcement learning state vector.

[0156] Specifically, after constructing the reinforcement learning state vector s for the current time step... t The vector is then fed into a policy network deployed locally on the switch for forward inference computation. In this embodiment, the policy network is structured as a lightweight multilayer perceptron, exemplarily comprising: an input layer of 14 neurons, two fully connected hidden layers (using the ReLU activation function) each containing 64 neurons, and an output layer. The number of neurons in the output layer is equal to the size |A| of the discretized congestion control parameter action space, and the activation function is Softmax to ensure that the sum of all elements in the output vector is 1 and all are positive values.

[0157] Furthermore, the action space design must be compatible with subsequent semantic adjustment vector fusion and hardware register configuration. For example, an ECN low-label threshold K is defined. min The set of possible values ​​is {10KB, 20KB, 30KB, 40KB}, and the high-labeling threshold K max The possible value set is {50KB, 60KB, 70KB, 80KB}. Flattening the Cartesian product of the two sets into a one-dimensional action list results in a total of |A| = 4 × 4 = 16 discrete actions. Each action index uniquely corresponds to a specific (K) set. min ,K max Configuration, for example, action index 0 corresponds to (10KB, 50KB), action index 5 corresponds to (20KB, 70KB), and action index 15 corresponds to (40KB, 80KB).

[0158] In this embodiment, the policy network processes the input s t Layer-by-layer matrix multiplication and non-linear activation are performed, ultimately outputting a 16-dimensional vector by the Softmax layer. For example, the output p of a certain inference iteration...t The values ​​are [0.02, 0.05, 0.03, 0.08, 0.12, 0.15, 0.10, 0.06, 0.09, 0.11, 0.07, 0.04, 0.03, 0.02, 0.02, 0.01]. Where p... t The 5th element (corresponding to action index 5, i.e., K) min =20KB, K max The value of 0.15 for (=70KB) indicates that in the current state s t Given the current policy parameters, the agent believes that the probability of choosing this configuration combination as the optimal decision is 15%.

[0159] This probability distribution vector represents the probability distribution of each selectable action in the discretized congestion control parameter action space. It comprehensively reflects the knowledge learned by the agent from historical experience regarding which action is more advantageous in which state.

[0160] Step S7.2: Random sampling is performed based on the selection probability distribution to determine the target action index, and the target action index is decoded into specific congestion control parameter configuration values.

[0161] According to one aspect of this application, this embodiment performs a transformation from a probability distribution to a specific discrete decision, and the random sampling strategy employed is key to balancing exploration and exploitation, as well as achieving semantically smooth guidance.

[0162] Specifically, the decision-making module does not simply choose the action with the highest probability (greedy strategy), but rather selects the action based on the output selection probability distribution p. t Perform random sampling. For example, use `np.random.choice(16, p=p)`. t Or equivalent hardware random number generation logic, with p t Each element value is used as a sampling weight to extract an action index 'a'. t In this example, action index 5 has a 15% probability of being selected, making it the option with the highest current probability, but there is still an 85% cumulative probability of selecting other actions.

[0163] In a preferred embodiment, this random sampling mechanism is deeply compatible with the soft actor-critic algorithm framework used in this invention. One of the core objectives of the SAC algorithm is to maximize cumulative reward while maximizing policy entropy, that is, to encourage the policy to maintain a certain degree of uncertainty. This synergizes with the concept of semantic guidance: the centralized controller shapes the terrain of the reward function through semantic adjustment vectors, guiding the direction of preference, but not forcing a unique path; the agent conducts random exploration with entropy regularization on the preference terrain, which can both approach the global optimum and avoid falling into local suboptimal policies too early, while maintaining adaptability to minor changes in traffic patterns.

[0164] Furthermore, the target action index a is obtained through sampling. t Then, the decision module calls the pre-stored action decoding mapping table to convert the index into a specific physical configuration value. For example, if a... t = 5, the mapping table returns entries {K min :20480, K max The threshold values ​​are 71680, meaning the low ECN labeling threshold is 20480 bytes (20KB) and the high ECN labeling threshold is 71680 bytes (70KB). These two values ​​are the specific congestion control parameter configuration values ​​that will take effect in the next cycle.

[0165] Step S7.3: Wait for the next time interval to arrive, observe the new reinforcement learning state vector, and call the dynamic reward generator to calculate the immediate reward value. The immediate reward value carries global semantic preferences because the reward calculation function has been reparameterized.

[0166] Specifically, in determining the target action a t After writing the corresponding configuration into the hardware register (see claim 9 for details of the configuration process), the agent waits for the next preset microsecond time interval Δt to arrive. When the new hardware timer is triggered, the state awareness module follows the complete process of steps S6.1 to S6.3 to collect, filter, normalize, and concatenate to construct a new reinforcement learning state vector s for the next time step. t+1 .

[0167] Furthermore, the decision-making module invokes the generated semantically injected dynamic reward generator to calculate the reward from state s. t Perform action a t Transition to state s t+1 The instant reward value r obtained in this process t The reward generator starts from the current state s. t+1 Extract link utilization txRate norm and normalized queue length qlen norm Substitute into the reparameterized reward formula: rt =σ(v i ,1)·txRate norm +σ(v i ,2)·11+qlen norm For example, suppose the currently effective semantic moderating vector is a configuration v that emphasizes low latency. i = [0.35, 0.88], mapping to obtain w T = 0.587, w L = 0.707. If txRate is observed. norm = 0.85, qlen norm = 0.03, then: throughput contribution: 0.587×0.85 =0.499; latency contribution: 0.707×1 / (1+0.03)=0.686; total reward (ignoring entropy term): approximately 1.185.

[0168] Conversely, if the semantic adjustment vector is configured with throughput preference, v_i = [0.82, -0.15, \ldots], the mapping yields w_T = 0.694 and w_L = 0.463. For identical observations, the reward calculation results are: throughput contribution: 0.694 × 0.85 = 0.590; latency contribution: 0.463 × 0.971 = 0.450; total reward: approximately 1.040.

[0169] In this embodiment, the numerical comparison clearly reveals the effect of semantic injection: under identical network state transitions, the instantaneous reward values ​​received by the agent differ significantly due to the different weight structures within the reward function. Under low latency preference, maintaining a low queue depth (qlen) norm =0.03) yielded a higher reward (1.185 vs 1.040); while under throughput preference, the same queue depth was considered too conservative, resulting in a relatively lower reward. It is this reward signal, reparameterized by global semantic preference, that continuously guides the agent's policy network in subsequent incremental updates towards a direction consistent with the upper-level semantic preference. Figure 1 The direction of evolution is slow.

[0170] Step S7.4: Store the experience tuples into the priority experience replay pool. The experience tuples include at least: the current state vector, the target action index, the immediate reward value, and the new state vector. Based on the batch data sampled from the priority experience replay pool, calculate the temporal difference error and the policy gradient, and perform incremental update processing on the weight parameters of the policy network and the value network.

[0171] According to one aspect of this application, this embodiment describes how a reinforcement learning agent uses empirical data generated from online interactions to continuously optimize its internal neural network parameters in order to achieve a gradual improvement in policy performance.

[0172] Specifically, after calculating the immediate reward r t Then, the decision-making module packages the complete decision transfer information into an experience tuple e. t = (s t , a t , r t , s t+1 For example, the tuple contains: a 14-dimensional current state vector, an integer action index, a scalar reward value, and a 14-dimensional next state vector.

[0173] Furthermore, the experience tuple is not immediately discarded after training, but is stored in a priority experience replay pool deployed in the switch's local control plane memory. In this embodiment, the capacity of the replay pool is exemplarily set to 10,000 experiences, managed in a circular queue. Unlike ordinary experience replay, the priority experience replay mechanism maintains an additional priority weight for each experience. This weight is proportional to the absolute value of the temporal difference error of the experience; the larger the error, the higher the degree of surprise of the experience to the current model, and the greater the probability of it being selected during sampling. This mechanism improves training efficiency, allowing the agent to focus more quickly on the state transitions most valuable for policy improvement.

[0174] In a preferred embodiment, the incremental update of model parameters is not performed in every decision cycle, but rather using a batch update strategy. For example, the update frequency is set to every N executions. update A training iteration is triggered after 100 action decisions. When triggered, the training thread randomly samples a batch of data from the priority experience replay pool according to priority weights. The batch size is set to B=64 for example.

[0175] In this embodiment, the following calculation process is performed based on the sampling batch: First, calculate the temporal difference error: For each experience in the batch, calculate the target value y using the target Q-Network; calculate the current Q-Network's value estimate for that experience; the squared difference between the two is the TD error loss. Calculate the gradient of this loss relative to the Q-Network parameters through backpropagation, and perform a one-step update of the Adam optimizer. Second, calculate the policy gradient: Use the updated Q-Network to evaluate the value of the action distribution output by the current policy network in state s, and calculate the SAC-specific policy gradient loss including an entropy term. Calculate the gradient of this loss relative to the policy network parameters through backpropagation, and also perform a one-step update. Third, soft update the target network: Perform a soft update on the parameters of the target Q-Network.

[0176] After the above batch-based incremental weight update process, the parameters of the policy network and the value network are obtained as optimized neural network weights. For example, after thousands of such updates, the policy network gradually learns that when the semantic adjustment vector emphasizes low latency, it should output a lower K-value when facing states with increasing queue depth. min / low K max The agent continuously learns online, adapting to the slow shifts in traffic characteristics and maintaining near-optimal congestion control. This ability allows the agent to adjust to the probability distribution of actions; conversely, it maintains a high threshold to ensure throughput.

[0177] like Figure 7 As shown, according to another aspect of this application, the steps of configuring the queue management register of a switch port based on congestion control configuration parameters include: Step S8.1: Convert the congestion control parameter configuration value obtained by decoding into a hardware-recognizable register write value according to the format specified in the switching chip hardware manual.

[0178] Specifically, after obtaining the specific congestion control parameter configuration value through the action decoding mapping table, for example, K. min =20480 bytes, K max =71680 bytes. These byte-based values ​​cannot be directly written to hardware registers, but must be formatted according to the hardware programming specifications of the target switching chip.

[0179] In this embodiment, taking a commercially available switching chip series widely deployed in current data centers as an example, the register in its queue management module used to configure the WRED / ECN threshold accepts unsigned integer values ​​at the buffer unit level. For example, referring to the chip's hardware manual reveals that the buffer unit size is CellSize = 254 bytes; K min The corresponding register is TM_WRED_MIN_THRESHOLD, with a bit width of 16 bits, which accepts threshold values ​​in cells; K max The corresponding register is TM_WRED_MAX_THRESHOLD, which also has a bit width of 16 bits.

[0180] Furthermore, the configuration execution module performs the following calculation transformation: K min Register value = 20480 ÷ 254 = 80.63 ≈ 80; K maxThe register value = 71680 ÷ 254 = 282.20 ≈ 282. The rounding operation (usually rounding down or rounding to the nearest integer) ensures that the written value is a valid integer. Then, these two decimal integers are converted to 16-bit hexadecimal representations: 80 = 0x0050; 282 = 0x011A.

[0181] In one alternative implementation, the conversion rules may differ if the switch uses different chip vendors or different queue management architectures. For example, some chips require thresholds to be in 16-byte granularity, or require the write to be a thousandth of the threshold relative to the total buffer capacity. The configuration execution module dynamically selects the correct conversion formula by reading the device tree or chip ID, generating a hardware-recognizable register write value that strictly matches the current hardware platform.

[0182] Step S9.2: Through PCIe BAR space mapping or kernel driver interface, the register write value is directly written to the hardware registers in the queue management module corresponding to the ECN low marking threshold and ECN high marking threshold; after the register write is completed, subsequent data packets entering the port queue will be subject to congestion marking judgment operation based on the newly configured threshold.

[0183] According to one aspect of this application, two typical paths for physically writing the converted register value into the switching chip hardware, and changes in data plane behavior after configuration takes effect.

[0184] Specifically, the configuration execution module selects one of the following two write paths depending on the switch system architecture: Path 1: PCIe BAR Space Mapping Method. On white-box switches using open network operating systems (such as SONiC), the register space of the switching chip is fully mapped to the physical address space of the host CPU via the PCIe bus. For example, the register base address of the queue management module is 0xFB000000, the offset of the TM_WRED_MIN_THRESHOLD register is 0x0120, and the offset of the TM_WRED_MAX_THRESHOLD register is 0x0124. The configuration execution module, through a user-space driver (such as libpciaccess), performs a 32-bit write operation to physical address 0xFB000120, writing the value 0x00000050; and writes 0x0000011A to physical address 0xFB000124. Because this operation is a direct memory-mapped write, the latency is extremely low, typically completed within tens of nanoseconds.

[0185] Path Two: Kernel Driver Interface Approach. On switches running a full Linux kernel, the hardware operations of the switching chip are encapsulated as kernel driver modules. For example, the driver exposes the ECN threshold configuration as a sysfs virtual file node. The configuration execution module writes the register value as a string to the corresponding node via a standard file write system call. The write callback function in the kernel driver receives this value, performs necessary lock protection and hardware write timing, and ultimately completes the register update.

[0186] In this embodiment, regardless of the write path used, the hardware registers have the characteristic of taking effect immediately. Once the write operation is completed and transmitted to the queue management hardware logic via the chip's internal synchronization bus, the new threshold will take effect in the next clock cycle (typically within single-digit nanoseconds).

[0187] Furthermore, the data plane behavior after the effect takes effect is as follows: For each data packet that subsequently enters the queue of this port, the hardware queue management module executes the following judgment logic: obtain the instantaneous depth Q of the current queue. current (Unit: buffer unit); If Q current If the queue depth is less than 80 (i.e., less than 80 × 254 = 20320 bytes), the packet is not marked, the IP header ECN field remains unchanged, and it enters the queue for normal scheduling and forwarding; if 80 ≤ Q current <282, hardware based on a (Q) current The probability of setting the CE flag in the ECN field of a data packet is calculated proportionally to -80. The flagging probability increases linearly with queue depth until it reaches 100% at a threshold of 282. If Q current If the queue depth reaches or exceeds 282 (i.e., the queue depth is 282 × 254 = 71628 bytes), all incoming packets are unconditionally set to the CE flag.

[0188] In one optional implementation, if the port supports a compatible mode for explicit congestion notification, packets marked with CE will continue to be forwarded normally to the outgoing port; if the queue depth further exceeds the preset packet loss threshold, the hardware will trigger a tail-drop policy. At this point, the congestion control parameters generated by the semantic-level collaborative decision-making framework have been fully deployed to the data plane and begin to regulate actual network traffic.

[0189] This invention addresses the core challenge of decision oscillation in existing centralized-distributed collaborative architectures from a semantic perspective. Existing solutions, such as the linear weighted fusion method of the local Q-value matrix and the global guidance matrix used in CoPTER, implicitly assume that the two types of decision information differ only at the level of precision. However, in-depth analysis reveals that the Q-value learned by the local reinforcement learning agent based on port-level observations has the semantic meaning of "the expected cumulative reward for selecting a certain threshold configuration under the current local observations," while the guidance matrix output by the global performance model has the semantic meaning of "the prediction of the impact of configuring a certain threshold on the overall network flow completion time under the current network traffic distribution." These two semantics are inherently incommensurable. When a local port encounters micro-burst congestion while the actual global network load is relatively light, the local Q-value tends to significantly reduce the ECN threshold to quickly empty the queue, while the global guidance matrix tends to maintain a higher threshold to ensure throughput. Weighting and summing these two semantically conflicting signals at the numerical level does not produce a true compromise, but rather a structural distortion of the decision plane. This manifests as spurious local extrema appearing on the fused Q-value surface, causing parameter configurations to repeatedly jump between adjacent decision cycles. This invention abandons the approach of numerical layer fusion, instead repositioning the output of the centralized controller as a semantic adjustment vector. This vector no longer directly intervenes in each specific threshold selection, but rather shapes the agent's behavioral tendencies at the decision preference level by reparameterizing the reward function of the distributed agent. This semantic layer guidance mechanism integrates global information into the local decision-making process as soft constraints rather than hard constraints, eliminating decision oscillations caused by semantic discrepancies. In mixed traffic load scenarios, the method of this invention reduces the switching frequency of ECN threshold configuration by approximately 45% compared to existing solutions, effectively ensuring the continuity and stability of the congestion control strategy.

[0190] Furthermore, this invention achieves temporal decoupling between slow global policy changes and rapid local execution changes through a dynamic injection mechanism of semantic adjustment vectors, thereby improving the network's adaptability to traffic bursts. In traditional centralized-distributed collaborative architectures, the process of the centralized controller generating guidance parameters and the distributed agent's local inference lacks effective coordination on a temporal scale. Although the controller updates the global view at second-level intervals, its output guidance values ​​directly participate in every microsecond-level local decision, resulting in a rigid binding between the update frequency of global information and the execution frequency of local decisions. When network traffic patterns change rapidly, the controller has not yet completed a new round of global optimization calculations, but the local agent has already made hundreds of parameter adjustments based on outdated global guidance, causing policy execution to lag behind traffic evolution. This invention creatively introduces a semantic adjustment vector as an intermediate representation layer, reconstructing the interaction between the centralized controller and the distributed agent from numerical instruction transmission to preference parameter configuration. The semantic adjustment vector is updated and issued by the controller at second-level intervals to reparameterize the reward calculation function within the agent, while the agent itself still independently executes state perception and action decision-making at microsecond-level intervals. This architectural design achieves decoupling of two key time scales: First, once the reward function is reconstructed by the semantic adjustment vector, it is internalized into the agent's inherent decision-making logic. The agent does not need to interact with the controller again in subsequent local decision-making cycles, greatly reducing its sensitivity to control plane communication latency. Second, the gradual change characteristic of the semantic adjustment vector ensures a smooth transition in global policy preferences. When the controller detects a shift in traffic patterns from steady state to large-scale incast, the semantic adjustment vector gradually increases the latency sensitivity dimension, and the agent's decision preference slowly drifts from throughput priority to queue control priority, avoiding performance shocks caused by sudden policy changes. Furthermore, the physical meaning of each dimension of the semantic adjustment vector is clear and its values ​​are interpretable. Network operators can monitor the changing trends of the semantic adjustment vector to intuitively perceive the current global congestion mode of the network and the direction of policy guidance applied by the controller, improving the system's observability and maintainability.

[0191] Furthermore, this invention addresses the consistency problem of end-to-end congestion response in multi-path load balancing scenarios by employing path semantic alignment constraints and graph attention coding mechanisms. The widespread deployment of equivalent multipath routing and packet spraying techniques in modern data center networks means that different packets from the same Transmission Control Protocol (TCP) flow may arrive at the receiver via different physical paths. While this design effectively improves link utilization, it presents a severe challenge of end-to-end semantic inconsistency for congestion control: on multiple paths traversed by a flow, the independently operating agents on each switch port may learn conflicting threshold adjustment strategies due to differences in local observations. For example, an upstream port on one path might detect a shallow queue and maintain an aggressively low ECN labeling rate, while a bottleneck port on another concurrent path might execute a conservatively high ECN labeling rate due to a deep queue. This causes the congestion feedback signal received at the source to fluctuate wildly between no congestion and severe congestion, resulting in unnecessary oscillations in the transport layer sliding window and severely impairing throughput performance. Existing solutions to this problem have significant shortcomings. While CoPTER performs global performance prediction through path decomposition, each port still independently weights and merges its components during the final decision-making process, lacking an end-to-end policy coordination mechanism. This invention systematically solves this problem from two levels. At the global awareness level, a graph attention network is used to uniformly encode the entire network topology and traffic distribution. The inherent characteristics of the graph attention mechanism allow ports with similar topological distances and sharing the same upstream or downstream connections to perceive each other's congestion status. The encoded congestion semantic embedding vector naturally carries spatial correlation information, rather than viewing each port in isolation. At the policy constraint level, this invention introduces a path semantic alignment constraint mechanism. For all ports on the same forwarding path, their semantic adjustment vectors are forcibly constrained within the neighborhood of the path semantic adjustment center vector, ensuring that the entire path adopts a consistent policy tendency when dealing with congestion, i.e., simultaneously tightening or simultaneously relaxing thresholds. This constraint effectively avoids signal cancellation or signal amplification distortion caused by policy conflicts between upstream and downstream ports. In simulation tests of multi-path incast scenarios, the method of this invention reduced the tail delay of the flow completion time by about 30% compared with existing solutions, verifying its technical advantages in ensuring semantic consistency of end-to-end congestion response.

[0192] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A method for adaptive adjustment of congestion control parameters based on semantic-level collaborative decision-making, characterized in that, include: Obtain global traffic statistics and topology link status data of the data center network. The global traffic statistics include flow distribution and queue status information at the switch port level. Based on global traffic statistics and topology link status data, graph attention encoding is performed to generate congestion semantic embedding vectors that represent the semantic role of ports in global network congestion propagation. These congestion semantic embedding vectors are then mapped to semantic adjustment vectors used to regulate distributed decision preferences. The semantic adjustment vector is sent to the switch port, and the reward calculation function of the reinforcement learning agent is reparameterized according to the semantic adjustment vector to obtain the dynamic reward generator after semantic injection. The transient congestion observation data of the switch port is collected, input into the reinforcement learning agent, and combined with the dynamic reward generator for decision-making and policy update processing to generate congestion control configuration parameters. Configure the queue management register of the switch port according to the congestion control configuration parameters to complete the adaptive adjustment of the congestion control parameters.

2. The method according to claim 1, characterized in that, The steps to generate congestion semantic embedding vectors include: For each switch port, the reported flow distribution histogram data, average packet length statistics, traffic burst index, and current queue length normalized value are processed by feature splicing and normalization to construct the node's initial feature matrix. The network physical link connection relationship and the corresponding link bandwidth utilization and link propagation delay are obtained, and weighted fusion and normalization are performed to generate a weighted adjacency matrix. The edge weights in the weighted adjacency matrix are used to characterize the congestion cost of the corresponding link. The initial feature matrix and weighted adjacency matrix of the nodes are input into the multi-head graph attention network to perform attention coefficient calculation and multi-head feature aggregation processing, and output the congestion semantic embedding vector corresponding to each port. Among them, the multi-head graph attention network is obtained through offline pre-training, and its training objective is to encode discrete port congestion propagation roles in a continuous vector space.

3. The method according to claim 2, characterized in that, The steps to map congestion semantic embedding vectors to semantic conditioning vectors include: The congestion semantic embedding vector is input into the semantic mapping network for forward inference processing. The semantic mapping network contains multiple fully connected layers and non-linear activation functions. Its network parameters are obtained through meta-learning training based on historical traffic data. The semantic mapping network performs feature compression and nonlinear transformation on the congestion semantic embedding vector, and outputs a low-dimensional original semantic adjustment vector. Each dimension of the original semantic adjustment vector corresponds to a congestion control preference, which includes at least: throughput preference coefficient, latency sensitivity coefficient, ECN labeling aggressiveness coefficient, and policy stability coefficient.

4. The method according to claim 3, characterized in that, Before sending the semantic conditioning vector to the switch port, the following is also included: Obtain the set of end-to-end forwarding paths in the data center network and determine the sequence of switch ports traversed by each path; For each path, the arithmetic mean of the original semantic adjustment vectors corresponding to all ports on that path is calculated to obtain the path semantic adjustment center vector; For each port belonging to the path, calculate the deviation between its original semantic adjustment vector and the path semantic adjustment center vector. If the deviation exceeds a preset constraint threshold, perform a numerical correction process to move closer to the path semantic adjustment center vector to obtain the corrected semantic adjustment vector. The revised semantic adjustment vector will be used as the semantic adjustment vector for distribution.

5. The method according to claim 1, characterized in that, The reward calculation function of the reinforcement learning agent is reparameterized, including: In the data plane or control plane of the switch port, read the issued and cached semantic adjustment vector, which is represented as a multi-dimensional floating-point array; Replace the weight coefficients in the original static reward function of the reinforcement learning agent with activation function values ​​that take the corresponding dimension values ​​of the semantic adjustment vector as independent variables; The activation function values ​​map the values ​​of each dimension of the semantic adjustment vector to a preset weight range through the Sigmoid function, so that the global semantic preference is injected into the optimization objective of the reinforcement learning agent in a continuously differentiable manner.

6. The method according to claim 5, characterized in that, The steps for collecting transient congestion observation data from switch ports and inputting it into the reinforcement learning agent include: The raw register values ​​are collected from the switch hardware queue management module at preset microsecond intervals. The raw register values ​​include at least: instantaneous queue depth value, outgoing port transmission rate count value, and ECN tag packet count value. The collected raw register values ​​are processed by an exponentially weighted moving average to filter out micro-burst noise and obtain smoothed observation data. The smoothed observation data is normalized and concatenated with the congestion control parameter configuration value of the previous decision cycle to construct the reinforcement learning state vector of the current time step. The reinforcement learning state vector is input into the policy network and value network of the reinforcement learning agent.

7. The method according to claim 6, characterized in that, The steps involved in decision-making and policy update processing using a dynamic reward generator include: The policy network outputs the selection probability distribution of each available action in the discretized congestion control parameter action space based on the input reinforcement learning state vector. Random sampling is performed based on the selection probability distribution to determine the target action index, and the target action index is decoded into specific congestion control parameter configuration values; Wait for the next time interval to arrive, observe the new reinforcement learning state vector, and call the dynamic reward generator to calculate the immediate reward value. The immediate reward value carries global semantic preferences because the reward calculation function has been reparameterized. The experience tuple is stored in the priority experience replay pool. The experience tuple includes at least: the current state vector, the target action index, the immediate reward value, and the new state vector. Based on batch data sampled from the priority experience replay pool, the temporal difference error and policy gradient are calculated, and incremental update processing is performed on the weight parameters of the policy network and the value network.

8. The method according to claim 7, characterized in that, The steps for configuring the queue management register of a switch port according to congestion control configuration parameters include: The congestion control parameter configuration values ​​obtained from decoding are converted into hardware-recognizable register write values ​​according to the format specified in the switching chip hardware manual. The register write values ​​are directly written to the hardware registers in the queue management module corresponding to the ECN low mark threshold and ECN high mark threshold through PCIe BAR space mapping or kernel driver interface. Once the register is written, subsequent data packets entering the port queue will be subject to congestion marking based on the newly configured threshold.

9. The method according to claim 2, characterized in that, When calculating the attention coefficient, the multi-head graph attention network introduces the edge weights in the weighted adjacency matrix as a scaling factor into the calculation of the attention coefficient, so that the neighboring nodes of the links with higher congestion costs obtain higher attention weights in the feature aggregation process of the current node.

Images