DCQCN multi-dimensional parameter collaborative optimization method and system based on reinforcement learning

Abstract

The invention is suitable for the technical field of parameter optimization, and provides a DCQCN multi-dimensional parameter collaborative optimization method and system based on reinforcement learning, and the method comprises the steps: collecting network topology structure data, flow characteristic data and network performance index data in real time; determining a parameter optimization demand according to a preset triggering condition, wherein the triggering condition comprises periodic triggering, performance threshold triggering or topology change triggering; performing ring structure detection on the node connection matrix by using a spectral clustering algorithm, when the detected maximum ring length is greater than a set threshold value, determining that the network topology structure is mesh topology, otherwise, determining that the network topology structure is tree topology; if the network topology structure is a tree topology, performing parameter optimization by adopting a three-level layered reinforcement learning architecture; if the network topology is mesh topology, performing parameter optimization by adopting a federal reinforcement learning framework; and the parameters of the reinforcement learning model are dynamically updated according to the network state feedback data after parameter adjustment, closed-loop optimization is completed, and the network stability is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of parameter optimization technology, and in particular to a DCQCN multi-dimensional parameter collaborative optimization method and system based on reinforcement learning. Background Art

[0002] The DCQCN (Decentralized Quantized Congestion Notification) flow control protocol, as the core mechanism of RDMA networks, can effectively alleviate network congestion and improve transmission efficiency by dynamically adjusting multi-dimensional parameters such as ECN marking thresholds and rate limits. Currently, traditional DCQCN parameter configuration relies primarily on manual experience and static rules, making it difficult to adapt to dynamically changing network loads and complex topologies. This is especially true when faced with bursty traffic, link failures, or heterogeneous business scenarios. Fixed parameter strategies can easily lead to global performance fluctuations or imbalanced local resource utilization. For example, in a tree topology, traffic fluctuations at edge nodes can cause cascading congestion due to the lack of a hierarchical coordination mechanism. In a mesh topology, the high coupling between nodes makes local parameter adjustments prone to policy conflicts. These performance bottlenecks make it difficult to meet the high transmission efficiency and operation and maintenance requirements of complex data networks.

[0003] In view of this, a DCQCN multi-dimensional parameter collaborative optimization method and system based on reinforcement learning is needed. Summary of the Invention

[0004] This application provides a DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning, which is used to solve the problem that DCQCN parameter configuration is difficult to meet the high requirements of transmission efficiency and operation and maintenance of complex data networks.

[0005] The first aspect of this application provides a DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning, including:

[0006] Real-time collection of network topology data, traffic characteristics data, and network performance indicator data, wherein the network topology data includes a node connection matrix and a device capability vector;

[0007] Determine parameter optimization requirements based on preset trigger conditions, including periodic triggering, performance threshold triggering, or topology change triggering. The performance threshold triggering is when the end-to-end delay exceeds a preset threshold or the packet loss rate exceeds a set threshold. The topology change triggering is when a network node or link is added or deleted.

[0008] A spectral clustering algorithm is used to detect the ring structure of the node connection matrix. When the maximum ring length detected is greater than a set threshold, the network topology is determined to be a mesh topology, and a three-level hierarchical reinforcement learning framework is used for parameter optimization. Otherwise, it is determined to be a tree topology, and a federated reinforcement learning framework is used for parameter optimization.

[0009] The network state is adjusted based on the optimized parameters, and the reinforcement learning model parameters are dynamically updated according to the feedback data of the adjusted network state to complete the closed-loop optimization.

[0010] Furthermore, the parameter optimization requirement is determined according to a preset trigger condition, and the trigger condition includes a periodic trigger, a performance threshold trigger, or a topology change trigger. The performance threshold trigger is when the end-to-end delay exceeds a preset threshold or the packet loss rate exceeds a set threshold. The topology change trigger is when a network node or link is added or deleted, including:

[0011] Determine performance violations based on end-to-end latency and packet loss, and topology changes based on the addition or deletion of network nodes or links.

[0012] Generate corresponding parameter optimization requests according to the performance limit exceeding event or the topology change event, and mark the trigger type tag;

[0013] The real-time data associated with the trigger event is used as the input of the optimization task. The parameter search space initialization strategy is selected according to the trigger type. The performance threshold trigger adopts historical parameter weight decay, and the topology change trigger adopts parameter reset.

[0014] Furthermore, the generating of corresponding parameter optimization requests according to the performance exceeding event or the topology changing event and marking of trigger type tags respectively include:

[0015] The trigger conditions that are met simultaneously are sorted by priority, where the priority order is that the performance threshold trigger is higher than the topology change trigger, and the topology change trigger is higher than the periodic trigger;

[0016] Multiple events of the same trigger type within a continuous time window are merged into a single optimization request, and the first trigger timestamp and cumulative number of events are recorded.

[0017] Furthermore, the node connection matrix is detected for ring structure using a spectral clustering algorithm. When the maximum ring length detected is greater than a set threshold, the network topology is determined to be a mesh topology, and a three-level hierarchical reinforcement learning framework is used for parameter optimization. Otherwise, the topology is determined to be a tree topology, and a federated reinforcement learning framework is used for parameter optimization, including:

[0018] Construct an undirected graph based on the node connection matrix of the network topology and remove node self-loop edges;

[0019] Calculating an unnormalized Laplace matrix based on the node connection matrix and the degree matrix, performing eigenvalue decomposition on the Laplace matrix, and generating a characteristic matrix containing eigenvectors corresponding to the first several smallest non-zero eigenvalues;

[0020] Normalizing the eigenvectors in the feature matrix and dividing the nodes into several subgraphs using a clustering algorithm;

[0021] Perform a depth-first search on each subgraph to detect simple cycles, and record the maximum cycle length of each subgraph;

[0022] If the maximum value of the maximum ring length of all subgraphs is the set threshold, it is determined to be a mesh topology, otherwise it is determined to be a tree topology.

[0023] Furthermore, the network topology is determined to be a mesh topology, and a three-level hierarchical reinforcement learning architecture is used for parameter optimization, including:

[0024] Calculate the Q value based on the current state and execute the action, while uploading the timing difference error to the convergence layer;

[0025] The convergence layer dynamically assigns weights based on the traffic load of each leaf node, weightedly aggregates the temporal difference error, generates policy gradients, and updates the global target network parameters. The updated parameters are sent to each leaf node to synchronize the local model.

[0026] The root node receives the policy gradient from the convergence layer, generates the target policy parameters and sends them to all leaf nodes.

[0027] Furthermore, the convergence layer dynamically assigns weights based on the traffic load of each leaf node, weightedly aggregates the temporal difference error, generates policy gradients and updates the global target network parameters, and sends the updated parameters to each leaf node to synchronize the local model, including:

[0028] The calculation formula of the timing difference error δ is:

[0029]

[0030] Where: r t For state s t Next, perform action a t The immediate reward value obtained after γ∈[0,1] is the discount factor used to balance the importance of current rewards and future rewards. For the next state s t+1 The estimated value of the maximum Q value among all possible actions, Q(s t ,a t ) is in state s t Next, perform action a t The current estimated Q value of

[0031] Aggregation error δ agg The calculation formula is:

[0032]

[0033] Where: N is the total number of nodes aggregated, w′ i is the weight of node i, δ i is the timing differential error of node i.

[0034] Furthermore, the method of determining that the network topology is a mesh topology and using a three-level hierarchical reinforcement learning architecture for parameter optimization also includes:

[0035] The root node calculates the importance weight of the gradient of each level through the attention mechanism, and the attention weight is generated by mapping the gradient features of the fully connected layer;

[0036] Train the global Q network with the KL divergence loss function to generate the target policy parameters;

[0037] The leaf node collects the network status feedback data after execution, calculates the deviation between the actual reward and the predicted reward, and updates the local DQN model parameters if the deviation exceeds the preset threshold.

[0038] Furthermore, the above-mentioned method determines that the topology is a tree topology, and adopts a federated reinforcement learning framework to perform parameter optimization, including:

[0039] A local deep deterministic policy gradient model is deployed on each node. The state space of the local deep deterministic policy gradient model includes link bandwidth utilization, end-to-end delay, and neighbor node queue depth, and the action space is continuous flow control parameter adjustment instructions.

[0040] The node adds Laplace noise to the gradient of the local deep deterministic policy gradient model to satisfy the differential privacy constraint, generates privacy-preserving gradients, and uploads them to the parameter server through an encrypted channel;

[0041] The parameter server aggregates the privacy-preserving gradients of all nodes using a federated averaging algorithm, generates global model parameters, and sends them to each node.

[0042] After receiving the global parameters, each node fuses the local deep deterministic policy gradient model with the global model parameters through a soft update strategy.

[0043] Furthermore, otherwise determining that the topology is a tree topology and using a federated reinforcement learning framework for parameter optimization also includes:

[0044] Global gradient The calculation formula is:

[0045]

[0046] Where: M is the total number of nodes in federated learning, w ′ i ′ is the weight of node i, is the privacy protection gradient of node i, is the local gradient of node i, λ is the noise scale;

[0047] The calculation formula for fusing global model parameters with local model parameters is:

[0048] θ local ←τθ global +(1―τ)θ local

[0049] Where: θ local are local model parameters, θ global is the global model parameter, and τ is the mixing coefficient, which is used to control the fusion ratio of global parameters and local parameters.

[0050] The second aspect of the embodiment of the present application provides a DCQCN multi-dimensional parameter collaborative optimization system based on reinforcement learning, including:

[0051] A real-time data acquisition unit, configured to acquire network topology data, traffic characteristic data, and network performance indicator data in real time. The network topology data includes a node connection matrix and a device capability vector.

[0052] A parameter optimization requirement determination unit is configured to determine parameter optimization requirements based on preset trigger conditions, wherein the trigger conditions include periodic triggering, performance threshold triggering, or topology change triggering. The performance threshold triggering is when the end-to-end delay exceeds a preset threshold or the packet loss rate exceeds a set threshold. The topology change triggering is when a network node or link is added or deleted.

[0053] a network topology determination unit, configured to perform ring structure detection on the node connection matrix using a spectral clustering algorithm, and determine that the network topology is a mesh topology when the maximum ring length detected is greater than a set threshold, otherwise it is determined to be a tree topology;

[0054] A first parameter optimization unit is used to optimize parameters using a three-level hierarchical reinforcement learning architecture if the network topology is a tree topology;

[0055] The second parameter optimization unit is used to optimize the parameters using a federated reinforcement learning framework if the network topology is a mesh topology;

[0056] The model parameter updating unit is used to adjust the network state based on the optimized parameters and dynamically update the reinforcement learning model parameters according to the adjusted network state feedback data to complete the closed-loop optimization.

[0057] It can be seen from the above technical solutions that the embodiments of the present invention have the following advantages:

[0058] Based on the determination of parameter optimization requirements, the present invention determines whether the current network topology is a tree topology or a mesh topology. If the network topology is a tree topology, a three-level hierarchical reinforcement learning architecture is used for parameter optimization. If the network topology is a mesh topology, a federated reinforcement learning framework is used for parameter optimization. The reinforcement learning model is used to jointly optimize multi-dimensional parameters such as ECN marking threshold, rate limit, and cache allocation, thereby overcoming the local suboptimal problem caused by traditional single-parameter optimization, improving global throughput and reducing end-to-end latency. The use of three-level hierarchical reinforcement learning or federated learning framework can improve the policy adaptability under different network structures, reduce the risk of cascade congestion under tree topology, and reduce policy conflicts between nodes in mesh topology. The present invention can reduce policy conflict events in complex data center environments, and provide an efficient and secure intelligent operation and maintenance solution for highly dynamic and highly concurrent data center networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The figure is a flow chart of an embodiment of a DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning in the present invention. DETAILED DESCRIPTION

[0060] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0061] Example 1

[0062] The implementation method in this embodiment can be implemented in the system, can be implemented in the server, and can also be implemented in the terminal, without specific limitation. The following will introduce the DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning in this application from the perspective of system implementation. Figure 1 , the method provided in the embodiment of the present application includes the following steps:

[0063] S11. Real-time collection of network topology data, traffic characteristics data and network performance index data, network topology data including node connection matrix and device capability vector;

[0064] In this embodiment, the node connection matrix is obtained as a binary matrix representing the physical connection relationship between nodes. The device capability vector includes hardware capability indicators such as node cache capacity and port rate, which are used to construct a network topology diagram and identify node capability constraints. Traffic feature data includes link bandwidth utilization and queue depth, and network performance indicator data includes end-to-end delay and packet loss rate. The monitored link bandwidth utilization is the percentage of real-time traffic to theoretical bandwidth, and the queue depth is the number of data packets to be forwarded in the cache queue, reflecting the network load distribution and congestion risk. The measured end-to-end delay is the transmission delay of the data packet from the source to the destination, and the packet loss rate is the proportion of data packets lost during the transmission process, which is used to quantify network transmission efficiency and service quality. These data are collected in real time by distributed probes or SDN controllers, and after standardized processing, they are used as the state space input of the reinforcement learning model, providing a multi-dimensional perception basis for subsequent parameter optimization decisions.

[0065] S12. Determine the parameter optimization requirements based on the preset trigger conditions. The trigger conditions include periodic triggers, performance threshold triggers, or topology change triggers. The performance threshold trigger is when the end-to-end delay exceeds a preset threshold or the packet loss rate exceeds a set threshold. The topology change trigger is when network nodes or links are added or deleted.

[0066] Step S12 includes the following:

[0067] S121 based on end-to-end delay and packet loss rate to determine the performance limit event, and based on the network node or link add or delete operation to determine the topology change event;

[0068] End-to-end delay and packet loss rate are continuously collected through probes and compared with preset thresholds. If the end-to-end delay is greater than the preset threshold and continues to exceed the stable time, or the packet loss rate is greater than the preset threshold and occurs N times in a row, it is determined to be a performance overrun event. When the network management interface is monitored and a node or link addition or deletion operation occurs, it is determined to be a topology change event. It should be noted that periodic triggering is to automatically trigger parameter optimization at preset time intervals (such as every 5 minutes) for the purpose of regularly maintaining the network status and preventing potential performance degradation. Its triggering mechanism is independent of real-time performance or topology changes, ensuring long-term policy updates and model convergence stability, and avoiding parameter aging problems caused by long-term non-optimization.

[0069] S122. Generate corresponding parameter optimization requests based on performance exceeding events or topology changing events, and mark the trigger type tag;

[0070] Performance overrun events generate performance optimization requests, including the overrun indicator and severity. Topology change events generate topology driver requests, including a snapshot of the changed topology and the scope of impact. Tag fields include event type, timestamp, and associated data index. This step also includes:

[0071] 1. Sort the trigger conditions that are met simultaneously by priority, with performance threshold triggers taking precedence over topology change triggers, which in turn take precedence over periodic triggers.

[0072] 2. Multiple events of the same trigger type within a continuous time window, such as continuous packet loss exceeding the limit, are merged into a single optimization request, and the first trigger timestamp and cumulative number of events are recorded.

[0073] S123. Use the real-time data associated with the trigger event as the optimization task input, and select the parameter search space initialization strategy based on the trigger type. The performance threshold trigger adopts historical parameter weight decay, and the topology change trigger adopts parameter reset.

[0074] The real-time data associated with the trigger event is encapsulated as the optimization task input, with a trigger type tag and timestamp attached. The associated real-time data includes the current link bandwidth utilization, queue depth, and historical parameter versions; the topology connection matrix after the change, the device capability vector, and the neighbor node status. The weight decay of historical parameters is used to retain recently valid parameters and narrow the search scope. The parameter reset method discards historical parameters and randomly initializes the weight decay of historical parameters from a uniform distribution to expand the exploration space.

[0075] The encapsulated data and initialization parameters obtained in this step can provide structured input data for the reinforcement learning model.

[0076] S13. Use the spectral clustering algorithm to detect the ring structure of the node connection matrix. When the maximum ring length detected is greater than the set threshold, the network topology is determined to be a mesh topology; otherwise, it is determined to be a tree topology.

[0077] Step S13 includes the following steps:

[0078] S131. Construct an undirected graph based on the node connection matrix of the network topology and remove node self-loop edges;

[0079] Obtain the node connectivity matrix of the network topology. Matrix elements are binary values (0 / 1) representing the physical connections between nodes. Convert the node connectivity matrix into an undirected graph G = (V, E), where V is the set of nodes and E is the set of edges. The existence of edges is determined by the nonzero elements in the matrix.

[0080] S132. Calculate the unnormalized Laplace matrix based on the node connection matrix and the degree matrix, perform eigenvalue decomposition on the matrix, and generate a characteristic matrix containing the eigenvectors corresponding to the first several smallest non-zero eigenvalues;

[0081] Directly use the input node connection matrix, element A ij Indicates that nodes i and j are connected; construct a diagonal matrix with diagonal elements D iiThe degree of node i (number of connected edges); calculate the unnormalized Laplace matrix L = D - A; perform eigenvalue decomposition on L, extract the eigenvectors corresponding to the first k smallest non-zero eigenvalues, and form the characteristic matrix F∈R n×k , where n is the number of nodes, and k is a constant.

[0082] S133. Normalize the eigenvectors in the feature matrix and divide the nodes into several subgraphs using a clustering algorithm;

[0083] Each row of the feature matrix F (node feature vector) is L2 normalized; the normalized feature vector is divided into clusters using the k-means algorithm, each corresponding to a subgraph; k is determined based on the eigenvalue distribution, where k is a constant.

[0084] S134. Perform a depth-first search on each subgraph to detect simple cycles, and record the maximum cycle length of each subgraph;

[0085] For each subgraph G i Perform a depth-first search (DFS) or backtracking-based cycle detection algorithm to traverse all possible simple cycles (no repeated nodes); the cycle length is based on the number of nodes, for example, the length of the cycle v1→v2→v3→v4→v1 is 4; record each subgraph G i The maximum ring length L detected in i .

[0086] S135. If the maximum value of the maximum ring length of all subgraphs is the set threshold, it is determined to be a mesh topology; otherwise, it is determined to be a tree topology.

[0087] Take the maximum length of the maximum ring of all subgraphs, that is, L cycle =max{L1,L2,…,L k}, here the threshold is set to 4, if L cycle =4, the network is determined to be a mesh topology; if L cycle ≠4 or acyclic (L cycle If a subgraph does not exist, it is determined to be a tree topology. If multiple loop lengths exist in the same subgraph, only the maximum length is retained for determination. If a subgraph is an isolated node (no edges), the loop detection step is skipped. Finally, the determination results are cross-validated, for example, by performing a breadth-first search to reconfirm topological connectivity and loop existence.

[0088] S14. If the network topology is a tree topology, use a three-level hierarchical reinforcement learning architecture for parameter optimization.

[0089] Step S14 includes the following:

[0090] S141. Calculate the Q value based on the current state and perform the action, while uploading the timing difference error to the convergence layer;

[0091] An independent Deep Q Network (DQN) is deployed at each leaf node, i.e., terminal device or edge node. Its state space includes local link bandwidth utilization, queue depth, and historical packet loss rate, and its action space is discrete flow control parameter adjustment instructions, such as ECN marking threshold, rate limit value, etc. The leaf node is based on the current state s t The Q value Q(s) of each action is calculated through the DQN network t ,a), select the ε-greedy strategy to execute the optimal action a t , and collect instant rewards r t , the reward function is the weighted inverse of local link utilization and delay. Calculate the time difference error δ, and sum δ and state transfer data (s t ,a t ,r t ,s t+1 ) is uploaded to the aggregation layer.

[0092] The calculation formula of the timing difference error δ is:

[0093]

[0094] Where: r t For state s t Next, perform action a t The immediate reward value obtained after γ∈[0,1] is the discount factor used to balance the importance of current rewards and future rewards. For the next state s t+1 The estimated value of the maximum Q value among all possible actions, Q(s t ,a t ) is in state s t Next, perform action a t The current estimated Q value of .

[0095] S142. The convergence layer dynamically assigns weights based on the traffic load of each leaf node, weights and aggregates the temporal difference error, generates a policy gradient, and updates the global target network parameters. The updated parameters are sent to each leaf node to synchronize the local model.

[0096] The aggregation layer receives the δ values of all child nodes and uses the weighted average method to generate the aggregate error, with the weights dynamically allocated according to the node traffic load.

[0097] Aggregation error δ agg The calculation formula is:

[0098]

[0099] Where: N is the total number of nodes aggregated, w′ i is the weight of node i, δi is the timing differential error of node i.

[0100] Based on δ agg Calculating policy gradients Update the global target network parameters θ of the local DQN through backpropagation ′ . The updated gradient Distribute to each leaf node to trigger local model synchronization.

[0101] S143. The root node receives the policy gradient from the convergence layer, generates the target policy parameters and sends them to all leaf nodes.

[0102] This step also includes the following:

[0103] 1. The root node calculates the importance weights of the gradients at each level through the attention mechanism. The attention weights are generated by mapping the gradient features of the fully connected layer.

[0104] 2. Train the global Q network using the KL divergence loss function to generate the target policy parameters;

[0105] 3. The leaf node collects the network status feedback data after execution and calculates the deviation between the actual reward and the predicted reward. If the deviation exceeds the preset threshold, the local DQN model parameters are updated.

[0106] The root node receives the policy gradients from each convergence layer and uses the attention mechanism to calculate the importance weights of the gradients of each layer. The formula is: in, is the mapping function of the fully connected layer. The weighted gradient is input into the strategy distillation module, and the knowledge distillation loss function is used. Train the global Q network to generate a unified policy parameter θ global . global Send it to all leaf nodes, replace their local DQN parameters, and ensure global policy consistency.

[0107] After implementing the new parameters, leaf nodes collect network state data for the next cycle, calculate the actual reward, and compare it with the predicted value to generate a reward deviation. If the deviation exceeds the tolerance threshold, online fine-tuning of the local DQN is triggered, using the Prioritized Experience Replay (PER) mechanism to resample historical high-error samples to accelerate model convergence. A fixed time window (e.g., 10 minutes) is set to force synchronization of global parameters to all nodes to prevent long-term policy drift.

[0108] The above steps adapt to the hierarchical characteristics of the tree topology and reduce decision latency through a three-level architecture consisting of local response at the leaf node, regional coordination at the aggregation layer, and global optimization at the root node.

[0109] S15. If the network topology is a mesh topology, use the federated reinforcement learning framework for parameter optimization.

[0110] Step S15 includes the following:

[0111] S151. Deploy a local deep deterministic policy gradient model at each node. The state space of the local model includes link bandwidth utilization, end-to-end delay, and neighbor node queue depth, and the action space is a continuous flow control parameter adjustment instruction;

[0112] Each node deploys a local Deep Deterministic Policy Gradient (DDPG) model. Its state space includes link bandwidth utilization, end-to-end latency, and neighbor node queue depths, and its action space consists of continuous flow control parameters such as ECN marking thresholds and rate adjustment ratios. The node generates deterministic actions based on its current state, performs parameter adjustments, and collects rewards. The reward function is a weighted sum of latency reduction and throughput improvement. The node updates the parameters of the local actor-critic network using an experience replay pool. Laplace noise is added to the local gradient to generate a privacy-preserving gradient.

[0113] S152. The node adds Laplace noise to the gradient of the local model to satisfy the differential privacy constraint, generates a privacy-preserving gradient, and uploads it to the parameter server via an encrypted channel.

[0114] The node uploads the privacy-preserving gradient to the parameter server through a TLS encrypted channel, along with the local data volume weight. The parameter server executes the federated averaging algorithm to calculate the global gradient and update the global DDPG model parameters. The calculation formula is:

[0115]

[0116] Where: M is the total number of nodes in federated learning, w ′ i ′ is the weight of node i, is the privacy protection gradient of node i, is the local gradient of node i, and λ is the noise scale.

[0117] S153. The parameter server aggregates the privacy-preserving gradients of all nodes using the federated averaging algorithm, generates global model parameters, and distributes them to each node.

[0118] The parameter server will global It is sent to all nodes, and after receiving it, the nodes fuse the local and global models through the soft update strategy. The calculation formula for fusing the global model parameters with the local model parameters is:

[0119] θ local ←τθ global+(1―τ)θ local

[0120] Where: θ local are local model parameters, θ global is the global model parameter, and τ is the mixing coefficient, which is used to control the fusion ratio of global parameters and local parameters.

[0121] The node runs the new parameters in the local environment and monitors performance indicators, such as the latency change rate. If the indicator degradation exceeds the tolerance threshold, it rolls back to the previous version parameters and triggers an alarm.

[0122] S154. After receiving the global parameters, each node fuses the local model and the global model parameters through a soft update strategy.

[0123] If the node is offline for a timeout, such as exceeding the aggregation period T = 5 minutes, the parameter server automatically removes its weight and recalculates w ′ i ′ This ensures aggregation effectiveness. Furthermore, delayed gradients are allowed to participate in subsequent aggregation rounds, and a timestamp alignment mechanism is used to prevent outdated updates from interfering with global data. Nodes continuously collect optimized network status data, calculate actual rewards, and compare them with predicted values. If the deviation continues to exceed the standard, additional local training rounds are performed and the exploration rate is adjusted. Each node's gradient upload time, privacy budget consumption, and model version number are recorded to support compliance checks and traceability of abnormal behavior.

[0124] The above steps achieve distributed parameter collaborative optimization under multi-node privacy protection through the federated reinforcement learning framework. Combining differential privacy noise addition with FedAvg global aggregation, it adapts to dynamic changes in mesh topology while ensuring data security, significantly reducing end-to-end latency and alleviating policy conflicts.

[0125] S16. Adjust the network state based on the optimized parameters, and dynamically update the reinforcement learning model parameters based on the adjusted network state feedback data to complete the closed-loop optimization.

[0126] After parameter adjustment, network status feedback data is collected in real time through embedded probes or SDN controllers, including end-to-end delay, packet loss rate, link bandwidth utilization, and queue depth. Calculate the compound reward value: Where D is the delay drop rate, L is the packet loss rate change, α and β are dynamic weight coefficients. t ,a t ,r t ,s t+1) is stored in an experience pool, and sampling priorities are set based on the temporal difference error. Small batches of samples are sampled from the experience pool according to priority, and the loss function is calculated. The model parameters are updated using the Adam optimizer. The updated model-generated parameters are input into the ns-3 simulator for shadow verification, requiring them to meet the constraints. If verification fails, a rollback mechanism is triggered to load the historically optimal parameters. The reward function weights are modified based on the failed data, and the exploration rate in the ε-greedy strategy is dynamically adjusted based on the model convergence. The optimized parameters are redeployed to the network equipment, and the next round of data collection and model training is initiated, forming a closed-loop iterative optimization.

[0127] Through the hierarchical and federated reinforcement learning framework of dynamic topology adaptation, the coordinated optimization of multi-dimensional parameters such as ECN thresholds and rate limits is achieved. Combined with privacy protection and closed-loop verification mechanisms, the risk of cascading congestion is reduced in tree topologies, and policy conflicts are reduced in mesh topologies. End-to-end latency is reduced while throughput is increased, effectively improving network stability.

[0128] Example 2

[0129] An embodiment of a DCQCN multi-dimensional parameter collaborative optimization system based on reinforcement learning of the present invention includes the following steps:

[0130] A real-time data acquisition unit is used to collect network topology data, traffic characteristic data, and network performance index data in real time. The network topology data includes a node connection matrix and a device capability vector.

[0131] A parameter optimization requirement determination unit is used to determine parameter optimization requirements based on preset trigger conditions. The trigger conditions include periodic triggering, performance threshold triggering, or topology change triggering. The performance threshold triggering is when the end-to-end delay exceeds a preset threshold or the packet loss rate exceeds a set threshold. The topology change triggering is when a network node or link is added or deleted.

[0132] a network topology determination unit, configured to perform ring structure detection on the node connection matrix using a spectral clustering algorithm, and determine that the network topology is a mesh topology when the maximum ring length detected is greater than a set threshold, otherwise it is determined to be a tree topology;

[0133] A first parameter optimization unit is used to optimize parameters using a three-level hierarchical reinforcement learning architecture if the network topology is a tree topology;

[0134] The second parameter optimization unit is used to optimize the parameters using a federated reinforcement learning framework if the network topology is a mesh topology;

[0135] The model parameter updating unit is used to adjust the network state based on the optimized parameters and dynamically update the reinforcement learning model parameters according to the adjusted network state feedback data to complete the closed-loop optimization.

[0136] The specific functions and uses of the units in this embodiment are similar to the steps in the aforementioned embodiment 1 and are not described in detail here.

[0137] It is understandable that those skilled in the art can, under the guidance of the above embodiments, combine various implementation methods in the above embodiments to obtain technical solutions of multiple implementation methods.

[0138] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning, characterized in that: include: Real-time collection of network topology data, traffic characteristics data, and network performance indicator data, wherein the network topology data includes a node connection matrix and a device capability vector; Determine parameter optimization requirements based on preset trigger conditions, including periodic triggering, performance threshold triggering, or topology change triggering. The performance threshold triggering is when the end-to-end delay exceeds a preset threshold or the packet loss rate exceeds a set threshold. The topology change triggering is when a network node or link is added or deleted. A spectral clustering algorithm is used to detect the ring structure of the node connection matrix. When the maximum ring length detected is greater than a set threshold, the network topology is determined to be a mesh topology, and a three-level hierarchical reinforcement learning framework is used for parameter optimization. Otherwise, it is determined to be a tree topology, and a federated reinforcement learning framework is used for parameter optimization. The network state is adjusted based on the optimized parameters, and the reinforcement learning model parameters are dynamically updated according to the feedback data of the adjusted network state to complete the closed-loop optimization.

2. The DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning according to claim 1 is characterized in that: The parameter optimization requirement is determined according to a preset trigger condition, and the trigger condition includes a periodic trigger, a performance threshold trigger, or a topology change trigger. The performance threshold trigger is when the end-to-end delay exceeds a preset threshold or the packet loss rate exceeds a set threshold. The topology change trigger is when a network node or link is added or deleted, including: Determine performance violations based on end-to-end latency and packet loss, and topology changes based on the addition or deletion of network nodes or links. Generate corresponding parameter optimization requests according to the performance limit exceeding event or the topology change event, and mark the trigger type tag; The real-time data associated with the trigger event is used as the input of the optimization task. The parameter search space initialization strategy is selected according to the trigger type. The performance threshold trigger adopts historical parameter weight decay, and the topology change trigger adopts parameter reset.

3. The DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning according to claim 2 is characterized in that: Generating corresponding parameter optimization requests according to the performance exceeding event or the topology changing event and marking the trigger type tag respectively includes: The trigger conditions that are met simultaneously are sorted by priority, where the priority order is that the performance threshold trigger is higher than the topology change trigger, and the topology change trigger is higher than the periodic trigger; Multiple events of the same trigger type within a continuous time window are merged into a single optimization request, and the first trigger timestamp and cumulative number of events are recorded.

4. The DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning according to claim 1 is characterized in that: The node connection matrix is detected for ring structure using a spectral clustering algorithm. When the maximum ring length detected is greater than a set threshold, the network topology is determined to be a mesh topology, and a three-level hierarchical reinforcement learning architecture is used for parameter optimization. Otherwise, it is determined to be a tree topology, and the federated reinforcement learning framework is used for parameter optimization, including: Construct an undirected graph based on the node connection matrix of the network topology and remove node self-loop edges; Calculating an unnormalized Laplace matrix based on the node connection matrix and the degree matrix, performing eigenvalue decomposition on the Laplace matrix, and generating a characteristic matrix containing eigenvectors corresponding to the first several smallest non-zero eigenvalues; Normalizing the eigenvectors in the feature matrix and dividing the nodes into several subgraphs using a clustering algorithm; Perform a depth-first search on each subgraph to detect simple cycles, and record the maximum cycle length of each subgraph; If the maximum value of the maximum ring length of all subgraphs is the set threshold, it is determined to be a mesh topology, otherwise it is determined to be a tree topology.

5. The DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning according to claim 1 is characterized in that: The network topology is determined to be a mesh topology, and a three-level hierarchical reinforcement learning architecture is used for parameter optimization, including: Calculate the Q value based on the current state and execute the action, while uploading the timing difference error to the convergence layer; The convergence layer dynamically assigns weights based on the traffic load of each leaf node, weightedly aggregates the temporal difference error, generates policy gradients, and updates the global target network parameters. The updated parameters are sent to each leaf node to synchronize the local model. The root node receives the policy gradient from the convergence layer, generates the target policy parameters and sends them to all leaf nodes.

6. The DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning according to claim 5 is characterized in that: The convergence layer dynamically assigns weights based on the traffic load of each leaf node, weightedly aggregates the temporal difference error, generates policy gradients and updates the global target network parameters, and sends the updated parameters to each leaf node to synchronize the local model, including: The calculation formula of the timing difference error δ is: Where: r t For state s t Next, perform action a t The immediate reward value obtained after γ∈[0,1] is the discount factor used to balance the importance of current rewards and future rewards. For the next state s t+1 The estimated value of the maximum Q value among all possible actions, Q(s t ,a t ) is in state s t Next, perform action a t The current estimated Q value of Aggregation error δ agg The calculation formula is: Where: N is the total number of nodes aggregated, w′ i is the weight of node i, δ i is the timing differential error of node i.

7. The DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning according to claim 6 is characterized in that: The network topology is determined to be a mesh topology, and a three-level hierarchical reinforcement learning architecture is used for parameter optimization, further comprising: The root node calculates the importance weight of the gradient of each level through the attention mechanism, and the attention weight is generated by mapping the gradient features of the fully connected layer; Train the global Q network with the KL divergence loss function to generate the target policy parameters; The leaf node collects the network status feedback data after execution, calculates the deviation between the actual reward and the predicted reward, and updates the local DQN model parameters if the deviation exceeds the preset threshold.

8. The DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning according to claim 1 is characterized in that: Otherwise, it is determined to be a tree topology, and the federated reinforcement learning framework is used for parameter optimization, including: A local deep deterministic policy gradient model is deployed on each node. The state space of the local deep deterministic policy gradient model includes link bandwidth utilization, end-to-end delay, and neighbor node queue depth, and the action space is continuous flow control parameter adjustment instructions. The node adds Laplace noise to the gradient of the local deep deterministic policy gradient model to satisfy the differential privacy constraint, generates privacy-preserving gradients, and uploads them to the parameter server through an encrypted channel; The parameter server aggregates the privacy-preserving gradients of all nodes using a federated averaging algorithm, generates global model parameters, and sends them to each node. After receiving the global parameters, each node fuses the local deep deterministic policy gradient model with the global model parameters through a soft update strategy.

9. The DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning according to claim 8 is characterized in that: Otherwise, it is determined to be a tree topology, and the federated reinforcement learning framework is used for parameter optimization, including: Global gradient The calculation formula is: Where: M is the total number of nodes in federated learning, w″ i is the weight of node i, is the privacy protection gradient of node i, is the local gradient of node i, λ is the noise scale; The calculation formula for fusing global model parameters with local model parameters is: i local ←tth global +(1―τ)θ local Where: θ local are local model parameters, θ global is the global model parameter, and τ is the mixing coefficient, which is used to control the fusion ratio of global parameters and local parameters.

10. A DCQCN multi-dimensional parameter collaborative optimization system based on reinforcement learning, characterized in that: The DCQCN multi-dimensional parameter collaborative optimization method based on reinforcement learning described in any one of claims 1 to 9 includes: A real-time data acquisition unit, configured to acquire network topology data, traffic characteristic data, and network performance indicator data in real time. The network topology data includes a node connection matrix and a device capability vector. A parameter optimization requirement determination unit is configured to determine parameter optimization requirements based on preset trigger conditions, wherein the trigger conditions include periodic triggering, performance threshold triggering, or topology change triggering. The performance threshold triggering is when the end-to-end delay exceeds a preset threshold or the packet loss rate exceeds a set threshold. The topology change triggering is when a network node or link is added or deleted. a network topology determination unit, configured to perform ring structure detection on the node connection matrix using a spectral clustering algorithm, and determine that the network topology is a mesh topology when the maximum ring length detected is greater than a set threshold, otherwise it is determined to be a tree topology; A first parameter optimization unit is used to optimize parameters using a three-level hierarchical reinforcement learning architecture if the network topology is a tree topology; The second parameter optimization unit is used to optimize the parameters using a federated reinforcement learning framework if the network topology is a mesh topology; The model parameter updating unit is used to adjust the network state based on the optimized parameters and dynamically update the reinforcement learning model parameters according to the adjusted network state feedback data to complete the closed-loop optimization.

Images