Multi-modal multi-objective optimization-based computing power network redundant path construction method and scheduling system

Abstract

The invention discloses a computing power network redundant path construction method based on multi-modal multi-objective optimization and a scheduling system, and belongs to the technical field of routing optimization in a computing power network. According to the method, firstly, for a constructed multi-objective optimization function including the minimum average time delay of all service traffic, the minimum maximum link bandwidth utilization rate and the maximum successful routing rate of all traffic, a transformer-based deep reinforcement learning and truncation selection strategy is adopted to realize multi-modal multi-objective optimization of routing decision; compared with the prior art that an equivalent path cannot be obtained only through an optimal path, the solving method provided by the invention can make an optimal choice for problems such as routing link faults on the premise of ensuring Qos.

Description

Technical Field

[0001] The present invention belongs to the field of routing optimization in computing power networks, and specifically relates to a computing power network redundant path construction method and scheduling system based on multi-modal and multi-objective optimization. Background Art

[0002] By integrating computing power with the network, the computing network overcomes the computing capacity limitations of isolated data centers and edge computing systems, building a new intelligent, efficient, and on-demand computing service system that integrates computing and networking. Computing routing is a key component of the computing network. It dynamically generates service scheduling strategies based on the state of multi-dimensional resources and services, such as network, computing, and storage. The continuous development of optimization algorithms has led researchers to transform routing problems into optimization problems, using existing optimization algorithms to solve traffic paths. However, most optimization methods focus on a single objective, such as latency or bandwidth. This may not truly improve network QoS. However, research on multi-objective optimization algorithms allows us to model routing problems as multi-objective optimization problems. For example, we can simultaneously optimize latency, bandwidth, resource utilization, packet loss rate, and reliability, thereby achieving low latency, high resource utilization, and load balancing in the network environment. However, computing networks are based on ubiquitous computing power. Link failures can occur in the links and routers that make up the network environment due to various reasons, such as difficult maintenance, interface failures, and fiber optic disconnections. For applications that strictly require low latency and high reliability, such as defense, healthcare, financial transactions, autonomous driving, and industrial automation, these failures must be completely eliminated. Therefore, alternative paths must be quickly found to ensure that services can be processed while meeting their QoS requirements. While rerouting mechanisms can locate alternative routes, they still require time and resources to calculate. Therefore, for applications with high network quality of service requirements, achieving imperceptible rerouting when a link failure occurs is a significant challenge.

[0003] Multimodal, multi-objective optimization algorithms can help us solve the above problems because they can obtain multiple equivalent decisions for a certain point on the Pareto frontier, or a decision on a local optimal solution. As long as the local optimal solution meets the minimum requirements of the service flow, such as latency and bandwidth, it can be used as a candidate path for rerouting.

[0004] Existing multimodal, multi-objective optimization algorithms are all modified versions of multi-objective evolutionary algorithms. They primarily focus on two aspects: 1. Improving search capabilities allows the algorithm to find more optimal solutions in the decision space. 2. Implementing appropriate environmental selection mechanisms ensures that points that are close in the goal space but far apart in the decision space are retained. Most existing evolutionary algorithms make decisions based on these two principles. However, while these methods can find Pareto frontiers—equivalent solutions related to the frontier—they suffer from unstable results, poor generalization, and significant influence of encoding and decoding. Summary of the Invention

[0005] Purpose of the invention: In response to the shortcomings of the above-mentioned existing technologies, the present invention provides a method for constructing redundant paths in a computing power network based on multimodal and multi-objective optimization, which uses deep reinforcement learning combined with evolutionary selection strategies to solve multimodal problems; the second purpose of the present invention is to provide a large-scale computing power routing link scheduling system.

[0006] Technical Solution: A method for constructing redundant paths in a computing network based on multimodal and multi-objective optimization. This method uses transformer-based deep reinforcement learning combined with a truncation selection strategy to achieve multimodal and multi-objective optimization of routing decisions. The steps include:

[0007] S1. Build a multi-objective routing decision optimization model based on basic network information. The multi-objective optimization function of this model includes minimizing the average delay of all service flows, minimizing the maximum link bandwidth utilization, and maximizing the successful routing rate of all flows:

[0008] Minimize the average delay F of all business flows Delay , the corresponding objective function is as follows:

[0009]

[0010] where on t Is the flag for whether the tth traffic is successfully routed, are the transmission delay, propagation delay, optical-electrical-optical signal conversion delay, and server processing delay of the service flow t respectively;

[0011] Minimize the maximum link bandwidth utilization F bandwdith , the corresponding objective function is as follows:

[0012]

[0013] Among them B i,j represents the total bandwidth resource size from node i to node j, b t is the bandwidth required for the t-th flow, Indicates whether the tth flow passes through the link from node i to node j;

[0014] Maximize the successful routing rate F of all traffic success , the corresponding objective function is as follows:

[0015]

[0016] Where T represents the total number of business flows;

[0017] S2. Determine the constraints for routing decisions, including path continuity, indivisibility of service traffic, and the limited bandwidth and server resources.

[0018] ① Ensure the indivisibility of traffic

[0019]

[0020] Among them, H, K, and S represent the terminal node set, routing node set, and cloud-edge server node set, respectively. Indicates whether the tth flow passes through the source node h t If the link to node j passes, then otherwise where j∈K∪S. It represents whether the t-th flow passes through the link from node i to node s. If so, the value is 1, otherwise the value is 0, where i∈H∪K,s∈S;

[0021] Constraint ① is used to ensure that traffic can only enter the access layer from one node and can only enter the target node from one node, thereby achieving traffic indivisibility;

[0022] ② Ensure the continuity of traffic

[0023]

[0024] This constraint ② indicates that after traffic enters a certain intermediate node m, it must exit from this intermediate node, thus ensuring the continuity of the path. Indicates whether the tth flow passes through the link from node i to node m. If so, the value is 1, otherwise the value is 0, where i∈H∪K,m∈K. Indicates whether the tth flow passes through the link from node m to node j. If so, the value is 1, otherwise the value is 0, where j∈K∪S;

[0025] ③ Ensure that the path has no loops

[0026]

[0027]

[0028] Constraint ③ ensures that the number of times traffic passes through any intermediate node m is 1, to ensure that the path is loop-free;

[0029] ④Limit resource usage to no more than the maximum

[0030]

[0031] Constraint ④ is used to limit the bandwidth usage of each link to not exceed the maximum bandwidth of the link, and to ensure that the computing and storage resource usage of each server does not exceed the maximum computing and storage resources. Where V = H ∪ K ∪ S represents the node set. Indicates whether the tth flow passes through the link from node i to node j (d). If so, the value is 1, otherwise the value is 0, where i∈V,d∈S. C d ,M d Respectively represent the computing resource size and storage resource size of server node d. t ,m t They represent the computing resources and storage resource requirements of the t-th flow respectively;

[0032] ⑤ Ensure the effectiveness of traffic delay

[0033]

[0034] Constraint ⑤ ensures that the delay cost of each flow route to the target node does not exceed its maximum allowable delay limit;

[0035] (3) A weight-aware deep reinforcement learning algorithm based on a proximal policy optimization algorithm and a truncation selection strategy is used to optimize multimodal and multi-objective routing decisions. This algorithm uses a proximal policy optimization algorithm to process the interaction between the agent and the environment, maximizes the cumulative reward through strategy adjustment, and follows a Markov decision process.

[0036] Set a set of weight combinations w 1 ,…,w N ,in is the weight of average delay, is the weight of the maximum bandwidth routing rate, The weight of successful routing rate;

[0037] The entire combinatorial optimization objective is decomposed into N sub-problems through N weight combinations, and DRL trains these sub-problems in turn;

[0038] During the training process, each solution is stored in the set Population. When the set size reaches a fixed value PopSize, it is regarded as a population, and the optimal solution and equivalent solutions are screened out through the selection strategy.

[0039] Furthermore, the basic network information includes network topology information, link bandwidth information, distance information between two points, resource information of the cloud edge server, computing resources and storage resources, as well as basic information of the flow, including the source node of the flow, the size of the flow data, the computing resource requirement of the flow, the storage resource requirement of the flow, the maximum allowable delay of the flow, and the bandwidth requirement of the flow.

[0040] This method solves the multi-objective routing decision optimization model based on the weight-aware deep reinforcement learning algorithm with PPO and truncation selection strategy (WADRL-PPO-TSS). It involves limiting the amplitude of each update by clipping the objective function to reduce the gradient variance while selecting the optimal solution and equivalent solutions through the selection strategy. The process also includes:

[0041] The entire combinatorial optimization objective is decomposed into N sub-problems through N weight combinations. The Transformer-based deep reinforcement learning algorithm trains these sub-problems in sequence. During the training process, each solution is stored in the set Population.

[0042] When the set size reaches a fixed value PopSize, it is regarded as a population, and the optimal solution and equivalent solutions are screened out through the selection strategy.

[0043] Furthermore, the method described in this paper employs a proximal policy optimization (PPO) algorithm, which limits the magnitude of each update by clipping the objective function, effectively reducing gradient variance and making the training process more stable. Therefore, by replacing the existing policy gradient algorithm with PPO, a weight-aware deep reinforcement learning algorithm based on PPO and a truncation selection strategy is developed to solve the multi-objective routing decision optimization problem.

[0044] The core of the PPO algorithm lies in the interaction between the agent and the environment. It maximizes the cumulative reward by adjusting the strategy, following the Markov decision process (MDP). Furthermore, the Markov decision process in solving the multi-objective routing decision optimization model includes:

[0045] State: Take the network topology G and traffic matrix T as static state, and the bandwidth matrix B t and server resources S t As a dynamic state, the dynamic state is dynamically updated due to resource consumption caused by traffic routing;

[0046] Action: At each time step t, the agent performs the action A t is the traffic T to be routed in this step t wherein, the present invention obtains feasible paths for all flows in advance and directly selects the corresponding path at time step t.

[0047] State transition function: After time step t, the first t flows have been routed. Considering that the static state remains unchanged, the dynamic state changes with the flow path allocation. The bandwidth resource matrix consumed by the tth flow is defined as The bandwidth matrix update formula is:

[0048]

[0049] where on t Indicates whether the tth flow is routed successfully;

[0050] When the traffic reaches the target node and the server resources are sufficient, the server resources are updated as follows:

[0051]

[0052] in Indicates the server resource set used by the tth flow;

[0053] Reward function: time step t performs a t After that, get the first t traffic paths and the corresponding average delay Maximum bandwidth utilization and successful routing rate Instant Reward R t Expressed as:

[0054]

[0055] in Represents F Delay , F bandwidth , F success The minimum and maximum values ​​of are solved by deep reinforcement learning respectively;

[0056] In the Markov decision process of solving the multi-objective routing decision optimization model, the goal of this method is to minimize the average delay and maximum bandwidth utilization, so its coefficient is set to a negative number;

[0057] Based on the above processing, although it can be predicted that the traffic can reach the target node, considering the delay and bandwidth resource limitations, the routing success rate is 0, that is, when When setting That is, when all routes fail, a very small reward value will be obtained.

[0058] Furthermore, the multi-objective routing decision optimization model described in the method can be divided into an encoding module, a decoding module, and a value assessment module. The processing tasks of each module include:

[0059] The encoding module is provided with a weight embedding layer, which embeds network topology information G, flow information T and action information. The action information refers to all possible paths of the flow that need to be decided at the current time step, which is recorded as This module embeds all the above information related to the network into a high-dimensional space and extracts key features through the self-attention mechanism;

[0060] In addition, the link's available bandwidth information B t and server resource information S t The weight information will change dynamically with the execution of the action. To this end, the present invention adds a weight embedding layer to embed the information of the three together.

[0061] The decoding module generates the probability distribution of feasible paths using the features extracted by the encoding module;

[0062] The value evaluation module evaluates the maximum benefit that can be obtained at the current time step t based on the embedded high-dimensional features.

[0063] Furthermore, because the network topology and traffic information matrices have different dimensions, their features need to be encoded separately as high-dimensional information. Network topology information contains multiple matrices, such as the bandwidth matrix B, the distance matrix L, and the target node resource matrix S. These matrices have different dimensions and cannot be processed simultaneously by a single encoder. Therefore, they need to be processed as input variables.

[0064] The specific processing tasks of each module of the multi-objective routing decision optimization model include:

[0065] (1) Encoding of network topology

[0066] Use g=(g0,g1,…,g E ) represents the information of each edge in the network topology, where Represents the information of the i-th link. Using linear transformation to process the characteristic information, the expression obtained is:

[0067]

[0068] use Represents the embedded high-dimensional space, where is the set of trainable parameters, d e is a hyperparameter, representing the dimension of the high-dimensional space;

[0069] After obtaining the initial embedding, the encoded information passes through the multi-head attention layer MHA to obtain the query Q m , key K m Sum V m , after which the attention value Am Calculated by the following formula:

[0070]

[0071] Perform softmax normalization on the similarity between the query and the key, and then multiply the value vector to obtain the mth attention value;

[0072] MHA(G (0) ) is calculated by the following formula, that is, all attention values ​​are combined and the weight W is used. O Generate the final multi-head attention output MHA(G (0) )=[A1,A2,…,A m ]W O ; Obtained from the attention value through the jump connection layer and BN layer Therefore, the output of the entire Encoder is recorded as G 1 ,

[0073] (2) Action information A at the current time step t t The traffic matrix information T is encoded and embedded in the same way as step (1), where the traffic matrix It is a matrix and is not processed;

[0074] (3) For the link available bandwidth matrix B t and target resource matrix S t The embedding of , considering that it will change as the action is executed, therefore a dynamic state embedding layer is set up to embed information about the dynamic state and the variable target weight, specifically including:

[0075] Available bandwidth matrix B t Expressed as in Indicates the two node numbers and available bandwidth of a link, using Represents the target available resource matrix. represents the node number of the i-th server,

[0076] Available computing resources and available storage resources are embedded using the following two formulas:

[0077]

[0078] (4) Decoding operation: At each time step t, the agent determines the state s t Select actions, including static states Dynamic Status and action information Expressed as:

[0079]

[0080] Use the multi-head attention layer MHA to obtain the context information of the state and get Q at ,K a ,V a ;

[0081] Using the formula To calculate the compatibility between the path and the state, the probability distribution of the simple path selection is calculated by compatibility, and the expression is:

[0082] P t =softmax(λ t )

[0083] (5) Based on deep reinforcement learning, high-quality solutions under different weight combinations are obtained, and the optimal solution and equivalent solution are screened and retained through a value evaluation module. The value evaluation module performs population screening based on the priority truncation strategy and the centroid truncation strategy. The two strategies respectively screen out excellent solutions from the individual perspective and also find equivalent solutions from the group perspective.

[0084] Since the decision variable of each individual I is T traffic, I.dec∈N T×len ,When calculating the Euclidean distance between individuals, it is necessary to first flatten the decision variables into a vector of dimension T×len.

[0085] In the process of population screening using the priority truncation strategy and the centroid truncation strategy, the Euclidean distance in the decision space is used to measure the similarity between individuals. The resulting distance matrix is ​​used to evaluate the distribution of individuals in the population, thereby guiding the screening and optimization of the population.

[0086] The training algorithm uses the proximal policy optimization algorithm to train network parameters. Each training cycle contains T time steps, generates T traffic paths as population individuals, and stores them in the Population set. When the cumulative number of training times reaches the update frequency uf, the PPO algorithm updates and synchronizes the new and old network parameters θ and θ through the strategy old After each PopSize cycle is completed, the current population is merged with the historical elite population MIA, and excellent individuals and equivalent solutions are screened based on the priority truncation and center truncation strategies, and then the Population is cleared for the next round of iteration.

[0087] The present invention also provides a link scheduling system for large-scale computing power routing. When the cloud service requested by the user cannot be processed at the current inference node, the scheduling system executes the described method to switch the equivalent path.

[0088] Beneficial Effects: Existing multimodal, multi-objective methods directly employ evolutionary algorithms for solution. However, in the field of multi-objective optimization, there are solutions that combine reinforcement learning with evolutionary algorithms. These methods employ reinforcement learning training first, then use the final results as the initial solution for the evolutionary algorithm, subjecting it to multiple crossover and mutation selection cycles to obtain the final solution set. The method provided by the present invention utilizes the evolutionary algorithm's selection strategy to select the optimal population for each solution set generated by reinforcement learning. After selection is complete, the next episode is entered. This effectively uses reinforcement learning as a method for generating new populations. Because the crossover selection strategy of evolutionary algorithms may not be able to find new routing solutions in the routing domain, deep reinforcement learning is used instead. Experimental results show that compared to existing multi-objective optimization techniques that focus solely on finding optimal solutions, the present invention can indeed find more equivalent solutions. Furthermore, the number of equivalent solutions increases exponentially with increasing the θ equivalence threshold. WADRL performs relatively poorly, while genetic evolution algorithms exhibit almost no equivalent solutions. Therefore, the present invention can increase the diversity of choices available to decision makers for highly multimodal problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 This is an application framework diagram of the scheduling system of the present invention;

[0090] Figure 2 This is an algorithm framework diagram for weight-aware deep reinforcement learning combined with evolutionary selection strategy;

[0091] Figure 3 Algorithm flow chart for the priority evolution truncation selection strategy;

[0092] Figure 4 Flowchart of the algorithm for selecting a strategy for centroid truncation;

[0093] Figure 5 This is the model training flow chart. DETAILED DESCRIPTION

[0094] In order to illustrate the technical solution disclosed in the present invention in detail, the present invention is further described below with reference to the accompanying drawings and examples.

[0095] Combine Figure 1 and Figure 2 As shown, the application of the present invention is as follows:

[0096] First, a central controller senses computing resources, orchestrates and deploys computing resources, and finally uses a routing algorithm to determine the final path. The multimodal, multi-objective optimization algorithm proposed in this invention is used for routing calculations, calculating the optimal routing path and its equivalent paths. When the central controller distributes the flow table, it simultaneously distributes the optimal path as the primary path and its equivalent paths as candidate paths to each router. When the network is in good condition, the routers will accept and forward traffic as normal. However, because the computing network is highly dynamic, a link on the primary path can suddenly become congested, causing most traffic on this link to timeout and fail to route. This invention proposes adding a special link status awareness service to the central controller to monitor link status at all times and perform fault analysis upon receiving fault reports. The purpose of fault analysis is to identify which traffic flows will be prevented from reaching the target server due to the fault. Furthermore, the algorithm analyzes whether the candidate paths generated by the algorithm can be directly routed as alternative paths without recalculation.

[0097] Figure 1 This is the algorithm framework diagram for weight-aware deep reinforcement learning combined with evolutionary selection strategy. Figure 2 A network architecture diagram illustrating the establishment and use of redundant paths in the computing network. The routing algorithm calculates two equivalent paths, with the primary path represented by the solid black line and the backup path by the dashed black line. Initially, traffic is routed along the primary path. When congestion occurs, the central controller immediately detects the congestion and analyzes all traffic currently or potentially affected by the congestion to determine whether alternative paths can bypass the congested link. If so, a notification is immediately issued and the routing tables of the relevant routers are updated. For some traffic, if the alternative path still traverses a congested link, re-detection and re-routing are performed. For this traffic in the diagram, since the alternative path does not traverse the congested link, if the traffic has not yet reached the end node of the congested link, it will be routed along the alternative path as if no congestion has occurred. Otherwise, it will be re-routed from the end node, saving re-routing time.

[0098] The present invention simulates the network environment. In order to simulate the network environment, the embodiment uses 56 nodes to form a network topology, including 16 host nodes, 12 cloud edge servers, and 28 intermediate nodes, including access nodes, aggregation nodes, base stations and backbone nodes. Secondly, in order to compare the impact of the increase in the number of flows on the algorithm, each algorithm is experimented in a network environment with 32, 48, 64, and 80 flows respectively. For the convenience of explanation, this embodiment uses 32 flows to illustrate the solution. The parameters used in the experiment are shown in Table 1.

[0099] Table 1. Ranges of indicators in the simulation environment

[0100]

[0101] In this example, WADRL-PPO-TSS (a weight-aware deep reinforcement learning algorithm based on PPO and truncation selection strategy provided by the present invention), WADRL, NSGAII-500, NSGAII-1000, and SAC-NSGAII algorithms are considered respectively. Taking into account multiple objectives such as average latency, maximum bandwidth utilization, and routing success rate, a multi-objective routing planning model is solved in a simulation environment to find the optimal solution and its equivalent solution. It is expressed as follows:

[0102] Objective function 1: Minimize the average delay F of all traffic flows Delay :

[0103]

[0104] where on t The flag for whether the t-th item, where t∈{1,2,…,32}, is successfully routed. If successful, the value is 1, otherwise, the value is 0. They are the transmission delay, propagation delay, OEO conversion delay, and server processing delay of service traffic t. The calculation method is as follows:

[0105]

[0106] where d t Indicates the packet size of the tth flow, in MB, ranging from 1 to 3. Indicates whether the tth flow passes through the link from node i to node j. If so, it is 1, otherwise it is 0. t The bandwidth required for the t-th flow, in MB / s.

[0107]

[0108] Because the link medium affects the link propagation speed, the propagation speed must be calculated separately for different media. w Represents the traffic propagation speed of the optical fiber link, in KM / s, w i,j Determine the link Whether it is optical fiber, if yes, it is 1, if not, it is 0. wl i,j and ts wl It represents wireless connection. i,j Represents the distance between links i and j, in kilometers.

[0109]

[0110] Where β = 0.01 represents the photoelectric-to-optical conversion delay, in ms.

[0111]

[0112] Where α = 5242800 represents the number of instructions required to process 1MB of data, in IPS. t The computing resource size required by the tth traffic flow is represented by . These four delays are finally unified into milliseconds.

[0113] Objective function 2: Minimize the maximum link bandwidth utilization F bandwdith

[0114]

[0115] Among them B i,j Indicates the total bandwidth resource size from node i to node j, in Gb / s.

[0116] Objective function 3: Maximize the successful routing rate F of all traffic success

[0117]

[0118] Where T represents the total number of service flows.

[0119] (2) Design routing decision constraints to ensure path rationality, including path continuity, indivisibility of service traffic, and limited bandwidth and server resources.

[0120] ① Ensure the indivisibility of traffic

[0121]

[0122] Where H, K, and S represent the terminal node set, routing node set (access layer, aggregation layer, core layer), and cloud-edge server node set, respectively. The above three formulas ensure that traffic can only enter the access layer from one node and can only reach the target node from one node, ensuring traffic indivisibility.

[0123] ② Ensure the continuity of traffic

[0124]

[0125] This means that after traffic enters an intermediate node, it must exit from the intermediate node, thus ensuring the continuity of the path.

[0126] ③ Ensure that the path has no loops

[0127]

[0128] Ensure that traffic passes through any intermediate node only once to ensure there are no loops on the path.

[0129] ④ Ensure that resource usage does not exceed the maximum

[0130]

[0131]

[0132] Ensure that the bandwidth usage of each link does not exceed the maximum bandwidth of the link, and ensure that the computing and storage resource usage of each server does not exceed the maximum computing and storage resources.

[0133] ⑤ Ensure the effectiveness of traffic delay

[0134]

[0135] Ensure that the delay cost of routing each flow to the destination node does not exceed its maximum allowed delay limit.

[0136] (3) Design a Transformer-based PPO combined with evolutionary selection strategy method for solving multimodal and multi-objective routing decision optimization, specifically:

[0137] The multi-objective problem is decomposed into 21 subproblems using the weight combination [[1, 0, 0], [0.8, 0, 0.2], [0.8, 0.2, 0], …, [0, 0, 1]]. The neural network parameters for each subproblem are initialized using the model of the previous subproblem, and each subproblem is trained sequentially. During training, each round is set to 32 time steps, and a complete solution is obtained in each round. Each solution is stored in the set Population. When the set size reaches 100, it is treated as a population, and the optimal solution and equivalent solutions are screened using a selection strategy.

[0138] Combine Figure 3-5 As shown in Figure 2, we use the proximal policy optimization (PPO) algorithm to limit the amplitude of each update by clipping the objective function, effectively reducing the gradient variance and making the training process more stable. Therefore, we use PPO to replace the original policy gradient algorithm.

[0139] The core of the PPO algorithm lies in the interaction between the agent and the environment. It maximizes the cumulative reward by adjusting the strategy and follows the Markov decision process (MDP). Specifically, it includes the following elements:

[0140] State: The state is divided into two categories: static and dynamic. The static state (network topology G and traffic matrix T) has nothing to do with the action; the dynamic state (bandwidth matrix Bt and server resources S t ) is dynamically updated due to resource consumption caused by traffic routing.

[0141] Action: At each time step t, the agent performs the action A t is the traffic T to be routed in this step t Get the feasible path of all traffic in advance, and directly use the traffic T at time step t. t Select from the set of feasible paths.

[0142] State transition function: After time step t, the first t flows have been routed. The static state remains unchanged, while the dynamic state changes with the flow path allocation. Define the bandwidth resource matrix consumed by the tth flow as The bandwidth matrix update formula is:

[0143]

[0144] where on t Indicates whether the tth flow is routed successfully.

[0145] When the traffic reaches the target node and the server resources are sufficient, the server resources are updated as follows:

[0146]

[0147] in Indicates the server resource set used by the tth flow.

[0148] Reward: execute a at time step t t After that, the first t traffic paths and their corresponding average delays can be obtained. Maximum bandwidth utilization and successful routing rate Instant Reward R t for:

[0149]

[0150] in Respectively represent F Delay , F bandwidth , F success The minimum and maximum values ​​of are solved by DRL respectively. In addition, the goal of the present invention is to minimize the average delay and maximum bandwidth utilization, so their coefficients are set to negative numbers.

[0151] However, although we can predict that the traffic can reach the target node, due to the time delay and bandwidth resource limitations, the routing success rate is 0, that is, when When setting That is, when all routes fail, a very small reward value will be obtained.

[0152] The encoding module not only needs to embed the network topology information G and traffic information T, but also needs to embed the action information (that is, all possible paths of the traffic that need to be decided at the current time step, recorded as ) is embedded. In addition, the link's available bandwidth information B t and server resource information S t The weight information changes dynamically as actions are executed. To this end, we add a weight embedding layer to embed all three pieces of information together. The encoding module embeds all of the aforementioned network-related information into a high-dimensional space and extracts key features through a self-attention mechanism. The decoding module uses the features extracted by the encoding module to generate a probability distribution of feasible paths. The value network uses these embedded high-dimensional features to estimate the maximum reward achievable at the current time step t.

[0153] Because the network topology and traffic information matrices have different dimensions, their features need to be encoded separately as high-dimensional information. Network topology information consists of multiple matrices, such as the bandwidth matrix B, the distance matrix L, and the target node resource matrix S. These matrices have different dimensions and cannot be processed simultaneously by a single encoder. Therefore, they need to be processed as input variables.

[0154] Use g=(g1,g2,…,g 65 ) represents the information of 65 edges in the topology, Represents the information of the i-th link. Traffic matrix T∈R 32×6 It is a matrix, so it does not need to be processed. Action information It is a simple path set. If the action set is too large, it will make the training difficult to converge. Therefore, the number of feasible paths for each host node is forced to be 10, but Of variable length, we obtain the longest path The number of nodes in each path is 8, and then the number of nodes in each path is filled to 8 with the target node of the path.

[0155] For the encoding process of network topology, linear transformation is first used to process feature information. The expression is:

[0156]

[0157] use Represents the high-dimensional space after embedding. After obtaining the initial embedding, the encoded information is processed through a multi-head attention layer (MHA), a skip connection layer, a batch normalization layer (BN), and a fully connected feedforward layer (FF) to generate the final embedding. MHA uses 8 heads, each with a dimension of 8. For each head m, m∈1,2,…,8, the query (Q m ), key (K m ) and value (V m ) is calculated as follows:

[0158] Q m =W Qm ×G (0)

[0159] K m =W (Km) ×G (0)

[0160] V m =W Vm ×G (0)

[0161] Where W Qm ,W Km ∈R 8×64 , and W Vm ∈R 64×64 After that, the attention value A m Calculated by the following formula:

[0162]

[0163] That is, the similarity between the query and the key is softmax normalized, and then multiplied by the value vector to obtain the mth attention value.

[0164] MHA(G (0) ) is calculated by the following formula, that is, all attention values ​​are combined and the weight W is used. O Generate the final multi-head attention output MHA(G (0) ).

[0165] MHA(G (0) )=[A1,A2,…,A8]W O

[0166] Through the skip connection layer and BN layer, we can obtain The formula is as follows:

[0167]

[0168] Therefore, the output of the entire Encoder is recorded as G 1 , expressed as follows:

[0169]

[0170] In addition to the topological information, the action information A at the current time step t t The same embedding method is also used for the traffic matrix information T. In addition, the link available bandwidth matrix B t and target resource matrix S t It will change as the action is executed. Therefore, a dynamic state embedding layer is set up to embed information about the dynamic state and the variable target weight.

[0171] Available bandwidth matrix B t Expressed as in Indicates the two node numbers and available bandwidth of a link. Represents the target available resource matrix. Represents the node number, available computing resources, and available storage resources of the i-th server. They are embedded using the following two formulas:

[0172]

[0173] The last step of the policy network is the decoding operation. At each time step t, the agent is based on the state s t Select actions, including static states Dynamic Status and action information Specifically expressed as:

[0174]

[0175] The MHA layer is used to obtain the contextual information of the state. The query information is defined as the state embedding, and the key value is defined as the embedding of the action information. The specific calculation is as follows:

[0176]

[0177] Using the formula To calculate the compatibility between the path and the state. Finally, the probability distribution of simple path selection is calculated by compatibility. The expression formula is:

[0178] P t =softmax(λ t )

[0179] Next, the obtained population is selected through the selection algorithm to select the optimal solution and equivalent solution. Since the decision variable of each individual I is 32 traffic, that is, I.dec∈N 32×8 ,When calculating the Euclidean distance between individuals, the decision variables need to be flattened into a vector of dimension 32×8.

[0180] The obtained distance matrix can be used to evaluate the distribution of each individual in the population, and then guide the screening and optimization of the population. The training algorithm uses the proximal policy optimization algorithm (PPO) to train the network parameters. Its process combines Figure 3-5 As shown in Figure 2. Each training cycle (epoch) contains 32 time steps (steps), generating 32 traffic paths as population individuals and storing them in the Population set. When the cumulative number of training times reaches the update frequency uf = 1680, PPO updates and synchronizes the new and old network parameters θ and θ through the strategy. old Furthermore, after every 100 cycles, the current population is merged with the historical elite population MIA, and excellent individuals and equivalent solutions are screened based on the priority truncation and center truncation strategies, and then the Population is cleared for the next round of iteration.

[0181] Due to the high complexity of network topology, it is almost impossible to find two completely equivalent solutions in practice. Based on the definition of multimodal multi-objective optimization, we give the following definition:

[0182] For a solution x on the Pareto front, if there exists a solution y that satisfies:

[0183] ||F(x)-F(y)||2≤θ (F(x)=[f1(x),f2(x),f3(x)] T )

[0184] Then y is called the θ-equivalent solution of x, where θ>0 is the preset equivalent threshold. In addition, since the dimensions and numerical ranges corresponding to each indicator are different, to ensure accurate judgment of equivalence between solutions, the objective function in each solution set needs to be normalized before calculating whether there is equivalence between solutions. The normalization formula is as follows:

[0185]

[0186] Run WADRL-PPO-TSS and other comparison algorithms in network environments with different numbers of flows, and record the number of points on the Pareto front of the obtained population as N. PF , and the number of θ-equivalent solutions to these points is denoted as N PS Record this in Table 2. The results show that, compared to multi-objective optimization methods that focus solely on finding excellent solutions, the proposed WADRL-PPO-TSS algorithm can indeed find more equivalent solutions. Furthermore, as the θ equivalent threshold increases, the number of equivalent solutions increases exponentially. WADRL performs relatively poorly, and the genetic evolution algorithm finds almost none. Therefore, the highly multimodal nature of our algorithm can increase the diversity of choices made by decision makers.

[0187] Table 2 Comparison of multimodality of various methods

[0188]

[0189]

[0190] Based on the application of the above method, a link switching scheduling system for large-scale computing power routing can be obtained. When the cloud service requested by the user cannot be processed at the current inference node, the scheduling system executes the described method to switch the equivalent path. For example, an enterprise or platform (such as ByteDance, Baidu, Tencent Cloud, etc.) provides users with large model (such as ChatGPT, Wenshengtu, audio and video generation) services, and user requests are scheduled to the optimal computing power cluster (GPU) to perform inference tasks. These clusters are deployed in multiple IDCs, edge nodes or cloud service areas across the country. Assuming that the inference node in the Beijing area is currently congested, the system can switch to the data center in Jinan, Tianjin or Hohhot to process user requests. These paths are equivalent paths with comparable computing power and similar service quality.

Claims

1. A method for constructing redundant paths in a computing network based on multi-modal and multi-objective optimization, characterized in that: This method is based on transformer-based deep reinforcement learning combined with a truncation selection strategy to achieve multimodal and multi-objective optimization of routing decisions. The steps include: S1. Build a multi-objective routing decision optimization model based on basic network information. The multi-objective optimization function of this model includes minimizing the average delay of all service flows, minimizing the maximum link bandwidth utilization, and maximizing the successful routing rate of all flows: Minimize the average delay F of all business flows Delay , the corresponding objective function is as follows: where on t Is the flag for whether the tth traffic is successfully routed, are the transmission delay, propagation delay, optical-electrical-optical signal conversion delay, and server processing delay of the service flow t respectively; Minimize the maximum link bandwidth utilization F bandwdith , the corresponding objective function is as follows: Among them B i,j represents the total bandwidth resource size from node i to node j, b t is the bandwidth required for the t-th flow, Indicates whether the tth flow passes through the link from node i to node j; Maximize the successful routing rate F of all traffic success , the corresponding objective function is as follows: Where T represents the total number of business flows; S2. Determine the constraints for routing decisions, including path continuity, indivisibility of service traffic, and the limited bandwidth and server resources. ① Ensure the indivisibility of traffic Among them, H, K, and S represent the terminal node set, routing node set, and cloud-edge server node set, respectively. Indicates whether the tth flow passes through the source node h t If the link to node j passes, then otherwise where j∈K∪S, It represents whether the t-th flow passes through the link from node i to node s. If so, the value is 1, otherwise the value is 0, where i∈H∪K,s∈S; Constraint ① is used to ensure that traffic can only enter the access layer from one node and can only enter the target node from one node, thereby achieving traffic indivisibility; ② Ensure the continuity of traffic This constraint ② indicates that after traffic enters a certain intermediate node m, it must exit from this intermediate node, thus ensuring the continuity of the path. Indicates whether the tth flow passes through the link from node i to node m. If so, the value is 1, otherwise the value is 0, where i∈H∪K,m∈K, Indicates whether the tth flow passes through the link from node m to node j. If so, the value is 1, otherwise the value is 0, where j∈K∪S; ③ Ensure that the path has no loops Constraint ③ ensures that the number of times traffic passes through any intermediate node m is 1, to ensure that the path is loop-free; ④Limit resource usage to no more than the maximum Constraint ④ is used to limit the bandwidth usage of each link to not exceed the maximum bandwidth of the link, and to ensure that the usage of computing resources and storage resources of each server does not exceed the maximum resources of computing resources and storage resources, where V = H ∪ K ∪ S represents the node set; Indicates whether the tth flow passes through the link from node i to node j (d). If so, the value is 1, otherwise the value is 0; C d ,M d They represent the computing resource size and storage resource size of server node d, c t ,m t They represent the computing resources and storage resource requirements of the t-th flow respectively; ⑤ Ensure the effectiveness of traffic delay Constraint ⑤ ensures that the delay cost of each flow route to the target node does not exceed its maximum allowable delay limit, t represents the maximum allowed delay of the t-th flow; (3) A weight-aware deep reinforcement learning algorithm based on a proximal policy optimization algorithm and a truncation selection strategy is used to optimize multimodal and multi-objective routing decisions. This algorithm uses a proximal policy optimization algorithm to process the interaction between the agent and the environment, maximizes the cumulative reward through strategy adjustment, and follows a Markov decision process. Set a set of weight combinations w 1 ,…,w N ,in is the weight of average delay, is the weight of the maximum bandwidth routing rate, The weight of successful routing rate; The entire combinatorial optimization objective is decomposed into N sub-problems through N weight combinations, and DRL trains these sub-problems in turn; During the training process, each solution is stored in the set Population. When the set size reaches a fixed value PopSize, it is regarded as a population, and the optimal solution and equivalent solutions are screened out through the selection strategy.

2. The method for constructing redundant paths in a computing network based on multi-modal and multi-objective optimization according to claim 1, characterized in that: The basic network information includes network topology information, link bandwidth information, distance information between two points, resource information of cloud edge servers, computing resources and storage resources, as well as basic information of the flow, including the source node of the flow, the size of the flow data, the computing resource requirement of the flow, the storage resource requirement of the flow, the maximum allowable delay of the flow, and the bandwidth requirement of the flow.

3. The method for constructing redundant paths in a computing network based on multi-modal and multi-objective optimization according to claim 1, characterized in that: This method solves the multi-objective routing decision optimization model using a weight-aware deep reinforcement learning algorithm based on PPO and a truncation selection strategy. It involves limiting the amplitude of each update by clipping the objective function to reduce the gradient variance while selecting the optimal solution and equivalent solutions through the selection strategy. The process also includes: The entire combinatorial optimization objective is decomposed into N sub-problems through N weight combinations. The Transformer-based deep reinforcement learning algorithm trains these sub-problems in sequence. During the training process, each solution is stored in the set Population. When the set size reaches a fixed value PopSize, it is regarded as a population, and the optimal solution and equivalent solutions are screened out through the selection strategy.

4. The method for constructing redundant paths in a computing network based on multimodal multi-objective optimization according to claim 1 or 3, characterized in that: The Markov decision process in solving the multi-objective routing decision optimization model includes: State: Take the network topology G and traffic matrix T as static state, and the bandwidth matrix B t and server resources S t As a dynamic state, the dynamic state is dynamically updated due to resource consumption caused by traffic routing; Action: At each time step t, the agent performs the action A t is the traffic T to be routed in this step t feasible paths; State transition function: After time step t, the first t flows have been routed. Considering that the static state remains unchanged, the dynamic state changes with the flow path allocation. The bandwidth resource matrix consumed by the tth flow is defined as The bandwidth matrix update formula is: where on t Indicates whether the tth flow is routed successfully; When the traffic reaches the target node and the server resources are sufficient, the server resources are updated as follows: in Indicates the server resource set used by the tth flow; Reward function: time step t performs a t After that, get the first t traffic paths and the corresponding average delay Maximum bandwidth utilization and successful routing rate Instant Reward R t Expressed as: in Represents F Delay , F bandwidth , F success The minimum and maximum values ​​of are solved by DRL respectively; In the Markov decision process of solving the multi-objective routing decision optimization model, the goal of this method is to minimize the average delay and maximum bandwidth utilization, so its coefficient is set to a negative number; Considering the time delay and bandwidth resource limitation, the routing success rate is set to 0, that is, when When setting That is, when all routes fail, a very small reward value will be obtained.

5. The method for constructing redundant paths in a computing network based on multi-modal multi-objective optimization according to claim 1 or 4, characterized in that: The multi-objective routing decision optimization model can be divided into an encoding module, a decoding module, and a value assessment module. The processing tasks of each module include: The encoding module is equipped with a weighted embedding layer that embeds network topology information G, traffic information T, and action information. This module embeds all the above network-related information into a high-dimensional space and extracts key features through a self-attention mechanism; The decoding module generates the probability distribution of feasible paths using the features extracted by the encoding module; The value evaluation module evaluates the maximum benefit that can be obtained at the current time step t based on the embedded high-dimensional features.

6. The method for constructing redundant paths in a computing network based on multi-modal and multi-objective optimization according to claim 5, characterized in that: The specific processing tasks of each module of the multi-objective routing decision optimization model include: (1) Encoding of network topology Use g=(g0,g1,…,g E ) represents the information of each edge in the network topology, where Represents the information of the i-th link. The expression obtained by processing the characteristic information using linear transformation is: use Represents the embedded high-dimensional space, where is the set of trainable parameters, d e is a hyperparameter, representing the dimension of the high-dimensional space; After obtaining the initial embedding, the encoded information passes through the multi-head attention layer MHA to obtain the query Q m , key K m Sum V m , after which the attention value A m Calculated by the following formula: Perform softmax normalization on the similarity between the query and the key, and then multiply the value vector to obtain the mth attention value; MHA(G (0) ) is calculated by the following formula, that is, all attention values ​​are combined and the weight W is used. O Generate the final multi-head attention output MHA(G (0) )=[A1,A2,…,A m ]W O ; Obtained from the attention value through the jump connection layer and BN layer Therefore, the output of the entire Encoder is recorded as G 1 , (2) Action information A at the current time step t t The traffic matrix information T is encoded and embedded in the same way as step (1), where the traffic matrix It is a matrix and is not processed; (3) For the link available bandwidth matrix B t and target resource matrix S t The embedding of , considering that it will change as the action is executed, therefore a dynamic state embedding layer is set up to embed information about the dynamic state and the variable target weight, specifically including: Available bandwidth matrix B t Expressed as in Indicates the two node numbers and available bandwidth of a link, using represents the target available resource matrix, where Indicates the node number of the i-th server; Available computing resources and available storage resources are embedded using the following two formulas: (4) Decoding operation: At each time step t, the agent determines the state s t Select actions, including static states Dynamic Status and action information Expressed as: Use the multi-head attention layer MHA to obtain the context information of the state and get Q at ,K a ,V a ; Using the formula To calculate the compatibility between the path and the state, the probability distribution of the simple path selection is calculated by compatibility, and the expression is: P t =softmax(λ t ) (5) Based on deep reinforcement learning, high-quality solutions under different weight combinations are obtained, and the optimal solution and equivalent solution are screened and retained through a value evaluation module. The value evaluation module performs population screening based on the priority truncation strategy and the centroid truncation strategy. The two strategies respectively screen out excellent solutions from the individual perspective and also find equivalent solutions from the group perspective.

7. The method for constructing redundant paths in a computing network based on multi-modal and multi-objective optimization according to claim 6, characterized in that: In the process of population screening using the priority truncation strategy and the centroid truncation strategy, the Euclidean distance in the decision space is used to measure the similarity between individuals. The resulting distance matrix is ​​used to evaluate the distribution of individuals in the population, thereby guiding the screening and optimization of the population. The training algorithm uses the proximal policy optimization algorithm to train network parameters. Each training cycle contains T time steps, generates T traffic paths as population individuals, and stores them in the Population set. When the cumulative number of training times reaches the update frequency uf, the PPO algorithm updates and synchronizes the new and old network parameters θ and θ through the strategy old After each PopSize cycle is completed, the current population is merged with the historical elite population MIA, and excellent individuals and equivalent solutions are screened based on the priority truncation and center truncation strategies, and then the Population is cleared for the next round of iteration.

8. The method for constructing redundant paths in a computing network based on multi-modal and multi-objective optimization according to claim 7, characterized in that: Since the decision variable of each individual I is T traffic, i.dec∈N T×len ,When calculating the Euclidean distance between individuals, it is necessary to first flatten the decision variables into a vector of dimension T×len.

9. A large-scale computing power routing link scheduling system, characterized in that: When the cloud service requested by the user cannot be processed at the current inference node, the scheduling system performs the method according to any one of claims 1 to 8 to switch to an equivalent path.

Images