A cloud gateway routing multi-objective optimization method based on reinforcement learning

Abstract

The application discloses a cloud gateway routing multi-objective optimization method and system based on reinforcement learning, and belongs to the technical field of cloud computing and network communication. The application aims to solve the problem that the existing cloud gateway routing strategy is difficult to cooperatively optimize multiple conflicting performance indicators in a dynamic network environment. The core of the method is: modeling the routing decision process as a Markov decision process, designing a feature space that integrates multi-dimensional network states, and defining routing selection as an action space; constructing a composite reward function that simultaneously considers delay, bandwidth utilization and routing stability; using a proximal policy optimization algorithm to train the policy network so that it can output the probability distribution of the optimal routing action according to the real-time network state. The application realizes the adaptive and multi-objective cooperative optimization of the cloud gateway routing strategy, and improves the overall network performance and stability.

Description

Technical Field

[0001] This invention relates to the field of cloud computing and network communication technology, specifically to a multi-objective optimization method for cloud gateway routing based on reinforcement learning. Background Technology

[0002] With the rapid development of cloud computing and big data, cloud gateways, as the core hub connecting different cloud environments and cloud-local networks, directly determine the overall operational efficiency and service quality of cloud networks through the performance of their routing strategies. Currently, cloud networks face challenges such as dynamically changing traffic, complex link states, and diversified business requirements. Traditional cloud gateway routing strategies (such as OSPF and ECMP) still have many shortcomings and are difficult to adapt to complex network environments.

[0003] The main shortcomings of existing technologies are as follows: First, they have a singular objective. Traditional routing strategies are mostly based on a single metric (such as hop count or latency) for decision-making, which cannot simultaneously optimize conflicting performance indicators such as latency, bandwidth utilization, and load balancing. This can easily lead to overload of some links and low utilization of network resources. Second, they lack adaptability. The policy parameters are statically configured and cannot be adjusted in real time according to dynamic disturbances such as network traffic bursts and link failures. This results in delayed response and is prone to causing network congestion. Third, they have poor routing stability. There is no dedicated constraint mechanism for route switching, which can easily lead to frequent route oscillations, increasing network overhead and affecting the stability of upper-layer services. Fourth, some reinforcement learning-based routing schemes are only applicable to simple discrete action spaces. The algorithms used (such as DQN) have poor training stability and have not been combined with the actual deployment requirements of cloud gateways, so they are still far from production applications. Summary of the Invention

[0004] This invention proposes a multi-objective optimization method for cloud gateway routing based on reinforcement learning, to address the problem that existing technologies cannot collaboratively optimize multiple performance indicators in dynamic cloud network environments and may lead to routing instability. The technical solution provided by this invention is as follows:

[0005] A multi-objective optimization method for cloud gateway routing based on reinforcement learning includes the following steps:

[0006] Step 1: Construct a Markov decision process model for cloud gateway routing optimization, defining a state space representing the real-time operating state of the network, an action space representing routing selection behavior, and a multi-objective composite reward function including latency optimization, bandwidth utilization, load balancing, and routing stability.

[0007] Step 2: Use the near-end policy optimization algorithm to train the policy network and the value network. The policy network is used to output the probability distribution of routing actions based on the network state.

[0008] Step 3: Deploy the trained policy network on the cloud gateway control plane, collect network status in real time, select routing actions based on the probability distribution output by the policy network, and generate and distribute routing decisions.

[0009] Preferably, the state space is: the state at time t. The high-dimensional feature vector obtained by the vector concatenation operation Concat is represented as: ;in, Let M be the real-time load rate vector of the M links; This is a vector representing the current queue lengths of the N output ports. Let P be the average end-to-end delay vectors of the P preset paths within the nearest time window T; Let M be the bandwidth utilization vector for the M links.

[0010] Preferably, the action space is: for a data flow arriving at the cloud gateway with a destination address prefix of dst, from all its available next-hop sets. Choose an exit as the action. .

[0011] The preferred multi-objective composite reward function is:

[0012] ;

[0013] in, The preset weighting coefficients satisfy... And the range of values ​​for each weighting coefficient is... This is used to adjust the priority of different optimization objectives;

[0014] To optimize the reward items for latency, To perform the action The path selected later End-to-end delay at any given moment This is the preset baseline delay;

[0015] This is a reward item for balanced bandwidth utilization. for The average bandwidth utilization of all links at any given time. for The standard deviation of bandwidth utilization of all links at any given time. To avoid the minimum value where the denominator is 0;

[0016] This is a congestion penalty item. for The maximum load rate of all links at any given time. For the preset congestion threshold, when Exceed When the overload level is high, negative rewards are generated, and the severity of the punishment increases with the degree of overload; conversely, there is no punishment when the overload level is low.

[0017] This is a penalty for routing stability. For indicator functions, when the current action Actions at the previous moment At the same time, This results in a negative reward; when the route is not switched, No punishment.

[0018] Preferably, step 2 trains the total loss function. This includes the policy network optimization objective function, the value function error term, and the policy entropy regularization term:

[0019] ;

[0020] in, Optimize the objective function for the policy network. , For coefficients, For value function, For actual returns, For strategy In state Entropy below.

[0021] Preferably, the policy network optimization objective function for:

[0022] ;

[0023] in, As expected, The probability ratio between the old and new strategies. For the estimation of the advantage function, This is for pruning hyperparameters.

[0024] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: Through multi-dimensional design of the state space, the real-time operating status of the network is comprehensively perceived; through multi-dimensional design of the composite reward function, objectives such as latency optimization, utilization balancing, congestion avoidance, and stability assurance are transformed into learnable reward signals for the agent; a near-end policy optimization algorithm is used to train the policy network, ensuring the stability and convergence of the algorithm, enabling the agent to continuously learn the optimal routing strategy from interactions with the environment. By deploying the trained policy network on the cloud gateway control plane, and dynamically selecting the optimal exit from all available next-hop sets when generating specific actions, no manual intervention is required. Attached Figure Description

[0025] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0026] Figure 1 This is the main flowchart of the method of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] To make the above-mentioned objectives, features and effects of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Example 1: A multi-objective optimization method for cloud gateway routing based on reinforcement learning, such as... Figure 1 As shown, it includes the following steps:

[0030] Step 1: Construct a Markov decision process model for cloud gateway routing optimization, defining the state space S representing the real-time operating state of the network, the action space A representing the routing selection behavior, and the multi-objective composite reward function R for collaborative optimization of latency, bandwidth utilization, load balancing, and routing stability.

[0031] Step 1.1, define the state space S representing the real-time operating state of the network, and the state at time t. The high-dimensional feature vector obtained through the vector concatenation operation Concat is specifically represented as follows: ;in, The real-time load rate vector of M links (dimension: M represents the total number of links managed by the cloud gateway, and the load factor is the ratio of the current link traffic to the link bandwidth. The current queue length vector for N output ports (dimension 1) (where N is the total number of output ports of the cloud gateway). The average end-to-end delay vector (with dimension 1) for P preset paths within the most recent time window T. P represents the total number of preset paths from the cloud gateway to each target network, and T represents the preset latency statistics window, ranging from 30s to 120s. The bandwidth utilization vector of M links (dimension 1) ,and One-to-one correspondence, representing the actual proportion of link bandwidth occupied.

[0032] Step 1.2: Define the action space A representing the routing behavior. For a data flow arriving at the cloud gateway with a destination address prefix of dst, select from all available next-hop sets. Choose an exit as the action. .

[0033] Step 1.3, define the multi-objective composite reward function R. Calculated using the following formula:

[0034] ;

[0035] in, The preset weighting coefficients satisfy... And the range of values ​​for each weighting coefficient is... This is used to adjust the priority of different optimization objectives;

[0036] To optimize the reward items for latency, To perform the action The path selected later End-to-end delay at any given moment The preset baseline delay (taken as the median of the historical average delay of all preset paths) is used when Less than hour, A positive reward is given if the conditions are met, and a negative reward is given if the conditions are not met.

[0037] This is a reward item for balanced bandwidth utilization. for The average bandwidth utilization of all links at any given time. for The standard deviation of bandwidth utilization of all links at any given time. To avoid extremely small values ​​where the denominator is 0 (the range of values ​​is...) The larger the value of this reward item, the more balanced the link bandwidth utilization.

[0038] This is a congestion penalty item. for The maximum load rate of all links at any given time. The preset congestion threshold (the value range is...) ),when Exceed When the overload level is high, negative rewards are generated, and the severity of the punishment increases with the degree of overload; conversely, there is no punishment when the overload level is low.

[0039] This is a penalty for routing stability. For indicator functions, when the current action Actions at the previous moment When different (i.e., when switching routes), This results in a negative reward; when the route is not switched, No punishment.

[0040] Step 2: Train the policy network using the proximal policy optimization algorithm. and value network The objective function for training the policy network. for:

[0041] ;

[0042] in, As expected, The probability ratio between the old and new strategies. For the estimation of the advantage function, For pruning hyperparameters; the total loss function for training. It also includes the value function error term and the policy entropy regularization term:

[0043] ;in, , For coefficients, For actual returns, For strategy In state Entropy below.

[0044] Step 3: Deploy the trained policy network on the cloud gateway control plane and collect network status data in real time. And based on the probability distribution output by the policy network Select route action It generates and distributes routing decisions.

[0045] This embodiment is a complete implementation of the method described in the invention, applied to enterprise-level multi-cloud interconnection cloud gateway scenarios, and adapted to cloud gateway devices with software-defined networking (SDN) architecture. It achieves multi-objective collaborative optimization of latency, bandwidth utilization, load balancing, and routing stability during the routing decision-making process. The specific implementation process is as follows:

[0046] In this embodiment, the cloud gateway is deployed in the enterprise headquarters data center, serving as the core forwarding hub connecting the enterprise's local private cloud, public cloud A, and public cloud B. It adopts an SDN-separated architecture, with the control plane responsible for generating and distributing routing decisions, and the data plane responsible for line-speed traffic forwarding. The basic network and algorithm core parameters are configured as follows:

[0047] Network topology parameters: The total number of physical links managed by the cloud gateway is M=8 (4 to public cloud A leased lines and 4 to public cloud B leased lines, each with a bandwidth of 10Gbps); the total number of output ports of the cloud gateway is N=8, corresponding one-to-one with the 8 links; the total number of preset available paths from the cloud gateway to each target network is P=12 (6 to public cloud A and 6 to public cloud B).

[0048] Time and threshold parameters: Delay statistics time window T=60s; Link congestion threshold τ=0.85; Avoid minimum values ​​where the denominator is 0. = 4; PPO algorithm for pruning hyperparameters =0.2; coefficient of error term in value function =0.5; Policy entropy regularization coefficient =0.01.

[0049] Reward function weight parameters: Preset weight coefficients satisfy Specifically, this refers to: latency optimization weights =0.35, bandwidth utilization balancing weight =0.25, congestion avoidance weight =0.25, route stability weight =0.15, all weight values ​​are within the range of (0,1).

[0050] Deployment and training environment: The policy network and value network were trained using the Python + PyTorch framework, and the simulation environment was built on the NS-3 network simulation platform. The trained model was deployed on an x86 server (CPU: Intel Xeon 8375C, memory: 32GB) in the cloud gateway control plane, and the decision-making was carried out using the OpenFlow 1.3 protocol.

[0051] Constructing a Markov Decision Process (MDP) model for cloud gateway routing optimization: This step models the real-time routing decision process of the cloud gateway as a discrete-time Markov decision process, completing the definition and implementation of the state space S, action space A, and multi-objective composite reward function R, providing an interactive environment for reinforcement learning agents.

[0052] The state of state space S at time t , is the high-dimensional feature vector obtained through the vector concatenation operation Concat, specifically expressed as: The rules for real-time acquisition, calculation, and implementation of each feature vector are as follows:

[0053] Link Real-Time Load Rate Vector The system has an 8×1 dimension and collects the current traffic of 8 links in real time with a period of 100ms. The load rate of a single link is calculated as: current traffic of the link / rated bandwidth of the link (10Gbps). The actual measured data at time t is... .

[0054] Output port queue length vector The dimension is 8×1, and the current data packet queue length (unit: KB) of 8 output ports is collected synchronously. The actual measurement at time t is obtained. .

[0055] Path average end-to-end delay vector The dimension is 12×1. The average end-to-end latency (in milliseconds) of 12 preset paths within the most recent 60-second time window is calculated. The results are obtained at time t. .

[0056] Link bandwidth utilization vector : Dimensions are 8×1, and One-to-one correspondence, real-time collection of the actual bandwidth utilization ratio of 8 links, measured at time t. .

[0057] By concatenating the dimensions of the above four vectors, we obtain a 36-dimensional high-dimensional state feature vector. As input to the policy network, it enables a full-dimensional perception of the network's operational status.

[0058] The core definition of action space A is: for a data stream arriving at the cloud gateway with a destination address prefix of dst, from all its available next-hop sets. In the process, select an exit as the routing action at time t. .

[0059] In this embodiment, the action space landing rules for different target networks are as follows:

[0060] Data streams with target address prefixes in the public cloud A network segment can use the next-hop set. This corresponds to 4 dedicated line links to public cloud A;

[0061] Data streams with target address prefixes in the public cloud B network segment can use the next-hop set. This corresponds to 4 dedicated line links to public cloud B;

[0062] Data streams whose target address prefix is ​​an enterprise private cloud network segment can use the next-hop set. This corresponds to two local core switch exits.

[0063] Routing actions The final choice is determined by the action probability distribution output by the policy network, realizing adaptive routing decision based on real-time network status.

[0064] The multi-objective composite reward function is the core guiding principle for agent learning. In this embodiment, the calculation formula for the reward function is as follows: The specific calculation and implementation rules for each item are as follows:

[0065] Latency optimization reward items The calculation formula is: The reference delay In this embodiment, the median historical average latency of 12 preset paths is used. =22ms. If the action is executed. Then, the end-to-end delay of the selected path at time t+1 is 19ms (less than the baseline delay), then... ≈0.146, a positive reward is obtained; if the latency is 25ms (greater than the baseline latency), then ≈-0.128, resulting in a negative reward.

[0066] Bandwidth utilization equalization incentive The calculation formula is: If the measured link bandwidth utilization at time t+1 Calculated ≈0.546, ≈0.098, therefore ≈5.56, the larger the value, the more balanced the link bandwidth utilization.

[0067] Congestion penalty items The calculation formula is: If the maximum load rate of all links at time t+1 is 0.72 (less than the congestion threshold of 0.85), then =0, no congestion penalty; if the maximum load rate is 0.92 (exceeding the threshold), then =-0.07, the higher the overload level, the greater the negative penalty.

[0068] Routing stability penalty The calculation formula is: in This is an indicator function. If the current action... Actions at the previous moment If consistent (no route switching), then 0, No penalty; if a route switch occurs, then... , This generates a fixed negative penalty, preventing frequent routing oscillations.

[0069] Based on the above calculation results, the action to be performed at time t can be obtained. The resulting composite reward value R serves as the core optimization objective for updating the parameters of the PPO algorithm.

[0070] Based on the MDP model constructed above, the Proximal Policy Optimization (PPO) algorithm is used to complete the policy network. and value network The training of the policy network, where the policy network is used to adjust the network state. The output routing action probability distribution is used to evaluate the value of the current state and assist in the calculation of the advantage function.

[0071] Policy Network: Employs a 3-layer fully connected neural network with an input layer dimension of 36 (compared to the state vector). (The dimensions are consistent). Hidden layer 1 has a dimension of 128, hidden layer 2 has a dimension of 64, the activation function of the hidden layers is ReLU, the output layer uses the Softmax activation function, and the output corresponds to the probability distribution of the next available action.

[0072] Value Network: It adopts a 3-layer fully connected neural network with an input layer dimension of 36, a hidden layer 1 dimension of 128, a hidden layer 2 dimension of 64, and the activation function of the hidden layers is ReLU. The output layer is 1-dimensional and outputs the value estimate of the current state.

[0073] Complete model training process:

[0074] Initialization configuration: Randomly initialize policy network parameters and value network parameters The total number of training iterations was set to 2000, with 1024 steps per iteration for trajectory acquisition and a batch size of 64. The optimizer used was Adam, and the learning rate was set to 3e-4. A discount factor was also specified. =0.99, GAE generalized advantage estimation parameter λ=0.95.

[0075] Experience trajectory acquisition: In the NS-3 network simulation environment, simulate the real traffic characteristics of an enterprise (including normal traffic and burst traffic), allowing the policy network to interact with the network environment in real time, and collect the data at each step. , , , The empirical trajectory dataset is constructed using the quintuple data of 'done', where 'done' is a boolean value (True / False) used to mark whether the current step is the termination step of the trajectory (e.g., done=True when the simulation environment is reset or the traffic scenario is switched, otherwise done=False).

[0076] Advantage function and payoff calculation: Based on the collected empirical trajectories, the generalized advantage estimation (GAE) method is used to calculate the advantage function at each step. Meanwhile, the actual return for each step is calculated using the Monte Carlo method. , serving as a monitoring signal for value network updates.

[0077] Network parameter update: Based on the pruning objective function of the PPO algorithm, the total training loss function is calculated as follows:

[0078] ;

[0079] The core of the pruning objective function is to limit the update range of the old and new policies to avoid abrupt policy changes, as shown in the following formula.

[0080] ;

[0081] In the formula Given the ratio of action probabilities between the old and new policies, the total loss function is minimized using the Adam optimizer to complete the iterative update of parameters in the policy network and value network.

[0082] Convergence Judgment and Model Saving: In the repeated trajectory acquisition-parameter update process, when the average cumulative reward fluctuation of 50 consecutive training rounds is less than 5% and the reward value tends to be stable, the model is judged to have converged, training is stopped, and the converged policy network weight parameters are saved for subsequent online deployment.

[0083] The trained and converged policy network is deployed on the SDN control plane of the cloud gateway to achieve real-time, adaptive routing decision generation and distribution. The complete execution process is as follows:

[0084] Real-time network status acquisition: The cloud gateway control plane collects link load rate, port queue length, path end-to-end latency, and link bandwidth utilization data from the forwarding devices in the data plane in real time, with a period of 100ms. It then performs feature cleaning and vector concatenation according to the definition of the state space to generate a real-time state feature vector. .

[0085] Online reasoning in policy networks: real-time state The trained policy network is input, and through forward propagation, the probability distribution of the next-hop action available for the corresponding target address is output. The inference time for a single data stream is less than 1ms, which meets the latency requirements for real-time routing decisions.

[0086] Routing action selection and decision-making: Based on the probability distribution output by the policy network, a greedy strategy is adopted in normal scenarios to select the action with the highest probability as the final routing action. In scenarios involving sudden traffic bursts, an ε-greedy sampling strategy is adopted, retaining a 10% exploration probability to adapt to sudden network changes. After determining the action, the corresponding OpenFlow flow table rules are generated and sent to the forwarding devices in the cloud gateway data plane to complete the route configuration update.

[0087] Continuous closed-loop optimization: Repeated state collection - online inference - decision distribution process to continuously adapt to real-time changes in network status; at the same time, continuously collect reward data of online interaction, and make incremental fine-tuning of the policy network based on the newly added online traffic data on a 24-hour cycle to adapt to long-term changes in the network environment and achieve continuous optimization of routing policies.

[0088] This embodiment compares the method of the present invention with traditional ECMP routing strategies, OSPF shortest path routing strategies, and DQN-based single-objective reinforcement learning routing strategies through simulation and physical environment tests. The core performance test results are as follows:

[0089] End-to-end latency performance: Under normal network traffic load of 70%, the average end-to-end latency of this method is 18.6ms, which is 32.1% lower than ECMP strategy and 24.4% lower than OSPF shortest path strategy; under traffic burst scenarios, the latency jitter is controlled within 5ms, which is far better than traditional strategies.

[0090] Bandwidth utilization and load balancing: The average bandwidth utilization of the entire network links in this method reaches 78.2%, which is 26.5% higher than that of the ECMP strategy. The standard deviation of the link bandwidth utilization is controlled within 0.08, with no single link overload, which completely solves the resource waste problem of "partial link congestion and partial link idleness" that is easy to occur in traditional strategies.

[0091] Congestion control capability: In sudden scenarios where the peak traffic exceeds the baseline traffic by 50%, the network congestion rate of this method is 2.1%, which is 89.3% lower than that of traditional strategies. There are no congestion situations lasting more than 3 seconds, effectively ensuring the quality of service.

[0092] Routing stability: The average route switching frequency of this method is 0.8 times / minute, which is 76.5% lower than the DQN routing strategy without stability constraints. There is no route oscillation phenomenon, which greatly reduces the network overhead and service interruption risk caused by route switching.

[0093] In summary, this embodiment fully implements the reinforcement learning-based multi-objective optimization method for cloud gateway routing described in this invention, achieving adaptive and multi-objective collaborative optimization of cloud gateway routing strategies, and effectively improving the overall operational efficiency and stability of the cloud network.

[0094] Example 2: The computer-readable storage medium of this example stores a computer program that, when executed by a processor, implements the steps of the reinforcement learning-based multi-objective optimization method for cloud gateway routing in Example 1.

[0095] The computer-readable storage medium in this embodiment can be an internal storage unit of the terminal, such as the terminal's hard disk or memory; the computer-readable storage medium in this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. equipped on the terminal; furthermore, the computer-readable storage medium can include both the terminal's internal storage unit and external storage devices.

[0096] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0097] Example 2: The computer device of this example includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the reinforcement learning-based cloud gateway routing multi-objective optimization method of Example 1.

[0098] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The memory can include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0099] Those skilled in the art will clearly understand that each implementation can be achieved using software plus the necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0100] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-objective optimization method for cloud gateway routing based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Construct a Markov decision process model for cloud gateway routing optimization, defining a state space representing the real-time operating state of the network, an action space representing routing selection behavior, and a multi-objective composite reward function including latency optimization, bandwidth utilization, load balancing, and routing stability. Step 2: Use the near-end policy optimization algorithm to train the policy network and the value network. The policy network is used to output the probability distribution of routing actions based on the network state. Step 3: Deploy the trained policy network on the cloud gateway control plane, collect network status in real time, select routing actions based on the probability distribution output by the policy network, and generate and distribute routing decisions.

2. The multi-objective optimization method for cloud gateway routing based on reinforcement learning according to claim 1, characterized in that, The state space is: the state at time t. The high-dimensional feature vector obtained by the vector concatenation operation Concat is represented as: ;in, Let M be the real-time load rate vector of the links; This is a vector representing the current queue lengths of the N output ports. Let P be the average end-to-end delay vector of the P preset paths within the nearest time window T; Let M be the bandwidth utilization vector for the M links.

3. The multi-objective optimization method for cloud gateway routing based on reinforcement learning according to claim 2, characterized in that, The action space is: for a data flow arriving at the cloud gateway with a destination address prefix of dst, from all its available next-hop sets. Choose an exit as the action. .

4. The multi-objective optimization method for cloud gateway routing based on reinforcement learning according to claim 3, characterized in that, The multi-objective composite reward function is: ； in, The preset weighting coefficients satisfy... And the range of values ​​for each weighting coefficient is 100%. This is used to adjust the priority of different optimization objectives; To optimize the reward items for latency, To perform the action The path selected later End-to-end delay at any given moment The preset baseline delay; This is a reward item for balanced bandwidth utilization. for The average bandwidth utilization of all links at any given time. for The standard deviation of bandwidth utilization of all links at any given time. To avoid the minimum value where the denominator is 0; This is a congestion penalty item. for The maximum load rate of all links at any given time. For the preset congestion threshold, when Exceed When the overload level is high, negative rewards are generated, and the severity of the punishment increases with the degree of overload; conversely, there is no punishment when the overload level is low. This is a penalty for routing stability. For indicator functions, when the current action Actions at the previous moment At the same time, This results in a negative reward; when the route is not switched, No punishment.

5. The multi-objective optimization method for cloud gateway routing based on reinforcement learning according to claim 1, characterized in that, Step 2: Train the total loss function This includes the policy network optimization objective function, the value function error term, and the policy entropy regularization term: ； in, Optimize the objective function for the policy network. , For coefficients, For value function, For actual returns, For strategy In state Entropy below.

6. The multi-objective optimization method for cloud gateway routing based on reinforcement learning according to claim 5, characterized in that, Policy network optimization objective function for: ； in, For the expectation, The probability ratio between the old and new strategies. For the estimation of the advantage function, This is for pruning hyperparameters.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the reinforcement learning-based multi-objective optimization method for cloud gateway routing as described in any one of claims 1-6.

8. A computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multi-objective optimization method for cloud gateway routing based on reinforcement learning as described in any one of claims 1-6.

Images