A method for optimizing cloud-edge computing network routing

Abstract

The present invention discloses a cloud-edge computing network routing optimization method in the field of computing network technology, including obtaining a high-dimensional computing node embedding representation and an edge embedding representation of the network topology structure of the cloud-edge computing network; obtaining a hidden state set of each link in the network topology structure when transmitting messages between computing nodes; predicting the future traffic of each link in the network topology structure based on the high-dimensional computing node embedding representation, edge embedding representation and hidden state set, and obtaining the state information of each link; converting a pre-constructed routing scheduling problem into a Markov decision process; based on the Markov decision process, using a multi-agent proximal strategy optimization algorithm to train the agent to obtain the optimal routing selection strategy; wherein the routing scheduling problem is constructed and obtained with the goal of minimizing the standard deviation of link utilization and task transmission time. The present invention solves the problem of low efficiency and poor flexibility of routing optimization in existing cloud-edge computing networks.

Description

Technical Field

[0001] The present invention relates to the field of computing power network technology, and in particular to a cloud-edge computing power network routing optimization method. Background Art

[0002] In recent years, with the rapid development of emerging technologies such as 5G, 6G networks, edge computing, the Internet of Things, and artificial intelligence, traditional cloud computing architecture has been unable to adapt to the stringent service requirements of applications for real-time network transmission, high bandwidth, low latency, etc. At the same time, the rise of large-scale model technologies such as ChatGPT and Sora has led to a sharp increase in demand for computing power. With the development of computing power and the popularization of new network technologies, computing power resources are showing a ubiquitous deployment state.

[0003] In order to solve these problems, the concept of computing power network came into being. The computing power network integrates the idle resources of edge devices to build a distributed multi-domain computing network, and provides service-based resource scheduling and service strategies on this basis, aiming to maximize the computing power of edge devices and realize resource sharing and optimal utilization.

[0004] However, in the face of an increasingly complex and dynamically changing network environment, how to efficiently schedule appropriate computing nodes to complete tasks has become a new challenge. In this context, routing optimization has become a key technology. Routing optimization can effectively manage the transmission path of data packets in a distributed network, which can not only maximize network performance but also improve resource utilization.

[0005] Existing routing optimization methods often rely on static routing tables, which cannot adapt to the dynamic changes in the network environment and the diverse application requirements. Once these static routing optimization strategies are deployed in actual networks, they often fail to achieve the desired performance results. In addition, most traditional optimization problems focus on minimizing the maximum link load. Such a design ignores the utilization of all links in the overall network and has certain limitations.

[0006] Although deep reinforcement learning methods provide new solutions for routing optimization. However, when applying deep reinforcement learning, existing routing optimization methods usually simply train the traffic-related data obtained from the data plane and control plane without fully considering the problems that may be caused by the dynamic changes in network topology. Therefore, existing routing optimization methods have certain limitations in application scenarios. Summary of the invention

[0007] The technical problem to be solved is: the routing optimization efficiency and poor flexibility in the existing cloud-edge computing network.

[0008] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0009] The present invention provides a cloud-edge computing network routing optimization method, comprising:

[0010] Obtain the network topology of the cloud-edge computing network;

[0011] Obtaining a high-dimensional computing power node embedding representation and an edge embedding representation of the network topology structure;

[0012] When messages are transmitted between computing nodes, the hidden state set of each link in the network topology is obtained;

[0013] Based on the high-dimensional computing power node embedding representation, edge embedding representation and hidden state set, the future traffic of each link in the network topology is predicted, and the state information of each link in the network topology is obtained;

[0014] The computing nodes of the network topology are used as intelligent agents, and the pre-constructed routing scheduling problem is converted into a Markov decision process according to the status information of each link in the network topology.

[0015] Based on the Markov decision process, a multi-agent proximal strategy optimization algorithm is used to train the agent to obtain the optimal routing strategy;

[0016] The routing scheduling problem is constructed with the goal of minimizing the link utilization standard deviation and the task transmission time.

[0017] Furthermore, the obtaining of the network topology structure of the cloud-edge computing network includes:

[0018] The network architecture of the cloud-edge computing network is defined to include a knowledge plane, a control plane, a management plane, and a data plane, and a weighted graph data structure is used to represent the network architecture of the cloud-edge computing network to form a network topology structure;

[0019] The knowledge plane is responsible for learning network behavior, receiving and processing network topology information collected by the management plane, converting the collected network topology information into network status knowledge through the Markov decision process, using the network status knowledge to formulate a routing optimization strategy for the cloud-edge computing network, and sending the routing optimization strategy to the control plane;

[0020] The control plane is responsible for collecting traffic information of the forwarding device to construct the view of the data plane, and converting the routing optimization policy sent by the knowledge plane into flow table rules, and sending the flow table rules to the forwarding device of the data plane;

[0021] The data plane is composed of a forwarding device and a link between the forwarding devices, and is responsible for controlling the forwarding rules of the traffic based on the flow table rules issued by the control plane and forwarding the traffic to the forwarding device for forwarding;

[0022] The management plane is responsible for storing basic information from the data plane and the control plane, and periodically providing global information about the network to the knowledge plane, wherein the global information includes network topology information.

[0023] Furthermore, the network topology structure is a directed graph.

[0024] Furthermore, the obtaining of a high-dimensional computing power node embedding representation and an edge embedding representation of the network topology structure includes:

[0025] Using a graph neural network to map the features of the computing nodes and edges of the network topology structure into a high-dimensional space; wherein the computing nodes represent the computing resources of the cloud-edge computing network, and the edges represent the network links of the cloud-edge computing network;

[0026] In high-dimensional space, computing nodes and edges receive information from neighboring nodes through a multi-layer message passing mechanism, and update their own computing node embedding representation and edge embedding representation;

[0027] After several rounds of information transmission, high-dimensional embedding representations of computing power nodes and edges are obtained.

[0028] Furthermore, when the message is transmitted between the computing nodes, the hidden state set of each link in the network topology structure is obtained, including:

[0029] Receive the hidden state of neighboring nodes after message exchange;

[0030] After combining the received hidden state with its own hidden state, the combined information is obtained;

[0031] Aggregate all the information after the combination processing to obtain aggregate information;

[0032] updating and calculating a new hidden state based on the received hidden state and the aggregate information;

[0033] After message transmission between computing nodes is completed, the characteristic vector of the network topology structure and the attributes of each computing node are predicted;

[0034] The predicted feature vector of the network topology structure and the attributes of each computing power node are used as a link hidden state set of the network topology structure.

[0035] Furthermore, the hidden states in the link hidden state set of the network topology structure are defined by the feature vectors of the corresponding links; when the dimension of the hidden state is not sufficient to represent all link features in the network topology structure, the feature vectors are completed by zero filling.

[0036] Furthermore, the routing scheduling problem is expressed as:

[0037] ;

[0038] In the formula, represents the weight of the standard deviation of link utilization, represents the weight of the task transmission time, represents the standard deviation of link utilization, Indicates j The task transmission time of each path, Indicates j path.

[0039] Furthermore, the computing nodes of the network topology structure are used as intelligent agents, and the pre-constructed routing scheduling problem is converted into a Markov decision process according to the state information of each link in the network topology structure, including:

[0040] Obtaining initial link characteristics and traffic requirements of each link, and constructing a discrete state space, wherein the state space includes all states of the agent;

[0041] Obtaining weight data of each link, and constructing an action space based on the weight data of each link of the network topology structure, wherein the action space contains all actions selected by the agent in a given state;

[0042] Define a reward value function according to the link utilization standard deviation and task completion time of the pre-constructed routing scheduling problem, wherein the reward value function includes the cumulative reward of the agent from the initial state to the terminal state;

[0043] Constructing a Markov decision process according to the state space, the action space and the reward value function;

[0044] In the Markov decision process, the agent selects an action according to the current state, and obtains a reward according to the selected action and state change, and continuously iterates and learns with the goal of maximizing the reward value.

[0045] Furthermore, based on the Markov decision process, a multi-agent proximal strategy optimization algorithm is used to train the agent to obtain an optimal routing selection strategy, including:

[0046] In the state space, an initial state of the network topology is obtained as an initial observation of each agent; a policy network of each agent is initialized with the goal of minimizing the standard deviation of link utilization and task transmission time, and the policy network is used to select an action in the action space according to the observation of the agent;

[0047] Repeat the following steps until the policy network converges to obtain an agent that can generate the optimal routing strategy, thereby obtaining the optimal routing strategy:

[0048] Each agent selects an action based on the current observation and policy network, and after executing the selected action, observes and records the new state of the network topology;

[0049] Calculate the reward value of each agent for performing an action according to the new state of the network topology and the reward value function;

[0050] Based on the reward value and the new state of the network topology, a proximal policy optimization algorithm is used to update the policy network of each agent.

[0051] Furthermore, the multi-agent proximal strategy optimization algorithm includes:

[0052] The loss function of the multi-agent proximal strategy optimization algorithm is constructed based on the loss value of the Actor network and the loss value of the Critic network; the Actor network and the Critic network output action values ​​and state values ​​respectively;

[0053] Using the loss function of the multi-agent proximal strategy optimization algorithm, back-propagation updates the neural network parameters, and through continuous iterative training, obtains an agent that can generate the optimal routing selection strategy;

[0054] The loss function of the multi-agent proximal strategy optimization algorithm is expressed as:

[0055] ;

[0056] In the formula, is the total loss value of the multi-agent proximal policy optimization algorithm, is the loss value of the Actor network, is the loss value of the Critic network The coefficient of Reward for Entropy The coefficient of are neural network parameters, for The state of the network topology at any moment, Select the best routing strategy.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The present invention transforms the routing scheduling problem into a Markov decision process. By treating the computing power nodes as intelligent agents and using the state information of each link in the network topology as input, the Markov decision process can systematically describe the decision-making process of the intelligent agent in different states, providing a theoretical framework for finding the optimal routing strategy. At the same time, it also combines the advantages of deep reinforcement learning and multi-agent systems, and can effectively solve the routing scheduling problem in a complex network environment. By using a multi-agent proximal strategy optimization algorithm to train the intelligent agent, it learns to choose the optimal action, that is, routing decision, under a given state, thereby obtaining the optimal routing selection strategy.

[0059] The present invention constructs the routing scheduling problem with the goal of minimizing the standard deviation of link utilization and task transmission time, which can ensure that network resources are used evenly and reduce task transmission time. This multi-objective joint optimization method helps to improve the overall performance of the network and user experience. At the same time, by clarifying the optimization goals, algorithms and models can be designed more specifically to improve solution efficiency and quality.

[0060] The present invention improves the learning efficiency and generalization performance of the multi-agent proximal strategy optimization algorithm, so that the multi-agent deep reinforcement learning method can effectively adapt to more complex network environments, and provide a stable and efficient routing scheduling solution for user services, effectively improving the robustness of the cloud-edge computing network routing optimization method, and is suitable for large-scale complex networks. Once a cloud-edge computing network link failure is detected or a link is overloaded, other links will be given priority and rerouting operations will be automatically performed. It can effectively reduce manual intervention and improve the elasticity of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of a process flow of a cloud-edge computing network routing optimization method provided by an embodiment of the present invention;

[0062] Figure 2 A schematic diagram of a domain controller network topology of a cloud-edge computing network provided in an embodiment of the present invention;

[0063] Figure 3 A schematic diagram of the network structure of a cloud-edge computing network routing optimization method provided by an embodiment of the present invention;

[0064] Figure 4 Schematic diagram of the training process of the multi-agent proximal strategy optimization algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0066] The core of the present invention is to provide a cloud-edge computing network routing optimization method to solve the problems of link load balancing and task transmission delay in the cloud-edge computing network. In order to enable people in the technical field to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0067] Example 1

[0068] like Figure 1 As shown, this embodiment provides a cloud edge computing network routing optimization method, including the following steps:

[0069] Step 1: Obtain the network topology of the cloud-edge computing network.

[0070] By obtaining the network topology, a global view of the entire cloud-edge computing network can be obtained, including the scale, structure, node distribution, and link characteristics of the cloud-edge computing network.

[0071] Step 2: Obtain a high-dimensional computing node embedding representation and an edge embedding representation of the network topology structure.

[0072] Through graph neural networks, the nodes and edges in the network topology are mapped to high-dimensional space to form high-dimensional computing node embedding representations and edge embedding representations, which can capture the complex relationships and hidden features between nodes and edges. High-dimensional computing node embedding representations and edge embedding representations provide richer information than the original network topology structure and also improve the generalization ability of the algorithm, enabling it to adapt to networks of different sizes and structures.

[0073] Step 3: When transmitting messages between computing nodes, obtain the hidden state set of each link in the network topology.

[0074] Through the message passing mechanism, such as the message passing neural network in the neural network in the figure, information is transmitted between computing nodes, and the hidden state set of each link is calculated. The hidden state can reflect the internal state information of the cloud-edge computing network during dynamic changes. At the same time, the hidden state set provides important input data for traffic prediction. By capturing the real-time state information of the network, the future traffic of the network topology can be predicted more accurately, thereby formulating a more reasonable routing strategy.

[0075] Step 4: Based on the high-dimensional computing power node embedding representation, edge embedding representation and hidden state set, predict the future traffic of each link in the network topology structure and obtain the state information of each link in the network topology structure.

[0076] By using machine learning or deep learning models, such as the gated recurrent unit (GRU), and predicting future traffic, we can perceive the changing trend of cloud-edge computing network traffic in advance, provide a decision-making basis for routing scheduling, thereby avoiding network congestion and improving the overall performance of the network.

[0077] Step 5: Take the computing nodes of the network topology as intelligent agents, and transform the pre-constructed routing scheduling problem into a Markov decision process based on the status information of each link in the network topology.

[0078] Through Markov decision process modeling, the complex routing scheduling problem is simplified into a series of decision problems, which is easy to solve.

[0079] Step 6: Based on the Markov decision process, a multi-agent proximal strategy optimization algorithm is used to train the agent to obtain the optimal routing selection strategy.

[0080] The multi-agent proximal strategy optimization algorithm is used to train the agent so that it can learn to choose the best action under a given state, that is, routing decision. By training the agent, the agent can have the ability to make autonomous decisions and adjust the routing strategy in real time according to the network status. It can effectively solve the routing scheduling problem in a complex network environment and obtain the optimal routing selection strategy.

[0081] The routing scheduling problem is constructed with the goal of minimizing the link utilization standard deviation and the task transmission time.

[0082] This embodiment clarifies that the optimization goal of the routing scheduling problem is to minimize the standard deviation of link utilization and task transmission time, which respectively reflect the balanced utilization of network resources and the level of transmission efficiency. By clarifying the optimization goal, the efficiency and quality of the solution are improved. At the same time, the optimization goal also guides the training process of the multi-agent proximal strategy optimization algorithm, so that the trained routing selection strategy is more in line with actual needs.

[0083] Example 2

[0084] The invention concept is the same as that of Example 1. This embodiment provides a cloud-edge computing network routing optimization method, which specifically includes the following steps:

[0085] Step 1: Obtain the network topology of the cloud-edge computing network.

[0086] The network architecture of the cloud-edge computing network is defined to include the knowledge plane, control plane, management plane, and data plane. The network architecture of the cloud-edge computing network is represented by a weighted graph data structure to form a network topology. The composition of the cloud-edge computing network is clarified, forming a clear network architecture model, which is convenient for understanding and analyzing the structure and function of the cloud-edge computing network.

[0087] The network structure of the cloud-edge computing network involved in this embodiment is a cross-domain computing network. Figure 2 As shown, there are five domains in the cloud-edge computing network. Each domain has a domain controller, multiple computing nodes, and links between domain controllers. There is also a cross-domain controller connecting each domain controller to form the entire cloud-edge computing network structure.

[0088] This embodiment also provides a network structure diagram of the cloud edge computing network routing optimization method, such as Figure 3 As shown in Figure 1, it includes the knowledge plane, control plane, data plane and management plane.

[0089] The knowledge plane includes a network traffic prediction module and a deep reinforcement learning module, which are responsible for learning network behavior, receiving and processing the network topology information collected by the management plane, converting the collected network topology information into network status knowledge through the Markov decision process, and using the northbound interface to transmit the network status knowledge to the control plane to formulate the routing optimization strategy of the cloud-edge computing network, and sending the routing optimization strategy to the control plane;

[0090] The control plane includes a network information perception module and a routing forwarding module, which are responsible for collecting traffic information of forwarding devices to build a view of the data plane, and converting the routing optimization strategy sent by the knowledge plane into flow table rules, and sending the flow table rules to the forwarding devices of the data plane;

[0091] The data plane includes a network information perception module and a routing forwarding module, which is composed of a forwarding device and a link between the forwarding devices, and is responsible for controlling the forwarding rules of the traffic based on the flow table rules issued by the control plane and forwarding the traffic to the forwarding device for forwarding;

[0092] The management plane includes a data preprocessing module, which is responsible for storing basic information from the data plane and the control plane, and regularly providing global information about the network to the knowledge plane, wherein the global information includes network topology information.

[0093] Step 2: Obtain a high-dimensional computing power node embedding representation and an edge embedding representation of the network topology structure, and obtain a hidden state set of each link in the network topology structure when transmitting messages between computing power nodes.

[0094] In this embodiment, the network topology is modeled by a weighted directed graph data structure, where the nodes in the directed graph represent the computing nodes in the cloud-edge computing network, and the edges represent the connections between the computing nodes. A Represents the connection between computing nodes, feature matrix X To represent the characteristics of each computing power node.

[0095] Using graph convolutional neural networks, a filter is designed in the Fourier domain to process computing nodes, so that computing nodes can pass through the first-order neighborhood to capture spatial features. Subsequently, by stacking multiple convolutional layers, the graph convolutional neural network can integrate and extract high-dimensional computing node embedding representations. The process of extracting high-dimensional computing node embedding representations is expressed as:

[0096] (1);

[0097] In the formula, For the l +1 layer of computing power node embedding representation, is the activation function, is the degree matrix, is the weight matrix, For the l The computing power nodes of the layer are embedded in the representation.

[0098] in, Represents the preprocessing process of embedding representation of high-dimensional computing nodes, where , represents the adjacency matrix with self-connection added, , is the adjacency matrix representing the connections between computing nodes, is the identity matrix.

[0099] From the process of extracting the embedded representation of high-dimensional computing power nodes, it can be seen that the message passing neural network and the gated recurrent unit can learn spatial features from the network topology structure. The gated recurrent unit obtains the embedded representation of high-dimensional computing power nodes and the spatial features of surrounding routers through convolution operations, thereby obtaining spatial dependencies.

[0100] The convolution operation of a layer of graph convolutional neural network can obtain the features of neighboring nodes, and the graph convolutional neural network can capture higher-order spatial features by stacking multiple convolutional layers.

[0101] The graph convolution operation of the two-layer graph convolutional neural network to capture spatial dependencies is expressed as:

[0102] (2);

[0103] In the formula, is the adjacency matrix A and the feature matrix X The graph convolution operation between is the activation function, is a linear rectification function, is the weight matrix from the input layer to the hidden layer, is the weight matrix from the hidden layer to the output layer.

[0104] In order to effectively capture the time dependency of each link information in the cloud-edge computing network, the present invention also adopts a gated recurrent neural unit architecture to effectively learn the dynamic changes of traffic data. Gated recurrent units are widely popular due to their high computational efficiency and small number of parameters. Compared with long short-term memory, gated recurrent units provide a simpler learning framework.

[0105] in, The hidden state at the moment is express, Network information at all times Indicates that the door is reset with Indicates that the control ignores the degree of information at the previous moment and updates the gate with The update gate acts like the forget gate and input gate in long short-term memory. The update gate determines what information needs to be forgotten and what information needs to be added. The memory content of the moment express, The hidden state at the moment is express.

[0106] Using the gated recurrent unit The hidden state at the moment and the current information are used as input to calculate In this way, the graph neural network model is able to preserve historical information and capture temporal dependencies while still incorporating information from the current moment.

[0107] Will Update door at all times It is expressed as:

[0108] (3);

[0109] Will Reset gate of time It is expressed as:

[0110] (4);

[0111] Will Neural network memory content at each moment It is expressed as:

[0112] (5);

[0113] Will Output status at the moment It is expressed as:

[0114] (6);

[0115] In the formula, is the activation function; for The hidden state of the moment, is the graph convolution operation, and To update the weights and biases during gate training; and To reset the weights and biases during gate training; Represents the activation function of the neural network, which is the hyperbolic tangent function. and They are the weight and bias of the neural network memory content during training.

[0116] In summary, the message passing neural network is used to capture network topology information, which can better represent and learn the characteristics of computing nodes in the cloud-edge computing network, and then predict the status of each link in the network topology structure.

[0117] Wherein, the step of obtaining a high-dimensional computing power node embedding representation and an edge embedding representation of the network topology structure includes:

[0118] Using a graph neural network to map the features of the computing nodes and edges of the network topology structure into a high-dimensional space; wherein the computing nodes represent the computing resources of the cloud-edge computing network, and the edges represent the network links of the cloud-edge computing network;

[0119] In high-dimensional space, computing nodes and edges receive information from neighboring nodes through a multi-layer message passing mechanism, and update their own computing node embedding representation and edge embedding representation;

[0120] After several rounds of information transmission, high-dimensional embedding representations of computing power nodes and edges are obtained.

[0121] When the message is transmitted between the computing nodes, the hidden state set of each link in the network topology structure is obtained, including:

[0122] Receive the hidden state of neighboring nodes after message exchange;

[0123] After combining the received hidden state with its own hidden state, the combined information is obtained;

[0124] Aggregate all the information after the combination processing to obtain aggregate information;

[0125] updating and calculating a new hidden state based on the received hidden state and the aggregate information;

[0126] After message transmission between computing nodes is completed, the characteristic vector of the network topology structure and the attributes of each computing node are predicted;

[0127] The predicted feature vector of the network topology structure and the attributes of each computing power node are used as a link hidden state set of the network topology structure.

[0128] Among them, the hidden state in the hidden state set of each link in the network topology structure is defined by the feature vector of the corresponding link. In the process of vectorizing the hidden state, when the dimension of the hidden state is not sufficient to represent all the link characteristics in the network topology structure, the feature vector is completed by zero filling.

[0129] In the above process, the network topology of the cloud-edge computing network is converted into a directed graph , the directed graph It includes computing nodes and edges. The computing nodes represent the computing resources of the cloud-edge computing network, and the edges represent the network links of the cloud-edge computing network. Represents the collection of computing nodes in the network topology. Represents a collection of edges in a network topology.

[0130] Step 3: Based on the high-dimensional computing power node embedding representation, edge embedding representation and hidden state set, predict the future traffic of each link in the network topology structure and obtain the state information of each link in the network topology structure.

[0131] By using message passing neural network and gated recurrent unit, based on high-dimensional node embedding representation, edge embedding representation and the hidden state set of each link in the network topology structure, the future traffic of each link in the network topology structure is predicted, and the state information of each link in the network topology structure is obtained.

[0132] Step 4: With the goal of minimizing the standard deviation of link utilization and task transmission time, construct a routing scheduling problem under the network architecture that optimizes the cloud-edge computing network.

[0133] The routing scheduling problem under the network architecture of the cloud-edge computing network is regarded as an optimization problem. The optimization target needs to consider the task completion time and network status, where the network status is indirectly represented by the link utilization.

[0134] The routing scheduling problem under the network architecture of the cloud-edge computing network is expressed as:

[0135] (7);

[0136] In the formula, represents the weight of the standard deviation of link utilization, represents the weight of the task transmission time, represents the standard deviation of link utilization, Indicates j The task transmission time of each path, Indicates j path.

[0137] in, , , .

[0138] Among them, link utilization will be maximized It is expressed as:

[0139] (8);

[0140] In the formula, For Link The allocated bandwidth, For Link The initial bandwidth, max( ) is the maximum value function, For any link To process, is the set of edges in the network topology.

[0141] The average link utilization It is expressed as:

[0142] (9);

[0143] In the formula, is the number of paths in the network topology.

[0144] Among them, the link utilization standard deviation It is expressed as:

[0145] (10);

[0146] Among them, the task transmission time It is expressed as:

[0147] (11);

[0148] In the formula, is the number of transmission paths between the source and the end point of the network topology, For Link link delay.

[0149] The optimization goal of this embodiment is as shown in formula (7), which is to minimize , try to obtain the smallest link utilization standard deviation and the smallest task completion time.

[0150] In the present invention, combined with the link utilization standard deviation and task completion time To solve the mathematical model of the routing scheduling problem under the network architecture of the cloud-edge computing network, and are two weights that affect routing scheduling and need to satisfy and and The values ​​are all By adjusting and The value of is used to adjust the optimization target of the mathematical model of the routing scheduling problem under the network architecture of the cloud-edge computing network. When is larger, the mathematical model of the routing scheduling problem under the network architecture of the cloud-edge computing network considers more the optimization of the maximum link utilization. When it is larger, the mathematical model of the routing scheduling problem under the network architecture of the cloud-edge computing network considers the optimization of task completion time more.

[0151] Formula (8) is the maximum link utilization in the network topology The calculation formula for each link is that each link has its own link utilization. It represents the maximum value of all link utilizations in the entire network. The smaller the maximum link utilization, the more balanced the bandwidth allocation in the network, which indirectly reflects the better network quality. Formula (9) is the calculation formula for the average value of link utilization in the network topology.

[0152] In order to focus on the global optimization of network performance, the standard deviation of the link utilization in the optimization target is used to calculate , as shown in formula (10). This setting allows optimizing the load of each topology, rather than just the link with the maximum link utilization. Formula (11) is the task completion time According to the calculation formula, the task completion time is originally composed of two parts: task transmission time and task calculation time.

[0153] The present invention mainly considers the impact of routing scheduling capabilities on computing scheduling time, so the task completion time is simplified to task transmission time for processing, and the task transmission time includes round-trip time. The one-way task transmission time is equal to the sum of all link delays on the transmission path.

[0154] Since the result of the task needs to be returned to the client after calculation at the computing power node, the task transmission time is twice the one-way task transmission time, so it is assumed that the transmission time and the time for returning the result are equal.

[0155] Step 5: Take the computing nodes of the network topology as intelligent agents and transform the pre-built routing scheduling problem into a Markov decision process according to the status information of each link in the network topology, including:

[0156] Obtaining initial link characteristics and traffic requirements of each link, and constructing a discrete state space, wherein the state space includes all states of the agent;

[0157] Obtaining weight data of each link, and constructing an action space based on the weight data of each link of the network topology structure, wherein the action space contains all actions selected by the agent in a given state;

[0158] Define a reward value function according to the link utilization standard deviation and task completion time of the pre-constructed routing scheduling problem, wherein the reward value function includes the cumulative reward of the agent from the initial state to the terminal state;

[0159] Constructing a Markov decision process according to the state space, the action space and the reward value function;

[0160] In the Markov decision process, the agent selects an action according to the current state, and obtains a reward according to the selected action and state change, and continuously iterates and learns with the goal of maximizing the reward value.

[0161] The reward value is defined as follows according to the optimization objective of minimizing the standard deviation of link utilization and task transmission time and formula (7): ,The reward value combines two key indicators, namely the standard deviation of link utilization in the cloud-edge computing network and the task completion time.

[0162] By adjusting and , the optimization target can be customized according to the specific needs of the actual network scenario. The standard deviation of all link utilization is used as the optimization target. The smaller the standard deviation of link utilization, the more balanced the bandwidth distribution in the network. The negative value is used as the reward value to make the agent converge faster. If the reward value is smaller, the link utilization and task completion time are larger, the network load balancing effect is worse, and the routing optimization strategy is worse; on the contrary, if the reward value is larger, the routing optimization strategy performance is better. In addition, if the agent's operation violates the constraints, the environment will impose a large negative value on the agent as a penalty. If all constraints are met, the penalty is 0.

[0163] Step 6: Based on the Markov decision process, a multi-agent proximal strategy optimization algorithm is used to train the agent to obtain an optimal routing strategy, including:

[0164] In the state space, an initial state of the network topology is obtained as an initial observation of each agent; a policy network of each agent is initialized with the goal of minimizing the standard deviation of link utilization and task transmission time, and the policy network is used to select an action in the action space according to the observation of the agent;

[0165] Repeat the following steps until the policy network converges to obtain an agent that can generate the optimal routing strategy, thereby obtaining the optimal routing strategy:

[0166] Each agent selects an action based on the current observation and policy network, and after executing the selected action, observes and records the new state of the network topology;

[0167] Calculate the reward value of each agent for performing an action according to the new state of the network topology and the reward value function;

[0168] Based on the reward value and the new state of the network topology, a proximal policy optimization algorithm is used to update the policy network of each agent.

[0169] Typical policy gradient algorithms suffer from inaccurate advantage estimation, which leads to data bias. If the policy changes too much in one update, it may deviate completely in the next sampling, causing the policy to be updated to a completely deviated position, thus forming a vicious cycle.

[0170] In the present invention, a multi-agent proximal policy optimization algorithm is used. Compared with the traditional policy gradient method, the multi-agent proximal policy optimization algorithm has been proven to have better data efficiency and lower training variance. The importance sampling in the multi-agent proximal policy optimization algorithm is mainly used to evaluate the difference between the new and old strategies. An importance sampling ratio that is too large or too small will limit the range of change of the new strategy, thereby preventing the strategy from going too far. In this way, the multi-agent proximal policy optimization algorithm can perform policy updates more stably, avoiding the vicious cycle problem commonly seen in traditional policy gradient methods.

[0171] The multi-agent proximal strategy optimization algorithm includes an Actor network and a Critic network, the Actor network and the Critic network output action values ​​and state values ​​respectively, and a loss function of the multi-agent proximal strategy optimization algorithm is constructed according to the loss value of the Actor network and the loss value of the Critic network. The loss function of the multi-agent proximal strategy optimization algorithm is used to back-propagate and update the neural network parameters, and through continuous iterative training, an agent capable of generating an optimal routing selection strategy is obtained;

[0172] The probability ratio between the new routing strategy and the old routing strategy It is expressed as:

[0173] (12);

[0174] In the formula, Neural network parameters for multi-agent proximal policy optimization algorithms, Additional neural network parameters created for the multi-agent policy optimization algorithm A copy of the Collected within a moment , probability ratio is used to represent the penalty coefficient, for The action of the moment, for The state of the network topology at any moment, is the best routing selection strategy, is the old routing strategy, where s represents the action, a represents the state, and r represents the reward value;

[0175] According to the next moment +1 bonus value The advantage function is calculated by the state value output by the Critic network, and the calculated advantage function is expressed as:

[0176] (13);

[0177] In the formula, for The advantage function at time, for The state value function at the moment; To accumulate 0 to M The reward value at that moment, 0 to M The time discount factor at the moment, l 0 to M Time representation within a moment; express M The state value function at time , for The time discount factor of the moment; a time discount factor close to 0 indicates a focus on short-term rewards, and a time discount factor close to 1 indicates a focus on long-term rewards;

[0178] According to the probability ratio between the new routing strategy and the old routing strategy And the advantage function, calculate the loss of the Actor network, and express the loss of the Actor network as:

[0179] (14);

[0180] In the formula, is the loss value of the Actor network, is the shear function, is the shear function Limit the probability ratio between the new routing strategy and the old routing strategy range, is a hyperparameter that determines the range of the upper and lower limits of the clipping function, For the sake of expectation;

[0181] Among them, the mean square error is used to calculate the loss of the Critic network, and the loss of the Critic network is expressed as:

[0182] (15);

[0183] In the formula, is the loss value of the Critic network, is the estimated state value of the Critic network, is the target value function of the Critic network. The value at the moment, where ;

[0184] The loss function of the multi-agent proximal strategy optimization algorithm is expressed as:

[0185] (16);

[0186] In the formula, is the total loss value of the multi-agent proximal policy optimization algorithm, is the loss value of the Actor network, is the loss value of the Critic network The coefficient of Reward for Entropy The coefficient of are neural network parameters, for The state of the network topology at any moment, Select the best routing strategy.

[0187] During the training phase, the corresponding actions are generated for each traffic demand through the link weights, and the routing path with the shortest link weight is calculated. Then, the corresponding reward value is determined based on the current network topology and bandwidth allocation. By continuously adjusting the link weights, the routing path that maximizes the reward value can be found to achieve global balance of the network load. When the training is completed, the final link weight obtained will guide the routing selection of subsequent traffic in the knowledge plane, control plane, management plane, and data plane in the network architecture of the cloud-edge computing network.

[0188] The training process of the multi-agent proximal policy optimization algorithm, such as Figure 4 shown.

[0189] To summarize, the present invention provides a cloud-edge computing network routing optimization method. First, by obtaining a high-dimensional embedding representation, the complex relationship and hidden features between nodes and edges are captured. Secondly, by obtaining the hidden state set of each link in the network topology structure, the internal state information of the cloud-edge computing network in the dynamic change process is obtained, which can more accurately grasp the real-time state of the network. By combining high-dimensional node embedding representation, edge embedding representation and hidden state set, the future traffic of each link in the network topology structure can be predicted, and the changing trend of the cloud-edge computing network traffic can be obtained.

[0190] The present invention also transforms the routing scheduling problem into a Markov decision process, which is an effective modeling method. By considering the computing power nodes as intelligent agents and using the state information of each link in the network topology structure as input, the Markov decision process can systematically describe the decision-making process of the intelligent agent in different states, providing a theoretical framework for finding the optimal routing strategy.

[0191] The present invention combines the advantages of deep reinforcement learning and multi-agent systems, and can effectively solve the routing scheduling problem in complex network environments. By using a multi-agent proximal strategy optimization algorithm to train the agent, it learns to choose the best action, i.e., routing decision, under a given state, thereby obtaining the optimal routing selection strategy.

[0192] The present invention constructs the routing scheduling problem with the goal of minimizing the standard deviation of link utilization and task transmission time, which can ensure that network resources are used evenly and reduce task transmission time. This multi-objective joint optimization method helps to improve the overall performance of the network and user experience. At the same time, by clarifying the optimization goals, algorithms and models can be designed more specifically to improve solution efficiency and quality.

[0193] The present invention improves the learning efficiency and generalization performance of the multi-agent proximal strategy optimization algorithm, so that the multi-agent deep reinforcement learning method can effectively adapt to more complex network environments, and provide a stable and efficient routing scheduling solution for user services, effectively improving the robustness of the cloud-edge computing network routing optimization method, and is suitable for large-scale complex networks. Once a cloud-edge computing network link failure is detected or a link is overloaded, other links will be given priority and rerouting operations will be automatically performed. It can effectively reduce manual intervention and improve the elasticity of the network.

[0194] The above is a detailed description of a cloud edge computing network routing optimization method provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. Similar parts between the various embodiments can refer to each other. Contents not described in detail in the specification of the present invention belong to the prior art known to professional and technical personnel in this field.

[0195] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0196] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0197] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0199] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A cloud-edge computing network routing optimization method, characterized in that: include: Obtain the network topology of the cloud-edge computing network; Obtaining a high-dimensional computing power node embedding representation and an edge embedding representation of the network topology structure; When messages are transmitted between computing nodes, the hidden state set of each link in the network topology is obtained; Based on the high-dimensional computing power node embedding representation, edge embedding representation and hidden state set, the future traffic of each link in the network topology is predicted, and the state information of each link in the network topology is obtained, including through the message passing mechanism, such as the message passing neural network in the neural network in the figure, to pass information between computing power nodes and calculate the hidden state set of each link; The computing nodes of the network topology are used as intelligent agents, and the pre-constructed routing scheduling problem is converted into a Markov decision process according to the status information of each link in the network topology. The routing scheduling problem is expressed as: ； In the formula, represents the weight of the standard deviation of link utilization, represents the weight of the task transmission time, represents the standard deviation of link utilization, Indicates j The task transmission time of each path, Indicates j Paths; Based on the Markov decision process, a multi-agent proximal strategy optimization algorithm is used to train the agent to obtain the optimal routing strategy; The routing scheduling problem is constructed with the goal of minimizing the link utilization standard deviation and the task transmission time.

2. The cloud-edge computing network routing optimization method according to claim 1 is characterized in that: The obtaining of the network topology structure of the cloud-edge computing network includes: The network architecture of the cloud-edge computing network is defined to include a knowledge plane, a control plane, a management plane, and a data plane, and a weighted graph data structure is used to represent the network architecture of the cloud-edge computing network to form a network topology structure; The knowledge plane is responsible for learning network behavior, receiving and processing network topology information collected by the management plane, converting the collected network topology information into network status knowledge through the Markov decision process, using the network status knowledge to formulate a routing optimization strategy for the cloud-edge computing network, and sending the routing optimization strategy to the control plane; The control plane is responsible for collecting traffic information of the forwarding device to construct the view of the data plane, and converting the routing optimization policy sent by the knowledge plane into flow table rules, and sending the flow table rules to the forwarding device of the data plane; The data plane is composed of a forwarding device and a link between the forwarding devices, and is responsible for controlling the forwarding rules of the traffic based on the flow table rules issued by the control plane and forwarding the traffic to the forwarding device for forwarding; The management plane is responsible for storing basic information from the data plane and the control plane, and periodically providing global information about the network to the knowledge plane, wherein the global information includes network topology information.

3. The cloud-edge computing network routing optimization method according to claim 2 is characterized in that: The network topology structure is a directed graph.

4. The cloud-edge computing network routing optimization method according to claim 1 is characterized in that: The obtaining of a high-dimensional computing node embedding representation and an edge embedding representation of the network topology structure includes: Using a graph neural network to map the features of the computing nodes and edges of the network topology structure into a high-dimensional space; wherein the computing nodes represent the computing resources of the cloud-edge computing network, and the edges represent the network links of the cloud-edge computing network; In high-dimensional space, computing nodes and edges receive information from neighboring nodes through a multi-layer message passing mechanism, and update their own computing node embedding representation and edge embedding representation; After several rounds of information transmission, high-dimensional embedding representations of computing power nodes and edges are obtained.

5. The cloud-edge computing network routing optimization method according to claim 4 is characterized in that: When the message is transmitted between the computing nodes, the hidden state set of each link in the network topology structure is obtained, including: Receive the hidden state of neighboring nodes after message exchange; After combining the received hidden state with its own hidden state, the combined information is obtained; Aggregate all the information after the combination processing to obtain aggregate information; updating and calculating a new hidden state based on the received hidden state and the aggregate information; After message transmission between computing nodes is completed, the characteristic vector of the network topology structure and the attributes of each computing node are predicted; The predicted feature vector of the network topology structure and the attributes of each computing power node are used as a hidden state set of each link in the network topology structure.

6. The cloud edge computing network routing optimization method according to claim 5 is characterized in that: The hidden state in the hidden state set of each link in the network topology is defined by the feature vector of the corresponding link; when the dimension of the hidden state is not sufficient to represent all the link features in the network topology, the feature vector is completed by zero padding.

7. The cloud-edge computing network routing optimization method according to claim 1, characterized in that: The computing nodes of the network topology structure are used as intelligent agents, and the pre-constructed routing scheduling problem is converted into a Markov decision process according to the status information of each link in the network topology structure, including: Obtaining initial link characteristics and traffic requirements of each link, and constructing a discrete state space, wherein the state space includes all states of the agent; Obtain weight data of each link, and construct an action space based on the weight data of each link, wherein the action space contains all actions selected by the agent in a given state; Define a reward value function according to the link utilization standard deviation and task completion time of the pre-constructed routing scheduling problem, wherein the reward value function includes the cumulative reward of the agent from the initial state to the terminal state; Constructing a Markov decision process according to the state space, the action space and the reward value function; In the Markov decision process, the agent selects an action according to the current state, and obtains a reward according to the selected action and state change, and continuously iterates and learns with the goal of maximizing the reward value.

8. The cloud edge computing network routing optimization method according to claim 7 is characterized in that: Based on the Markov decision process, a multi-agent proximal strategy optimization algorithm is used to train the agent to obtain the optimal routing strategy, including: In the state space, an initial state of the network topology is obtained as an initial observation of each agent; a policy network of each agent is initialized with the goal of minimizing the standard deviation of link utilization and task transmission time, and the policy network is used to select an action in the action space according to the observation of the agent; Repeat the following steps until the policy network converges to obtain an agent that can generate the optimal routing strategy, thereby obtaining the optimal routing strategy: Each agent selects an action based on the current observation and policy network, and after executing the selected action, observes and records the new state of the network topology; Calculate the reward value of each agent for performing an action according to the new state of the network topology and the reward value function; Based on the reward value and the new state of the network topology, a proximal policy optimization algorithm is used to update the policy network of each agent.

9. The cloud-edge computing network routing optimization method according to claim 8, characterized in that: The multi-agent proximal strategy optimization algorithm includes: The loss function of the multi-agent proximal strategy optimization algorithm is constructed based on the loss value of the Actor network and the loss value of the Critic network; the Actor network and the Critic network output action values ​​and state values ​​respectively; Using the loss function of the multi-agent proximal strategy optimization algorithm, back-propagation updates the neural network parameters, and through continuous iterative training, obtains an agent that can generate the optimal routing selection strategy; The loss function of the multi-agent proximal strategy optimization algorithm is expressed as: ； In the formula, is the total loss value of the multi-agent proximal policy optimization algorithm, is the loss value of the Actor network, is the loss value of the Critic network The coefficient of Reward for Entropy The coefficient of are neural network parameters, for The state of the network topology at any moment, Select the best routing strategy.

Images