A virtual network mapping method based on deep reinforcement learning

Abstract

This invention discloses a virtual network mapping method based on deep reinforcement learning. This method characterizes the physical network and virtual network as weighted undirected graphs, encodes them into a network topology vector using a graph neural convolutional network, and then inputs the network topology vector and the network resource state vector into a policy neural network in a scheduler, which is trained using a proximal policy optimization algorithm. Compared with existing technologies, this method dynamically adjusts scheduling decisions based on the current environmental state, coordinates node mapping and link mapping, improves the mapping benefit-cost ratio while improving physical network resource utilization, and reduces physical network cluster energy consumption. The model is simple and easy to train, significantly improving user experience and providing technical support for the field of virtual network mapping.

Description

Technical Field

[0001] The present invention relates to the field of virtual network mapping technology, in particular to virtual network mapping under cloud computing network virtualization technology and a virtual network mapping method based on deep reinforcement learning. Background Art

[0002] With technological advancements, cloud computing has become increasingly widely used. Cloud computing centers provide numerous physical servers and virtual machines and containers running on them. Software-defined networking and network virtualization allow multiple tenants to request and use computing and network resources on demand through a pay-as-you-go system. A key challenge is how to fully utilize physical network resources to serve a wider range of tenants with diverse network topologies and resource requirements. Network virtualization decouples network services from the underlying physical hardware, allowing multiple heterogeneous network structures to share a single physical network. It has become an effective resource sharing technology. Infrastructure providers leverage network virtualization technology to provide customized network topologies, or virtual networks, to different tenants, using their best efforts to provide the required resources (such as CPU capacity and network bandwidth). Each virtual network consists of a set of virtual nodes and virtual links between them. The process of mapping a virtual network to a physical network is called virtual network mapping. Virtual network mapping involves two phases: node mapping and link mapping. Node mapping maps virtual nodes to physical nodes, while link mapping maps virtual links to physical links. Virtual network mapping maps all virtual nodes and virtual links to corresponding physical nodes and links while satisfying resource constraints, allocating resources to tenants' virtual network mapping requests. Virtual network mapping is an important and challenging task in network virtualization for optimizing resource utilization and improving network performance. The virtual network mapping problem has been proven to be NP-hard, meaning that an exact solution is impossible in large network environments. To address this problem, many heuristic algorithms have been proposed. However, most algorithms simply use artificial rules to sort and map virtual network requests and nodes. A key drawback of these algorithms is their non-adaptive nature: the mapping strategy is predetermined and cannot be adjusted and optimized based on actual operational feedback and real-time network conditions.

[0003] In recent years, deep reinforcement learning (RL) has garnered widespread attention across various fields, offering a promising approach for solving VNE problems. The general RL paradigm typically involves an intelligent agent that learns through continuous interaction with its external environment. Initially, the agent begins with no knowledge and gradually learns through exploration, performing actions that change the state of the external environment. The environment provides the agent with a reward signal, which it then dynamically refines based on the next generation of actions. Through continuous interaction with the environment, the agent gradually learns a strategy that maximizes cumulative reward over the long term. After training, the model can automatically handle the generation of embedded policies. RL's strong adaptive self-learning capabilities make it well-suited for solving complex problems in dynamic environments, including VNEs. However, most algorithms treat node mapping and link mapping as separate stages, lacking coordination between the two, resulting in low resource utilization. Automatically adjusting the mapping strategy based on changes in network state and coordinating node and link mappings is crucial for improving physical network resource utilization, enhancing the cost-benefit ratio, and reducing energy consumption.

[0004] The existing virtual network mapping algorithm treats node mapping and link mapping as two independent stages, lacks coordination between node mapping and link mapping, and the mapping strategy cannot automatically adjust according to changes in network status, which leads to low cluster resource utilization and failure to consider both benefits and energy consumption. Summary of the Invention

[0005] The purpose of the present invention is to provide a virtual network mapping method based on deep reinforcement learning to address the deficiencies of the existing technology. The method adopts the method of treating the physical network and the virtual network as a weighted undirected graph, and uses a graph convolutional network (GCN) to encode them into a network topology structure vector. The network topology structure vector and the network resource state vector are jointly input into the policy neural network in the scheduler, and the proximal policy optimization algorithm (PPO) is used to train it. This method can simultaneously take into account the dynamic changes of network resources and the dependency between node mapping and link mapping, and dynamically adjust the mapping strategy according to the cluster resource situation to improve cluster resource utilization and increase the mapping benefit-cost ratio, thereby reducing the energy consumption of the physical network cluster.

[0006] The specific technical solution for achieving the purpose of the present invention is: a virtual network mapping method based on deep reinforcement learning, which specifically includes the following steps:

[0007] S1. Modeling of virtual network mapping model

[0008] 1) Physical network: The physical network can be viewed as a weighted undirected graph Gs = (Ns, Es, An, Al), where Ns represents the set of physical network computing nodes, Es represents the set of physical network links, An represents node attributes (such as CPU processing power, memory space, and node reliability), and Al represents link attributes (such as bandwidth, delay, and packet loss rate). The present invention uses the CPU processing power of each physical node as the physical node attribute and the bandwidth of each physical network as the physical link attribute. Therefore, each physical node There is a weight It represents the node The number of CPU computing resources available on . Similarly, each edge There is also a weight It represents the link e s The current available bandwidth. In the VNE problem, a virtual link in the virtual network corresponds to a physical routing path, which is composed of one or more physical links in the physical network. s To represent G s The set of all feasible paths in Refers to the node and The physical path between The available bandwidth of each path is expressed as B(P s ) indicates that it depends on the minimum remaining bandwidth of the bottleneck link along the physical path, that is,

[0009] 2) Virtual network request: Based on the virtual network mapping request received from the tenant, a virtual network topology is established on the shared physical network. The virtual network can also be viewed as a weighted undirected graph Gv = (Nv, Ev, Rn, Rl), where Nv represents the set of virtual network computing nodes, Ev represents the set of virtual network links, and Rn and Rl represent the virtual node and link requests of physical resources, respectively. A virtual network request VNR can be expressed as VNR = (Gv, ta, td), where ta and td represent the arrival and end time of the virtual network request, respectively. When a virtual network request is generated with a certain network topology structure and arrives at the physical network at time ta, if the resources of the physical network meet the resources Rn and Rl of the virtual network request, the virtual network request uses these resources until they are released at time td. If the remaining resources of the physical network are insufficient to meet the virtual network request or the embedded algorithm cannot find a suitable mapping solution, the mapping request of the virtual network is rejected. The present invention takes the CPU processing capacity of each virtual node as the virtual node attribute and the bandwidth of each virtual network as the virtual link attribute. Therefore, each virtual node Assigned a weight It represents a virtual node The minimum CPU capacity required. Similarly, each virtual link There is also a weight This refers to the virtual node and Virtual links between Minimum required bandwidth capacity.

[0010] 3) Virtual Network Mapping Problem: The virtual network embedding problem can be defined as a mapping from a virtual network to a physical network: M:Gv->Gs. In cloud data centers, VN requests arrive and terminate at irregular intervals. The key responsibility of the VNE is to dynamically allocate the required physical network resources when these VN requests arrive. The ultimate goal of the VNE is to find the appropriate solution for all VNs to maximize overall benefits and minimize overall energy consumption. Generally, a VNE process can be decomposed into two stages: node mapping and link mapping. Node mapping is the process of mapping each VN's virtual node to a specific physical node under node resource constraints. Equation (j) below represents the node resource constraint: the CPU resources of the virtual node must be less than or equal to the CPU resources of the mapped physical node. Virtual nodes in the same virtual network cannot be mapped to the same physical node. Equation (k) indicates that if any two virtual nodes in the same virtual network are mapped to the same physical node, then these two virtual nodes must be the same virtual node. Virtual nodes in different virtual networks can be mapped to the same physical node.

[0011]

[0012]

[0013] Among them, M n Represents a virtual node map.

[0014] In contrast, link mapping maps each virtual link of a VN to an appropriate physical path consisting of one or more physical links under the link resource constraints. Specifically, a link mapping is found: satisfy

[0015] S2. Reinforcement Learning Modeling

[0016] The basic idea of ​​reinforcement learning is that an agent observes the state of its environment, takes actions based on the observed state to interact with the environment, and continuously optimizes its strategy based on the rewards given by the environment to maximize the expected long-term return. The three elements of reinforcement learning are the state space, action space, and reward mechanism described below:

[0017] 1) State Space

[0018] The state of the environment observed by the agent Indicates the physical network status, Indicates the virtual network status of the requested mapping. Represents the available CPU resource vector of the physical node, B s represents the available bandwidth vector of the physical link, Represents the mapping state vector of the physical node, C v Represents the CPU resource vector of the virtual node, B v represents the virtual link bandwidth resource vector, Represents the virtual node mapping state vector.

[0019] 2) Action Space

[0020] The physical network has n nodes Let action space A={a0,a1,a2,...,a n}, when the agent performs action a i When the virtual node to be mapped is mapped to the physical node In particular, the present invention provides an action a n Used to indicate that the current virtual node is not mapped and set action a n The benefit is that it provides the smart mapper with the ability to reject the current virtual network mapping request, thereby potentially making a scheduling decision that is more optimal from a global perspective.

[0021] 3) Reward Mechanism

[0022] After completing the link mapping or node mapping requested by the VN, the mapping reward is calculated by the algorithm as follows:

[0023] A. For a VN mapping request, if the number of mapping actions performed by the agent exceeds the maximum number of mapping actions set, the mapping fails and the reward is -100;

[0024] B. If the node resources of the physical network are insufficient, the reward is 0;

[0025] C. If the physical node selected for the action has already been mapped, or the available CPU resources of the physical node selected for the action are less than the resources required by the virtual node, the reward is calculated using the following formula (a):

[0026]

[0027] D. For a VN being mapped, vn_embed_nodesnum is the number of nodes already mapped, and vn_nodesnum is the number of nodes in the VN. For example, if a VNR has four virtual nodes, the reward for a failed mapping of the first node is -40, the reward for a failed mapping of the second node is -60, and so on, until the reward for a failed mapping of the last node is -100.

[0028] E. If after a virtual node is mapped, the virtual link where it is located fails to map due to insufficient physical network resources, the VN mapping fails and the reward is -100.

[0029] F. If the virtual node and the virtual link where the virtual node is located are successfully mapped, the reward is calculated according to the following formula (b):

[0030]

[0031] Among them, Rev is the revenue generated by mapping virtual node j, and PN is the energy consumption generated by mapping virtual node j.

[0032] S3, virtual network mapping based on the proximal policy optimization (PPO) algorithm

[0033] 1) Neural network initialization

[0034] The Proximal Policy Optimization (PPO) algorithm is based on the actor-critic architecture. Therefore, we first need to initialize three neural networks, namely the current policy network π θ and the old policy network And the value network Q. The policy network is used to output the probability of each action performed by the agent in a certain state, while the value network is used to evaluate the quality of the state.

[0035] 2) Check whether there is a virtual network mapping request in the current time slice t. If there is a virtual network mapping request in the current time slice t, start mapping the virtual network.

[0036] 3) Use the graph convolutional neural network (GCN) to embed the physical network and the virtual network state to be mapped into a fixed-size vector, which is then combined with the available CPU resource vector of the physical node, the available bandwidth vector of the physical link, the mapping state vector of the physical node, the CPU resource vector of the virtual node, the mapping state vector of the virtual node, and the bandwidth resource vector of the virtual link to form the state s that the current intelligent scheduler can observe. t .

[0037] 4) Change the state s t Input to the policy network π θ In the strategy network, the probability of all actions is output. Then, sampling is performed based on the probability to obtain action a.t . Then calculate the current reward r t , and {s t , a t , r t}Stored in the memory pool.

[0038] 5) Enter the next time t+1 and repeat steps (2) to (4) until the preset deadline t is reached. max , or the maximum number of steps per round T, or all virtual network mapping requests are mapped.

[0039] 6) Take out data {s0, a0, r0, s1, a1, r1...s T , a T , r T}, change the state s t Input to the value network Q, use the value network to estimate v(s t ), and then use the following formulas (e) to (f) to calculate the action advantage A t :

[0040] μ t =r t +γν(s t+1 )-v(s t ) (e);

[0041] A t =μ t +(γλ)μ t+1 +...+(γλ) T-t+1 μ T-1 (f).

[0042] Among them, γ and λ are hyperparameters, representing the discount factor and the number of steps respectively; in this invention, γ is taken as 0.99 and λ is taken as 0.95; r t represents the reward obtained at time t; v(s t ) and v(s t+1 ) represent the state values ​​at time t and time t+1 respectively.

[0043] 7) Take out data {s0, a0, r0, s1, a1, r1...s T , a T , r T}, change the state s t Input into the policy network π θ and strategic networks Get in state s t The action probability distribution π under θ (a t |s t )and Calculate the importance weight r t (θ), and L clip (θ) is used as the loss function, back propagated, and the policy network π is updated θ , the importance weight r t (θ) is calculated by the following formula (g):

[0044]

[0045] The L clip (θ) is calculated by the following formula (h);

[0046]

[0047] Wherein, ε is a hyperparameter, and in this invention, ε=0.2; clip(r t (θ), 1-ε, 1+ε) means when r t When (θ) is greater than 1+ε, it takes the value of 1+ε. t When (θ) is less than 1-ε, it takes the value of 1-ε; the min function means taking the smaller value of the two. Expressing expectation.

[0048] 8) Repeat the above steps 6) to 7) several times, and then use the new policy network parameters to overwrite the old policy network parameters, that is,

[0049] 9) Calculate the discounted return G according to the following formula (i): t , and then use the discounted return G t And the predicted value v(s) of the value network Q t ) to make a difference, use the mean square error MSE as the loss function, back propagate, and update the value network parameters:

[0050] G t =r t+1 +γr t+2 +γ 2 r t+3 +...+γ T-1 r T+1 +γ T-t+1 v(s t+1 ) (i).

[0051] 10) Clear the memory pool and repeat steps 2) to 9) for several rounds until the environment reward obtained by the intelligent scheduler converges to the optimal value or suboptimal value.

[0052] Compared with the existing technology, the present invention takes into account the dynamic changes of network resources and the dependency between node mapping and link mapping, and dynamically adjusts the mapping strategy according to the cluster resource situation to improve cluster resource utilization and increase the mapping benefit-cost ratio, further reducing the energy consumption of the physical network cluster. The cluster resource utilization is high, the model is simple and easy to train, and it can significantly improve the user experience, providing technical support for the field of virtual network mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the deep reinforcement learning model constructed for the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described and explained in detail below with specific implementation:

[0055] Example 1

[0056] See Figure 1 , follow the steps below to perform virtual network mapping:

[0057] S1. Modeling of virtual network mapping model

[0058] 1) Physical network: The physical network can be viewed as a weighted undirected graph Gs = (Ns, Es, An, Al), where Ns represents the set of physical network computing nodes and Es represents the set of physical network links. An represents node attributes (such as CPU processing power, memory space, and node reliability), and Al represents link attributes (such as bandwidth, delay, and packet loss rate). The present invention uses the CPU processing power of each physical node as a node attribute and the bandwidth of each physical network as a link attribute. Therefore, each physical node There is a weight It represents the node The number of CPU computing resources available on . Similarly, each edge There is also a weight It represents the link e s The current available bandwidth. In the VNE problem, a virtual link in the virtual network corresponds to a physical routing path, which consists of one or more physical links in the physical network. Here we use P s To represent G s The set of all feasible paths in Refers to the node and The physical path between The available bandwidth of each path is expressed as B(P s ) indicates that it depends on the minimum remaining bandwidth of the bottleneck link along the physical path, that is,

[0059] 2) Virtual network request: Based on the virtual network mapping request received from the tenant, a virtual network topology is established on the shared physical network. The virtual network can also be viewed as a weighted undirected graph Gv = (Nv, Ev, Rn, Rl), where Nv represents the set of virtual network computing nodes, Ev represents the set of virtual network links, and Rn and Rl represent the virtual node and link requests for physical resources, respectively. A virtual network request VNR can be expressed as VNR = (Gv, ta, td), where ta and td represent the arrival and end time of the virtual network request, respectively. When a virtual network request is generated with a certain network topology structure and arrives at the physical network at time ta, if the resources of the physical network meet the resources Rn and Rl of the virtual network request, the virtual network request uses these resources until they are released at time td. If the remaining resources of the physical network are insufficient to meet the virtual network request or the embedded algorithm cannot find a suitable mapping solution, the mapping request of the virtual network is rejected. The present invention takes the CPU processing capacity of each node as a node attribute and the bandwidth of each physical network as a link attribute. Therefore, each virtual node Assigned a weight It represents a virtual node The minimum CPU capacity required. Similarly, each virtual link There is also a weight This refers to the virtual node and Virtual links between Minimum required bandwidth capacity.

[0060] 3) Virtual Network Mapping Problem: The virtual network embedding problem can be defined as a mapping from a virtual network to a physical network: M:Gv->Gs. In cloud data centers, VN requests arrive and terminate at irregular intervals. The key responsibility of the VNE is to dynamically allocate the required physical network resources when these VN requests arrive. The ultimate goal of the VNE is to find the appropriate solution for all VNs to maximize overall benefits and minimize overall energy consumption. Generally, a VNE process can be decomposed into two stages: node mapping and link mapping. Node mapping is the process of mapping each VN's virtual node to a specific physical node under node resource constraints. Equation (j) below represents the node resource constraint, namely, the CPU resources of the virtual node must be less than or equal to the CPU resources of the mapped physical node. Virtual nodes in the same virtual network cannot be mapped to the same physical node. Equation (k) indicates that if any two virtual nodes in the same virtual network are mapped to the same physical node, then these two virtual nodes must be the same virtual node. Virtual nodes in different virtual networks can be mapped to the same physical node.

[0061]

[0062]

[0063] Among them, M n Represents a virtual node map.

[0064] In contrast, link mapping maps each virtual link of a VN to an appropriate physical path consisting of one or more physical links under the link resource constraints. Specifically, a link mapping is found: satisfy

[0065] S2. Reinforcement Learning Modeling

[0066] The basic idea of ​​reinforcement learning is that an agent observes the state of its environment, takes actions based on the observed state to interact with the environment, and continuously optimizes its strategy based on the rewards given by the environment to maximize the expected long-term return. The three elements of reinforcement learning are the state space, action space, and reward mechanism described below:

[0067] 1) State Space

[0068] The state of the environment observed by the agent Indicates the physical network status; Indicates the virtual network status of the requested mapping; C s Represents the available CPU resource vector of the physical node; B s The available bandwidth vector representing the physical link; Represents the mapping state vector of the physical node; C v Represents the CPU resource vector of the virtual node; B v represents the virtual link bandwidth resource vector; Represents the virtual node mapping state vector.

[0069] 2) Action Space

[0070] The physical network has n nodes Let action space A={a0,a1,a1,...,a n}, when the agent performs action a i When the virtual node to be mapped is The present invention sets an action a n Used to indicate that the current virtual node is not mapped and set action a n The benefit is that it provides the smart mapper with the ability to reject the current virtual network mapping request, thereby potentially making a scheduling decision that is more optimal from a global perspective.

[0071] 3) Reward Mechanism

[0072] After completing the link mapping or node mapping requested by the VN, the mapping reward is calculated by the algorithm as follows:

[0073] A. For a VN mapping request, if the number of mapping actions performed by the agent exceeds the maximum number of mapping actions set, the mapping fails and the reward is -100.

[0074] B. If the node resources of the physical network are insufficient, the reward is 0.

[0075] C. If the physical node selected for the action has already been mapped, or the available CPU resources of the physical node selected for the action are less than the resources required by the virtual node, the reward is calculated using the following formula (a):

[0076]

[0077] D. For a VN being mapped, vn_embed_nodesnum is the number of nodes already mapped, and vn_nodesnum is the number of nodes in the VN. For example, if a VNR has four virtual nodes, the reward for a failed mapping of the first node is -40, the reward for a failed mapping of the second node is -60, and so on, until the reward for a failed mapping of the last node is -100.

[0078] E. If after a virtual node is mapped, the virtual link where it is located fails to map due to insufficient physical network resources, the VN mapping fails and the reward is -100.

[0079] F. If the virtual node and the virtual link where the virtual node is located are successfully mapped, the reward is calculated according to the following formula (b):

[0080]

[0081] Among them, Rev is the revenue generated by mapping virtual node j, and PN is the energy consumption generated by mapping virtual node j.

[0082] S3, virtual network mapping based on the proximal policy optimization (PP0) algorithm

[0083] 1) Neural network initialization: The Proximal Policy Optimization (PPO) algorithm is based on the actor-critic architecture. Therefore, we first need to initialize three neural networks, namely the current policy network π θ and the old policy network And the value network Q. The policy network is used to output the probability of each action performed by the agent in a certain state, while the value network is used to evaluate the quality of the state.

[0084] 2) Check whether there is a virtual network mapping request in the current time slice t. If there is a virtual network mapping request in the current time slice t, start mapping the virtual network.

[0085] 3) Use the graph convolutional neural network (GCN) to embed the physical network and the virtual network state to be mapped into a fixed-size vector. Then, it is combined with the available CPU resource vector of the physical node, the available bandwidth vector of the physical link, the mapping state vector of the physical node, the CPU resource vector of the virtual node, the mapping state vector of the virtual node, and the bandwidth resource vector of the virtual link to form the state s that the current intelligent scheduler can observe. t .

[0086] 4) Change the state s t Input to the policy network π θ In the strategy network, the probability of all actions is output. Then sampling is performed based on the probability to obtain action a t . Then calculate the current reward r t , and {s t , a t , r t}Stored in the memory pool.

[0087] 5) Enter the next time t+1 and repeat steps (2) to (4) until the preset deadline t is reached. max , or the maximum number of steps per round T, or all virtual network mapping requests are mapped.

[0088] 6) Take out data {s0, a0, r0, s1, a1, r1...s T , a T , r T}, change the state s t Input to the value network Q, use the value network to estimate v(s t ), and then use the following formulas (e) to (f) to calculate the action advantage A t :

[0089] μ t =r t +γν(s t+1 )-v(s t ) (e);

[0090] A t =μ t +(γλ)μ t+1 +...+(γλ) T-t+1 μ T-1 (f).

[0091] Among them, γ and λ are hyperparameters, representing the discount factor and the number of steps respectively; in this invention, γ is taken as 0.99 and λ is taken as 0.95; r t represents the reward obtained at time t; v(s t ) and v(s t+1 ) represent the state values ​​at time t and time t+1 respectively.

[0092] 7) Take out data {s0, a0, r0, s1, a1, r1...s T , a T , r T}, change the state s t Input into the policy network π θ and strategic networks Get in state s t The action probability distribution π under θ (a t |s t )and Calculate the importance weight r t (θ), and L clip (θ) is used as the loss function, back propagated, and the policy network π is updated θ , the importance weight r t (θ) is calculated by the following formula (g):

[0093]

[0094] The L clip (θ) is calculated by the following formula (h);

[0095]

[0096] Wherein, ε is a hyperparameter, and in the present invention, ε=0.2, clip(r t (θ), 1-ε, 1+ε) means when r t When (θ) is greater than 1+ε, it takes the value of 1+ε. t When (θ) is less than 1-ε, it takes the value of 1-ε. The min function means taking the smaller value of the two. Expressing expectation.

[0097] 8) Repeat the above steps 6) to 7) several times, and then use the new policy network parameters to overwrite the old policy network parameters, that is,

[0098] 9) Calculate the discounted return G according to the following formula (i): t , and then use the discounted return G t And the predicted value v(s) of the value network Q t) to make a difference, use the mean square error MSE as the loss function, back propagate, and update the value network parameters:

[0099] G t =r t+1 +γr t+2 +γ 2 r t+3 +...+γ T-t r T+1 +γ T-t+1 v(s t+1 ) (i).

[0100] 10) Clear the memory pool and repeat steps 2) to 9) for several rounds until the environment reward obtained by the intelligent scheduler converges to the optimal value or suboptimal value.

[0101] The above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A virtual network mapping method based on deep reinforcement learning, characterized in that: The virtual network mapping method specifically includes the following steps: 1) Perform virtual network mapping modeling based on the collected multi-tenant customized virtual network mapping request information and physical network cluster resource information. The physical network is a weighted undirected graph Gs represented by the following formula (a): Gs =(Ns,Es,An,Al) (a); Among them, Ns is the set of physical network computing nodes; Es is the set of physical network links; An is the node attribute; Al is the link attribute; 2) Conduct reinforcement learning modeling based on the virtual network model and the physical network cluster resource model, which specifically includes the design of the state space, action space, and reward mechanism; 3) The mapper is trained using the Proximal Policy Optimization (PPO) algorithm in deep reinforcement learning. The Proximal Policy Optimization (PPO) algorithm uses three neural networks based on the actor-critic architecture, which are the current policy network, , the old policy network and value network Q; The step 3) specifically includes: 3-1: Neural Network Initialization Initialize the current policy network , the old policy network and the value network Q three neural networks, the policy network and strategic networks It is used to output the probability of each action performed by the agent in a certain state; the value network Q is used to evaluate the quality of the state; 3-2: Check whether there is a virtual network mapping request in the current time slice t. If there is a virtual network mapping request in the current time slice t, start mapping the virtual network; 3-3: Use the graph convolutional neural network (GCN) to embed the physical network and the virtual network state to be mapped into a fixed-size vector, which is then combined with the available CPU resource vector of the physical node, the available bandwidth vector of the physical link, the mapping state vector of the physical node, the CPU resource vector of the virtual node, the mapping state vector of the virtual node, and the bandwidth resource vector of the virtual link to form the state observable by the current intelligent scheduler. ; 3-4: Change status Input to the policy network The output is the probability of all actions, and the actions are sampled according to the probability. , then calculate the current reward , and Store to memory pool; 3-5: Go to the next time t+1 and repeat steps 3-2 to 3-4 until the preset deadline is reached. , the maximum number of steps per round T or all virtual network mapping requests are mapped; 3-6: Retrieving data from the memory pool , the state Input to the value network Q, use the value network Q to estimate , and calculate the action advantage by the following formulas (e) to (f) : （e）； （f）； in, are hyperparameters, representing the discount factor and the number of steps respectively; is the reward obtained at time t; and are the state values ​​at time t and time t+1 respectively; 3-7: Retrieving data from the memory pool , the state Input into the policy network and the old policy network , respectively and , calculate the importance weight , and As loss function, back propagate and update the policy network , the importance weight Calculated by the following formula (g): （g）； described From the following formula (h) ; ] （h）； in, is a hyperparameter; Indicates when Greater than When ,when When , the value is ; The min function means taking the smaller value of the two; Expressing hope; 3-8: Repeat steps 3-6 to 3-7 several times, and then overwrite the old policy network parameters with the updated policy network parameters, that is, ; 3-9: Calculate the discounted return using the following formula (i) : （i）； Use Discount Returns and the predicted value of the value network Q Use mean square error (MSE) as the loss function, back propagate, and update the parameters of the value network Q; 3-10: Clear the memory pool and repeat steps 3-2 to 3-9 for several rounds until the environmental reward obtained by the intelligent scheduler converges to the optimal value or suboptimal value.

2. The virtual network mapping method based on deep reinforcement learning according to claim 1 is characterized in that: The step 1) specifically includes: 1-1: Based on the virtual network mapping request received from the tenant, a virtual network topology is established on the shared physical network. The virtual network is represented as a weighted undirected graph Gv by the following formula (b): Gv=(Nv,Ev,Rn,Rl) (b); Where Nv is the set of virtual network computing nodes; Ev is the set of virtual network links; Rn and Rl are the virtual node and link requests VNR for the underlying resources respectively; 1-2: A virtual network request VNR is represented by the following formula (c): VNR=(Gv,ta,td) (c); Among them, ta and td represent the arrival and end time of the virtual network request respectively; 1-3: When a virtual network request (VNR) is generated with a certain virtual network topology and arrives at the physical network at time ta, if the physical network resources meet the virtual network's requested resources Rn and Rl, the virtual network request uses these resources until they are released at time td. If the remaining resources in the physical network are insufficient to meet the virtual network request or the embedded algorithm cannot find a suitable mapping solution, the virtual network mapping request is rejected. The embedding algorithm maps the virtual network VNE to the physical network: M:Gv->Gs. In a cloud data center, virtual network requests arrive and end irregularly; The embedding algorithm defines virtual network embedding as a mapping from a virtual network to a physical network: M:Gv->Gs. This mapping dynamically allocates the required physical network resources when these virtual network VN requests arrive. In a cloud data center, virtual network VN requests arrive and end irregularly.

3. The virtual network mapping method based on deep reinforcement learning according to claim 1, characterized in that: The state space in step 2) is the state of the environment observed by the agent ,in Indicates the physical network status, ; Represents the available CPU resource vector of the physical node; The available bandwidth vector representing the physical link; Represents the mapping state vector of the physical node; Indicates the virtual network status of the requested mapping. ; Represents the CPU resource vector of the virtual node; represents the virtual link bandwidth resource vector; Represents the virtual node mapping state vector; The action space is a physical network with n nodes , let the action space A = { }, when the agent performs an action When the virtual node to be mapped is mapped to the physical node , set the action Used to indicate that the current virtual node is not mapped; After completing the link mapping or node mapping requested by the VN, the reward mechanism calculates the mapping reward according to the following mechanism: A. For a VN mapping request, if the number of mapping actions performed by the agent exceeds the maximum number of mapping actions set, the mapping fails and the reward is -100; B. If the node resources of the physical network are insufficient, the reward is 0; C. If the physical node selected for the action has already been mapped, or the available CPU resources of the physical node selected for the action are less than the resources required by the virtual node, the reward is calculated using the following formula (a): （a）； D. For a VN being mapped, vn_embed_nodesnum is the number of nodes already mapped, and vn_nodesnum is the number of nodes in the VN. If a VNR has 4 virtual nodes, the reward for the first node mapping failure is -40, the second node is -60, and so on. The reward for the last node mapping failure is -100. E. If after a virtual node is mapped, the virtual link where it is located fails to map due to insufficient physical network resources, the VN mapping fails and the reward is -100; F. If the virtual node and the virtual link where the virtual node is located are successfully mapped, the reward is calculated according to the following formula (b): （b）； Among them, Rev (n v j ) is the revenue generated by mapping virtual node j; PN j is the energy consumption generated by mapping virtual node j.

Images