Virtual network mapping method and device based on improved deep reinforcement learning

Abstract

The invention discloses a virtual network mapping method and device based on improved deep reinforcement learning, and the method comprises the steps: building a network system diagram model based on a physical network and a virtual network, and enabling network nodes and links in the network system diagram model to be converted into multi-dimensional vectors, and enabling the multi-dimensional vectors to be used for regularizing network topology and forming a system state; key spatial features in the physical network state are automatically extracted as input by adopting a depth map convolutional neural network, and a target action is selected and executed through a strategy network; calculating a reward signal according to the action execution result and the network performance index, and updating a strategy network parameter according to the reward signal by using a gradient descent algorithm; and obtaining the trained target depth map convolutional neural network and the target strategy network, and determining a target performance evaluation result.

Description

Technical Field

[0001] The present application relates to the technical field of network virtualization, and more specifically, to a virtual network mapping method and device based on improved deep reinforcement learning. Background Art

[0002] With the rapid development of information technology and the increasing number of network applications, Virtual Network Embedding (VNE) technology plays an increasingly important role in the field of network virtualization. The core goal of VNE technology is to map the requirements of virtual networks (VN) to the existing physical networks (Substrate Network, SN) to achieve efficient resource utilization and flexible deployment of network services. However, the current VNE technology faces the following major problems in practical applications:

[0003] 1. Limitations of static optimization algorithms: Most existing VNE solutions are based on static optimization algorithms, which are obviously insufficient in the face of rapid changes and uncertainties in the network environment. They cannot effectively cope with dynamic changes in network topology and traffic patterns, such as the increase and decrease of nodes and fluctuations in traffic.

[0004] 2. Resource utilization and mapping quality: Traditional methods have limitations in key performance indicators such as resource utilization, mapping success rate and mapping quality, and are difficult to meet the needs of large-scale networks and dynamic changes.

[0005] 3. Insufficient adaptability and robustness: With the expansion of network scale and the increase in complexity of network environment, existing VNE technology faces challenges in adaptability and robustness, and it is difficult to ensure stable services under various network conditions.

[0006] 4. Computational complexity and scalability: For large-scale VNE problems, existing algorithms often consume a lot of computing resources and time to find the optimal solution, which limits their solution scale and feasibility of practical applications.

[0007] 5. Quality of Service (QoS) Guarantee: In a multi-tenant and dynamic network environment, it is a challenge to ensure that the QoS requirements of each virtual network request (VNR) are met. Existing methods are insufficient in terms of resource allocation and QoS guarantee. Summary of the invention

[0008] In response to at least one defect or improvement need in the prior art, the present invention provides a virtual network mapping method and device based on improved deep reinforcement learning, which solves the problem that the existing VNE cannot effectively cope with the dynamic changes of network topology and traffic patterns, improves the solving efficiency and mapping quality of VNE problems, and enhances the adaptability, robustness and security of the network.

[0009] To achieve the above-mentioned purpose, according to a first aspect of the present invention, a virtual network mapping method based on improved deep reinforcement learning is provided, the method comprising: establishing a network system graph model based on a physical network and a virtual network, wherein the network nodes and links in the network system graph model are converted into multidimensional vectors for regularizing the network topology and constituting the system state; using a deep graph convolutional neural network to automatically extract key spatial features in the physical network state as input, selecting and executing a target action through a policy network; calculating a reward signal based on the action execution result and the network performance indicator, and using a gradient descent algorithm to update the policy network parameters based on the reward signal; obtaining the trained target deep graph convolutional neural network and target policy network, and determining a target performance evaluation result.

[0010] In an exemplary embodiment, the use of a deep graph convolutional neural network to automatically extract key spatial features in the physical network state as input includes: extracting multiple key spatial features based on network nodes, wherein the key spatial features include node resource utilization, the sum of bandwidth utilization of links connected to the nodes, remaining resources, the sum of remaining bandwidths of links connected to the nodes, the degree of the node, the average distance to other mapped nodes, and the node proximity centrality; after the key spatial features of each layer are extracted, a rectified linear unit activation function is used.

[0011] In an exemplary embodiment, the method of using a deep graph convolutional neural network to automatically extract key spatial features in the physical network state as input, selecting a target action through a policy network and executing the action includes: using the policy network to forward propagate the current state to obtain a probability distribution of each action; selecting the target action actually taken based on the output of the policy network; constructing a virtual label based on the target action and the output of the policy network; calculating the difference between the probability distribution of the actual execution of the target action and the probability distribution predicted by the policy network, and determining a loss function.

[0012] In an exemplary embodiment, the reward signal is calculated according to the action execution result and the network performance index, and the policy network parameters are updated according to the reward signal using the gradient descent algorithm, including: calculating the mapping benefit after successful mapping and the mapping cost during the mapping process, and taking the ratio of the mapping benefit to the mapping cost as the score of the reward signal; and passing the score of the reward signal to each weight of the deep graph convolutional neural network through back propagation and the gradient descent algorithm for updating.

[0013] In an exemplary embodiment, before establishing a network system graph model based on the physical network and the virtual network, the method further includes: initializing network parameters of the deep graph convolutional neural network for feature extraction, initializing policy network parameters of the learning agent; and performing an algorithm loop after setting hyperparameters, wherein the hyperparameters include a learning rate, a discount factor, and an exploration rate.

[0014] In an exemplary embodiment, after initializing the network parameters of the deep graph convolutional neural network for feature extraction and initializing the policy network parameters of the learning agent, the method further includes: using the agent to perform actions in the environment to learn the optimal network mapping strategy; the agent interacts with the environmental state to perform actions and receive feedback, wherein the environmental state includes the topological structure of the underlying network, the resource usage of nodes and links, and the characteristics of the VNR, and the action space contains all actions that the agent can perform, which is used to allocate nodes and links to the VNR.

[0015] According to the second aspect of the present invention, a virtual network mapping device based on improved deep reinforcement learning is also provided, which includes: an establishment unit, used to establish a network system graph model based on a physical network and a virtual network, wherein the network nodes and links in the network system graph model are converted into multidimensional vectors for regularizing the network topology and constituting the system state; an input unit, used to automatically extract key spatial features in the physical network state using a deep graph convolutional neural network as input, select a target action through a policy network and execute it; an update unit, used to calculate a reward signal based on the action execution result and the network performance indicator, and use a gradient descent algorithm to update the policy network parameters based on the reward signal; a determination unit, used to obtain the trained target deep graph convolutional neural network and target policy network, and determine the target performance evaluation result.

[0016] According to a third aspect of the present invention, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned virtual network mapping method based on improved deep reinforcement learning when running.

[0017] According to a fourth aspect of the present invention, there is also provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the virtual network mapping method based on improved deep reinforcement learning through the computer program.

[0018] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0019] (1) The present invention provides a virtual network mapping method based on improved deep reinforcement learning, which can realize dynamic mapping of virtual networks and optimize the utilization of network resources while maintaining rapid response and adaptability to changes in the network environment. By extracting key network features through deep graph convolutional neural networks, higher quality mapping decisions can be made and the embedding quality can be improved.

[0020] (2) By adopting a virtual network mapping method based on improved deep reinforcement learning provided by the present invention, deep reinforcement learning is combined with graph convolutional neural network to improve the intelligence level of mapping strategy. It can realize dynamic optimization and multi-objective optimization, dynamically adapt to changes in the network environment, and adjust the mapping strategy in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic flow chart of an optional virtual network mapping method based on improved deep reinforcement learning provided in an embodiment of the present application;

[0023] Figure 2 An optional virtual network request schematic diagram provided for an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the structure of an optional virtual network mapping algorithm based on improved deep reinforcement learning provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of an optional training process of a virtual network mapping algorithm based on improved deep reinforcement learning provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of an optional virtual network mapping device based on improved deep reinforcement learning provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0029] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0030] According to one aspect of the embodiments of the present application, a virtual network mapping method based on improved deep reinforcement learning is provided. Figure 1 The present invention describes a virtual network mapping method based on improved deep reinforcement learning provided in an embodiment of the present application.

[0031] Figure 1 is a flow chart of an optional virtual network mapping method based on improved deep reinforcement learning provided in an embodiment of the present application, such as Figure 1 As shown, the process of the method may include the following steps:

[0032] S102, establishing a network system graph model based on the physical network and the virtual network, wherein the network nodes and links in the network system graph model are converted into multi-dimensional vectors for regularizing the network topology and constituting the system state;

[0033] S104, using a deep graph convolutional neural network to automatically extract key spatial features in the physical network state as input, and selecting and executing the target action through the policy network;

[0034] S106, calculating a reward signal according to the action execution result and the network performance index, and using a gradient descent algorithm to update the policy network parameters according to the reward signal;

[0035] S108, obtaining the trained target deep graph convolutional neural network and target policy network, and determining the target performance evaluation result.

[0036] A virtual network mapping method based on improved deep reinforcement learning provided in an embodiment of the present application can be used to solve the dynamic and uncertainty problems in virtual network embedding (VNE).

[0037] Optionally, you can first perform abstract modeling of the network system to establish a network system graph model. Figure 2 , using graph theory knowledge, physical networks (SN) and virtual networks (VN) can be abstracted into graph models. In this model, network nodes and links are converted into multidimensional vectors to regularize the network topology and constitute the system state. The modeling of SN and VN is abstracted based on the knowledge of graph theory. For example, SN can be represented as a graph G = (V, E). In SN, V represents the physical nodes in the network, such as routers and switches, and E represents the physical links connecting these nodes; in VN, V represents the nodes that need to be embedded, and E represents the links required to connect these nodes.

[0038] The processing power and memory size of the node are represented by resource vectors, which explicitly specify the required processing power and memory size, and are quantified by determining the amount of CPU memory available for each node. For links, they are quantified by the amount of bandwidth and the maximum delay time. The remaining bandwidth is also an important indicator of the data transmission capacity it can support, and is represented by the value after subtracting the currently allocated bandwidth. These resource requirements are represented in the VNE model by mathematical formulas or constraints to guide the optimization algorithm to find a mapping solution that meets the requirements of all virtual nodes and links, while maximizing the utilization of SN resources.

[0039] Furthermore, if Figure 3 As shown, a deep graph convolutional neural network can be used to automatically extract key spatial features in the physical network state as input. It should be noted that the key spatial features include node resource utilization, the sum of bandwidth utilization of links connected to the node, remaining resources, the sum of remaining bandwidth of links connected to the node, node degree, average distance to other mapped nodes, and node proximity centrality. Then, the target action can be selected and executed through the policy network, and the reward signal is calculated based on the action execution results and network performance indicators. The gradient descent algorithm is used to update the policy network parameters based on the reward signal. Obtain the trained target deep graph convolutional neural network and target policy network to determine the target performance evaluation result.

[0040] Through the above steps S102 to S108, a network system graph model is established based on the physical network and the virtual network, wherein the network nodes and links in the network system graph model are converted into multidimensional vectors for regularizing the network topology and constituting the system state; a deep graph convolutional neural network is used to automatically extract key spatial features in the physical network state as input, and a target action is selected and executed through a policy network; a reward signal is calculated based on the action execution result and the network performance index, and a gradient descent algorithm is used to update the policy network parameters based on the reward signal; the trained target deep graph convolutional neural network and target policy network are obtained, and the target performance evaluation result is determined, thereby improving the solution efficiency and mapping quality of the VNE problem, while enhancing the adaptability, robustness and security of the network.

[0041] In an exemplary embodiment, using a deep graph convolutional neural network to automatically extract key spatial features in a physical network state as input includes:

[0042] S11, extracting multiple key spatial features based on network nodes, wherein the key spatial features include node resource utilization, sum of bandwidth utilization of links connected to the node, remaining resources, sum of remaining bandwidth of links connected to the node, node degree, average distance to other mapped nodes, and node proximity centrality;

[0043] S12, after the key spatial features are extracted at each layer, the rectified linear unit activation function is used.

[0044] In the present application embodiment, Figure 3 As shown in the figure, a deep graph convolutional neural network (DGCN) can be used to automatically extract key spatial features in the network. Seven key features based on nodes are extracted, namely, node resource utilization, sum of bandwidth utilization of links connected to nodes, remaining resources, sum of remaining bandwidth of links connected to nodes, node degree, average distance to other mapped nodes, and node proximity centrality. After feature extraction at each layer, the network is activated by a rectified linear unit RELU function.

[0045] In an exemplary embodiment, a deep graph convolutional neural network is used to automatically extract key spatial features in the physical network state as input, and a target action is selected and executed through a policy network, including:

[0046] S21, use the policy network to forward propagate the current state to obtain the probability distribution of each action;

[0047] S22, select the actual target action to be taken according to the output of the policy network;

[0048] S23, constructing a virtual label according to the target action and the output of the policy network;

[0049] S24, calculating the difference between the probability distribution of actually executing the target action and the probability distribution predicted by the policy network, and determining the loss function.

[0050] In the present application embodiment, Figure 4 As shown in Figure 1, when training the policy network, the cross entropy loss function can be used to adjust the network parameters through the back propagation algorithm to minimize the loss and optimize the policy network.

[0051] Optionally, the loss function can be calculated by using the policy network to forward propagate the current state to obtain the probability distribution of each action, and then select the actual action based on the output of the policy network. And based on the actual action taken and the output of the policy network, a virtual label (one-hot encoding) is constructed, where the position corresponding to the action taken is 1, and the rest is 0. Finally, the cross entropy loss is calculated, which is the difference between the probability distribution of the actual action and the probability distribution predicted by the policy network. In the classification problem, if there is a probability distribution y of a category (usually obtained by the Softmax function), and a target distribution y′ (usually the real label of the one-hot encoding), the cross entropy loss function can be expressed as:

[0052]

[0053] Here, M is the number of categories, y′ c is the one-hot encoding of the cth category in the target distribution (if the sample belongs to category y c , then y c =1, otherwise y c =0), y c is the probability that the model predicts that the sample belongs to category c. The parameters of the policy network can be updated through the gradient descent algorithm to reduce the value of the loss function. Through continuous training, the cross entropy is continuously reduced and the effect is continuously optimized.

[0054] In an exemplary embodiment, a reward signal is calculated based on the action execution result and the network performance indicator, and a gradient descent algorithm is used to update the policy network parameters based on the reward signal, including:

[0055] S31, calculating the mapping benefit after successful mapping and the mapping cost during the mapping process, and taking the ratio of the mapping benefit to the mapping cost as the score of the reward signal;

[0056] S32, passing the score of the reward signal to each weight of the deep graph convolutional neural network through back propagation and gradient descent algorithm for updating.

[0057] In an embodiment of the present application, in the process of reward function design, optionally, a reward function can be designed based on the ratio of revenue to cost to guide the optimization direction of the reinforcement learning algorithm and ensure that the algorithm can learn towards goals such as improving resource utilization and reducing latency. Here, mapping revenue (Revenue): that is, calculating the revenue after successful mapping, based on resource requirements, CPU and bandwidth requirements in VNR. Mapping cost (Cost): Calculate the cost of the mapping process, which is related to the amount of network resources used, such as links and node resources allocated to the virtual network. Revenue-to-cost ratio (Revenue / Cost Ratio): Evaluate the efficiency of the mapping strategy by comparing the mapping revenue and cost, and pass it to the various weights of DGCN as the final score through back propagation and gradient operations. The specific calculation formula is as follows:

[0058]

[0059] Here, n, n′, m, and m′ represent virtual nodes, nodes embedded in the actual network, VLs, and links embedded in the actual network, respectively.

[0060] From the above formula, we can see that since the virtual node and the embedded node have the same resources, the first half of this formula is always 1, so this formula can be changed to:

[0061]

[0062] Specifically, an optional example of the reward function calculation process is:

[0063] start

[0064] Define the mapping profit calculation function:

[0065] Mapping revenue = Calculated mapping revenue (VNR)

[0066] Mapping cost = calculate mapping cost (VNR, link mapping)

[0067] Reward = Mapping Benefit / Mapping Cost

[0068] Parameter update = optimization algorithm (current strategy, reward)

[0069] Finish

[0070] In an exemplary embodiment, before establishing the network system graph model based on the physical network and the virtual network, the method further includes:

[0071] S41, initialize the network parameters of the deep graph convolutional neural network for feature extraction and initialize the policy network parameters of the learning agent;

[0072] S42, after setting hyper parameters, perform an algorithm cycle, where the hyper parameters include a learning rate, a discount factor, and an exploration rate.

[0073] In an exemplary embodiment, after initializing the network parameters of the deep graph convolutional neural network for feature extraction and initializing the policy network parameters of the learning agent, the method further includes:

[0074] S51, uses an agent to perform actions in the environment to learn the optimal network mapping strategy;

[0075] S52, the agent interacts with the environment state to perform actions and receive feedback, where the environment state includes the topology of the underlying network, the resource usage of nodes and links, and the characteristics of the VNR. The action space contains all actions that the agent can perform and is used to allocate nodes and links to the VNR.

[0076] In the present application embodiment, Figure 4 As shown in the figure, in the algorithm design, the VNE problem can be modeled as a reinforcement learning (RL) problem, and the corresponding algorithm structure is designed. The agent can continuously learn and optimize its strategy by exploring and utilizing the environment to achieve effective network mapping and resource management. In this way, the algorithm can effectively learn and adapt to complex network environments and provide solutions to the VNE problem.

[0077] Optionally, DGCRL-VNE (Decorrelated Global Centralized Ranking Loss for Virtual Network Embedding) is a method for solving the virtual network embedding (VNE) problem, and its pseudo code is as follows:

[0078] initialization:

[0079] - Initialize DGCN network parameters for feature extraction

[0080] - Initialize the policy network parameters of the reinforcement learning agent

[0081] -Set hyperparameters such as learning rate, discount factor, exploration rate ε

[0082] Loop (for each virtual network request VNR):

[0083] 1.VNR login:

[0084] -Receive new virtual network requests (VNRs)

[0085] 2. Feature extraction:

[0086] -Use the DGCN network to extract features from the current physical network state

[0087] 3. Action selection:

[0088] -Use the extracted features as input and select an action through the policy network

[0089] - Implement a softmax strategy to balance exploration and exploitation

[0090] 4. Execute the action:

[0091] -Perform selected actions in the physical network, such as allocating nodes and links

[0092] 5. Reward calculation:

[0093] - Calculate reward signals based on action results and network performance metrics

[0094] 6. Termination condition check:

[0095] - Check if the termination condition is met

[0096] 7. Parameter update:

[0097] -Use gradient descent algorithm to update policy network parameters according to reward signal

[0098] - Such as using policy gradient method

[0099] 8. Performance feedback:

[0100] - Collect performance feedback such as mapping success rate, resource utilization, and mapping quality

[0101] 9. Hyperparameter Tuning:

[0102] -Adjust hyperparameters based on performance feedback

[0103] 10. Loop to step 1 until the termination condition is met

[0104] Output:

[0105] -Return the trained DGCN network and policy network, as well as the performance evaluation results

[0106] It should be noted that the agent is an algorithmic entity that performs actions in the environment to learn the optimal network mapping strategy. The environment represents the SN of the VNE problem, and the agent interacts with it to perform actions and receive feedback. The state space defines all the environmental states observed by the agent, which usually includes the topology of the underlying network, the resource usage of nodes and links, and the characteristics of the VNR, etc. The state changes continuously with the action. The action space contains all the actions that the agent can perform, such as the nodes and links that can be allocated to the VNR, the adjustment of resource allocation strategies, etc. Each action changes the current state and causes the transfer of the environment state. The reward function is a key part in reinforcement learning, which provides immediate feedback based on the actions performed by the agent and the results produced.

[0107] Finally, system integration and real-time mapping can be performed, that is, all components are integrated into one system to ensure smooth data flow and real-time decision-making, and realize rapid mapping and resource allocation of virtual networks.

[0108] Through this embodiment, dynamic mapping of virtual networks can be achieved, and utilization of network resources can be optimized, while maintaining rapid response and adaptability to changes in the network environment.

[0109] According to another aspect of an embodiment of the present application, a virtual network mapping device for implementing the above-mentioned virtual network mapping method based on improved deep reinforcement learning is also provided. Figure 5 is a structural diagram of an optional virtual network mapping device based on improved deep reinforcement learning according to an embodiment of the present application, such as Figure 5 As shown, the device may include:

[0110] An establishing unit 502 is used to establish a network system graph model based on the physical network and the virtual network, wherein the network nodes and links in the network system graph model are converted into multi-dimensional vectors for regularizing the network topology and constituting the system state;

[0111] An input unit 504 is used to automatically extract key spatial features in the physical network state using a deep graph convolutional neural network as input, select a target action through a policy network and execute it;

[0112] An updating unit 506, configured to calculate a reward signal according to the action execution result and the network performance indicator, and update the policy network parameters according to the reward signal using a gradient descent algorithm;

[0113] The determination unit 508 is used to obtain the trained target deep graph convolutional neural network and target policy network, and determine the target performance evaluation result.

[0114] It should be noted that the establishing unit 502 in this embodiment can be used to execute the above step S102, the input unit 504 in this embodiment can be used to execute the above step S104, the updating unit 506 in this embodiment can be used to execute the above step S106, and the determining unit 508 in this embodiment can be used to execute the above step S108.

[0115] Through the above modules, a network system graph model is established based on the physical network and the virtual network, wherein the network nodes and links in the network system graph model are converted into multidimensional vectors to regularize the network topology and constitute the system state; a deep graph convolutional neural network is used to automatically extract key spatial features in the physical network state as input, and the target action is selected and executed through the policy network; the reward signal is calculated based on the action execution results and network performance indicators, and the policy network parameters are updated according to the reward signal using the gradient descent algorithm; the trained target deep graph convolutional neural network and target policy network are obtained, and the target performance evaluation results are determined, thereby improving the solution efficiency and mapping quality of the VNE problem, while enhancing the adaptability, robustness and security of the network.

[0116] In an exemplary embodiment, the input unit comprises:

[0117] An extraction module is used to extract multiple key spatial features based on network nodes, wherein the key spatial features include node resource utilization, the sum of bandwidth utilization of links connected to the node, remaining resources, the sum of remaining bandwidth of links connected to the node, node degree, average distance to other mapped nodes, and node proximity centrality;

[0118] The activation module is used for each layer after the key spatial feature extraction, through the rectified linear unit activation function.

[0119] In an exemplary embodiment, the input unit comprises:

[0120] The acquisition module is used to use the policy network to forward propagate the current state and obtain the probability distribution of each action;

[0121] A selection module, used for selecting the target action actually taken according to the output of the policy network;

[0122] A construction module, configured to construct a virtual label according to the target action and the output of the policy network;

[0123] The calculation module is used to calculate the difference between the probability distribution of the actual execution of the target action and the probability distribution predicted by the policy network, and determine the loss function.

[0124] In an exemplary embodiment, the computing unit comprises:

[0125] A calculation module, used to calculate the mapping benefit after successful mapping and the mapping cost during the mapping process, and use the ratio of the mapping benefit to the mapping cost as the score of the reward signal;

[0126] An updating module is used to transfer the score of the reward signal to each weight of the deep graph convolutional neural network through a back propagation and gradient descent algorithm for updating.

[0127] In an exemplary embodiment, the apparatus further comprises:

[0128] An initialization unit, used to initialize the network parameters of the deep graph convolutional neural network for feature extraction and to initialize the policy network parameters of the learning agent;

[0129] A setting unit is used to set hyper parameters and then perform an algorithm cycle, wherein the hyper parameters include a learning rate, a discount factor, and an exploration rate.

[0130] In an exemplary embodiment, the apparatus further comprises:

[0131] A learning unit, configured to learn an optimal network mapping strategy by using the agent to perform actions in an environment;

[0132] An execution unit is used for the agent to interact with the environment state to perform actions and receive feedback, wherein the environment state includes the topology of the underlying network, the resource usage of nodes and links, and the characteristics of the VNR, and the action space contains all actions that the agent can perform and is used to allocate nodes and links to the VNR.

[0133] It should be noted here that the examples and scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiments. It should be noted that the above-mentioned modules as part of the device can run in a hardware environment and can be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0134] According to another aspect of the embodiments of the present application, a storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above virtual network mapping methods based on improved deep reinforcement learning in the embodiments of the present application.

[0135] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0136] S1, establishing a network system graph model based on the physical network and the virtual network, wherein the network nodes and links in the network system graph model are converted into multi-dimensional vectors for regularizing the network topology and constituting the system state;

[0137] S2, uses a deep graph convolutional neural network to automatically extract key spatial features from the physical network state as input, selects the target action through the policy network and executes it;

[0138] S3, calculates the reward signal based on the action execution results and network performance indicators, and uses the gradient descent algorithm to update the policy network parameters according to the reward signal;

[0139] S4, obtain the trained target deep graph convolutional neural network and target policy network, and determine the target performance evaluation result.

[0140] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0141] Among them, computer-readable storage media may include, but are not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0142] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned virtual network mapping method based on improved deep reinforcement learning is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0143] Figure 6 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 6 As shown, it includes a processor 602, a communication interface 604, a memory 606 and a communication bus 608, wherein the processor 602, the communication interface 604, and the memory 606 communicate with each other through the communication bus 608, wherein,

[0144] Memory 606, used to store computer programs;

[0145] The processor 602 is used to implement the following steps when executing the computer program stored in the memory 606:

[0146] S1, establishing a network system graph model based on the physical network and the virtual network, wherein the network nodes and links in the network system graph model are converted into multi-dimensional vectors for regularizing the network topology and constituting the system state;

[0147] S2, uses a deep graph convolutional neural network to automatically extract key spatial features from the physical network state as input, selects the target action through the policy network and executes it;

[0148] S3, calculates the reward signal based on the action execution results and network performance indicators, and uses the gradient descent algorithm to update the policy network parameters according to the reward signal;

[0149] S4, obtain the trained target deep graph convolutional neural network and target policy network, and determine the target performance evaluation result.

[0150] Optionally, the communication bus may be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The communication interface is used for communication between the electronic device and other devices.

[0151] The memory may include RAM, or may include nonvolatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0152] As an example, the memory 606 may include, but is not limited to, the establishment unit 502, the input unit 504, the update unit 506, and the determination unit 508 in the virtual network mapping device based on improved deep reinforcement learning. In addition, other module units in the virtual network mapping device based on improved deep reinforcement learning may also be included but are not limited to, which will not be repeated in this example.

[0153] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0154] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0155] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0156] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0157] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0158] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program code.

[0161] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0162] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0163] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A virtual network mapping method based on improved deep reinforcement learning, characterized in that: include: A network system graph model is established based on the physical network and the virtual network, wherein the network nodes and links in the network system graph model are converted into multi-dimensional vectors for regularizing the network topology and constituting the system state; A deep graph convolutional neural network is used to automatically extract key spatial features from the physical network state as input, and the target action is selected and executed through the policy network; Calculate reward signals based on action execution results and network performance indicators, and use the gradient descent algorithm to update the policy network parameters based on the reward signals; Obtain the trained target deep graph convolutional neural network and target policy network, and determine the target performance evaluation results.

2. The virtual network mapping method based on improved deep reinforcement learning according to claim 1, characterized in that: The method of automatically extracting key spatial features in the physical network state using a deep graph convolutional neural network as input includes: Extracting multiple key spatial features based on network nodes, wherein the key spatial features include node resource utilization, sum of bandwidth utilization of links connected to the node, remaining resources, sum of remaining bandwidth of links connected to the node, node degree, average distance to other mapped nodes, and node proximity centrality; After the key spatial features of each layer are extracted, a rectified linear unit activation function is used.

3. The virtual network mapping method based on improved deep reinforcement learning according to claim 1, characterized in that: The method of using a deep graph convolutional neural network to automatically extract key spatial features in the physical network state as input, selecting a target action through a policy network and executing the action includes: Use the policy network to forward propagate the current state to obtain the probability distribution of each action; According to the output of the policy network, selecting the target action actually taken; constructing a virtual label according to the target action and the output of the policy network; The difference between the probability distribution of the actual execution of the target action and the probability distribution predicted by the policy network is calculated to determine the loss function.

4. The virtual network mapping method based on improved deep reinforcement learning according to claim 1, characterized in that: The reward signal is calculated according to the action execution result and the network performance index, and the strategy network parameters are updated according to the reward signal using the gradient descent algorithm, including: Calculate the mapping benefit after successful mapping and the mapping cost during the mapping process, and use the ratio of the mapping benefit to the mapping cost as the score of the reward signal; The score of the reward signal is passed to each weight of the deep graph convolutional neural network through back propagation and gradient descent algorithm for updating.

5. The virtual network mapping method based on improved deep reinforcement learning according to claim 1, characterized in that: Before establishing the network system graph model based on the physical network and the virtual network, the method further includes: Initializing the network parameters of the deep graph convolutional neural network for feature extraction and initializing the policy network parameters of the learning agent; After setting the hyperparameters, the algorithm is cycled, wherein the hyperparameters include learning rate, discount factor, and exploration rate.

6. The virtual network mapping method based on improved deep reinforcement learning according to claim 5, characterized in that: After initializing the network parameters of the deep graph convolutional neural network for feature extraction and initializing the policy network parameters of the learning agent, the method further includes: Using the agent to perform actions in the environment to learn an optimal network mapping strategy; The agent interacts with the environment state to perform actions and receive feedback, wherein the environment state includes the topology of the underlying network, the resource usage of nodes and links, and the characteristics of the VNR, and the action space contains all actions that the agent can perform and is used to allocate nodes and links to the VNR.

7. A virtual network mapping device based on improved deep reinforcement learning, characterized in that: include: An establishing unit, used for establishing a network system graph model based on a physical network and a virtual network, wherein network nodes and links in the network system graph model are converted into multi-dimensional vectors for regularizing network topology and constituting a system state; An input unit, which is used to automatically extract key spatial features in the physical network state using a deep graph convolutional neural network as input, select a target action through a policy network, and execute it; An update unit, which is used to calculate the reward signal based on the action execution results and network performance indicators, and use the gradient descent algorithm to update the policy network parameters according to the reward signal; The determination unit is used to obtain the trained target deep graph convolutional neural network and target policy network, and determine the target performance evaluation result.

8. The virtual network mapping device based on improved deep reinforcement learning according to claim 1, characterized in that: The input unit comprises: An extraction module is used to extract multiple key spatial features based on network nodes, wherein the key spatial features include node resource utilization, the sum of bandwidth utilization of links connected to the node, remaining resources, the sum of remaining bandwidth of links connected to the node, node degree, average distance to other mapped nodes, and node proximity centrality; The activation module is used for each layer after the key spatial feature extraction, through the rectified linear unit activation function.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

Images