Deep Q learning-based virtual network function placement method with high and low orbit fusion

Through the high and low orbit converged virtual network function placement algorithm based on deep Q learning, communication delay and energy consumption in the integrated satellite Internet network in the world are optimized, synchronous satellite coverage is improved, and complex network topology design and orbital height are solved. Challenge, achieving efficient network performance and resource utilization.

CN118945669APending Publication Date: 2024-11-12BEIJING UNIV OF TECH
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202411164410.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the integrated satellite Internet network of the world, communication delay and bandwidth requirements caused by complex network topology design and different orbital heights are difficult to optimize, and the existing technology is difficult to effectively solve.

Method used

Adopt the high and low orbit converged virtual network function placement algorithm based on deep Q learning, optimize communication delay, energy consumption and synchronous satellite coverage by establishing satellite constellation network model and service function chain model, and optimize network services using dual deep Q network (DDQN) Deployment of the function chain.

Benefits of technology

It realizes efficient placement and management of network service function chains in multi-layer satellite networks, optimizes communication delay and resource consumption, improves the coverage rate of synchronous satellites, and improves network performance and resource utilization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945669A_ABST
    Figure CN118945669A_ABST
Patent Text Reader

Abstract

The invention discloses a high-orbit and low-orbit fused virtual network function placement method based on deep Q learning, which is oriented to the field of network function virtualization in the satellite internet and mainly adopts a deep reinforcement learning method to optimize the deployment of a network service function chain. The invention realizes a virtual network function placement algorithm with high and low orbit fusion based on deep Q learning. An SEC network model containing low earth orbit and geosynchronous orbit satellites is designed, a mathematical model for satellite constellation network service placement is established, and key performance indexes such as communication delay, energy consumption and geosynchronous satellite coverage rate are emphatically considered. An optimization algorithm based on the DDQN is developed, detailed definitions of states, actions, rewards and a neural network structure are given, and deployment of a network service function chain SFC is optimized through the DDQN of the double deep Q network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of satellite Internet technology, in particular to the application of network function virtualization (NFV) technology. Through deep reinforcement learning methods, the deployment of network service function chains is optimized to achieve optimization of key performance indicators such as communication delay, energy consumption and synchronous satellite coverage. Background Art

[0002] With the rapid development of information technology and communication technology, modern society is increasingly dependent on communication networks. The growth of global Internet users and the surge in data transmission demand require the continuous evolution of communication networks to meet higher speeds, wider coverage and stronger reliability. In this context, satellite Internet has become an important technology to make up for the lack of ground network coverage and realize communication connections in remote areas due to its unique advantages. It not only supports traditional communication services, but also plays a key role in emergency communications, global navigation, environmental monitoring and other fields.

[0003] The main features of satellite Internet include its wide coverage, support for broadcast and multicast communication methods, and high availability. These features enable satellite Internet to demonstrate unique advantages in wireless TV broadcasting, global positioning systems, and large-scale network coverage. Especially in areas with remote locations or complex terrain, satellite Internet can effectively supplement the deficiencies of ground infrastructure.

[0004] As global communication technology moves towards 6G, the role of satellite Internet becomes more important. 6G is expected to support higher data transmission rates and lower latency. To achieve this goal, the integration of satellite Internet becomes one of the key steps. In the 6G network architecture, satellite Internet can serve as an important supplement to the ground network, especially in achieving seamless coverage around the world. It can ensure that high-quality communication services can be enjoyed in remote areas and areas such as oceans that are difficult to cover with traditional networks. In addition, satellite networks also show their unique value in network redundancy design, disaster recovery and emergency response.

[0005] my country is making solid progress in the development of low earth orbit (LEO) internet satellites. With inspiration from the European commercial satellite development model, my country is gradually adopting a general satellite product development strategy based on market demand, while also emphasizing the permanent development and maintenance of the technical team. This new development model breaks through the traditional satellite development model, especially in terms of mass production, rapid deployment and cost reduction.

[0006] Nowadays, the development of satellite Internet is driven by a variety of technical factors, including Network Function Virtualization (NFV), Virtual Network Function (VNF) and Service Function Chain (SFC). These technologies not only improve the efficiency of satellite network management and operation, but also promote the flexibility and responsiveness of network services.

[0007] The integrated space-ground satellite Internet network integrates low-orbit satellite networks, synchronous satellites and ground facilities to form a unified network to provide Internet functional services. This network architecture aims to overcome the limitations of traditional ground networks and single satellite networks, and provide global Internet coverage and seamless connectivity. However, the design and implementation of this system face many challenges and difficulties, such as complex network topology design, different communication delays and bandwidth requirements caused by satellite networks at different orbital altitudes, etc. In this context, the network function virtualization (NFV) technology of the 5G core network can provide an effective solution for the optimization of the integrated space-ground satellite Internet network. The combination of the 5G core network and NFV technology improves the efficiency and flexibility of the network, supports multiple network slices, dynamic resource scheduling, edge computing and fault recovery, and ensures the optimization of network performance.

[0008] Based on the 5G core network NFV technology, the present invention proposes a virtual network function placement algorithm based on high-orbit and low-orbit fusion based on deep Q learning for the satellite Internet environment. Summary of the invention

[0009] This paper implements a virtual network function placement algorithm based on deep Q learning for high- and low-orbit fusion. A SEC network model containing low-Earth orbit and geosynchronous orbit satellites is designed, and a mathematical model for satellite constellation network service placement is established, focusing on key performance indicators such as communication delay, energy consumption and synchronous satellite coverage. An optimization algorithm based on DDQN is developed, and detailed definitions of states, actions, rewards and neural network structures are given. The deployment of network service function chains (SFCs) is optimized through a dual deep Q network (DDQN).

[0010] The technical solution adopted by the present invention is a virtual network function placement method for high-orbit and low-orbit fusion based on deep Q learning. First, a satellite constellation network service placement model is established, including a SEC network model and a service function chain model; the SEC network model describes the distribution of satellites on different orbits and their computing resource limitations, and the service function chain model defines a series of virtual network functions VNFs that ground users' network requests need to pass through; the optimization goal of network service function chain placement is to minimize total delay and total energy consumption, and improve the coverage of synchronous satellites; each SFC request is regarded as a cycle, and node deployment is a step. The decision of the edge is solved by the Dijkstra algorithm, and the decision of the node is converted into a Markov decision process MDP problem; the state of the agent is represented as physical computing resource information, the action represents the selection of a physical node for VNF deployment, and the reward setting takes into account resources, energy consumption, delay and high-orbit coverage factors; the neural network structure adopts a fully connected layer, the input is the physical network state matrix, and the output is the Q value of each action; finally, the model is optimized through the training and testing process, and the experience replay pool is used to store samples, and the target Q network parameters are regularly updated to optimize the placement of the network service function chain.

[0011] Step 1: Establish a satellite constellation network service placement model;

[0012] Step 1.1: Two models;

[0013] Establish satellite constellation network model and service function chain model, and clearly define the problems to be solved and the optimization goals from the physical level and network level respectively;

[0014] Satellite edge computing SEC network model and service function chain model;

[0015] 1)SEC network model;

[0016] The satellite constellation network includes a SEC network; define the SEC network G = (V, E), where the satellites are distributed in orbits of different altitudes, V is a group of satellites, and E is a set of links between low earth orbit satellites, i.e., LEO satellites, and geosynchronous satellites, i.e., GEO satellites; each LEO satellite and GEO satellite orbits the earth in an orbit of a given altitude to provide various network services to ground users; let v j ∈V is a satellite, where 1≤j≤|V|; each satellite v j Each of them is equipped with a certain amount of computing resources to implement various VNFs in the form of software running in virtual machines or containers; j For satellite v j The amount of available computing resources provided at time t; the NFV request of the ground user is selected to be sent to the satellite with the closest transmission distance, and the satellites communicate with each other through laser inter-satellite links; at time t, the SEC network topology is static;

[0017] 2) Service function chain model;

[0018] In current communication networks, the network request data of each ground user needs to be processed by a series of virtual network functions (VNFs). VNFs together form a service function chain that implements a network service.

[0019] Step 1.2: Problem definition;

[0020] By establishing the SEC network model and service function chain model, the optimization problem of network service function chain placement is defined;

[0021] In a given SEC network, the data of each network request issued by the user at a specific time t needs to be processed by the service function chain located in the satellite; the total delay and total energy consumption are the key performance indicators of the SEC network processing network service function chain requests: (1) the total delay is the sum of the total transmission delay of the link; (2) the total energy consumption is the difference between the sum of the energy consumption of all VNF requests in the deployed link and the satellite energy replenishment;

[0022] This gives definitions of the relevant variables to express the problem mathematically:

[0023] (1) Performance indicators:

[0024] Let δ D , δ E are the delay and energy consumption of processing SFC requests by the SEC network respectively; let δ G It is an integer in the range of [0,1], indicating the coverage of the synchronous satellite;

[0025] (2) Decision variables:

[0026] Let binary decision variables Represents service function chain SC k The mth VNF function f in m Deployed on LEO satellite v at time t j Above; let the binary decision variable z p , indicating the request R k The data is routed through path p at time t; let n be the number of physical nodes on which VNF deployment has been completed;

[0027] (3) Other parameters

[0028] Let f m is the computing resource required by the VNF; let α and Charge represent the power consumption coefficient and energy replenishment coefficient of the physical node respectively; the delay in path p is represented by delay p express;

[0029] (4) Overview

[0030] Let P be the network service function chain placement problem in the SEC network solved by the algorithm; let δ D , δ E are the delay and energy consumption of processing SFC requests by the SEC network respectively; let δ G is an integer in the range of [0,1], indicating the coverage of the synchronous satellite; let vj∈V be a satellite, where 1≤j≤|V|, f m Represents service function chain SC k The mth VNF function in ; binary decision variable Represents service function chain SC k The mth VNF function f in m is deployed on LEO satellite vj at time t; Ck represents the computing resource capacity of the satellite; the binary decision variable zp represents the service function chain SC k All VNF functions in the final determination have only one or no request path placed; let fm be the computing resources required by VNF; let α and Charge represent the power consumption coefficient and energy replenishment coefficient of the physical node respectively; the delay in path p is represented by delayp; G represents the synchronous satellite, where the synchronous satellite number is set to be after all LEO satellites;

[0031] Taking into account the computing resource capacity constraints on each satellite, the algorithm must be able to solve the optimal path to minimize the delay and energy consumption of processing network service function chain SFC requests and improve the coverage of synchronous satellites; the optimization goal of network service function chain placement in the SEC network is expressed as:

[0032] (P):minδ D +minδ E +maxδ G (1)

[0033] Constraints:

[0034]

[0035] The calculation method of performance indicators is:

[0036] δ D =z p ·delay p (5)

[0037]

[0038] Among them, (1) shows that the optimization goal of the algorithm is to minimize the delay and energy consumption of processing service function chain SFC requests and improve the coverage of synchronous satellites; Inequality (2) shows that the computing resource capacity of the satellite cannot be violated; Equation (3) shows that each VNF can only be placed on the satellite of the request path, where Φ(p) is the set of satellites in path p; Constraint (4) represents the variable y j m (t) and z p is a binary decision variable; Equation (5) shows that the total delay δ D is the sum of the total transmission delay of link p; Equation (6) shows that the total energy consumption is the sum of the energy consumption of all VNF requests minus the difference in energy replenishment; Equation (7) shows that if there is a synchronous satellite in the service function chain placement path, the synchronous satellite coverage rate is 1, otherwise it is 0.

[0039] Step 2 includes:

[0040] Step 2.1: Problem Statement Based on Reinforcement Learning

[0041] 1) Status setting

[0042] The state of the agent is represented as the computing resource information of the physical network, so it is necessary to convert the relevant state in the physical network into a discrete state matrix; the definition of the state matrix is ​​as follows: the physical network node feature matrix N represents the computing resource information of the physical node, and its form is N = [R], where R represents the available computing resources of the node; each state matrix represents the physical network state at a certain moment;

[0043] 2) Action settings

[0044] The action of the agent is represented by selecting a physical node to deploy the current virtual network function VNF on the node; the action settings are as follows:

[0045] (1) Action space;

[0046] The action space defines all possible actions that the agent can take at each time step; the action space is the set of all physical nodes, and the agent needs to select a physical node to deploy the current VNF; therefore, the action space A is the action set {0, 1, 2, N-1}, where N represents the number of physical nodes; each action a∈A corresponds to the index of a physical node;

[0047] (2) Action definition;

[0048] Each action represents the selection of a physical node for VNF deployment; action a represents the deployment of the VNF to be deployed to the physical node with index a;

[0049] (3) Action selection strategy;

[0050] During the training process, the agent selects actions through the following two strategies: first, exploration: randomly selecting a physical node for VNF deployment to enhance the exploration of the environment; then, exploitation: using the Q network to select the physical node with the largest Q value for VNF deployment to make the best decision using the existing information; the balance between exploration and exploitation is achieved through the ∈-greedy strategy;

[0051] (4) Action execution;

[0052] After selecting an action, the agent executes the action in the environment, updates the environment state and obtains rewards. First, the agent executes the action to deploy the current VNF ​​to the selected physical node, then updates the state, that is, updates the node features of the physical network diagram, and calculates the reward based on the deployment results. Finally, the agent determines whether the current VNF ​​deployment is completed and ends the execution of this action.

[0053] 3) Reward setting;

[0054] The reward value is an important indicator for the agent to judge the effectiveness of the action; the setting of rewards in the algorithm takes into account whether resources are sufficient, energy consumption, latency, and high-orbit coverage reward factors;

[0055] (1) Failure reward;

[0056] If the computing resources of the selected physical node are insufficient to deploy the current virtual network function VNF, a large negative reward (-100) will be returned and the deployment will be marked as failed;

[0057] (2) Success rewards;

[0058] After the current VNF ​​is successfully deployed, the list of recorded deployment plans will be updated and the VNF index currently being processed will be increased. For latency, when the number of deployed nodes exceeds 1, the Dijkstra algorithm is used to calculate the shortest path latency and the maximum latency delay is updated. For energy consumption, the actual energy consumption of the current deployment and the energy consumption e in the reward will be calculated first, and then accumulated into the total energy consumption energy. In addition, if any of the deployed nodes is within the range of a high-orbit satellite, a high-orbit coverage reward will be given.

[0059] Finally, the total reward is calculated based on energy consumption, latency, and high-orbit coverage.

[0060] 4) Neural network structure;

[0061] The input of the neural network structure is the physical network state matrix, and the input size is N×1, where N is the number of nodes in the physical network, indicating the computing resources of each node; the first layer is a fully connected layer, using the linear rectification function ReLU as the activation function, and the size is 128; the second layer also uses the linear rectification function ReLU as the activation function, and the size is also 128; the output size is N×1, indicating the Q value of each action, and the node with the largest Q value is selected as the deployment position;

[0062] Step 2.2: The training and testing process of the virtual network function placement algorithm based on high and low orbit fusion based on deep Q learning is as follows;

[0063] First, the training and testing environments are initialized. The algorithm will define the physical network, including the features of nodes and edges, and generate the feature matrix and adjacency matrix of the physical network when the environment is initialized. Next, an experience replay pool is created to store samples collected during training. Then, the local Q network and the target Q network are defined, and the parameters of the target Q network are set to be consistent with those of the local Q network.

[0064] Then the model is trained and tested. During the training process, the environment, experience replay pool and Q network are initialized first. At the beginning of each training round, the environment is reset and the state of the current physical network is obtained. Each VNF of the current SFC is deployed in turn, and the ∈-greedy strategy is used to select the action, that is, the deployment location. After the action is executed, the environment state is updated, the reward is calculated, and the sample is stored in the experience replay pool. At the same time, the loss of the target Q value and the current Q value is calculated, and the loss function is minimized by the gradient descent algorithm. The Q network parameters are updated and the target Q network parameters are updated regularly. Then, samples are sampled from the experience replay pool for training, and the current training round ends when all VNF nodes are deployed. Finally, the average reward and optimization target of the current round are verified, and the optimal model is saved. During the test process, the current environment state is obtained, and then the model is used to predict the Q value of each action, and the action with the largest Q value is selected as the deployment location. When all VNF nodes are deployed, the model test ends.

[0065] Based on the 5G core network NFV technology, this paper proposes a deep Q-learning-based high-orbit and low-orbit fusion virtual network function placement algorithm for the satellite Internet environment. The research will focus on how to efficiently place and manage network service function chains in multi-layer satellite networks to optimize network performance and resource utilization. This includes the study of service function chain optimization and resource allocation strategies to minimize communication delays and resource consumption in complex satellite network topologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 An example diagram of a SEC network model.

[0067] Figure 2 Service function chain diagram composed of VNFs.

[0068] Figure 3 Diagram of the neural network structure.

[0069] Figure 4 Algorithm training and testing flow chart. DETAILED DESCRIPTION

[0070] The main process can be divided into the following two steps: establishing a satellite constellation network service placement model and specific algorithm design.

[0071] Step 1: Build a satellite constellation network service placement model

[0072] Step 1.1: Two Models

[0073] To study the placement of service function chains in satellite constellation networks, it is necessary to establish a satellite constellation network model and a service function chain model at the same time, and to clearly define the problems that this study wants to solve and the optimization goals from the physical level and network level respectively.

[0074] The following introduces the Satellite Edge Computing (SEC) network model and service function chain model.

[0075] 1.SEC Network Model

[0076] Attached Figure 1 The figure shows a satellite constellation network, which includes an example of a SEC network. Define the SEC network G = (V, E), where the satellites are distributed in orbits of different altitudes, V is a group of satellites, and E is a set of links between low earth orbit satellites (LEO satellites) and geostationary earth orbit satellites (GEO satellites). Each LEO satellite and GEO satellite orbits the earth at an orbit of a given altitude, providing various network services to ground users. Let v j ∈V is a satellite, where 1≤j≤|V|. Each satellite v j Each satellite is equipped with a certain amount of computing resources to implement various VNFs in the form of software running in virtual machines or containers. Due to size constraints, the satellite has limited computing resources and cannot process all NFV requests issued by ground users. Therefore, let C j For satellite v j The amount of available computing resources provided at time t. The NFV request of the ground user is selected to be sent to the satellite with the closest transmission distance, and the satellites communicate with each other through laser inter-satellite links. At time t, the SEC network topology is static.

[0077] 2. Service Function Chain Model

[0078] In the current communication network, the network request data of each ground user needs to be processed by a series of virtual network functions (VNFs), such as Figure 2 These VNFs together form a service function chain that can implement a network service.

[0079] Step 1.2: Problem Definition

[0080] By building the SEC network model and service function chain model, the optimization problem of network service function chain placement can now be precisely defined.

[0081] In a given SEC network, the data of each network request issued by the user at a specific time t needs to be processed by the service function chain located in the satellite. The total delay and total energy consumption are the key performance indicators of the SEC network processing network service function chain requests: (1) The total delay is the sum of the total transmission delay of the link; (2) The total energy consumption is the difference between the sum of the energy consumption of all VNF requests in the deployed link and the satellite energy replenishment (solar charging).

[0082] This gives definitions of the relevant variables to express the problem mathematically:

[0083] (4) Performance indicators:

[0084] Let δ D , δ E are the delay and energy consumption of the SEC network processing SFC requests respectively. Let δ G It is an integer in the range of [0,1], indicating the coverage of the synchronous satellite.

[0085] (5) Decision variables:

[0086] Let binary decision variables Represents service function chain SC k The mth VNF function f in m Deployed on LEO satellite v at time t j Let the binary decision variable z p , indicating the request R k The data of is routed through path p at time t. Let n be the number of physical nodes where VNF deployment has been completed.

[0087] (6) Other parameters

[0088] Let f m is the computing resource required by the VNF. Let α and Charge represent the power consumption coefficient and energy replenishment (solar charging) coefficient of the physical node respectively. The delay in path p is represented by delay p express.

[0089] Taking into account the computing resource capacity constraints on each satellite, the algorithm must be able to solve the optimal path to minimize the delay and energy consumption of processing network service function chain (SFC) requests and improve the coverage of synchronous satellites. Therefore, the optimization goal of network service function chain placement in the SEC network can be expressed as:

[0090] (P):minδ D +minδ E +maxδ G (1)

[0091] Constraints:

[0092]

[0093] The calculation method of performance indicators is:

[0094] δ D =z p ·delay p (5)

[0095]

[0096]

[0097] Among them, (1) shows that the optimization goal of the algorithm is to minimize the delay and energy consumption of processing service function chain (SFC) requests and improve the coverage of synchronous satellites. Inequality (2) shows that the computing resource capacity of the satellite cannot be violated. Equation (3) shows that each VNF can only be placed on the satellite of the request path, where Φ(p) is the set of satellites in path p. Constraint (4) means that the variable y j m (t) and z p is a binary decision variable. Equation (5) shows that the total delay δ D is the sum of the total transmission delay of link p. Equation (6) shows that the total energy consumption is the sum of the energy consumption of all VNF requests minus the difference in energy replenishment (solar charging). Equation (7) shows that if there is a synchronous satellite in the service function chain placement path, the synchronous satellite coverage rate is 1, otherwise it is 0.

[0098] Step 2: Algorithm Design

[0099] Step 2.1: Problem Statement Based on Reinforcement Learning

[0100] The optimization problem proposed in step 1 is an integer linear programming problem that is difficult to solve. In order to solve this problem, a virtual network function placement algorithm based on high-low rail fusion based on deep Q learning was developed. The algorithm divides the problem into node decision and edge decision, and regards each SFC request as a cycle (episode). The entire cycle includes the deployment of multiple nodes, and each node deployment is regarded as a step (step). Among them, the edge decision is solved by the Dijkstra algorithm, and the node decision is converted into a Markov decision process (MDP) problem. The definitions of specific states, actions and rewards and the neural network structure are given below.

[0101] 1. Status settings

[0102] The state of the agent is represented as the computing resource information of the physical network, so it is necessary to convert the relevant state in the physical network into a discrete state matrix. The definition of the state matrix is ​​as follows: The physical network node feature matrix N represents the computing resource information of the physical node, and its form is N = [R], where R represents the available computing resources of the node. Each state matrix represents the state of the physical network at a certain moment.

[0103] 2. Action Settings

[0104] The action of the agent is represented by selecting a physical node to deploy the current virtual network function (VNF) on the node. The action settings are as follows:

[0105] (1) Action Space

[0106] The action space defines all possible actions that the agent can take at each time step. In this problem, the action space is the set of all physical nodes, and the agent needs to select a physical node to deploy the current VNF. Therefore, the action space A: the action set {0,1,2,N-1}, where N represents the number of physical nodes. Each action a∈A corresponds to the index of a physical node.

[0107] (2) Action definition

[0108] Each action represents the selection of a physical node for VNF deployment. Specifically, action a represents the deployment of the current VNF ​​to be deployed to the physical node with index a.

[0109] (3) Action selection strategy

[0110] During training, the agent selects actions through the following two strategies: First, exploration: randomly select a physical node for VNF deployment to enhance exploration of the environment. Then, exploitation: use the Q network to select the physical node with the largest Q value for VNF deployment to make the best decision using the existing information. The balance between exploration and exploitation is achieved through the ∈-greedy strategy.

[0111] (4) Action Execution

[0112] After selecting an action, the agent executes the action in the environment, updates the environment state and obtains rewards. Specifically, the agent first executes the action, deploys the current VNF ​​to the selected physical node, then updates the state, that is, updates the node features of the physical network diagram, and then calculates the reward based on the deployment results. Finally, the agent determines whether the current VNF ​​deployment is completed and ends the execution of this action.

[0113] 3. Reward setting

[0114] The reward value is an important indicator for the agent to judge the effectiveness of the action. The setting of the reward in the algorithm mainly considers factors such as whether the resources are sufficient, energy consumption, latency, and high-orbit coverage rewards.

[0115] (1) Failure Reward

[0116] If the computing resources of the selected physical node are insufficient to deploy the current virtual network function (VNF), a large negative reward (-100) will be returned and the deployment will be marked as failed.

[0117] (2) Success Rewards

[0118] After the current VNF ​​is successfully deployed, the list of recorded deployment plans will be updated and the VNF index currently being processed will be increased. For latency, when the number of deployed nodes exceeds 1, the Dijkstra algorithm is used to calculate the shortest path latency and update the maximum latency delay. For energy consumption, the actual energy consumption of the current deployment and the energy consumption e in the reward will be calculated first, and then accumulated into the total energy consumption energy. In addition, if any of the deployed nodes is within the range of a high-orbit satellite, a high-orbit coverage reward will be given.

[0119] Finally, the total reward is calculated based on comprehensive energy consumption, latency, and high-orbit coverage. The reward setting fully considers a variety of influencing factors to ensure the comprehensiveness and balance of the optimization goals in order to achieve better training results.

[0120] 4. Neural Network Structure

[0121] The neural network structure is attached Figure 3:The input of the model is the physical network state matrix, and the input size is N×1, where N is the number of nodes in the physical network, indicating the computing resources of each node. The first layer is a fully connected layer, using the linear rectifier function (ReLU) as the activation function, and the size is 128. The second layer also uses the linear rectifier function (ReLU) as the activation function, and the size is also 128. The output size of the model is N×1, which represents the Q value of each action (deployment location). The algorithm will select the node with the largest Q value as the deployment location.

[0122] Step 2.2: Algorithm flow and pseudocode

[0123] 1. Algorithm Process

[0124] The training and testing process of the virtual network function placement algorithm based on high and low orbit fusion based on deep Q learning is shown in the attached Figure 4 .

[0125] First, the training and testing environments are initialized. The algorithm will define the physical network, including the features of nodes and edges, and generate the feature matrix and adjacency matrix of the physical network when the environment is initialized. Next, an experience replay pool is created to store samples collected during training. Then, the local Q network and the target Q network are defined, and the parameters of the target Q network are set to be consistent with those of the local Q network.

[0126] Then the model is trained and tested. Figure 4 As shown in the left figure, during the training process, the environment, experience replay pool and Q network are first initialized. At the beginning of each training round, the environment is reset and the state of the current physical network is obtained. Each VNF of the current SFC is deployed in turn, and the ∈-greedy strategy is used to select the action (deployment location). After the action is executed, the environment state is updated, the reward is calculated, and the sample is stored in the experience replay pool. At the same time, the loss of the target Q value and the current Q value is calculated, and the loss function is minimized by the gradient descent algorithm. The Q network parameters are updated and the target Q network parameters are updated regularly. Then, samples are sampled from the experience replay pool for training, and the current training round ends when all VNF nodes are deployed. Finally, the average reward and optimization target of the current round are verified, and the optimal model is saved. As shown in the attached figure Figure 4 As shown in the figure on the right, during the test, the current environment state is first obtained, then the model is used to predict the Q value of each action, and the action with the largest Q value is selected as the deployment location. When all VNF nodes are deployed, the model test ends.

[0127] 2. Algorithm pseudocode

[0128] enter:

[0129] config: contains configuration parameters (training round N, target network update round n, number of VNFs T,

[0130] Experience pool capacity M, sampling size m)

[0131] physical_nodes: list of nodes in the physical network

[0132] env, env_test: training and testing environment objects (including status and methods)

[0133] Output:

[0134] best_reward_model: the best performing model (including the optimal neural network weight parameters)

[0135] 1. Initialize the environment, experience replay pool and Q network, the weight parameter θ of the prediction network, and the weight parameter θ' of the target network

[0136] 2.for elpisode_num=1 to N do:

[0137] 3. Initialize round reward reward = 0

[0138] 4. Load training data and reset the environment

[0139] 5. for t=1 to T do:

[0140] 6. Get the current state features sfcr_features and physical_features

[0141] 7. Use ∈-greedy strategy to select actions (deployment locations)

[0142] 8. Execute the action and get the next state next_obs and reward reward

[0143] 9. Replace {S t ,a t ,S t+1 ,R t}Deposit and update the experience replay pool

[0144] 10.

[0145] 11. Randomly sample m samples from the experience replay pool and calculate the target Q value Q target and predicted Q value Q expected ,

[0146] 12. Calculate the MSE loss function loss and use the Adam optimization algorithm to update the weight parameter θ of the prediction network

[0147] 13. end if

[0148] 14. Update the target Q network every n steps, θ'=θ

[0149] 15. endfor

[0150] 16. Verify the average reward and optimization target of the current round and save the optimal model

[0151] 17.endfor.

Claims

1. A virtual network function placement method based on high- and low-orbit fusion based on deep Q-learning, Features: Firstly, a satellite constellation network service placement model is established, including the SEC network model and the service function chain model. The SEC network model describes the distribution of satellites in different orbits and their computing resource constraints, while the service function chain model defines a series of virtual network functions (VNFs) that ground users' network requests need to go through. The optimization goal of network service function chain placement is to minimize the total delay and total energy consumption and improve the coverage of synchronous satellites. Each SFC request is regarded as a cycle and node deployment is a step. The decision of the edge is solved by the Dijkstra algorithm, and the decision of the node is converted into a Markov decision process (MDP) problem. The state of the agent is represented by physical computing resource information, and the action represents the selection of a physical node for VNF deployment. The reward setting takes into account resources, energy consumption, delay and high orbit coverage factors. The neural network structure adopts a fully connected layer, with the input being the physical network state matrix and the output being the Q value of each action. Finally, the model is optimized through the training and testing process, and the experience replay pool is used to store samples, and the target Q network parameters are updated regularly to optimize the placement of the network service function chain.

2. According to the method for placing virtual network functions based on high and low orbit fusion based on deep Q learning in claim 1, Features: Step 1: Establish a satellite constellation network service placement model; Step 1.1: Two models; Establish satellite constellation network model and service function chain model, and clearly define the problems to be solved and the optimization goals from the physical level and network level respectively; Satellite edge computing SEC network model and service function chain model; 1)SEC network model; The satellite constellation network includes a SEC network; define the SEC network G = (V, E), where the satellites are distributed in orbits of different altitudes, V is a group of satellites, and E is a set of links between low earth orbit satellites, i.e., LEO satellites, and geosynchronous satellites, i.e., GEO satellites; each LEO satellite and GEO satellite orbits the earth in an orbit of a given altitude to provide various network services to ground users; let v j ∈V is a satellite, where 1≤j≤|V|; each satellite v j Each of them is equipped with a certain amount of computing resources to implement various VNFs in the form of software running in virtual machines or containers; j For satellite v j The amount of available computing resources provided at time t; the NFV request of the ground user is selected to be sent to the satellite with the closest transmission distance, and the satellites communicate with each other through laser inter-satellite links; at time t, the SEC network topology is static; 2) Service function chain model; In current communication networks, the network request data of each ground user needs to be processed by a series of virtual network functions (VNFs). VNFs together form a service function chain that implements a network service. Step 1.2: Problem definition; By establishing the SEC network model and service function chain model, the optimization problem of network service function chain placement is defined; In a given SEC network, the data of each network request issued by the user at a specific time t needs to be processed by the service function chain located in the satellite; the total delay and total energy consumption are the key performance indicators of the SEC network processing network service function chain requests: (1) the total delay is the sum of the total transmission delay of the link; (2) the total energy consumption is the difference between the sum of the energy consumption of all VNF requests in the deployed link and the satellite energy replenishment; This gives definitions of the relevant variables to express the problem mathematically: (1) Performance indicators: Let δ D , δ E are the delay and energy consumption of processing SFC requests by the SEC network respectively; let δ G It is an integer in the range of [0,1], indicating the coverage of the synchronous satellite; (2) Decision variables: Let binary decision variables Represents service function chain SC k The mth VNF function f in m Deployed on LEO satellite v at time t j Above; let the binary decision variable z p , indicating the request R k The data is routed through path p at time t; let n be the number of physical nodes on which VNF deployment has been completed; (3) Other parameters; Let f m is the computing resource required by the VNF; let α and Charge represent the power consumption coefficient and energy replenishment coefficient of the physical node respectively; the delay in path p is represented by delay p express; (4) Overview; Let the network service function chain placement problem in the SEC network solved by this method be P; let δ D , δ E are the delay and energy consumption of processing SFC requests by the SEC network respectively; let δ G is an integer in the range of [0,1], indicating the coverage of the synchronous satellite; let vj∈V be a satellite, where 1≤j≤|V|, f m Represents service function chain SC k The mth VNF function in ; binary decision variable Represents service function chain SC k The mth VNF function f in m It is deployed on LEO satellite vj at time t; Ck represents the computing resource capacity of the satellite; Binary decision variable zp, representing the service function chain SC k All VNF functions in the final determination have only one or no request path placed; let fm be the computing resources required by VNF; let α and Charge represent the power consumption coefficient and energy replenishment coefficient of the physical node respectively; the delay in path p is represented by delayp; G represents the synchronous satellite, and the synchronous satellite number is set to be after all LEO satellites; The optimization goal of placing the network service function chain in the SEC network is expressed as: (P):minδ D +minδ E +maxδ G (1) Constraints: The calculation method of performance indicators is: δ D =z p ·delay p (5) Among them, (1) shows that the optimization goal is to minimize the delay and energy consumption of processing service function chain SFC requests and improve the coverage of synchronous satellites; Inequality (2) shows that the computing resource capacity of the satellite cannot be violated; Equation (3) shows that each VNF can only be placed on the satellite of the request path, where Φ(p) is the set of satellites in path p; Constraint (4) means that the variable y j m (t) and z p is a binary decision variable; Equation (5) shows that the total delay δ D is the sum of the total transmission delay of link p; Equation (6) shows that the total energy consumption is the sum of the energy consumption of all VNF requests minus the difference in energy replenishment; Equation (7) shows that if there is a synchronous satellite in the service function chain placement path, the synchronous satellite coverage rate is 1, otherwise it is 0.

3. According to claim 1, a method for placing virtual network functions based on high and low orbit fusion based on deep Q learning, Features: Step 2 includes: Step 2.1: Problem description based on reinforcement learning; 1) Status setting; The state of the agent is represented as the computing resource information of the physical network, so it is necessary to convert the relevant state in the physical network into a discrete state matrix; the definition of the state matrix is ​​as follows: the physical network node feature matrix N represents the computing resource information of the physical node, and its form is N = [R], where R represents the available computing resources of the node; each state matrix represents the physical network state at a certain moment; 2) Action settings; The action of the agent is represented by selecting a physical node to deploy the current virtual network function VNF on the node; the action settings are as follows: (1) Action space; The action space defines all possible actions that the agent can take at each time step; the action space is the set of all physical nodes, and the agent needs to select a physical node to deploy the current VNF; therefore, the action space A is the action set {0, 1, 2, N-1}, where N represents the number of physical nodes; each action a∈A corresponds to the index of a physical node; (2) Action definition; Each action represents the selection of a physical node for VNF deployment; action a represents the deployment of the VNF to be deployed to the physical node with index a; (3) Action selection strategy; During the training process, the agent selects actions through the following two strategies: first, exploration: randomly selecting a physical node for VNF deployment to enhance the exploration of the environment; then, exploitation: using the Q network to select the physical node with the largest Q value for VNF deployment to make the best decision using the existing information; the balance between exploration and exploitation is achieved through the ∈-greedy strategy; (4) Action execution; After selecting an action, the agent executes the action in the environment, updates the environment state and obtains rewards. First, the agent executes the action to deploy the current VNF ​​to the selected physical node, then updates the state, that is, updates the node features of the physical network diagram, and calculates the reward based on the deployment results. Finally, the agent determines whether the current VNF ​​deployment is completed and ends the execution of this action. 3) Reward setting; The reward value is an important indicator for the agent to judge the effectiveness of the action; the setting of the reward takes into account whether the resources are sufficient, energy consumption, latency, and high-orbit coverage reward factors; (1) Failure reward; If the computing resources of the selected physical node are insufficient to deploy the current virtual network function VNF, a large negative reward (-100) will be returned and the deployment will be marked as failed; (2) Success rewards; After the current VNF ​​is successfully deployed, the list of recorded deployment plans will be updated and the VNF index currently being processed will be increased. For latency, when the number of deployed nodes exceeds 1, the Dijkstra algorithm is used to calculate the shortest path latency and the maximum latency delay is updated. For energy consumption, the actual energy consumption of the current deployment and the energy consumption e in the reward will be calculated first, and then accumulated into the total energy consumption energy. In addition, if any of the deployed nodes is within the range of a high-orbit satellite, a high-orbit coverage reward will be given. Finally, the total reward is calculated based on energy consumption, latency, and high-orbit coverage. 4) Neural network structure; The input of the neural network structure is the physical network state matrix, and the input size is N×1, where N is the number of nodes in the physical network, indicating the computing resources of each node; the first layer is a fully connected layer, using the linear rectification function ReLU as the activation function, and the size is 128; the second layer also uses the linear rectification function ReLU as the activation function, and the size is also 128; the output size is N×1, indicating the Q value of each action, and the node with the largest Q value is selected as the deployment position; Step 2.2: The training and testing process of the virtual network function placement method based on high and low orbit fusion based on deep Q learning is as follows; First, the training and testing environments are initialized; the physical network is defined, including the features of nodes and edges, and the feature matrix and adjacency matrix of the physical network are generated when the environment is initialized; then, an experience replay pool is created to store the samples collected during the training process; then, the local Q network and the target Q network are defined, and the parameters of the target Q network are set to be consistent with those of the local Q network; Then the model is trained and tested. During the training process, the environment, experience replay pool and Q network are initialized first. At the beginning of each training round, the environment is reset and the state of the current physical network is obtained. Each VNF of the current SFC is deployed in turn, and the ∈-greedy strategy is used to select the action, that is, the deployment location. After the action is executed, the environment state is updated, the reward is calculated, and the sample is stored in the experience replay pool. At the same time, the loss of the target Q value and the current Q value is calculated, and the loss function is minimized by the gradient descent algorithm. The Q network parameters are updated and the target Q network parameters are updated regularly. Then, samples are sampled from the experience replay pool for training, and the current training round ends when all VNF nodes are deployed. Finally, the average reward and optimization target of the current round are verified, and the optimal model is saved. During the test process, the current environment state is obtained, and then the model is used to predict the Q value of each action, and the action with the largest Q value is selected as the deployment location. When all VNF nodes are deployed, the model test ends.

Citation Information

Cited By

  • VNF multi-dimensional resource demand prediction method based on synchronous and asynchronous space-time characteristic fusion

    CN119740095A

  • A VNF-based multidimensional resource demand forecasting method based on the fusion of synchronous and asynchronous spatiotemporal characteristics.

    CN119740095B

  • Mixed edge cloud-oriented satellite-ground fusion network service function chain arrangement method

    CN119967446A

  • High-reliability service function chain deployment method based on dual deep Q network

    CN120389784A

  • High-reliability service function chaining deployment method based on double deep Q network

    CN120389784B