SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning

Abstract

The invention relates to an SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning, and the method comprises the following steps: S1, building a three-layer SDN / NFV satellite network architecture: building a satellite network architecture with an application management layer, a control layer and a data forwarding layer, and achieving the dynamic network mapping through a virtual topology mode; s2, SFC deployment problem modeling is defined, specifically, the SFC deployment problem is modeled through MDP, meanwhile, limitation on SFC constraints is added in definition, and SFC-oriented QoS constraints and QoE are deployed for comprehensive optimization; and S3, solving an optimal deployment method through a dynamic deployment decision algorithm: providing an arrangement method based on deep reinforcement learning for an SFC dynamic deployment problem, the arrangement method having the capability of capturing a network environment state change sequence, maximizing QoE under the condition of satisfying the QoS constraint of the SFC, and realizing optimal dynamic VNF deployment. According to the method, multiple network indexes are calculated, meanwhile, the resource constraint property of SFC chain deployment is considered in MDP modeling, penalty terms are distributed for different constraint conditions in a traditional reward mechanism, and comprehensive optimization of QoS constraint and QoE experience facing SFC is achieved.

Description

Technical Field

[0001] The present invention relates to the field of satellite networks, and in particular to a dynamic SFC deployment method for SDN / NFV satellite networks based on deep reinforcement learning. Background Art

[0002] The construction of the ground network relies on the ground wireless network infrastructure. It is difficult to deploy related infrastructure in some areas with harsh climate and terrain conditions or in vast and sparsely populated areas. Therefore, the coverage capacity of the ground network communication system is limited. In addition, in outdoor environments with high uncertainty, the ground-side infrastructure is easily affected by natural disasters or human factors, resulting in relatively high equipment maintenance costs. As an extension of the ground network, the satellite network is not constrained by physical environmental conditions. Its wider coverage can ensure effective network services in remote areas or disaster emergencies. At the same time, the ground network can supplement the scarce network resources and high transmission delay of the satellite network itself. The integration of satellite networks and ground networks can alleviate the load of ground networks, improve the network service performance in special areas or at times, and provide a more flexible networking method.

[0003] There are still some problems with traditional satellite networks in practical applications: 1. Configuration is too slow. In traditional satellite networks, a satellite is only configured when it passes through a ground station, which results in a long time to configure the global network. 2. Low scheduling efficiency. Given the static decision-making strategies and overly decentralized management models adopted by traditional satellite networks, it is difficult for them to adapt to the requirements of dynamic user tasks and fine-grained resource management in the modern communications field. 3. Service delivery limitations. In the field of satellite services, the deployment of related services usually follows a specific supplier layout, aiming to achieve the execution of specific functions. In this context, some new technologies such as Software-Defined Networking (SDN) and Network Functions Virtualization (NFV) can help us effectively deal with current problems.

[0004] Software-defined networking decouples the traditional control plane and data plane, avoiding the inefficiency of network updates caused by the tight coupling between traditional switch hardware and its built-in algorithms, and adopts a logical unified control strategy to achieve more efficient network management. Network function virtualization is committed to virtualizing traditional hardware modules into software modules VNF (Virtual Network Function), so as to configure services more flexibly. Service Function Chain (SFC) has become a paradigm for these two propositions. The service function chain defines a set of ordered or partially ordered VNFs with certain sorting constraints, namely the SFC request flow. To build virtual links and complete the deployment of VNFs at the physical infrastructure level, the first task is to find a path between the service terminal and the user. The implementation of this path must meet the resource constraints of all requests. The choice of path has a significant impact on the performance of the SFC request flow.

[0005] At present, satellite networks based on SDN and NFV still lack an effective architecture that is widely agreed upon by the industry, and the designed control and management algorithms often perform static scheduling, and the controller cannot obtain satellite link status information in real time to make effective scheduling. At the same time, SFC deployment research usually only focuses on the constraints and optimization of QoS-related indicators, and rarely pays attention to QoE optimization research. Although there is a certain connection between QoS and QoE, the relationship between the two is nonlinear. Therefore, how to comprehensively optimize the SFC dynamic deployment strategy for different QoS indicators and user experience quality QoE in satellite networks while meeting the QoS constraint requirements of SFC is an urgent problem to be solved. Summary of the invention

[0006] In order to improve the flexible networking capability of satellite networks and address the dynamic SFC deployment optimization problem in satellite networks, the present invention proposes a dynamic SFC deployment method for SDN / NFV satellite networks based on deep reinforcement learning.

[0007] The SDN / NFV satellite network dynamic SFC deployment based on deep reinforcement learning of the present invention comprises the following steps:

[0008] S1. Build a three-layer SDN / NFV satellite network architecture: Build a satellite network architecture with application management layer, control layer, and data forwarding layer, and use virtual topology mode to achieve dynamic network mapping;

[0009] S2. Define the SFC deployment problem modeling: Model the SFC deployment problem with MDP, add restrictions on SFC constraints in the definition, and deploy SFC-oriented QoS constraints and QoE for comprehensive optimization;

[0010] S3. Solve the optimal deployment method through dynamic deployment decision algorithm: For the SFC dynamic deployment problem, a deep reinforcement learning-based orchestration method is proposed, which has the ability to capture the sequence of network environment state changes, maximize QoE while satisfying the QoS constraints of SFC, and achieve optimal dynamic VNF deployment.

[0011] Furthermore, in S1, the steps for building a three-layer SDN / NFV satellite network architecture are as follows:

[0012] The application management layer is composed of the satellite network management center, which is responsible for policy calculation and resource management policy functions, and transmits information with the control layer below through the northbound interface; the GEO layer satellite is responsible for collecting satellite network link status information and sending it to SNMC, and SNMC then formulates deployment strategies based on this information and specific user needs and sends it back to the GEO layer; the control layer includes the SDN controller placed on the GEO satellite. Through the northbound interface, the GEO layer satellite can send the collected link status information to SNMC, receive instructions from SNMC, and continue to distribute to the lower layer satellites to achieve centralized control logic; the data forwarding layer is composed of MEO and LEO satellites, which only transmit data and do not carry any global control capabilities; different service functions are embedded in virtual machines deployed on LEO, and the VNF deployment and configuration strategies involved are determined by SNMC. The traffic in the satellite network is forwarded to users through different VNF ​​instances, and the paths through different VNFs during the forwarding process are decided by SNMC; the virtual topology method subdivides the topology changes of the satellite's rotation around the earth into static topologies of multiple time periods, so that each satellite can obtain the topology map of all different time periods in a complete operation cycle, realizing adaptive network management.

[0013] Further, S2 includes the following steps:

[0014] S21. Define the deployment of SFC based on QoE / QoS optimization;

[0015] S22, perform MDP modeling on the SFC deployment problem;

[0016] S23. Determine a reward model that reflects QoE under QoS or other constraints.

[0017] Furthermore, S21 specifically includes the following steps:

[0018] The SFC request flow includes N network functions; each network function can be represented by a VNF instance t i Complete service delivery, where i∈{1, 2, …, N}, and each VNF instance t i All have M i candidate VNF instances; let ins ijRepresents a VNF instance t i The j-th VNF instance, where i∈{1, 2, …, N}, j∈{1, 2, …, M i The QoE / QoS-aware SFC deployment problem, i.e., the QoE-optimized SFC deployment problem that satisfies the QoS constraints of the SFC request flow, is defined as follows:

[0019]

[0020] Among them, x ij ∈{0, 1} indicates whether to select ins ij , x ij =1 represents selection, x ij = 0 means no selection to complete the i-th network function required by the SFC request; the SFC deployment problem is to start from the t-th i In this instance, one and only one VNF instance is selected for the ith network function to be deployed to realize the network service; C represents the set of all service function chains that can satisfy the SFC request in terms of functional constraints, qoe c represents the end-to-end QoE of the service function chain c∈C; qos c This L-dimensional vector represents the QoS metric of the service function chain c∈C, q con This L-dimensional vector represents the QoS constraints of the SFC request flow; it is assumed that the first K dimensions of the QoS vector are positive indices and the remaining dimensions are negative indices.

[0021] Further, S22 includes the following steps:

[0022] MDP is defined by a five-element group {S, A, P, R, γ}, where S is the set of possible states that may appear during the definition process; A represents a finite set of possible actions; P describes a finite set of possible state transitions after performing actions in different states; R contains the immediate benefits that can be obtained by completing a specific action in a specific state; the γ∈[0,1] discount factor is introduced to consider the time decay effect when accumulating the return value; the solution of MDP is the strategy, and the goal of the strategy is that in the state s∈S, the selected action a∈A should maximize the cumulative reward; the quality of the strategy π(s,a) is evaluated by the cumulative reward; two value functions are defined, namely the state value function V π (s) and the action-value function Q π (s, a); V π (s) represents the expected cumulative reward obtained by taking actions according to a certain strategy in a given state s, Q π (s, a) represents the expected cumulative reward after taking action a in a given state s, and the mathematical relationship is as follows:

[0023]

[0024] where r t represents the immediate reward of step t∈{1, 2, …, T}, E(·) is the operator for finding the mathematical expectation; finding the optimal solution of MDP is the process of finding the best strategy that can maximize the state value function:

[0025] π * * = argmaxV π (s) (4) According to the Bellman optimality equation, the optimal strategy π * The following conditions must be met:

[0026]

[0027] Furthermore, in S22, the deployment problem of SFC is modeled according to the MDP model as follows:

[0028] 1) S: Each state s∈S represents the network environment, including the network topology, the QoS / QoE status of different VNF ​​instances, the functions and QoS requirements of the SFC request being processed;

[0029] 2) A: Each action a∈A represents the selection of a directly connected VNF instance from the currently available VNF instances. For the i-th network function, there are M i Actions;

[0030] 3) P: Transition probability It indicates the probability that the QoS / QoE state of the network changes from s to s′ under the condition that the VNF instance selects action a;

[0031] 4) R: The immediate reward r∈R represents the selected VNF instance ins ij Immediate QoE contribution to the functional chain;

[0032] The SFC deployment scheme modeled by MDP is to find an optimal service function chain c*∈C under strategy π, where C represents a finite set of service function chains such that:

[0033]

[0034] Further, S23 includes the following steps:

[0035] QoE gain, QoS constraint penalty, and OPEX penalty constitute the components of the incremental SFC deployment operational reward model;

[0036] Objective evaluation for QoE can calculate QoE from measurable indicators. QoE is calculated based on the measured QoS indicators and the psychological perception of the end user. Using the Weber-Fechner law and the QoE-QoS index interdependence hypothesis, QoE can be derived from QoS indicators. The nonlinear relationship between QoS and QoE is shown in the following formula:

[0037] dQoS∝QoS·dQoE,WFL(7)

[0038] dQoE'∝QoE'·dQoS,IQX (8)

[0039] For positive QoS indicators, the corresponding QoE is derived according to WFL, and for negative QoS indicators, the corresponding QoE is derived according to IQX, as shown below:

[0040]

[0041] in is the t-th QoS metric of service function chain c, α p , β p , γ p ,θ p , α n , β n , γ n and θ n It is a constant parameter used to fine-tune the QoS / QoE relationship;

[0042] For a certain SFC, the QoE gain of the entire chain is calculated as follows:

[0043]

[0044] In a given state, select all VNF instances that make up chain c by taking actions ij s, and finally a chain is constructed; the QoE gain of chain c is as follows:

[0045]

[0046] Define chain c based on QoS indicator qos c The penalties resulting from the QoS constraint qcon are as follows:

[0047]

[0048] Where P is defined as a sufficiently large constant to penalize QoS constraints that have been violated;

[0049] VNF instance ins ij The operating expense costs are defined as follows:

[0050]

[0051] in, represents the OPEX of starting the corresponding virtual machine of the i-th network function in SFC, represents the OPEX of starting the corresponding VNF instance of its network function; the OPEX penalty of using a deployed VNF instance is opex normal ;

[0052] The OPEX penalty for chain c is the sum of the OPEX penalties of all VNF instances deployed along this SFC:

[0053]

[0054] According to the definition of QoE gain, QoS constraint penalty and OPEX penalty, the real-time return of deployment chain c can be defined as follows:

[0055]

[0056] Among them, the real-time reward r of chain c c Evenly distribute among the VNF instances participating in chain c:

[0057]

[0058] Furthermore, S3 specifically includes the following steps:

[0059] S31. Determine a VNF instance selection strategy;

[0060] S32. Propose a SFC dynamic scheduling method based on deep reinforcement learning.

[0061] Furthermore, S31 specifically includes the following steps:

[0062] Use the ∈-greedy algorithm to balance the exploration and exploitation tendencies by selecting to execute random actions with probability ∈ and executing the greedy strategy with probability 1-∈:

[0063]

[0064] Further, S32 specifically includes the following steps:

[0065] The action value function Q(s, a) is the long-term accumulation of the real-time reward r. Each time a VNF instance ins is selected ij , you will get a real-time reward r for selecting this instance. ij , the real-time reward is used to update the selected ins ij The action value function Q(s, ins ij) for reference in future decisions; the deep Q network uses real-time rewards r ij s to the cumulative return Q(s, ins ij )s for fitting; DQN uses a convolutional neural network to generate a fitting value Q(s, a; θ) to approximate the true Q(s, a); CNN can complete the update of state s and generate Q(s, ins ij ; θ), and finally guide ins according to the strategy ij To evaluate the performance of the neural network, the mean square error is selected as the loss function for training the network. The definition of the loss function is as follows:

[0066]

[0067] Accordingly, the gradient of the loss function can be obtained as follows:

[0068]

[0069] In the continuous iteration of the neural network, through gradient descent and back propagation, the optimal action value function Q(s, ins) of the i-th network function can be continuously approached. ij );

[0070] By rewriting the three CNN layers in the original DQN network structure and replacing the last CNN layer with an LSTM layer, DQN is upgraded to rDQN. The LSTM layer enables rDQN to capture state sequences, and by effectively integrating network state information between different time nodes, rDQN can learn the changes in network state information. The rDQN-SFC algorithm is used to fit the action value function and complete the strategy selection instance deployment.

[0071] Beneficial Effects

[0072] The method proposed in the present invention can provide an excellent dynamic SFC deployment solution for the satellite network in a highly dynamic satellite network environment. The present invention establishes a three-layer SDN / NFV satellite network architecture, builds an "application management layer-control layer-data forwarding layer" satellite network architecture and a satellite network dynamic SFC deployment model.

[0073] The present invention deploys NFV services at the LEO layer of the three-layer SDN / NFV network architecture, delivers corresponding network services to users along the SFC orchestration chain while forwarding data, and implements virtual topology based on the dynamic characteristics of satellites.

[0074] The present invention uses deep reinforcement learning to train the SFC dynamic deployment solution. The training basis of reinforcement learning is established on the MDP model, and the SFC dynamic deployment problem in the satellite network is modeled as an MDP model.

[0075] The present invention calculates a variety of network indicators and optimizes the user's QoE experience. At the same time, the resource constraint nature of SFC chain deployment is taken into account in MDP modeling, and penalty items are assigned to different constraints in the traditional reward mechanism, thereby achieving comprehensive optimization of SFC-oriented QoS constraints and QoE experience, and effectively improving the user's intuitive experience.

[0076] The present invention adopts the rDQN-SFC dynamic deployment algorithm and utilizes the "memory" characteristics of the recurrent neural network to give the network itself the ability to capture sequence information. The network can "remember" through the memory unit which previous state the current state is derived from. With the information of state change, the rDQN-SFC algorithm can make more adaptive deployment decisions for the dynamic deployment problem of SFC. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is a schematic diagram of the three-layer SDN / NFV satellite network architecture in the present invention;

[0078] Figure 2 It is a schematic diagram of the virtual topology of the satellite network in the present invention;

[0079] Figure 3 Schematic diagram of the DQN network structure in the present invention;

[0080] Figure 4 Schematic diagram of the LSTM unit structure in the present invention;

[0081] Figure 5 Schematic diagram of the principle of the rDQN-SFC algorithm in the present invention;

[0082] Figure 6 It is a schematic diagram of the QoE performance comparison of the random algorithm, brute force algorithm, traditional DQN algorithm and rDQN-SFC algorithm in the present invention;

[0083] Figure 7 It is a schematic diagram comparing the QoS constraint satisfaction of the random algorithm, brute force algorithm, traditional DQN algorithm and rDQN-SFC algorithm in the present invention;

[0084] Figure 8 This is a schematic diagram comparing the average QoE, error rate, and average response time of the random algorithm, brute force algorithm, traditional DQN algorithm, and rDQN-SFC algorithm after iterative training in the present invention. DETAILED DESCRIPTION

[0085] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0086] The SDN / NFV satellite network dynamic SFC deployment based on deep reinforcement learning of the present invention comprises the following steps:

[0087] S1. Build a three-layer SDN / NFV satellite network architecture: Build a satellite network architecture with application management layer, control layer, and data forwarding layer, and use virtual topology mode to achieve dynamic network mapping;

[0088] The three-layer SDN / NFV satellite network architecture includes three levels: application management layer, control layer and data forwarding layer.

[0089] The application management layer is composed of the Satellite Network Management Center (SNMC). SNMC is responsible for policy calculation, resource management policy and other functions, and can transmit information with the control layer below through the northbound interface. The GEO layer satellite is responsible for collecting satellite network link status information and sending it to SNMC. SNMC then formulates deployment strategies based on this information and specific user needs and sends them back to the GEO layer. The control layer includes the SDN controller placed on the GEO satellite. Through the northbound interface, the GEO layer satellite can send the collected link status information to SNMC, receive instructions from SNMC, and continue to distribute to the lower layer satellites to achieve centralized control logic. The data forwarding layer is composed of MEO and LEO satellites, which only transmit data and do not carry any global control capabilities. Different service functions will be embedded in the virtual machines deployed on LEO. The VNF deployment and configuration strategies involved are determined by SNMC. The traffic in the satellite network is forwarded to users through different VNF ​​instances. The paths through different VNFs during the forwarding process are decided by SNMC. The virtual topology method subdivides the topology changes of a satellite orbiting the earth into static topologies of multiple time periods, so that each satellite can obtain the topology map of all different time periods within a complete operating cycle, realizing adaptive network management.

[0090] S2. Define the SFC deployment problem modeling: Model the SFC deployment problem with MDP, add restrictions on SFC constraints in the definition, and deploy SFC-oriented QoS constraints and QoE for comprehensive optimization;

[0091] S2 includes the following steps:

[0092] S21. Define the deployment of SFC based on QoE / QoS optimization:

[0093] An SFC request flow includes N network functions (for example, traffic passes through firewall, DPI, etc. in sequence). Each network function can be represented by a VNF instance t iComplete service delivery, where i∈{1, 2, …, N}, and each VNF instance t i All have M i candidate VNF instances. ij Represents a VNF instance t i The j-th VNF instance, where i∈{1, 2, ..., N}, j∈{1, 2, ..., M i The QoE / QoS-aware SFC deployment problem is a QoE-optimized SFC deployment problem that satisfies the QoS constraints of the SFC request flow, and is defined as follows:

[0094]

[0095] Among them, x ij ∈{0, 1} indicates whether to select ins ij (x ij =1 represents selection, x ij = 0 means no selection) to complete the i-th network function required by the SFC request. The SFC deployment problem is to start from the t-th i In the example, one and only one VNF instance is selected for the ith network function to be deployed to realize the network service. C represents the set of all service function chains that can satisfy the SFC request in terms of functional constraints. c represents the end-to-end QoE of the service function chain c∈C. qos c This L-dimensional vector represents the QoS metric of the service function chain c∈C, q con This L-dimensional vector represents the QoS constraints of the SFC request flow. For the sake of generality, we assume that the first K dimensions of the QoS vector are positive indicators (the larger the value, the better, such as bandwidth), and the remaining dimensions are negative indicators (the smaller the value, the better, such as delay).

[0096] S22. MDP modeling for SFC deployment problem:

[0097] MDP is usually defined as a five-element group {S, A, P, R, γ}, where S defines the set of possible states in the process; A represents a finite set of possible actions; P describes a finite set of possible state transitions after performing actions in different states; R contains the immediate benefits that can be obtained by completing a specific action in a specific state; the discount factor γ∈[0, 1] is introduced to consider the time decay effect when accumulating the return value. The solution of MDP is called a strategy. The goal of the strategy is that the selected action a∈A should maximize the cumulative reward under the state s∈S. The quality of the strategy π(s, a) is not determined by the immediate reward r∈R, but is evaluated by the cumulative reward. Define two value functions, namely the state value function V and the state value function V. π(s) and the action-value function Q π (s, a). π (s) represents the expected cumulative reward obtained by taking actions according to a certain strategy in a given state s, Q π (s, a) represents the expected cumulative reward after taking action a in a given state s, and the mathematical relationship is as follows:

[0098]

[0099] where r t represents the immediate reward of step t∈{1, 2, …, T}, and E(·) is the operator for finding the mathematical expectation. Finding the optimal solution of MDP is the process of finding the best strategy that can maximize the state value function:

[0100] π * = argmaxV π (s) (24)

[0101] According to the Bellman optimality equation, the optimal strategy π * The following conditions must be met:

[0102]

[0103] The deployment problem of SFC is modeled according to the MDP model as follows:

[0104] 5) S: Each state s∈S represents the network environment, including the network topology, QoS / QoE status of different VNF ​​instances, the functions and QoS requirements of the SFC request being processed, etc.

[0105] 6) A: Each action a∈A represents the selection of a directly connected VNF instance from the currently available VNF instances. Obviously, for the i-th network function, there are M i actions (that is, there are M i options).

[0106] 7) P: Transition probability It represents the probability that the QoS / QoE state of the network changes from s to s′ under the condition that the VNF instance selects action a. However, for the SFC deployment problem in this study, the size of the state transition probability is unknown, so the SFC deployment problem is a model-free reinforcement learning problem.

[0107] 8) R: The immediate reward r∈R represents the selected VNF instance ins ij Immediate QoE contribution to the functional chain.

[0108] The SFC deployment solution modeled by MDP is to find an optimal service function chain c under strategy π* ∈C (C represents a finite set of service function chains), such that:

[0109]

[0110] S23. Determine a reward model that reflects QoE under QoS or other constraints:

[0111] QoE gain, QoS constraint penalty, and OPEX penalty constitute the components of the incremental SFC deployment operational reward model.

[0112] Objective evaluation for QoE can calculate QoE from measurable indicators without the need for end-user participation in the survey, making it possible to automate QoE evaluation. In objective evaluation, QoE can be calculated based on measured QoS indicators and the psychological perception of end users. Using the Weber-Fechner Law (WFL) and the Exponential Interdependency of QoE and QoS (IQX) hypothesis, QoE can be derived from QoS indicators, where the nonlinear relationship between QoS and QoE is shown in the following formula:

[0113] dQoS∝QoS·dQoE,WFL (27)

[0115] dQoE'∝QoE'·dQoS,IQX (28)

[0117] For positive QoS indicators, the corresponding QoE can be derived according to WFL, while for negative QoS indicators, the corresponding QoE can be derived according to IQX, as shown below:

[0118]

[0119] in is the t-th QoS metric of service function chain c, α p , β p , γ p ,θ p , α n , β n , γ n and θ n It is a constant parameter used to fine-tune the QoS / QoE relationship.

[0120] Then for a certain SFC, the QoE gain of the entire chain is calculated as follows:

[0121]

[0122] In a given state, select all VNF instances that make up chain c by taking actions ij s, and finally construct chain c. The QoE gain of chain c is as follows:

[0123]

[0124] The key to using reinforcement learning to solve the QoS / QoE-based SFC deployment problem is to enforce the QoS constraints of the SFC while in the learning process. However, if the QoS constraints of the chain are explicitly specified in the reinforcement learning process like the mathematical programming method, this will cause the original problem to become an NP-hard problem, and it is difficult to find a feasible algorithm to solve the problem in polynomial time. There is a contradiction between QoE gain and QoS constraints. High-intensity resource consumption will continuously narrow the gap between the current QoS indicators and QoS constraints, increasing the possibility of violating QoS constraints and no longer suitable for maintaining high-intensity resource consumption. Design a negative reward indicator, which can be called "penalty", to balance the reward amount. The smaller the gap between the indicator and the constraint, the greater the penalty relative to the reward; when a QoS indicator has violated the corresponding QoS constraint, the penalty will be quite "severe". Define chain c based on the QoS indicator qos c The penalties resulting from the QoS constraint qcon are as follows:

[0125]

[0126] Where P is defined as a sufficiently large constant to penalize QoS constraints that have been violated.

[0127] The deployment of VNF instances can be divided into deploying SFC directly using the deployed VNF instances or deploying a new VNF instance on a node with available resources. Compared with directly using the VNF instances that have been deployed on a node, deploying a new VNF instance on a node will consume high OPEX, so OPEX costs will also be considered a penalty for rewards. This makes it more inclined to give priority to using existing VNF instances when deploying SFC, rather than constantly creating new VNF instances at a high deployment cost and inefficiently. VNF instance ins ij The operating expense costs are defined as follows:

[0128]

[0129] in, represents the OPEX of starting the corresponding virtual machine of the i-th network function in SFC, It represents the OPEX of starting the corresponding VNF instance of its network function. Once a VNF instance is instantiated on demand and created using network resources, it can be used by other SFC deployments. The OPEX penalty for using a deployed VNF instance is opex normal , which includes operational expenses such as energy consumption which cannot be ignored in normal network operations.

[0130] The OPEX penalty for chain c is the sum of the OPEX penalties of all VNF instances deployed along this SFC:

[0131]

[0132] According to the definition of QoE gain, QoS constraint penalty and OPEX penalty, the real-time return of deployment chain c can be defined as follows:

[0133]

[0134] Among them, the real-time reward r of chain c c Evenly distribute among the VNF instances participating in chain c:

[0135]

[0136] S3. Solve the optimal deployment method through dynamic deployment decision algorithm: For the SFC dynamic deployment problem, a deep reinforcement learning-based orchestration method is proposed, which has the ability to capture the sequence of network environment state changes, maximize QoE while satisfying the QoS constraints of SFC, and achieve optimal dynamic VNF deployment.

[0137] S3 includes the following steps:

[0138] S31. Determine the VNF instance selection strategy:

[0139] For a given network function, the reinforcement learning agent has two strategies for selecting the VNF instance of this function: (1) It is more inclined to select new VNF instances that have not been executed yet, enhance the perception of network conditions and increase the probability of executing this VNF instance as the best decision (i.e. exploration). (2) It is more inclined to repeatedly select the known best VNF ​​instance according to the current network conditions and obtain the maximum known reward in a more conservative way (i.e. utilization).

[0140] Excessive "exploration" may face the risk of failing to maximize rewards, while excessive "utilization" may face the risk of missing out on better alternatives. At the same time, the available resources of the corresponding nodes in the network may also face exhaustion due to excessive utilization (for example, some nodes may be overloaded due to repeated selections too frequently). Therefore, in order to adapt to certain random characteristics of the dynamic network environment, when managing the VNF instance selection problem, in addition to considering the network status information composed of QoE / QoS, the probability distribution of action selection should also be considered.

[0141] The greedy strategy is to repeatedly select the function with the highest value Q π (s, a) action. The action for Q defined in the SFC deployment scenario π The accumulated real-time reward r of (s, a) ij It is not a constant value at all times. The ∈-greedy algorithm is used to balance the exploration and utilization tendency problems by selecting to execute random actions with a probability of ∈ and executing the greedy strategy with a probability of 1-∈.

[0142]

[0143] S32. Propose a SFC dynamic scheduling method based on deep reinforcement learning (rDQN-SFC):

[0144] The action value function Q(s, a) is the long-term accumulation of the real-time reward r. Each time a VNF instance ins is selected ij , you will get a real-time reward r for selecting this instance. ij , the real-time reward will be used to update the selected ins ij The action value function Q(s, ins ij ) for future decision making. The Deep Q Network (DQN) can use real-time rewards r ij s to the cumulative return Q(s, ins ij )s for fitting. DQN uses a convolutional neural network (CNN) to generate fitting values ​​Q(s, a; θ) to approximate the true Q(s, a). CNN can complete the update of state s and generate Q(s, ins ij ; θ), and finally guide ins according to the strategy ij In order to evaluate the performance of the neural network, the mean square error is selected as the loss function for training the network. The definition of the loss function is as follows:

[0145]

[0146] Accordingly, the gradient of the loss function can be obtained as follows:

[0147]

[0148] In the continuous iteration of the neural network, through gradient descent and back propagation, the optimal action value function Q(s, ins) of the i-th network function can be continuously approached. ij ).

[0149] The network uses the experience replay mechanism, and subsequent training will continue to randomly sample from previously trained data and add it to the training, which can alleviate the temporal correlation of samples and improve data utilization. In order to make the training process more stable, the network consists of two independent parts, the evaluation network and the target network, and the structures of the two are exactly the same. The difference between them is that the network parameters of the evaluation network are updated after each iteration, while the network parameters of the target network are not updated in real time with the iteration. Instead, they are temporarily frozen and updated once after every C iterations.

[0150] Certain improvements are made to the DQN network structure to enable the DQN network to capture the sequence of state changes. Obviously, the addition of the Long Short Term Memory (LSTM) network can give the DQN network a more powerful "memory" function and capture the dynamic and continuous changes of network state information. The LSTM model belongs to the category of Recurrent Neural Network (RNN), which shows its unique advantages in processing time stream data and has a stronger ability to learn long-term dependent information. In the LSTM architecture, its core components are called "cells", which play the role of memory units. Each cell contains two main input elements: one is the cell state h of the previous time step and the other is the state h of the cell in the previous time step. t-1 , and the second is the current input information x t The "cell" consists of four interacting layers, using a structure called a "gate" to give the cell the ability to "remember" or "forget" the information. The gate is a key means to control the flow of information, and its core idea is to achieve selective passage of information. Specifically, this mechanism usually relies on a sigmoid layer to map the input data to a range of 0 to 1, thereby controlling the degree of information passing. LSTM first relies on the forget gate to determine the cell state C from the previous time. t-1 The useless information is removed from the input gate, and the useful information in the new information is filtered out and stored in the cell state. t-1 Updated to C tFinally, the output gate is used for filtering to determine which parts of the cell state will be output as h. With this structure, LSTM has a better memory ability for long sequences and can make more reliable predictions based on previous state information.

[0151] By rewriting the three CNN layers in the original DQN network structure and replacing the last CNN layer with an LSTM layer, DQN is upgraded to rDQN. The existence of the LSTM layer enables rDQN to capture state sequences. By effectively integrating network state information between different time nodes, rDQN can learn the potential knowledge of changes in network state information. With the help of LSTM, rDQN can obtain an SFC deployment solution with better QoE performance.

[0152] Based on the modeling of the problem and the construction of the deep reinforcement network model in the previous section, the rDQN-SFC algorithm is used to fit the action value function and complete the strategy selection instance deployment. The pseudo code of the rDQN-SFC algorithm is as follows:

[0153]

[0154] where Q is the Q(s, ins of the i-th network function ij )s, obviously it is an M i dimensional vector. EPISODE is the number of training rounds. REQUEST is the number of SFC requests used for training. In the VNF instance selection in row 9, the selected instance ins ij It must be able to connect to the current instance and adapt to the dynamic network topology environment. In the strategy selection process of algorithm execution, the ∈-greedy algorithm is used to balance the tendency of "exploration" and "utilization" by adjusting the hyperparameter ∈.

[0155] In one embodiment of the present invention, Figure 1 As shown, this implementation method constructs a satellite network architecture consisting of three layers: application management layer, control layer and data forwarding layer. Among them, the application management layer is orchestrated by SNMC for the overall network. SNMC is responsible for policy calculation, resource management strategy and other functions. The control layer is composed of GEO satellites, and the SDN controller is deployed on GEO. GEO has the ability to divide and refine the control domain it is responsible for, and can collect and integrate various types of link resource data information through link perception. The data forwarding layer is composed of satellites in MEO and LEO orbits. It does not have control and management capabilities, retains basic access functions and forwarding functions, and deploys NFV functions at the LEO layer. Different service functions will be embedded in virtual machines deployed on LEO. The VNF deployment and configuration strategies involved are all determined by SNMC.

[0156] In one embodiment of the present invention, the satellite network virtual topology principle is used as follows: Figure 2 As shown in the figure. The virtual topology method uses a virtual strategy based on the satellite operation cycle, which divides the topology changes of the satellite's rotation around the earth into static topologies of multiple time periods. When each time period is divided into short enough, it can be considered that the network topology is approximately stable and unchanged in each specific time period. This processing method enables each satellite to obtain the topology map of all different time periods in a complete operation cycle, providing strong support for achieving more adaptive network management.

[0157] In one embodiment of the present invention, the DQN network structure is as follows: Figure 3 As shown in the figure. The core of DQN is to use a deep neural network to approximate the Q-value function to estimate the Q-value of each state-action pair. The structure of this neural network is usually a deep feedforward neural network, which contains an input layer, a hidden layer, and an output layer. The hidden layer includes a convolutional layer and a fully connected layer, which are used to learn the abstract representation of state features. Combined with technologies such as experience replay and target network, sample efficiency and stability can be improved.

[0158] In one embodiment of the present invention, the LSTM memory cell structure is as follows: Figure 4 As shown in Figure 1. The gate is a key means for LSTM to control the flow of information. Its core idea is to achieve selective passage of information. The two main input information are the cell state h of the previous time step and t-1 and the current input information x t The forget gate determines the LSTM cell state C from the previous time. t-1 The input gate selects the useful content in the new information and stores it in the cell state. The cell state C t-1 Updated to C t The output gate performs the final filtering to determine which information in the cell state will be output as h t Through this structure, LSTM has a better memory ability for long sequences.

[0159] In one embodiment of the present invention, the principle diagram of the rDQN-SFC algorithm is as follows: Figure 5 The rDQN-SFC algorithm maintains the overall training efficiency in the iterative learning process through asynchronous updates of the current value network and the target value network and the random sampling mechanism of the playback memory unit.

[0160] A. Problem Formulation

[0161] 1) SFC deployment problem definition

[0162] a. SFC deployment optimization based on QoS / QoE

[0163] An SFC request flow includes N network functions (for example, traffic passes through firewall, DPI, etc. in sequence). Each network function can be represented by a VNF instance t i Complete service delivery, where i∈{1, 2, …, N}, and each VNF instance t i All have M i candidate VNF instances. ij Represents a VNF instance t i The j-th VNF instance, where i∈{1, 2, …, N}, j∈{1, 2, …, M i The QoE / QoS-aware SFC deployment problem is a QoE-optimized SFC deployment problem that satisfies the QoS constraints of the SFC request flow, and is defined as follows:

[0164]

[0165] Among them, x ij ∈{0, 1} indicates whether to select ins ij (x ij =1 represents selection, x ij = 0 means no selection) to complete the i-th network function required by the SFC request. The SFC deployment problem is to start from the t-th i In the example, one and only one VNF instance is selected for the ith network function to be deployed to realize the network service. C represents the set of all service function chains that can satisfy the SFC request in terms of functional constraints. c represents the end-to-end QoE of the service function chain c∈C. qos c This L-dimensional vector represents the QoS metric of the service function chain c∈C, q con This L-dimensional vector represents the QoS constraints of the SFC request flow. For the sake of generality, we assume that the first K dimensions of the QoS vector are positive indicators (the larger the value, the better, such as bandwidth), and the remaining dimensions are negative indicators (the smaller the value, the better, such as delay).

[0166] b. QoE reward model building

[0167] Positive and negative QoS indicators have different derivation formulas, as shown below:

[0168]

[0169] in is the t-th QoS metric of service function chain c, α p , β p , γ p ,θ p , α n , β n , γn and θ n It is a constant parameter used to fine-tune the QoS / QoE relationship.

[0170] Then for a certain SFC, the QoE gain of the entire chain is calculated as follows:

[0171]

[0172] In a given state, select all VNF instances that make up chain c by taking actions ij s, and finally construct chain c. The QoE gain of chain c is as follows:

[0173]

[0174] Define chain c based on QoS indicator qos c The penalties resulting from the QoS constraint qcon are as follows:

[0175]

[0176] Where P is defined as a sufficiently large constant to penalize QoS constraints that have been violated.

[0177] VNF instance ins ij The operating expense costs are defined as follows:

[0178]

[0179] in, represents the OPEX of starting the corresponding virtual machine of the i-th network function in SFC, It represents the OPEX of starting the corresponding VNF instance of its network function. Once a VNF instance is instantiated on demand and created using network resources, it can be used by other SFC deployments. The OPEX penalty for using a deployed VNF instance is opex normal , which includes operational expenses such as energy consumption which cannot be ignored in normal network operations.

[0180] The OPEX penalty for chain c is the sum of the OPEX penalties of all VNF instances deployed along this SFC:

[0181]

[0182] According to the definition of QoE gain, QoS constraint penalty and OPEX penalty, the real-time return of deployment chain c can be defined as follows:

[0183]

[0184] Among them, the real-time reward r of chain c c Evenly distribute among the VNF instances participating in chain c:

[0185]

[0186] 2) VNF selection strategy

[0187] The ∈-greedy algorithm balances the exploration and exploitation tendencies by choosing to execute random actions with probability ∈ and executing the greedy strategy with probability 1-∈.

[0188]

[0189] 3) Parameter definition and value

[0190] The satellite network simulation parameter definitions and their values ​​are shown in Table 1:

[0191] Table 1 Satellite network parameter settings

[0192]

[0193] The rDQN network parameter settings are shown in Table 2:

[0194] Table 2: DQN parameter settings

[0195]

[0196] The rDQN network architecture settings are shown in Table 3:

[0197] Table 3 rDQN network structure settings

[0198]

[0199] 4) MDP modeling definition for the satellite network SFC dynamic optimization deployment problem

[0200] V π (s) and Q π (s, a) has the following mathematical relationship:

[0201]

[0202] where r t represents the immediate reward of step t∈{1, 2, …, T}, and E(·) is the operator for finding the mathematical expectation. Finding the optimal solution of MDP is the process of finding the best strategy that can maximize the state value function:

[0203] π * arg maxV π (s) (14)

[0204] The best strategy π * The following conditions must be met:

[0205]

[0206] B. Solution Method Proposed in This Implementation

[0207] In one embodiment of the present invention, the specific process of solving the satellite network SFC dynamic deployment problem by a SDN / NFV satellite network dynamic SFC deployment method (rDQN-SFC) based on deep reinforcement learning is as follows.

[0208] Step 1: Input the SFC request flow set, initialize the rDQN parameters with random weights, and set the empty memory pool D.

[0209] Step 2: Initialize chain c for each arriving SFC request flow REQUEST.

[0210] Step 3: For each function in the request flow, use the ∈-greedy algorithm to select an instance ins ij .

[0211] Step 4: Get the new network state s′ and find r ij , store (s, ins in D ij , r ij , s′)

[0212] Step 5: Randomly sample from the memory pool M and train the weight of Q.

[0213] Step 6: If the time interval reaches C, update the weight of Q

[0214] Step 7: Return to step 2 and repeat until EPISODE times are reached.

[0215] The pseudo code for this implementation is as follows:

[0216]

[0217] In one embodiment of the present invention, the rDQN-SFC deployment algorithm based on QoS / QoE in the present invention is simulated and tested by TensorFlow, and the performance of the rDQN-SFC algorithm is compared horizontally with the brute force search algorithm, the random search algorithm and the traditional DQN algorithm. The SFC deployment strategy obtained by the brute force search algorithm can ensure that the deployed SFC has the best service performance, that is, the corresponding QoE is the highest, while the SFC deployment strategy given by the random search algorithm is the SFC deployment scheme with the fastest deployment time within the feasibility range. Except for the newly added LSTM layer in rDQN, the network structure of the original DQN algorithm and the network architecture of the rDQN-SFC algorithm are exactly the same. It is assumed that the entire network topology contains 10 VNF types, each of which contains 10 VNF instances, and 100 SFC requests are generated each time. The performance of the four algorithms in terms of QoE performance, QoS constraint satisfaction, and response time was compared, and a total of 1000 rounds of training tests were performed.

[0218] The experimental environment settings are shown in Table 4:

[0219] Table 4 Experimental environment settings

[0220]

[0221] In one embodiment of the present invention, Figure 6 and Figure 7 The results show that the SFC deployment performance of the rDQN-SFC algorithm and the original DQN algorithm is between the random search algorithm and the brute force search algorithm. The brute force search algorithm ensures that the SFC deployment scheme has the best QoE performance and the lowest QoS constraint violation rate. The random search algorithm only gives a functionally feasible chain and does not delve into the QoE performance of the chain. Therefore, the QoE performance of the deployed chain is the lowest and QoS constraints are often violated. Both the original DQN and the rDQN-SFC algorithm have shown a certain learning ability and can gradually approach the best QoE and the lowest error rate after training. In the initial stage, the performance of the DQN and rDQN-SFC algorithms is not much different. As the number of training rounds continues to advance, the "memory" function of the rDQN-SFC algorithm gradually emerges. The ability to capture state sequences helps the rDQN-SFC algorithm to make a more reasonable SFC deployment, and ultimately shows a lower error rate and higher QoE performance than DQN. At the same time, during the training process, the deployment effect of the DQN algorithm has relatively large fluctuations, the algorithm performance is not stable enough, and the convergence is not obvious enough. However, the rDQN-SFC algorithm has a smaller fluctuation amplitude and can converge faster. In comparison, the algorithm performance is more stable.

[0222] In one embodiment of the present invention, Figure 8This shows that the SFC deployment scheme obtained by the rDQN-SFC algorithm shows high QoE performance and can fully meet the QoS constraints, but there are still violations of QoS constraints. This is because the QoS constraints are modeled as penalty items (i.e. scalars) for rewards in the modeling, which will cause a loss in deployment accuracy and ultimately lead to the possibility of violating QoS constraints. The specific results of the average QoE evaluation and deployment error rate of the four algorithms are shown in Table 5:

[0223] Table 5 Index evaluation of four algorithms

[0224]

[0225] The specific response time results of the four algorithms are compared in Table 6:

[0226] Table 6 Average response time of four algorithms

[0227]

[0228] Although brute force search has the best deployment results, its response time is the slowest, which is a considerable order of magnitude difference compared with the other three algorithms. However, this is not easy to accept in practical applications, especially for those time-sensitive applications. Compared with the static brute force search algorithm, the rDQN-SFC algorithm has a faster deployment response speed. Because the addition of LSTM layers makes the network dependent on time series and the hidden layers in the network have more parameters to participate in the operation, the response speed of the rDQN-SFC algorithm is slightly slower than that of the original DQN. However, considering that the deployment performance of the original DQN is not good enough and very unstable, such a difference in response speed is still within an acceptable range. In general, the rDQN-SFC algorithm has a better time-benefit cost-effectiveness. After the decision agent is iteratively trained, the decision deployment time complexity of the rDQN-SFC algorithm is almost constant. To some extent, the rDQN-SFC algorithm can be regarded as a heuristic algorithm for brute force search, aiming to achieve more agile SFC orchestration while maintaining high QoE performance. In this regard, the algorithm based on rDQN-SFC balances QoE performance and QoS constraints.

[0229] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A dynamic SFC deployment method for SDN / NFV satellite network based on deep reinforcement learning, the method comprising the following steps: S1. Build a three-layer SDN / NFV satellite network architecture: Build a satellite network architecture with application management layer, control layer, and data forwarding layer, and use virtual topology mode to achieve dynamic network mapping; S2. Define the SFC deployment problem modeling: Model the SFC deployment problem with MDP, add restrictions on SFC constraints in the definition, and deploy SFC-oriented QoS constraints and QoE for comprehensive optimization; S3. Solve the optimal deployment method through dynamic deployment decision algorithm: For the SFC dynamic deployment problem, a deep reinforcement learning-based orchestration method is proposed, which has the ability to capture the sequence of network environment state changes, maximize QoE while satisfying the QoS constraints of SFC, and achieve optimal dynamic VNF deployment.

2. According to claim 1, the SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning is characterized in that: In S1, the steps to build a three-layer SDN / NFV satellite network architecture are as follows: The application management layer is composed of the satellite network management center, which is responsible for policy calculation and resource management policy functions, and transmits information with the control layer below through the northbound interface; the GEO layer satellite is responsible for collecting satellite network link status information and sending it to SNMC, which then formulates deployment strategies based on the information and specific user needs and sends it back to the GEO layer; The control layer includes the SDN controller placed on the GEO satellite, which sends the collected link status information to the SNMC through the northbound interface GEO layer satellite, receives instructions from the SNMC, and continues to distribute to the lower layer satellite to achieve centralized control logic; The data forwarding layer is composed of MEO and LEO satellites for data transmission. Different service functions are embedded in virtual machines deployed on LEO. The VNF deployment and configuration strategies involved are determined by SNMC. The traffic in the satellite network is forwarded to users via different VNF ​​instances. The paths through different VNFs during the forwarding process are decided by SNMC. The virtual topology method subdivides the topology changes of the satellite during its rotation around the earth into static topologies of multiple time periods, so that each satellite can obtain the topology map of all different time periods within a complete operation cycle, realizing adaptive network management.

3. The SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning according to claim 1 is characterized in that: S2 includes the following steps: S21. Define the deployment of SFC based on QoE / QoS optimization; S22, perform MDP modeling on the SFC deployment problem; S23. Determine a reward model that reflects QoE under QoS or other constraints.

4. The SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning according to claim 3 is characterized in that: S21 specifically includes the following steps: The SFC request flow includes N network functions; each network function can be represented by a VNF instance t i Complete service delivery, where i∈{1, 2, …, N}, and each VNF instance t i All have M i candidate VNF instances; let ins ij Represents a VNF instance t i The j-th VNF instance, where i∈{1, 2, ..., N}, j∈{1, 2, ..., M i The QoE / QoS-aware SFC deployment problem, i.e., the QoE-optimized SFC deployment problem that satisfies the QoS constraints of the SFC request flow, is defined as follows: Among them, x ij ∈{0, 1} indicates whether to select ins ij , x ij =1 represents selection, x ij = 0 means no selection to complete the i-th network function required by the SFC request; the SFC deployment problem is to start from the t-th i In this instance, one and only one VNF instance is selected for the ith network function to be deployed to realize the network service; C represents the set of all service function chains that can satisfy the SFC request in terms of functional constraints, qoe c represents the end-to-end QoE of the service function chain c∈C; qos c This L-dimensional vector represents the QoS metric of the service function chain c∈C, q con This L-dimensional vector represents the QoS constraints of the SFC request flow; it is assumed that the first K dimensions of the QoS vector are positive indices and the remaining dimensions are negative indices.

5. The SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning according to claim 3 is characterized in that: S22 includes the following steps: MDP is defined by a five-element group {S, A, P, R, γ}, where S is the set of possible states that may appear during the definition process; A represents a finite set of possible actions; P describes a finite set of possible state transitions after performing actions in different states; R contains the immediate benefits that can be obtained by completing a specific action in a specific state; the γ∈[0,1] discount factor is introduced to consider the time decay effect when accumulating the return value; the solution of MDP is the strategy, and the goal of the strategy is that in the state s∈S, the selected action a∈A should maximize the cumulative reward; the quality of the strategy π(s,a) is evaluated by the cumulative reward; two value functions are defined, namely the state value function V π (s) and the action-value function Q π (s, a); V π (s) represents the expected cumulative reward obtained by taking actions according to a certain strategy in a given state s, Q π (s, a) represents the expected cumulative reward after taking action a in a given state s, and the mathematical relationship is as follows: where r t represents the immediate reward of step t∈{1, 2, …, T}, E(·) is the operator for finding the mathematical expectation; finding the optimal solution of MDP is the process of finding the best strategy that can maximize the state value function: p * =argmaxV π (s) (4) According to the Bellman optimality equation, the optimal strategy π * The following conditions must be met:

6. The SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning according to claim 3 is characterized in that: In S22, the deployment problem of SFC is modeled according to the MDP model as follows: 1) S: Each state s∈S represents the network environment, including the network topology, the QoS / QoE status of different VNF ​​instances, the functions and QoS requirements of the SFC request being processed; 2) A: Each action a∈A represents the selection of a directly connected VNF instance from the currently available VNF instances. For the i-th network function, there are M i Actions; 3) P: Transition probability It indicates the probability that the QoS / QoE state of the network changes from s to s′ under the condition that the VNF instance selects action a; 4) R: The immediate reward r∈R represents the selected VNF instance ins ij Immediate QoE contribution to the functional chain; The SFC deployment solution modeled by MDP is to find an optimal service function chain c under strategy π * ∈C, C represents a finite set of service function chains such that:

7. The SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning according to claim 3 is characterized in that: S23 includes the following steps: QoE gain, QoS constraint penalty, and OPEX penalty constitute the components of the incremental SFC deployment operational reward model; Objective evaluation for QoE can calculate QoE from measurable indicators. QoE is calculated based on the measured QoS indicators and the psychological perception of the end user. Using the Weber-Fechner law and the QoE-QoS index interdependence hypothesis, QoE can be derived from QoS indicators. The nonlinear relationship between QoS and QoE is shown in the following formula: dQoS∝QoS·dQoE,WFL (7) dQoE′∝QoE′·dQoS, IQX (8) For positive QoS indicators, the corresponding QoE is derived according to WFL, and for negative QoS indicators, the corresponding QoE is derived according to IQX, as shown below: in is the t-th QoS metric of service function chain c, α p , β p , γ p ,θ p , α n , β n , γ n and θ n It is a constant parameter used to fine-tune the QoS / QoE relationship; For a certain SFC, the QoE gain of the entire chain is calculated as follows: In a given state, select all VNF instances that make up chain c by taking actions ij s, and finally construct chain c; the QoE gain of chain c is as follows: Define chain c based on QoS indicator qos c The penalties resulting from the QoS constraint qcon are as follows: Where P is defined as a sufficiently large constant to penalize QoS constraints that have been violated; VNF instance ins ij The operating expense costs are defined as follows: in, represents the OPEX of starting the corresponding virtual machine of the i-th network function in SFC, represents the OPEX of starting the corresponding VNF instance of its network function; the OPEX penalty of using a deployed VNF instance is opex normal ; The OPEX penalty for chain c is the sum of the OPEX penalties of all VNF instances deployed along this SFC: According to the definition of QoE gain, QoS constraint penalty and OPEX penalty, the real-time return of deployment chain c can be defined as follows: Among them, the real-time reward r of chain c c Evenly distribute among the VNF instances participating in chain c:

8. The SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning according to claim 1, characterized in that: S3 specifically includes the following steps: S31. Determine a VNF instance selection strategy; S32. Propose a SFC dynamic scheduling method based on deep reinforcement learning.

9. The SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning according to claim 7, characterized in that: S31 specifically includes the following steps: Use the ∈-greedy algorithm to balance the exploration and exploitation tendencies by selecting to execute random actions with probability ∈ and executing the greedy strategy with probability 1-∈:

10. The SDN / NFV satellite network dynamic SFC deployment method based on deep reinforcement learning according to claim 1, characterized in that: S32 specifically includes the following steps: The action value function Q(s, a) is the long-term accumulation of the real-time reward r. Each time a VNF instance ins is selected ij , you will get a real-time reward r for selecting this instance. ij , the real-time reward is used to update the selected ins ij The action value function Q(s, ins ij ) for reference in future decisions; the deep Q network uses real-time rewards r ij s to the cumulative return Q(s, ins ij )s for fitting; DQN uses a convolutional neural network to generate a fitting value Q(s, a; θ) to approximate the true Q(s, a); CNN can complete the update of state s and generate Q(s, ins ij ; θ), and finally guide ins according to the strategy ij To evaluate the performance of the neural network, the mean square error is selected as the loss function for training the network. The definition of the loss function is as follows: Accordingly, the gradient of the loss function can be obtained as follows: In the continuous iteration of the neural network, through gradient descent and back propagation, the optimal action value function Q(s, ins) of the i-th network function can be continuously approached. ij ); By rewriting the three CNN layers in the original DQN network structure and replacing the last CNN layer with an LSTM layer, DQN is upgraded to rDQN. The LSTM layer enables rDQN to capture state sequences, and by effectively integrating network state information between different time nodes, rDQN can learn the changes in network state information. The rDQN-SFC algorithm is used to fit the action value function and complete the strategy selection instance deployment.

Images