Satellite internet resource scheduling method based on deep reinforcement learning

Abstract

The invention relates to a satellite internet resource scheduling method based on deep reinforcement learning, and aims to solve the problem that in an SGIN environment, an unstable link easily causes increase of a VNF backup AoI and thus it is difficult to ensure that the VNF can be quickly recovered when a fault occurs. A VNF state transition and failure recovery model is established for the problem that the AoI of VNF backup is increased due to link instability, and a VNF backup model perceived by the AoI is established to adjust bandwidth resources to optimize the AoI. In order to solve the problem that the VNF backup is far away from the original VNF due to relative movement between network nodes, a VNF backup migration mechanism is introduced. In order to make an adaptive optimization strategy for services with different time delay sensitivities, a time delay sensitive factor is introduced as a weight, an optimization problem is established by taking maximization of VNF backup income as an optimization target, and the optimization problem is solved. A simulation result shows that the scheme can reduce the interruption time delay of the VNF with a serious fault under the condition of reasonably utilizing network resources.

Description

Technical Field

[0001] The present invention relates to the field of satellite Internet, and in particular to a satellite Internet resource scheduling method based on deep reinforcement learning. Background Art

[0002] Satellite is a complex system in space. It consists of many components. If any of the components fails due to software failure, CPU overload or exceeding the temperature threshold, the satellite may not work properly. In addition, there are many factors in the space environment, such as solar wind, solar flares, sunspots, etc., which will also affect the normal operation of the satellite. Once a satellite node fails, it will directly affect the function of satellite communication, resulting in a large amount of network delay and even service interruption. Usually, when a failure occurs in the SFC, the user will automatically retry to continue the connection. If the orchestrator can mitigate the impact of the failure before the identifiable time interval, the user will not experience service interruption caused by the failure. Most failures can be predicted by monitoring the changes in the status information of the VNF on the node (such as resource overload and temperature) and using DRL, and then preparing for failure recovery based on the prediction results. However, VNF migration when a VNF fails will generate a large amount of interruption delay, which seriously affects service reliability. Using a predictive migration solution to predict the failure in advance can reduce the migration delay to a certain extent, but the migration of misprediction will also generate additional delay and resource waste. Therefore, in order to further ensure the reliability of SFC while making reasonable use of network resources, the present invention completes the fault recovery of VNF by pre-backup, and improves the performance loss caused by prediction errors. Specifically, for satellite nodes that are prone to failure, backup VNFs are deployed on other servers in advance, and the backup VNFs are started immediately when the satellite nodes fail, while the original node VNFs are deleted to save resources. If the VNF backup is not prepared in time due to prediction errors, the failed VNF needs to be urgently migrated immediately to reduce some interruption delays.

[0003] Existing literature has studied the rapid recovery of VNFs that fail in ground network nodes, but most of them focus on stateless backup placement and availability optimization. Among them, in order to improve resource utilization efficiency, a new sharing mechanism of redundancy and multi-tenant technology is proposed, and an availability-aware VNF deployment solution is proposed to calculate the modified availability of each VNF based on a given shared redundant block. Existing methods also propose a backup resource allocation model that considers the importance of the probability of functional and backup server failures and intermediate servers, and a reliability-aware resource allocation algorithm to reduce the cost of redundant backup VNFs. The above studies do not consider the synchronous backup of VNF state information, and offline backup solutions are difficult to apply to dynamic network environments, so they may not be able to quickly recover when VNFs fail. In order to solve the problem of rapid recovery of VNFs by predicting failures, algorithms such as machine learning are used to predict failures and formulate backup plans in advance. To ensure service continuity, a master-slave VNF structure is used, and an active path recovery strategy is adopted to quickly recover VNFs.

[0004] However, most of the existing research is based on ground networks and cannot be directly applied to complex SGINs. On the one hand, unlike fixed ground base stations, the relative movement speed between LEO satellites in different orbits is very fast, so the satellite server where the VNF backup is located may move away from the original VNF as the satellite moves; on the other hand, the interruption of the physical link increases the AoI of the backup VNF, which requires more recovery time when starting the VNF backup. Excessive recovery time will cause service interruption and make it difficult to ensure service quality. In addition, existing research also rarely considers that different services have different delay sensitivities, and the backup of services with different delay sensitivities should have different decision weights. Since the available network resources on satellites are relatively scarce compared to ground networks, too many VNF backups may occupy a large amount of network resources. Therefore, different VNF ​​backup decisions should be made for VNFs with different delay sensitivities and different fault severity to achieve a compromise between delay and network resources. Summary of the invention

[0005] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: unstable links in the SGIN environment easily lead to an increase in the VNF backup AoI, which makes it difficult to ensure that the VNF can be quickly recovered when a failure occurs.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solution: a satellite Internet resource scheduling method based on deep reinforcement learning, comprising the following steps:

[0007] S1: Combining the high coverage of MEO satellites and the high dynamics of LEO satellites, a satellite-ground fusion network SDMSGIN ​​model under software-defined multi-layer satellites is established. The physical layer topology of the SDMSGIN ​​model is represented as a weighted undirected graph G P =(N P , L P ), where N P is the set of physical nodes n, L P is the physical link L nn′ (n,n′∈N P ), define L nn′ (t) represents time slot t physical link L nn′ The remaining computing data capacity of each physical node n in time slot t is defined as The remaining capacity of storage resources is Each physical link L nn′ The remaining bandwidth capacity is

[0008] The virtual application layer where SFC is located is abstracted as a weighted directed graph G V =(F V , L V ), where F V is the VNF set, L V is a set of virtual links, define the SFC set as U, and define Denotes the set of k VNFs that make up the u-th SFC, and defines L u is the virtual link between two adjacent VNFs in the uth SFC A collection of virtual links. The bandwidth requirement is Defining binary variables Indicates the mapping of virtual links and defines The computing resource requirements are Storage data requirements are

[0009] S2: A VNF state transfer and fault recovery model is established to solve the problem of increased AoI of VNF backup due to link instability. In order to specifically describe how to use VNF backup to help the failed VNF to recover quickly, an AoI-aware VNF backup model is established to adjust bandwidth resources to optimize AoI.

[0010] S3: In the AoI-aware VNF backup model, a VNF backup migration mechanism is introduced to solve the problem that the relative movement between network nodes causes the VNF backup to be far away from the original VNF. An adaptive bandwidth adjustment strategy is introduced to allocate larger bandwidth in more urgent situations to quickly synchronize status data.

[0011] S4: Introduce the delay sensitivity factor as a weight to make adaptive optimization strategies for services with different delay sensitivities. Under the constraints of tolerable delay and resources, establish an optimization problem with maximizing the VNF backup benefit as the optimization goal.

[0012] S5: A method combining LSTM and DPPO is proposed to solve the optimization problem proposed in S4. First, the optimization problem is transformed into MDP, and a four-tuple (S, A, P, R) is defined to represent the state space, action space, state transition probability and reward function respectively;

[0013] The VNF state transition probability is predicted through LSTM, and the LSTM is trained using the state sequence before the VNF time slot t. When the maximum number of iterations is reached, the trained LSTM is obtained and the new state is output.

[0014] The backup strategy is optimized using the distributed reinforcement learning algorithm DDPO, which is suitable for handling complex environments and coping with large-scale state spaces. When DDPO reaches the maximum number of iterations, the strategy obtained at this time is the optimal strategy.

[0015] Furthermore, in S2, the process of establishing the VNF state transfer and fault recovery model is as follows:

[0016] The states of VNF are set as normal state, warning state and critical state. The transition probabilities between the three states of VNF are:

[0017] In normal state The probability that a VNF remains in a normal state at time slot t is is the probability of entering the warning state. If the VNF enters the warning state, the probability In warning state, The probability of completing the repair and returning to normal state. Let the binary variable and Respectively represent The normal state, warning state and critical state of a VNF in time slot t. If a VNF is in one of the three states at time slot t, the binary variable corresponding to the state is 1, otherwise it is 0.

[0018] set up Indicates The time that a VNF is in the warning state when a fault occurs in time slot t, and define Indicates the backup trigger waiting time threshold. The server should immediately back up the VNF to other available nodes and transmit status data in real time. After that, it returns to normal state. If the probability of transfer is less than , the backup of the VNF is deleted to save node resources. The probability of reaching a critical state is 1. A VNF that enters a critical state cannot be restored to other states and will stay in this state, i.e., the absorbing state, with a probability of 1. Obviously, if the VNF is not backed up at this time, it will cause service interruption and the VNF can only be urgently migrated to other nodes. If the VNF has been backed up in the above warning stage and then transferred to a critical state, it will be restored to a normal state with a probability of 1.

[0019] Since different types of services have different degrees of latency sensitivity, different backup trigger waiting time thresholds need to be designed for different services. Specifically, services with high latency sensitivity require lower latency thresholds. To prevent service degradation, define The delay sensitivity factor is The more delay-sensitive the service is, the larger the value is. The value of is expressed as:

[0020]

[0021] Among them, e represents the natural constant, T w Indicates the basic waiting time.

[0022] Furthermore, in S2, the process of establishing the AoI-aware VNF backup model is as follows: the AoI at the end of the t-th time slot is expressed as:

[0023]

[0024] in, is the AoI at the end of the t-th time slot, t p is the end time of the pth time slot, p is a natural number, τ is the length of the time slot, is time slot t The node n where the backup is needed The size of the state data packet sent by node n′, Occupies bandwidth for logical links used to deliver VNF status in real time.

[0025] t time slot The accumulated amount of data not transferred It is expressed as:

[0026]

[0027] in, is the number of forwarding hops between the original VNF and the backup VNF, The backup interruption duration parameter indicates VNF backup The sum of the interruption durations between node n and the original VNF in time slot t.

[0028] Furthermore, in S3, the adaptive bandwidth adjustment strategy is to define the bandwidth adjustment factor as It is expressed as:

[0029]

[0030] in, is the delay sensitivity factor, The probability of the VNF state transitioning to a serious failure predicted by the agent.

[0031] The AoI at the end of the tth time slot is expressed as:

[0032]

[0033] Furthermore, in S3, the VNF backup migration mechanism is:

[0034] The AoI at the end of the tth time slot is expressed as:

[0035]

[0036] in, Indicates VNF backup on node n Migrate to node n' at time slot t.

[0037] If the VNF is not backed up in advance due to prediction errors, the emergency migration of the VNF will cause a lot of interruption delays. The only way to reduce this delay is to increase the migration bandwidth urgently. Since the information difference between the node to be migrated and the original node is the size of the entire VNF data to be migrated, the emergency migration of this data will cause interruption delays. Therefore, this migration delay is equivalent to AoI, and a binary variable is set Indicates that the emergency migration occurs, otherwise The AoI is finally expressed as:

[0038]

[0039] in, The total amount of data that needs to be transferred to migrate the VNF;

[0040]

[0041] in, Storage resource requirements for backup.

[0042] Furthermore, in S4, the process of establishing the optimization problem with maximizing the VNF backup benefit as the optimization goal is:

[0043] Assume that the migration of VNF backup is completed instantly, each VNF can have at most one backup at the same time, and to ensure that traffic is not split, introduce other constraints: in, and They represent the deployment of VNF backup and the deployment of state synchronization link respectively;

[0044] Each VNF and its backup cannot be in the same node, which introduces another constraint: in, VNF deployment status.

[0045] Introducing resource constraints:

[0046]

[0047] in, They are The computing resource requirements for backup and The computing resource requirements, They are Backup storage resource requirements and Storage resource requirements, Calculate the resource capacity for each physical node n's appetite in time slot t, The remaining capacity of the storage resource;

[0048] The sum of the bandwidth occupied by the virtual links between VNFs of adjacent nodes, the logical synchronization links for transmitting status, and the migrated backup VNF is less than the remaining available bandwidth capacity of the physical link:

[0049]

[0050] in, Virtual Link The mapping situation, Virtual Link The broadband demand is the remaining bandwidth capacity in each physical link;

[0051] Defining binary variables if Backup has been performed in time slot t and backup recovery can be performed in this time slot. Otherwise 0: Among them, IF(.) is an indicator function. If true, IF(.) is 1, otherwise IF(.) is 0;

[0052] Introducing tolerable delay constraints: in is the critical state of the VNF in time slot t;

[0053] Define the optimization objective function It consists of two parts: gain (t) and cost (t):

[0054] in, is the critical state of the VNF at time slot t+1, is the warning status of the VNF in time slot t.

[0055] For the cost part of the objective function, it is divided into three parts: Cost(t) = Cost1(t) + Cost2(t) + Cost3(t)

[0056]

[0057] in, is the probability that VNF ​​completes recovery from warning state to normal state in time slot t+1, is the normal state of VNF at time slot t+1, is the normal state of the VNF in time slot t;

[0058]

[0059] in, is the probability that the VNF is in the warning state at time slot t+1, is the warning status of the VNF in time slot t+1, is the warning status of the VNF in time slot t;

[0060]

[0061] in, is the probability that the VNF enters the severe state from the warning state at the t+1 time slot.

[0062] Furthermore, in 54, the state space, action space, state transition probability and reward function are respectively:

[0063] State space: Define s(t) = {Res(t), L(t), P(t)}∈S to represent the SDMSGIN ​​network state space of time slot t, where The state space representing the VNF resource requirements, L(t) = {l nn′ (t)|n∈N p} represents the state space of the physical link, The state space representing the three key state transition probabilities of the VNF;

[0064] Action space: define a(t) = {X(t), Y(t), Z(t)}∈A, where and represents the action space of VNF backup decision and state synchronization link mapping decision, is the action space of backup migration decision.

[0065] State transition probability: The transition probability P represents the probability that the agent transfers to the next state s(t+1) after taking action a(t) in state s(t), expressed as p(s(t+1)|s(t), a(t).

[0066] Reward function: To maximize the reward R, the reward for time slot t is expressed by the optimization objective:

[0067] If the constraints in P1 are not met, then: r(t) = -1 / ζ, where ζ represents an infinitesimal quantity that tends to 0, thereby imposing a large penalty value on decisions that violate the constraints. In order to consider future benefits while considering immediate rewards, the reward discount factor for the t time slot is set to γ∈(0,1), and its cumulative discounted reward is: Where k is the number of iterations. Obviously, the more iterations, the smaller the cumulative reward. It represents the decision of taking action a(t) in state s(t). The strategy value function is used to judge the quality of the strategy in time slot t: Q π (s(t), a(t)) = E[R π (t)|s(t),a(t)]

[0068] Among them, Q π (s(t), a(t)) is the strategy value function, and E[.] is the expectation.

[0069] It is expressed by the iteration of Bellman equation: Q π (s(t), a(t)) = E[r(t) + γQ π (s(t+1), a(t+1))], where r(t) is the reward function.

[0070] The optimal decision π* is expressed as:

[0071] Furthermore, in S5, the process of obtaining the optimal decision is:

[0072] Input: state sequence {s(1), s(2), ...s(t), G P =(N P , L P ), G V =(F V , L V), discount factor γ, objective function update frequency F, number of threads N thread , the number of global network iterations M all , the number of local network iterations M part , learning rates α1 and α2;

[0073] Output: optimal strategy π * ;

[0074] 1) Initialize LSTM network parameters and global Actor network parameters And the global Critic network parameters θ, initialize the Actor network parameters of the i-th agent as The parameter of the Critic network of the i-th agent is θ i ;

[0075] 2) Set ecod = 1

[0076] 3) Set episode = 1;

[0077] 4) Set thread = 1;

[0078] 5) Initialize the environment S and obtain the initial state s' from the SDN controller;

[0079] 6) t = 1 to T;

[0080] 7) Sequentially extract the state sequence {s(1), s(2), …s(t)} from the state data before time t;

[0081] 8) Input {s(1), s(2), …s(t)} into LSTM;

[0082] 9) The probability parameters (μ, σ) of the state s(t+1) are calculated through the LSTM network and the fully connected layer. 2 );

[0083] 10) According to the probability of state s(t+1), the parameters (μ, σ) are 2 ) Sampling determines the state s(t+1);

[0084] 11) Calculate the VNF backup trigger threshold based on the VNF state transition probability prediction result

[0085]

[0086] Among them, e represents the natural constant, T w Indicates the basic waiting time;

[0087] 12) The SDN controller observes the environment state s and selects the strategy π(s) from the local actor network i(t)|a i (t)) select action a(t);

[0088] 13) If the tolerable delay constraint, resource constraint, other constraint 1 and other constraint 2 are satisfied at the same time, then

[0089] Execute action a(t), obtain reward r(t), transfer to new state s(t+1), and execute the next step;

[0090] Otherwise, the reward value r(t) = -1 / ζ, and the action a(t) is reselected from the local Actor network and 12) is executed;

[0091] 14) Determine thread>N thread Is it true? If yes, execute the next step, otherwise set thread = thread + 1 and return 5);

[0092] 15) Update the global Actor network parameters according to the following formula and global critic network parameter θ;

[0093]

[0094] θ=θ+α2Δθ

[0095] in, is the gradient of the global Actor network, Δθ is the gradient of the global Critic network, α1 and α2 are the learning rates of the global Actor network and the global Critic network respectively, L clip (.) and L(.) are the loss function and loss function after pruning, respectively;

[0096] 16) Update the state sequence {s(1), s(2), ...s(t), s(t+1)};

[0097] 17) Use r(t) and s(t+1) to update the LSTM network parameters;

[0098] 18) Determine episode>M part Is it true? If yes, go to the next step, otherwise set episode = episode + 1 and return to 4);

[0099] 19) Determine if ecod>M all Is it true? If yes, execute the next step, otherwise set ecod = ecod + 1 and return to 3);

[0100] 20) Output the optimal strategy.

[0101] The relationship between the global Actor network and the local Actor network:

[0102] Correspondence: The global actor network is a centralized network that maintains a set of parameters that are a "summary" or "average" of the parameters of multiple local actor networks. Each local actor network is a copy of the global actor network and is used to make decisions and learn independently in different environment instances or different threads. Synchronization: Local actor networks periodically synchronize their parameters from the global actor network to ensure that they are kept up to date. This helps reduce deviations caused by parameter differences between different local networks. Asynchronous updates: In each local environment, the local actor network interacts with the environment independently and calculates gradients based on its experience (state, action, reward, etc.). These gradients are then sent to the global actor network to update the parameters of the global network. The update of the global network is asynchronous, that is, there is no need to wait for all local networks to complete their updates.

[0103] The relationship between the global critic network and the local critic network:

[0104] Correspondence: Similar to the relationship between the global actor network and the local actor network, the global critic network also maintains a set of parameters that are the summary or average of the parameters of multiple local critic networks. Each local critic network evaluates the value of the action selected by the corresponding local actor network and calculates the loss and gradient. Synchronization: Like the local actor network, the local critic network also periodically synchronizes its parameters from the global critic network to maintain the consistency and accuracy of its evaluation. Asynchronous update: The local critic network calculates the loss and gradient based on the experience of the local actor network (including state, action, and reward), and sends these gradients to the global critic network for asynchronous update. This update mechanism helps the global critic network more accurately evaluate the value of actions in different states.

[0105] The global actor network and the global critic network are centralized, and they aggregate the information and parameters of multiple local networks (local actor networks and local critic networks). The local networks are responsible for making decisions and learning independently in their respective environments, and feeding back the learned experience (through gradients) to the global network for updating. This combination of global and local helps improve the parallelism and learning efficiency of the algorithm.

[0106] Compared with the prior art, the present invention has at least the following advantages:

[0107] The method of the present invention effectively solves the challenge of VNF backup in SGIN and improves the reliability of the network and the efficiency of resource utilization. Specifically, it reduces the interruption delay: by predicting faults and backing up in advance, and quickly starting the backup VNF when a fault occurs, the interruption delay caused by VNF failure is significantly reduced. Improve business reliability: through the pre-backup and backup migration mechanism, it ensures that the continuity and reliability of the business can be maintained even under the dynamic changes of the SGIN environment. Optimize resource utilization: through the AoI perception model and optimization algorithm, the network resources are reasonably allocated, so that while ensuring the quality of service, the waste of resources is avoided. Strong adaptability: the algorithm can adapt to the dynamic changes in the SGIN environment and adjust the backup strategy in real time to cope with the changing network conditions. Improve prediction accuracy: the VNF status is predicted through the LSTM network, which improves the prediction accuracy of VNF failures, so as to make more accurate backup decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 This is the SDMSGIN ​​network field diagram.

[0109] Figure 2 It is a VNF state transfer and failure recovery framework.

[0110] Figure 3 Schematic diagram of the LSTM algorithm training process.

[0111] Figure 4 Effective VNF backup ratio under different algorithms.

[0112] Figure 5 Average interruption delay of VNF under different bandwidth adjustment strategies.

[0113] Figure 6 Average interruption delay of VNF under different backup migration mechanisms. DETAILED DESCRIPTION

[0114] The present invention is described in further detail below.

[0115] First, in order to solve the problem of increased AoI of VNF backup caused by link instability, a VNF state transfer and fault recovery model was established, and an AoI-aware VNF backup model was established to adjust bandwidth resources to optimize AoI. Secondly, in order to solve the problem that the relative movement between network nodes causes the VNF backup to be far away from the original VNF, a VNF backup pre-migration mechanism was introduced. Finally, in order to make adaptive optimization strategies for services with different delay sensitivities, the delay sensitivity factor was introduced as a weight, and an optimization problem was established with maximizing the VNF backup benefit as the optimization goal. An algorithm combining LSTM and DPPO was proposed to solve it. Simulation results show that the scheme can reduce the interruption delay of VNFs with serious failures while making reasonable use of network resources.

[0116] A satellite Internet resource scheduling method based on deep reinforcement learning is described as follows:

[0117] First, combining the high coverage of MEO satellites and the high dynamics of LEO satellites, a Software Defined Multi-layer Satellite Ground Integrated Network (SDMSGIN) model is established under the software-defined multi-layer satellite to create a highly dynamic and widely covered network environment for VNF backup.

[0118] Next, in response to the link instability problem in SDMSGIN, the real-time monitoring of the SDN controller accurately detects the VNF status and link status, and dynamically adjusts the backup strategy to meet the needs of rapid recovery when VNF fails. In addition, considering that the relative movement of nodes may cause the distance between the backup node and the original node to increase, the VNF backup migration mechanism is introduced to ensure that the backup always remains on the optimal node, thereby minimizing the fault recovery time.

[0119] Secondly, in order to solve the optimization problem of resource allocation and delay sensitivity of different services during the backup process, an AoI-aware model was constructed. By taking the AoI optimization after successful backup as the benefit and the possible self-repair and emergency migration interruption during the backup process as the cost, corresponding optimization weights were assigned to services with different delay sensitivities. Then, the optimization problem was established under tolerable delay and resource constraints with maximizing the decision benefit after deducting the cost as the optimization goal.

[0120] Finally, in order to solve the problem of uncertain state transition probability when VNF fails, a state prediction algorithm based on LSTM is designed to predict the state information. In order to solve the problem of long model training time and high overhead of single-agent VNF ​​backup algorithm, an AoI-aware VNF hierarchical backup algorithm based on DPPO is proposed. The VNF backup strategy is optimized in real time by predicting the backup needs of VNF through multi-agents, and the optimal VNF backup strategy is obtained through continuous training.

[0121] 1. SDMSGIN ​​model

[0122] Compared with LEO satellites, MEO satellites have a larger delay but a larger coverage area, and then have more stable satellite-to-ground links and inter-layer links between different satellite layers. Therefore, the present invention combines multiple layers of MEO satellites and LEO satellites and integrates the ground network to establish the SDMSGIN ​​model, such as Figure 1 The ground central control server is responsible for the central management of the entire SGIN, including the deployment, configuration and optimization of SFC, as well as the scheduling of network resources such as bandwidth, computing and storage; GEO satellites with ultra-large coverage can serve as auxiliary controllers and work together with the ground central control server to manage SDMSGIN; MEO satellites and LEO satellites are equipped with servers that can process and forward data.

[0123] To model SDMSGIN, its physical layer topology is represented as a weighted undirected graph G P =(N p , L P ), where N p is the set of physical nodes n, L P is the physical link L nn′ (n,n′∈N P ). Due to the high-speed movement of the satellite, the physical link is periodically disconnected and connected, so L is defined nn′ (t) represents the time slot t of the physical link L nn′ The connection state, where L nn′ (t) = 1 indicates physical link L nn′ In the connected state at time slot t, L nn′ (t) = 0 indicates that the physical link L nn′ In order to describe the network resource usage, the remaining computing data capacity of each physical node n in time slot t is defined as The remaining capacity of storage resources is Each physical link L nn′ The remaining bandwidth capacity is

[0124] The virtual application layer where SFC is located is abstracted as a weighted directed graph GV =(F V , L V ), where F V is the VNF set, L V is a set of virtual links. Define the SFC set as U and define Denotes the set of k VNFs that make up the u-th SFC, and defines L u is the virtual link between two adjacent VNFs in the uth SFC A collection of virtual links. The bandwidth requirement is Defining binary variables Indicates the mapping of virtual links. When the virtual link is t timeslot Mapped to physical link L nn′ hour otherwise Defining binary variables Indicates the VNF deployment status. In time slot t, the node n is deployed otherwise definition The computing resource requirements are Storage data requirements are

[0125] 2. VNF state transfer and fault recovery model

[0126] State transfer and failure recovery frameworks such as Figure 2 As shown. The three states of VNF are set as normal state, warning state and severe state. Among them, the normal state indicates that the result caused by the failure of VNF is at a tolerable level; the warning state indicates that the failure of VNF has begun to produce recognizable performance degradation. If it is still in the warning state for a period of time, it means that the failure may not be able to return to normal in a short time. At this time, it is necessary to back up the VNF to other available nodes and transmit status information in real time; the severe state indicates that the failure of VNF is very serious and cannot be restored to normal within the tolerable time. The VNF backup instance needs to be started immediately, and in order to reduce the load of the node, the failed VNF instance needs to be deleted.

[0127] The transition probability between the three states of VNF is defined as: represents the probability that the VNF in the normal state will still remain in the normal state in the next time slot t, is the probability of entering the warning state. If the VNF enters the warning state, the probability In warning state, The probability of completing the repair and returning to normal state. represents the probability of entering a serious state. Let the binary variable and They represent normal state, warning state and critical state respectively. If in that state, it is equal to 1, otherwise it is equal to 0. Indicates the time that the VNF is in the warning state when a fault occurs, and defines Indicates the backup trigger waiting time threshold. Indicates that the server should immediately back up the VNF to other available nodes and transmit the status data in real time. After that, it returns to normal state (with If the probability of transfer is high, the backup of the VNF is deleted to save node resources. The probability of reaching a critical state is 1. A VNF that enters a critical state cannot be restored to other states and will stay in this state (absorbing state) with a probability of 1. Obviously, if the VNF is not backed up at this time, it will cause service interruption and the VNF can only be urgently migrated to other nodes. If the VNF has been backed up in the above warning stage and then transferred to a critical state, it will be restored to a normal state with a probability of 1.

[0128] Since different types of services have different degrees of latency sensitivity, different backup trigger waiting time thresholds need to be designed for different services. Specifically, services with high latency sensitivity require lower latency thresholds. To prevent service degradation, define service The delay sensitivity factor is The more delay-sensitive the service is, the larger the value is. The value of can be expressed as:

[0129]

[0130] Wherein, e represents a natural constant, which is introduced to weaken the excessive influence of probability on the backup trigger waiting time threshold, and Tw represents the basic waiting time.

[0131] 3. Establish an AoI-aware VNF backup model

[0132] Since VNF generates configuration information, event logs and other status data during operation, in order to quickly recover and start the VNF instance from a failure, an additional logical synchronization link is required to transmit status information in real time so that the status data between the backup node and the original node VNF instance can be synchronized in real time. The backup is Assume that the VNF data packet is generated and transmitted at the beginning of each time slot. The node n where the backup is needed The size of the state data packet sent by node n′ is Assume that the bandwidth occupied by the logical link used to transmit VNF ​​status in real time is AoI is used to express the time difference between the VNF backup instance and the main VNF instance, that is, the difference between the time when the VNF backup instance receives the data packet and the time when the data packet is generated. Let t p is the end time of the pth time slot (p is a natural number), and the length of a time slot is τ. The AoI at the end time of the tth time slot is expressed as:

[0133]

[0134] in, is the AoI at the end of the t-th time slot, t p is the end time of the pth time slot, p is a natural number, τ is the length of the time slot, is time slot t The node n where the backup is needed The size of the state data packet sent by node n′, Occupies bandwidth for logical links used to deliver VNF status in real time;

[0135] Since the link disconnection status can be predicted by the network topology, the link disconnection time can be avoided by adjusting the sending of data packets. Therefore, it is assumed that data packets will not be sent during the disconnection process, that is, it is assumed that there will be no packet loss. In this case, the time actually available for data transmission in each time slot is Then in each time slot, the amount of data that the backup node can receive is Considering that the original VNF and the backup VNF may pass through multiple physical links, the amount of data not transmitted in each time slot is in is the number of forwarding hops between the original VNF and the backup VNF. When the pth time slot ends, the total amount of untransmitted data will be the sum of the amount of untransmitted data in all previous time slots. Therefore, the t time slot The accumulated amount of data not transferred It is expressed as:

[0136]

[0137] in, is the number of forwarding hops between the original VNF and the backup VNF, The backup interruption duration parameter indicates VNF backup The sum of the interruption durations between node n and the original VNF in time slot t.

[0138] 4. Adaptive bandwidth adjustment strategy

[0139] For more urgent situations, network resources should be sacrificed first to avoid service interruption, so the delay sensitivity factor Larger services should be allocated larger bandwidth and the probability of transitioning to a critical failure should be predicted by the agent. The larger the bandwidth, the larger it should be. Therefore, an adaptive bandwidth adjustment strategy is proposed, that is, to allocate a larger bandwidth to the more urgent situations to quickly synchronize the state data, and define the bandwidth adjustment factor as It is expressed as:

[0140]

[0141] in, is the delay sensitivity factor, The probability of the VNF state transitioning to a serious fault predicted by the agent;

[0142] The AoI at the end of the tth time slot is expressed as:

[0143]

[0144] 5. VNF backup and migration mechanism

[0145] Due to the relative movement of the satellite, the interruption of the link will lead to an increase in AoI. The relative movement between the backup node and the original VNF node may increase the number of physical links and thus increase the forwarding hops, which will also increase AoI. In order to solve the above problems, a VNF backup migration mechanism is introduced, that is, the VNF backup instances with too long cumulative interruption time should be pre-migrated to other nodes to reduce the routing forwarding hops. Although VNF instances can be pre-migrated, the synchronization of state data will change AoI. Indicates VNF backup on node n Migrate to node n' at time slot t.

[0146] The AoI at the end of the tth time slot is expressed as:

[0147]

[0148] in, Indicates VNF backup on node n Migrate to node n′ at time slot t;

[0149] If the VNF is not backed up in advance due to prediction errors, the emergency migration of the VNF will cause a lot of interruption delays. The only way to reduce this delay is to increase the migration bandwidth urgently. Since the information difference between the node to be migrated and the original node is the size of the entire VNF data to be migrated, the emergency migration of this data will cause interruption delays. Therefore, this migration delay is equivalent to AoI, and a binary variable is set Indicates that the emergency migration occurs, otherwise The AoI is finally expressed as:

[0150]

[0151] in, The total amount of data required to transfer the VNF, including VNF status data and re-

[0152] The sum of the data size required for the newly instantiated VNF;

[0153]

[0154] in, Storage resource requirements for backup.

[0155] 6. Create an optimization problem

[0156] The backup of the failed VNF transferred from the warning state to the serious state is an effective backup, while the backup when it recovers to the normal state is an invalid backup. Invalid backup will occupy too many network resources, but if there is no backup in advance when the VNF transfers to the serious state, it will cause service interruption. The present invention takes the optimization of AoI under the effective backup of VNF as the benefit, and the optimization of AoI and service interruption under the invalid backup as the cost, and jointly considers the benefits and costs of VNF backup and optimized AoI under the constraints of network resources and tolerable delay to establish an optimization problem.

[0157] The backup of a failed VNF transferred from the warning state to the critical state is an effective backup, while the backup that exists when it recovers to the normal state is an invalid backup. Invalid backups will occupy too many network resources, but if there is no backup in advance when the VNF transfers to the critical state, it will cause service interruption. The optimization of AoI under the effective backup of VNF is regarded as the benefit, and the optimization of AoI and service interruption under the invalid backup are regarded as the cost. Under the constraints of network resources and tolerable delay, the benefits and costs of VNF backup and optimized AoI are jointly considered to establish an optimization problem.

[0158] Defining binary variables and Respectively represent the deployment of VNF backup and the deployment of state synchronization link. The backup of is mapped to node n during time slot t, then Otherwise Similarly, if during time slot t, and its backup The state synchronization link between the two is embedded into the physical link L nn′ , in, then otherwise, Since the migration of VNF backup is done in advance, the migration process of VNF backup will not affect the original VNF processing data. Therefore, it is assumed that the migration of VNF backup is completed instantly. Each VNF can have only one backup at the same time. In order to ensure that the traffic is not split, other constraints are introduced:

[0159]

[0160] in, and They represent the deployment of VNF backup and the deployment of state synchronization link respectively;

[0161] If a VNF fails on a node server, all VNFs of this type on this node will stop working. Each VNF and its backup cannot be in the same node, which introduces another constraint:

[0162]

[0163] in, VNF deployment status.

[0164] Once the failure of a VNF reaches a critical level, the backup instance of the VNF should be started immediately. Therefore, the backup instance should be guaranteed to have remaining resource capacity in its nodes and path links. In each time slot, the computing resource capacity and storage resource capacity occupied by all VNF instances and backup instances on the node should not exceed the remaining available resource capacity, introducing resource constraints:

[0165]

[0166] in, They are The computing resource requirements for backup and The computing resource requirements, They are Backup storage resource requirements and Storage resource requirements, Calculate the resource capacity for each physical node n's appetite in time slot t, The remaining capacity of the storage resource;

[0167] The sum of the bandwidth occupied by the virtual links between VNFs of adjacent nodes, the logical synchronization links for transmitting status, and the migrated backup VNF is less than the remaining available bandwidth capacity of the physical link:

[0168]

[0169] in, Virtual Link The mapping situation, Virtual Link The broadband demand is the remaining bandwidth capacity in each physical link;

[0170] Defining binary variables if Backup has been performed in time slot t and backup recovery can be performed in this time slot. Otherwise 0:

[0171]

[0172] Among them, IF(.) is an indicator function. If true, IF(.) is 1, otherwise IF(.) is 0;

[0173] When a VNF in the warning state is transferred to the critical state, if a VNF backup instance is deployed in advance, the time it takes to start this VNF backup instance to transfer the remaining state data is the size of the VNF backup AoI; if there is no VNF backup instance, the time it takes to urgently migrate the VNF is the equivalent AoI size. Both of these situations will cause interruption delays. In order to prevent the interruption delay from being too large and causing a violation of the service level agreement, it is necessary to The fault recovery of VNF is completed by starting the backup instance or urgently migrating VNF, and the tolerable delay constraint is introduced: in, is the critical status of the VNF in time slot t.

[0174] Will The reason why is used as the tolerable coefficient is that different delay-sensitive services have different tolerable delays. The more delay-sensitive the service is, the smaller the tolerable delay should be, and the less delay-sensitive the service is, the larger the tolerable delay should be.

[0175] In order for VNF to recover quickly when a failure occurs, the AoI of VNF backup should be minimized to reduce the time to synchronize the remaining state data. However, effective backup and fault recovery will only generate benefits from AoI optimization if and only if the prediction of VNF state transition probability is correct. Otherwise, the optimization of AoI in the case of wrong prediction will be the cost of occupying additional network resources. Therefore, the optimization objective function is defined as It consists of two parts: gain (t) and cost (t):

[0176] For the benefit part of the objective function, it should be the AoI when the VNF completes the backup in the warning state at time slot t and transfers to the severe state at time slot t+1. At this time, the smaller the AoI of the objective function, the greater the benefit. In addition, minimizing the AoI for services with different delay sensitivity has different benefits. The delay sensitivity is high ( The larger the value is, the greater the benefit is. Therefore, the benefit part of the objective function is expressed as:

[0177]

[0178] in, is the critical state of the VNF at time slot t+1, is the warning status of the VNF in time slot t;

[0179] The cost part of the objective function is divided into three parts: Cost(t)=Cost1(t)+Cost2(t)+Cost3(t).

[0180] The first part of the cost Cost1(t) is generated when the VNF is in the warning state at time slot t and backs up the VNF, but due to a prediction error, the VNF is successfully repaired at time slot t+1 and transferred to the normal state. The AoI optimization of the VNF previously occupied the cost of additional network resources. In addition, in order to reduce the impact of unexpectedness on the solution of the optimization goal, the lower the probability of this situation, the lower the cost should be. Therefore, Cost1(t) is expressed as:

[0181]

[0182] in, is the probability that VNF ​​completes recovery from warning state to normal state in time slot t+1, is the normal state of VNF at time slot t+1, is the normal state of the VNF in time slot t;

[0183] The second part of the cost, Cost2(t), is the backup of the VNF when the VNF is in the warning state at time slot t, and it is still in the warning state at time slot t+1. The backup and optimization of AoI at this time are premature backups, and also the cost of occupying less network resources. However, the priority of business uninterrupted should be greater than the occupation of network resources. In order to avoid sacrificing reliability to reduce the consumption of network resources, the weight of this cost should be further weakened, taking the square of the probability of this situation. Therefore, Cost2(t) is expressed as:

[0184]

[0185] in, is the probability that the VNF is in the warning state at time slot t+1, is the warning status of the VNF in time slot t+1, is the warning status of the VNF in time slot t;

[0186] The third part of the cost Cost3(t) is the VNF in the warning state at time slot t without backup VNF, but transferred to the severe state at time slot t+1. If the VNF is in the warning state without backup VNF, but transferred to the warning state due to prediction error, the equivalent of the emergency migration VNF interruption delay will be generated. Similarly, in order to reduce the impact of unexpectedness on the optimization solution, the state transition probability is weighted, and Cost3(t) is expressed as:

[0187]

[0188] in, is the probability that the VNF enters the severe state from the warning state at the t+1 time slot.

[0189] 7. MDP Model

[0190] The selection decision of VNF backup, the migration decision of backup and the bandwidth adjustment decision not only affect the network performance at the current moment, but also affect the future state and decision. In order to capture the sequential and cumulative effects of such decisions and provide a basis for finding the long-term optimal strategy, the optimization problem is transformed into an MDP model and a four-tuple (S, A, P, R) is defined to represent the state space, action space, state transition probability and reward function respectively.

[0191] State space: Define s(t) = {Res(t), L(t), P(t)}∈S to represent the SDMSGIN ​​network state space of time slot t, where The state space representing the VNF resource requirements, L(t) = {l nn′ (t)|n∈N p} represents the state space of the physical link, The state space representing the three key state transition probabilities of the VNF;

[0192] Action space: define a(t) = {X(t), Y(t), Z(t)}∈A, where and represents the action space of VNF backup decision and state synchronization link mapping decision, is the action space of backup migration decision.

[0193] State transition probability: The transition probability P represents the probability that the agent transfers to the next state s(t+1) after taking action a(t) in state s(t), expressed as p(s(t+1)|s(t), a(t).

[0194] Reward function: To maximize the reward R, the reward for time slot t is expressed by the optimization objective:

[0195] If the constraints in P1 are not met, then: r(t) = 1 / ζ, where ζ represents an infinitesimal quantity that tends to 0; the reward discount factor for time slot t is set to γ∈(0, 1), then its cumulative discounted reward is: Where k is the number of iterations. Obviously, the more iterations, the smaller the cumulative reward. It represents the decision of taking action a(t) in state s(t). The strategy value function is used to judge the quality of the strategy in time slot t: Qπ(s(t), a(t))=E[R π (t)|s(t),a(t)], where Q π (s(t), a(t)) is the strategy value function, and E[.] is the expectation.

[0196] It is expressed by the iteration of Bellman equation: Q π (s(t), a(t)) = E[r(t) + γQ π (s(t+1), a(t+1))], where r(t) is the reward function.

[0197] The optimal decision π* is expressed as:

[0198] 7. State prediction algorithm based on LSTM

[0199] like Figure 3 The figure shows the framework of the probability distribution parameters of the policy network output, which shows the training process of the policy network using the LSTM network. Specifically, in order to capture the long-term dependencies of the data, first, the LSTM network captures a sequence of past states {s(1), s(2), ...s(t)}. These states are mainly the state transition probabilities P(t) of the VNF observed by the agent in the historical dynamic network environment. This sequence is sequentially input into the LSTM network. After repeated processing, the LSTM network outputs a hidden state sequence, in which each element represents the output of the LSTM unit. Then, these outputs pass through a fully connected layer to generate the probability distribution parameters of the state s(t+1). Finally, the probability distribution of the output of the output layer includes the mean μ and the variance σ 2 , where the mean μ represents the average value of the predicted state, reflecting the most likely case of the state s(t+1) predicted by the LSTM network based on the input data, and the variance σ 2 Indicates the uncertainty of the prediction or the dispersion of the predicted values ​​around the mean. The smaller the variance, the more confident the model is in its predicted state transition probability, and the lower the uncertainty of the prediction result. The larger the variance, the higher the uncertainty of the prediction. During the training phase, the agent selects the backup and migration strategies of the agent by sampling from this distribution to ensure that the strategy selection is random and exploratory. As training progresses, the variance σ 2, gradually decreases, indicating that the network gradually stabilizes during learning. After the training is completed, the VNF status prediction results are obtained. The agent can determine the backup and migration operations of the VNF based on the mean of the decision, and consider the variance to evaluate the uncertainty and risk of the decision. The training process of the LSTM-based VNF status prediction algorithm is shown in Algorithm 1.

[0200] The policy network introduces the LSTM training process

[0201]

[0202] 8. VNF backup algorithm based on DPPO

[0203] Each VNF is regarded as an intelligent agent, and is equipped with a pair of Actor network and Critic network. The Actor network parameters of the i-th intelligent agent are The parameter of the Critic network of the i-th agent is θ i ; First, each agent will explore the network environment, collect the network resource information required by VNF, the state data of the physical link of SDMSGIN, and the three states of VNF, and perform VNF backup behavior based on the collected state data, and calculate the immediate reward. Through interaction with the network environment, the local PPO (proximal policy optimization algorithm) network of each agent can gradually optimize the VNF backup strategy based on the reward information obtained. For each agent, the Critic part of its PPO network uses the state value function and advantage function to evaluate the current output strategy of the Actor network, where the cumulative reward expectation that the i-th agent can obtain by adopting the mapping strategy in the current state is expressed as the state value function:

[0204] V(s i (t)) = E[r i (t)+γV(s i (t+1))]

[0205] The agent is in state s i (t) Take action a i (t) can get a cumulative expected value, that is, the state action value function is expressed as:

[0206] Q(s i (t), a i (t)) = E[r i (t)+YQ(s i (t+1), a i (t+1))]

[0207] And use advantage function A π (s i (t), a i(t)) to evaluate the action a made by agent i i (t) For the current state s i (t) advantages and disadvantages:

[0208] A π (s i (t), a i (t)) = Q(s i (t), a i (t))-V(s i (t))

[0209] There are usually methods such as TD advantage estimation, Q value advantage estimation and maximum entropy advantage estimation for advantage estimation of policy gradient reinforcement learning. However, considering that this model contains dynamic environmental factors, such as fault transfer probability and bandwidth adjustment, in order to effectively update the advantage estimation in this dynamic environment and help the agent better understand the impact of environmental changes on long-term benefits and make better decisions, a more flexible generalized advantage estimation is used. The advantage function of agent i is expressed as:

[0210]

[0211] In the formula, T is the step size of a training, and the parameter λ is the discount attenuation factor. When making VNF backup strategies and adjusting bandwidth, long-term benefits need to be considered, especially when considering the optimization of AoI. GAE improves the efficiency and stability of the policy gradient algorithm by adjusting the trade-off between deviation and variance, allowing the agent to find a balance between immediate rewards and long-term benefits, which helps to formulate better backup strategies and bandwidth adjustment strategies. i (t) is the TD error, expressed as:

[0212] δ i (t) = r i (t)+γ(V(s i (t+1))-V(s i (t))

[0213] Through the continuous evaluation of the Critic network of agent i, the TD error will gradually become smaller. In order to measure the difference between the predicted value and the marked value in this process, the loss function is introduced:

[0214]

[0215] In order to limit the update range of the new strategy, thereby reducing the sensitivity of the learning rate and improving the stability of the algorithm, the DPPO algorithm adds the action ratio ρ(φ i ) parameter to represent the important sampling weight. Let the new strategy be π new (s i(t)|a i (t),φ i ), the old strategy is π old (s i (t)|a i (t),φ i ) small ratio ρ(φ i )<1 means that the agent prefers to choose the old strategy, otherwise it chooses the new strategy. The action ratio is expressed as follows:

[0216] Too large a strategy update can lead to unstable learning. In order to ensure that the agent does not lose its learning direction due to excessive changes when exploring backup strategies and bandwidth adjustment decisions, a PPO algorithm with a clipped optimization target is used to prevent the new strategy from seriously deviating from the old strategy. The loss function after clipping is expressed as:

[0217] L clip (φ i )=E[min(A(s i (t), a i (t))clip(ρ(φ i ), 1-δ, 1+δ), ρ(φ i )A(s i (t), a i (t)))]

[0218] Where δ is the hyperparameter of the clipping strategy update amplitude, clip(ρ(φ i ), 1-δ, 1+δ) represents the ratio of the old and new strategies to the new strategies The part beyond the interval (1-δ, 1+δ) is trimmed as follows:

[0219]

[0220] The parameters of the global PPO network are updated by the loss functions of the local Critic and Actor networks, where the gradient of the global Actor network is and parameters And the update method of the gradient Δθ and parameter θ of the global Critic network is:

[0221]

[0222] φ=φ+α1Δφ

[0223]

[0224] θ=θ+α2Δθ

[0225] in, is the gradient of the global Actor network, Δθ is the gradient of the global Critic network, α1 and α2 are the learning rates of the global Actor network and the global Critic network respectively, L clip (.) and L(.) are the pruned loss function and the global network parameters after each round of training of the loss function, respectively. The specific training process of the algorithm is shown in Algorithm 2.

[0226]

[0227] 9. Simulation and Evaluation

[0228] In order to evaluate the performance of the method of the present invention, experimental simulation was carried out on a computer with an AMD Ryzen 7 processor and 16GB of memory. First, the satellite network topology was built in the STK software, and the generated data was saved. Then, a custom environment, that is, the system model part, was built in the ycharm2022 software, and the data generated in STK was imported into the custom environment. Then, the Pytorch toolkit was used to build LSTM. Finally, the DPPO algorithm was used to continuously train the intelligent agent to obtain various experimental results.

[0229] (1) First, six polar orbits with an altitude of 1000 km and six LEO satellite constellations on each orbit, and three MEO satellite constellations with an altitude of 10,000 km and an inclination of 550 and four satellites on each orbit were built in STK. Twelve ground stations were set up. After 1800s of simulation, link visibility report data was obtained and applied to a custom environment built by Py charm software. The custom environment was built using the network system simulation parameters shown in Table 1. The LSTM model is responsible for capturing the time series dependency of VNF states, while the DPPO algorithm optimizes the network backup and resource allocation strategies based on these prediction results. The present invention combines LSTM and DPPO algorithms to design VNF state prediction and backup strategy optimization. Distributed reinforcement learning is used to process large-scale state space and action space, and a data set is constructed. After the agent runs 20 times in the environment, the generated data trajectory is saved in the buffer as a batch data set. In the LSTM-DPPO algorithm, the LSTM network configuration includes 2 hidden layers, each with 100 hidden units. The DPPO part uses a two-layer fully connected network with 256 neurons in each layer to meet complex decision-making needs. The learning rate of the Critic and Actor networks is set to 0.002, the clipping parameter is set to 0.2, and the target network update frequency is 5. The overall configuration of the algorithm aims to balance learning efficiency and prediction accuracy while maintaining adaptability to dynamic network environments.

[0230] Table 1 Simulation parameters

[0231]

[0232] (2) Performance evaluation

[0233] When the learning rate is 0.0005, the cumulative reward value is too small, resulting in a small step size for updating the model parameters, causing the model to move slowly in the parameter space, and thus slowing down the convergence of the reward value, and making it difficult to converge to a higher reward value; when the learning rate is 0.002, the value is too large, resulting in a large step size for updating the parameters. Although this speeds up the convergence of the algorithm, the large amplitude of the parameter update causes a deviation from the optimal solution and is very unstable; when the learning rate is 0.001, it is relatively stable and can converge to a higher reward value.

[0234] The LSTM-DPPO algorithm converges faster and is more stable, while the LSTM-PPO algorithm and the LSTM-SAC algorithm converge more slowly and have relatively large fluctuations. This is because the DPPO algorithm uses multi-agent distributed exploration of the network environment. Although this leads to a higher algorithm space complexity, parallel processing reduces the algorithm time complexity, allowing it to more efficiently find a broader solution space.

[0235] When a VNF failure is transferred from a warning state to a serious state, if the VNF is backed up in advance for rapid recovery, it is considered an effective backup. The higher the effective VNF backup ratio within a period of time, the higher the accuracy of the algorithm in predicting serious VNF failures. Figure 4 The effective VNF backup ratios of the LSTM-DPPO algorithm, the LSTM-PPO algorithm, and the LSTM-SAC algorithm are compared. It can be seen that the effective VNF backup ratio of the LSTM-DPPO algorithm is significantly higher than that of the LSTM-PPO algorithm and the LSTM-SAC algorithm, and the performance degradation rate is also significantly lower as the number of VNFs increases.

[0236] Compared with LSTM-PPO and LSTM-SAC,the LSTM-DPPO algorithm improves the accuracy by about 11.06%N 17.93%,which shows that the LSTM-DPPO algorithm is more accurate in predicting the state transition probability of VNF,and thus can make better VNF backup decisions.

[0237] Figure 5The adaptive bandwidth adjustment strategy proposed in the present invention is compared with the fixed bandwidth allocation strategy and the random bandwidth allocation strategy for the average interruption delay of VNFs with different delay sensitivities. It can be obviously seen that the average interruption delay of VNFs under the adaptive bandwidth adjustment strategy proposed in the present invention is greatly reduced, which is about 41.43%-46.37% lower than that of the fixed bandwidth allocation strategy, and the degree of reduction is greater for VNFs with higher delay sensitivity. On the one hand, the adaptive bandwidth adjustment strategy proposed in the present invention will adjust the bandwidth of the state synchronization link according to different state transition probabilities and different delay sensitivities. For VNFs with a higher probability of transitioning to a severe state predicted by LSTM, a larger bandwidth will be allocated to reduce the backup AoI, thereby reducing the interruption delay when the VNF recovers from a fault. On the other hand, when LSTM mispredicts that a VNF in a warning state transitions to a severe state, no backup is performed, and the VNF and state data need to be urgently migrated. The adaptive bandwidth adjustment strategy will also adjust the bandwidth required for the emergency migration of the VNF according to different delay sensitivities. The higher the delay sensitivity, the larger the allocated bandwidth, so the more the interruption delay is reduced.

[0238] Figure 6 The average interruption delay of VNF under the backup migration mechanism proposed in the present invention is compared with that under the non-backup migration mechanism and the backup random migration mechanism. It can be seen that the average interruption delay of VNF under the backup migration mechanism proposed in the present invention is reduced by about 37.23%-43.62% compared with that under the non-backup migration mechanism. This is because the unstable links and relative movement of nodes in the SGIN environment will lead to the increase of VNF backup AoI. The backup VNF is migrated close to the original VNF, or migrated to a location with less interruption time of the state synchronization link, so that the backup AoI is reduced. In addition, the migration mechanism also adopts an adaptive bandwidth adjustment strategy, so that it does not occupy too many network resources while ensuring service reliability.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A satellite Internet resource scheduling method based on deep reinforcement learning, characterized in that: The steps include: S1: Combining the high coverage of MEO satellites and the high dynamics of LEO satellites, a satellite-ground fusion network SDMSGIN ​​model under software-defined multi-layer satellites is established. The physical layer topology of the SDMSGIN ​​model is represented as a weighted undirected graph G P =(N P , L P ), where N P is the set of physical nodes n, L P is the physical link L nn′ (n,n′∈N P ), define L nn′ (t) represents time slot t physical link L nn′ The remaining computing data capacity of each physical node n in time slot t is defined as The remaining capacity of storage resources is Each physical link L nn′ The remaining bandwidth capacity is The virtual application layer where SFC is located is abstracted as a weighted directed graph G V =(F V , L V ), where F V is the VNF set, L V is a set of virtual links, define the SFC set as U, and define Denotes the set of k VNFs that make up the u-th SFC, and defines L u is the virtual link between two adjacent VNFs in the uth SFC A collection of virtual links The bandwidth requirement is Defining binary variables Indicates the mapping of virtual links and defines The computing resource requirements are Storage data requirements are S2: A VNF state transfer and fault recovery model was established to solve the problem of increased AoI of VNF backup caused by link instability. An AoI-aware VNF backup model was established to adjust bandwidth resources to optimize AoI. S3: Introduce a VNF backup migration mechanism in the AoI-aware VNF backup model to solve the problem that the VNF backup is far away from the original VNF due to the relative movement between network nodes, and introduce an adaptive bandwidth adjustment strategy to allocate larger bandwidth in more urgent situations; S4: Introduce the delay sensitivity factor as a weight to make adaptive optimization strategies for services with different delay sensitivities. Under the constraints of tolerable delay and resources, establish an optimization problem with maximizing the VNF backup benefit as the optimization goal. S5: A method combining LSTM and DPPO is proposed to solve the optimization problem proposed in S4. First, the optimization problem is transformed into MDP, and a four-tuple (S, A, P, R) is defined to represent the state space, action space, state transition probability and reward function respectively. The VNF state transition probability is predicted through LSTM, and the LSTM is trained using the state sequence before the VNF time slot t. When the maximum number of iterations is reached, the trained LSTM is obtained and the new state is output; The distributed reinforcement learning algorithm DDPO is used to optimize the backup strategy. When DDPO reaches the maximum number of iterations, the strategy obtained is the optimal strategy.

2. A satellite Internet resource scheduling method based on deep reinforcement learning as claimed in claim 1, characterized in that: In S2, the process of establishing the VNF state transfer and fault recovery model is as follows: The states of VNF are set as normal state, warning state and critical state; the transition probabilities between the three states of VNF are: In normal state The probability that a VNF remains in a normal state at time slot t is is the probability of entering the warning state. If the VNF enters the warning state, the probability In warning state, The probability of completing the repair and returning to normal state; let the binary variable and Respectively represent The normal state, warning state and critical state of a VNF in time slot t. If a VNF is in one of the three states at time slot t, the binary variable corresponding to the state is 1, otherwise it is 0; set up Indicates The time that a VNF is in the warning state when a fault occurs in time slot t, and define Indicates the backup trigger waiting time threshold; if The server should immediately back up the VNF to other available nodes and transmit status data in real time. After that, it returns to normal state. If the probability of transfer is high, the backup of the VNF is deleted to save node resources; the VNF in the warning state is The probability of reaching a critical state is 1. A VNF that enters a critical state cannot be restored to other states and will stay in this state, i.e., the absorbing state, with a probability of 1. If the VNF has been backed up and transferred to a critical state in the above warning stage, it will be restored to a normal state with a probability of 1. definition The delay sensitivity factor is The more delay-sensitive the service is, the larger the value is. The value of is expressed as: Where e represents a natural constant, T w Indicates the basic waiting time.

3. A satellite Internet resource scheduling method based on deep reinforcement learning as claimed in claim 2, characterized in that: In S2, the process of establishing the AoI-aware VNF backup model is as follows: The AoI at the end of the tth time slot is expressed as: in, is the AoI at the end of the t-th time slot, t p is the end time of the pth time slot, p is a natural number, τ is the length of the time slot, is time slot t The node n where the backup is needed The size of the state data packet sent by node n′, Occupies bandwidth for logical links used to deliver VNF status in real time; t time slot The accumulated amount of data not transferred It is expressed as: in, is the number of forwarding hops between the original VNF and the backup VNF, The backup interruption duration parameter indicates VNF backup The sum of the interruption durations between node n and the original VNF in time slot t.

4. A satellite Internet resource scheduling method based on deep reinforcement learning as claimed in claim 3, characterized in that: In S3, the adaptive bandwidth adjustment strategy is: Define the bandwidth adjustment factor as It is expressed as: in is the delay sensitivity factor, The probability of the VNF state transitioning to a serious fault predicted by the agent; The AoI at the end of the tth time slot is expressed as:

5. A satellite Internet resource scheduling method based on deep reinforcement learning as claimed in claim 4, characterized in that: In S3, the VNF backup migration mechanism is: The AoI at the end of the tth time slot is expressed as: in, Indicates VNF backup on node n Migrate to node n′ at time slot t; If the VNF is not backed up in advance due to prediction errors, the emergency migration of the VNF will cause a lot of interruption delays. The only way to reduce this delay is to increase the migration bandwidth urgently. Since the information difference between the node to be migrated and the original node is the size of the entire VNF data to be migrated, the emergency migration of this data will cause interruption delays. Therefore, this migration delay is equivalent to AoI, and a binary variable is set Indicates that the emergency migration occurs, otherwise The AoI is finally expressed as: in, The total amount of data that needs to be transferred to migrate the VNF; in Storage resource requirements for backup.

6. A satellite Internet resource scheduling method based on deep reinforcement learning as claimed in claim 5, characterized in that: In S4, the process of establishing the optimization problem with maximizing the VNF backup benefit as the optimization goal is: Assume that the migration of VNF backup is completed instantly, each VNF can have at most one backup at the same time, and to ensure that traffic is not split, introduce other constraints: in and They represent the deployment of VNF backup and the deployment of state synchronization link respectively; Each VNF and its backup cannot be in the same node, which introduces another constraint: in VNF deployment status; Introducing resource constraints: in, They are The computing resource requirements for backup and The computing resource requirements, They are Backup storage resource requirements and Storage resource requirements, Calculate the resource capacity for each physical node n's appetite in time slot t, The remaining capacity of the storage resource; The sum of the bandwidth occupied by the virtual links between VNFs of adjacent nodes, the logical synchronization links for transmitting status, and the migrated backup VNF is less than the remaining available bandwidth capacity of the physical link: in, Virtual Link The mapping situation, Virtual Link The broadband demand is the remaining bandwidth capacity in each physical link; Defining binary variables if Backup has been performed in time slot t and backup recovery can be performed in this time slot. Otherwise 0: Among them, IF(.) is an indicator function. If true, IF(.) is 1, otherwise IF(.) is 0; Introducing tolerable delay constraints: in, is the critical state of the VNF in time slot t; Define the optimization objective function It consists of two parts: gain (t) and cost (t): in, is the critical state of the VNF at time slot t+1, is the warning status of the VNF in time slot t; For the cost part of the objective function, it is divided into three parts: Cost(t)=Cost1(t)+Cost2(t)+Cost3(t); in, is the probability that VNF ​​completes recovery from warning state to normal state in time slot t+1, is the normal state of VNF at time slot t+1, is the normal state of VNF in time slot t; in, is the probability that the VNF is in the warning state at time slot t+1, is the warning status of the VNF in time slot t+1, is the warning status of the VNF in time slot t; in, is the probability that the VNF enters the severe state from the warning state at the t+1 time slot.

7. A satellite Internet resource scheduling method based on deep reinforcement learning as claimed in claim 6, characterized in that: In 54, the state space, action space, state transition probability and reward function are respectively: State space: Define s(t) = {Res(t), L(t), P(t)}∈S to represent the SDMSGIN ​​network state space of time slot t, where The state space representing the VNF resource requirements, L(t) = {l nn′ (t)|n∈Np} represents the state space of the physical link, The state space representing the three key state transition probabilities of the VNF; Action space: define a(t) = {X(t), Y(t), Z(t)}∈A, where and represents the action space of VNF backup decision and state synchronization link mapping decision, is the action space of backup migration decisions; State transition probability: The transition probability P represents the probability that the agent will transfer to the next state s(t+1) after taking action a(t) in state s(t), expressed as p(s(t+1)|s(t), a(t); Reward function: To maximize the reward R, the reward for time slot t is expressed by the optimization objective: If the constraints in P1 are not met, then: r(t) = 1 / ζ, where ζ represents an infinitesimal quantity that tends to 0; the reward discount factor for time slot t is set to γ∈(0, 1), then its cumulative discounted reward is: Where k is the number of iterations. Obviously, the more iterations, the smaller the cumulative reward. It represents the decision of taking action a(t) in state s(t). The strategy value function is used to judge the quality of the strategy in time slot t: Q π (s(t), a(t)) = E[R π (t)|s(t),a(t)], where Q π (s(t), a(t)) is the strategy value function, and E[.] is the expectation; It is expressed by the iteration of Bellman equation: Q π (s(t), a(t)) = E[r(t) + γQ π (s(t+1), a(t+1))], where r(t) is the reward function; The optimal decision π* is expressed as:

8. A satellite Internet resource scheduling method based on deep reinforcement learning as claimed in claim 7, characterized in that: In S5, the process of obtaining the optimal decision is: Input: state sequence {s(1), s(2), …s(t), G P =(N P ,L P ), G V =(F V ,L V ), discount factor γ, objective function update frequency F, number of threads N thread , the number of global network iterations M all , the number of local network iterations M part , learning rates α1 and α2; Output: optimal strategy π * ; 1) Initialize LSTM network parameters and global Actor network parameters And the global Critic network parameters θ, initialize the Actor network parameters of the i-th agent as The parameter of the Critic network of the i-th agent is θ i ; 2) Set ecod = 1 3) Set episode = 1; 4) Set thread = 1; 5) Initialize the environment S and obtain the initial state s' from the SDN controller; 6) t = 1 to T; 7) Sequentially extract the state sequence {s(1), s(2), …s(t)} from the state data before time t; 8) Input {s(1), s(2), …s(t)} into LSTM; 9) The probability parameters (μ, σ) of the state s(t+1) are calculated through the LSTM network and the fully connected layer. 2 ); 10) According to the probability of state s(t+1), the parameters (μ, σ) are 2 ) Sampling determines the state s(t+1); 11) Calculate the VNF backup trigger threshold based on the VNF state transition probability prediction result Among them, e represents the natural constant, T w Indicates the basic waiting time; 12) The SDN controller observes the environment state s and selects the strategy π(s) from the local actor network i (t)|a i (t)) select action a(t); 13) If the tolerable delay constraint, resource constraint, other constraint 1 and other constraint 2 are satisfied at the same time, then Execute action a(t), obtain reward r(t), transfer to new state s(t+1), and execute the next step; Otherwise, the reward value r(t) = -1 / ζ, and the action a(t) is reselected from the local Actor network and 12) is executed; 14) Determine thread>N thread Is it true? If yes, execute the next step, otherwise set thread = thread + 1 and return 5); 15) Update the global Actor network parameters according to the following formula and global critic network parameter θ; θ=θ+α2Δθ in, is the gradient of the global Actor network, Δθ is the gradient of the global Critic network, α1 and α2 are the learning rates of the global Actor network and the global Critic network respectively, L clip (.) and L(.) are the loss function and loss function after pruning, respectively; 16) Update the state sequence {s(1), s(2), …s(t), s(t+1)}; 17) Use r(t) and s(t+1) to update the LSTM network parameters; 18) Determine episode>M part Is it true? If yes, go to the next step, otherwise set episode = episode + 1 and return to 4); 19) Determine if ecod>M all Is it true? If yes, execute the next step, otherwise set ecod = ecod + 1 and return to 3); 20) Output the optimal strategy.

Images