Satellite internet virtual network function deployment and scheduling optimization method
By combining Lyapunov optimization and deep reinforcement learning, a joint latency-resource constraint model is constructed to optimize the deployment and scheduling of virtual network functions (VNFs) in satellite networks. This solves the problems of low resource utilization and insufficient static deployment strategies in traditional satellite networks, achieving efficient and robust VNF deployment and scheduling, and improving the long-term stability and resource utilization efficiency of the system.
Patent Information
- Application Number
- CN202511697818.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-02-10
AI Technical Summary
Traditional satellite networks, designed for specific missions, have infrastructure and network resources that are isolated, which limits the dynamic sharing of physical resources. This results in low resource utilization and high operating costs. Furthermore, static VNF deployment strategies may lead to over-configuration and insufficient resource utilization, making it difficult to achieve efficient and robust VNF deployment and scheduling in dynamic satellite network environments.
By combining Lyapunov optimization theory with deep reinforcement learning, a joint latency-resource constraint model for node data queues and virtual cost queues is constructed. Through the action decomposition structure method, the deployment and scheduling of virtual network functions are optimized. The Actor-Critic strategy is adopted to select the near-optimal joint deployment and scheduling scheme under Lyapunov constraints, thereby achieving unified control over latency stability and resource budget.
It significantly enhances the long-term stability and resource utilization efficiency of satellite internet systems, improves scheduling success rate and system performance assurance capabilities in dynamic environments, and avoids the problems of resource waste and over-configuration in traditional methods.
Smart Images
Figure CN121508623A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of satellite internet communication technology, and in particular to a method for optimizing the deployment and scheduling of satellite internet virtual network functions. Background Technology
[0002] Satellite internet will play a vital role in future communication systems, providing global services, especially for users in remote areas such as oceans and uninhabited regions. Compared to geostationary orbit (GEO) and medium Earth orbit (MEO) satellites, low Earth orbit (LEO) satellites have attracted a large number of users. Traditional satellite networks, designed for specific missions, suffer from isolated infrastructure and network resources, limiting dynamic sharing of physical resources and lacking collaborative capabilities, leading to low resource utilization and high operating costs. To advance fine-grained resource management and dynamic traffic scheduling, software-defined satellite networks (SDNs) are implemented by applying Virtual Network Function (VNF) and SDN technologies. On-board functions are virtualized into VNFs, and services are provided on demand by SFCs deployed on multiple LEO satellite nodes. However, achieving coordinated deployment of SDN and NFV to ensure high-quality service delivery still faces many challenges: On the one hand, due to the limited energy consumption, heat dissipation, and size of satellite nodes, the contradiction between the scarce communication resources of a single satellite node and the surging service demand from users becomes increasingly prominent. A task request must coordinate multiple nodes, that is, multiple VNFs are deployed on different LEO satellite nodes to share network resources; On the other hand, the dynamic nature of satellite movement and the uncertainty of VNF resource requirements mean that static VNF deployment strategies may lead to over-configuration and insufficient resource utilization.
[0003] Some existing works have attempted to apply reinforcement learning to network resource scheduling, enabling agents to learn dynamic policies online to adapt to environmental changes. However, without considering long-term constraints (such as average latency or energy budget), policies that solely optimize for immediate performance may lead to unstable results. For example, a policy focused solely on maximizing immediate throughput might overuse the resources of a particular satellite node, resulting in severe queue backlogs or energy overruns. Furthermore, classical Markov Decision Process (MDP) solutions struggle to be solved in real-time when the state and action spaces are vast, and reinforcement learning training also faces the complexity of high-dimensional decision-making. Therefore, an innovative method is urgently needed that can balance short-term decision optimization with long-term performance guarantees to achieve efficient and robust VNF deployment and scheduling in dynamic satellite network environments.
[0004] Lyapunov optimization theory provides a powerful tool for solving the aforementioned problems. By introducing virtual queues and Lyapunov functions, Lyapunov optimization transforms optimization problems with long-term constraints into time-slot-by-time decision problems. It allows for decision-making in each time slot based on the current queue state without requiring prior knowledge of future statistics, theoretically guaranteeing system queue stability and approximating optimal long-term performance. However, in complex satellite network scenarios, relying solely on analytical Lyapunov control strategies may struggle to directly obtain the global optimum, especially when discrete function deployment choices and resource scheduling decisions involving combinations are involved. Therefore, combining Lyapunov optimization with modern reinforcement learning, utilizing learning algorithms to approximate the optimal decision strategy while simultaneously using Lyapunov methods to provide long-term stability-guided strategy optimization, represents a potentially effective solution. Summary of the Invention
[0005] The purpose of this invention is to provide a satellite internet virtual network function deployment and scheduling optimization method based on Lyapunov optimization constraints, so as to solve the problems of large system latency and unstable resource scheduling caused by limited on-board computing power, dynamic changes in link topology and tight resource constraints during the deployment of service function chains.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] A method for optimizing the deployment and scheduling of satellite internet virtual network functions based on Lyapunov optimization constraints includes the following steps:
[0008] S1: Construct a joint latency-resource constraint model in the space-ground integrated satellite network, which includes node data queues and node virtual cost queues. The data queues constrain latency in the long term, while the virtual cost queues constrain CPU, memory, and bandwidth resources in the long term.
[0009] S2: Based on the data queue and the virtual resource cost queue, the Lyapunov drift-penalty upper bound is derived, which transforms the joint optimization problem of long-term latency and resource cost into a time-slot-wise solvable optimization problem, where the optimization term for each time slot is strictly derived by the drift-penalty upper bound;
[0010] S3: The action decomposition structure method based on deep reinforcement learning is adopted to model the virtual network function deployment action, link mapping action and node resource allocation action as sub-actions respectively. The near-optimal joint deployment and scheduling scheme is selected under Lyapunov constraints through the Actor-Critic strategy.
[0011] Furthermore, the satellite network includes multiple low-Earth orbit satellites and ground edge nodes; each satellite node provides link forwarding while possessing computing and storage capabilities, and can selectively deploy virtual network functions (VNFs) under the constraints of node data queues and virtual cost queues; each satellite is connected to neighboring satellites through two intra-orbit links and two inter-orbit links to support cross-node mapping and resource scheduling of service function chains.
[0012] Furthermore, the resource cost model quantifies the computational resource consumption, memory usage, link bandwidth usage, and node energy consumption during the service function chain deployment process into a unified resource cost, and introduces node resource budgeting when constructing the virtual cost queue. This allows resource costs to not only participate in single-slot cost calculations but also influence long-term resource stability through a virtual cost queue, thereby achieving dynamic constraints and long-term budget control over node resources. The cost is:
[0013] ;
[0014] The method takes into account the operation of a time-slotted system, with each service request... Includes a function chain Its deployment decision on node i is determined by a binary variable. The mapping from the i-th virtual link of the u-th request to the physical link (m, n) is represented by a binary variable. (t) represents.
[0015] The system's propagation delay consists of two parts: propagation delay and VNF computation delay. The propagation delay of service u in time slot t is expressed as:
[0016] ;
[0017] in, The propagation delay of the physical link l in time slot t is used to characterize the dynamic changes in distance and time delay caused by changes in the inter-satellite topology as the orbit moves.
[0018] The computation latency of service u is expressed as:
[0019] ;
[0020] in, Let be the computation and processing latency of the u-th virtual network function on node m, which varies between time slots depending on node resource allocation and service load. Therefore, the end-to-end time slot latency of service u is:
[0021] ;
[0022] In each time slot, the satellite network dynamically determines the deployment location and mapping path of each function based on the current satellite link availability and changes in node computing power in response to the continuously arriving SFC requests, in order to minimize the long-term average end-to-end propagation latency. Its optimization objective is: ;
[0023] in Determine the latency weights for different services. To make deployment decisions on time slot sequences With mapping decision Joint decision-making strategy.
[0024] Furthermore, the Lyapunov drift-penalty optimization framework establishes an actual data queue to represent the backlog of service requests and introduces a virtual cost queue to represent the utilization of resource budget. By minimizing the sum of the drift term and penalty term of the Lyapunov function in each time slot, it achieves stable control of the data queue and ensures that the long-term average resource consumption does not exceed a predetermined budget threshold. Each satellite node m maintains a data queue. This indicates the queue dynamics of the service backlog at this node in time slot t:
[0025] ;
[0026] in This represents the traffic volume newly arriving in time slot t and allocated to node m. Indicates the node's processing rate;
[0027] According to Little's law, if V If it is bounded, the node service latency remains stable.
[0028] Existing reinforcement learning methods constrain resource consumption using weighted loss or heuristic rewards, but these methods struggle to guarantee that resources will not be over-constrained in the long run and that nodes will not remain continuously overloaded. The proposed method constructs a node-level virtual cost queue.
[0029] ;
[0030] The consumption of resources such as CPU, memory, and bandwidth is explicitly included in long-term stability constraints, and Lyapunov optimization is used to ensure that resource usage does not exceed the budget in the time-averaged sense. This queue mechanism can suppress excessive resource consumption, ensure long-term resource stability, and improve the robustness of the algorithm.
[0031] Define the Lyapunov function:
[0032] ;
[0033] Its drift amount is:
[0034] ;
[0035] The method derives the upper bound of the Lyapunov drift penalty by deriving the queue dynamics formula:
[0036] ;
[0037] The penalty coefficient V is used to adjust the relative priority between reducing latency and reducing resource consumption in the system. When V is large, it tends to save the resource consumption portion of the resource reward. The impact of V will be amplified. In low-energy scenarios such as when satellite power supply is tight, the strategy will choose nodes with sufficient resources to place VNF or avoid migration that would lead to increased costs. When V is small, it will be more inclined to reduce latency. The part of the reward related to queues will be amplified. When facing sudden tasks, the strategy will allocate more CPUs to process queue tasks more actively.
[0038] Therefore, the optimization objective can be rewritten as:
[0039] ;
[0040] Furthermore, by constructing an adaptive stochastic decision-making model with Lyapunov stability constraints, the deployment and scheduling of virtual network functions are optimized;
[0041] The model includes:
[0042] (1) State vector
[0043] ;
[0044] Where G(t) represents the link bandwidth, propagation delay, and link availability; This represents the task data queue of each node; This represents the resource cost queue for each node, used for dynamic control of long-term resource constraints. It indicates the available resource status of each node in terms of CPU, memory, and bandwidth.
[0045] (2) The action space adopts an action decomposition structure.
[0046] Deployment actions:
[0047] ;
[0048] Route mapping action:
[0049] ;
[0050] Resource allocation actions:
[0051] ;
[0052] The aforementioned actions are used to control the placement of virtual network functions, service link mapping, and node resource allocation, respectively. This action decomposition framework significantly reduces the combinatorial explosion caused by action coupling, transforming the policy search space from a high-dimensional area to a controllable region.
[0053] (3) The transition probability is Besides being affected by task arrival rate and link availability, it also depends on the gradient direction of the Lyapunov function, when the system drift... At this time, the action space automatically shrinks to the low-cost feasible region to achieve node-level dynamic stability. This adaptive transfer mechanism differs from traditional fixed-probability transfer MDPs, and can adjust the policy distribution in real time during the learning process.
[0054] (4) The reward function is In each time slot t, based on the current queue length Cost queue Service speed and resource costs Construct instant rewards:
[0055] ;
[0056] The original drift-penalty objective is approximated by maximizing the aforementioned reward. This reward function is not empirically set but rigorously derived from theory, enabling simultaneous control over queue pressure, service rate, and resource consumption, with the policy update direction aligned with the system stability objective.
[0057] Furthermore, step S3, which describes solving the Lyapunov long-run optimization problem using an action decomposition structure algorithm based on deep reinforcement learning, specifically includes the following steps:
[0058] S31: Construct a set of actionable actions, which combines conditions such as the remaining resources of the node, the queue length and the link connectivity to eliminate illegal actions caused by insufficient resources or unreachable topology, and ensure that the policy output is schedulable.
[0059] S32: Given that VNF orchestration involves multi-dimensional resource and path joint optimization, within each time slot, the actor network outputs a joint deployment-scheduling action based on its current state and set of available actions; the system then performs resource allocation and path mapping accordingly, completes cost calculation, and generates policy feedback.
[0060] S33: Based on the queue changes and resource consumption after the action is executed, construct an instant reward function with Lyapunov drift and resource cost as its core to measure the execution effect of the current strategy under time delay control and resource constraints.
[0061] S34: The critic network uses immediate rewards to evaluate the long-term value of the current action and updates the actor network parameters through the policy gradient method, so that the policy gradually converges to the optimal direction that balances cost and latency objectives.
[0062] Furthermore, step S3, which describes solving the Lyapunov long-run optimization problem using an action decomposition structure algorithm based on deep reinforcement learning, specifically includes the following steps:
[0063] Step 1) Initialize the data queue, virtual cost queue, training interval, parameters of the actor network and critic network, experience replay memory, drift-penalty coefficient V, learning rate, and discount factor for each node. ;
[0064] Step 2) For each satellite node m and for each time slot t, obtain the currently arriving service function chain requests and data queues. and virtual cost queue The system state s(t) of the current time slot is formed based on the node resource status and link availability.
[0065] Step 3) Construct a set of actionable actions. Based on the size of the node task queue and the virtual cost queue, impose constraints on deployment actions, link mapping actions, and resource allocation actions in the current time slot: If a node is overloaded, prohibit its corresponding virtual network function deployment actions; if a link is unavailable or propagation latency exceeds the limit, prohibit its corresponding mapping actions; if a node's resource budget is tight, limit the scope of its corresponding resource allocation actions; thus forming the set of actionable actions for the current time slot. ;
[0066] Step 4) The actor network determines the state s(t) and action space. Output joint action And submit the action to the environment;
[0067] Step 5) After the environment executes the joint action, the environment updates the system state based on the action execution result and calculates the immediate reward. .
[0068] Step 6) Store in the experience replay memory;
[0069] Step 7) If t mod δT = 0, then sample a batch of historical samples for training;
[0070] Step 8) Update the action value function Q(s,a) of the critic network using the sampled samples through the temporal difference (TD) method;
[0071] Step 9) Update the policy parameters of the actor network based on the value gradient of the critics' feedback, so that the policy tends to minimize the drift-penalty objective;
[0072] Step 10) If the Lyapunov drift or reward function change is below the threshold for several consecutive time slots, the policy training is considered to have converged, and the current joint deployment and resource scheduling policy is output. Otherwise, return to Step 2 to continue iterating.
[0073] Compared with the prior art, the beneficial effects of the present invention are:
[0074] I. This invention introduces Lyapunov optimization theory into the deployment and scheduling of Service Function Chains (SFC) in satellite internet, establishes a dual feedback mechanism of task data queue and resource virtual cost queue, realizes unified modeling and dynamic control of latency stability and resource budget constraints, and significantly enhances the long-term stability of the system.
[0075] Second, this invention proposes a reinforcement learning modeling method based on three-dimensional action decomposition (deployment, routing, resource allocation). While avoiding the dimensional explosion problem of traditional joint action strategies, it can flexibly cope with dynamic topology and link availability changes of satellite nodes, thereby improving the convergence efficiency of the strategy and the adaptability of the system.
[0076] Third, this invention constructs a set of feasible actions based on the current state in each time slot, and tailors the action space according to multiple factors such as node load, link status and resource budget, effectively avoiding the evaluation overhead of infeasible or high-cost solutions, and improving the effectiveness of the learning process and the feasibility of the scheduling strategy.
[0077] Fourth, this invention proposes an instant reward function design based on the Lyapunov drift-penalty criterion. It does not rely on empirical settings but is theoretically derived from system constraint optimization. This ensures that the training direction of the reinforcement learning strategy is strictly aligned with the global optimization goal of "queue stability + resource constraints + latency control", thereby improving the reliability of the deployment strategy.
[0078] Fifth, this invention designs a dynamic convergence determination mechanism in policy training, which performs closed-loop monitoring based on Lyapunov drift and real-time reward changes, realizing stable convergence determination of reinforcement learning in dynamic environments, and improving the system's practicality and online deployability.
[0079] VI. This invention addresses the resource constraints and dynamic topology characteristics of satellite internet. Through a Lyapunov-guided reinforcement learning algorithm, it achieves an adaptive, scalable, and highly robust service function chain deployment optimization scheme, significantly outperforming existing heuristic and static optimization methods. It demonstrates higher scheduling success rates and system performance assurance capabilities in dynamic environments. Attached Figure Description
[0080] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0081] Figure 1 This is a network scene diagram provided in Embodiment 1 of the present invention;
[0082] Figure 2 This is a flowchart of the satellite internet virtual network function deployment and scheduling optimization algorithm based on Lyapunov optimization constraints in Embodiment 1 of the present invention. Detailed Implementation
[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0084] The purpose of this invention is to provide a method for optimizing the deployment and scheduling of satellite internet virtual network functions based on Lyapunov optimization constraints, aiming to improve the long-term service stability of satellite dynamic networks under latency constraints and resource-limited conditions.
[0085] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0086] Example 1
[0087] like Figure 1 and Figure 2 As shown, this invention is a method for optimizing the deployment and scheduling of satellite internet virtual network functions based on Lyapunov optimization constraints, specifically including the following steps:
[0088] like Figure 1 As shown, the satellite-ground integrated internet is modeled as a directed graph. ,in This represents the set of all nodes in the system, including low-Earth orbit satellite nodes and ground edge nodes; This represents the set of inter-satellite links, satellite-to-ground links, and ground links. Each node... The computing resource capacity and storage resource capacity are respectively used as and Indicates that the link The bandwidth capacity is expressed as The propagation delay is .
[0089] The method takes into account the operation of a time-slotted system, with each service request... Includes a function chain Its deployment decision on node i is determined by a binary variable. The mapping from the i-th virtual link of the u-th request to the physical link (m, n) is represented by a binary variable. (t) represents.
[0090] like Figure 2 The present invention provides a method for optimizing the deployment and scheduling of satellite internet virtual network functions based on Lyapunov optimization constraints. The method includes the following steps:
[0091] S1: Construct a joint latency-resource constraint model in the space-ground integrated satellite network, which includes node data queues and node virtual cost queues. The data queues constrain latency in the long term, while the virtual cost queues constrain CPU, memory, and bandwidth resources in the long term.
[0092] S2: Based on the data queue and the virtual resource cost queue, the Lyapunov drift-penalty upper bound is derived, which transforms the joint optimization problem of long-term latency and resource cost into a time-slot-wise solvable optimization problem, where the optimization term for each time slot is strictly derived by the drift-penalty upper bound;
[0093] S3: The action decomposition structure method based on deep reinforcement learning is adopted to model the virtual network function deployment action, link mapping action and node resource allocation action as sub-actions respectively. The near-optimal joint deployment and scheduling scheme is selected under Lyapunov constraints through the Actor-Critic strategy.
[0094] The satellite network comprises multiple low-Earth orbit satellites and ground edge nodes; each satellite node provides link forwarding while possessing computing and storage capabilities, and can selectively deploy virtual network functions (VNFs) under the constraints of node data queues and virtual cost queues; each satellite is connected to neighboring satellites through two intra-orbit links and two inter-orbit links to support cross-node mapping and resource scheduling of service function chains.
[0095] The resource cost model quantifies the computational resource consumption, memory usage, link bandwidth usage, and node energy consumption during the service function chain deployment process into a unified resource cost, and introduces node resource budgeting when constructing the virtual cost queue. This allows resource costs to not only participate in single-slot cost calculations but also influence long-term resource stability through a virtual cost queue, thereby achieving dynamic constraints and long-term budget control over node resources. The cost is:
[0096] ;
[0097] The method takes into account the operation of a time-slotted system, with each service request... Includes a function chain Its deployment decision on node i is determined by a binary variable. The mapping from the i-th virtual link of the u-th request to the physical link (m, n) is represented by a binary variable. (t) represents.
[0098] The system's propagation delay consists of two parts: propagation delay and VNF computation delay. The propagation delay of service u in time slot t is expressed as:
[0099] ;
[0100] in, The propagation delay of the physical link l in time slot t is used to characterize the dynamic changes in distance and time delay caused by changes in the inter-satellite topology as the orbit moves.
[0101] The computation latency of service u is expressed as:
[0102] ;
[0103] in, Let be the computation and processing latency of the u-th virtual network function on node m, which varies between time slots depending on node resource allocation and service load. Therefore, the end-to-end time slot latency of service u is:
[0104] ;
[0105] In each time slot, the satellite network dynamically determines the deployment location and mapping path of each function based on the current satellite link availability and changes in node computing power in response to the continuously arriving SFC requests, in order to minimize the long-term average end-to-end propagation latency. Its optimization objective is: ;
[0106] in Determine the latency weights for different services. To make deployment decisions on time slot sequences With mapping decision Joint decision-making strategy.
[0107] The Lyapunov drift-penalty optimization framework establishes an actual data queue to represent the backlog of business requests and introduces a virtual cost queue to represent the utilization of resource budget. By minimizing the sum of the drift and penalty terms of the Lyapunov function in each time slot, it achieves stable control of the data queue and ensures that the long-term average resource consumption does not exceed a predetermined budget threshold. Each satellite node m maintains one data queue. This indicates the service backlog at this node in time slot t, and its queue dynamics are as follows:
[0108] ;
[0109] in This represents the traffic volume newly arriving in time slot t and allocated to node m. This indicates the processing rate of the node.
[0110] According to Little's law, if V If it is bounded, the node service latency remains stable.
[0111] Existing reinforcement learning methods constrain resource consumption using weighted loss or heuristic rewards, but these methods struggle to guarantee that resources will not be over-constrained in the long run and that nodes will not remain continuously overloaded. The proposed method constructs a node-level virtual cost queue.
[0112] ;
[0113] The consumption of resources such as CPU, memory, and bandwidth is explicitly included in long-term stability constraints, and Lyapunov optimization is used to ensure that resource usage does not exceed the budget in the time-averaged sense. This queue mechanism can suppress excessive resource consumption, ensure long-term resource stability, and improve the robustness of the algorithm.
[0114] Define the Lyapunov function:
[0115] ;
[0116] Its drift amount is:
[0117] ;
[0118] The method derives the upper bound of the Lyapunov drift penalty by deriving the queue dynamics formula:
[0119] ;
[0120] The penalty coefficient V is used to adjust the relative priority between reducing latency and reducing resource consumption in the system. When V is large, it tends to save the resource consumption portion of the resource reward. The impact of V will be amplified. In low-energy scenarios such as when satellite power supply is tight, the strategy will choose nodes with sufficient resources to place VNF or avoid migration that would lead to increased costs. When V is small, it will be more inclined to reduce latency. The part of the reward related to queues will be amplified. When facing sudden tasks, the strategy will allocate more CPUs to process queue tasks more actively.
[0121] Therefore, the optimization objective can be rewritten as:
[0122] ;
[0123] The method optimizes the deployment and scheduling of virtual network functions by constructing an adaptive stochastic decision model with Lyapunov stability constraints.
[0124] The model includes:
[0125] (1) State vector ;
[0126] Where G(t) represents the link bandwidth, propagation delay, and link availability; This represents the task data queue of each node; This represents the resource cost queue for each node, used for dynamic control of long-term resource constraints. It indicates the available resource status of each node in terms of CPU, memory, and bandwidth.
[0127] (2) The action space adopts an action decomposition structure.
[0128] Deployment actions: ;
[0129] Route mapping action: ;
[0130] Resource allocation actions: ;
[0131] The aforementioned actions are used to control the placement of virtual network functions, service link mapping, and node resource allocation, respectively. This action decomposition framework significantly reduces the combinatorial explosion caused by action coupling, transforming the policy search space from a high-dimensional area to a controllable region.
[0132] (3) The transition probability is Besides being affected by task arrival rate and link availability, it also depends on the gradient direction of the Lyapunov function, when the system drift... At this time, the action space automatically shrinks to the low-cost feasible region to achieve node-level dynamic stability. This adaptive transfer mechanism differs from traditional fixed-probability transfer MDPs, and can adjust the policy distribution in real time during the learning process.
[0133] (4) The reward function is In each time slot t, based on the current queue length Cost queue Service speed and resource costs Construct instant rewards:
[0134] ;
[0135] The original drift-penalty objective is approximated by maximizing the aforementioned reward. This reward function is not empirically set but rigorously derived from theory, enabling simultaneous control over queue pressure, service rate, and resource consumption, with the policy update direction aligned with the system stability objective.
[0136] Step S3, which describes solving the Lyapunov long-run optimization problem using an action decomposition structure algorithm based on deep reinforcement learning, specifically includes the following steps:
[0137] S31: Construct a set of actionable actions, which combines conditions such as the remaining resources of the node, the queue length and the link connectivity to eliminate illegal actions caused by insufficient resources or unreachable topology, and ensure that the policy output is schedulable.
[0138] S32: Given that VNF orchestration involves multi-dimensional resource and path joint optimization, within each time slot, the actor network outputs a joint deployment-scheduling action based on its current state and set of available actions; the system then performs resource allocation and path mapping accordingly, completes cost calculation, and generates policy feedback.
[0139] S33: Based on the queue changes and resource consumption after the action is executed, construct an instant reward function with Lyapunov drift and resource cost as its core to measure the execution effect of the current strategy under time delay control and resource constraints.
[0140] S34: The critic network uses immediate rewards to evaluate the long-term value of the current action and updates the actor network parameters through the policy gradient method, so that the policy gradually converges to the optimal direction that balances cost and latency objectives.
[0141] Furthermore, step S3, which describes solving the Lyapunov long-run optimization problem using an action decomposition structure algorithm based on deep reinforcement learning, specifically includes the following steps:
[0142] Step 1) Initialize the data queue, virtual cost queue, training interval, parameters of the actor network and critic network, experience replay memory, drift-penalty coefficient V, learning rate, and discount factor for each node. ;
[0143] Step 2) For each satellite node m and for each time slot t, obtain the currently arriving service function chain requests and data queues. and virtual cost queue The system state s(t) of the current time slot is formed based on the node resource status and link availability.
[0144] Step 3) Construct a set of actionable actions. Based on the size of the node task queue and the virtual cost queue, impose constraints on deployment actions, link mapping actions, and resource allocation actions in the current time slot: If a node is overloaded, prohibit its corresponding virtual network function deployment actions; if a link is unavailable or propagation latency exceeds the limit, prohibit its corresponding mapping actions; if a node's resource budget is tight, limit the scope of its corresponding resource allocation actions; thus forming the set of actionable actions for the current time slot. ;
[0145] Step 4) The actor network determines the state s(t) and action space. Output joint action And submit the action to the environment;
[0146] Step 5) After the environment executes the joint action, the environment updates the system state based on the action execution result and calculates the immediate reward. .
[0147] Step 6) Store in the experience replay memory;
[0148] Step 7) If t mod δT = 0, then sample a batch of historical samples for training;
[0149] Step 8) Update the action value function Q(s,a) of the critic network using the sampled samples through the temporal difference (TD) method;
[0150] Step 9) Update the policy parameters of the actor network based on the value gradient of the critics' feedback, so that the policy tends to minimize the drift-penalty objective;
[0151] Step 10) If the Lyapunov drift or reward function change is below the threshold for several consecutive time slots, the policy training is considered to have converged, and the current joint deployment and resource scheduling policy is output. Otherwise, return to Step 2 to continue iterating.
[0152] Based on the above, Figure 1 and Figure 2Accordingly, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the satellite internet virtual network function deployment and scheduling optimization method of any of the above embodiments.
[0153] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.
[0154] Based on the above, Figure 1 and Figure 2 To achieve the above objectives, this application also provides a computer device, including a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the steps of the satellite internet virtual network function deployment and scheduling optimization method of any of the above embodiments.
[0155] Optionally, the computer device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB ports, card reader ports, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.
[0156] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components, or combine certain components, or have different component arrangements.
[0157] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and stores the hardware and software resources of a computer device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software within the physical device.
[0158] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for optimizing the deployment and scheduling of satellite internet virtual network functions based on Lyapunov optimization constraints, characterized in that, The method includes: S1: Construct a joint latency-resource constraint model in the space-ground integrated satellite network, which includes node data queues and node virtual cost queues. The data queues constrain latency in the long term, while the virtual cost queues constrain CPU, memory, and bandwidth resources in the long term. S2: Based on the data queue and the virtual resource cost queue, the Lyapunov drift-penalty upper bound is derived, which transforms the joint optimization problem of long-term latency and resource cost into a time-slot-wise solvable optimization problem, where the optimization term for each time slot is strictly derived by the drift-penalty upper bound; S3: The action decomposition structure method based on deep reinforcement learning is adopted to model the virtual network function deployment action, link mapping action and node resource allocation action as sub-actions respectively. The near-optimal joint deployment and scheduling scheme is selected under Lyapunov constraints through the Actor-Critic strategy.
2. The satellite internet virtual network function deployment and scheduling optimization method according to claim 1, characterized in that, The satellite-ground integrated network comprises multiple low-Earth orbit satellites and ground edge nodes; each satellite node provides link forwarding while possessing computing and storage capabilities, and can selectively deploy virtual network functions (VNFs) under the constraints of node data queues and virtual cost queues; each satellite is connected to neighboring satellites through two intra-orbit links and two inter-orbit links to support cross-node mapping and resource scheduling of service function chains.
3. The satellite internet virtual network function deployment and scheduling optimization method according to claim 1, characterized in that, The resource cost model quantifies the computational resource consumption, memory usage, link bandwidth usage, and node energy consumption during the service function chain deployment process into a unified resource cost, and introduces node resource budgeting when constructing the virtual cost queue. This allows resource costs to not only participate in single-slot cost calculations but also influence long-term resource stability through a virtual cost queue, thereby achieving dynamic constraints and long-term budget control over node resources. The cost is: 。 4. The satellite internet virtual network function deployment and scheduling optimization method according to claim 1, characterized in that, The method takes into account the operation of a time-slotted system, with each service request... Includes a function chain Its deployment decision on node i is determined by a binary variable. This indicates that the mapping of the i-th virtual link to the physical link (m, n) for the u-th request is represented by a binary variable. (t) indicates that the propagation delay of the system includes two parts: propagation delay and VNF computation delay; The propagation delay of service u in time slot t is expressed as: ； The propagation delay of the physical link l in time slot t is used to characterize the dynamic changes in distance and time delay caused by changes in the inter-satellite topology as the orbit moves. The computation latency of service u is expressed as: ； Let be the computation and processing latency of the u-th virtual network function on node m, which varies between time slots depending on node resource allocation and service load; therefore, the end-to-end time slot latency of service u is: ； In each time slot, the satellite network dynamically determines the deployment location and mapping path of each function based on the current satellite link availability and changes in node computing power in response to the continuously arriving SFC requests, in order to minimize the long-term average end-to-end propagation latency. Its optimization objective is: ; in Determine the latency weights for different services. To make deployment decisions on time slot sequences With mapping decision Joint decision-making strategy.
5. The satellite internet virtual network function deployment and scheduling optimization method according to claim 1, characterized in that, The Lyapunov drift-penalty optimization framework establishes an actual data queue to represent the backlog of business requests and introduces a virtual cost queue to represent the utilization of resource budget. By minimizing the sum of the drift and penalty terms of the Lyapunov function in each time slot, it achieves stable control of the data queue and ensures that the long-term average resource consumption does not exceed a predetermined budget threshold. Each satellite node m maintains one data queue. This indicates the queue dynamics of the service backlog at this node in time slot t: ； in This represents the traffic volume newly arriving in time slot t and allocated to node m. Indicates the node's processing rate; According to Little's law, if V If the nodes are bounded, then the node service latency remains stable; Existing reinforcement learning methods constrain resource consumption using weighted loss or heuristic rewards, but these methods struggle to guarantee that resources will not be over-constrained in the long run and that nodes will not remain continuously overloaded. The proposed method constructs a node-level virtual cost queue. ； The consumption of resources such as CPU, memory, and bandwidth is explicitly included in long-term stability constraints, and Lyapunov optimization is used to ensure that resource usage does not exceed the budget on a time average basis. This queue mechanism can suppress excessive resource consumption, ensure long-term resource stability, and improve the robustness of the algorithm. Define the Lyapunov function: ； Its drift amount is: ； The method derives the upper bound of the Lyapunov drift penalty by deriving the queue dynamics formula: ； The penalty coefficient V is used to adjust the relative priority between reducing latency and reducing resource consumption in the system. When V is large, it tends to save the resource consumption portion of the resource reward. The impact of V will be amplified. In low-energy scenarios such as when satellite power supply is tight, the strategy will choose nodes with sufficient resources to place VNF or avoid migration that would lead to increased costs. When V is small, it will be more inclined to reduce latency. The part of the reward related to queues will be amplified. When facing sudden tasks, the strategy will allocate more CPUs to process queue tasks more actively. Therefore, the optimization objective can be rewritten as: 。 6. The satellite internet virtual network function deployment and scheduling optimization method according to claim 1, characterized in that, The deployment and scheduling of virtual network functions are optimized by constructing an adaptive stochastic decision model with Lyapunov stability constraints. The model includes: (1) State vector ; Where G(t) represents the link bandwidth, propagation delay, and link availability; This represents the task data queue of each node; This represents the resource cost queue for each node, used for dynamic control of long-term resource constraints. This indicates the available resource status of each node in terms of CPU, memory, and bandwidth. (2) The action space adopts an action decomposition structure. Deployment actions: ； Route mapping action: ； Resource allocation actions: ； The above actions are used to control the placement of virtual network functions, service link mapping, and node resource allocation respectively. This action decomposition framework significantly reduces the combinatorial explosion caused by action coupling, and transforms the policy search space from a high-dimensional area to a controllable area. (3) The transition probability is Besides being affected by task arrival rate and link availability, it also depends on the gradient direction of the Lyapunov function, when the system drift... When the action space automatically shrinks to the low-cost feasible region, it achieves node-level dynamic stability. This adaptive transfer mechanism differs from the traditional fixed-probability transfer MDP and can adjust the policy distribution in real time during the learning process. (4) The reward function is In each time slot t, based on the current queue length Cost queue Service speed and resource costs Construct instant rewards: ； The original drift-penalty objective is approximated by maximizing the aforementioned reward. This reward function is not empirically set but rigorously derived from theory, enabling simultaneous control of "queue pressure, service rate, and resource consumption," with the policy update direction aligned with the system stability objective.
7. The satellite internet virtual network function deployment and scheduling optimization method according to claim 1, characterized in that, Step S3, which describes solving the Lyapunov long-run optimization problem using an action decomposition structure algorithm based on deep reinforcement learning, specifically includes the following steps: S31: Construct a set of actionable actions, which combines conditions such as the remaining resources of the node, the queue length and the link connectivity to eliminate illegal actions caused by insufficient resources or unreachable topology, and ensure that the policy output is schedulable. S32: Given that VNF orchestration involves multi-dimensional resource and path joint optimization, in each time slot, the actor network outputs a joint deployment-scheduling action based on the current state and the set of actionable actions; the system then performs resource allocation and path mapping, completes cost calculation, and forms policy feedback accordingly. S33: Based on the queue changes and resource consumption after the action is executed, construct an instant reward function with Lyapunov drift and resource cost as the core to measure the execution effect of the current strategy under time delay control and resource constraints; S34: The critic network uses immediate rewards to evaluate the long-term value of the current action and updates the actor network parameters through the policy gradient method, so that the policy gradually converges to the optimal direction that balances cost and latency objectives.
8. The satellite internet virtual network function deployment and scheduling optimization method according to claim 1, characterized in that, Step S3, which describes solving the Lyapunov long-run optimization problem using an action decomposition structure algorithm based on deep reinforcement learning, specifically includes the following steps: Step 1) Initialize the data queue, virtual cost queue, training interval, parameters of the actor network and critic network, experience replay memory, drift-penalty coefficient V, learning rate, and discount factor for each node. ; Step 2) For each satellite node m and for each time slot t, obtain the currently arriving service function chain requests and data queues. and virtual cost queue The system state s(t) of the current time slot is formed based on the node resource status and link availability. Step 3) Construct a set of actionable actions. Based on the size of the node task queue and the virtual cost queue, impose constraints on deployment actions, link mapping actions, and resource allocation actions in the current time slot: If a node is overloaded, prohibit its corresponding virtual network function deployment actions; if a link is unavailable or propagation latency exceeds the limit, prohibit its corresponding mapping actions; if a node's resource budget is tight, limit the scope of its corresponding resource allocation actions; thus forming the set of actionable actions for the current time slot. ; Step 4) The actor network determines the state s(t) and action space. Output joint action And submit the action to the environment; Step 5) After the environment executes the joint action, the environment updates the system state based on the action execution result and calculates the immediate reward. ; Step 6) Store in the experience replay memory; Step 7) If t mod δT = 0, then sample a batch of historical samples for training; Step 8) Update the action value function Q(s,a) of the critic network using the sampled samples through the temporal difference (TD) method; Step 9) Update the policy parameters of the actor network based on the value gradient of the critics' feedback, so that the policy tends to minimize the drift-penalty objective; Step 10) If the Lyapunov drift or reward function change is lower than the threshold in several consecutive time slots, it is determined that the policy training has converged, and the current joint deployment and resource scheduling policy is output; otherwise, return to step 2 to continue iterating.