Time delay optimization calculation unloading strategy for satellite-ground fusion network

Abstract

The invention relates to a time delay optimization calculation unloading strategy for a satellite-to-ground fusion network, and the strategy comprises the following steps: 1, constructing a satellite-to-ground fusion network system model which comprises a network model, a satellite-to-ground link communication model, a queue model, a local equipment calculation model, an LEO satellite calculation model, and a ground cloud calculation center calculation model; problem description is carried out; step 2, based on a Lyapunov stability theory, decoupling a long-term mixed integer nonlinear programming optimization problem into sub-optimization problems of each time slot; 3, converting each time slot calculation unloading problem into a Markov decision process, and solving by using a deep reinforcement learning algorithm; designing a double-strategy network model structure of the Actor module to generate a strategy; and finally, exploring a better unloading strategy by using a ternary order-preserving quantizer, and accelerating model convergence. According to the method, the dynamic regulation and control of the terminal and the low-orbit satellite task queue can be realized, and the long-term stability of the unloading process and the continuous availability of computing resources are improved.

Description

Technical Field

[0001] The present invention relates to the field of satellite network communication technology, and in particular to a delay optimization calculation unloading strategy for a satellite-ground fusion network. Background Art

[0002] Edge devices (such as monitoring terminals or sensor nodes) play a vital role in environmental monitoring tasks such as hydrology, water quality, and meteorology in remote areas. However, these devices often generate computationally intensive data processing tasks during operation, which poses a significant challenge to the computational latency of task execution. Mobile edge computing technology can effectively enhance the computing power of edge devices, enabling them to offload local computing tasks to edge servers, which then complete the task calculations and return the results, thus meeting the task's latency requirements. However, due to the remote location, limited network coverage, and weak infrastructure of remote areas, traditional mobile edge computing architectures for land-based areas are difficult to directly apply. In this context, edge computing networks deployed in remote areas not only face challenges such as fluctuating wireless communication channel quality and dynamic changes in edge computing resources, but also need to consider the energy constraints of edge nodes with limited power supply capabilities and the risk of service interruptions caused by environmental changes. These factors can significantly affect the efficiency of computing offloads and response latency.

[0003] In recent years, neural networks have achieved breakthroughs in fields such as computer vision, speech recognition, and natural language processing, demonstrating powerful feature extraction and modeling capabilities. Convex optimization methods, due to their excellent mathematical interpretability and global optimality, also play a vital role in resource allocation and computational task scheduling. Combining neural networks with convex optimization methods leverages the strengths of both: neural networks can learn complex system dynamics, while convex optimization can efficiently solve computational offloading problems based on these learned features, ensuring the feasibility and optimality of computational resource allocation.

[0004] Facing the task computing requirements of ground terminal devices, a stochastic computing offloading decision algorithm combining deep reinforcement learning and traditional optimization methods is designed. The steps of this algorithm are as follows: First, environmental information is collected, including the wireless channel conditions and task queue lengths of each user. After obtaining this data, it is input into a deep neural network (DNN). The DNN generates computing offloading decisions based on this real-time provided environmental information, indicating how each user processes its tasks, that is, deciding whether to process locally or through LEO satellites. To further optimize these offloading decisions, an action quantization method generates multiple candidate offloading decision sets. For each candidate offloading decision, a traditional convex optimization algorithm is then applied to solve the related resource allocation problem. The solution of this part takes into account the current wireless channel conditions and the task requirements of users, aiming to find the best balance among various performance metrics such as power consumption and latency. After evaluating all candidate offloading decisions and their corresponding resource allocation decisions, the best-performing offloading decision and its corresponding resource allocation strategy are selected according to the corresponding optimization objectives. The selected best decision is used to update the task queue to ensure that tasks are completed within the scheduled time and to prepare for task processing in the next time period.

[0005] The joint optimization framework of this neural network and convex optimization also has good online adaptability. It can update the neural network parameters in real time according to the changing environmental state and adjust the offloading strategy without relying on future information prediction, and is applicable to complex and changing network environments. Compared with traditional static or rule-based methods, this method has advantages in dealing with resource competition and environmental dynamics, can improve the stability of task offloading, and thus meet the computing requirements.

[0006] In the existing technologies, such as the Chinese patent with the publication number CN114866133B, it discloses a computing offloading method for satellite cloud-edge collaborative computing. It constructs a satellite cloud-edge collaborative system with a three-layer network architecture of cloud-edge-terminal, takes the current channel state as the input, outputs the offloading decision corresponding to the minimized computing cost, uses order-preserving quantization to ensure the consistency of the order of offloading decision variables before and after quantization and the algorithm performance and complexity, and eliminates unnecessary offloading decisions by adaptively adjusting the quantization value K<subgt;t< / subgt;, ensuring the performance of this algorithm. The technical solution of the present invention can significantly reduce the algorithm complexity and the execution time of the algorithm while ensuring that the system generates a computing cost close to the minimum.

[0007] The aforementioned existing technical solutions suffer from the following drawbacks: Although the offloading method combining neural networks with convex optimization demonstrates strong adaptability and decision-making quality in dynamic environments, it still suffers from deficiencies in practical applications. First, traditional convex optimization methods, when combined with reinforcement learning or neural network frameworks, are highly dependent on model parameters and lack flexibility. To ensure the convexity of the problem being solved, the system model often needs to be simplified or strong assumptions made, which to some extent limits the model's expressive power.

[0008] In addition, the solution process of convex optimization usually involves iterative calculations. The calculation speed in complex scenarios is often slower than the forward propagation of pure neural networks, making it difficult to meet the requirements of computational offloading scenarios with high computational timeliness. Secondly, the action quantization methods currently used are mostly based on a "two-choice" offloading method (such as choosing between local and LEO satellites), which to a certain extent limits the flexibility of decision-making. When faced with a more complex multi-access edge computing environment (such as the coexistence of local nodes, low-Earth satellites, and ground edge servers), the existing action quantization mechanism is difficult to expand and cannot effectively model and distinguish three or more offloading options, thereby affecting the comprehensiveness and precision of the offloading strategy.

[0009] Finally, this approach fails to fully account for the environmental dynamics brought about by orbital changes in low-Earth orbit satellites. Due to the periodic motion of LEO satellites, their channel availability conditions can rapidly change over time and space. Failure to incorporate satellite orbital dynamics into offloading strategies can compromise the reliability of these strategies, presenting a key challenge that current approaches must address in remote computing scenarios. Summary of the Invention

[0010] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide a delay-optimized computational offloading strategy for satellite-ground converged networks. The technical problem to be solved is how to offload computationally intensive tasks in the dynamic environment of satellite-ground converged networks to reduce task processing latency. Specifically, a delay-optimized offloading strategy for computationally intensive tasks in satellite-ground converged networks is proposed.

[0011] The above-mentioned object of the present invention is achieved through the following technical solutions:

[0012] A delay optimization calculation offloading strategy for satellite-ground converged networks includes the following steps:

[0013] Step 1: Construct a satellite-ground fusion network system model, which includes a network model, a satellite-ground link communication model, a queue model, a local device computing model, a LEO satellite computing model, and a ground cloud computing center computing model, and describe the problem.

[0014] Step 2: Based on Lyapunov stability theory, the long-term mixed integer nonlinear programming optimization problem is decoupled into sub-optimization problems for each time slot. The long-term average delay optimization under the task queue stability constraint is achieved by minimizing the Lyapunov drift penalty function.

[0015] Step 3: Model the computation offloading problem for each time slot as a Markov decision process and solve it using a deep reinforcement learning algorithm. The deep reinforcement learning algorithm with an actor module dual-strategy network structure generates offloading decisions and resource allocation strategies. The first strategy network outputs a ternary offloading decision and computational resource allocation, while the second strategy network outputs communication resource allocation.

[0016] Step 4: Map the continuous action space into discrete ternary offloading decisions through the ternary rank-preserving quantization method, and optimize the strategy selection in combination with the dynamic noise exploration mechanism. Execute the optimal offloading decision based on the current environmental state, update the task queue and system state, and iterate the optimization until convergence.

[0017] As a further technical solution of the present invention: in step 1, the satellite-to-ground link communication model in the satellite-to-ground fusion network system model calculates the satellite-to-ground link transmission rate based on the Shannon formula:

[0018]

[0019] in and Is user gu r In time slot t i The transmission power and channel gain when It is a satellite n k , the channel gain, It is a satellite n k Assigned to ground terminal equipment r bandwidth resources, It is from gu r To Satellite n k Free space loss, L t is the other link loss, K is the Boltzmann constant, and T is the noise temperature, where It is expressed as follows:

[0020]

[0021] in It is a ground terminal device r To Satellite n k The link length between them, c and F are the speed of light and the communication center frequency.

[0022] As a further technical solution of the present invention: in step 1, the dynamic update of the task queue in the system model satisfies:

[0023] Ground terminal equipment queue:

[0024]

[0025] Low Earth Orbit Satellite Fleet:

[0026]

[0027] where at r (i) at k (i) are the task arrival amount, ct r (i) ct k (i) is the task processing capacity.

[0028] As a further technical solution of the present invention: in step 2, the Lyapunov drift penalty function is defined as:

[0029]

[0030] Where ΔL(Z(i)) is the Lyapunov drift of the queue length, V is the penalty parameter, and C(i) is the total execution delay within the time slot;

[0031] By minimizing the function, the long-term optimization goal is transformed into an immediate resource allocation problem for each time slot.

[0032] As a further technical solution of the present invention: in step 3, the dual-strategy network structure includes:

[0033] The first policy network inputs the environment state information Including the channel gain, remaining energy, local and satellite task queue lengths of ground terminal equipment, outputting continuous ternary offloading decision signals and calculating resource allocation ratios;

[0034] The second policy network inputs the environment state information Including the intermediate results of offloading decisions, task attributes and maximum bandwidth constraints, and outputting the communication resource allocation ratio;

[0035] The first strategy network and the second strategy network store samples through independent experience pools and update network parameters based on the Adam algorithm.

[0036] As a further technical solution of the present invention: in the step 3, a Critic module is also included, which is used to evaluate the candidate unloading decisions generated by the Actor module and select the optimal strategy based on the reward function; the reward function is consistent with the objective function of the optimization target P2, that is, minimizing the weighted delay and queue backlog.

[0037] As a further technical solution of the present invention: in step 4, the ternary order-preserving quantization method includes the following steps:

[0038] Multiply the continuous action value by the number of unloaded objects and divide it into the intervals of [0,0.5], [0.5,1.5], and [1.5,2.5], which correspond to local processing, satellite computing, and ground cloud computing center computing, respectively;

[0039] The quantization boundary is dynamically adjusted according to the decision deviation distance, the nearest offloading target is prioritized, and the policy diversity is expanded by adding Gaussian noise.

[0040] As a further technical solution of the present invention: in step 4, the update rule of the dynamic noise exploration mechanism is:

[0041] ∈ i =max(∈ i ·ep,∈ min ) (43);

[0042] Where ep∈[0,1] is a hyperparameter, ∈ min is the minimum exploration noise value, which is used to balance strategy exploration and exploitation.

[0043] As a further technical solution of the present invention: the strategy also includes:

[0044] In the communication resource allocation, the Softmax function is used to normalize the bandwidth allocation ratio to ensure the total bandwidth constraint of the satellite;

[0045] In local computing mode, the bandwidth allocation ratio is forced to zero to avoid resource waste.

[0046] In summary, the present invention includes at least one of the following beneficial technical effects:

[0047] 1. The present invention discloses a latency-optimized computational offloading strategy for satellite-ground converged networks, which includes a reinforcement learning algorithm with a dual-strategy network structure to further enhance the flexibility and real-time performance of computational offloading decisions. On the one hand, a policy network is introduced to replace the traditional convex optimization module, significantly improving computational efficiency while maintaining the effectiveness of resource allocation. This overcomes the shortcomings of convex optimization methods, such as complex parameter settings and slow solution speed. On the other hand, a new action quantization method is designed, breaking through the limitation of traditional methods that only support "two-party offloading selection" and expanding support for three-party offloading decisions among local nodes, near-Earth satellites, and ground edge servers, thereby improving the adaptability and generalization of the offloading strategy in multi-access environments. In addition, considering the highly dynamic nature of satellite orbits, this paper incorporates communication conditions and energy acquisition conditions that change with orbital changes into the modeling process. By dynamically sensing the connection availability and energy constraints between satellites and ground equipment, the offloading strategy is further optimized, thereby effectively improving the stability and sustainable operation capabilities of the system while ensuring service continuity and accessibility.

[0048] 2. Compared with the prior art, the present invention has three main contributions:

[0049] (1) Traditional convex optimization often requires multiple iterations to converge in highly dynamic environments, and it is difficult to meet offloading requirements in scenarios with high latency requirements. However, the dual-strategy network in the present invention generates computational offloading decisions and computational and communication resource allocation strategies in two stages, and the decision delay is lower than the iterative solution process.

[0050] (2) The existing two-choice quantization method is limited to local terminal device-low-orbit satellite or local terminal device-edge server. The ternary rank-preserving quantization designed by the present invention supports the three-way offloading of local terminal device-low-orbit satellite-ground cloud computing center in the satellite-ground fusion network, significantly enhancing the generalization ability of the strategy in complex environments.

[0051] (3) In response to the uncertainties of energy acquisition fluctuations, link status fluctuations, and random arrival of ground tasks caused by satellite orbit changes in the dynamic satellite-ground fusion network, the present invention introduces a Lyapunov optimization framework to decompose the system's long-term task processing delay minimization goal into a deterministic offloading strategy and resource allocation sub-problem for each time slot, dynamically manage the task queue backlog, and thus significantly improve the stability of offloading decisions and the sustainable utilization of computing resources under the condition of dynamic orbit changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a diagram of the computing architecture based on the spatial computing power network in the present invention.

[0053] Figure 2 This is the structural diagram of the dual-strategy network model in the present invention.

[0054] Figure 3 This is an example diagram of the quantification of offloading decisions in the present invention. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application; it is obvious that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0056] With the advancement of sensing and artificial intelligence technologies, the application of the Internet of Things in remote areas (such as smart agriculture and environmental monitoring) requires high computing efficiency, but ground-based terminal devices have limited computing capabilities. Transmitting tasks to cloud computing centers via terrestrial networks results in high latency. Inspired by Mobile Edge Computing (MEC), deploying MEC servers can bring computing resources down to the cloud, improving response speed. However, in remote areas with complex terrain (such as deserts, oceans, and mountains), the deployment of base stations and MEC servers is limited, making it difficult to meet computing needs. As a supplement to terrestrial networks, LEO satellite networks are unaffected by terrain and use inter-satellite link (ISL) technology to build a global mesh network for seamless coverage.

[0057] By combining LEO satellite networks with ground networks and deploying MEC servers, tasks can be transferred to satellite computing or relayed to cloud centers, significantly reducing latency. Due to the dynamic nature of satellite orbits, the length of the satellite-to-ground link changes with changes in the satellite orbit, resulting in variable satellite-to-ground link transmission latency during communication. Furthermore, the satellite charging rate fluctuates with the movement of the satellite's orbital position, affecting task offloading decisions. Finally, when using reinforcement learning models to collaboratively optimize computing and communication resources, decisions are often made through a single policy network due to the high complexity of the dynamic environment, resulting in limited model convergence speed. Therefore, to address the above issues, how to consider the changes in satellite-to-ground communication conditions and the impact of satellite energy fluctuations caused by changes in LEO satellite orbits, and design a computational offloading strategy that optimizes the long-term task processing latency of the satellite-to-ground integrated network while meeting energy consumption constraints has become a hot research topic.

[0058] The technical problem to be solved by the present invention is how to offload computationally intensive tasks in a dynamic environment of a satellite-ground fusion network to reduce task processing delay. Specifically, a delay-optimized offloading strategy for computationally intensive tasks in a satellite-ground fusion network is proposed.

[0059] This paper proposes a delay-optimized computational offloading strategy that integrates Lyapunov optimization and reinforcement learning. To address uncertainties such as satellite orbit changes, link status fluctuations, and dynamic task arrival, this strategy models task offloading in a satellite-ground fusion network as a long-term computational offloading problem. Lyapunov stability theory is used to guide reinforcement learning to learn offloading and resource allocation strategies within each time slot, transforming the long-term delay control objective into short-term solvable subproblems. This allows for dynamic regulation of terminal and low-orbit satellite task queues, improving the long-term stability of the offloading process and the continuous availability of computing resources.

[0060] The object to be protected by this invention is a satellite-ground fusion network computing offloading strategy for latency optimization, with the following protection points:

[0061] Dual-strategy network structure: The present invention proposes a dual-strategy network architecture, which inputs system status information and outputs offloading decisions and communication resource allocation strategies respectively, to achieve decoupling optimization of computing tasks and resource scheduling.

[0062] Ternary offloading quantization method: A ternary rank-preserving quantization method is designed to efficiently convert continuous actions into ternary offloading decisions. It combines error distance and exploration mechanism to improve the accuracy and diversity of strategy selection.

[0063] Example 1:

[0064] The present invention discloses a delay optimization calculation offloading strategy for a satellite-ground fusion network, comprising the following steps:

[0065] Step 1: Construct a satellite-ground fusion network system model, which includes a network model, a satellite-ground link communication model, a queue model, a local device computing model, a LEO satellite computing model, and a ground cloud computing center computing model, and describe the problem.

[0066] Specifically, the system model is further explained:

[0067] Network model:

[0068] like Figure 1 As shown in the figure, the space computing network architecture consists of R ground terminal devices GU, a low-orbit satellite constellation equipped with MEC servers, and a ground cloud center. Over time, the satellite that has been providing services to the R GUs and remains connected will leave this area. Then, another LEO satellite that can cover the area will provide computing services for the area again. This cyclical process ensures the continuous formation of a powerful satellite edge computing network.

[0069] In this invention, MEC servers are deployed on LEO satellites to provide low-latency computing services to GUs in remote scenarios beyond the reach of terrestrial networks. Three different modes are available for processing GU computing tasks. In Mode I, tasks are processed locally on ground devices. In Mode II, tasks are offloaded to LEO satellites for execution. In Mode III, computing tasks are transmitted via the LEO satellite network to a terrestrial cloud computing center for execution.

[0070] For ease of expression, define GU={gu1,…,gu r ,…gu R} represents the ground terminal equipment set, GS={gs l} represents the set of ground cloud computing centers, N={n1,…,n k ,…n M} is a set of low-orbit satellites. S={s1…s n …s V} is a set of tasks, T={t1…t i …t L} represents a time set. In addition, the link (gu r ,n k ) is from the ground terminal equipment gu r To Satellite n k Satellite-to-ground link, (n j ,n k ) is from satellite n j to n k Since there are three different offloading modes, an offloading strategy is constructed for each task. n The uninstallation strategy is described as follows:

[0071]

[0072]

[0073]

[0074] in Represents task s n will be executed locally, Indicates that the task will not be executed locally. and Similar to the above, the task is decided whether to be calculated on the LEO satellite or in the ground cloud computing center.

[0075] Satellite-to-ground link communication model:

[0076] In time slot t i Shannon's formula is used to calculate the satellite-ground link (SGL) from gpr to n k The achievable transfer rates are as follows:

[0077]

[0078] in and Is user gu r In time slot t i The transmission power and channel gain when It is a satellite n k , the channel gain, It is a satellite n k Assigned to ground terminal equipment r bandwidth resources, It is from gu r To Satellite n k Free space loss, L t is the other link loss, K is the Boltzmann constant, and T is the noise temperature, where It is expressed as follows:

[0079]

[0080] in It is a ground terminal device r To Satellite n k The link length between them, c and F are the speed of light and the communication center frequency.

[0081] Queue Model:

[0082] GU and LEO satellites have a task queue to store the incoming tasks. i At the beginning, r With at r (i)(bits)The number of computing tasks that have arrived. r (i) is expressed as

[0083] at r (i) = sd n (6);

[0084] where sd n It's a task n Data size, sd n Follow independent and identically distributed exponential distributions, and E[at r (i)]=λ r , where λ r is the task arrival rate. i ,gu r There is a computation task ct(i) that needs to be processed. For task s n For example, if This means that the task will be calculated locally on the ground terminal device. This means that the task will be transmitted to the satellite or transmitted to the ground cloud computing center through the low-orbit satellite network for calculation. r There is ct r (i) The bit task needs to be processed, ct r (i) is represented as follows:

[0085]

[0086] The computing tasks that have arrived but not yet unloaded will be stored in the task queue. The queue length varies in different time slots, and the user r In time slot t i The calculated length is Dynamic representation is:

[0087]

[0088] in

[0089] On LEO satellites, there is also a task queue to store computing tasks that have been unloaded but not yet executed. k (i) represents the number of satellites n k The size of the mission data from the ground terminal equipment is expressed as:

[0090]

[0091] in represents the computation offloading decision, if This means that the computational task s n Will be on satellite n k Execute on. ct k (i) represents the time slot t i Satellite n k The amount of data that has been calculated. Similarly, satellite n k The task queue is represented as:

[0092]

[0093] in

[0094] Local device computing model:

[0095] In the local computing model, computing tasks of ground terminal devices will be performed locally. n The local energy consumption cost is:

[0096]

[0097] in represents the computation offloading decision, μ mob,r For ground terminal equipment r The effective energy coefficient, f r,n It's gu r Assigned to tasks n In addition, local computing latency Expressed as:

[0098]

[0099] Among them, c n It is the CPU cycles required to process 1 bit of computation.

[0100] In time slot t i+1 ,gu r The remaining energy is calculated as follows:

[0101]

[0102] in It is a ground terminal device r In time slot t i Harvested energy, Indicates the maximum energy storage value of the ground terminal equipment.

[0103] In addition, tasks n The local queuing delay is:

[0104]

[0105] in and Calculate the task start time and task arrival time.

[0106] LEO satellite calculation model:

[0107] In the LEO satellite computing model, computing tasks will be performed in the LEO satellite. n In satellite n k The energy consumption is calculated as follows:

[0108]

[0109] in represents the computation offloading decision, μ mob,k Indicates satellite n k The effective energy coefficient, f k,n is n k Assigned to tasks n For task s n , satellite calculation delay Expressed as:

[0110]

[0111] In time slot t i+1 , n k The remaining energy is calculated as follows:

[0112]

[0113] in is the satellite in time slot t i Harvested energy, Indicates the maximum energy storage value of the satellite.

[0114] In addition, tasks n The satellite queuing delay is:

[0115]

[0116] The satellite-to-ground link transmission delay is:

[0117]

[0118] in Indicates that in time slot t i Ground terminal equipment r With satellite n k The transmission rate between.

[0119] Computing model of ground cloud computing center:

[0120] In this computing model, the task is performed at the ground cloud computing center gs l Processing. Therefore, task s n Processing delay The calculation is as follows:

[0121]

[0122] where f l,n Represents the ground cloud computing center gs l Processing tasks n In addition, the transmission delay of the ground cloud computing center is:

[0123]

[0124] Among them, M j,k ∈{0,1} is a binary variable representing the intersatellite link (n j ,n k ) Whether to transfer tasks, is the length of the intersatellite communication link, For ground terminal equipment rTo Satellite n k The transmission delay, Indicates the number of satellites n k To the ground cloud computing center gs l transmission delay.

[0125] Problem description:

[0126] The execution delay of a task includes computation delay, transmission delay and queuing delay. Therefore, the time slot t i The total execution delay of the inner task is expressed as follows:

[0127] C(i)=∑ n∈S d n (twenty two);

[0128]

[0129] The long-term average execution delay of a task is:

[0130]

[0131] Based on the above system model, an optimization problem is formulated to minimize the long-term average execution delay of tasks. In this problem, under the premise of long-term task queue stability, the computation offloading and resource allocation decisions are jointly optimized for the computing architecture. Finally, the optimization problem is formulated as

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141]

[0142] where x(i) = [X1(i), X2(i)…X R (i)] represents the time slot t i Unloading decision of ground terminal equipment, is the bandwidth resource allocated by satellite to ground terminal equipment, f gu (i)=[f1(i),f2(i),…f R (i)] is the computing resource allocated to the ground terminal equipment, f sa (i)=[f k,1 (i),f k,2 (i),…f k,R (i)] is the computing resource allocated by the satellite, f gs (i)=[f l,1 (i),f l,2 (i),…f l,R (i)] is the computing resources allocated by the ground cloud computing center.

[0143] In addition, (25a) indicates that the computing power of the GU does not exceed its maximum computing power. (25b) and (25c) indicate that the computing power allocated to the task is less than the maximum computing power of the LEO satellite or the ground cloud computing center. (25d) limits the bandwidth resources allocated to the GU. (25e) and (25f) limit the minimum safe energy of the GU and the LEO satellite. (25g) and (25h) maintain the computing queue at a stable computing rate so that all arriving computing tasks can be executed within a limited time. (25j) ensures that the task is processed locally, on the LEO satellite, or in the ground cloud computing center. Since Problem P1 includes discrete variables and continuous variables, Problem P1 is a long-term MINLP problem and is difficult to solve using conventional methods.

[0144] Step 2: Based on Lyapunov stability theory, the long-term mixed integer nonlinear programming optimization problem is decoupled into sub-optimization problems for each time slot. The long-term average delay optimization under the task queue stability constraint is achieved by minimizing the Lyapunov drift penalty function.

[0145] Specifically, the present invention solves the optimization problem proposed above using a dynamic computational method based on a combination of Lyapunov optimization and reinforcement learning. This method dynamically allocates computing and communication resources between ground-based terminal devices and LEO satellites, achieving optimal offloading results. Based on the premise of a stable computing task queue, the problem of minimizing long-term task execution delays is solved. Based on the Lyapunov optimization algorithm, the quadratic Lyapunov queue model for this system model is defined as:

[0146]

[0147] in In addition, to ensure the stability of the queue, a Lyapunov drift function is introduced, which represents the time slot t i To time slot t i+1The growth amount of the Lyapunov function is designed to optimize performance indicators, such as minimizing latency or reducing power consumption, while ensuring the stability of all queues, which is expressed as follows

[0148] ΔL(Z(i))=E[L(Z(i+1))-L(Z(i))|Z(i)] (27);

[0149] At the same time, it is also necessary to define a minimum solution of the Lyapunov drift function and the objective function based on the Lyapunov drift penalty function, and solve its minimum value by defining the following equation:

[0150]

[0151] Where V>0 is a penalty parameter, which is used to balance the stability of the task queue and the objective function. According to the Lyapunov drift penalty function theory, the above functions are transformed and scaled, and the formula (28) is combined to obtain

[0152]

[0153] By summing the queues on both sides of formula (29), we get

[0154]

[0155] Similarly, the dynamic transformation of the LEO satellite fleet is as follows

[0156]

[0157] By adding the LEO satellite arrays in equation (31), we get

[0158]

[0159] By summing formula (30) and formula (32), we get

[0160]

[0161] Next, we get ΔL(Z(i)). According to the expectation on both sides of formula (31), it can be expressed as

[0162]

[0163] because

[0164] as well as

[0165]

[0166] as well as

[0167]

[0168] Therefore, the upper limit of ΔL(Z(i)) is:

[0169]

[0170] in

[0171] W=W1+W2+W3 (39)

[0172] The task arrivals in each time slot are modeled as independent and identically distributed, which means that at r (i) is not affected by the existing queue length. Finally, by retaining the variables related to the control action and removing the constant term, problem P1 is rewritten as

[0173]

[0174] Step 3: Model the computation offloading problem for each time slot as a Markov decision process and solve it using a deep reinforcement learning algorithm. The deep reinforcement learning algorithm with an actor module dual-strategy network structure generates offloading decisions and resource allocation strategies. The first strategy network outputs a ternary offloading decision and computational resource allocation, while the second strategy network outputs communication resource allocation.

[0175] Step 4: Map the continuous action space into discrete ternary offloading decisions through the ternary rank-preserving quantization method, and optimize the strategy selection in combination with the dynamic noise exploration mechanism. Execute the optimal offloading decision based on the current environmental state, update the task queue and system state, and iterate the optimization until convergence.

[0176] Specifically, the optimization problem described by P2 involves the multi-dimensional state characteristics of communication and computing resources. Traditional static scheduling strategies, due to their inherent limitations, are difficult to flexibly adapt to dynamic changes in the network environment and real-time adjustments to task requirements. To address this challenge, the present invention formalizes the task scheduling decision process as a Markov decision process and designs a computational offloading strategy for latency optimization based on a deep reinforcement learning algorithm to solve this problem. The optimization problem in this invention is specifically defined as follows:

[0177] State space: Set the state space S to

[0178] in By channel gain Residual energy As well as local equipment and LEO satellite task queues constitute;

[0179] where x(i)|ω j(i) is the state space Output of offloading decision, s n For task information, The maximum communication resource.

[0180] Action space: Set the action space A to

[0181] in for Action output To calculate the offloading decision, Make decisions on the corresponding computing resource allocation;

[0182] for The action output of Make decisions about communication resource allocation.

[0183] Reward function: Set the reward function R to the objective function - P2.

[0184] Actor module: The Actor module consists of two policy network submodules and an action quantizer. Figure 2 As shown, in time slot t i In , the parameters of the first policy network are denoted as φ1(i).

[0185] Take the environment variable φ1(i) as the input of the first policy network, in The output of the first output layer of the first policy network uses a sigmoid activation function and is later quantized into feasible ternary decisions.

[0186] in Output by the second output layer of the first policy network using a sigmoid activation function.

[0187] Since problem P2 requires a ternary unloading decision, it is necessary to convert the continuous unloading decision This can be converted into a ternary offloading decision. This is done by multiplying by the number of objects being unloaded. And Explore better decisions by adding noise that decreases over time.

[0188] The quantization function can balance exploration and historical experience training when generating offloading decisions to ensure training convergence. The first set of offloading decisions enhanced from {x(i)|ω1(i)} yes:

[0189]

[0190] Then, the next M is generated by applying a ternary order-preserving quantizer i / 2-1 uninstall decision.

[0191] Then the distance between each offloading decision and its two decision boundaries is calculated. Figure 3 As shown,

[0192] in Indicates the decision deviation distance from the current decision to the GU decision, which is less than This means it is more likely to be quantified as a local offloading decision first.

[0193] Each task will have two decision deviation distances, which are converted using the function G(x) Then, the lower and upper bounds of the decision deviation distances of multiple tasks are calculated simultaneously, and the nearest offloading decision is shifted to the decision target with the smallest distance in turn until M is satisfied. i / 2-1 quantitative unloading decision.

[0194] For convenience, an example is used to illustrate the ternary lawful quantizer method mentioned above. The decision value of {x(i)|ω1(i)} is [1,1,2,0,0]. However, the lower limit of its decision deviation distance is [0.1,0.2,0.5,0.4,0.4], and the upper limit of its decision deviation distance is [0.9,0.8,0.5,0.6,0.6]. It can be seen that 0.1 is the smallest, so the unloading decision changes from [1,1,2,0,0] to [2,1,2,0,0]. Continuing this pattern, if M i / 2=7, then the remaining offloading decisions will become [1,1,2,2,0],[1,1,2,0,1],[1,1,0,0,0],[1,1,1,0,0]. i / 2 offloading decisions, generating additive Gaussian noise Uninstall decision and limit The remaining offloading decisions are then obtained using a ternary rank-preserving quantization method.

[0195] For the second policy network, the parameters are represented as φ2(i), the environment variables φ2(i) are used as the input of the second policy network and the communication resource allocation is output

[0196] in Since GUs share the same LEO satellite bandwidth, a softmax activation function is used as the output layer to make effective decisions.

[0197] To explore better decisions, at the beginning of each time slot, the system updates the exploration noise according to the following rules:

[0198] ∈ i =max(∈ i ·ep,∈ min ) (43);

[0199] Where ep∈[0,1] is a hyperparameter, ∈ min is the minimum exploration noise value.

[0200] Next, the offload decision is converted into a binary mask and multiplied with the bandwidth decision to set the locally computed bandwidth allocation ratio to 0. The bandwidth allocation ratio of non-local offload is then normalized according to the decision target to obtain the final bandwidth allocation.

[0201] Critic module: This Critic module selects the optimal uninstallation decision by setting an appropriate reward. For the uninstallation decision provided by the Actor module, the critic module evaluates and selects the optimal uninstallation decision.

[0202] Policy Update module: based on environment variables and And obtain the best calculation offloading decision through the Actor module and Critic module set up As the first policy network label sample, As the second policy network labeled samples, these labeled samples will be used to update the parameters of each policy network. In the experiment, two experience pools with limited capacity are set up to store labeled samples separately. When the capacity is exceeded, the newly generated samples will replace the old ones.

[0203] Env module: To solve the P2 problem, it uses DNN to obtain the optimal offloading decision. Therefore, the system environment performs joint offloading decisions And simulate new data arriving Then the Env module updates the queue data Finally observe the new environment variables and As the input of the next time slot policy network, and start a new iteration from the Actor module.

[0204] The present invention transforms the delay-optimized satellite-ground fusion network computation offloading strategy problem into an MDP problem and uses DRL to solve the problem, as shown in the following Algorithm 1:

[0205]

[0206]

[0207] The implementation principle of this invention is as follows: This invention discloses a latency-optimized computational offloading strategy for satellite-ground converged networks. This strategy incorporates a reinforcement learning algorithm with a dual-strategy network structure to further enhance the flexibility and real-time performance of computational offloading decisions. By introducing a strategy network to replace the traditional convex optimization module, this significantly improves computational efficiency while maintaining effective resource allocation, overcoming the shortcomings of convex optimization methods, such as complex parameter settings and slow solution speed. Furthermore, a novel action quantization method is designed, breaking through the limitation of traditional methods that only support "two-party offloading selection" and expanding support for three-party offloading decisions involving local nodes, near-Earth satellites, and ground edge servers. This improves the adaptability and generalization of the offloading strategy in multi-access environments. Furthermore, considering the highly dynamic nature of satellite orbits, this paper incorporates orbit-dependent communication conditions and energy availability into the modeling process. By dynamically sensing the connection availability and energy constraints between satellites and ground equipment, the offloading strategy is further optimized, effectively improving the stability and sustainable operation of the system while ensuring service continuity and accessibility.

[0208] The embodiments of this specific implementation method are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A delay optimization calculation offloading strategy for satellite-ground fusion network, characterized by: The following steps are involved: Step 1: Construct a satellite-ground fusion network system model, which includes a network model, a satellite-ground link communication model, a queue model, a local device computing model, a LEO satellite computing model, and a ground cloud computing center computing model, and describe the problem. Step 2: Based on Lyapunov stability theory, the long-term mixed integer nonlinear programming optimization problem is decoupled into sub-optimization problems for each time slot. The long-term average delay optimization under the task queue stability constraint is achieved by minimizing the Lyapunov drift penalty function. Step 3: Model the computation offloading problem for each time slot as a Markov decision process and solve it using a deep reinforcement learning algorithm. The deep reinforcement learning algorithm with an actor module dual-strategy network structure generates offloading decisions and resource allocation strategies. The first strategy network outputs a ternary offloading decision and computational resource allocation, while the second strategy network outputs communication resource allocation. Step 4: Map the continuous action space into discrete ternary offloading decisions through the ternary rank-preserving quantization method, and optimize the strategy selection in combination with the dynamic noise exploration mechanism. Execute the optimal offloading decision based on the current environmental state, update the task queue and system state, and iterate the optimization until convergence.

2. The delay optimization calculation offloading strategy for satellite-ground fusion network according to claim 1 is characterized in that: In step 1, the satellite-to-ground link communication model in the satellite-to-ground fusion network system model calculates the satellite-to-ground link transmission rate based on the Shannon formula: in and Is user gu r In time slot t i The transmission power and channel gain when It is a satellite n k , the channel gain, It is a satellite n k Assigned to ground terminal equipment r bandwidth resources, It is from gu r To Satellite n k Free space loss, L t is the other link loss, K is the Boltzmann constant, and T is the noise temperature, where It is expressed as follows: in It is a ground terminal device r To Satellite n k The link length between them, c and F are the speed of light and the communication center frequency.

3. The delay optimization calculation offloading strategy for satellite-ground fusion network according to claim 1 is characterized in that: In step 1, the dynamic update of the task queue in the system model satisfies: Ground terminal equipment queue: Low Earth Orbit Satellite Fleet: where at r (i) at k (i) are the task arrival amount, ct r (i) ct k (i) is the task processing capacity.

4. The delay optimization calculation offloading strategy for satellite-ground fusion network according to claim 1 is characterized in that: In step 2, the Lyapunov drift penalty function is defined as: Where ΔL(Z(i)) is the Lyapunov drift of the queue length, V is the penalty parameter, and C(i) is the total execution delay within the time slot; By minimizing the function, the long-term optimization goal is transformed into an immediate resource allocation problem for each time slot.

5. The delay optimization calculation offloading strategy for satellite-ground fusion network according to claim 1 is characterized in that: In step 3, the dual-strategy network structure includes: The first policy network inputs the environment state information Including the channel gain, remaining energy, local and satellite task queue lengths of ground terminal equipment, outputting continuous ternary offloading decision signals and calculating resource allocation ratios; The second policy network inputs the environment state information Including the intermediate results of offloading decisions, task attributes and maximum bandwidth constraints, and outputting the communication resource allocation ratio; The first strategy network and the second strategy network store samples through independent experience pools and update network parameters based on the Adam algorithm.

6. The delay optimization calculation offloading strategy for satellite-ground fusion network according to claim 1 is characterized in that: In step 3, a Critic module is also included to evaluate the candidate offloading decisions generated by the Actor module and select the optimal strategy based on a reward function; the reward function is consistent with the objective function of the optimization target P2, that is, minimizing the weighted delay and queue backlog.

7. The delay optimization calculation offloading strategy for satellite-ground fusion network according to claim 1 is characterized in that: In step 4, the ternary order-preserving quantization method includes the following steps: Multiply the continuous action value by the number of unloaded objects and divide it into [0,0.5], [0.5,1.5], [1.5,2.5] The intervals correspond to local processing, satellite computing, and ground cloud computing center computing; The quantization boundary is dynamically adjusted according to the decision deviation distance, the nearest offloading target is prioritized, and the policy diversity is expanded by adding Gaussian noise.

8. The delay optimization calculation offloading strategy for satellite-ground fusion network according to claim 1 is characterized in that: In step 4, the update rule of the dynamic noise exploration mechanism is: ∈ i =max(∈ i ·ep,∈ min ) (43); Where ep∈[0,1] is a hyperparameter, ∈ min is the minimum exploration noise value, which is used to balance strategy exploration and exploitation.

9. The delay optimization calculation offloading strategy for satellite-ground fusion network according to claim 1 is characterized in that: The strategy also includes: In the communication resource allocation, the Softmax function is used to normalize the bandwidth allocation ratio to ensure the total bandwidth constraint of the satellite; In local computing mode, the bandwidth allocation ratio is forced to zero to avoid resource waste.

Images