A task scheduling and resource slicing method and system for a low earth orbit satellite network

Abstract

The application provides a task scheduling and resource slicing method and system for a low-orbit satellite network, wherein the low-orbit satellite network receives a service request initiated by a terminal user; a current resource state set of each satellite is obtained, a task scheduling mechanism is constructed based on reinforcement learning to generate a task scheduling strategy according to a network state; a current resource allocation set of each satellite is obtained and a satellite resource slicing utility function is constructed; a resource slicing mechanism is constructed based on a heuristic algorithm, each satellite resource slicing candidate solution is regarded as a particle with the same electric charge, in one iteration, the fitness value of each candidate solution is calculated according to the utility function to perform evaluation, the candidate solution with a higher fitness value receives a higher electric charge, and the candidate solution with a lower fitness value receives a lower electric charge; the candidate solution is updated according to the optimal fitness value and the worst fitness value of the overall satellite, multiple iterations are performed to obtain an optimal solution, and a resource slicing strategy is generated. The method provided by the application can flexibly adjust resource allocation, improve processing efficiency and resource utilization.

Description

Technical Field

[0001] The present invention relates to the field of low-orbit satellite network technology, and in particular to a task scheduling and resource slicing method and system for low-orbit satellite networks. Background Art

[0002] With the vigorous development of next-generation communication and network technologies such as 5G and Wi-Fi 6, a large number of new user services such as the Internet of Vehicles, VR / AR, 4K / 8K, and smart cities have emerged, greatly improving people's quality of life and promoting social development. However, due to infrastructure limitations, traditional Internet access is often difficult in remote areas. With the continuous development of low-Earth orbit (LEO) satellite technology, LEO satellite services have shown great potential in fields such as communications and earth observation. Satellite networks have the characteristics of wide coverage and broadcast communications, and can cover remote areas that are difficult to reach with traditional terrestrial networks, providing these areas with reliable Internet connections. At the same time, LEO constellations can improve service quality with lower signal propagation delays, which means that LEO constellations have the potential to quickly and accurately process delay-sensitive and computationally intensive services from mobile users. With the continuous improvement of satellite communication technology, satellite networks are gradually becoming an important part of future networks.

[0003] Although low-orbit satellites are expected to provide global communication services and have the advantage of short propagation delays, they still face many challenges when handling user computing services. First, the high dynamics of satellites lead to frequent periodic link interruptions, which poses a significant challenge to communication stability. Frequent topology changes will affect end-to-end data transmission, which will significantly affect the user experience in real-time applications and multimedia transmission. At the same time, due to the high dynamics of satellites, it is necessary to dynamically adjust the resource slicing decisions of multiple service flows to meet business needs. The distributed resource slicing strategy of low-orbit satellites needs to be continuously updated. In addition, due to the limited resources of low-orbit satellites, the traditional static resource slicing method leads to low resource utilization and difficulty in flexible load balancing, which brings challenges to the resource management of low-orbit satellite constellations.

[0004] The existing technical solution introduces low-orbit satellites, synchronous satellites and ground stations into the three-layer system network of resource slicing, such as Figure 1As shown. In this hierarchical structure, synchronous satellites share the computing tasks of satellites, while low-orbit satellites, synchronous satellites and ground stations are independent individuals that can independently process user-generated requests. This three-layer system provides users with a more flexible and efficient way to utilize resources. The sequential decision-making of low-orbit satellites, synchronous satellites and ground stations is studied, and this resource slicing problem is modeled as a three-layer Stackelberg game. In this game, the ground station acts as a leader, the synchronous satellite acts as a follower, and the satellite acts as a sub-follower. This model takes into account the hierarchy and mutual influence between each level to more comprehensively understand and optimize the resource slicing strategy. The research method includes the application of a multi-agent reinforcement learning algorithm, which aims to search for the Nash equilibrium of the game to learn the optimal resource management strategy. Through this intelligent learning method, the system can adaptively optimize the allocation of resources to adapt to the ever-changing satellite environment and user needs. However, this technical solution also has at least the following three defects:

[0005] (1) This technical solution relies too much on resource monitoring decisions among various components when allocating resources. Although user computing tasks can directly obtain services provided by satellite nodes, low-orbit satellite resource slicing decisions must be coordinated with ground stations and space stations in the global topology. Moreover, the ground station acts as a leader and the synchronous satellite acts as a follower. This solution relies too much on resource monitoring decisions among various components during the resource slicing process. Although user computing tasks can directly obtain services provided by satellite nodes, low-orbit satellite resource slicing decisions must be coordinated with ground stations and space stations in the global topology. As a result, this architecture has the risk of single point failure and faces challenges in increasing link redundant traffic.

[0006] (2) When the business runs on a low-orbit satellite node that is close to full load and the actual resources used exceed the pre-allocated amount, the three-layer resource slicing network cannot expand the resources, resulting in task processing failure and resource waste. It is impossible to dynamically slice the satellite node resources to meet the needs of user services.

[0007] (3) It is difficult to use multi-agent reinforcement learning to handle computationally intensive tasks in a multi-satellite distributed collaborative scenario. The resource slicing mechanism for low-orbit satellites using multi-agent reinforcement learning is very complex and needs to consider many factors, such as the available computing resources and remaining energy of all satellites, and the link delay and capacity between all satellites. At the same time, when considering the optimization of ground stations and space stations, multi-agent reinforcement learning will make it more difficult to achieve resource slicing strategy learning due to the increase in the action state space. Summary of the Invention

[0008] In view of this, an embodiment of the present invention provides a task scheduling and resource slicing method and system for low-orbit satellite networks to eliminate or improve one or more defects in the existing technology, and solve the problems of waste of computing and storage resources, low efficiency of resource slicing resources, and poor satellite distributed coordination effect in the current low-orbit satellite networks.

[0009] In one aspect, the present invention provides a method for task scheduling and resource slicing for a low-orbit satellite network. The method is performed in a low-orbit satellite network having a distributed structure. The method comprises the following steps:

[0010] Receive service requests initiated by end users;

[0011] Obtaining a current resource status set of each satellite; the resource status set includes at least a task generation status, a communication status between each satellite, and a communication status between each satellite and the ground;

[0012] A task scheduling mechanism is constructed based on reinforcement learning. The state space is constructed based on the task generation state, data queue state, and communication state between satellites and between satellites and the ground. The action space is constructed based on the task scheduling performed by each satellite. The reward function is constructed by minimizing the task processing cost. The task scheduling mechanism is trained using experience replay to generate a corresponding task scheduling strategy based on the current state.

[0013] Obtain the current resource allocation set of each satellite and construct the satellite resource slice utility function;

[0014] A resource slicing mechanism is constructed based on a heuristic algorithm. Candidate solutions for resource slicing on each satellite are considered to have the same charge. In one iteration, the fitness value of each candidate solution is calculated according to the utility function for evaluation. Candidate solutions with higher fitness values ​​receive higher charges, while those with lower fitness values ​​receive lower charges. Candidate solutions are updated based on the optimal and worst fitness values ​​of the entire satellite. In multiple iterations, the interactions between charges are simulated so that the charges are concentrated near the optimal solution to obtain the optimal solution and generate a resource slicing strategy.

[0015] Task scheduling and resource slicing are performed on each satellite according to the task scheduling strategy and the resource slicing strategy, and the results are fed back to the end user after the task is completed.

[0016] In some embodiments of the present invention, a current resource slice set of each satellite is obtained, where the set is defined as:

[0017] Y=[Y 1 ,...,Y v ,...,Y V ];

[0018]

[0019] Where Y represents the candidate solution set of resource slices of all satellites; γ com，v represents the communication resource slice ratio of candidate solution v; γ comp，v Indicates the computing resource slice ratio of the candidate solution v; Indicates the size of the candidate solution v when it is the d-th resource slice.

[0020] In some embodiments of the present invention, the utility function is defined as:

[0021] Bene=[Bene 1 ,...,Bene v ,...,Bene V ];

[0022] Bene is calculated by the following equation:

[0023]

[0024] Where Bene represents the utility value of the candidate solution v; ζ j represents the weight of computing task j, j∈J; Γ j，m (t) represents the generation status of computing task j of satellite m; Indicates whether computing task j is scheduled from satellite m to satellite n; x k，j，m (t) represents the data size of computing task j; Indicates the computing resource slice ratio; F n represents the total computing power of satellite n; represents the transmission rate from satellite n to satellite m.

[0025] In some embodiments of the present invention, calculating the fitness value of each candidate solution according to the utility function for evaluation further includes:

[0026] The calculation formula for the position of the optimal fitness value obtained by the charged particle corresponding to each candidate solution at any time is:

[0027]

[0028] in, represents the position of the optimal fitness value obtained by particle i in dimension d at time t+1; Bene(·) represents the utility function; P v (t) represents the position of the global optimal fitness value at time t; Y v (t+1) represents the candidate solution v at time t+1; Represents the size of the candidate solution v when it is the d-th resource slice at time t+1.

[0029] In some embodiments of the present invention, the method further comprises:

[0030] The Coulomb constant is calculated based on the current number of iterations and the preset total number of iterations. The calculation formula is:

[0031]

[0032] Wherein, K(t) represents the Coulomb constant; K0 represents the initial value; β represents a constant; iteration represents the current number of iterations; and max iteration represents the preset total number of iterations.

[0033] The Coulomb force between particles is calculated based on the Coulomb constant, and the calculation formula is:

[0034]

[0035] in, represents the Coulomb force exerted by particle h on particle i in dimension d of time t; K(t) represents the Coulomb constant; Q i (t) represents the charge of particle i; Q h (t) represents the charge of particle h; The location representing the fitness value of particle h of dimension d; represents the size of particle i at time t when it is the dth resource slice; R ih (t) represents the Euclidean distance between particles i and h; ε represents a small positive constant;

[0036] Calculate the total Coulomb force on each particle using the following formula:

[0037]

[0038] in, represents the sum of the Coulomb forces on particle i at time t and dimension d from other particles; rand(·) represents a uniform random number in [0, 1]; Represents the Coulomb force exerted by particle h on particle i in dimension d at time t.

[0039] In some embodiments of the present invention, the method further comprises:

[0040] The electric field of the particle is calculated by dividing the total Coulomb force on the particle by the charge of the particle. The calculation formula is:

[0041]

[0042] in, represents the electric field of particle i in dimension d at time t; represents the total Coulomb force on particle i in dimension d at time t from other particles; Q i (t) represents the charge of particle i;

[0043] According to Newton's second law, the acceleration of the particle is calculated as follows:

[0044]

[0045] in, represents the acceleration of particle i in dimension d at time t; Q i (t) represents the charge of particle i; represents the electric field of particle i in dimension d at time t; M i (t) represents the mass of particle i.

[0046] In some embodiments of the present invention, in each iteration, the updated expressions of the particle's velocity and position are:

[0047]

[0048]

[0049] in, represents the velocity of particle i at time t and dimension d; rand(·) represents a uniform random number in [0, 1]; represents the acceleration of particle i in dimension d at time t; represents the position of particle i in dimension d at time t.

[0050] In some embodiments of the present invention, updating the candidate solution according to the optimal fitness value and the worst fitness value of the entire satellite further includes:

[0051] The charge function of the particle is defined as:

[0052]

[0053] Among them, q i (t) represents the charge of particle i at time t; Bene i (t) represents the utility value of particle i at time t; best(t) represents the optimal fitness value; worst(t) represents the worst fitness value;

[0054] The calculation formulas of the optimal fitness value and the worst fitness value are respectively:

[0055] best(t)=max(Bene v (t)), i∈(1, 2,...,V);

[0056] worst(t)=min(Bene v (t)), i∈(1, 2,...,V);

[0057] Among them, Bene v (t) represents the utility value of particle v at time t;

[0058] According to the charge function, the charge update expression of the particle is:

[0059]

[0060] Among them, Q i (t) represents the updated charge of particle i.

[0061] On the other hand, the present invention provides a task scheduling and resource slicing system for a low-orbit satellite network. When the system is executed, the steps of any one of the task scheduling and resource slicing methods for a low-orbit satellite network mentioned above are implemented. The system includes:

[0062] End users, used to initiate service requests;

[0063] A low-orbit satellite network is composed of satellite nodes and is provided with a task scheduling mechanism and a resource slicing mechanism. The mechanism is used to generate a task scheduling strategy and a resource slicing strategy according to the state of the low-orbit satellite network, perform task scheduling and resource slicing on the satellite nodes, and feedback the results to the end user after the task processing is completed.

[0064] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any one of the methods mentioned above when executed by a processor.

[0065] The beneficial effects of the present invention are at least:

[0066] The present invention provides a task scheduling and resource slicing method and system for low-orbit satellite networks, including: a low-orbit satellite network receives a service request initiated by a terminal user; obtains resource information of each satellite, computing task status information, etc.; constructs a small-time-scale low-orbit satellite network task scheduling mechanism based on reinforcement learning, uses a Markov decision process to achieve optimal scheduling of computing tasks, and performs reasonable scheduling based on the actual energy loss and time delay of task processing, thereby solving the problems of mismatch between computing task service requirements caused by high dynamics of satellites in traditional task processing methods, and resource waste caused by mismatch between static resources of low-orbit satellites and computing task requirements, so as to minimize network energy consumption and task processing delay and improve task processing performance; at the same time, obtains the current resource allocation set of each satellite and constructs a satellite resource slicing mechanism. The system uses a utility function to calculate the fitness value of each candidate solution according to the utility function. The system updates the candidate solution according to the optimal and worst fitness values ​​of the entire satellite, and performs multiple iterations to obtain the optimal solution and generate a resource slicing strategy. The system uses a heuristic algorithm to slicing distributed resources, which can flexibly adjust resource allocation, adapt to the high-speed movement of satellites and the rapid changes in computing tasks, and maximize the overall network service capabilities. At the same time, it can reduce communication delays, increase data transmission speeds, reduce system energy consumption, and improve global processing efficiency and resource utilization.

[0067] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0068] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0070] Figure 1 This is a schematic diagram of the steps of a task scheduling and resource slicing method for a low-orbit satellite network in one embodiment of the present invention.

[0071] Figure 2This is a flowchart of a task scheduling and resource slicing method for a low-orbit satellite network in one embodiment of the present invention.

[0072] Figure 3 This is a schematic diagram of the structure of a task scheduling and resource slicing system for a low-orbit satellite network in one embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0074] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0075] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0076] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0077] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0078] It should be emphasized here that the step marks mentioned below do not limit the order of the steps, but it should be understood that the steps can be executed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be executed simultaneously.

[0079] In order to solve the problems of waste of computing and storage resources, low efficiency of resource slicing, and poor satellite distributed coordination in the current low-orbit satellite network, the present invention proposes a task scheduling and resource slicing method for low-orbit satellite networks. The method is executed in the low-orbit satellite network, such as Figure 1 As shown, the method includes the following steps S101 to S106:

[0080] Step S101: receiving a service request initiated by a terminal user.

[0081] Step S102: Acquire the current resource status set of each satellite, wherein the resource status set at least includes the mission generation status, the communication status between each satellite, and the communication status between each satellite and the ground.

[0082] Step S103: Construct a task scheduling mechanism based on reinforcement learning, construct a state space based on the task generation state, data queue state, and communication state between satellites and between satellites and the ground, construct an action space based on the task scheduling performed by each satellite, and construct a reward function based on minimizing the task processing cost; use experience replay to train the task scheduling mechanism to generate a corresponding task scheduling strategy based on the current state.

[0083] Step S104: Obtain the current resource allocation set of each satellite and construct a satellite resource slice utility function.

[0084] Step S105: A resource slicing mechanism is constructed based on a heuristic algorithm. The candidate solutions for resource slicing of each satellite are regarded as particles with the same charge. In one iteration, the fitness value of each candidate solution is calculated according to the utility function for evaluation. The candidate solution with a higher fitness value receives a higher charge, and the candidate solution with a lower fitness value receives a lower charge. The candidate solution is updated according to the optimal fitness value and the worst fitness value of the entire satellite. In multiple iterations, the interaction between charges is simulated so that the charges gather near the optimal solution to obtain the optimal solution and generate a resource slicing strategy.

[0085] Step S106: Perform task scheduling and resource slicing on each satellite according to the task scheduling strategy and resource slicing strategy, and feedback the results to the end user after the task is completed.

[0086] like Figure 2 , which is a flowchart of a task scheduling and resource slicing method for low-orbit satellite networks.

[0087] In step S101, an end user initiates a service request to a low-orbit satellite network. In this context, end users are Internet of Remote Things (IoRT) users, encompassing any single device or device cluster utilizing the services of a low-orbit satellite edge computing network. These devices include not only terrestrial devices but also those on the ocean and in the air.

[0088] The end user initiates a service request to the low-orbit satellite network, which processes the data and then feeds back the results to the end user, completing the computing task.

[0089] In step S102, first obtain the initial (current) resource state set, specifically: the task generation state Γ of satellite node n n (t) Whether the task is scheduled from satellite node n to satellite node m The task queue sequence Ψ of satellite node nn，j (t), the communication status between satellite node n and satellite node m, that is, the data transmission rate R from satellite node n to satellite node m n，m (t), the communication status between satellite node n and the ground, that is, the data transmission rate R from satellite node n to ground 0 n，0 (t). If the task queue reaches its maximum capacity, some tasks may be discarded and lost. Therefore, it is also necessary to obtain the discarded task amount Ψ of the task sequence k of satellite node n k，j，n (t).

[0090] At the same time, the computing task status of each satellite is obtained, including: task sequence k j，n ∈Γ j，n (t), the computing resources c required to process computing task j j , task sequence k j，n The transmission delay D k，j，n (t), mainly includes the task transmission delay, data buffer waiting delay and execution delay, task sequence k j，n Energy loss E k，j，n (t), which mainly includes the transmission energy consumption and processing energy consumption of the task.

[0091] In step S103, a task scheduling mechanism is constructed based on reinforcement learning.

[0092] First, the optimization goal of satellite task scheduling is constructed. Taking into account the user's service quality and the limited satellite resources, the task processing delay and energy consumption are minimized. At the same time, in order to ensure the task completion rate, the present invention combines the task processing delay, energy consumption, and overflow queue as the cost of task processing, as shown in formula (1):

[0093]

[0094] In formula (1), π represents the task scheduling strategy; t represents the time slot, t∈T; n represents the satellite node, n∈N; j represents the computing task, j∈J; c t represents the task processing cost at time slot t.

[0095] The task processing cost at time slot t can be expressed as formula (2):

[0096]

[0097] In formula (2), E k，j，n (t) represents the energy consumption of task sequence k; D k，j，n (t) represents the transmission delay of task sequence k; Ψ k，j，n (t) represents the amount of tasks discarded in task sequence k; K E , KD , K Ψ is the performance gain coefficient.

[0098] The main obstacle to optimizing the goal is managing long-term constraints. In this paper, the Lyapunov technique is introduced to transform the problem. Specifically:

[0099] First, a delay deficit queue is used to represent the level of achievement of long-term delay constraints to guide the agent to meet these constraints. First, a delay deficit queue {Y j.t} j∈J , its dynamic update process is shown in formula (3):

[0100]

[0101] In formula (3), Y j，t+1 represents the delay deficit queue of computing task j in time slot t+1; d j，t represents the processing delay of computing task j in time slot t; represents the maximum tolerable delay of computing task j in time slot t; Y j.t represents the latency deficit queue for computing task j in time slot t.

[0102] In order to express the satisfaction degree of the delay requirement, the Lyapunov function is introduced, as shown in formula (4):

[0103] L(Y j.t )=Y j.t / twenty four)

[0104] To ensure compliance with the long-term delay constraint, the Lyapunov function should be as small as possible. Therefore, a Lyapunov drift is introduced to track the change of the Lyapunov function between two consecutive time slots, as shown in formula (5):

[0105] Δ(Y j.t )=L(Y j.t+1 )-L(Y j.t ); (5)

[0106] In formula (4), Δ(Y j.t ) represents Lyapunov drift; Y j，t+1 Y represents the delay deficit queue of computing task j in time slot t+1; j.t represents the latency deficit queue for computing task j in time slot t.

[0107] The upper limit of Lyapunov drift is limited as shown in formula (6):

[0108]

[0109] In formula (6), d j，trepresents the processing delay of computing task j in time slot t; represents the maximum tolerable delay of computing task j in time slot t; is a constant, represents the maximum achievable delay of computing task j.

[0110] According to Lyapunov optimization theory, the above optimization objective can be simplified to minimizing the drift plus the penalty term, as shown in formula (7):

[0111]

[0112] In formula (7), It represents the weighted constant for adjusting each optimization indicator to meet the long-term energy consumption, load balance and queue overflow constraints. The annotations of other physical quantities refer to the above and will not be repeated here.

[0113] On this basis, a task scheduling mechanism based on reinforcement learning is constructed.

[0114] Since the task scheduling policy at time slot t will be affected by the previous time slot, the scheduling process can be modeled as a Markov decision process (MDP). Based on the delay deficit queue introduced above, the following state space, action space, and reward function can be obtained, as shown in formulas (8), (9), and (10), respectively:

[0115] s t ={Γ n (t),φ n，j (t), R n，m (t), R n，0 (t)|n,m∈N,j∈J}; (8)

[0116] In formula (8), Γ n (t) represents the task generation state; φ n，j (t) indicates the data queue status; R n，m (t) represents the communication status between satellite n and satellite m; R n，0 (t) represents the communication status between satellite n and the ground.

[0117]

[0118] In formula (9), Indicates the status of whether the task is scheduled from satellite n to satellite m; k represents the task sequence.

[0119]

[0120] In formula (10), t represents the time slot, t∈T; n represents the satellite node, n∈N; j represents the computing task, j∈J; E k，j，n(t) represents the energy consumption of task sequence k; D k，j，n (t) represents the transmission delay of task sequence k; Ψ k，j，n (t) represents the amount of tasks discarded in task sequence k; K E , K D , K Ψ is the performance gain coefficient.

[0121] In some embodiments, the present invention proposes a task scheduling algorithm based on a double-delayed deep deterministic policy gradient (TD3) algorithm. Deep reinforcement learning algorithms can sometimes achieve good performance, but they also encounter function approximation problems, resulting in overestimation and invalid policies. Based on the deep reinforcement learning algorithm, TD3 implements the following three improvements to address the overestimation bias problem: 1) clipped double-Q network, 2) delayed policy update, and 3) target policy smoothing regularization. At the same time, the TD3 task scheduling algorithm consists of many training segments, each of which consists of several steps. An event corresponds to a schedule window, and a step corresponds to a schedule slot.

[0122] In some embodiments, the task scheduling mechanism is trained by using experience replay, which mainly includes the following steps:

[0123] Initialize the Critic network, Actor network, and target network.

[0124] In each time slot, the agent retrieves the current state of the low-orbit satellite network from the state space and generates an action based on the current state and the policy. In this step, the present invention adds exploratory noise to enhance the diversity of training samples, helping to quickly discover the optimal policy. The selected action then yields a reward and the next time slot state.

[0125] The current state, action, reward value and next time slot state are constructed into a four-tuple {s t , a t , r t , s t+1}, stored in the experience replay memory.

[0126] In each time slot, a mini-batch of samples M is sampled from the experience replay memory b , used to train and update the Critic network, Actor network and target network.

[0127] The following is a further explanation of the network parameter update.

[0128] For the critic network, a dual-Q network is used for each update, with the network with a smaller Q value as the Q target to mitigate the negative impact of overestimation. The update process of the dual-Q network is shown in formula (11):

[0129]

[0130] In formula (11), r t represents the reward function; γ represents the discount factor; Denote the parameter θ i Q’ Q function of s t+1 Indicates the status of time slot t+1; Represents the output of the target policy.

[0131] The critic network is updated by minimizing the loss. The update principle is shown in formula (12):

[0132]

[0133] In formula (12), L(θ i Q ) represents the loss, and its calculation formula is shown in formula (13):

[0134]

[0135] In formula (13), M b represents a small batch of samples; Represents the predicted value.

[0136] For the Actor network, the update frequency of the Actor network should be lower than that of the Critic network to ensure that the estimation error is reduced before updating the policy. Therefore, the Actor network is updated only after the Critic network has been updated a preset number of times.

[0137] The update principle of the Actor network is shown in formula (14):

[0138]

[0139] In formula (14), J(φ) represents the gradient of the Q function with respect to the policy parameter φ; M b represents a small batch of samples; represents the Q function with parameter θ1; π φ (s) represents the gradient of policy π with respect to policy parameter φ.

[0140] While reducing the update frequency, the target network is updated using soft update. The calculation formula is shown in formula (15):

[0141] θ i Q’ ←τθ i Q +(1-τ)θ i Q’ ; (15)

[0142] In formula (15), τ represents the soft update factor.

[0143] The learning process of the proposed TD3 task scheduling algorithm alternates between critic and actor updates until convergence. Furthermore, because the algorithm is offline, offline pre-training is feasible in practical applications, requiring minimal time and resource investment and making it more adaptable to limited satellite resources.

[0144] Based on the aforementioned task scheduling mechanism, after outputting a satellite computing task scheduling strategy based on the current network status, the satellite will then perform a series of operations according to the optimized task scheduling strategy to effectively schedule computing tasks and meet dynamically changing task requirements. Based on the optimized task scheduling strategy, the satellite node ensures that it can adapt to the current task requirements and network status, comprehensively considering the status of task generation, communication, and data queues to make optimal scheduling decisions to maximize overall system performance.

[0145] In step S104, firstly, the current (initial) resource allocation set of each satellite is obtained.

[0146] In some embodiments, the set of candidate solutions for all satellites to use for a resource slice pair is defined as equation (16):

[0147] Y=[Y 1 ,...,Y v ,...,Y V ]; (16)

[0148] In formula (16), Y represents the candidate solution set of resource slices of all satellites.

[0149] Among them, the candidate solution v can be expressed as formula (17):

[0150]

[0151] In formula (17), γ com，v represents the communication resource slice ratio of candidate solution v; γ comp，v Indicates the computing resource slice ratio of the candidate solution v; Indicates the size of the candidate solution v when it is the d-th resource slice.

[0152] Each candidate solution represents the resource configuration of a low-orbit satellite. Each candidate solution is a charged particle with the same charge, possessing a random initial position and velocity in an electrostatic field. It should be noted that in this disclosure, candidate solutions for each satellite resource slice are considered particles with the same charge. The candidate solutions are denoted by v, and the particles are denoted by i.

[0153] Then, a utility function of satellite resource slicing is constructed. In the present invention, the position of each particle in the search space corresponds to the utility value of the candidate solution to the resource slicing problem.

[0154] In some embodiments, the utility function of a candidate solution for a resource slice (i.e., the position function of the corresponding particle in the search space) can be defined as formula (18):

[0155] Bene=[Bene 1 ,...,Bene v ,...,Bene V ]; (18)

[0156] Wherein, Bene is calculated by formula (19):

[0157]

[0158] In formula (19), Bene represents the utility value of the candidate solution v, that is, the position of the corresponding particle in the search space; ζ j represents the weight of computing task j, j∈J; Γ j，m (t) represents the generation status of computing task j of satellite m; Indicates whether computing task j is scheduled from satellite m to satellite n; x k，j，m (t) represents the data size of computing task j; Indicates the computing resource slice ratio; F n represents the total computing power of satellite n; represents the transmission rate from satellite n to satellite m.

[0159] In step S105, a resource slicing mechanism is constructed based on a heuristic algorithm.

[0160] In each iteration, based on the above utility function, the position of the global optimal utility value and the position of the individual optimal utility value of each particle are calculated.

[0161] In some embodiments, the position of the optimal utility value (fitness value) obtained by a particle at any time is shown in formula (20):

[0162]

[0163] in, represents the position of the optimal fitness value obtained by particle i in dimension d at time t+1; Bene(·) represents the utility function; P v (t) represents the position of the global optimal fitness value at time t; Y v (t+1) represents the candidate solution v at time t+1; Represents the size of the candidate solution v when it is the d-th resource slice at time t+1.

[0164] The position of the global optimal fitness value of all fitness values ​​is given by P best =Y best In order to quickly find the optimal solution, during the iteration process, the candidate solution with a higher fitness value receives a higher charge, and the candidate solution with a lower fitness value receives a lower charge.

[0165] In some embodiments, the calculation formulas for the optimal fitness value and the worst fitness value are shown in formula (21) and formula (22), respectively:

[0166] best(t)=max(Bene v (t)), i∈(1, 2,...,V); (21)

[0167] worst(t)=min(Bene v (t)), i∈(1, 2,...,V); (22)

[0168] Among them, Bene v (t) represents the utility value of the candidate solution v (particle i) at time t.

[0169] In an electrostatic field, the change in velocity or acceleration of any charged particle is equal to the force applied to the system divided by the particle's mass. In this invention, the Coulomb constant used to calculate the Coulomb force is a function of the current iteration number and the preset total number of iterations, and decreases exponentially to control the algorithm's search accuracy.

[0170] In some embodiments, the Coulomb constant is calculated by equation (23):

[0171]

[0172] In formula (23), K(t) represents the Coulomb constant; K0 represents the initial value; β represents a constant; iteration represents the current number of iterations; and max iteration represents the preset total number of iterations.

[0173] In an electrostatic field, charged particles move under the action of Coulomb force. The current velocity of any charged particle is equal to the sum of its previous velocity and the change in velocity. The acceleration of any charged particle is equal to the force applied to the system divided by the mass of the particle.

[0174] In some embodiments, in the search space, at time t, the Coulomb force exerted by particle h on particle i is as shown in formula (24):

[0175]

[0176] In formula (24), represents the Coulomb force exerted by particle h on particle i in dimension d at time t; K(t) represents the Coulomb constant; Q i (t) represents the charge of particle i; Q h (t) represents the charge of particle h; The location representing the fitness value of particle h of dimension d; represents the size of particle i at time t when it is the dth resource slice; R ih (t) represents the Euclidean distance between particles i and h; ε represents a small positive constant.

[0177] In some embodiments, in the search space, at time t, the sum of the Coulomb forces exerted on particle i by other particles is as shown in formula (25):

[0178]

[0179] In formula (25), represents the sum of the Coulomb forces on particle i at time t and dimension d from other particles; rand(·) represents a uniform random number in [0, 1]; Represents the Coulomb force exerted by particle h on particle i in dimension d at time t.

[0180] In some embodiments, in the search space, the electric field of particle i in dimension d at time t can be calculated using formula (26):

[0181]

[0182] In formula (26), represents the electric field of particle i in dimension d at time t; represents the total Coulomb force on particle i in dimension d at time t from other particles; Q i (t) represents the charge of particle i.

[0183] According to Newton's second law, the acceleration of particle i in dimension d at time t can be calculated using formula (27):

[0184]

[0185] In formula (27), represents the acceleration of particle i in dimension d at time t; Q i (t) represents the charge of particle i; represents the electric field of particle i in dimension d at time t; M i (t) represents the mass of particle i.

[0186] According to formula (27), the velocity and position update equations of charged particle i can be obtained, as shown in formula (28) and formula (29), respectively:

[0187]

[0188]

[0189] In formula (28) and formula (29), represents the velocity of particle i at time t and dimension d; rand(·) represents a uniform random number in [0, 1]; represents the acceleration of particle i in dimension d at time t; represents the position of particle i in dimension d at time t.

[0190] In order to allow the candidate solution to quickly approach the optimal solution, it is necessary to allow particles with higher fitness values ​​to have higher charges in order to generate a greater Coulomb force. Therefore, the charge function is defined as shown in formula (30):

[0191]

[0192] In formula (30), q i (t) represents the charge of particle i at time t; Bene i (t) represents the utility value of particle i at time t; best(t) represents the optimal fitness value; and worst(t) represents the worst fitness value. Where best(t) and worst(t) are calculated by formula (21) and formula (22), respectively.

[0193] Then the charge of particle i at time t can be updated by formula (31):

[0194]

[0195] In formula (31), Q i (t) represents the updated charge of particle i. According to formula (31), the charge value range of all particles is [0, 1].

[0196] Based on the above update method, multiple iterations are performed to simulate the interaction between charges so that the charges gather near the optimal solution. After reaching the preset total number of iterations, the optimal solution is obtained and a resource slicing strategy is generated.

[0197] In step S106, after obtaining the resource slicing strategy based on step S105, the satellite will perform a series of operations according to the optimized resource slicing scheme to effectively deploy computing services and meet dynamically changing mission requirements. Based on the optimized resource slicing strategy, the satellite node will dynamically adjust its resource allocation to ensure that the resource slices can adapt to the current mission requirements and network status, including reallocating computing, communication, and storage resources to maximize overall system performance.

[0198] In some embodiments, as Figure 2 As can be seen from the overall flow chart, the present invention proposes a dual-time-scale optimization framework, including iterative solutions for large-time-scale (minute-level) resource slicing and small-time-scale (second-level) task scheduling, to obtain the optimal task scheduling and resource slicing strategy. As shown in the flow chart, after the first iteration of the task scheduling problem, the task scheduling strategy is obtained and fed back to the resource slicing part to solve the latest resource slicing strategy. Subsequently, a second iteration is performed to solve the task scheduling strategy and resource slicing strategy again in turn to determine the new task scheduling strategy and resource slicing strategy. After multiple iterations until the task scheduling mechanism converges, the optimal solution for the task scheduling strategy and resource slicing is obtained, thereby optimizing the computing service performance.

[0199] The task scheduling and resource slicing method for low-orbit satellite networks provided by the present invention can be implemented and used when satellite network communication service providers deploy satellite networks and provide user services, such as Figure 3 The structure diagram of the present invention is shown in FIG.

[0200] During the satellite networking process, the low-orbit satellites of the satellite network communication service provider are operating normally in orbit to form a computing network. Figure 3 The method shown in the figure constructs a low-orbit satellite computing network. In this networking method, each low-orbit satellite functions as a ground-orbit computing node and a low-orbit satellite resource slicing node, capable of distributed task scheduling, resource allocation, and task processing. Specifically, the platform's network controller is used to perceive the network link topology and implement network management and control capabilities, while the resource manager primarily performs end-user computing service acquisition, task scheduling, and resource slicing.

[0201] When providing user services, the end user presses Figure 3 The network communication method shown in the figure sends a service request to the low-orbit satellite of the satellite network communication service provider, and then the low-orbit satellite computing network is connected according to Figure 2The task scheduling and resource slicing processes shown implement task scheduling and resource slicing, and handle compute-intensive, latency-sensitive, and other user-initiated services. Specifically, on small timescales, the system schedules tasks at a high frequency. When the deadline for a larger timescale is reached, task scheduling decisions and resource slicing decisions are mutually input and converge through iteration to obtain the optimal decision for both. This ensures efficient use of low-orbit satellite resources while ensuring service performance requirements and user service quality.

[0202] The present invention also provides a task scheduling and resource slicing system for low-orbit satellite networks, which, when executed, implements the steps of the task scheduling and resource slicing method for low-orbit satellite networks, such as Figure 3 As shown, including:

[0203] End users initiate service requests. End users include any single device or device cluster that deploys network services using distributed resource slices on low-orbit satellites. These devices include not only land-based devices but also ocean-based and airborne devices.

[0204] The low-orbit satellite network, comprised of satellite nodes, exhibits a distributed structure and features task scheduling and resource slicing mechanisms. Each satellite node can handle computing services requested by end users, as well as perform task scheduling and resource slicing operations from the network. Each satellite can schedule tasks in seconds and slice resources in minutes, achieving efficient resource allocation. Furthermore, because the low-orbit satellite network covers end users worldwide, any satellite in the network can immediately and autonomously respond with service processing regardless of the end user's request. This distributed management and control network architecture and capabilities ensure the dynamic autonomy of the distributed network of a giant low-orbit constellation. Through distributed resource slicing, the satellite network can flexibly adjust resource allocation, maximize overall network service capabilities, and improve global processing efficiency and resource utilization.

[0205] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the task scheduling and resource slicing method for low-orbit satellite networks are implemented.

[0206] Corresponding to the above method, the present invention also provides a device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0207] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0208] In summary, the present invention provides a task scheduling and resource slicing method and system for low-orbit satellite networks, including: a low-orbit satellite network receives a service request initiated by a terminal user; obtains resource information of each satellite, computing task status information, etc.; constructs a small-time-scale low-orbit satellite network task scheduling mechanism based on reinforcement learning, uses a Markov decision process to achieve optimal scheduling of computing tasks, and performs reasonable scheduling based on the actual energy loss and time delay of task processing, thereby solving the problems of mismatch between computing task service requirements caused by high dynamics of satellites in traditional task processing methods, and resource waste caused by mismatch between static resources of low-orbit satellites and computing task requirements, so as to minimize network energy consumption and task processing delays and improve task processing performance; at the same time, obtains the current resource allocation set of each satellite and constructs a satellite resource allocation system. Source slicing utility function; a resource slicing mechanism is constructed based on a heuristic algorithm, and the candidate solutions of each satellite resource slice are regarded as particles with the same charge. In one iteration, the fitness value of each candidate solution is calculated according to the utility function for evaluation. The candidate solution with a higher fitness value receives a higher charge, and the candidate solution with a lower fitness value receives a lower charge; the candidate solution is updated according to the optimal fitness value and the worst fitness value of the entire satellite, and multiple iterations are performed to obtain the optimal solution and generate a resource slicing strategy; based on the heuristic algorithm, distributed resource slicing can flexibly adjust resource allocation to adapt to the high-speed movement of satellites and the rapid changes in computing tasks, maximize the overall network service capabilities, and at the same time, reduce communication delays, increase data transmission speeds, reduce system energy consumption, and improve global processing efficiency and resource utilization.

[0209] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0210] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0211] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0212] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A task scheduling and resource slicing method for low-orbit satellite networks, characterized in that: The method is performed in a low-orbit satellite network, wherein the low-orbit satellite network has a distributed structure, and comprises the following steps: Receive service requests initiated by end users; Obtaining a current resource status set of each satellite; the resource status set includes at least a task generation status, a communication status between each satellite, and a communication status between each satellite and the ground; A task scheduling mechanism is constructed based on reinforcement learning. The state space is constructed based on the task generation state, data queue state, and communication state between satellites and between satellites and the ground. The action space is constructed based on the task scheduling performed by each satellite. The reward function is constructed by minimizing the task processing cost. The task scheduling mechanism is trained using experience replay to generate a corresponding task scheduling strategy based on the current state. Obtain the current resource allocation set of each satellite and construct the satellite resource slice utility function; A resource slicing mechanism is constructed based on a heuristic algorithm. Candidate solutions for resource slicing on each satellite are considered to have the same charge. In one iteration, the fitness value of each candidate solution is calculated according to the utility function for evaluation. Candidate solutions with higher fitness values ​​receive higher charges, while those with lower fitness values ​​receive lower charges. Candidate solutions are updated based on the optimal and worst fitness values ​​of the entire satellite. In multiple iterations, the interactions between charges are simulated so that the charges are concentrated near the optimal solution to obtain the optimal solution and generate a resource slicing strategy. Performing task scheduling and resource slicing on each satellite according to the task scheduling strategy and the resource slicing strategy, and feeding back results to the end user after the task is completed; The set of candidate solutions for the satellite resource slice is defined as: ； in, The calculation formula is: ； in, Represents the set of candidate solutions for resource slices of all satellites; Represents a candidate solution Communication resource slice ratio; Represents a candidate solution The computing resource slice ratio; Represents a candidate solution For the The size of the resource when slicing; The utility function is defined as: ； in, Calculated by the following equation: ； in, Represents a candidate solution Utility value of Represents a computing task The weight of ; Indicates satellite Computational tasks Generate status; Represents a computing task Whether from satellite Dispatching to satellite Status; Represents a computing task The data size; Indicates the computing resource slice ratio; Indicates satellite Total computing power; Indicates satellite to satellite The transmission rate.

2. The task scheduling and resource slicing method for low-orbit satellite networks according to claim 1, characterized in that: Calculating the fitness value of each candidate solution according to the utility function for evaluation, further comprising: The calculation formula for the position of the optimal fitness value obtained by the charged particle corresponding to each candidate solution at any time is: ； in, Indicates time Dimensions Particles The location of the best fitness value obtained; represents the utility function; Indicates time The location of the global optimal fitness value; Indicates time Candidate solutions for ; Indicates time Candidate solutions For the The size of the resource when slicing.

3. The task scheduling and resource slicing method for low-orbit satellite networks according to claim 1, characterized in that: The method further comprises: The Coulomb constant is calculated based on the current number of iterations and the preset total number of iterations. The calculation formula is: ； in, represents the Coulomb constant; Indicates the initial value; represents a constant; represents the current iteration number; Indicates the total number of preset iterations; The Coulomb force between particles is calculated based on the Coulomb constant, and the calculation formula is: ； in, Indicates time Dimensions Particles Acting on particles Coulomb force; represents the Coulomb constant; Represents particles Charge; Represents particles Charge; Representation Dimension Particles The location of the fitness value; Indicates time particle For the The size of the resource when slicing; Represents particles and particles The Euclidean distance between represents a very small positive constant; Calculate the total Coulomb force on each particle using the following formula: ； in, Indicates time Dimensions Particles The sum of the Coulomb forces from other particles; represents a uniform random number in [0,1]; Indicates time Dimensions Particles Acting on particles Coulomb force.

4. The task scheduling and resource slicing method for low-orbit satellite networks according to claim 3, characterized in that: The method further comprises: The electric field of the particle is calculated by dividing the total Coulomb force on the particle by the charge of the particle. The calculation formula is: ； in, Indicates time Dimensions Particles The electric field; Indicates time Dimensions Particles The sum of the Coulomb forces from other particles; Represents particles Charge; According to Newton's second law, the acceleration of the particle is calculated as follows: ； in, Indicates time Dimensions Particles acceleration; Represents particles Charge; Indicates time Dimensions Particles The electric field; Represents particles quality.

5. The task scheduling and resource slicing method for low-orbit satellite networks according to claim 4, characterized in that: In each iteration, the updated expressions of the particle's velocity and position are: ； ； in, Indicates time Dimensions Particles speed; represents a uniform random number in [0,1]; Indicates time Dimensions Particles acceleration; Indicates time Dimensions Particles location.

6. The task scheduling and resource slicing method for low-orbit satellite networks according to claim 1, characterized in that: Update candidate solutions based on the optimal fitness value and the worst fitness value of the entire satellite, and also include: The charge function of the particle is defined as: ； in, Indicates time particle Charge; Indicates time particle Utility value of represents the optimal fitness value; represents the worst fitness value; The calculation formulas of the optimal fitness value and the worst fitness value are respectively: ； ； in, Indicates time particle Utility value of According to the charge function, the charge update expression of the particle is: ； in, Represents the updated particle of charge.

7. A task scheduling and resource slicing system for low-orbit satellite networks, characterized in that: When the system is executed, the steps of the task scheduling and resource slicing method for a low-orbit satellite network according to any one of claims 1 to 6 are implemented, and the system includes: End users, used to initiate service requests; A low-orbit satellite network is composed of satellite nodes and is provided with a task scheduling mechanism and a resource slicing mechanism. The mechanism is used to generate a task scheduling strategy and a resource slicing strategy according to the state of the low-orbit satellite network, perform task scheduling and resource slicing on the satellite nodes, and feedback the results to the end user after the task processing is completed.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images