A satellite network intelligent resource scheduling method based on reinforcement learning

Abstract

The application discloses a kind of satellite network intelligent resource scheduling methods based on reinforcement learning, belong to satellite network resource scheduling technical field.The following steps are included: satellite resource information and on-board mission state information are collected;Determine the limiting condition, state, action and reward value function of algorithm according to the collected information;Send the relevant information obtained to the reinforcement learning action module to perform satellite and link selection operation and reward value function calculation;Satellite server forwards the task according to the action result.The present application considers the bandwidth required by on-board task, duration and other information, as well as satellite idle computing capacity, storage resources, inter-satellite link available bandwidth and other information, realizes the intelligent scheduling of ground user task or cloud data center task in inter-satellite, reduces the delay of inter-satellite forwarding, and improves the communication performance of on-board cloud data center.

Description

Technical Field

[0001] This invention belongs to the field of satellite network resource scheduling technology, and in particular relates to a satellite network intelligent resource scheduling method based on reinforcement learning. Background Technology

[0002] With the continuous development of next-generation low- and medium-Earth orbit (LEO) satellite constellation technology, satellite networks utilizing LEO satellite constellations can effectively overcome the problem of insufficient terrestrial network service capabilities. Compared to traditional terrestrial networks, satellite networks offer advantages such as global coverage, deployment unrestricted by geographical or terrain conditions, and strong system resilience. Therefore, an increasing number of users are accessing internet services through satellite networks. However, given the high complexity, high dynamism, and resource scarcity inherent in LEO satellite networks, achieving efficient and intelligent resource scheduling to allocate limited resources across different services has become a critical challenge.

[0003] Currently, classic resource scheduling algorithms for satellites include First-In-First-Out (FIFO), Shortest Job First (SJF), Priority Queues, and Weighted Fair Queues. However, these traditional algorithms incur a significant proportion of the total resource overhead when scheduling user services in complex and resource-scarce environments. Therefore, combining them with more efficient AI technologies has become a practical solution. For example, genetic algorithms can be used to select tasks for scheduling based on dynamically updated information, calculate fitness values ​​using total task weights and task completion times as parameters, and then use an optimized genetic algorithm to find the optimal solution among different scheduling schemes. Alternatively, the Hungarian algorithm can be used to calculate the optimal resource allocation scheme that satisfies the maximum QoE criterion, thereby improving user experience quality while ensuring resource utilization. While such algorithms improve upon the shortcomings of classic algorithms, they still have limitations in scheduling specific bandwidth service resources, multi-satellite joint scheduling, and the needs of special services. To address these issues, reinforcement learning technology, which has emerged in recent years, has better intelligent solution capabilities for solving complex optimization decision problems. Faced with the challenges of resource scheduling in low- and medium-Earth orbit satellite networks, characterized by multi-satellite collaborative operation, all-time and all-space interconnection, and massive user base, which presents challenges such as high complexity, low dynamism, resource scarcity, and a large amount of demand information, the most reasonable resource scheduling method can be achieved through intelligent decision-making using reinforcement learning algorithms. Summary of the Invention

[0004] To address the existing technical problems, this invention provides a satellite network intelligent resource scheduling method based on reinforcement learning, which can not only adapt to complex and dynamic satellite network systems, but also reduce the latency problem of resource scheduling.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A satellite network intelligent resource scheduling method based on reinforcement learning includes the following steps:

[0007] Step 1: Collect satellite resource information and on-board mission status information;

[0008] Step 2: Determine the constraints, states, actions, and reward value function based on the collected information;

[0009] Step 3: Send the collected information to the reinforcement learning action module to perform satellite and link selection operations and calculate the reward value function;

[0010] Step 4: The satellite server forwards the task based on the action results.

[0011] Furthermore, the specific method for step 1 is as follows:

[0012] Step 1a: Deploy a satellite resource sensing module on the satellite to obtain satellite resource status information, including satellite number (SN). i Available computing resources (SCPU) i With available memory resources SM i ;

[0013] Step 1b: Deploy a mission information collection module on the satellite to collect mission requests from ground users or sub-mission requests from ground core cloud nodes that need to be processed by the satellite, including the bandwidth (WBW) required for each sub-mission. k and task duration τ;

[0014] Step 1c, design the visual constraint flags for the source and destination satellite nodes:

[0015] FLAG = α*β*δ

[0016]

[0017]

[0018]

[0019] Where α, β, and δ represent the elevation angle, azimuth angle, and viewing angle of the binary star link establishment under Earth's obstruction, respectively. min α max These represent the minimum and maximum pitch angles of the link terminal, respectively; β min β max These represent the minimum and maximum azimuth angles of the link terminal, respectively; δ max To determine the maximum possible angle of view for establishing a binary star link under Earth's obstruction.

[0020] Furthermore, the specific method for step 2 is as follows:

[0021] Step 2a: Based on the Actor-Critic (AC) algorithm, establish multiple agents that interact with their environment, with each agent containing a copy of the environment; construct an Asynchronous Advantage ActorCritic (A3C) agent based on actor-critic, where actors are used to generate actions and critics are used to evaluate and critique current policies. By processing the rewards obtained from the environment, the critics adopt the TD-error method.

[0022] Step 2b, construct the network state set at time t as follows:

[0023] S n ={SLA,SBW OD SCPU D SM D WBW k ,τ};

[0024] Where SLA stands for link angle and SBW stands for link angle. OD Available bandwidth resources for the link between source and destination satellite nodes;

[0025] Step 2c, construct action set A = {N} i} contains all actions that the decision-making algorithm can choose;

[0026] Step 2d: Construct the behavioral action strategy for the reinforcement learning system:

[0027] π(s,a)=π(a|s)=p(A=a|S=s)

[0028]

[0029] Where S is the set of state spaces and A is the set of action spaces;

[0030] Step 2e: Construct the reward system for the reinforcement learning system. The reward is represented by the γ-discounted reward, which represents the cumulative reward obtained from the current state to the future state. The future reward value is multiplied by a discount factor γ, where γ is a real number between 0 and 1, representing the magnitude of the future reward's influence. t This means that the further back in time the reward has been, the smaller its impact becomes, as shown below:

[0031]

[0032] Where R represents the immediate reward from the environment, s0 is the initial state of the agent, and s t It is the state of the agent at time t;

[0033] Step 2f: Construct the state transition probabilities of the reinforcement learning system:

[0034]

[0035] The state transition probability represents the probability of performing action a. n Later, the network environment changed, and S... n Transfer to S n+1 The probability of;

[0036] Step 2g: Construct the reward function for the reinforcement learning system: Execute action a n Afterwards, the environment will provide a reward value r. n To measure the quality of action execution; for each step, the reward value r n It is determined by the time required to process the entire subtask; the reward function is set as follows:

[0037]

[0038] Among them, T d Indicates processing latency, determined by the response speed of the wavelength routing device; T w The propagation delay is represented by dividing the interstellar distance by the speed of light; T t The transmission delay is represented by T, which is obtained by dividing the task bandwidth by the communication rate. p Indicates queuing delay;

[0039] Let f o It is the threshold of the reward function f. If f o If the value is greater than or equal to f, the action is considered valid and the reward value is 1; otherwise, the action is considered invalid and the reward value is -1.

[0040] Furthermore, the specific method for step 3 is as follows:

[0041] Step 3a involves transitioning from the reinforcement learning state to a set of Markov decision processes. The current state of the environment depends only on the state at the previous time step and is independent of previous states. Therefore, the state-value function simplifies to:

[0042]

[0043] Where p is the state transition probability;

[0044] Step 3b: Use the state-action value function to evaluate the quality of choosing an action in a given state, i.e.:

[0045]

[0046] Among them, Q *(s,a) means that in state s, action a is chosen and in the subsequent policy selection process, the optimal policy is used to obtain a larger cumulative reward.

[0047] The state-value function and the state-action-value function satisfy the following relationship:

[0048] V * (s)=Q * (s,a)

[0049] Step 3c, the optimal action strategy is:

[0050]

[0051] In step 3d, when updating the action strategy, the strategy is adjusted and optimized and actions are selected based on the state value function, rather than the action selection is based on the immediate reward function.

[0052] Step 3e: The neural network for the AC algorithm is a classic fully connected structure. The actor network has a 100*6 matrix as input, three hidden layers, and outputs a 100-dimensional column vector; the critic network outputs the reward value.

[0053] Rewards are accumulated through discounts. To determine this; the task is broken down into 5 sub-tasks, therefore the AC algorithm needs to be run 5 times; the goal of reinforcement learning is to maximize the cumulative reward, i.e.

[0054] Step 3f: The Star-Cloud Data Center calculates the time corresponding to each subtask for each node based on the current environment, selects the node and reward value {i,f} corresponding to the minimum time of the k-th subtask, and uploads it to the global administrator; the global administrator then makes further selections, ensuring that the nodes corresponding to each selected sub-file are different, thus ensuring that all sub-files are stored simultaneously.

[0055] The beneficial effects of this invention are as follows:

[0056] 1. The resource scheduling method provided by this invention treats all data as a whole, fuses the data through a neural network algorithm, and incorporates historical data of state parameters as input neurons into the neural network algorithm, thereby ensuring the time of satellite network missions and enabling it to adapt to more complex systems.

[0057] 2. The method of the present invention comprehensively considers information such as the bandwidth and duration required by the on-board mission, as well as information such as the satellite's idle computing power, storage resources, and available bandwidth of inter-satellite links, to realize intelligent scheduling of ground user missions or cloud data center missions between satellites, reduce the latency of inter-satellite forwarding, and improve the communication performance of on-board cloud data centers. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the network framework provided by the present invention;

[0059] Figure 2 Here is a diagram of the algorithm framework;

[0060] Figure 3 A diagram illustrating reinforcement learning;

[0061] Figure 4 This is a diagram of the state transition process;

[0062] Figure 5 This is the overall algorithm flowchart. Detailed Implementation

[0063] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] A satellite network intelligent resource scheduling method based on reinforcement learning includes the following steps:

[0065] Step 1: Collect satellite resource information and on-board mission status information;

[0066] Step 2: Determine the algorithm's constraints, states, actions, and reward function based on the collected information;

[0067] Step 3: Send the obtained relevant information to the reinforcement learning action module to perform satellite and link selection operations and calculate the reward value function;

[0068] Step 4: The satellite server forwards the task based on the action results.

[0069] Step 1 is as follows:

[0070] Step 1a: Place a satellite resource sensing module on the satellite to acquire satellite resource status information, mainly including the satellite number (SN). i Available computing resources (SCPU) i With available memory resources SM i wait.

[0071] Step 1b: Deploy a mission information collection module on the satellite to collect mission requests from ground users or sub-mission requests from ground core cloud nodes that need to be processed by the satellite. This mainly includes the bandwidth (WBW) required for each sub-mission. k And the task duration τ.

[0072] Step 1c, and These are the source satellite node and destination satellite node numbers, respectively. (SBW) OD The available bandwidth resources for the link between source and destination satellite nodes. Design SBW.OD Constraint Formula. Due to the high-speed motion of the satellite and real-time changes in topology, the link angle also changes constantly. The link angle is represented by SLA. FLAG represents the visible constraints between the source and destination satellite nodes, mainly including link constraints and orbital constraints. The specific judgment function of FLAG is as follows:

[0073] FLAG = α*β*δ

[0074]

[0075]

[0076]

[0077] Where, α min α max These represent the minimum and maximum pitch angles of the link terminal, respectively; β min β max These represent the minimum and maximum azimuth angles of the link terminal, respectively; δ max To determine the maximum possible angle of view for establishing a binary star link under Earth's obstruction.

[0078] Step 2 is as follows:

[0079] Step 2a: The AC algorithm reduces the impact of task continuity on the final result and can asynchronously update the global model. It utilizes multiple agents interacting with their respective environments, each agent containing a copy of the environment. A specific A3C agent is based on an actor-critic, where actors are used to generate actions, and critics are used to evaluate and critique current policies by processing rewards obtained from the environment. The TD-error method is typically used as the critic to evaluate the actors' performance.

[0080] Step 2b: The state of the network environment is the basis for decision-making. To arrive at a decision, it is necessary to obtain network environment information. Therefore, the set of network states at time t is:

[0081] S n ={SLA,SBW OD SCPU D SM D WBW k ,τ};

[0082] Step 2c: The action set contains all actions that the decision algorithm can choose. Corresponding to the decision variables in the dynamic programming model above, the corresponding action set A is A = {N}. i};

[0083] Step 2d: Besides the agent and its environment, the main elements of a reinforcement learning system include the action policy, environmental reward, and state-value function. The policy, which guides the agent in selecting actions within a specific environment, is the core of reinforcement learning. It is usually represented by S*A→[0,1]. Another policy representation is S→A, indicating that in state s, the probability of action A being executed is 1, and the probability of other actions being executed is 0. Therefore, the policy is defined as:

[0084] π(s,a)=π(a|s)=p(A=a|S=s)

[0085]

[0086] Where S is the set of state spaces and A is the set of action spaces. The learning objective of reinforcement learning is to continuously update and optimize the policy during the iterative process of interaction with the environment, so that the agent can maximize long-term reward benefits according to the policy.

[0087] Step 2e: The reward is an evaluation of the environmental state or the agent's actions. It can be used to guide the agent in policy optimization and updates, and can also serve as a learning objective for the agent. This reward value must be objective and unchangeable by the agent. The goal of reinforcement learning is to maximize the total reward value obtained. The reward function is the environment's immediate evaluation of the current state and action, a short-term signal; while the state value function represents the agent's expected cumulative reward, a long-term representation. The agent may receive a lower immediate reward but potentially a higher cumulative reward.

[0088] State-value functions are typically represented using gamma-discounted rewards. Gamma-discounted rewards represent the cumulative reward from the current state to future states, where the future reward value is multiplied by a discount factor γ. γ is a real number between 0 and 1, representing the magnitude of the future reward's influence. t This means that the further back in time the reward has been, the smaller its impact becomes, as shown below:

[0089]

[0090] Where R represents the immediate reward from the environment, s0 is the initial state of the agent, and s t It is the state of the agent at time t.

[0091] Step 2f: The state transition probability refers to the probability of state transition when performing action a. n Later, the network environment changed, and S... n Transfer to S n+1The probability of state transition. Since queue tasks change rapidly, the network state is considered constant; therefore, the state at the next moment is determined solely by the state at the current moment. Thus, the state transition probability is expressed as:

[0092]

[0093] Step 2g: Reward function: Execute action a n Afterwards, the environment will provide a reward value r. n To measure the quality of action execution. For each step, the reward function r n It is determined by the time required to process the entire subtask, and the reward function is set as follows:

[0094]

[0095] Among them, T d Indicates processing latency, determined by the response speed of the wavelength routing device; T w The propagation delay is represented by dividing the interstellar distance by the speed of light; T t The transmission delay is represented by T, which is obtained by dividing the task bandwidth by the communication rate. p This indicates the queuing delay. Let f o This is the threshold of the reward function f. If f o If the value is greater than or equal to f, the action is considered valid and the reward value is 1; otherwise, the action is considered invalid and the reward value is -1.

[0096] Step 3 is as follows:

[0097] Step 3a: Since the state transition in reinforcement learning is essentially a set of Markov decision processes, the current state of the environment depends only on the state at the previous moment and is independent of previous states. Therefore, the state-value function can be simplified to:

[0098]

[0099] Where p is the state transition probability.

[0100] Step 3b: Use the state-action value function to evaluate the quality of choosing an action in a given state, i.e.:

[0101]

[0102] Q * (s,a) represents choosing action a in state s, where the optimal policy in the subsequent policy selection process yields a larger cumulative reward. From the above analysis, we can see that the state-value function and the state-action-value function satisfy the following relationship:

[0103] V * (s)=Q *(s,a)

[0104] Step 3c, further, yields the optimal action strategy as follows:

[0105]

[0106] Step 3d: Based on the above analysis, the state-value function is a reward function that considers the long-term perspective and can serve as an optimization objective function in the process of the agent interacting and learning with the environment. Furthermore, when updating action policies, the policy adjustment and action selection should be based on the state-value function, rather than on the immediate reward function. This allows the agent to focus more on long-term cumulative gains.

[0107] Step 3e: The neural network for the AC algorithm is a classic fully connected structure. The Actor network has a 100*6 matrix as its input layer, three hidden layers, and a 100-dimensional column vector (softmax function) as its output. The Critic network outputs the reward value.

[0108] Rewards need to be accumulated through discounts. To determine this, the task is broken down into 5 sub-tasks, therefore the AC algorithm needs to be run 5 times. The goal of reinforcement learning is to maximize the cumulative reward, i.e.

[0109] Step 3f: The Star-Cloud Data Center calculates the time corresponding to each subtask for each node based on the current environment, selects the node and reward value {i,f} corresponding to the minimum time for calculating the k-th subtask, and uploads this information to the global administrator. The global administrator then makes further selections, ensuring that each selected sub-file corresponds to a different node, thus guaranteeing that all sub-files are stored simultaneously.

[0110] Here is a more specific example:

[0111] A reinforcement learning-based intelligent resource scheduling method for satellite networks. Figure 1 The diagram shown is a framework diagram of a satellite network. Figure 2 To reinforce the learning algorithm framework diagram, Figure 3 Reinforcement learning diagram, Figure 4 The state transition process diagram combines reinforcement learning algorithms with satellite network resource scheduling. Figure 5 The overall algorithm flowchart is shown below. Figure 5 As shown, it includes the following steps:

[0112] Step 1: Collect satellite resource information and on-board mission status information;

[0113] Step 2: Determine the algorithm's constraints, states, actions, and reward function based on the collected information;

[0114] Step 3: Send the obtained relevant information to the reinforcement learning action module to perform satellite and link selection operations and calculate the reward value function;

[0115] Step 4: The satellite server forwards the task based on the action results.

[0116] Step 1 is as follows:

[0117] Step 1a: Place a satellite resource sensing module on the satellite to acquire satellite resource status information, mainly including the satellite number (SN). i Available computing resources (SCPU) i With available memory resources SM i wait.

[0118] Step 1b: Deploy a mission information collection module on the satellite to collect mission requests from ground users or sub-mission requests from ground core cloud nodes that need to be processed by the satellite. This mainly includes the bandwidth (WBW) required for each sub-mission. k And the task duration τ.

[0119] Step 1c, and These are the source satellite node and destination satellite node numbers, respectively. (SBW) OD The available bandwidth resources for the link between source and destination satellite nodes. Design SBW. OD Constraint Formula. Due to the high-speed motion of the satellite and real-time changes in topology, the link angle also changes constantly. The link angle is represented by SLA. FLAG represents the visible constraints between the source and destination satellite nodes, mainly including link constraints and orbital constraints. The specific judgment function of FLAG is as follows:

[0120] FLAG = α*β*δ

[0121]

[0122]

[0123]

[0124] Where, α min α max These represent the minimum and maximum pitch angles of the link terminal, respectively; β min β max These represent the minimum and maximum azimuth angles of the link terminal, respectively; δ max To determine the maximum possible angle of view for establishing a binary star link under Earth's obstruction.

[0125] Step 2 is as follows:

[0126] Step 2a: The AC algorithm reduces the impact of task continuity on the final result and can asynchronously update the global model. It utilizes multiple agents interacting with their respective environments, each agent containing a copy of the environment. A specific A3C agent is based on an actor-critic, where actors are used to generate actions, and critics are used to evaluate and critique current policies by processing rewards obtained from the environment. The TD-error method is typically used as the critic to evaluate the actors' performance.

[0127] Step 2b: The state of the network environment is the basis for decision-making. To arrive at a decision, it is necessary to obtain network environment information. Therefore, the set of network states at time t is:

[0128] S n ={SLA,SBW OD SCPU D SM D WBW k ,τ};

[0129] Step 2c: The action set contains all actions that the decision algorithm can choose. Corresponding to the decision variables in the dynamic programming model above, the corresponding action set A is A = {N}. i};

[0130] Step 2d: Besides the agent and its environment, the main elements of a reinforcement learning system include the action policy, environmental reward, and state-value function. The policy, which guides the agent in selecting actions within a specific environment, is the core of reinforcement learning. It is usually represented by S*A→[0,1]. Another policy representation is S→A, indicating that in state s, the probability of action A being executed is 1, and the probability of other actions being executed is 0. Therefore, the policy is defined as:

[0131]

[0132] Where S is the set of state spaces and A is the set of action spaces. The learning objective of reinforcement learning is to continuously update and optimize the policy during the iterative process of interaction with the environment, so that the agent can maximize long-term reward benefits according to the policy.

[0133] Step 2e: The reward is an evaluation of the environmental state or the agent's actions. It can be used to guide the agent in policy optimization and updates, and can also serve as a learning objective for the agent. This reward value must be objective and unchangeable by the agent. The goal of reinforcement learning is to maximize the total reward value obtained. The reward function is the environment's immediate evaluation of the current state and action, a short-term signal; while the state value function represents the agent's expected cumulative reward, a long-term representation. The agent may receive a lower immediate reward but potentially a higher cumulative reward.

[0134] State-value functions are typically represented using gamma-discounted rewards. Gamma-discounted rewards represent the cumulative reward from the current state to future states, where the future reward value is multiplied by a discount factor γ. γ is a real number between 0 and 1, representing the magnitude of the future reward's influence. t This means that the further back in time the reward has been, the smaller its impact becomes, as shown below:

[0135]

[0136] Where R represents the immediate reward from the environment, s0 is the initial state of the agent, and s t It is the state of the agent at time t.

[0137] Step 2f: The state transition probability refers to the probability of state transition when performing action a. n Later, the network environment changed, and S... n Transfer to S n+1 The probability of state transition. Since queue tasks change rapidly, the network state is considered constant; therefore, the state at the next moment is determined solely by the state at the current moment. Thus, the state transition probability is expressed as:

[0138]

[0139] Step 2g: Reward function: Execute action a n Afterwards, the environment will provide a reward value r. n To measure the quality of action execution. For each step, the reward function r n It is determined by the time required to process the entire subtask, and the reward function is set as follows:

[0140]

[0141] Among them, T d Indicates processing latency, determined by the response speed of the wavelength routing device; T w The propagation delay is represented by dividing the interstellar distance by the speed of light; T t The transmission delay is represented by T, which is obtained by dividing the task bandwidth by the communication rate.p This indicates the queuing delay. Let f o This is the threshold of the reward function f. If f o If the value is greater than or equal to f, the action is considered valid and the reward value is 1; otherwise, the action is considered invalid and the reward value is -1.

[0142] Step 3 is as follows:

[0143] Step 3a: Since the state transition in reinforcement learning is essentially a set of Markov decision processes, the current state of the environment depends only on the state at the previous moment and is independent of previous states. Therefore, the state-value function can be simplified to:

[0144]

[0145] Where p is the state transition probability.

[0146] Step 3b: Use the state-action value function to evaluate the quality of choosing an action in a given state, i.e.:

[0147]

[0148] Q * (s,a) represents choosing action a in state s, where the optimal policy in the subsequent policy selection process yields a larger cumulative reward. From the above analysis, we can see that the state-value function and the state-action-value function satisfy the following relationship:

[0149] V * (s)=Q * (s,a)

[0150] Step 3c, further, yields the optimal action strategy as follows:

[0151]

[0152] Step 3d: Based on the above analysis, the state-value function is a reward function that considers the long-term perspective and can serve as an optimization objective function in the process of the agent interacting and learning with the environment. Furthermore, when updating action policies, the policy adjustment and action selection should be based on the state-value function, rather than on the immediate reward function. This allows the agent to focus more on long-term cumulative gains.

[0153] Step 3e: The neural network for the AC algorithm is a classic fully connected structure. The Actor network has a 100*6 matrix as its input layer, three hidden layers, and a 100-dimensional column vector (softmax function) as its output. The Critic network outputs the reward value.

[0154] Rewards need to be accumulated through discounts. To determine this, the task is broken down into 5 sub-tasks, therefore the AC algorithm needs to be run 5 times. The goal of reinforcement learning is to maximize the cumulative reward, i.e.

[0155] Step 3f: The Star-Cloud Data Center calculates the time corresponding to each subtask for each node based on the current environment, selects the node and reward value {i,f} corresponding to the minimum time for calculating the k-th subtask, and uploads this information to the global administrator. The global administrator then makes further selections, ensuring that each selected sub-file corresponds to a different node, thus guaranteeing that all sub-files are stored simultaneously.

[0156] In summary, this invention is primarily applied to spaceborne cloud data center scenarios. A spaceborne cloud data center consists of ground-based core cloud nodes and satellite edge computing nodes. The ground-based core cloud nodes possess several distributed computing centers, each composed of a large-scale, high-performance computer cluster, providing services such as model training, large-scale complex task processing, and data storage. The satellite edge computing nodes include several medium-Earth orbit (MEO) satellites, responsible for controlling low-Earth orbit (LEO) remote sensing satellites and receiving remote sensing data. They also possess cloud computing platforms with independent decision-making capabilities, enabling them to receive and execute tasks. This invention collects satellite resource information and onboard mission status information, and uses the AC reinforcement learning scheduling algorithm to allocate resources to appropriate satellite edge servers for execution based on the mission's requirements, achieving intelligent task forwarding between satellites and reducing communication latency.

[0157] The technical principles of the present invention have been described above with reference to specific embodiments. These descriptions are merely for explaining the principles of the invention and should not be construed as limiting the scope of protection of the invention in any way. Based on this explanation, those skilled in the art can conceive of other specific embodiments of the invention without creative effort, and these embodiments will all fall within the scope of protection of the present invention.

Claims

1. A satellite network intelligent resource scheduling method based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Collect satellite resource information and on-board mission status information; Step 2: Based on the AC algorithm, establish multiple agents that interact with their environment. Each agent contains a copy of the environment, constructing an A3C agent based on the actor-commentator model; determine the constraints, states, actions, and reward value functions based on the collected information. The behavioral action strategy is as follows: Where S is the set of state spaces and A is the set of action spaces; The reward is as follows: Where R represents the immediate reward for environmental feedback. This represents the initial state of the agent. γ represents the state of the agent at time t, and γ represents the discounted reward. The state transition probability is: State transition probability represents the probability of performing an action. Later, the network environment changed, from Transferred to The probability of; The reward function is: in, Indicates processing delay. Indicates the propagation delay. Indicates transmission delay. Indicates queuing delay; Step 3: Send the collected information to the reinforcement learning action module to perform satellite and link selection operations and calculate the reward value function; when updating the action policy, adjust and optimize the policy and select actions based on the state value function; Step 4: The satellite server forwards the task based on the action results.

2. The intelligent resource scheduling method for satellite networks based on reinforcement learning according to claim 1, characterized in that, The specific steps for step 1 are as follows: Step 1a: Deploy a satellite resource sensing module on the satellite to obtain satellite resource status information, including satellite number. Available computing resources With available memory resources ; Step 1b: Deploy a mission information collection module on the satellite to collect mission requests from ground users or sub-mission requests from ground core cloud nodes that need to be processed by the satellite, including the bandwidth required for each sub-mission. and task duration ; Step 1c: Design the visual constraints for the source and destination satellite nodes. : Wherein, α, β, and δ represent the elevation angle, azimuth angle, and viewing angle of the binary star link establishment under Earth's obstruction, respectively. , These represent the minimum and maximum pitch angles of the link terminal, respectively. , These represent the minimum and maximum azimuth angles of the link terminal, respectively. To determine the maximum possible angle of view for establishing a binary star link under Earth's obstruction.

3. The intelligent resource scheduling method for satellite networks based on reinforcement learning according to claim 2, characterized in that, The specific steps for step 2 are as follows: Step 2a: Based on the AC algorithm, establish multiple agents that interact with their own environment, with each agent containing a copy of the environment; construct an A3C agent based on actor-commentator, where actors are used to generate actions and commentators are used to evaluate and criticize the current policy. By processing the rewards obtained from the environment, the commentator adopts the TD-error method. Step 2b, construct the network state set at time t as follows: ； Where SLA stands for Link Angle. Available bandwidth resources for the link between source and destination satellite nodes; Step 2c, construct the action set It includes all actions that the decision-making algorithm can choose; Step 2d: Construct the behavioral action strategy for the reinforcement learning system: Where S is the set of state spaces and A is the set of action spaces; Step 2e: Construct the reward system for the reinforcement learning system. The reward is represented by the γ-discounted reward, which represents the cumulative reward obtained from the current state to future states. The future reward value is multiplied by a discount factor γ, where γ is a real number between 0 and 1, representing the magnitude of the future reward's influence. t This means that the further back in time the reward has been, the smaller its impact becomes, as shown below: Where R represents the immediate reward for environmental feedback. This represents the initial state of the agent. It is the state of the agent at time t; Step 2f: Construct the state transition probabilities of the reinforcement learning system: State transition probability represents the probability of performing an action. Later, the network environment changed, from Transferred to The probability of; Step 2g: Construct the reward function for the reinforcement learning system: execute actions Afterwards, the environment will provide a reward value. To measure the quality of action execution; for each step, a reward value is given. It is determined by the time required to process the entire subtask; the reward function is set as follows: in, This indicates the processing latency, which is determined by the response speed of the wavelength routing device. This represents the propagation delay, obtained by dividing the interstellar distance by the speed of light. The transmission delay is represented by dividing the task bandwidth by the communication rate. Indicates queuing delay; make It is a reward function The threshold, if If the action is successful, the action is considered valid and the reward value is 1; otherwise, the action is considered invalid and the reward value is -1.

4. The intelligent resource scheduling method for satellite networks based on reinforcement learning according to claim 3, characterized in that, The specific steps for step 3 are as follows: Step 3a involves transitioning from the reinforcement learning state to a set of Markov decision processes. The current state of the environment depends only on the state at the previous time step and is independent of previous states. Therefore, the state-value function simplifies to: Where p is the state transition probability; Step 3b: Use the state-action value function to evaluate the quality of choosing an action in a given state, i.e.: in, This means that if action a is chosen in state s, then in the subsequent policy selection process, the optimal policy will be used to obtain a larger cumulative reward. The state-value function and the state-action-value function satisfy the following relationship: Step 3c, the optimal action strategy is: Step 3d: When updating the action strategy, the strategy is adjusted and optimized and actions are selected based on the state value function, rather than the action selection is based on the immediate reward function. Step 3e: The neural network for the AC algorithm is a classic fully connected structure. The actor network has a 100*6 matrix as input, three hidden layers, and outputs a 100-dimensional column vector; the critic network outputs the reward value. Rewards are accumulated through discounts. To determine this; the task is broken down into 5 sub-tasks, therefore the AC algorithm needs to be run 5 times; the goal of reinforcement learning is to maximize the cumulative reward, i.e. ; Step 3f: The Nebula Data Center calculates the time corresponding to each subtask for each node based on the current environment, and selects the node and reward value corresponding to the minimum time for calculating the k-th subtask. The data is then uploaded to the global administrator, who then makes further selections. Each selected sub-file corresponds to a different node, ensuring that all sub-files are stored simultaneously.

Images