A task scheduling method for unified sky-ground network

Abstract

This invention discloses a task scheduling method for a unified sky-ground network. The method comprises the following steps: constructing a task-oriented communication scenario within the unified sky-ground network; determining the target problem for task scheduling and modeling it as a Markov decision process; constructing a policy-value neural network architecture based on multi-agent proximal policy optimization; training the constructed policy-value neural network architecture to obtain a trained policy-value neural network; dynamically updating information age based on the trained policy-value neural network and interacting with the real environment; and applying policies based on real-time feedback. This method effectively improves scheduling efficiency and optimizes task allocation by combining a reinforcement learning framework with network architecture characteristics and time-varying constraints.

Description

Technical Field

[0001] The present invention relates to the field of communications, and in particular to a task scheduling method oriented to a unified system sky-ground network. Background Art

[0002] In a unified, integrated sky-ground network, where spectrum resources are shared across multiple layers of access points, task scheduling presents numerous challenges. Access points only have limited access to information within their coverage area, making it difficult to understand the global state and thus making global optimization extremely difficult. Furthermore, spectrum allocation is subject to spatial and temporal interference and must be performed under dynamically changing constraints, further complicating resource allocation. More critically, as the number of users and channels increases, the action space of access points grows exponentially, posing extremely high computational complexity to the design and optimization of scheduling strategies. These challenges make it difficult for traditional optimization algorithms to adapt to the dynamically changing network environment and the global coordination requirements of multiple access points. Therefore, existing technologies have significant limitations in solving task scheduling problems in integrated sky-ground networks, and an innovative solution that can effectively address these complexities and challenges is urgently needed. Summary of the Invention

[0003] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a task scheduling method for a unified system sky-ground network, which can effectively improve scheduling efficiency and optimize task allocation by combining a reinforcement learning framework with network architecture characteristics and time-varying constraints.

[0004] The object of the present invention is achieved through the following technical solution: a task scheduling method for a unified system sky-ground network, comprising the following steps:

[0005] Build a mission-oriented communication scenario in a unified sky-ground network;

[0006] Determine the target problem of task scheduling and model the target problem as a Markov decision process;

[0007] Build a policy-value neural network architecture based on multi-agent proximal policy optimization;

[0008] Train the constructed strategy-value neural network architecture to obtain a trained strategy-value neural network, and dynamically update the information age based on the interaction between the trained strategy-value neural network and the real environment. , and apply strategies based on real-time feedback.

[0009] The beneficial effects of the present invention are: based on the multi-agent proximal strategy optimization algorithm, the present invention can target network architecture characteristics and time-varying constraints, combine with the reinforcement learning framework, effectively improve scheduling efficiency and optimize task allocation, reduce algorithm complexity, and provide a new theoretical basis and technical method for this research field. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of the method of the present invention;

[0011] Figure 2 This is a schematic diagram of the unified system sky-ground network scenario;

[0012] Figure 3 Schematic diagram of the task scheduling architecture based on multi-agent proximal strategy optimization. DETAILED DESCRIPTION

[0013] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0014] like Figure 1 As shown in the figure, a task scheduling method for a unified sky-ground network is proposed. The method is based on multi-agent proximal strategy optimization. The task scheduling architecture is as follows: Figure 3 As shown, the following steps are included:

[0015] Build a mission-oriented communication scenario in a unified sky-ground network;

[0016] like Figure 2 As shown, in this scenario, considering U ground users, a communication satellite, M A drone, N Ground base stations, a total of K Access points provide services to users. For the sake of index clarity, the communication satellite index is defined as k = 0, the drone index is , the base station index is Determine the target problem of task scheduling and model the target problem as a Markov decision process;

[0017] The scheduling goal is to minimize the user's Age of Information (AoI) and the energy consumption of the access point.

[0018] First, the problem is modeled as a Markov Decision Process (MDP), which consists of the following four parts:

[0019] Action: In time t Time access point k The action vector is defined as ,in express The first ( u,p ) elements. Specifically: Indicates access point k Resource Block p In time t Assigned to users u ,and Indicates an unassigned channel. Each resource block is a channel, and the total number of channels is defined as P .

[0020] Status: Access Point k The state space Including information age and transmission record vector Two key components:

[0021] Information age: Information age measures the freshness of information received by users. For a given user u , in time t When the information age Defined as:

[0022]

[0023] in: t is the current time, The timestamp of the last time the information was updated or received, indicating when the information was last updated.

[0024] Transmission record vector: Due to the propagation delay between the satellite and the UAV, the mission data packet cannot reach the user end immediately. In order to monitor the transmission process of the mission data packet, an access point is defined k With users u The transmission record vector between is: ,in: represents the set of natural numbers, Indicates access point k With users u When the access point k Send the task data package to the user U hour, The corresponding element of will be set to 1 and incremented by 1 at each time step until it reaches Once you arrive , the element will be reset to 0. The d Elements The update steps are summarized as follows:

[0025]

[0026] State transfer: In a unified sky-ground network, whether the state transfers depends on whether the user successfully receives the data packet. Based on the transmission record vector, the user's information age transfer can be expressed as:

[0027]

[0028] Among them: I is an indicator function that satisfies The condition is 1, otherwise it is 0. In the first case, when any base station is a user u Provide services, will be reset to 1; in the second case, if the user u is not served by any base station and does not receive service data packets from drones or satellites, then Increment by 1, that is ; In the third case, if no base station is serving the user u service, but successfully received data packets from drones or satellites, The smallest one between the current information age of the device itself and the information age of the received data packet is selected as the new information age.

[0029] Reward: The reward function combines the current information age and energy consumption and is expressed as:

[0030]

[0031] in, U , P , K Represents the total number of users, total number of channels and total number of access points, Indicates access point k Serve u The energy required to be consumed.

[0032] Based on the above Markov decision process modeling, users u The average information age is defined as:

[0033]

[0034] in, represents the mathematical expectation. u The average energy expenditure is defined as:

[0035]

[0036] Combining the two, the overall objective function can be written as:

[0037]

[0038] in, and are the weights of information age and energy consumption respectively. The joint optimization problem is established as:

[0039]

[0040]

[0041]

[0042] in, k and For two different access points, and Access Point k and and users u Formula (9) limits the resource block to be used by at most one user, and formula (10) takes into account the time and space interference in the network. Together, they ensure the rationality of task scheduling.

[0043] To solve this problem, an architecture based on Multi-Agent Proximal Policy Optimization (MAPPO) is proposed.

[0044] Policy network (Actor, also called action network): The policy network defines the policy , the parameters are , combined and recorded as The network is used to optimize access points k Action distribution In order to realize resource allocation under the condition of mutual blindness, the Softmax function is used to reconstruct the actions of the access points into ,in, Indicates access point k Channel p Assign to user u , 0 means no channel is allocated. To further mitigate the impact of spatiotemporal constraints, the input state dimension is filtered based on access point coverage: specifically, satellites only consider the transmission record vectors of users they cover, while drones and base stations include the information age and transmission record vectors of users they cover. The loss function is defined as:

[0045]

[0046] in, and Represent the current strategy and the original strategy respectively, and the advantage function is expressed as , is the set threshold, Indicates that for the input value x , and the given upper and lower limits min and max , the clip function will x Restricted to [ min, max ] range; if x If it exceeds the range, it will be truncated to the boundary value; min Represents the minimum function;

[0047] Value Network (Critic): The value network has parameters And by using the value function ( is the total state space, is a real number domain) to estimate the expected cumulative reward to evaluate the strategy . When performing an action Afterwards, the access point received an immediate reward and move to the next state The input of the value network includes the information age of all users, and the goal is to minimize The mean square error between the cumulative reward and the target. The loss function is expressed as:

[0048]

[0049] in, Indicates access point k The state-value function of express In the value function The value below.

[0050] Train the constructed strategy-value neural network architecture to obtain a trained strategy-value neural network, and dynamically update the information age based on the interaction between the trained strategy-value neural network and the real environment. , and apply strategies based on real-time feedback.

[0051] The algorithm implementation mainly consists of two parts:

[0052] 1) Offline training: It can be divided into two parts: environment simulation and offline training.

[0053] Environmental simulation: Based on the current state and actions , transfer to the next state , and calculate the average information age and average energy consumption according to formula (5) and formula (6). Using the current state ,action and global state The immediate reward is calculated by formula (4).

[0054] Offline training: At each time step, store the experience tuple according to the simulated environment ,in: represents the reward vector, Represents the probability distribution of actions. By extracting data from the experience pool, the policy network and value network are trained using formulas (11) and (12).

[0055] 2) Online application: Use the offline trained policy network to interact with the real environment and dynamically update the information age , and apply existing strategies based on real-time feedback:

[0056] Each access point obtains real-time environmental status based on local observations, including real-time information age and transmission record vector , the trained action network is input to generate its own action decision, and resource blocks are dynamically allocated to the covered users (that is, each access point chooses whether to use the resource block and which resource block to use to provide services to the user based on the input information), thereby realizing task scheduling.

[0057] Specifically, the trained policy network interacts with a real unified sky-ground network environment, bringing the offline trained policy into real-time service scheduling scenarios for real-time testing. Each access point (satellite, drone, base station) obtains the state of the environment based on its local observations and generates its own action decisions through the trained actor network, dynamically allocating resource blocks to the covered users. According to formula (3), real-time update: when the base station is the user u The age is reset to 1 when service is provided; it increments by 1 when there is no base station service and no data packets are received from satellites or drones; and when a data packet is received from a drone or satellite, the smaller of the current age and the age of the received packet is selected. A real-time feedback mechanism evaluates the effectiveness of the policy by monitoring the age of all users and the energy consumption of all access points. The smaller the weighted sum of the two, the more effective the policy. The system dynamically adjusts resource allocation based on these metrics to ensure optimal overall performance while meeting coverage constraints and avoiding spatial and temporal interference.

[0058] The trained policy network is directly applied to real-time scheduling decisions, determining at each time step which resource block each access point uses to serve which user. By optimizing the objective function (Equation 7), the system achieves a balance between the freshness of user information and access point energy consumption.

[0059] By considering the spatiotemporal interference constraints caused by the ripple effect (Formula 10), this scheduling method can effectively avoid channel conflicts and adjust the scheduling according to different task requirements. (Information Age) and The weight of the system energy consumption is calculated to achieve flexible task priority management. In addition, through dynamic resource allocation, the method adapts to network changes and ensures the effective execution of time-sensitive tasks in the complex and changing unified system sky-ground network environment.

[0060] The foregoing description is a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Instead, the present invention can be used in other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A task scheduling method for a unified sky-ground network, characterized by: The following steps are involved: Build a mission-oriented communication scenario in a unified sky-ground network; Determine the target problem of task scheduling and model the target problem as a Markov decision process; The objective problem of the task scheduling is to minimize the user's information age and the energy consumption of the access point; Modeling the target problem as a Markov decision process includes the following four parts: (1) Action: in time t Time access point k The action vector is defined as ,in express The first ( u,p ) elements, Indicates access point k Resource Block p In time t Assigned to users u ,and Indicates access point k Resource Block p In time t Not assigned to user u ; Each resource block is a channel, and the total number of channels is defined as P ; (2) Status: Access Point k The state space Including information age and transmission record vector Two key components: Information age: Information age measures the freshness of information received by users. For a given user u , in time t When the information age Defined as: in, t is the current time, It is the timestamp of the last time the information was updated or received, indicating the time when the information was last updated; Transmission record vector: Due to the propagation delay between the satellite and the UAV, the mission data packet cannot reach the user end immediately. In order to monitor the transmission process of the mission data packet, an access point is defined k With users u The transmission record vector between is: ,in: represents the set of natural numbers, Indicates access point k With users u The propagation delay between When the access point k Send the task data package to the user U hour, The corresponding element of will be set to 1 and incremented by 1 at each time step until it reaches Once you arrive , the element will be reset to 0, The d Elements The update steps are summarized as follows: (3) State transfer: In a unified sky-ground network, whether the state transfers depends on whether the user successfully receives the data packet. Based on the transmission record vector, the user's information age transfer is expressed as: in, is an indicator function that satisfies 1 if the condition is met, 0 otherwise; In the first case, when any base station is a user u Provide services, will be reset to 1; In the second case, if the user u is not served by any base station and does not receive service data packets from drones or satellites, then Increment by 1, that is ; In the third case, if no base station is serving the user u service, but successfully received data packets from drones or satellites, The smallest of the current information age and the information age of the received data packet is selected as the new information age; (4) Reward: The reward function combines the current information age and energy consumption, expressed as: in, U , P , K Represents the total number of users, total number of channels and total number of access points, Indicates access point k Serve u The amount of energy required; Based on the above Markov decision process modeling, users u The average information age is defined as: in, represents the mathematical expectation, user u The average energy expenditure is defined as: The overall objective function is: in, and are the weights of information age and energy consumption respectively, and the joint optimization problem is established as: in, k and For two different access points, and Access Point k and and users u The propagation delay between them, formula (9) limits the resource block to be used by at most one user, and formula (10) takes into account the time and space interference in the network. The two together ensure the rationality of task scheduling; Build a policy-value neural network architecture based on multi-agent proximal policy optimization; Train the constructed strategy-value neural network architecture to obtain a trained strategy-value neural network, and dynamically update the information age based on the interaction between the trained strategy-value neural network and the real environment. , and apply strategies based on real-time feedback.

2. The method for task scheduling in a unified sky-ground network according to claim 1, characterized in that: The mission-oriented communication scenario includes a communication satellite, M A drone, N Ground base stations, a total of K An access point provides services to users, and the access point is a communication satellite, a drone or a ground base station. ; For the sake of index clarity, the communication satellite index is defined as k = 0, the drone index is , the base station index is .

3. The method for task scheduling in a unified sky-ground network according to claim 1, characterized in that: The strategy-value neural network architecture based on multi-agent proximal strategy optimization includes: Policy Network: The policy network defines the policy , the parameters are , combined and recorded as ; This network is used to optimize access points k Action distribution To achieve resource allocation under mutually blind conditions, the Softmax function is used to reconstruct the actions of the access points into ,in, Indicates access point k Channel p Assign to user u ,0 means no channel is allocated; To mitigate the impact of spatiotemporal constraints, the input state dimension is filtered based on access point coverage: satellites only consider the transmission record vectors of covered users, while drones and base stations include the information age and transmission record vectors of covered users. The loss function is defined as: in, and Represent the current strategy and the original strategy respectively, and the advantage function is expressed as , is the set threshold, Indicates that for the input value x , and the given upper and lower limits min and max , the clip function will x Restricted to [ min, max ] range; if x If it exceeds the range, it will be truncated to the boundary value; Represents the minimum function; Value Network: The value network has parameters And by using the value function Estimating expected cumulative rewards to evaluate strategies , is the total state space, is the field of real numbers; In performing actions Afterwards, the access point received an immediate reward and move to the next state , the input of the value network includes the information age of all users, and the goal is to minimize The mean square error between the cumulative reward and the target, the loss function is expressed as: in, Indicates access point k The state-value function of express In the value function The value below.

4. The method for task scheduling in a unified sky-ground network according to claim 3, characterized in that: The training of the constructed policy-value neural network architecture to obtain a trained policy-value neural network architecture includes: Environmental simulation: Based on the current state and actions , transfer to the next state , and calculate the average information age and average energy consumption according to formula (5) and formula (6), using the current state ,action and global state Calculate the immediate reward using formula (4); Offline training: At each time step, store the experience tuple according to the simulated environment ,in: represents the reward vector, Represents the logarithmic probability distribution of actions. By extracting data from the experience pool, the policy network and value network are trained using formulas (11) and (12).

5. The method for task scheduling in a unified sky-ground network according to claim 4, characterized in that: The trained policy-value neural network interacts with the real environment, dynamically updates the information age, and applies the policy based on real-time feedback, using only the trained policy network, including: Each access point obtains real-time environmental status based on local observations, including real-time information age and transmission record vector , input the trained policy network to generate its own action decision, dynamically allocate resource blocks for the covered users, and thus realize task scheduling.

Images