Edge task scheduling method based on multi-agent proximal strategy optimization for offline training

Abstract

The present invention discloses an edge task scheduling method based on multi-agent proximal strategy optimization offline training. First, UAVs are deployed in an area without ground infrastructure, and a temporary network is formed with low-orbit satellites. The task scheduling cost is defined according to the task scheduling delay and energy consumption; then the task scheduling cost minimization problem is modeled as a discrete partially observable Markov decision process, and the task scheduling problem in the network is trained offline using a multi-agent proximal strategy optimization algorithm through offline simulation of environmental data to obtain an approximately optimal scheduling strategy model; finally, the model parameters obtained by offline training are pre-deployed in the UAV, and when a computing task request is received, the model is directly called for online reasoning to obtain the optimal scheduling strategy. The present invention effectively avoids the high computing load brought by online training, reduces the task scheduling cost, and improves the overall scheduling efficiency of the system.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing technology, and specifically relates to an edge task scheduling method based on multi-agent proximal strategy optimization offline training. Background Art

[0002] After natural disasters such as earthquakes, floods, and landslides, ground communication infrastructure is often severely damaged, disrupting traditional network services and creating significant challenges for information transmission and rescue coordination in disaster-stricken areas. In recent years, collaborative air-space-ground networks consisting of unmanned aerial vehicles (UAVs) and low-Earth orbit (LEO) satellites, combined with mobile edge computing (MEC) technology, have become a key tool for building emergency communication systems. In post-disaster emergency networks, ground users' requests for assistance and data processing tasks require timely responses. Therefore, efficient scheduling of computing tasks within these temporary emergency communication systems is crucial.

[0003] However, existing methods typically only consider either service deployment or load balancing for task scheduling, rarely considering both simultaneously. This single scheduling mechanism leads to uneven task distribution, node overload, and difficulty adapting to the actual distribution of network services after a disaster, reducing the overall system scheduling efficiency. For example, the applicant, Hangzhou Dianzi University, disclosed a task scheduling method based on service deployment updates in its patent application document "Task Scheduling Method and System Based on Service Deployment Updates" (filing date: October 31, 2022, application number: 202211347782.7, application publication number: CN115696588 A, the content of which is still citationable). This task scheduling method based on service deployment updates has the following shortcomings: it does not consider load balancing during task scheduling. When computing tasks surge in the network, uneven task distribution can lead to node overload. Moreover, if a user cannot find an edge server deploying the required service within communication range, the task may not be offloaded. The patent application document "A Task Load Balancing Scheduling Method Based on Cloud-Edge-End Collaboration" (filing date: May 15, 2024, application number: 202410602801.9, application publication number: CN118394520 A, the content of which is still citationable) discloses a task scheduling method that considers load balancing. This load balancing-based task scheduling method has the following shortcomings: It assumes that all required services are available on the edge server during task scheduling, which is inconsistent with the actual situation of limited edge server storage capacity and reduces the overall scheduling efficiency of the system.

[0004] With the development of artificial intelligence (AI) technology, some researchers have begun using reinforcement learning algorithms to solve task scheduling problems. However, existing reinforcement learning algorithms often rely on online training, which results in high computational energy consumption and significant computational latency. This not only increases the cost of task scheduling but also severely impacts the real-time response capabilities of emergency communication networks. For example, the applicant, Jilin University, disclosed a MEC task scheduling method based on proximal policy optimization in its patent application, "Method for MEC Task Scheduling in Internet of Vehicles Based on Proximal Policy Optimization" (filing date: August 18, 2023, application number: 202311041910.X, publication number: CN117014832 A, the content of which is still citationable). This MEC task scheduling method based on proximal policy optimization suffers from the following shortcomings: Proximal policy optimization is optimized through online training, which results in additional energy consumption and latency, increasing task scheduling costs. Furthermore, the online training method consumes computing resources, placing a high computational load on the MEC system.

[0005] In summary, the existing methods for post-disaster emergency network edge task scheduling have the problems of low overall system scheduling efficiency, and the online algorithm training increases the computational load, resulting in high task scheduling costs. Summary of the Invention

[0006] Purpose of the Invention: Existing methods for post-disaster emergency network edge task scheduling only consider service deployment or load control mechanisms, leading to uneven node loads or difficulty adapting to the distribution of post-disaster network services, resulting in low overall system scheduling efficiency. Furthermore, when using reinforcement learning algorithms to solve task scheduling problems, online training methods are used, which results in a high computational load, increases the cost of task scheduling, and seriously affects the real-time response capability of emergency communication networks. This invention proposes an edge task scheduling method based on offline training optimized by a multi-agent proximal strategy to address the shortcomings of the above technologies.

[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0008] A method for scheduling edge tasks based on multi-agent proximal strategy optimization for offline training includes the following steps:

[0009] S1: Deploy drones in an area without ground infrastructure and establish a temporary network with low-orbit satellites. Incorporate service deployment restrictions and load control mechanisms. In this temporary network, the edge server on each drone deploys only a subset of services, while the edge server on the low-orbit satellite deploys all services.

[0010] S2: Define task scheduling delays separately and task scheduling energy consumption , and define the task scheduling cost according to the task scheduling delay and the task scheduling energy consumption; the task scheduling delay Including task upload time , task migration time and task calculation time ;Task scheduling energy consumption Including calculation of energy consumption , migration energy consumption and hovering energy consumption ;

[0011] S3: The task scheduling cost minimization problem is modeled as a discrete partially observable Markov decision process, where the discrete partially observable Markov decision process includes a local observation state, a global state, an action space, and a reward function.

[0012] S4: Through offline simulation of environmental data, a multi-agent proximal strategy optimization algorithm is used to perform offline training on the task scheduling problem in the network to obtain an approximately optimal scheduling strategy model; the offline simulation environmental data is obtained through the digital twin simulation platform and contains state-action-reward tuples , as data input for training the multi-agent proximal policy optimization algorithm;

[0013] S5: The model parameters obtained through offline training are pre-deployed in the UAV. When a computing task request is received, the model is directly called for online inference to obtain the optimal scheduling strategy.

[0014] Furthermore, the step S1 is specifically as follows:

[0015] S1-1: Deploy drones in areas without ground infrastructure and establish a temporary network with low-orbit satellites:

[0016] Deployment in areas without ground infrastructure UAVs, and with Low Earth Orbit Satellites and Ground users form a temporary network; drones , low-orbit satellites and users The sets of these three types of nodes are represented as 、 and Among them, drones and low-orbit satellites are equipped with edge servers to provide computing services to ground users

[0017] S1-2: Add service deployment restrictions and load control mechanisms:

[0018] make Indicates the number of services in the network, Represents the set of services in the network. Each drone-mounted edge server only deploys a part of the services, i.e. ,in, Indicates drone The edge servers deployed on low-orbit satellites deploy a collection of services; while the edge servers deployed on low-orbit satellites deploy all services , the computing tasks generated by ground users can only be dispatched to edge servers with corresponding services for processing. For example, if the ground user generates an image recognition computing task, the computing task needs to be dispatched to the edge server deployed with the image recognition service for processing;

[0019] Add a load control mechanism to the edge server on the drone. Received to ground users Computational tasks After that, it is first stored in the buffer; if the drone Have services that handle the computing tasks, and the drone Check that the occupancy rate of its own buffer is lower than the load threshold When , the calculation task will be directly Process locally; otherwise, the load control mechanism is triggered to transfer the computing task Forwarded to the load buffer below the load threshold And have a drone that can handle the corresponding service of the computing task If the drone No matching drones found within the communication range , then the calculation task Forwarded to low-orbit satellite On processing.

[0020] Furthermore, the step S2 is specifically as follows:

[0021] S2-1: Define task scheduling delays separately and task scheduling energy consumption :

[0022] The UAV task processing mode is divided into three types: local processing, forwarding to other UAVs for processing, and forwarding to low-orbit satellites for processing; therefore, task scheduling delays Including task upload time , task migration time and task calculation time :

[0023] (1)

[0024] drones Task scheduling energy consumption Including calculation of energy consumption , migration energy consumption and hovering energy consumption :

[0025] (2)

[0026] S2-2: Delay according to task scheduling and task scheduling energy consumption Defining Scheduling Costs :

[0027] The goal of the task scheduling problem is to find the optimal task scheduling strategy , minimize the scheduling cost; the cost of task scheduling Defined as:

[0028] (3)

[0029] in, and Represent the weight factors of task scheduling delay and UAV task scheduling energy consumption, and satisfy ; and They represent the maximum task scheduling delay and the maximum UAV task scheduling energy consumption respectively.

[0030] Furthermore, the step S3 is specifically as follows:

[0031] Each drone Considered as an agent, a discrete partially observable Markov decision process consists of the following elements:

[0032] (1) Local observation state: in the time slot , the agent collects environmental information within the communication range and observes the local state is defined as:

[0033] (4)

[0034] in, For drones The computational load; For drones A collection of service deployments; For drones The task queue status; For drones The channel status; For drones Remaining computing resources;

[0035] (2) Global state: in time slot , global state Contains all drone local observation status , Global unprocessed task queue and global computing task request information , global state is defined as:

[0036] (5)

[0037] (3) Action space: in time slots , use action Indicates drone Action space, action is defined as:

[0038] (6)

[0039] Constraints: and ;

[0040] in, is the target node, which is a drone or a low-orbit satellite; Indicates the target node Deployed the corresponding services, Indicates the target node Buffer occupancy Below load threshold ;

[0041] Reward function: The goal of task scheduling is to find the optimal task scheduling strategy , minimize the scheduling cost, so in time slot , the reward obtained by the agent after interacting with the environment is defined as:

[0042] (7)

[0043] in, is the resource overlimit penalty coefficient, ranging from 0 to 1; For drones Assigned to computing tasks computing resources, For drones Available computing resources.

[0044] Furthermore, the step S4 is specifically as follows:

[0045] S4-1: Simulate environmental data through offline simulation:

[0046] First, a digital twin simulation platform is constructed. Based on the temporary network topology structure of UAV-LOW-Earth orbit satellite-ground users defined in steps S1 and S2, the simulation parameters are clarified, including the UAV Taiwan, low-orbit satellite terrestrial users Services deployed per drone , load threshold , task generation rate During the simulation process, the computing task flow generated dynamically by ground users is simulated, and the computing tasks are generated using the Poisson distribution model. , each task carries the service type requirement and records the task upload time , migration time and calculation time , in order to obtain the task scheduling delay Data; Recording Drone Task calculation energy consumption , migration energy consumption and hovering energy consumption , in order to obtain the drone Task scheduling energy consumption data; at the same time, in each time slot Generate local observation state , including drones Computational load , drones Service deployment collection , drones Task queue status , drones Channel status , drones Remaining computing resources ; Construct state-action-reward tuple , as data input for training the multi-agent proximal policy optimization algorithm;

[0047] S4-2: Use the multi-agent proximal policy optimization algorithm to perform offline training on the task scheduling problem in the network and obtain an approximately optimal scheduling policy model:

[0048] Take the state-action-reward tuple obtained in step S4-1 As input, the task scheduling problem in the network is trained offline, and the loss function of the multi-agent proximal policy optimization algorithm is Defined as:

[0049] (8)

[0050] in, represents the action probability ratio, is the scheduling policy parameter, is the advantage function, is the clipping range parameter, ∈[0.1,0.3]; Represents the probability ratio of the action Crop, and the crop range is [ ]; advantage function is defined as:

[0051] (9)

[0052] in, Time slot The value function when Time slot The value function when is the discount coefficient, ranging from 0.9 to 0.99.

[0053] Furthermore, the step S5 is specifically as follows:

[0054] S5-1: Pre-deploy the model parameters obtained from offline training on the drone:

[0055] Compress the approximate optimal scheduling strategy model obtained through offline training in S4; and pre-deploy the compressed model parameters to the edge server of the UAV;

[0056] S5-2: When a computing task request is received, the model is directly called for online reasoning to obtain the optimal scheduling strategy:

[0057] When a drone Receiving ground users Computational tasks When , first obtain the local observation state , including drones Computational load , drones Service deployment collection , drones Task queue status , drones Channel status , drones Remaining computing resources ; Then the local observation state Input the deployed policy model for forward reasoning and output the optimal scheduling policy .

[0058] The advantages and technical effects of the present invention are as follows:

[0059] The present invention first performs task scheduling by simultaneously considering service deployment conditions and incorporating a load control mechanism, thereby evenly distributing tasks in accordance with the actual service deployment of the edge network, avoiding node task overload, and improving the overall scheduling efficiency of the system. Secondly, through offline simulation of environmental data, the multi-agent proximal policy optimization algorithm is used to perform offline training on the task scheduling problem in the network to obtain a near-optimal scheduling policy model, avoiding the additional computing load, energy consumption, and delay brought about by the online training process of the multi-agent proximal policy optimization algorithm, reducing the task scheduling cost, and enabling the system to quickly respond to task requests. Finally, after simulation verification, when the UAV computing power is set to 0.6 (billion cycles / second), the method of the present invention reduces the task scheduling cost by approximately 40.3% compared with the online trained multi-agent proximal policy optimization (MAPPO) method.

[0060] In summary, the present invention effectively avoids the high computational load of online training, reduces task scheduling costs, and improves the overall task scheduling efficiency and real-time response capability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is an overall flow chart of an embodiment of the present invention;

[0062] Figure 2 is a network architecture diagram of an embodiment of the present invention;

[0063] Figure 3 Schematic diagram of a load control mechanism implemented by an edge server mounted on a drone according to an embodiment of the present invention;

[0064] Figure 4 Schematic diagram of three task scheduling processing modes for a UAV according to an embodiment of the present invention;

[0065] Figure 5 This is a comparison chart of task scheduling cost simulation results using the method provided by the present invention and the online trained MAPPO method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0067] This embodiment proposes a method for scheduling edge tasks based on multi-agent proximal strategy optimization for offline training. The overall flow chart is as follows: Figure 1 As shown, the following steps are included:

[0068] S1: Deploy drones in an area without ground infrastructure and establish a temporary network with low-orbit satellites. Incorporate service deployment restrictions and load control mechanisms. In this temporary network, the edge servers on each drone deploy only a portion of the services, while the edge servers on the low-orbit satellite deploy all services. The specific steps are as follows:

[0069] S1-1: Deploy drones in areas without ground infrastructure and establish a temporary network with low-orbit satellites:

[0070] like Figure 2 As shown, deployment in areas without ground infrastructure UAVs, and with Low Earth Orbit Satellites and Ground users form a temporary network. In this embodiment, =8, =3, =12; UAV , low-orbit satellites and users The sets of these three types of nodes are represented as 、 and Among them, drones and low-orbit satellites are equipped with edge servers to provide computing services to ground users;

[0071] S1-2: Add service deployment restrictions and load control mechanisms:

[0072] make Indicates the number of services in the network. In this embodiment, =10; Represents the set of services in the network. Each drone-mounted edge server only deploys a part of the services, i.e. ,in, Indicates drone The edge servers deployed on low-orbit satellites deploy a collection of services; while the edge servers deployed on low-orbit satellites deploy all services , the computing tasks generated by ground users can only be dispatched to edge servers with corresponding services for processing. For example, if the ground user generates an image recognition computing task, the computing task needs to be dispatched to the edge server deployed with the image recognition service for processing;

[0073] like Figure 3 As shown, a load control mechanism is added to the edge server carried by the drone. Received to ground users Computational tasks After that, it is first stored in the buffer; if the drone Have services that handle the computing tasks, and the drone Check that the occupancy rate of its own buffer is lower than the load threshold When , the calculation task will be directly Process locally; otherwise, the load control mechanism is triggered to transfer the computing task Forwarded to the load buffer below the load threshold And have a drone that can handle the corresponding service of the computing task If the drone No matching drones found within the communication range , then the calculation task Forwarded to low-orbit satellite In this embodiment, the load threshold is taken as .

[0074] S2: Define task scheduling delays separately and task scheduling energy consumption , and delay according to task scheduling and task scheduling energy consumption Defining task scheduling costs ; The task scheduling delay Including task upload time , task migration time and task calculation time ;Task scheduling energy consumption Including calculation of energy consumption , migration energy consumption and hovering energy consumption , the specific steps are as follows:

[0075] S2-1: Define task scheduling delays separately and task scheduling energy consumption :

[0076] like Figure 4 As shown in the figure, the UAV task processing mode is divided into three types: local processing, forwarding to other UAVs for processing, and forwarding to low-orbit satellites for processing; therefore, the task scheduling delay Including task upload time , task migration time and task calculation time :

[0077] (1)

[0078] drones Task scheduling energy consumption Including calculation of energy consumption , migration energy consumption and hovering energy consumption :

[0079] (2)

[0080] S2-2: Delay according to task scheduling and task scheduling energy consumption Defining Scheduling Costs :

[0081] The goal of the task scheduling problem is to find the optimal task scheduling strategy , minimize the scheduling cost; the cost of task scheduling Defined as:

[0082] (3)

[0083] in, and Represent the weight factors of task scheduling delay and UAV task scheduling energy consumption, and satisfy ; In this embodiment, =0.4 and =0.6; and They represent the maximum task scheduling delay and the maximum UAV task scheduling energy consumption respectively.

[0084] S3: Model the task scheduling cost minimization problem as a discrete partially observable Markov decision process. The discrete partially observable Markov decision process includes local observation state, global state, action space, and reward function. The specific steps are as follows:

[0085] Each drone Considered as an agent, a discrete partially observable Markov decision process consists of the following elements:

[0086] (1) Local observation state: in the time slot , the agent collects environmental information within the communication range and observes the local state is defined as:

[0087] (4)

[0088] in, For drones The computational load; For drones A collection of service deployments; For drones The task queue status; For drones The channel status; For drones Remaining computing resources;

[0089] (2) Global state: in time slot , global state Contains all drone local observation status , Global unprocessed task queue and global computing task request information , global state is defined as:

[0090] (5)

[0091] (3) Action space: in time slots , use action Indicates drone Action space, action is defined as:

[0092] (6)

[0093] Constraints: and ;

[0094] in, is the target node, which is a drone or a low-orbit satellite; Indicates the target node Deployed the corresponding services, Indicates the target node Buffer occupancy Below load threshold ;

[0095] Reward function: The goal of task scheduling is to find the optimal task scheduling strategy , minimize the scheduling cost, so in time slot , the reward obtained by the agent after interacting with the environment is defined as:

[0096] (7)

[0097] in, The resource overlimit penalty coefficient is between 0 and 1. =0.1; For drones Assigned to computing tasks computing resources, For drones Available computing resources.

[0098] S4: Through offline simulation of environmental data, a multi-agent proximal strategy optimization algorithm is used to perform offline training on the task scheduling problem in the network to obtain an approximately optimal scheduling strategy model; the offline simulation environmental data is obtained through the digital twin simulation platform and contains state-action-reward tuples , as the data input for training the multi-agent proximal policy optimization algorithm, the specific steps are as follows:

[0099] S4-1: Simulate environmental data through offline simulation:

[0100] First, a digital twin simulation platform is constructed. Based on the temporary network topology structure of UAV-LOW-Earth orbit satellite-ground users defined in steps S1 and S2, the simulation parameters are clarified, including the UAV Taiwan, low-orbit satellite terrestrial users Services deployed per drone , load threshold =80%, task generation rate tasks / second; During the simulation process, the computing task flow generated dynamically by ground users is simulated, and the computing tasks are generated using the Poisson distribution model. , each task carries the service type requirement and records the task upload time , migration time and calculation time , in order to obtain the task scheduling delay Data; Recording Drone Task calculation energy consumption , migration energy consumption and hovering energy consumption , in order to obtain the drone Task scheduling energy consumption data; at the same time, in each time slot Generate local observation state , including drones Computational load , drones Service deployment collection , drones Task queue status , drones Channel status , drones Remaining computing resources ; Construct state-action-reward tuple , as data input for training the multi-agent proximal policy optimization algorithm;

[0101] S4-2: Use the multi-agent proximal policy optimization algorithm to perform offline training on the task scheduling problem in the network and obtain an approximately optimal scheduling policy model:

[0102] Take the state-action-reward tuple obtained in step S4-1 As input, the task scheduling problem in the network is trained offline, and the loss function of the multi-agent proximal policy optimization algorithm is Defined as:

[0103] (8)

[0104] in, represents the action probability ratio, is the scheduling policy parameter, is the advantage function, is the clipping range parameter, ∈[0.1,0.3], in this embodiment, =0.2; Represents the probability ratio of the action Crop, and the crop range is [ ]; advantage function is defined as:

[0105] (9)

[0106] in, Time slot The value function when Time slot The value function when is the discount coefficient, which ranges from 0.9 to 0.99. =0.94; This embodiment passes Rounds of offline simulation iterative training to minimize the loss function and scheduling costs , maximize the reward ;in, Indicates the number of offline simulation iterations. In this embodiment, =200.

[0107] S5: The model parameters obtained through offline training are pre-deployed in the UAV. When a computing task request is received, the model is directly called for online inference to obtain the optimal scheduling strategy. The specific steps are as follows:

[0108] S5-1: Pre-deploy the model parameters obtained from offline training on the drone:

[0109] The near-optimal scheduling strategy model obtained through offline training in S4 is compressed to reduce the space occupied by the model. The compressed model parameters are pre-deployed to the UAV's edge server so that the UAV can quickly call it when receiving computing task requests from ground users.

[0110] S5-2: When a computing task request is received, the model is directly called for online reasoning to obtain the optimal scheduling strategy:

[0111] When a drone Receiving ground users Computational tasks When , first obtain the local observation state , including drones Computational load , drones Service deployment collection , drones Task queue status , drones Channel status , drones Remaining computing resources ; Then the local observation state Input the deployed policy model for forward reasoning and output the optimal scheduling policy .

[0112] The comparison results of task scheduling cost simulation using the method provided by the present invention and the MAPPO method for online training are as follows: Figure 5 This embodiment is simulated in Matlab R2021a, using a computer with 32GB of memory and an x64-based Intel Core i5-12400F CPU processor. The specific parameters of all simulations in this embodiment are listed in Table 1.

[0113] Table 1 Simulation parameters

[0114] ,

[0115] from Figure 5 Simulation results show that the task scheduling cost of the method provided by the present invention is significantly reduced as the UAV's computing power increases. This is because the increased UAV computing power allows the system to process more tasks within the same scheduling cycle, thereby reducing the load overflow caused by task accumulation and reducing task processing costs. The online training MAPPO method performs relatively poorly. When the UAV computing power is set to 0.6 (billion cycles per second), the method provided by the present invention reduces the task scheduling cost by approximately 40.3% compared to the online training MAPPO method.

[0116] In summary, the present invention effectively avoids the high computational load brought by online training, reduces task scheduling costs, and improves the overall task scheduling efficiency of the system.

[0117] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for a person skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions claimed to be protected by the present invention.

Claims

1. A method for scheduling edge tasks based on multi-agent proximal strategy optimization for offline training, characterized in that: The method comprises the following steps: S1: Deploy drones in areas without ground infrastructure, establish a temporary network with low-orbit satellites, add service deployment restrictions, and the computing tasks generated by ground users can only be dispatched to edge servers with corresponding services for processing; add a load control mechanism to the edge servers carried by drones. Received to ground users Computational tasks After that, it is first stored in the buffer; if the drone Have services that handle the computing tasks, and the drone Check that the occupancy rate of its own buffer is lower than the load threshold When , the calculation task will be directly Process locally; Otherwise, the load control mechanism is triggered and the computing task Forwarded to the load buffer below the load threshold And have a drone that can handle the corresponding service of the computing task If the drone No matching drones found within the communication range , then the calculation task Forwarded to low-orbit satellite In the temporary network, the edge server carried by each drone deploys only a part of the services, and the edge server carried by the low-orbit satellite deploys all services. S2: Define task scheduling delays separately and task scheduling energy consumption , and the cost of task scheduling is calculated based on task scheduling delay and task scheduling energy consumption Defined as: in, and Represent the weight factors of task scheduling delay and UAV task scheduling energy consumption, and satisfy ; and They represent the maximum task scheduling delay and the maximum UAV task scheduling energy consumption respectively; represents a collection of drones, represents a collection of ground users; the task scheduling delay Including task upload time , task migration time and task calculation time ;Task scheduling energy consumption Including calculation of energy consumption , migration energy consumption and hovering energy consumption ; S3: The task scheduling cost minimization problem is modeled as a discrete partially observable Markov decision process, where the discrete partially observable Markov decision process includes a local observation state, a global state, an action space, and a reward function. S4: Through offline simulation of environmental data, a multi-agent proximal strategy optimization algorithm is used to perform offline training on the task scheduling problem in the network to obtain an approximately optimal scheduling strategy model; the offline simulation environmental data is obtained through the digital twin simulation platform and contains state-action-reward tuples , as data input for training the multi-agent proximal policy optimization algorithm; S5: The model parameters obtained through offline training are pre-deployed in the UAV. When a computing task request is received, the model is directly called for online inference to obtain the optimal scheduling strategy.

2. The edge task scheduling method based on multi-agent proximal strategy optimization offline training according to claim 1 is characterized in that: The step S1 is specifically as follows: S1-1: Deploy drones in areas without ground infrastructure and establish a temporary network with low-orbit satellites: Deployment in areas without ground infrastructure UAVs, and with Low Earth Orbit Satellites and Ground users form a temporary network; drones , low-orbit satellites and users The sets of these three types of nodes are represented as 、 and Among them, drones and low-orbit satellites are equipped with edge servers to provide computing services to ground users; S1-2: Add service deployment restrictions and load control mechanisms: make Indicates the number of services in the network, Represents the set of services in the network. Each drone-mounted edge server only deploys a part of the services, i.e. ,in, Indicates drone The edge servers deployed on low-orbit satellites deploy a collection of services; while the edge servers deployed on low-orbit satellites deploy all services .

3. The edge task scheduling method based on multi-agent proximal strategy optimization offline training according to claim 1 is characterized in that: The step S2 is specifically as follows: S2-1: Define task scheduling delays separately and task scheduling energy consumption : The UAV task processing mode is divided into three types: local processing, forwarding to other UAVs for processing, and forwarding to low-orbit satellites for processing; therefore, task scheduling delays Including task upload time , task migration time and task calculation time : (1) drones Task scheduling energy consumption Including calculation of energy consumption , migration energy consumption and hovering energy consumption : (2) Then, according to the task scheduling delay and task scheduling energy consumption Defining Scheduling Costs .

4. The edge task scheduling method based on multi-agent proximal strategy optimization offline training according to claim 1 is characterized in that: In step S3, the task scheduling cost minimization problem is modeled as a discrete partially observable Markov decision process, which includes a local observation state, a global state, an action space, and a reward function, as follows: Each drone Considered as an agent, a discrete partially observable Markov decision process consists of the following elements: (1) Local observation state: in the time slot , the agent collects environmental information within the communication range and observes the local state is defined as: (3) in, For drones The computational load; For drones A collection of service deployments; For drones The task queue status; For drones The channel status; For drones Remaining computing resources; (2) Global state: in time slot , global state Contains all drone local observation status , Global unprocessed task queue and global computing task request information , global state is defined as: (4) (3) Action space: in time slots , use action Indicates drone Action space, action is defined as: (5) Constraints: and ; in, is the target node, which is a drone or a low-orbit satellite; Indicates the target node Deployed the corresponding services, Indicates the target node Buffer occupancy Below load threshold ; Reward function: The goal of task scheduling is to find the optimal task scheduling strategy , minimize the scheduling cost, so in time slot , the reward obtained by the agent after interacting with the environment is defined as: (6) in, is the resource overlimit penalty coefficient, ranging from 0 to 1; For drones Assigned to computing tasks computing resources, For drones Available computing resources.

5. The edge task scheduling method based on multi-agent proximal strategy optimization offline training according to claim 1, characterized in that: The step S4 is specifically as follows: S4-1: Simulate environmental data through offline simulation: First, a digital twin simulation platform is constructed. Based on the temporary network topology structure of UAV-LOW-Earth orbit satellite-ground users defined in steps S1 and S2, the simulation parameters are clarified, including the UAV Taiwan, low-orbit satellite terrestrial users Services deployed per drone , load threshold , task generation rate During the simulation process, the computing task flow generated dynamically by ground users is simulated, and the computing tasks are generated using the Poisson distribution model. , each task carries the service type requirement and records the task upload time , migration time and calculation time , in order to obtain the task scheduling delay Data; Recording Drone Task calculation energy consumption , migration energy consumption and hovering energy consumption , in order to obtain the drone Task scheduling energy consumption data; at the same time, in each time slot Generate local observation state , including drones Computational load , drones Service deployment collection , drones Task queue status , drones Channel status , drones Remaining computing resources ; Construct state-action-reward tuple , as data input for training the multi-agent proximal policy optimization algorithm; S4-2: Use the multi-agent proximal policy optimization algorithm to perform offline training on the task scheduling problem in the network and obtain an approximately optimal scheduling policy model: Take the state-action-reward tuple obtained in step S4-1 As input, the task scheduling problem in the network is trained offline, and the loss function of the multi-agent proximal policy optimization algorithm is Defined as: (7) in, represents the action probability ratio, is the scheduling policy parameter, is the advantage function, is the clipping range parameter, ∈[0.1,0.3]; Represents the probability ratio of the action Crop, and the crop range is [ ]; advantage function is defined as: (8) in, Time slot The value function when Time slot The value function when is the discount coefficient, ranging from 0.9 to 0.

99.

6. The edge task scheduling method based on multi-agent proximal strategy optimization offline training according to claim 1, characterized in that: The step S5 is specifically as follows: S5-1: Pre-deploy the model parameters obtained from offline training on the drone: Compress the approximate optimal scheduling strategy model obtained through offline training in S4; and pre-deploy the compressed model parameters to the edge server of the UAV; S5-2: When a computing task request is received, the model is directly called for online reasoning to obtain the optimal scheduling strategy: When a drone Receiving ground users Computational tasks When , first obtain the local observation state , including drones Computational load , drones Service deployment collection , drones Task queue status , drones Channel status , drones Remaining computing resources ; Then the local observation state Input the deployed policy model for forward reasoning and output the optimal scheduling policy .

Images