Multi-machine collaborative task scheduling method for aerospace network edge computing scene

Abstract

The invention provides a multi-machine collaborative task scheduling method for an aerospace network edge computing scene, and relates to the technical field of aerospace network edge computing. The unmanned aerial vehicle collects user tasks based on a pre-constructed task collection model (based on a Markov decision process and multi-agent reinforcement learning modeling) to maximize the task collection rate of the current time slot; when the user task is unloaded to the target unmanned aerial vehicle, task scheduling optimization is carried out based on a depth deterministic strategy gradient algorithm, the task is scheduled to other unmanned aerial vehicles in the multi-unmanned aerial vehicle cluster, or a satellite communicating with the multi-unmanned aerial vehicle cluster is scheduled; therefore, other unmanned aerial vehicles and / or satellites in the multi-unmanned aerial vehicle cluster and the target unmanned aerial vehicle are subjected to task cooperative calculation, and the task completion time delay of the user is minimized. According to the invention, the problem of user distribution density difference is solved, the problem of unmanned aerial vehicle resource limitation is overcome, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of aerospace network edge computing, and in particular to a multi-machine collaborative task scheduling method for aerospace network edge computing scenarios. Background Art

[0002] With the development of space network technology, edge computing, as an emerging computing model, has been widely used in various scenarios. As an important mobile node, drones play an important role in various application scenarios, especially in edge computing scenarios. In the space network edge computing scenario, drones, as important computing nodes, undertake a large number of data processing tasks.

[0003] In existing technologies, drones can collect and process data through their own sensors and communication modules, while providing computing services to other devices. However, the capabilities of a single drone are limited, and its coverage and computing resources are insufficient to meet the needs of large areas or high-density users.

[0004] In addition, due to the uneven distribution of users, drones in some areas may be overloaded, while drones in other areas may be lightly loaded. For example, a large number of users may gather in certain areas, causing drones in these areas to be overloaded, while drones in other areas may be underloaded. This load imbalance will not only waste the processing power of some drones, but also increase the delay of data transmission, thereby affecting the overall network service quality and user experience. Summary of the invention

[0005] The purpose of this application is to provide a multi-machine collaborative task scheduling method for aerospace network edge computing scenarios to alleviate the above-mentioned technical problems existing in the prior art.

[0006] In a first aspect, the present invention provides a multi-machine collaborative task scheduling method for an aerospace network edge computing scenario, the method comprising: In response to a user task initiated by a ground user, a target drone in a multi-drone cluster is matched, and the target drone collects the user task based on a pre-built task collection model; wherein the pre-built task collection model is a model for maximizing the task collection rate based on Markov decision process and multi-agent reinforcement learning modeling; When the user task is offloaded to the target UAV, other UAVs in the multi-UAV cluster are scheduled through a pre-constructed task delay model, or a satellite communicating with the multi-UAV cluster is scheduled, so that other UAVs and / or satellites in the multi-UAV cluster can perform task collaborative calculation with the target UAV to minimize the completion delay of the user task; wherein, the task delay model is a model constructed based on a deep deterministic policy gradient algorithm to optimize the task scheduling strategy.

[0007] In an optional implementation, the step of constructing the task collection model includes: Constructing an association matrix between the user tasks initiated by the ground user and the drones in the multi-drone cluster; the association matrix is ​​used to characterize the association relationship between the user tasks initiated by the ground user and the drones in the current time slot; Determine the task collection rate of the current time slot according to the association matrix, the first task data volume of publishing the user task, the second data volume of collecting the user task by the drone, the first location information of the ground user, and the second location information of the drone; During the flight process of the UAV for task collection, the flight trajectory of the UAV is optimized based on flight constraints and Markov decision process, and the task collection model of the UAV is determined based on the maximized task collection rate of the UAV corresponding to the optimized flight trajectory.

[0008] In an optional implementation, the task collection rate of the current time slot is determined according to the association matrix, the first task data volume of publishing the user task, the second data volume of the user task collected by the drone, the first location information of the ground user, and the second location information of the drone, and is expressed by the following formula: =

[0009] in, is the task collection rate, N is the number of drones, M is the number of ground users, For user at time t i The amount of tasks generated, is the user at time t j The amount of tasks generated, For drones i and users j The association relationship between them, t is the time slot.

[0010] In an optional embodiment, during the flight process of the UAV performing task collection, the flight trajectory of the UAV is optimized based on flight constraints and a Markov decision process, and a task collection model of the UAV is determined based on a maximized task collection rate of the UAV corresponding to the optimized flight trajectory, including: A corresponding intelligent agent is deployed on each UAV. In each time slot, the intelligent agent interacts with the environment based on the flight constraints, takes a first intelligent agent action according to the first intelligent agent state, and obtains a first immediate reward after the intelligent agent performs the first intelligent agent action; wherein the first immediate reward is related to the task collection rate; the flight constraints include the flight speed constraints of the UAV, the flight angle constraints, the movement range constraints of the UAV and the ground user, and the collision distance constraints between the UAVs; When the agent interacts with the environment, a policy network and a value network are initialized for each agent, and the preset strategy is used to interact with the environment, and the corresponding agent state, agent action and immediate reward are obtained; the advantage value of each time step is calculated by generalized advantage estimation; the objective function of the multi-agent reinforcement learning algorithm is maximized, and the value function of the multi-agent reinforcement learning algorithm is minimized, and this step is repeated until convergence to determine the task collection model of the drone.

[0011] In an optional implementation, when the user task is unloaded to the target drone, scheduling other drones in the multi-drone cluster or scheduling satellites communicating with the multi-drone cluster through a pre-built task delay model, so that other drones and / or satellites in the multi-drone cluster perform task collaborative computing with the target drone, including: When the user task is unloaded to the target UAV, an initial task delay model is constructed; wherein the initial task delay model is a model related to communication delay, calculation delay and transmission delay; The satellite end is determined as the central scheduling agent. The satellite end obtains the task information and calculation queue status of all drones, optimizes the task scheduling strategy based on the deep deterministic policy gradient algorithm, and obtains the target task delay model. The target task delay model is used to schedule other drones in the multi-drone cluster, or to schedule satellites communicating with the multi-drone cluster, so that other drones and / or satellites in the multi-drone cluster can perform task collaborative calculation with the target drone.

[0012] In an optional implementation, the satellite end is determined as a central scheduling agent, the satellite end obtains the task information of all drones and calculates the queue status, optimizes the task scheduling strategy based on the deep deterministic policy gradient algorithm, and obtains the target task delay model, including: Initialize the Actor network and Critic network of the deep deterministic policy gradient algorithm, and initialize the experience replay pool; The satellite terminal is determined as a central scheduling agent, and for each user task, a second agent action corresponding to the central scheduling agent is selected according to the current strategy, and after executing the second agent action, a second instant reward and a next second agent state are obtained, and stored in the experience replay pool; Randomly sampling a batch of samples from the experience replay pool, calculating the loss through the loss function, and updating the parameters of the policy network and the evaluation network; The above steps are used to determine the continuous action strategy corresponding to minimizing the task completion delay, so as to determine the target task delay model.

[0013] In an optional implementation, matching a target drone in a multi-drone cluster includes: Based on the location information of the ground user, a drone in the multi-drone cluster within a flight coverage range corresponding to the location information is matched, and the drone is determined as a target drone.

[0014] In an optional embodiment, the method further comprises: When the user task corresponds to multiple drones that can collect the user task, the target drone in the multi-drone cluster is matched based on the load of the drone.

[0015] In the second aspect, the present invention provides a multi-machine collaborative task scheduling system for aerospace network edge computing scenarios, which is used to implement the multi-machine collaborative task scheduling method for aerospace network edge computing scenarios described in the aforementioned implementation manner, wherein the system includes a satellite terminal, a multi-UAV cluster and multiple ground users, wherein the multi-UAV cluster includes multiple UAVs.

[0016] In a third aspect, the present invention provides a drone, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the multi-machine collaborative task scheduling method for aerospace network edge computing scenarios as described in any of the aforementioned implementations.

[0017] The beneficial effects of the multi-machine collaborative task scheduling method for aerospace network edge computing scenarios provided by this application are as follows: By modeling the operation scenario as a Markov decision process (MDP) and using the model-based multi-agent proximal policy optimization (MAPPO) algorithm to perform trajectory planning for the target drone, the task acquisition efficiency is maximized; the deep deterministic policy gradient (DDPG) algorithm is used to promote collaboration between drones, making efficient scheduling decisions for user requests and effectively reducing the average task response time, thereby effectively improving the problem of uneven drone load distribution. This method not only solves the challenges brought by differences in user distribution density, but also overcomes the problem of drone resource limitations, achieving a significant improvement in task processing efficiency and overall system performance, thereby ensuring that a high level of service quality can be maintained even in complex dynamic environments, improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a multi-machine collaborative task scheduling method for an aerospace network edge computing scenario provided in an embodiment of the present application; Figure 2 A schematic diagram of an air-space network edge computing scenario provided in an embodiment of the present application; Figure 3 A flowchart of another multi-machine collaborative task scheduling method for an aerospace network edge computing scenario provided in an embodiment of the present application; Figure 4 A structural logic diagram of a drone provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0022] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0023] The present application embodiment provides a multi-machine collaborative task scheduling method for aerospace network edge computing scenarios, see Figure 1 As shown, the method mainly includes the following steps: Step S110, in response to a user task initiated by a ground user, a target drone in a multi-drone cluster is matched, and the target drone collects the user task based on a pre-built task collection model.

[0024] The above-mentioned pre-built task collection model is a model for maximizing the task collection rate based on Markov decision process and multi-agent reinforcement learning modeling. This model is a task collection model with the goal of maximizing the task collection rate. The embodiment of the present application expresses the model optimization problem as a Markov decision process (MDP), including the definition of state, action and reward. In the specific implementation, the multi-agent reinforcement learning algorithm MAPPO (Multi-Agent Proximal Policy Optimization) is used to optimize the action selection of the drone. The MAPPO algorithm includes the steps of initialization of the policy network and the value network, trajectory sampling, advantage estimation calculation, policy update and value function update. In this way, the algorithm can gradually optimize the action strategy of each drone and maximize the task collection rate while ensuring safety.

[0025] In one embodiment, when a ground user initiates a user task through a specific interface or device (such as a smartphone application, a web interface, etc.), the request will first be received by the system. Next, the system parses the request to determine the specific requirements of the user task, such as the target location of the task, the type of data to be collected, the time limit, the priority and other key information. The system maintains a multi-drone cluster, and the drones in the multi-drone cluster are distributed in different locations and may have different capabilities and characteristics. According to the requirements of the user task, the system needs to select a suitable drone (that is, the target drone) from the cluster to collect the task.

[0026] The above-mentioned matching of the target drone in the multi-drone cluster can match the drones in the multi-drone cluster within the flight coverage corresponding to the location information based on the location information of the ground user, and determine the drone as the target drone. When the user task corresponds to multiple drones that can collect the user task, the target drone in the multi-drone cluster can also be matched based on the drone's load, for example, drones with less load are preferentially selected for collection.

[0027] Step S120, when the user task is unloaded to the target UAV, other UAVs in the multi-UAV cluster are scheduled through the pre-built task delay model, or the satellite communicating with the multi-UAV cluster is scheduled, so that other UAVs and / or satellites in the multi-UAV cluster can perform task collaborative calculation with the target UAV to minimize the user task completion delay.

[0028] Among them, the task delay model is a model for constructing an optimized task scheduling strategy based on a deep deterministic policy gradient algorithm. The task delay model is a delay model that includes communication delay, computational delay, and queuing delay. The task delay model is determined by minimizing the average completion delay of the computing task. In one embodiment, a Markov decision process (MDP) is used to model the task scheduling problem, and the satellite end is defined as a central scheduling agent to achieve unified decision-making. To solve this problem, a deep deterministic policy gradient (DDPG) algorithm is used to optimize the task scheduling strategy. The optimization steps of the DDPG algorithm include network initialization, exploration and experience collection, network update, and target network update. In this way, the load balancing problem in the collaborative computing of drone clusters and satellites can be effectively solved.

[0029] The multi-machine collaborative task scheduling method for the edge computing scenario of the aerospace network provided in the embodiment of the present application ensures the maximization of the task acquisition rate by modeling the operation scenario as a Markov decision process (MDP) and using the model-based multi-agent proximal policy optimization (MAPPO) algorithm to execute the trajectory planning of the target drone; the deep deterministic policy gradient (DDPG) algorithm is used to promote the collaboration between drones, and it can make efficient scheduling decisions for user requests, effectively reducing the average task response time, thereby effectively improving the problem of uneven drone load distribution. This method not only solves the challenges brought by differences in user distribution density, but also overcomes the problem of drone resource limitations, achieving a significant improvement in task processing efficiency and overall system performance, thereby ensuring that a high level of service quality can be maintained even in complex dynamic environments, and improving user experience.

[0030] For ease of understanding, the multi-machine collaborative task scheduling method for aerospace network edge computing scenarios provided in an embodiment of the present application is described in detail below.

[0031] First, this application is applied to the edge computing scenario of space network, see Figure 2 As shown, in this scenario, there is a satellite node, UAV nodes and The total simulation time of the whole system is T, which is evenly divided into K time slots. Task collection, drone movement decision, and task scheduling are all carried out in time slots. In each time slot, users will generate tasks according to a certain probability.

[0032] For tasks initiated by ground user m, the triplet express, Indicates the size of the task data. Indicates the computing resources required for the task, Indicates the maximum tolerable delay of the task.

[0033] Ground users can only move in an area with a length of X and a width of Y. The position coordinates of ground users are expressed as At the end of each time slot, the user moves randomly, and the mobility model is as follows:

[0034] in is the maximum moving speed of the user, , represents the actual speed of the user, is the time slot size, The user's movement angle.

[0035] The drone flies over the area at a constant altitude of H, and its position is expressed as Considering the limited coverage of a single drone, this application defines the drone coverage radius as R. During the task collection process, a drone can only collect user tasks within its coverage area. For the case where a single user is within the coverage area of ​​multiple drones, drones with less load are given priority for collection.

[0036] In one implementation, the step of constructing the task collection model may include the following steps 1-1 to 1-3: Step 1-1, construct an association matrix between user tasks initiated by ground users and drones in a multi-drone cluster; the association matrix is ​​used to characterize the association relationship between user tasks initiated by ground users and drones in the current time slot.

[0037] The relationship between the user task m initiated by the ground user and the drone n can be represented as an association matrix : .

[0038] Interpretation of the correlation matrix: N represents the number of drones; M represents the number of user tasks initiated by ground users; t represents the time point; element o n,m (t) Represents the association relationship between UAV n and user task m initiated by ground user at time point t.

[0039] Step 1-2, determine the task collection rate of the current time slot based on the association matrix, the first task data volume of the user task released, the second data volume of the user task collected by the drone, the first location information of the ground user, and the second location information of the drone. Considering the limited coverage of a single drone, multiple drones are combined to collect tasks and achieve preliminary load balancing. According to the above association matrix The association relationship between drones and users in the current time slot can be obtained, and the tasks collected by drone n are , so the task collection rate modeling of the current time slot can be expressed by the following formula: =

[0040] in, is the task collection rate, N is the number of drones, M is the number of ground users, is the user at time t i The amount of tasks generated, is the user at time t j The amount of tasks generated, For drones i and users j The association relationship between them, t is the time slot.

[0041] Based on the above model, an optimization problem can be established to maximize the task collection rate, which can be expressed as:

[0042] Step 1-3, during the flight process of the UAV collecting tasks, the flight trajectory of the UAV is optimized based on the flight constraints and the Markov decision process, and the task collection model of the UAV is determined based on the maximum task collection rate of the UAV corresponding to the optimized flight trajectory. Specifically, the steps 1-3.1 and 1-3.2 may be further included: Step 1-3.1, deploy the corresponding intelligent agent on each UAV, and in each time slot, enable the intelligent agent to interact with the environment based on the flight constraints, take the first intelligent agent action according to the first intelligent agent state, and obtain the first immediate reward after the intelligent agent performs the first intelligent agent action; wherein, the first immediate reward is related to the task collection rate.

[0043] Markov decision process (MDP) modeling is performed based on the optimization problem. In this application, each drone deploys an agent to optimize the flight trajectory of the drone and maximize the task collection rate. It mainly includes three parts: state, action and reward. In each time slot, the agent interacts with the environment, takes action a according to its own state s, makes the environment enter a new state, and obtains an immediate reward r. The details are as follows: The state of each agent includes its own location information, the location information of all users, and the task data size of all users, which is defined as:

[0044] in is the position of agent i, is the location set of all users, A collection of task data sizes for all users.

[0045] The action of each agent includes the flight speed and flight angle of the drone, which are defined as:

[0046] in is the flight speed parameter, For the flight angle, the movement model of the drone is as follows:

[0047] After the agent performs a specific action, it will receive feedback from the environment, which is the reward. In this application, the reward of the agent is defined as:

[0048] in is the total task collection rate, The penalty set for a drone flying outside of its range of motion is used to ensure that the drone flies within its range of motion. Represents the penalty set when the distance between drones is less than the minimum safe distance, ensuring that drones avoid collisions. , and is the weight coefficient, which is used to balance the importance of different objectives.

[0049] The above flight constraints include the flight speed constraints of the drone, the flight angle constraints, the movement range constraints of the drone and the ground user, and the collision distance constraints between drones. Specifically,

[0050] in C 1 and C2 is the constraint on the flight speed and flight angle of the drone. C 3~ C 6 is the restriction on the movement range of drones and users. C 7 is the constraint on the collision distance between drones.

[0051] Steps 1-3.2, when the agent interacts with the environment, initialize the policy network and value network for each agent, use the preset strategy to interact with the environment, and obtain the corresponding agent state, agent action and immediate reward; calculate the advantage value of each time step through generalized advantage estimation; maximize the objective function of the multi-agent reinforcement learning algorithm, and minimize the value function of the multi-agent reinforcement learning algorithm, repeat this step until convergence, and determine the task collection model of the drone.

[0052] In order to optimize the action selection of the drone, this application uses the multi-agent reinforcement learning algorithm MAPPO (Multi-Agent Proximal Policy Optimization). MAPPO is a multi-agent extension of the PPO algorithm, which can handle scenarios where multiple agents learn simultaneously.

[0053] The MAPPO algorithm optimizes the drone’s motion by following these steps: 1. Policy network and value network initialization: Initialize the policy network for each agent and value network .

[0054] 2. Sampling trajectory: using the current strategy Interact with the environment and collect data such as status, actions, rewards, etc.

[0055] 3. Calculate the advantage estimate: Use the generalized advantage estimate (GAE) to calculate the advantage value at each time step .

[0056] 4. Strategy update: maximize the objective function , while ensuring that the difference between the old and new policies is not too great:

[0057] in is the cropping parameter, is the old strategy before parameter update, is the policy entropy, is the entropy hyperparameter.

[0058] 5. Value function update: Minimize the loss function of the value network:

[0059] 6. Repeat steps 2-5 until convergence or the predetermined number of iterations is reached.

[0060] In this way, the MAPPO algorithm can gradually optimize the action strategy of each drone to maximize the task collection rate while ensuring safety. The algorithm will weigh flight speed, angle selection, and cooperation with other drones to achieve the best overall performance.

[0061] Furthermore, when the user task is unloaded to the target UAV, the pre-built task delay model is used to schedule other UAVs in the multi-UAV cluster, or to schedule satellites communicating with the multi-UAV cluster, so that they can perform task collaborative computing with the target UAV. In specific implementation, the following steps 2-1 to 2-3 may be included: Step 2-1, when the user task is offloaded to the target UAV, an initial task delay model is constructed; wherein the initial task delay model is a model related to communication delay, computing delay and transmission delay.

[0062] The task completion process mainly includes communication delay, calculation delay and transmission delay.

[0063] 1. Communication delay mainly includes unloading delay and scheduling delay: 1) Unloading delay is the time required to unload task m to drone n:

[0064] in, is the transmission rate between drone n and user m, defined as:

[0065] Indicates drone The channel bandwidth, is the maximum transmission power of the user equipment, is the noise power, is the channel gain between UAV n and user equipment m.

[0066] 2) Scheduling delay is the maximum transmission delay of task m during the task scheduling process.

[0067]

[0068] Where j is the number of the drone unloaded from task m, is the unloading ratio of UAV j to UAV n, is the task offloading ratio of UAV j dispatched to the satellite. and are the transmission rates between UAV n and UAV j and the transmission rates between UAV j and the satellite, respectively. is defined as:

[0069] in, represents the channel bandwidth, is the maximum transmission power of the UAV. represents the channel gain between UAV n and UAV j.

[0070] The transmission rate between the drone and the satellite is defined as follows:

[0071] in, represents the channel bandwidth, is the path loss of the communication channel between the UAV and the satellite.

[0072] Therefore, the communication delay is .

[0073] 2. For task m, its computational latency is defined as follows

[0074] in Indicates the number of cycles required for task m to calculate one unit of task data, For the processing power of the drone, The processing power of the satellite.

[0075] At the same time, considering the limited processing power of the drone, a computing queue is maintained on the drone side for task processing. After the task is dispatched to other drones according to the unloading ratio, it will be stored in the drone's computing queue according to the arrival time. Since the computing power and storage capacity of the satellite are much greater than that of the drone, the satellite side does not maintain a computing queue, and the computing tasks dispatched to the satellite can be calculated immediately without waiting. The computing queue delay of task m is the sum of the computing delays of the previous tasks, that is,

[0076] In summary, the total completion delay of a task = computation delay + queuing delay + communication delay, communication delay = unloading delay + scheduling delay, that is, the total completion delay for task m is: .

[0077] In step 2-2, the satellite end is determined as the central scheduling agent. The satellite end obtains the task information and calculation queue status of all drones, optimizes the task scheduling strategy based on the deep deterministic policy gradient algorithm, and obtains the target task delay model.

[0078] Based on the task delay model constructed in the above steps, the total completion delay of the task initiated by user m in time slot t is This application aims to minimize the average completion delay of computing tasks and proposes a task scheduling decision optimization scheme. The optimization problem is modeled as follows:

[0079]

[0080] Among them, C1 is the constraint on the task scheduling variable, ensuring that the sum of the task scheduling ratios is 1.

[0081] The embodiment of the present application uses the Markov Decision Process (MDP) to model the task scheduling problem of collaborative computing between drone clusters and satellites. Taking into account the problem that the difference in load of each drone in a heterogeneous drone cluster may lead to inconsistent state dimensions, preferably, the present application defines the satellite end as a central scheduling agent to achieve unified decision-making. The satellite end can obtain the task information and calculation queue status of all drones, so as to perform global optimization. In view of the fact that scheduling decisions need to be made independently for each task, the Deep Deterministic Policy Gradient (DDPG) algorithm is used to optimize the task scheduling strategy. As a deep reinforcement learning method based on the actor-critic architecture, the DDPG algorithm can effectively handle decision-making problems in continuous action space and improve the accuracy of decision optimization in this application scenario.

[0082] The state of the agent includes the data size of the current task and the calculation queue status of all drones in the previous time slot, which is defined as follows:

[0083] in, The calculation queue of drone n in time slot t-1 is updated as follows:

[0084] Considering that the task is divisible and can be dispatched to other drones and satellites in a certain ratio, this application defines the agent as the dispatch ratio of task n, which is defined as follows:

[0085] After outputting the action, the agent will interact with the environment and obtain the task completion delay, which is the reward. .

[0086] Furthermore, in the specific implementation, the following specific steps 2-2.1 to 2-2.4 may be included: Step 2-2.1, initialize the Actor network and Critic network of the deep deterministic policy gradient algorithm, and initialize the experience replay pool; Step 2-2.2: Determine the satellite as the central dispatching agent for each user task , select the second agent action corresponding to the central dispatching agent according to the current strategy (In the early stage of training, some noise can be added to increase the exploration rate) and execute the second agent action After that, get the second instant reward and the next second agent state , and Deposit into the experience replay pool; Step 2-2.3, randomly sample a batch of samples from the experience replay pool, calculate the loss through the loss function, and update the parameters of the policy network and the evaluation network; Step 2-2.4, determine the continuous action strategy corresponding to minimizing the task completion delay through the above steps to determine the target task delay model.

[0087] Step 2-3, through the target task delay model, other drones in the multi-drone cluster, or scheduling satellites communicating with the multi-drone cluster, so that they can perform task collaborative calculations with the target drone.

[0088] In this way, the DDPG algorithm is used for decision optimization, which can gradually optimize the task scheduling strategy and learn a continuous action strategy that can minimize the task completion delay, thereby enhancing the decision-making of drone scheduling and effectively reducing the average task response time.

[0089] In summary, the embodiments of the present application ensure the maximization of task acquisition efficiency by modeling the operation scenario as a Markov decision process (MDP) and using the model-based multi-agent proximal policy optimization (MAPPO) algorithm to perform trajectory planning of the target UAV; the deep deterministic policy gradient (DDPG) algorithm is used to promote collaboration between UAVs, and efficient scheduling decisions can be made for user requests, effectively reducing the average task response time, thereby effectively improving the problem of uneven load distribution of UAVs.

[0090] In the actual task collection and task scheduling process, Figure 3 Another multi-machine collaborative task scheduling method for aerospace network edge computing scenarios is shown, including the following steps: Step S1: For ground users in any time slot, the drone cluster collects user location information and data size information to determine its local state of the environment. The local state refers to the state observed by each drone agent. In this simulation system, the local state is [the location information of the current drone, the location information of all users, and the task size of all users].

[0091] In step S2, the agent observes the local state and outputs trajectory optimization decisions.

[0092] Step S3: The drone flies and collects tasks according to the action output by the agent, and obtains the total task collection rate of the current time slot as the task reward of the agent. Step S4: For the tasks unloaded to the drone, the satellite makes a task scheduling decision. For any task, the task data size and the reinforcement of each drone queue in the previous time slot are obtained as the input of the task scheduling agent, and the actor network outputs the task scheduling decision.

[0093] Step S5, schedule the task according to the scheduling decision, count the task completion delay as a reward for the scheduling algorithm, update the environment, and enter the next state.

[0094] Step S6, the user moves randomly and enters the next time slot, and the user generates tasks according to a certain probability. In practical applications, the entire simulation duration is T. In order to facilitate simulation, it is pre-divided into K time slots. At the beginning of each time slot, the user will generate tasks according to a certain probability. After that, the drone adjusts the flight trajectory to collect tasks and complete the coordinated scheduling of tasks.

[0095] In summary, the above-mentioned intelligent agent can dynamically adjust the flight path and task scheduling strategy of the UAV according to environmental changes and user behavior, so as to better adapt to complex real-life scenarios; through the decision-making process of the intelligent agent, the optimization of the UAV's flight path and the rationalization of task allocation are ensured, thereby improving the resource utilization of the entire system.

[0096] Furthermore, the multi-machine collaborative task scheduling system for aerospace network edge computing scenarios provided in the embodiments of the present application is used to implement the multi-machine collaborative task scheduling method for aerospace network edge computing scenarios described in the aforementioned implementation method. Its implementation principle and the technical effects produced are the same as those of the aforementioned method embodiments. For the sake of brief description, for the parts not mentioned in the embodiment of the multi-machine collaborative task scheduling device for aerospace network edge computing scenarios, reference can be made to the corresponding contents in the aforementioned multi-machine collaborative task scheduling method embodiment for aerospace network edge computing scenarios. The system includes a satellite terminal, a multi-drone cluster and multiple ground users, wherein the multi-drone cluster includes multiple drones.

[0097] The present application also provides a drone, such as Figure 4 As shown, it is a structural diagram of the UAV, wherein the electronic device 100 includes a processor 41 and a memory 40, the memory 40 stores computer executable instructions that can be executed by the processor 41, and the processor 41 executes the computer executable instructions to implement any of the above-mentioned multi-machine collaborative task scheduling methods for aerospace network edge computing scenarios.

[0098] exist Figure 4 In the illustrated embodiment, the electronic device further includes a bus 42 and a communication interface 43 , wherein the processor 41 , the communication interface 43 and the memory 40 are connected via the bus 42 .

[0099] Among them, the memory 40 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 42 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 42 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0100] The processor 41 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 41. The above processor 41 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 41 reads the information in the memory and combines its hardware to complete the steps of the multi-machine collaborative task scheduling method for the aerospace network edge computing scenario of the aforementioned embodiment.

[0101] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0102] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0103] In the description of the present application, it should be noted that the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-machine collaborative task scheduling method for aerospace network edge computing scenarios, characterized in that: The method comprises: In response to a user task initiated by a ground user, a target drone in a multi-drone cluster is matched, and the target drone collects the user task based on a pre-built task collection model; wherein the pre-built task collection model is a model for maximizing the task collection rate based on Markov decision process and multi-agent reinforcement learning modeling; When the user task is offloaded to the target UAV, other UAVs in the multi-UAV cluster are scheduled through a pre-constructed task delay model, or a satellite communicating with the multi-UAV cluster is scheduled, so that other UAVs and / or satellites in the multi-UAV cluster can perform task collaborative calculation with the target UAV to minimize the completion delay of the user task; wherein, the task delay model is a model constructed based on a deep deterministic policy gradient algorithm to optimize the task scheduling strategy.

2. The multi-machine collaborative task scheduling method for aerospace network edge computing scenarios according to claim 1 is characterized in that: The steps of constructing the task collection model include: Constructing an association matrix between the user tasks initiated by the ground user and the drones in the multi-drone cluster; the association matrix is ​​used to characterize the association relationship between the user tasks initiated by the ground user and the drones in the current time slot; Determine the task collection rate of the current time slot according to the association matrix, the first task data volume of publishing the user task, the second data volume of collecting the user task by the drone, the first location information of the ground user, and the second location information of the drone; During the flight process of the UAV for task collection, the flight trajectory of the UAV is optimized based on flight constraints and Markov decision process, and the task collection model of the UAV is determined based on the maximized task collection rate of the UAV corresponding to the optimized flight trajectory.

3. The multi-machine collaborative task scheduling method for aerospace network edge computing scenarios according to claim 2 is characterized in that: The task collection rate of the current time slot is determined according to the association matrix, the first task data volume of the user task released, the second data volume of the user task collected by the drone, the first location information of the ground user, and the second location information of the drone, and is expressed by the following formula: = in, is the task collection rate, N is the number of drones, M is the number of ground users, For user at time t i The amount of tasks generated, is the user at time t j The amount of tasks generated, For drones i and users j The association relationship between them, t is the time slot.

4. The multi-machine collaborative task scheduling method for aerospace network edge computing scenarios according to claim 2 is characterized in that: During the flight process of the UAV for task collection, the flight trajectory of the UAV is optimized based on flight constraints and a Markov decision process, and the task collection model of the UAV is determined based on the maximum task collection rate of the UAV corresponding to the optimized flight trajectory, including: A corresponding intelligent agent is deployed on each UAV. In each time slot, the intelligent agent interacts with the environment based on the flight constraints, takes a first intelligent agent action according to the first intelligent agent state, and obtains a first immediate reward after the intelligent agent performs the first intelligent agent action; wherein the first immediate reward is related to the task collection rate; the flight constraints include the flight speed constraints of the UAV, the flight angle constraints, the movement range constraints of the UAV and the ground user, and the collision distance constraints between the UAVs; When the agent interacts with the environment, a policy network and a value network are initialized for each agent, and the preset strategy is used to interact with the environment, and the corresponding agent state, agent action and immediate reward are obtained; the advantage value of each time step is calculated by generalized advantage estimation; the objective function of the multi-agent reinforcement learning algorithm is maximized, and the value function of the multi-agent reinforcement learning algorithm is minimized, and this step is repeated until convergence to determine the task collection model of the drone.

5. The multi-machine collaborative task scheduling method for aerospace network edge computing scenarios according to claim 1 is characterized in that: When the user task is unloaded to the target drone, other drones in the multi-drone cluster are scheduled by a pre-built task delay model, or a satellite communicating with the multi-drone cluster is scheduled, so that other drones and / or satellites in the multi-drone cluster perform task collaborative computing with the target drone, including: When the user task is unloaded to the target UAV, an initial task delay model is constructed; wherein the initial task delay model is a model related to communication delay, calculation delay and transmission delay; The satellite end is determined as the central scheduling agent. The satellite end obtains the task information and calculation queue status of all drones, optimizes the task scheduling strategy based on the deep deterministic policy gradient algorithm, and obtains the target task delay model. The target task delay model is used to schedule other drones in the multi-drone cluster, and / or schedule satellites communicating with the multi-drone cluster, so that other drones and / or satellites in the multi-drone cluster can perform task collaborative calculations with the target drone.

6. The multi-machine collaborative task scheduling method for aerospace network edge computing scenarios according to claim 5 is characterized in that: The satellite end is determined as the central scheduling agent. The satellite end obtains the task information and calculation queue status of all drones, optimizes the task scheduling strategy based on the deep deterministic policy gradient algorithm, and obtains the target task delay model, including: Initialize the Actor network and Critic network of the deep deterministic policy gradient algorithm, and initialize the experience replay pool; Determine the satellite end as the central scheduling agent, select the second agent action corresponding to the central scheduling agent according to the current strategy for each user task, obtain the second instant reward and the next second agent state after executing the second agent action, and store them in the experience replay pool; Randomly sampling a batch of samples from the experience replay pool, calculating the loss through the loss function, and updating the parameters of the policy network and the evaluation network; The above steps are used to determine the continuous action strategy corresponding to minimizing the task completion delay, so as to determine the target task delay model.

7. The multi-machine collaborative task scheduling method for aerospace network edge computing scenarios according to claim 1 is characterized in that: Match target drones in a multi-drone cluster, including: Based on the location information of the ground user, a drone in the multi-drone cluster within a flight coverage range corresponding to the location information is matched, and the drone is determined as a target drone.

8. The multi-machine collaborative task scheduling method for aerospace network edge computing scenarios according to claim 1 or 7 is characterized in that: The method further comprises: When the user task corresponds to multiple drones that can collect the user task, the target drone in the multi-drone cluster is matched based on the load of the drone.

9. A multi-machine collaborative task scheduling system for aerospace network edge computing scenarios, characterized in that: A task scheduling method for multi-machine collaboration for aerospace network edge computing scenarios according to claim 1 is used to implement the system, which includes a satellite terminal, a multi-UAV cluster and multiple ground users, wherein the multi-UAV cluster includes multiple UAVs.

10. A drone, characterized in that: It includes a processor and a memory, the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the multi-machine collaborative task scheduling method for aerospace network edge computing scenarios as described in any one of claims 1 to 8.

Images