Space-space integrated task processing method and device based on hierarchical reinforcement learning

Abstract

The invention provides an air-space integrated task processing method and device based on hierarchical reinforcement learning, and relates to the technical field of mobile edge computation.The method comprises the steps that after the states of all unmanned aerial vehicles in a current time slot are determined, a target multi-agent deep reinforcement learning model is used for processing the states of all the unmanned aerial vehicles, and the target multi-agent deep reinforcement learning model is used for processing the states of all the unmanned aerial vehicles; obtaining a task unloading proportion decision of each unmanned aerial vehicle, then constructing a state of a center satellite based on the task unloading proportion decisions of all unmanned aerial vehicles, and processing the state of the center satellite by using a target single-agent deep reinforcement learning model; and obtaining a task allocation proportion decision of the task quantity unloaded to the space-based network by each unmanned aerial vehicle on each satellite. According to the method, a task deployment problem is decomposed into a task unloading problem and a task allocation problem, so that the complexity of an air-space integrated task processing problem is effectively reduced, the action space dimension of an intelligent agent is reduced, and the convergence performance and the processing efficiency of the air-space integrated task processing method are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile edge computing, and in particular to a method and device for processing air-space integrated tasks based on hierarchical reinforcement learning. Background Art

[0002] With the continuous development of 5G, 6G has been proposed, which aims to extend the ground mobile communication network to the sky and build an ultra-wideband mobile communication system connecting the sky, air, sea and land. The integrated space-air-ground network has become one of the core research architectures of the 6G network. In addition, in the ground communication network, mobile edge computing provides a new paradigm for various IoT applications, providing lower latency and more secure transmission compared to cloud computing. As an emerging paradigm that combines edge computing with air-based networks, integrated space-air edge computing has attracted widespread attention due to its potential to support various applications that require low latency and high throughput in dynamic environments. However, efficient resource management, including task allocation and computation offloading, remains a key challenge for integrated space-air edge computing systems.

[0003] Deep reinforcement learning combines the perception ability of deep learning with the decision-making ability of reinforcement learning, realizes end-to-end learning, can solve various complex problems in real-world scenarios, and has attracted widespread attention. In recent years, many studies have applied deep reinforcement learning to edge computing scenarios based on space-air integration. However, as the optimization problem becomes more and more complex, the state and action space of the agent may grow significantly, leading to the dimensionality hell problem and bringing challenges to algorithm convergence and efficiency. In summary, the existing space-air integrated task processing methods have technical problems such as high convergence difficulty and low processing efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for integrated space-air mission processing based on hierarchical reinforcement learning, so as to alleviate the technical problems of high convergence difficulty and low processing efficiency in the integrated space-air mission processing method in the prior art.

[0005] In a first aspect, the present invention provides an air-space integrated task processing method based on hierarchical reinforcement learning, including: determining the status of all drones in the current time slot; wherein the status of the target drone includes: the distance between the mapping point of the central satellite on the drone plane and the target drone, the processing time required for the task queue of the target drone at the start of the current time slot, the amount of tasks generated by the target drone in the current time slot, the index of the target drone and the time slot index; the target drone represents any drone among all drones in the air-based network; the central satellite represents the satellite with the least computing resources in the space-based network; the status of all drones is processed using a target multi-agent deep reinforcement learning model to obtain a task offloading ratio decision for each drone; wherein the target multi-agent deep reinforcement learning model includes multiple drone agents, and each drone agent is connected to the air-based network. The drones correspond one to one, the observation state of the drone agent is the state of the corresponding drone, and the action of the drone agent is the task offloading ratio decision of the corresponding drone; determine the task queue processing time and the computing resources required for the task queue of each satellite in the space-based network at the start of the current time slot; construct the state of the central satellite based on the task queue processing time of all satellites, the computing resources required for the task queue, the task offloading ratio decision of all drones and the time slot index; use the target single-agent deep reinforcement learning model to process the state of the central satellite, and obtain the task allocation ratio decision on each satellite for the amount of tasks unloaded from each drone to the space-based network; among them, the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation ratio decision.

[0006] Optionally, the target multi-agent deep reinforcement learning model is a plurality of first performer networks in the MATD3 network, and the target single-agent deep reinforcement learning model is a second performer network in the TD3 network; before determining the state of all drones in the current time slot, it also includes: initializing the network parameters of the MATD3 network and the network parameters of the TD3 network; initializing the system environment state, and under the current network parameters and system environment state, continuously generating first experience data and storing it in the drone agent experience replay device, and continuously generating second experience data and storing it in the satellite agent experience replay device; wherein the first experience data includes: the observation state, action, first reward of the drone agent in the current time slot and the observation state of the next time slot, and the second experience data includes: the observation state, action, second reward of the satellite agent in the current time slot and the observation state of the next time slot; after determining that the number of groups of the first experience data in the drone agent experience replay device reaches When it is determined that the number of groups of second experience data in the satellite agent experience replayer reaches a multiple of the second preset update step number, a second number of second experience data is extracted from the satellite agent experience replayer to train the TD3 network to update its network parameters; when it is determined that the network parameters of the MATD3 network or the network parameters of the TD3 network are updated, if the number of groups of first experience data in the drone agent experience replayer does not reach the target number, the first experience data and the second experience data continue to be generated under the updated network parameters; otherwise, the step of initializing the system environment state is returned until the preset rounds are reached to obtain the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model.

[0007] Optionally, under the current network parameters and system environment, first experience data is continuously generated and stored in the UAV agent experience replay device, and second experience data is continuously generated and stored in the satellite agent experience replay device, including: under the current network parameters and system environment, the observation state of each UAV agent in the MATD3 network in the current time slot is obtained, and the action of each UAV agent in the current time slot is determined based on the corresponding first performer network; the observation state of the satellite agent in the current time slot is constructed based on the action of all UAV agents in the current time slot, and the observation state of the satellite agent in the current time slot is processed by the second performer network to obtain the action of the satellite agent in the current time slot; based on the observation state and action of each UAV agent in the current time slot, the satellite agent is determined. Determine the observation state of each drone agent in the next time slot, and determine the action of each drone agent in the next time slot based on the corresponding first performer network; construct the observation state of the satellite agent in the next time slot based on the actions of all drone agents in the next time slot; calculate the first reward obtained by each drone agent after performing the action in the current time slot and the second reward obtained by the satellite agent after performing the action in the current time slot; store the observation state, action, first reward of all drone agents in the current time slot and the observation state of the next time slot as the first experience data in the drone agent experience replay device, and store the observation state, action, second reward of the satellite agent in the current time slot and the observation state of the next time slot as the second experience data in the satellite agent experience replay device.

[0008] Optionally, the training objective of the MATD3 network and the TD3 network is to minimize the system cost, and the calculation process of the system cost includes: calculating the first total delay of the task that the target UAV leaves to process locally in the current time slot, and calculating the second total delay of the task that the target UAV offloads to the target satellite for processing in the current time slot; wherein the target satellite represents any satellite in the space-based network; taking the maximum delay among the first total delay and all second total delays as the processing delay of the task generated by the target UAV in the current time slot; calculating the average value of the processing delay of the tasks generated by all UAVs in the current time slot, and taking the average value as the delay cost; calculating the first energy consumption of the task that the target UAV leaves to process locally in the current time slot, and calculating the second energy consumption of the task that the target UAV offloads to the target satellite for processing in the current time slot; calculating the energy consumption cost based on the first energy consumption of all UAVs and the second energy consumption of all satellites in the current time slot; and calculating the system cost of the current time slot based on the delay cost and the energy consumption cost.

[0009] Optionally, calculating the first total delay of the tasks left to be processed locally by the target UAV in the current time slot includes: taking the processing time required for the task queue of the target UAV at the start time of the current time slot as the first waiting delay of the tasks left to be processed locally by the target UAV in the current time slot; determining the amount of tasks left to be processed locally by the target UAV in the current time slot based on the task offloading ratio decision of the target UAV in the current time slot and the amount of tasks generated by the target UAV in the current time slot; determining the first computing delay of the tasks left to be processed locally by the target UAV in the current time slot based on the amount of tasks processed locally, the computing resources required for the target UAV to process 1 bit of data, and the computing power of the target UAV; determining the first total delay of the tasks left to be processed locally by the target UAV in the current time slot based on the first waiting delay and the first computing delay.

[0010] Optionally, calculating the second total delay of the task unloaded from the target UAV to the target satellite for processing in the current time slot includes: calculating the processing delay of the current task queue of the target satellite; wherein the current task queue represents the task queue of the target satellite before receiving the task unloaded from the target UAV; calculating the transmission delay of the target UAV to unload the task to the target satellite; using the maximum delay of the processing delay and the transmission delay of the current task queue as the second waiting delay of the task unloaded from the target UAV to the target satellite for processing in the current time slot; determining the second calculation delay of the task unloaded from the target UAV to the target satellite for processing in the current time slot based on the amount of tasks unloaded from the target UAV to the target satellite for processing in the current time slot, the computing resources required for the target satellite to process 1 bit of data, and the computing power of the target satellite; determining the second total delay of the task unloaded from the target UAV to the target satellite for processing in the current time slot based on the second waiting delay and the second calculation delay.

[0011] Optionally, the first reward obtained by each drone agent after performing the action in the current time slot and the second reward obtained by the satellite agent after performing the action in the current time slot are calculated, including: based on the first total delay and the first energy consumption of the task left by the target drone for local processing in the current time slot, calculating the first total cost of the task left by the target drone for local processing in the current time slot; based on the second total delay and the second energy consumption of the task offloaded from the target drone to each satellite for processing in the current time slot, calculating the second total cost of the task offloaded from the target drone to the satellite for processing in the current time slot; based on the first total cost, the second total cost, the first waiting delay and the system cost, calculating the first reward obtained by the drone agent corresponding to the target drone after performing the action in the current time slot; based on all the second total costs and system costs of the tasks offloaded from all drones to the satellite for processing in the current time slot, calculating the second reward obtained by the satellite agent after performing the action in the current time slot.

[0012] In a second aspect, the present invention provides an integrated air-space task processing device based on hierarchical reinforcement learning, comprising: a first determination module, used to determine the state of all drones in the current time slot; wherein the state of the target drone includes: the distance between the mapping point of the central satellite on the drone plane and the target drone, the computing resources required for the task queue of the target drone at the start of the current time slot, the amount of tasks generated by the target drone in the current time slot, the index of the target drone and the time slot index; the target drone represents any drone among all drones in the air-based network; the central satellite represents the satellite with the least computing resources in the space-based network; the first processing module is used to process the state of all drones using a target multi-agent deep reinforcement learning model to obtain a task offloading ratio decision for each drone; wherein the target multi-agent deep reinforcement learning model includes multiple drone agents, and each drone agent is related to a drone in the air-based network. One-to-one correspondence, the observation state of the UAV agent is the state of the corresponding UAV, and the action of the UAV agent is the task offloading ratio decision of the corresponding UAV; the second determination module is used to determine the task queue processing time of each satellite in the space-based network at the start of the current time slot and the computing resources required for the task queue; the construction module is used to construct the state of the central satellite based on the task queue processing time of all satellites, the computing resources required for the task queue, the task offloading ratio decision of all UAVs and the time slot index; the second processing module is used to process the state of the central satellite using the target single-agent deep reinforcement learning model to obtain the task allocation ratio decision on each satellite for the amount of tasks unloaded from each UAV to the space-based network; wherein, the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation ratio decision.

[0013] In a third aspect, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the aforementioned integrated air-space task processing method based on hierarchical reinforcement learning is implemented.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the aforementioned integrated air-space task processing method based on hierarchical reinforcement learning is implemented.

[0015] Aiming at the task deployment problem in the space-air integrated edge computing scenario, the present invention provides a space-air integrated task processing method based on hierarchical reinforcement learning, the first layer solves the task unloading problem of UAV tasks, and the second layer solves the task allocation problem of satellites. Specifically, after determining the status of all UAVs in the current time slot, the target multi-agent deep reinforcement learning model is first used to process the status of all UAVs to obtain the task unloading ratio decision of each UAV, thereby solving the task unloading problem of UAV tasks in the first layer, and then the status of the central satellite is constructed based on the task unloading ratio decision of all UAVs, and the target single-agent deep reinforcement learning model is used to process the status of the central satellite to obtain the task allocation ratio decision of each UAV unloaded to the space-based network on each satellite, thereby solving the task allocation problem of the satellite in the second layer, and then realizing the decomposition of the task deployment problem into the task unloading problem and the task allocation problem, effectively reducing the complexity of the space-air integrated task processing problem, reducing the dimension of the action space of the agent, thereby improving the convergence performance and processing efficiency of the space-air integrated task processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of a method for processing an air-space integrated task based on hierarchical reinforcement learning provided in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of the scenario model of integrated air-space edge computing;

[0019] Figure 3 This is a schematic diagram of the framework of the air-space integrated task processing method based on hierarchical reinforcement learning;

[0020] Figure 4 A functional module diagram of an air-space integrated task processing device based on hierarchical reinforcement learning provided by an embodiment of the present invention;

[0021] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0025] Embodiment 1

[0026] Figure 1 A flowchart of a method for processing an air-space integrated task based on hierarchical reinforcement learning is provided in an embodiment of the present invention, such as Figure 1 As shown, the method specifically comprises the following steps:

[0027] Step S102, determining the status of all drones in the current time slot.

[0028] Among them, the status of the target UAV includes: the distance between the mapping point of the central satellite on the UAV plane and the target UAV, the processing time required for the task queue of the target UAV at the start of the current time slot, the amount of tasks generated by the target UAV in the current time slot, the index of the target UAV and the time slot index; the target UAV represents any UAV among all UAVs in the air-based network; the central satellite represents the satellite with the least computing resources in the space-based network.

[0029] Specifically, for the task deployment problem in the space-air integrated edge computing scenario, the embodiment of the present invention proposes a hierarchical reinforcement learning framework. Figure 2 This is a schematic diagram of the scenario model of integrated space-air edge computing. Figure 2It can be seen that the system consists of three parts: a space-based network composed of multiple low-orbit satellites, an air-based network composed of multiple drones, and a ground terminal network, and the ground terminal network can only communicate through the air-based network. Specifically, each drone swarm serves a ground area, providing communication coverage and computing support for users on the ground. Each satellite provides services for a drone swarm, providing communication coverage and computing support for drones. Each drone generates a task in each time slot. The drone can process part of the task locally and choose to offload the other part to the satellite for processing. At the same time, since different satellites need to process different amounts of tasks, the central satellite with insufficient computing resources can assign tasks to satellites with sufficient computing resources for collaborative processing through intersatellite links. In the present invention and embodiments, it is assumed that the system is quasi-static, that is, within a time slot, environmental factors such as the wireless channel gain and the position of the drone remain unchanged.

[0030] The method provided in the embodiment of the present invention can provide a task deployment scheme for any time slot in the task cycle of the space-air integrated network. To perform task deployment in the current time slot, it is first necessary to determine the status of all drones in the current time slot t and the status of the target drone u in the current time slot t. The status includes: the distance between the center satellite's mapping point on the drone plane and the target drone u The processing time required for the task queue of target drone u at the beginning of the current time slot t The amount of tasks generated by the target UAV u at the current time slot t The target UAV’s index u and time slot index t.

[0031] Specifically, in the embodiment of the present invention, the three-dimensional spatial position coordinates of the central satellite and the three-dimensional spatial position coordinates of each drone are known, and all drones are at the same flight altitude. Therefore, the three-dimensional spatial position coordinates of the mapping point of the central satellite on the drone plane can be determined, and the distance formula can be used to calculate

[0032] The processing time required for the task queue of target drone u at the beginning of the current time slot t in, represents the computing resources required by the task queue of target UAV u at the beginning of the current time slot t (in cycles), f u represents the computing power of the target drone u (unit: GHz), represents the computing resources required by the task queue of target UAV u at the beginning of time slot t-1. If If it is equal to 0, it means that there is no queued task in the previous time slot. δ represents the length of a time slot. c uIt represents the computing resources required by the target drone u to process 1 bit of data (unit: cycles / bit). It represents the amount of tasks generated by the target UAV u in the time slot t-1 that are retained for local processing (unit: bit), represents the computing resources that the target UAV u can process in a time slot, Indicates processing The computing resources required.

[0033] Step S104, using the target multi-agent deep reinforcement learning model to process the status of all drones to obtain the task offloading ratio decision of each drone.

[0034] Among them, the target multi-agent deep reinforcement learning model includes multiple UAV agents, each UAV agent corresponds one-to-one to a UAV in the air-based network, the observation state of the UAV agent is the state of the corresponding UAV, and the action of the UAV agent is the task offloading ratio decision of the corresponding UAV.

[0035] Figure 3 This is a schematic diagram of the framework of the space-air integrated task processing method based on hierarchical reinforcement learning. Figure 3 It can be seen that the embodiment of the present invention decomposes the task processing problem in the integrated space-air network into two layers of sub-problems. The first layer solves the task offloading problem of UAV tasks, and the second layer solves the task allocation problem of satellites. In the embodiment of the present invention, the first layer adopts a multi-agent deep reinforcement learning model, and the second layer adopts a single-agent deep reinforcement learning model. By pre-training the models in the first and second layers, the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model can be obtained.

[0036] The target multi-agent deep reinforcement learning model includes multiple UAV agents. The number of UAV agents is consistent with the number of UAVs in the air-based network and corresponds one to one. To obtain the task offloading ratio decision of the target UAV, it is only necessary to input the state of the target UAV into the corresponding UAV agent. The action output by the UAV agent is the task offloading ratio decision of the target UAV. in, represents the task offloading ratio decision of target UAV u in the current time slot t, represents the proportion of the target UAV u offloading the tasks generated in the current time slot t to the satellite, It represents the proportion of the tasks generated by the target UAV u in the current time slot t that are kept for local processing,

[0037] Step S106, determining the task queue processing duration of each satellite in the space-based network and the computing resources required by the task queue at the start time of the current time slot.

[0038] Step S108, construct the status of the central satellite based on the task queue processing time of all satellites, the computing resources required for the task queue, the task offloading ratio decision of all drones and the time slot index.

[0039] In the embodiment of the present invention, the state of the central satellite cs in the current time slot t in, It represents the set of task queue processing durations of all satellites in the space-based network at the start of the current time slot t. represents the set of computing resources required by the task queues of all satellites in the space-based network at the beginning of the current time slot t, represents the set of task offloading ratio decisions of all drones, Represents a collection of drones.

[0040] Among them, the task queue processing time of the target satellite s in the space-based network at the beginning of the current time slot t is in, represents the computing resources required by the task queue of satellite s at the start of the current time slot t (unit: cycles), f s represents the computing power of satellite s (unit: GHz), represents the computing resources required by the task queue of satellite s at the beginning of time slot t-1. If If it is equal to 0, it means that there is no queued task in the previous time slot, c s It represents the computing resources required by satellite s to process 1 bit of data (unit: cycles / bit). represents the total task amount (bit) assigned to satellite s in time slot t-1, Indicates the computing resources (cycles) that the satellite can process in one time slot. Indicates processing The computing resources required.

[0041] Step S110, using the target single-agent deep reinforcement learning model to process the state of the central satellite, and obtain the task allocation ratio decision on each satellite for the task volume unloaded by each drone to the space-based network.

[0042] Among them, the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation ratio decision.

[0043] There is only one satellite agent in the target single-agent deep reinforcement learning model, and this agent corresponds to the central satellite. After obtaining the state of the central satellite, the state of the central satellite is input into the satellite agent. The action output by the satellite agent is the task allocation ratio decision of each UAV to the space-based network on each satellite. in, represents the set of task allocation ratios of all satellites for the amount of tasks unloaded to the Tianji network by the target UAV u at the current time slot t, represents the task allocation ratio of the target UAV u to the Tianji network in the current time slot t on the satellite S, Represents the collection of satellites in the Tianji network.

[0044] Aiming at the task deployment problem in the space-air integrated edge computing scenario, the present invention provides a space-air integrated task processing method based on hierarchical reinforcement learning, the first layer solves the task unloading problem of UAV tasks, and the second layer solves the task allocation problem of satellites. Specifically, after determining the status of all UAVs in the current time slot, the target multi-agent deep reinforcement learning model is first used to process the status of all UAVs to obtain the task unloading ratio decision of each UAV, thereby solving the task unloading problem of UAV tasks in the first layer, and then the status of the central satellite is constructed based on the task unloading ratio decision of all UAVs, and the target single-agent deep reinforcement learning model is used to process the status of the central satellite to obtain the task allocation ratio decision of each UAV unloaded to the space-based network on each satellite, thereby solving the task allocation problem of the satellite in the second layer, and then realizing the decomposition of the task deployment problem into the task unloading problem and the task allocation problem, effectively reducing the complexity of the space-air integrated task processing problem, reducing the dimension of the action space of the agent, thereby improving the convergence performance and processing efficiency of the space-air integrated task processing method.

[0045] The above describes in detail how to apply the model to complete task deployment after the first and second layer model training is completed. The following describes in detail how to obtain the models in the first and second layers.

[0046] In an optional embodiment, the target multi-agent deep reinforcement learning model is a plurality of first performer networks in the MATD3 network, and the target single-agent deep reinforcement learning model is a second performer network in the TD3 network; before determining the states of all drones in the current time slot, the method of the present invention further comprises the following steps:

[0047] Step S201, initializing network parameters of the MATD3 network and network parameters of the TD3 network.

[0048] Step S202, initialize the system environment state, under the current network parameters and system environment state, continuously generate first experience data and store it in the UAV agent experience replay device, and continuously generate second experience data and store it in the satellite agent experience replay device; wherein the first experience data includes: the observation state, action, first reward of the UAV agent in the current time slot and the observation state of the next time slot, and the second experience data includes: the observation state, action, second reward of the satellite agent in the current time slot and the observation state of the next time slot.

[0049] Step S203, when it is determined that the number of groups of first experience data in the drone intelligent agent experience replayer reaches a multiple of the first preset update step number, extract a first number of first experience data from the drone intelligent agent experience replayer to train the MATD3 network to update its network parameters.

[0050] Step S204, when it is determined that the number of groups of second experience data in the satellite agent experience replayer reaches a multiple of the second preset update step number, extract a second amount of second experience data from the satellite agent experience replayer to train the TD3 network to update its network parameters.

[0051] Step S205, when it is determined that the network parameters of the MATD3 network or the network parameters of the TD3 network have been updated, if the number of groups of first experience data in the drone agent experience playback device has not reached the target number, then under the updated network parameters, continue to generate the first experience data and the second experience data; otherwise, return to the step of initializing the system environment state until the preset rounds are reached to obtain the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model.

[0052] In the embodiment of the present invention, in order to solve the problem of unloading the UAV task, since multiple UAV agents are required, the first layer uses the MATD3 network. In order to solve the problem of satellite task allocation, since only one satellite agent is required, the second layer uses the TD3 network. Figure 3, each drone agent has an actor network (i.e., the first actor network mentioned above), a target actor network, a first critic network, a first target critic network, a second critic network, and a second target critic network. The satellite agent also has an actor network (i.e., the second actor network mentioned above), a target actor network, a first critic network, a first target critic network, a second critic network, and a second target critic network. By using two sets of critic networks (critic1 and critic2) and corresponding target critic networks (target critic1 and target critic2), the smaller value of the two can be taken when calculating the target value, thereby suppressing the problem of network overestimation.

[0053] In order to obtain the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model, the network parameters of the MATD3 network and the TD3 network can be initialized by random initialization, and the capacity of the experience playback container of the UAV agent and the satellite agent can be initialized. The above model adopts a multi-round training method, each round includes a specified number of execution steps, and each step will generate a set of experience data.

[0054] At the beginning of each round of training, the system environment state needs to be initialized. The initialization content includes: setting the time slot to time slot 0, setting the task queue sequence of the satellite and the drone to empty, resetting the initial position of the drone, resetting the computing resources of the drone and the satellite, etc. After the initialization is completed, each agent can make corresponding actions according to its own observation state, and then determine its observation state in the next time slot, and calculate the rewards obtained by each agent according to its actions.

[0055] The embodiment of the present invention uses the observation state, action, first reward and observation state of the drone agent in the current time slot as a set of experience data of the drone agent (i.e., first experience data), and uses the observation state, action, second reward and observation state of the satellite agent in the current time slot as a set of experience data of the satellite agent (i.e., second experience data). The first experience data and the second experience data are entered into respective experience playback devices.

[0056] If the number of groups of first experience data in the drone agent experience replayer reaches a multiple of the first preset update step number, it is necessary to extract a first amount of first experience data from the drone agent experience replayer to train the MATD3 network so as to update the network parameters of the actor network and the critic network in the network.

[0057] If the number of groups of second experience data in the satellite agent experience replayer reaches a multiple of the second preset update step number, it is necessary to extract a second amount of second experience data from the satellite agent experience replayer to train the TD3 network so as to update the network parameters of the actor network and the critic network in the network.

[0058] After the network parameters are updated, if the number of groups of the first experience data in the drone agent experience replayer does not reach the target number, that is, the number of execution steps specified for a round of training is not reached, it is necessary to continue to generate the first experience data and the second experience data under the updated network parameters. Otherwise, it means that one round of training is completed, and the step of initializing the system environment state is returned to start a new round of training until the preset rounds are reached. The training can be ended to obtain the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model.

[0059] In an optional implementation, in the above step S202, under the current network parameters and system environment, continuously generating first experience data and storing it in the drone agent experience replay device, and continuously generating second experience data and storing it in the satellite agent experience replay device, specifically includes the following steps:

[0060] Step S2021, under the current network parameters and system environment, obtain the observation state of each drone agent in the MATD3 network in the current time slot, and determine the action of each drone agent in the current time slot based on the corresponding first performer network.

[0061] Step S2022, construct the observation state of the satellite agent in the current time slot based on the actions of all drone agents in the current time slot, and use the second performer network to process the observation state of the satellite agent in the current time slot to obtain the action of the satellite agent in the current time slot.

[0062] Step S2023, based on the observation state and action of each drone agent in the current time slot, determine the observation state of each drone agent in the next time slot, and determine the action of each drone agent in the next time slot based on the corresponding first performer network.

[0063] Step S2024, construct the observation state of the satellite agent in the next time slot based on the actions of all drone agents in the next time slot.

[0064] Step S2025, calculate the first reward obtained by each drone agent after performing the action in the current time slot and the second reward obtained by the satellite agent after performing the action in the current time slot.

[0065] Step S2026, stores the observation state, action, first reward and observation state of all drone agents in the current time slot as first experience data in the drone agent experience replay device, and stores the observation state, action, second reward and observation state of the satellite agent in the current time slot as second experience data in the satellite agent experience replay device.

[0066] Specifically, the observed state of the target UAV agent corresponding to the target UAV u at the current time slot t is The first performer network of the target UAV agent is used to determine its action in the current time slot: Then we get the set of task offloading ratio decisions for all drones: And get the set of observation states of all UAV agents in the current time slot:

[0067] The task offloading decision of the UAV will affect the size of the task uploaded to the satellite. Therefore, only after determining the actions of all UAV agents at the current time slot t, can the observation state of the satellite agent corresponding to the central satellite cs at the current time slot t be obtained: That is, the decision of the UAV agent affects the state of the satellite agent as a priori decision. After obtaining the observed state of the satellite agent at the current time slot t, the second actor network of the satellite agent is used to determine its action at the current time slot:

[0068] Each drone agent can determine its observation state in the next time slot based on its observation state and action in the current time slot, and then use its own first performer network to determine its action in the next time slot, and then construct the observation state of the satellite agent in the next time slot. According to the observation states of all UAV agents in the next time slot, the observation state set can be constructed

[0069] The drone agent can obtain the first reward after performing the action in the current time slot, and the satellite agent can obtain the second reward after performing the action in the current time slot. In the embodiment of the present invention, the first experience data is expressed as: The second empirical data is expressed as: in, represents the set of first rewards R1, and R2 represents the set of second rewards.

[0070] In an optional implementation manner, the training objective of the MATD3 network and the TD3 network is to minimize the system cost. In the embodiment of the present invention, the calculation process of the system cost includes the following steps:

[0071] Step S301, calculate the first total delay of the task that the target UAV leaves to be processed locally in the current time slot, and calculate the second total delay of the task that the target UAV offloads to the target satellite for processing in the current time slot; wherein the target satellite represents any satellite in the space-based network.

[0072] Step S302: Use the maximum delay among the first total delay and all the second total delays as the processing delay of the task generated by the target UAV in the current time slot.

[0073] Specifically, if the first total delay of the task that the target UAV u stays in the local processing in the current time slot t is The second total delay of the task unloaded from the target UAV u to the target satellite s in the current time slot t is T s_u_t,exe , then the processing delay of the task generated by the target UAV u in the current time slot t is: in,

[0074] Step S303, calculate the average value of the processing delay of the tasks generated by all drones in the current time slot, and use the average value as the delay cost.

[0075] The delay cost of the tasks generated by all drones in the current time slot is: Among them, N u Represents the total number of drones in the air-based network.

[0076] Step S304, calculating the first energy consumption of the task that the target UAV keeps for local processing in the current time slot, and calculating the second energy consumption of the task that the target UAV offloads to the target satellite for processing in the current time slot.

[0077] In the embodiment of the present invention, the first energy consumption of the task left by the target drone u for local processing in the current time slot t is Calculated by the following formula: Among them, κ u represents the energy consumption coefficient of the target UAV u, It represents the amount of tasks generated by the target UAV u in the current time slot t that are retained for local processing.

[0078] The second energy consumption E of the task that the target UAV u unloads to the target satellite s in the current time slot t s_u_t Calculated by the following formula: Among them, κ s represents the energy consumption coefficient of the target satellite s, d s_u_tIt represents the amount of tasks that the target UAV u unloads to the target satellite s for processing in the current time slot t.

[0079] Step S305, calculating the energy consumption cost based on the first energy consumption of all UAVs and the second energy consumption of all satellites in the current time slot.

[0080] The energy consumption cost of the tasks generated by all drones in the current time slot is: Among them, r u represents the energy consumption weight of the UAV processing task, r s represents the energy consumption weight of satellite processing tasks, r u +r s =1.

[0081] Step S306: Calculate the system cost of the current time slot based on the delay cost and the energy consumption cost.

[0082] The system cost is calculated as: Among them, w t represents the weight of the delay cost, w e represents the weight of energy consumption cost, w t +w e =1.

[0083] In an optional implementation manner, in the above step S301, calculating the first total delay of the task left by the target UAV for local processing in the current time slot specifically includes the following contents:

[0084] The processing time required by the task queue of the target UAV at the beginning of the current time slot is used as the first waiting delay of the task left by the target UAV for local processing in the current time slot. That is, It represents the first waiting delay of the task that the target UAV u stays to process locally in the current time slot t.

[0085] Based on the task offloading ratio decision of the target UAV in the current time slot and the amount of tasks generated by the target UAV in the current time slot, the amount of tasks left to be processed locally by the target UAV in the current time slot is determined. Specifically, the task offloading ratio decision of the target UAV u in the current time slot t is known. The ratio of the tasks generated by the target UAV u in the current time slot t to be processed locally is included in Will The amount of tasks generated by the target UAV u at the current time slot t By multiplying them, we can get the task amount generated by the target UAV u in the current time slot t and retained for local processing.

[0086] Based on the amount of tasks processed locally, the computing resources required by the target drone to process 1 bit of data, and the computing power of the target drone, the first computing delay of the task left for local processing by the target drone in the current time slot is determined. in, Represents the first computational delay of the task that the target UAV u leaves to process locally in the current time slot t.

[0087] Based on the first waiting delay and the first calculation delay, the first total delay of the task left by the target UAV for local processing in the current time slot is determined. The formula for the first total delay is:

[0088] In an optional implementation manner, in the above step S301, calculating the second total delay of the task unloaded from the target UAV to the target satellite for processing in the current time slot specifically includes the following contents:

[0089] Calculate the processing delay of the current task queue of the target satellite; the current task queue refers to the task queue of the target satellite before receiving the task unloaded by the target UAV. Specifically, since the satellite may need to process multiple tasks in each time slot, when a task b generated in time slot t needs to be processed, there is a task a generated in time slot t before task b, so there will be a queuing delay for the satellite to process tasks. Assuming that the task assigned to the target satellite s by the target UAV u in time slot t is the kth task on the target satellite s (the relationship between k and t is uncertain), then D can be set s,k represents the computing resources required by the target satellite s to complete the first k-1 tasks before starting to process the kth task, that is, D s,k Represents the computing resources required to process the current task queue; where, D s,k-1 represents the computing resources required by the target satellite s to complete the first k-2 tasks before starting to process the k-1th task. represents the task amount of the k-1th task in the current task queue on the target satellite s. It can be seen that the processing delay of the current task queue of the target satellite s is:

[0090] Calculate the transmission delay of the target UAV to unload the task to the target satellite. Specifically, the tasks unloaded by the target UAV to each satellite in the space-based network are allocated by the central satellite. If the target satellite is the central satellite, then the transmission delay of the target UAV u to unload the task to the target satellite in the current time slot t is for: in, represents the amount of tasks that the target UAV u unloads to the satellite for processing in the current time slot t. This amount of tasks is the proportion of the tasks generated by the target UAV u in the current time slot t that are unloaded to the satellite. The amount of tasks generated by the target UAV u at the current time slot t The result of multiplication, represents the uplink data transmission rate of the target UAV u in the current time slot t; if the target satellite is not the central satellite, the transmission delay of the target UAV to offload the task to the target satellite includes not only It also includes the inter-satellite transmission delay when the central satellite assigns the task to the target satellite. d s_u_t represents the amount of tasks that the target UAV u unloads to the target satellite s for processing in the current time slot t, Represents the data transmission rate between the central satellite and the target satellite s in the current time slot t.

[0091] The maximum delay of the processing delay and transmission delay of the current task queue is used as the second waiting delay of the task unloaded from the target UAV to the target satellite in the current time slot. That is, the second waiting delay of the task unloaded from the target UAV u to the target satellite s in the current time slot t is expressed as: in, Indicates that the target satellite s is the central satellite, Indicates that the target satellite s is not the central satellite.

[0092] Based on the amount of tasks that the target UAV unloads to the target satellite for processing in the current time slot, the computing resources required by the target satellite to process 1 bit of data, and the computing power of the target satellite, the second computing delay of the tasks that the target UAV unloads to the target satellite for processing in the current time slot is determined. That is, Among them, T s_u_t,com Represents the second computational delay of the task unloaded from the target UAV u to the target satellite s in the current time slot t.

[0093] Based on the second waiting delay and the second calculation delay, the second total delay of the task unloaded from the target UAV to the target satellite for processing in the current time slot is determined. The formula for the second total delay is: T s_u_t,exe =T s_u_t,wait +T s_u_t,com .

[0094] In an optional implementation, the above step S2025, calculating the first reward obtained by each drone agent after performing the action in the current time slot and the second reward obtained by the satellite agent after performing the action in the current time slot, specifically includes the following contents:

[0095] Based on the first total delay and the first energy consumption of the task left by the target UAV for local processing in the current time slot, the first total cost of the task left by the target UAV for local processing in the current time slot is calculated. In the embodiment of the present invention, the formula for the first total cost of the task left by the target UAV u for local processing in the current time slot t is:

[0096] Based on the second total delay and the second energy consumption of the task unloaded from the target UAV to each satellite in the current time slot, the second total cost of the task unloaded from the target UAV to the satellite in the current time slot is calculated. In the embodiment of the present invention, the formula for the second total cost of the task unloaded from the target UAV u to the satellite in the current time slot t is:

[0097] Based on the first total cost, the second total cost, the first waiting delay and the system cost, the first reward obtained by the drone agent corresponding to the target drone after executing the action in the current time slot is calculated. The formula for the first reward obtained by the drone agent corresponding to the target drone u after executing the action in the current time slot t is: c1 is the compensation coefficient to prevent the first reward R1 from being too high. Based on the formula of the first reward, it can be seen that the drone expects each task to be processed locally and as little as possible without queuing. Therefore, the first reward is positively correlated with the first total cost, and the first reward is negatively correlated with the second total cost and the first waiting delay.

[0098] Based on the second total cost and system cost of all tasks unloaded by all drones to the satellite for processing in the current time slot, the second reward obtained by the satellite agent after performing the action in the current time slot is calculated. The formula for the second reward obtained by the satellite agent after performing the action in the current time slot t is: Both c2 and c3 represent normalization coefficients, which are used to normalize the second reward. Both the first reward and the second reward are used to train the critic network of the corresponding agent, and the smaller Q value obtained by the critic network is used to train the actor network of the corresponding agent.

[0099] In summary, in order to reduce the complexity of the task deployment problem, the method provided by the embodiment of the present invention decomposes the problem into two layers of sub-problems. The first layer solves the task offloading ratio decision problem of the UAV, and the second layer solves the task allocation ratio decision problem of the satellite. At the same time, the decision of the first layer of sub-problems provides a priori decisions for the second layer of sub-problems. By using the hierarchical reinforcement learning framework to alternately optimize the process, not only the complexity of each sub-problem is reduced, but also the learning efficiency is improved. In addition, in order to improve the flexibility of the hierarchical reinforcement learning framework, based on the hierarchical problem, the embodiment of the present invention uses the reinforcement learning algorithms MATD3 and TD3 to solve the two-layer sub-problems. Therefore, the method provided by the embodiment of the present invention can obtain the real-time scheduling strategy of the UAV task in a dynamic environment, thereby generating efficient task offloading ratio and task allocation ratio decisions based on the dynamic system state. Compared with other existing algorithms, the method of the present invention has good performance in terms of task processing delay, energy consumption and operating performance, and can effectively improve the overall performance of the system.

[0100] Embodiment 2

[0101] An embodiment of the present invention also provides an integrated air-space task processing device based on hierarchical reinforcement learning, which is mainly used to execute the integrated air-space task processing method based on hierarchical reinforcement learning provided in the above-mentioned embodiment one. The following is a specific introduction to the integrated air-space task processing device based on hierarchical reinforcement learning provided in an embodiment of the present invention.

[0102] Figure 4 A functional module diagram of an air-space integrated task processing device based on hierarchical reinforcement learning provided by an embodiment of the present invention, such as Figure 4 As shown, the device mainly includes: a first determination module 10, a first processing module 20, a second determination module 30, a construction module 40, and a second processing module 50, wherein:

[0103] The first determination module 10 is used to determine the status of all drones in the current time slot; wherein the status of the target drone includes: the distance between the mapping point of the central satellite on the drone plane and the target drone, the processing time required for the task queue of the target drone at the start of the current time slot, the amount of tasks generated by the target drone in the current time slot, the index of the target drone and the time slot index; the target drone represents any drone among all drones in the air-based network; the central satellite represents the satellite with the least computing resources in the space-based network.

[0104] The first processing module 20 is used to process the states of all drones using the target multi-agent deep reinforcement learning model to obtain a task offloading ratio decision for each drone; wherein the target multi-agent deep reinforcement learning model includes multiple drone agents, each drone agent corresponds one-to-one to a drone in the air-based network, the observed state of the drone agent is the state of the corresponding drone, and the action of the drone agent is the task offloading ratio decision for the corresponding drone.

[0105] The second determination module 30 is used to determine the task queue processing time of each satellite in the space-based network and the computing resources required by the task queue at the start time of the current time slot.

[0106] The construction module 40 is used to construct the status of the central satellite based on the task queue processing time of all satellites, the computing resources required for the task queue, the task offloading ratio decision of all drones and the time slot index.

[0107] The second processing module 50 is used to process the state of the central satellite using the target single-agent deep reinforcement learning model to obtain the task allocation ratio decision on each satellite for the amount of tasks unloaded from each drone to the space-based network; wherein the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation ratio decision.

[0108] In view of the task deployment problem in the space-air integrated edge computing scenario, an embodiment of the present invention provides a space-air integrated task processing device based on hierarchical reinforcement learning, the first layer solves the task offloading problem of UAV tasks, and the second layer solves the task allocation problem of satellites. Specifically, after determining the status of all UAVs in the current time slot, the target multi-agent deep reinforcement learning model is first used to process the status of all UAVs to obtain the task offloading ratio decision of each UAV, thereby solving the task offloading problem of UAV tasks at the first layer, and then the status of the central satellite is constructed based on the task offloading ratio decision of all UAVs, and the target single-agent deep reinforcement learning model is used to process the status of the central satellite to obtain the task allocation ratio decision of each UAV unloaded to the space-based network on each satellite, thereby solving the task allocation problem of the satellite at the second layer, and then realizing the decomposition of the task deployment problem into the task offloading problem and the task allocation problem, effectively reducing the complexity of the space-air integrated task processing problem, reducing the dimension of the action space of the agent, thereby improving the convergence performance and processing efficiency of the space-air integrated task processing method.

[0109] Optionally, the target multi-agent deep reinforcement learning model is a plurality of first performer networks in a MATD3 network, and the target single-agent deep reinforcement learning model is a second performer network in a TD3 network; the device further comprises the following modules:

[0110] The initialization module is used to initialize the network parameters of the MATD3 network and the network parameters of the TD3 network.

[0111] The experience acquisition module is used to initialize the system environment state, and under the current network parameters and system environment state, continuously generate first experience data and store it in the UAV agent experience replay device, and continuously generate second experience data and store it in the satellite agent experience replay device; wherein the first experience data includes: the observation state, action, first reward of the UAV agent in the current time slot and the observation state of the next time slot, and the second experience data includes: the observation state, action, second reward of the satellite agent in the current time slot and the observation state of the next time slot.

[0112] The first training module is used to extract a first number of first experience data from the drone intelligent agent experience replayer to train the MATD3 network to update its network parameters when it is determined that the number of groups of first experience data in the drone intelligent agent experience replayer reaches a multiple of a first preset update step number.

[0113] The second training module is used to extract a second amount of second experience data from the satellite agent experience replayer to train the TD3 network and update its network parameters when it is determined that the number of groups of second experience data in the satellite agent experience replayer reaches a multiple of the second preset update step number.

[0114] The iteration module is used to continue generating the first experience data and the second experience data under the updated network parameters when it is determined that the network parameters of the MATD3 network or the network parameters of the TD3 network have been updated. If the number of groups of the first experience data in the drone agent experience replayer has not reached the target number, the iteration module returns to the step of initializing the system environment state until the preset rounds are reached to obtain the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model.

[0115] Optionally, the experience acquisition module includes:

[0116] The acquisition and determination unit is used to obtain the observation state of each drone agent in the MATD3 network in the current time slot under the current network parameters and system environment, and determine the action of each drone agent in the current time slot based on the corresponding first performer network.

[0117] The first construction unit is used to construct the observation state of the satellite agent in the current time slot based on the actions of all drone agents in the current time slot, and use the second performer network to process the observation state of the satellite agent in the current time slot to obtain the action of the satellite agent in the current time slot.

[0118] A determination unit is used to determine the observation state of each drone agent in the next time slot based on the observation state and action of each drone agent in the current time slot, and determine the action of each drone agent in the next time slot based on the corresponding first performer network.

[0119] The second construction unit is used to construct the observation state of the satellite agent in the next time slot based on the actions of all the drone agents in the next time slot.

[0120] The calculation unit is used to calculate the first reward obtained by each UAV agent after performing the action in the current time slot and the second reward obtained by the satellite agent after performing the action in the current time slot.

[0121] The storage unit is used to store the observation state, action, first reward and observation state of all drone agents in the current time slot as first experience data in the drone agent experience replay device, and to store the observation state, action, second reward and observation state of the satellite agent in the current time slot as second experience data in the satellite agent experience replay device.

[0122] Optionally, the training objective of the MATD3 network and the TD3 network is to minimize the system cost, and the apparatus further includes:

[0123] The first calculation module is used to calculate the first total delay of the task that the target UAV keeps processing locally in the current time slot, and to calculate the second total delay of the task that the target UAV unloads to the target satellite for processing in the current time slot; wherein the target satellite represents any satellite in the space-based network.

[0124] The third determination module is used to use the maximum delay among the first total delay and all the second total delays as the processing delay of the task generated by the target UAV in the current time slot.

[0125] The second calculation module is used to calculate the average processing delay of the tasks generated by all drones in the current time slot, and use the average value as the delay cost.

[0126] The third calculation module is used to calculate the first energy consumption of the task that the target UAV keeps for local processing in the current time slot, and to calculate the second energy consumption of the task that the target UAV offloads to the target satellite for processing in the current time slot.

[0127] The fourth calculation module is used to calculate the energy consumption cost based on the first energy consumption of all drones and the second energy consumption of all satellites in the current time slot.

[0128] The fifth calculation module is used to calculate the system cost of the current time slot based on the delay cost and the energy consumption cost.

[0129] Optionally, the first computing module is specifically used for:

[0130] The processing time required by the task queue of the target UAV at the beginning of the current time slot is used as the first waiting delay of the task left by the target UAV for local processing in the current time slot.

[0131] Based on the task offloading ratio decision of the target UAV in the current time slot and the amount of tasks generated by the target UAV in the current time slot, the amount of tasks that the target UAV leaves to process locally in the current time slot is determined.

[0132] Based on the amount of tasks processed locally, the computing resources required by the target UAV to process 1 bit of data, and the computing power of the target UAV, the first computing delay of the task that the target UAV leaves to process locally in the current time slot is determined.

[0133] Based on the first waiting delay and the first calculation delay, the first total delay of the task left to be processed locally by the target UAV in the current time slot is determined.

[0134] Optionally, the first computing module is further configured to:

[0135] Calculate the processing delay of the current task queue of the target satellite; wherein the current task queue represents the task queue of the target satellite before receiving the task unloaded by the target UAV.

[0136] Calculate the transmission delay for the target UAV to offload the task to the target satellite.

[0137] The maximum delay of the processing delay and the transmission delay of the current task queue is used as the second waiting delay of the task unloaded from the target UAV to the target satellite for processing in the current time slot.

[0138] Based on the amount of tasks that the target UAV offloads to the target satellite for processing in the current time slot, the computing resources required for the target satellite to process 1 bit of data, and the computing power of the target satellite, the second computing delay of the task that the target UAV offloads to the target satellite for processing in the current time slot is determined.

[0139] Based on the second waiting delay and the second calculation delay, the second total delay of the task unloaded from the target UAV to the target satellite for processing in the current time slot is determined.

[0140] Optionally, the computing unit is specifically configured to:

[0141] Based on the first total delay and the first energy consumption of the task that the target UAV leaves to process locally in the current time slot, the first total cost of the task that the target UAV leaves to process locally in the current time slot is calculated.

[0142] Based on the second total delay and second energy consumption of the tasks unloaded from the target UAV to each satellite for processing in the current time slot, the second total cost of the tasks unloaded from the target UAV to the satellite for processing in the current time slot is calculated.

[0143] Based on the first total cost, the second total cost, the first waiting delay and the system cost, the first reward obtained by the drone agent corresponding to the target drone after performing the action in the current time slot is calculated.

[0144] Based on all the second total costs and system costs of the tasks offloaded by all UAVs to the satellite for processing in the current time slot, the second reward obtained by the satellite agent after performing the action in the current time slot is calculated.

[0145] Embodiment 3

[0146] See also Figure 5An embodiment of the present invention provides an electronic device, which includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.

[0147] The memory 61 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0148] The bus 62 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0149] Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device defined by the process disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0150] The processor 60 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 60. The above processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.

[0151] A computer program product of a method and device for integrated air-space mission processing based on hierarchical reinforcement learning provided in an embodiment of the present invention includes a computer-readable storage medium storing non-volatile program codes executable by a processor. The instructions included in the program code can be used to execute the methods in the previous method embodiments. For specific implementations, please refer to the method embodiments, which will not be repeated here.

[0152] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0153] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0154] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0155] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0156] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0157] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing air-space integrated tasks based on hierarchical reinforcement learning, characterized in that: include: Determine the status of all drones in the current time slot; wherein the status of the target drone includes: the distance between the mapping point of the central satellite on the drone plane and the target drone, the processing time required for the task queue of the target drone at the start of the current time slot, the task volume generated by the target drone in the current time slot, the index of the target drone and the time slot index; the target drone represents any drone among all drones in the air-based network; the central satellite represents the satellite with the least computing resources in the space-based network; The target multi-agent deep reinforcement learning model is used to process the states of all drones to obtain a task offloading ratio decision for each drone; wherein the target multi-agent deep reinforcement learning model includes multiple drone agents, each drone agent corresponds one-to-one to a drone in the air-based network, the observed state of the drone agent is the state of the corresponding drone, and the action of the drone agent is the task offloading ratio decision for the corresponding drone; Determine the task queue processing time and the computing resources required for the task queue of each satellite in the space-based network at the start time of the current time slot; Constructing the state of the central satellite based on the task queue processing time of all satellites, the computing resources required by the task queue, the task offloading ratio decision of all drones and the time slot index; The state of the central satellite is processed by using a target single-agent deep reinforcement learning model to obtain a task allocation ratio decision on each satellite for the amount of tasks unloaded from each drone to the space-based network; wherein the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation ratio decision.

2. The method for processing air-space integrated tasks based on hierarchical reinforcement learning according to claim 1, characterized in that: The target multi-agent deep reinforcement learning model is a plurality of first performer networks in the MATD3 network, and the target single-agent deep reinforcement learning model is a second performer network in the TD3 network; Before determining the status of all drones in the current time slot, it also includes: Initialize the network parameters of the MATD3 network and the TD3 network; Initialize the system environment state, and under the current network parameters and system environment state, continuously generate first experience data and store it in the UAV agent experience replay device, and continuously generate second experience data and store it in the satellite agent experience replay device; wherein the first experience data includes: the observation state, action, first reward and observation state of the UAV agent in the current time slot, and the second experience data includes: the observation state, action, second reward and observation state of the satellite agent in the current time slot; When it is determined that the number of groups of first experience data in the drone agent experience replayer reaches a multiple of a first preset update step number, extracting a first quantity of first experience data from the drone agent experience replayer to train the MATD3 network to update its network parameters; In the case where it is determined that the number of groups of second experience data in the satellite agent experience replayer reaches a multiple of a second preset update step number, extracting a second amount of second experience data from the satellite agent experience replayer to train the TD3 network to update its network parameters; When it is determined that the network parameters of the MATD3 network or the network parameters of the TD3 network have been updated, if the number of groups of first experience data in the drone agent experience playback device has not reached the target number, then under the updated network parameters, continue to generate the first experience data and the second experience data; otherwise, return to the step of initializing the system environment state until the preset rounds are reached to obtain the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model.

3. The method for processing air-space integrated tasks based on hierarchical reinforcement learning according to claim 2, characterized in that: Under the current network parameters and system environment, continuously generating first experience data and storing it in the unmanned aerial vehicle agent experience replay device, and continuously generating second experience data and storing it in the satellite agent experience replay device, including: Under the current network parameters and system environment, obtain the observation state of each drone agent in the MATD3 network in the current time slot, and determine the action of each drone agent in the current time slot based on the corresponding first performer network; Constructing the observation state of the satellite agent in the current time slot based on the actions of all drone agents in the current time slot, and processing the observation state of the satellite agent in the current time slot using the second performer network to obtain the action of the satellite agent in the current time slot; Determine the observation state of each drone agent in the next time slot based on the observation state and action of each drone agent in the current time slot, and determine the action of each drone agent in the next time slot based on the corresponding first performer network; Constructing the observation state of the satellite agent in the next time slot based on the actions of all drone agents in the next time slot; Calculate a first reward obtained by each of the UAV agents after performing an action in the current time slot and a second reward obtained by each of the satellite agents after performing an action in the current time slot; The observation state, action, first reward and observation state of the next time slot of all the drone agents in the current time slot are stored in the drone agent experience replay device as the first experience data, and the observation state, action, second reward and observation state of the satellite agent in the current time slot are stored in the satellite agent experience replay device as the second experience data.

4. The method for processing space-air integrated tasks based on hierarchical reinforcement learning according to claim 3 is characterized in that: The training objectives of the MATD3 network and the TD3 network are to minimize the system cost. The calculation process of the system cost includes: Calculate a first total delay of the task that the target UAV leaves to be processed locally in the current time slot, and calculate a second total delay of the task that the target UAV offloads to the target satellite for processing in the current time slot; wherein the target satellite represents any satellite in the space-based network; The maximum delay among the first total delay and all the second total delays is used as the processing delay of the task generated by the target UAV in the current time slot; Calculate the average processing delay of tasks generated by all drones in the current time slot, and use the average as the delay cost; Calculate the first energy consumption of the task that the target UAV keeps for local processing in the current time slot, and calculate the second energy consumption of the task that the target UAV offloads to the target satellite for processing in the current time slot; Calculate the energy consumption cost based on the first energy consumption of all drones and the second energy consumption of all satellites in the current time slot; Based on the delay cost and the energy consumption cost, the system cost of the current time slot is calculated.

5. The method for processing space-air integrated tasks based on hierarchical reinforcement learning according to claim 4 is characterized in that: Calculating the first total delay of the task that the target UAV remains to process locally in the current time slot, including: The processing time required by the task queue of the target UAV at the start time of the current time slot is used as the first waiting delay of the task left by the target UAV for local processing in the current time slot; Based on the task offloading ratio decision of the target UAV in the current time slot and the task amount generated by the target UAV in the current time slot, determine the task amount of the target UAV to be processed locally in the current time slot; Determine a first computing delay of the task left for local processing by the target drone in the current time slot based on the task amount processed locally, the computing resources required for the target drone to process 1 bit of data, and the computing capacity of the target drone; Based on the first waiting delay and the first calculation delay, a first total delay of the task left to be processed locally by the target UAV in the current time slot is determined.

6. The method for processing space-air integrated tasks based on hierarchical reinforcement learning according to claim 5, characterized in that: Calculating the second total delay of the task unloaded from the target UAV to the target satellite for processing in the current time slot, including: Calculate the processing delay of the current task queue of the target satellite; wherein the current task queue represents the task queue of the target satellite before receiving the task unloaded by the target UAV; Calculating the transmission delay of the target UAV to offload the task to the target satellite; The maximum delay of the processing delay of the current task queue and the transmission delay is used as the second waiting delay of the task that the target UAV is unloaded to the target satellite for processing in the current time slot; Determine a second computing delay of the task that is unloaded from the target UAV to the target satellite for processing in the current time slot based on the amount of tasks that are unloaded from the target UAV to the target satellite for processing in the current time slot, the computing resources required for the target satellite to process 1 bit of data, and the computing capacity of the target satellite; Based on the second waiting delay and the second calculation delay, a second total delay for the task unloaded from the target UAV to the target satellite for processing in the current time slot is determined.

7. The method for processing space-air integrated tasks based on hierarchical reinforcement learning according to claim 6, characterized in that: Calculating a first reward obtained by each of the UAV agents after performing an action in the current time slot and a second reward obtained by the satellite agent after performing an action in the current time slot, including: Calculate a first total cost of the task that the target UAV leaves to be processed locally in the current time slot based on a first total delay and a first energy consumption of the task that the target UAV leaves to be processed locally in the current time slot; Calculate a second total cost of the task unloaded from the target UAV to the satellite for processing in the current time slot based on the second total delay and the second energy consumption of the task unloaded from the target UAV to each satellite for processing in the current time slot; Based on the first total cost, the second total cost, the first waiting delay and the system cost, calculating a first reward obtained by the drone agent corresponding to the target drone after executing an action in the current time slot; Based on all second total costs of tasks offloaded by all drones to satellites for processing in the current time slot and the system cost, the second reward obtained by the satellite agent after performing the action in the current time slot is calculated.

8. An air-space integrated task processing device based on hierarchical reinforcement learning, characterized in that: include: The first determination module is used to determine the status of all drones in the current time slot; wherein the status of the target drone includes: the distance between the mapping point of the central satellite on the drone plane and the target drone, the computing resources required by the task queue of the target drone at the start time of the current time slot, the task volume generated by the target drone in the current time slot, the index of the target drone and the time slot index; the target drone represents any drone among all drones in the air-based network; the central satellite represents the satellite with the least computing resources in the space-based network; The first processing module is used to process the states of all drones using a target multi-agent deep reinforcement learning model to obtain a task offloading ratio decision for each drone; wherein the target multi-agent deep reinforcement learning model includes a plurality of drone agents, each drone agent corresponds one-to-one to a drone in the air-based network, the observed state of the drone agent is the state of the corresponding drone, and the action of the drone agent is the task offloading ratio decision for the corresponding drone; A second determination module is used to determine the task queue processing time and the computing resources required for the task queue of each satellite in the space-based network at the start time of the current time slot; A construction module, configured to construct the state of the central satellite based on the task queue processing time of all satellites, the computing resources required by the task queue, the task offloading ratio decision of all drones and the time slot index; The second processing module is used to process the state of the central satellite using the target single-agent deep reinforcement learning model to obtain the task allocation ratio decision on each satellite for the amount of tasks unloaded from each drone to the space-based network; wherein the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation ratio decision.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method for processing integrated air and space tasks based on hierarchical reinforcement learning as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the air-space integrated task processing method based on hierarchical reinforcement learning as described in any one of claims 1 to 7.

Images