Space-air integrated task processing method and device based on hierarchical reinforcement learning

Abstract

The application provides a space-air integrated task processing method and device based on hierarchical reinforcement learning, relates to the technical field of mobile edge computing, and comprises the following steps: after determining the states of all unmanned aerial vehicles (UAVs) in a current time slot, the states of all UAVs are processed by using a target multi-agent deep reinforcement learning model to obtain a task offloading ratio decision of each UAV, then the state of a central satellite is constructed based on the task offloading ratio decisions of all UAVs, and the state of the central satellite is processed by using a target single-agent deep reinforcement learning model to obtain a task allocation ratio decision of the task quantity of each UAV offloaded to the space network on each satellite. The method effectively reduces the complexity of the space-air integrated task processing problem by decomposing the task deployment problem into a task offloading problem and a task allocation problem, reduces the action space dimension of the agent, and thus improves the convergence performance and processing efficiency of the space-air integrated task processing method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mobile edge computing, and in particular to a space-air integrated task processing method and device based on hierarchical reinforcement learning. BACKGROUND

[0002] With the continuous development of 5G, 6G has been proposed, aiming to extend the ground mobile communication network to the sky, and build a super wideband mobile communication system connecting the sky, sea and land. The space-air-ground integrated network has become one of the core research architectures of 6G network. In addition, in the ground communication network, mobile edge computing provides a new paradigm for various Internet of Things applications, providing lower latency and more secure transmission compared to cloud computing. Space-air integrated edge computing, as a new paradigm combining edge computing with space-based networks, has attracted widespread attention due to its potential to support various applications requiring low latency and high throughput in dynamic environments. However, efficient resource management, including task allocation and computation offloading, remains a key challenge for space-air integrated edge computing systems.

[0003] Deep reinforcement learning combines the perception ability of deep learning and the decision-making ability of reinforcement learning, realizes end-to-end learning, and can solve various complex problems in real scenarios, which has attracted widespread attention. In recent years, many studies have applied deep reinforcement learning to space-air integrated edge computing scenarios. However, as the optimization problem becomes more complex, the state and action space of the agent may grow significantly, leading to the curse of dimensionality and posing challenges to algorithm convergence and efficiency. In summary, the existing space-air integrated task processing method has the technical problems of high convergence difficulty and low processing efficiency. SUMMARY

[0004] The present application relates to the technical field of mobile edge computing, and in particular to a space-air integrated task processing method and device based on hierarchical reinforcement learning.

[0005] In a first aspect, the present application provides a space-air integrated task processing method based on hierarchical reinforcement learning, comprising: determining the state of all unmanned aerial vehicles (UAVs) in a current time slot; wherein the state of a target UAV includes the distance between the mapping point of a central satellite in the UAV plane and the target UAV, the processing time required by the task queue of the target UAV at the start time of the current time slot, the task quantity generated by the target UAV in the current time slot, the index of the target UAV, and the time slot index; the target UAV represents any UAV in all UAVs in the air-based network; the central satellite represents the satellite with the least computing resources in the space-based network; processing the state of all UAVs by using a target multi-agent deep reinforcement learning model to obtain the task offloading ratio decision of each UAV; wherein the target multi-agent deep reinforcement learning model includes a plurality of UAV agents, each UAV agent corresponds to a UAV in the air-based network, the observation state of the UAV agent is the state of the corresponding UAV, and the action of the UAV agent is the task offloading ratio decision of the corresponding UAV; determining the task queue processing time and the required computing resources of the task queue of each satellite in the space-based network at the start time of the current time slot; constructing the state of the central satellite based on the task queue processing time, the required computing resources of the task queue of all satellites, the task offloading ratio decision of all UAVs, and the time slot index; processing the state of the central satellite by using a target single-agent deep reinforcement learning model to obtain the task allocation ratio decision of the task quantity offloaded by each UAV to the space-based network on each satellite; wherein the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation ratio decision.

[0006] Optionally, the target multi-agent deep reinforcement learning model is a plurality of first actor networks in a MATD3 network, and the target single-agent deep reinforcement learning model is a second actor network in a TD3 network; before determining the state of all UAVs in the current time slot, the method further comprises: initializing network parameters of the MATD3 network and network parameters of the TD3 network; initializing a system environment state, continuously generating first experience data under the current network parameters and the system environment state and storing the first experience data in a UAV agent experience replay buffer, and continuously generating second experience data and storing the second experience data in a satellite agent experience replay buffer; wherein the first experience data comprises an observation state of the UAV agent in the current time slot, an action, a first reward, and an observation state of the next time slot, and the second experience data comprises an observation state of the satellite agent in the current time slot, an action, a second reward, and an observation state of the next time slot; in a case where the number of groups of the first experience data in the UAV agent experience replay buffer reaches a multiple of a first preset update step number, a first number of the first experience data is extracted from the UAV agent experience replay buffer to train the MATD3 network to update the network parameters thereof; in a case where the number of groups of the second experience data in the satellite agent experience replay buffer reaches a multiple of a second preset update step number, a second number of the second experience data is extracted from the satellite agent experience replay buffer to train the TD3 network to update the network parameters thereof; in a case where the network parameters of the MATD3 network or the network parameters of the TD3 network are updated, if the number of groups of the first experience data in the UAV agent experience replay buffer does not reach a target number, the first experience data and the second experience data are continuously generated under the updated network parameters; otherwise, the step of initializing the system environment state is returned until a preset round is reached, and the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model are obtained.

[0007] Optionally, under the current network parameters and system environment, the first experience data is continuously generated and stored in the UAV agent experience replay, and the second experience data is continuously generated and stored in the satellite agent experience replay, including: under the current network parameters and system environment, the observation state of each UAV agent in the MATD3 network at the current time slot is obtained, and the action of each UAV agent at the current time slot is determined based on the corresponding first actor network; the observation state of the satellite agent at the current time slot is constructed based on the actions of all UAV agents at the current time slot, and the observation state of the satellite agent at the current time slot is processed by the second actor network to obtain the action of the satellite agent at the current time slot; the observation state of each UAV agent at the next time slot is determined based on the observation state and action of each UAV agent at the current time slot, and the action of each UAV agent at the next time slot is determined based on the corresponding first actor network; the observation state of the satellite agent at the next time slot is constructed based on the actions of all UAV agents at the next time slot; the first reward obtained after each UAV agent executes the action at the current time slot and the second reward obtained after the satellite agent executes the action at the current time slot are calculated; the observation state, action, first reward and observation state at the next time slot of all UAV agents at the current time slot are stored as first experience data in the UAV agent experience replay, and the observation state, action, second reward and observation state at the next time slot of the satellite agent at the current time slot are stored as second experience data in the satellite agent experience replay.

[0008] Optionally, the training target of the MATD3 network and the TD3 network is the minimization of the system cost, and the calculation process of the system cost includes: calculating the first total time delay of the task left by the target UAV for local processing at the current time slot, and calculating the second total time delay of the task unloaded by the target UAV to the target satellite for processing at the current time slot; wherein the target satellite represents any satellite in the space-based network; the first total time delay and the maximum time delay in all second total time delays are taken as the processing time delay of the task generated by the target UAV at the current time slot; the average value of the processing time delays of all UAVs at the current time slot is calculated, and the average value is taken as the time delay cost; the first energy consumption of the task left by the target UAV for local processing at the current time slot is calculated, and the second energy consumption of the task unloaded by the target UAV to the target satellite for processing at the current time slot is calculated; the energy consumption cost is calculated based on the first energy consumption of all UAVs at the current time slot and the second energy consumption of all satellites; the system cost at the current time slot is calculated based on the time delay cost and the energy consumption cost.

[0009] Optionally, the first total delay of the task left by the target UAV for local processing in the current time slot is calculated, including: taking the processing time required by the task queue of the target UAV at the start of the current time slot as the first waiting delay of the task left by the target UAV for local processing in the current time slot; determining the amount of tasks left by the target UAV for local processing in the current time slot based on the task offloading proportion of the target UAV in the current time slot and the amount of tasks generated by the target UAV in the current time slot; determining the first calculation delay of the task left by the target UAV for local processing in the current time slot based on the amount of tasks for local processing, the computing resources required by the target UAV for processing 1-bit data, and the computing capacity of the target UAV; and determining the first total delay of the task left by the target UAV for local processing in the current time slot based on the first waiting delay and the first calculation delay.

[0010] Optionally, the second total delay of the task offloaded by the target UAV to the target satellite for processing in the current time slot is calculated, including: calculating the processing delay of the current task queue of the target satellite; wherein the current task queue represents the task queue of the target satellite before receiving the task offloaded by the target UAV; calculating the transmission delay of the task offloaded by the target UAV to the target satellite; taking the maximum delay between the processing delay of the current task queue and the transmission delay as the second waiting delay of the task offloaded by the target UAV to the target satellite for processing in the current time slot; determining the second calculation delay of the task offloaded by the target UAV to the target satellite for processing in the current time slot based on the amount of tasks offloaded by the target UAV to the target satellite for processing in the current time slot, the computing resources required by the target satellite for processing 1-bit data, and the computing capacity of the target satellite; and determining the second total delay of the task offloaded by the target UAV to the target satellite for processing in the current time slot based on the second waiting delay and the second calculation delay.

[0011] Optionally, the first reward obtained after each UAV agent performs an action in the current time slot and the second reward obtained after the satellite agent performs an action in the current time slot are calculated, including: calculating the first total cost of the task left by the target UAV for local processing in the current time slot based on the first total delay and the first energy consumption; calculating the second total cost of the task offloaded by the target UAV to the satellite for processing in the current time slot based on the second total delay and the second energy consumption; calculating the first reward obtained after the target UAV corresponding UAV agent performs an action in the current time slot based on the first total cost, the second total cost, the first waiting delay, and the system cost; and calculating the second reward obtained after the satellite agent performs an action in the current time slot based on all the second total costs of the tasks offloaded by all the UAVs to the satellite for processing in the current time slot and the system cost.

[0012] In a second aspect, the present application provides a space-air integrated task processing device based on hierarchical reinforcement learning, comprising: a first determining module configured to determine the states of all unmanned aerial vehicles (UAVs) in a current time slot; wherein the state of a target UAV includes the distance between the mapping point of a central satellite in a UAV plane and the target UAV, the computing resources required by the task queue of the target UAV at the start time of the current time slot, the task quantity generated by the target UAV in the current time slot, the index of the target UAV, and the time slot index; the target UAV represents any UAV in all UAVs in an air-based network; the central satellite represents a satellite with the least computing resources in a space-based network; a first processing module configured to process the states of all UAVs by using a target multi-agent deep reinforcement learning model to obtain the task offloading ratio decision of each UAV; wherein the target multi-agent deep reinforcement learning model includes a plurality of UAV agents, each UAV agent corresponds to a UAV in the air-based network, the observation state of the UAV agent is the state of the corresponding UAV, and the action of the UAV agent is the task offloading ratio decision of the corresponding UAV; a second determining module configured to determine the task queue processing time length and the computing resources required by the task queue of each satellite in the space-based network at the start time of the current time slot; a constructing module configured to construct the state of the central satellite based on the task queue processing time length, the computing resources required by the task queue of all satellites, the task offloading ratio decision of all UAVs, and the time slot index; and a second processing module configured to process the state of the central satellite by using a target single-agent deep reinforcement learning model to obtain the task allocation ratio decision of the task quantity offloaded by each UAV to the space-based network on each satellite; wherein the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation ratio decision.

[0013] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor implements the aforementioned space-air integrated task processing method based on hierarchical reinforcement learning when executing the computer program.

[0014] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer instructions, and the computer instructions are executed by a processor to implement the aforementioned space-air integrated task processing method based on hierarchical reinforcement learning.

[0015] For the task deployment problem in the space-air integration edge computing scene, the application provides a space-air integration task processing method based on hierarchical reinforcement learning. The first layer solves the task offloading problem of unmanned aerial vehicles, and the second layer solves the task allocation problem of satellites. Specifically, after determining the states of all unmanned aerial vehicles in the current time slot, the states of all unmanned aerial vehicles are processed by using a target multi-agent deep reinforcement learning model to obtain the task offloading ratio decision of each unmanned aerial vehicle, thereby solving the task offloading problem of unmanned aerial vehicles in the first layer. Then, the state of the central satellite is constructed based on the task offloading ratio decision of all unmanned aerial vehicles, and the state of the central satellite is processed by using a target single-agent deep reinforcement learning model to obtain the task allocation ratio decision of the task quantity unloaded by each unmanned aerial vehicle to the space-based network on each satellite, thereby solving the task allocation problem of satellites in the second layer. Thus, the task deployment problem is decomposed into the task offloading problem and the task allocation problem, the complexity of the space-air integration task processing problem is effectively reduced, the action space dimension of the agent is reduced, and the convergence performance and processing efficiency of the space-air integration task processing method are improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the specific embodiments or the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 A flowchart of a space-air integration task processing method based on hierarchical reinforcement learning provided by an embodiment of the application;

[0018] Figure 2 A scene model schematic diagram of space-air integration edge computing;

[0019] Figure 3 A framework schematic diagram of the space-air integration task processing method based on hierarchical reinforcement learning;

[0020] Figure 4 A functional module diagram of a space-air integration task processing device based on hierarchical reinforcement learning provided by an embodiment of the application;

[0021] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the application. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0024] Some embodiments of the present application will be described in detail below with reference to the drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0025] Embodiment one

[0026] Figure 1 A flow chart of a space-air integrated task processing method based on hierarchical reinforcement learning provided by the embodiments of the present application is shown in Figure 1 The method specifically includes the following steps:

[0027] Step S102, determining the state of all unmanned aerial vehicles in the current time slot.

[0028] The state of the target unmanned aerial vehicle includes: the distance between the mapping point of the central satellite in the unmanned aerial vehicle plane and the target unmanned aerial vehicle, the processing time required by the task queue of the target unmanned aerial vehicle at the start time of the current time slot, the task amount generated by the target unmanned aerial vehicle in the current time slot, the index of the target unmanned aerial vehicle and the time slot index; the target unmanned aerial vehicle represents any unmanned aerial vehicle in all unmanned aerial vehicles in the air-based network; the central satellite represents the satellite with the least computing resources in the space-based network.

[0029] Specifically, for the task deployment problem in the space-air integrated edge computing scene, the embodiments of the present application propose a hierarchical reinforcement learning framework, Figure 2 A scene model schematic diagram of space-air integrated edge computing is shown in accordance with Figure 2It can be known that the system is composed of three parts: a space-based network composed of multiple low-orbit satellites, an air-based network composed of multiple unmanned aerial vehicles, and a ground terminal network, and the ground terminal network can only communicate through the air-based network. Specifically, each unmanned aerial vehicle group serves a ground area and provides communication coverage and computing support for users on the ground. Each satellite provides services for a group of unmanned aerial vehicles and provides communication coverage and computing support for the unmanned aerial vehicles. Each unmanned aerial vehicle generates a task in each time slot, and the unmanned aerial vehicle can process part of the task locally and select another part to be unloaded to the satellite for processing. At the same time, since different satellites need to process different amounts of tasks, a central satellite with insufficient computing resources can distribute tasks to satellites with sufficient computing resources for collaborative processing through inter-satellite links. In the present application and embodiments, it is assumed that the system is quasi-static, that is, the environmental factors such as wireless channel gain and position of the unmanned aerial vehicle remain unchanged within a time slot.

[0030] The method provided by the embodiments of the present application can provide a task deployment scheme for any time slot within a task period of the space-air integrated network. To perform task deployment in the current time slot, the state of all unmanned aerial vehicles in the current time slot t needs to be determined first, and the state of the target unmanned aerial vehicle u in the current time slot t is The state includes: the distance between the mapping point of the central satellite in the unmanned aerial vehicle plane and the target unmanned aerial vehicle u The processing time length required by the task queue of the target unmanned aerial vehicle u at the start time of the current time slot t The amount of tasks generated by the target unmanned aerial vehicle u in the current time slot t The index u of the target unmanned aerial vehicle and the time slot index t.

[0031] Specifically, in the embodiments of the present application, the three-dimensional spatial position coordinates of the central satellite and the three-dimensional spatial position coordinates of each unmanned aerial vehicle are known, and all the unmanned aerial vehicles are at the same flight altitude, so the three-dimensional spatial position coordinates of the mapping point of the central satellite in the unmanned aerial vehicle plane can be determined, and the distance can be calculated according to the distance formula

[0032] The processing time length required by the task queue of the target unmanned aerial vehicle u at the start time of the current time slot t Wherein, represents the computing resources (unit: cycles) required by the task queue of the target unmanned aerial vehicle u at the start time of the current time slot t, f u represents the computing capability (unit: GHz) of the target unmanned aerial vehicle u, represents the computing resources required by the task queue of the target unmanned aerial vehicle u at the start time of the t-1 time slot, and if is equal to 0, it indicates that there is no queued task in the previous time slot, and δ represents the length of a time slot, udenotes the computing resource required by the target UAV u to process 1 bit of data (unit: cycles / bit), denotes the amount of tasks generated by the target UAV u in the t-1 time slot and reserved for local processing (unit: bit), denotes the computing resource that the target UAV u can process in one time slot, denotes the computing resource required by the target UAV u to process the task.

[0033] Step S104, processing the states of all UAVs by using the target multi-agent deep reinforcement learning model to obtain the task offloading ratio decision of each UAV.

[0034] Among them, the target multi-agent deep reinforcement learning model includes a plurality of UAV agents, each UAV agent corresponds to a UAV in the space-based network, the observation state of the UAV agent is the state of the corresponding UAV, and the action of the UAV agent is the task offloading ratio decision of the corresponding UAV.

[0035] Figure 3 It is a schematic diagram of the framework of the space-air integrated task processing method based on hierarchical reinforcement learning, according to Figure 3 It can be known that the task processing problem in the space-air integrated network is decomposed into two sub-problems in the embodiment of the present application, the first layer solves the task offloading problem of the UAV task, and the second layer solves the task allocation problem of the satellite. In the embodiment of the present application, the first layer adopts a multi-agent deep reinforcement learning model, and the second layer adopts a single-agent deep reinforcement learning model. By pre-training the models in the first layer and the second layer, the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model can be obtained.

[0036] The target multi-agent deep reinforcement learning model includes a plurality of UAV agents, the number of UAV agents is consistent with and corresponds to the number of UAVs in the space-based network. In order to obtain the task offloading ratio decision of the target UAV, only the state of the target UAV needs to be input into the corresponding UAV agent, and the action output by the UAV agent is the task offloading ratio decision of the target UAV Among them, denotes the task offloading ratio decision of the target UAV u in the current time slot t, denotes the proportion of the tasks generated by the target UAV u in the current time slot t and offloaded to the satellite, denotes the proportion of the tasks generated by the target UAV u in the current time slot t and reserved for local processing,

[0037] Step S106, determining the task queue processing time length and the computing resource required by the task queue of each satellite in the space-based network at the start time of the current time slot.

[0038] Step S108, based on the task queue processing time length of all satellites, the required computing resources of the task queue, the task offloading ratio decision of all UAVs and the time slot index, the state of the center satellite is constructed.

[0039] In the embodiment of the application, the state of the center satellite cs at the current time slot t Wherein, represents the set of task queue processing time lengths of all satellites in the space-based network at the start time of the current time slot t, represents the set of computing resources required by the task queue of all satellites in the space-based network at the start time of the current time slot t, represents the set of task offloading ratio decisions of all UAVs, represents the set of UAVs.

[0040] Wherein, the task queue processing time length of the target satellite s in the space-based network at the start time of the current time slot t Wherein, represents the computing resources required by the task queue of satellite s at the start time of the current time slot t (unit: cycles), f s represents the computing capacity of satellite s (unit: GHz), represents the computing resources required by the task queue of satellite s at the start time of the t-1 time slot, if equal to 0, indicating that there is no queued task in the previous time slot, s represents the computing resources required by satellite s to process 1 bit of data (unit: cycles / bit), represents the total amount of tasks (bit) allocated to satellite s in the t-1 time slot, represents the computing resources (cycles) that satellite can process in a time slot, represents the computing resources required to process .

[0041] Step S110, the state of the center satellite is processed by using the target single-agent deep reinforcement learning model, and the task allocation ratio decision of the task offloaded by each UAV to the space-based network on each satellite is obtained.

[0042] Wherein, the satellite agent in the target single-agent deep reinforcement learning model corresponds to the center satellite, the observation state of the satellite agent is the state of the center satellite, and the action of the satellite agent is the task allocation ratio decision.

[0043] The target single-agent deep reinforcement learning model has only one satellite agent, and the agent corresponds to a central satellite. After obtaining the state of the central satellite, the state of the central satellite is input into the satellite agent, and the action output by the satellite agent is the task allocation ratio decision of the task quantity unloaded by each unmanned aerial vehicle into the space-based network on each satellite wherein, represents a set of task allocation ratios of the task quantity unloaded by the target unmanned aerial vehicle u into the space-based network on all satellites in the current time slot t, represents the task allocation ratio of the task quantity unloaded by the target unmanned aerial vehicle u into the space-based network on the satellite S in the current time slot t, represents a set of satellites in the space-based network.

[0044] For the task deployment problem in the space-air integration edge computing scene, the present application provides a space-air integration task processing method based on hierarchical reinforcement learning. The first layer solves the task offloading problem of unmanned aerial vehicle tasks, and the second layer solves the task allocation problem of satellites. Specifically, after determining the state of all unmanned aerial vehicles in the current time slot, the state of all unmanned aerial vehicles is processed by using a target multi-agent deep reinforcement learning model to obtain the task offloading ratio decision of each unmanned aerial vehicle, thereby solving the task offloading problem of unmanned aerial vehicle tasks in the first layer. Then, the state of the central satellite is constructed based on the task offloading ratio decision of all unmanned aerial vehicles, and the state of the central satellite is processed by using a target single-agent deep reinforcement learning model to obtain the task allocation ratio decision of the task quantity unloaded by each unmanned aerial vehicle into the space-based network on each satellite, thereby solving the task allocation problem of satellites in the second layer. Thus, the task deployment problem is decomposed into the task offloading problem and the task allocation problem, the complexity of the space-air integration task processing problem is effectively reduced, the action space dimension of the agent is reduced, and the convergence performance and processing efficiency of the space-air integration task processing method are improved.

[0045] In the above, the method of applying the model to complete task deployment after the training of the first layer and the second layer model is described in detail. The following describes how to obtain the model in the first layer and the second layer.

[0046] In an optional embodiment, the target multi-agent deep reinforcement learning model is a plurality of first actor networks in a MATD3 network, and the target single-agent deep reinforcement learning model is a second actor network in a TD3 network. Before determining the state of all unmanned aerial vehicles in the current time slot, the method further includes the following steps:

[0047] Step S201, initializing the network parameters of the MATD3 network and the network parameters of the TD3 network.

[0048] Step S202, initialize the system environment state, continuously generate the first experience data into the unmanned aerial vehicle agent experience replayer under the current network parameters and the system environment state, and continuously generate the second experience data into the satellite agent experience replayer; wherein the first experience data includes: the observation state, action, first reward and next time slot observation state of the unmanned aerial vehicle agent in the current time slot, and the second experience data includes: the observation state, action, second reward and next time slot observation state of the satellite agent in the current time slot.

[0049] Step S203, in the case where the number of groups of the first experience data in the unmanned aerial vehicle agent experience replayer reaches a multiple of the first preset update step number, extract the first number of the first experience data from the unmanned aerial vehicle agent experience replayer to train the MATD3 network to update the network parameters thereof.

[0050] Step S204, in the case where the number of groups of the second experience data in the satellite agent experience replayer reaches a multiple of the second preset update step number, extract the second number of the second experience data from the satellite agent experience replayer to train the TD3 network to update the network parameters thereof.

[0051] Step S205, in the case where the network parameters of the MATD3 network or the network parameters of the TD3 network exist updates, if the number of groups of the first experience data in the unmanned aerial vehicle agent experience replayer does not reach the target number, continue to generate the first experience data and the second experience data under the updated network parameters; otherwise, return to the step of initializing the system environment state until the preset round is reached to obtain the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model.

[0052] In the embodiment of the application, in order to solve the task offloading problem of the unmanned aerial vehicle task, since multiple unmanned aerial vehicle agents need to be used, the first layer adopts the MATD3 network. In order to solve the task allocation problem of the satellite, since only one satellite agent needs to be used, the second layer adopts the TD3 network. Reference Figure 3Each of the UAV agents has an actor network (i.e., the first actor network in the above), a target actor network, a critic1 network, a target critic1 network, a critic2 network and a target critic2 network. The satellite agent also has an actor network (i.e., the second actor network in the above), a target actor network, a critic1 network, a target critic1 network, a critic2 network and a target critic2 network. By using two critic networks (critic1 and critic2) and the corresponding target critic networks (target critic1 and target critic2), the smaller value between the two can be taken when calculating the target value, thereby suppressing the problem of network overestimation.

[0053] To obtain the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model, the network parameters of the MATD3 network and the network parameters of the TD3 network can be initialized by random initialization, and the capacity of the experience replay container of the UAV agent and the satellite agent can be initialized. The above model uses a multi-round training method, and each round includes a specified number of execution steps. After each step is executed, a set of experience data is generated.

[0054] At the beginning of each round of training, the system environment state needs to be initialized, including: setting the time slot to 0 time slot, setting the task queuing sequence of the satellite and the UAV to empty, resetting the initial position of the UAV, resetting the computing resources of the UAV and the satellite, etc. After initialization, each agent can make corresponding actions according to its own observation state, and then determine its observation state in the next time slot, and calculate the reward obtained by each agent according to the action performed.

[0055] The observation state, action, first reward and observation state in the next time slot of the UAV agent in the current time slot are taken as a set of experience data (i.e., first experience data) of the UAV agent, and the observation state, action, second reward and observation state in the next time slot of the satellite agent in the current time slot are taken as a set of experience data (i.e., second experience data) of the satellite agent. The first experience data and the second experience data are put into the respective experience replayers.

[0056] If the number of groups of the first experience data in the UAV agent experience replay reaches a multiple of the first preset update step number, the first quantity of the first experience data needs to be extracted from the UAV agent experience replay to train the MATD3 network to update the network parameters of the actor network and the critic network in the network.

[0057] If the number of groups of the second experience data in the satellite agent experience replay reaches a multiple of the second preset update step number, the second quantity of the second experience data needs to be extracted from the satellite agent experience replay to train the TD3 network to update the network parameters of the actor network and the critic network in the network.

[0058] After the network parameter update, if the number of groups of the first experience data in the UAV agent experience replay does not reach the target quantity, that is, does not reach the execution step number specified in a round of training, the first experience data and the second experience data need to be continuously generated under the updated network parameters, otherwise, it is indicated that a round of training is completed, and the step of initializing the system environment state is returned to start a new round of training, until a preset number of rounds is reached, that is, the training can be ended to obtain the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model.

[0059] In an optional implementation, in the step S202, the first experience data is continuously generated under the current network parameters and the system environment and stored in the UAV agent experience replay, and the second experience data is continuously generated under the current network parameters and the system environment and stored in the satellite agent experience replay, and the specific steps include the following steps.

[0060] In the step S2021, the observation state of each UAV agent in the current time slot in the MATD3 network is obtained under the current network parameters and the system environment, and the action of each UAV agent in the current time slot is determined based on the corresponding first actor network.

[0061] In the step S2022, the observation state of the satellite agent in the current time slot is constructed based on the actions of all the UAV agents in the current time slot, and the action of the satellite agent in the current time slot is obtained by processing the observation state of the satellite agent in the current time slot by using the second actor network.

[0062] In the step S2023, the observation state of each UAV agent in the next time slot is determined based on the observation state and the action of each UAV agent in the current time slot, and the action of each UAV agent in the next time slot is determined based on the corresponding first actor network.

[0063] In the step S2024, the observation state of the satellite agent in the next time slot is constructed based on the actions of all the UAV agents in the next time slot.

[0064] Step S2025, the first reward obtained after each UAV agent executes the action in the current time slot and the second reward obtained after the satellite agent executes the action in the current time slot are calculated.

[0065] Step S2026, the observation state, action, first reward and observation state in the next time slot of all UAV agents in the current time slot are stored into the UAV agent experience replay as first experience data, and the observation state, action, second reward and observation state in the next time slot of the satellite agent in the current time slot are stored into the satellite agent experience replay as second experience data.

[0066] Specifically, the observation state of the target UAV agent corresponding to the target UAV u in the current time slot t is The action of the target UAV agent in the current time slot is determined by using the first actor network of the target UAV agent: Further, the set of task offloading ratio decisions of all UAVs is obtained And the set of observation states of all UAV agents in the current time slot is obtained:

[0067] The task offloading decision of the UAV will affect the task size uploaded to the satellite, therefore, only after the actions of all UAV agents in the current time slot t are determined, the observation state of the satellite agent corresponding to the central satellite cs in the current time slot t can be obtained: That is, the decision of the UAV agent as a kind of prior decision affects the state of the satellite agent. After the observation state of the satellite agent in the current time slot t is obtained, the action of the satellite agent in the current time slot is determined by using the second actor network of the satellite agent:

[0068] Each UAV agent can determine its observation state in the next time slot according to its observation state and action in the current time slot, and then determine its action in the next time slot by using its own first actor network, and further construct the observation state of the satellite agent in the next time slot According to the observation states of all UAV agents in the next time slot, the observation state set can be constructed

[0069] The first reward can be obtained after the UAV agent executes the action in the current time slot, and the second reward can be obtained after the satellite agent executes the action in the current time slot, in the embodiment of the application, the first experience data is represented as: The second experience data is represented as: Wherein, R1 represents the set of first rewards, and R2 represents the second reward.

[0070] In an alternative implementation, the training target of the MATD3 network and the TD3 network is system cost minimization. In the embodiment of the application, the calculation process of the system cost comprises the following steps:

[0071] In step S301, the first total time delay of the task processed by the target UAV locally in the current time slot is calculated, and the second total time delay of the task unloaded by the target UAV to the target satellite for processing in the current time slot is calculated. The target satellite represents any satellite in the space-based network.

[0072] In step S302, the first total time delay and the maximum time delay among all the second total time delays are taken as the processing time delay of the task generated by the target UAV in the current time slot.

[0073] Specifically, if the first total time delay of the task processed by the target UAV u locally in the current time slot t is The second total time delay of the task processed by the target UAV u unloaded to the target satellite s in the current time slot t is T s_u_t,exe The processing time delay of the task generated by the target UAV u in the current time slot t is: Wherein,

[0074] In step S303, the average value of the processing time delay of the task generated by all the UAVs in the current time slot is calculated, and the average value is taken as the time delay cost.

[0075] The time delay cost of the task generated by all the UAVs in the current time slot is: Wherein, N u represents the total number of UAVs in the air-based network.

[0076] In step S304, the first energy consumption of the task processed by the target UAV locally in the current time slot is calculated, and the second energy consumption of the task unloaded by the target UAV to the target satellite for processing in the current time slot is calculated.

[0077] In the embodiment of the application, the first energy consumption E of the task processed by the target UAV u locally in the current time slot t is calculated by the following formula: Wherein, κ u represents the energy consumption coefficient of the target UAV u, and represents the amount of tasks retained by the target UAV u for local processing in the current time slot t.

[0078] The second energy consumption E s_u_t of the task processed by the target UAV u unloaded to the target satellite s in the current time slot t is calculated by the following formula: Wherein, κ s represents the energy consumption coefficient of the target satellite s, and d s_u_tdenotes the amount of tasks that the target UAV u offloads to the target satellite s to process in the current time slot t.

[0079] In step S305, the energy consumption cost is calculated based on the first energy consumption of all UAVs and the second energy consumption of all satellites in the current time slot.

[0080] The energy consumption cost of the tasks generated by all UAVs in the current time slot is: wherein, r u denotes the energy consumption weight of the UAV processing tasks, r s denotes the energy consumption weight of the satellite processing tasks, r u + r s = 1.

[0081] In step S306, the system cost in the current time slot is calculated based on the delay cost and the energy consumption cost.

[0082] The formula of the system cost is: wherein, w t denotes the weight of the delay cost, w e denotes the weight of the energy consumption cost, w t + w e = 1.

[0083] In an optional embodiment, in the step S301, the first total delay of the tasks that the target UAV leaves to process locally in the current time slot is calculated, which specifically includes the following contents:

[0084] The processing time length required by the task queue of the target UAV at the start time of the current time slot is taken as the first waiting delay of the tasks that the target UAV leaves to process locally in the current time slot. That is, denotes the first waiting delay of the tasks that the target UAV u leaves to process locally in the current time slot t.

[0085] Based on the task offloading proportion decision of the target UAV in the current time slot and the amount of tasks generated by the target UAV in the current time slot, the amount of tasks that the target UAV leaves to process locally in the current time slot is determined. Specifically, given the task offloading proportion decision of the target UAV u in the current time slot t which includes the proportion of the tasks generated by the target UAV u in the current time slot t that is kept to be processed locally is multiplied by the amount of tasks generated by the target UAV u in the current time slot t, i.e. , the amount of tasks that the target UAV u generates in the current time slot t and keeps to be processed locally

[0086] ​Based on the amount of tasks processed locally, the computing resources required for the target UAV to process 1 bit of data, and the computing capability of the target UAV, a first computing delay of the target UAV staying in local processing of tasks in the current time slot is determined. That is, wherein, represents the first computing delay of the target UAV u staying in local processing of tasks in the current time slot t.

[0087] Based on the first waiting delay and the first computing delay, a first total delay of the target UAV staying in local processing of tasks in the current time slot is determined. The formula of the first total delay is:

[0088] In an optional implementation, in the step S301, the second total delay of the target UAV unloading tasks to the target satellite for processing in the current time slot is calculated, specifically including the following contents:

[0089] The processing delay of the current task queue of the target satellite is calculated; wherein the current task queue represents the task queue of the target satellite before receiving the tasks unloaded by the target UAV. Specifically, since the satellite may need to process multiple tasks in each time slot, when a task b generated in a t time slot needs to be processed, there is still a task a generated in a t time slot in front of the task b that needs to be processed, so there will be a queuing delay when the satellite processes the tasks. Assuming that the task of the target UAV u in the t time slot is allocated to the target satellite s, and the task is the kth task on the target satellite s (the relationship between k and t is uncertain), then D s,k represents the computing resources required for the target satellite s to complete the first k-1 tasks before starting to process the kth task, that is, D s,k represents the computing resources required for processing the current task queue; wherein, D s,k-1 represents the computing resources required for the target satellite s to complete the first k-2 tasks before starting to process the k-1th task, represents the amount of tasks of the k-1th task in the current task queue on the target satellite s. Therefore, the processing delay of the current task queue of the target satellite s is:

[0090] The transmission delay of the target UAV unloading tasks to the target satellite is calculated. Specifically, the tasks unloaded by the target UAV to each satellite in the space-based network are allocated by the center satellite, and if the target satellite is the center satellite, the transmission delay of the target UAV u unloading tasks to the target satellite in the current time slot t is : wherein, represents the amount of tasks unloaded by the target UAV u to the satellite in the current time slot t, which is the proportion of the tasks generated by the target UAV u in the current time slot t unloaded to the satellite the amount of tasks generated by the target UAV u in the current time slot t the result of multiplication, denotes the uplink data transmission rate of the target UAV u in the current time slot t; if the target satellite is not the central satellite, the transmission delay of the target UAV unloading tasks to the target satellite includes the inter-satellite transmission delay of the central satellite assigning tasks to the target satellite d s_u_t denotes the amount of tasks unloaded by the target UAV u to the target satellite s for processing in the current time slot t, denotes the data transmission rate between the central satellite and the target satellite s in the current time slot t.

[0091] the maximum delay between the processing delay and the transmission delay of the current task queue is taken as the second waiting delay of the tasks unloaded by the target UAV to the target satellite s for processing in the current time slot. That is, the second waiting delay of the tasks unloaded by the target UAV u to the target satellite s for processing in the current time slot t is represented as: wherein, denotes that the target satellite s is the central satellite, denotes that the target satellite s is not the central satellite.

[0092] based on the amount of tasks unloaded by the target UAV to the target satellite s for processing in the current time slot, the computing resources required by the target satellite for processing 1 bit of data, and the computing capability of the target satellite, the second computing delay of the tasks unloaded by the target UAV to the target satellite s for processing in the current time slot is determined. That is, wherein, T s_u_t,com denotes the second computing delay of the tasks unloaded by the target UAV u to the target satellite s for processing in the current time slot t.

[0093] based on the second waiting delay and the second computing delay, the second total delay of the tasks unloaded by the target UAV to the target satellite s for processing in the current time slot is determined. The formula of the second total delay is: s_u_t,exe T s_u_t,wait total s_u_t,com .

[0094] In an alternative embodiment, the step S2025 of calculating the first reward obtained after each UAV agent performs the action in the current time slot and the second reward obtained after the satellite agent performs the action in the current time slot specifically includes the following contents:

[0095] based on the first total delay and the first energy consumption of the tasks left by the target UAV for local processing in the current time slot, the first total cost of the tasks left by the target UAV for local processing in the current time slot is calculated. In the embodiment of the application, the formula of the first total cost of the tasks left by the target UAV u for local processing in the current time slot t is:

[0096] Based on the second total latency and the second energy consumption of the task unloaded by the target UAV to each satellite in the current time slot, a second total cost of the task unloaded by the target UAV to the satellite for processing in the current time slot is calculated. In the embodiment of the application, the formula of the second total cost of the task unloaded by the target UAV u to the satellite for processing in the current time slot t is as follows:

[0097] Based on the first total cost, the second total cost, the first waiting latency and the system cost, a first reward obtained after the action of the UAV agent corresponding to the target UAV is executed in the current time slot is calculated. The formula of the first reward obtained after the action of the UAV agent corresponding to the target UAV u is executed in the current time slot t is as follows: c1 is a compensation coefficient for preventing the first reward R1 from being too high. Based on the formula of the first reward, the UAV expects each task to be processed locally by the UAV and to be queued as little as possible, so the first reward is positively correlated with the first total cost, and the first reward is negatively correlated with the second total cost and the first waiting latency.

[0098] Based on all the second total costs of the tasks unloaded by all the UAVs to the satellite for processing in the current time slot and the system cost, a second reward obtained after the action of the satellite agent is executed in the current time slot is calculated. The formula of the second reward obtained after the action of the satellite agent is executed in the current time slot t is as follows: c2 and c3 both represent normalization coefficients for normalizing the second reward. The first reward and the second reward are both used for training the critic network of the corresponding agent, and a smaller Q value obtained by the critic network is used for training the actor network of the corresponding agent.

[0099] In summary, in order to reduce the complexity of the task deployment problem, the method provided by the embodiment of the application divides the problem into two sub-problems, the first layer solves the task offloading ratio decision problem of the UAV, and the second layer solves the task allocation ratio decision problem of the satellite. At the same time, the decision of the first layer sub-problem provides a prior decision for the second layer sub-problem. Through the process of alternating optimization by using the hierarchical reinforcement learning framework, the complexity of each sub-problem is reduced, and the learning efficiency is improved. In order to improve the flexibility of the hierarchical reinforcement learning framework, based on the hierarchical problem, the reinforcement learning algorithms MATD3 and TD3 are used to solve the two sub-problems, so that the real-time scheduling strategy of the UAV task in a dynamic environment can be obtained by using the method provided by the embodiment of the application, thereby generating efficient task offloading ratio and task allocation ratio decisions based on the dynamic system state. Compared with other existing algorithms, the method has good performance in terms of task processing delay, energy consumption and running performance, and can effectively improve the overall performance of the system.

[0100] Example Two

[0101] The embodiment of the present application also provides a space-air integrated task processing device based on hierarchical reinforcement learning, which is mainly used for executing the space-air integrated task processing method based on hierarchical reinforcement learning provided in the above example one, and the following will make a specific introduction to the space-air integrated task processing device based on hierarchical reinforcement learning provided in the embodiment of the present application.

[0102] Figure 4 The functional module diagram of the space-air integrated task processing device based on hierarchical reinforcement learning provided in the embodiment of the present application is shown in the figure, and the device mainly comprises: a first determination module 10, a first processing module 20, a second determination module 30, a construction module 40, and a second processing module 50, wherein: Figure 4

[0103] The first determination module 10 is used for determining the states of all unmanned aerial vehicles in the current time slot, wherein the state of a target unmanned aerial vehicle includes the distance between the mapping point of a central satellite in the unmanned aerial vehicle plane and the target unmanned aerial vehicle, the processing time required by the task queue of the target unmanned aerial vehicle at the start time of the current time slot, the task quantity generated by the target unmanned aerial vehicle in the current time slot, the index of the target unmanned aerial vehicle, and the time slot index; the target unmanned aerial vehicle represents any unmanned aerial vehicle in all unmanned aerial vehicles in the air-based network; and the central satellite represents the satellite with the least computing resources in the space-based network.

[0104] The first processing module 20 is used for processing the states of all unmanned aerial vehicles by using a target multi-agent deep reinforcement learning model to obtain the task offloading ratio decision of each unmanned aerial vehicle; wherein the target multi-agent deep reinforcement learning model includes a plurality of unmanned aerial vehicle agents, each unmanned aerial vehicle agent corresponds to one unmanned aerial vehicle in the air-based network, the observation state of the unmanned aerial vehicle agent is the state of the corresponding unmanned aerial vehicle, and the action of the unmanned aerial vehicle agent is the task offloading ratio decision of the corresponding unmanned aerial vehicle.

[0105] The second determination module 30 is used for determining the task queue processing time and the required computing resources of the task queue of each satellite in the space-based network at the start time of the current time slot.

[0106] The construction module 40 is used for constructing the state of the central satellite based on the task queue processing time, the required computing resources of the task queue, the task offloading ratio decision of all unmanned aerial vehicles, and the time slot index.

[0107] ​The second processing module 50 is configured to process the state of the central satellite by using the target single-agent deep reinforcement learning model, to obtain the task allocation proportion decision of the task quantity unloaded by each UAV into the space-based network on each satellite; wherein the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation proportion decision.

[0108] For the task deployment problem in the space-air integration edge computing scene, the embodiment of the application provides a space-air integration task processing device based on hierarchical reinforcement learning. The first layer solves the task offloading problem of the UAV task, and the second layer solves the task allocation problem of the satellite. Specifically, after determining the states of all UAVs in the current time slot, the states of all UAVs are processed by using the target multi-agent deep reinforcement learning model to obtain the task offloading proportion decision of each UAV, so that the task offloading problem of the UAV task is solved in the first layer. Then, the state of the central satellite is constructed based on the task offloading proportion decision of all UAVs, and the state of the central satellite is processed by using the target single-agent deep reinforcement learning model to obtain the task allocation proportion decision of the task quantity unloaded by each UAV into the space-based network on each satellite, so that the task allocation problem of the satellite is solved in the second layer. Thus, the task deployment problem is decomposed into the task offloading problem and the task allocation problem, the complexity of the space-air integration task processing problem is effectively reduced, the action space dimension of the agent is reduced, and the convergence performance and processing efficiency of the space-air integration task processing method are improved.

[0109] Optionally, the target multi-agent deep reinforcement learning model is a plurality of first actor networks in the MATD3 network, and the target single-agent deep reinforcement learning model is a second actor network in the TD3 network; and the device further comprises the following modules.

[0110] The initialization module is configured to initialize the network parameters of the MATD3 network and the network parameters of the TD3 network.

[0111] The experience acquisition module is configured to initialize the system environment state, continuously generate first experience data under the current network parameters and the system environment state and store the first experience data into the UAV agent experience replay buffer, and continuously generate second experience data and store the second experience data into the satellite agent experience replay buffer; wherein the first experience data comprises the observation state, the action, the first reward and the observation state of the next time slot of the UAV agent in the current time slot, and the second experience data comprises the observation state, the action, the second reward and the observation state of the next time slot of the satellite agent in the current time slot.

[0112] The first training module is configured to extract a first quantity of first experience data from the UAV agent experience replayer to train the MATD3 network to update network parameters of the MATD3 network, in a case where a quantity of groups of the first experience data in the UAV agent experience replayer reaches a multiple of a first preset update step number.

[0113] The second training module is configured to extract a second quantity of second experience data from the satellite agent experience replayer to train the TD3 network to update network parameters of the TD3 network, in a case where a quantity of groups of the second experience data in the satellite agent experience replayer reaches a multiple of a second preset update step number.

[0114] The iteration module is configured to, in a case where the network parameters of the MATD3 network or the network parameters of the TD3 network exist to be updated, if the quantity of groups of the first experience data in the UAV agent experience replayer does not reach a target quantity, continue to generate the first experience data and the second experience data under the updated network parameters; or return to the step of initializing the system environment state until a preset round is reached, to obtain a target multi-agent deep reinforcement learning model and a target single-agent deep reinforcement learning model.

[0115] Optionally, the experience acquisition module comprises:

[0116] The acquisition and determination unit is configured to acquire, under current network parameters and a system environment, an observation state of each UAV agent in a current time slot in the MATD3 network, and determine an action of each UAV agent in the current time slot based on a corresponding first actor network.

[0117] The first construction unit is configured to construct an observation state of the satellite agent in the current time slot based on the actions of all the UAV agents in the current time slot, and process the observation state of the satellite agent in the current time slot by using a second actor network to obtain an action of the satellite agent in the current time slot.

[0118] The determination unit is configured to determine an observation state of each UAV agent in a next time slot based on the observation state and the action of each UAV agent in the current time slot, and determine an action of each UAV agent in the next time slot based on a corresponding first actor network.

[0119] The second construction unit is configured to construct an observation state of the satellite agent in the next time slot based on the actions of all the UAV agents in the next time slot.

[0120] The calculation unit is configured to calculate a first reward obtained after each UAV agent executes the action in the current time slot, and a second reward obtained after the satellite agent executes the action in the current time slot.

[0121] a storage unit configured to store, as first experience data, observation states, actions, first rewards, and observation states of next time slots of all UAV agents in a current time slot into a UAV agent experience replacer, and store, as second experience data, observation states, actions, second rewards, and observation states of next time slots of the satellite agent in the current time slot into a satellite agent experience replacer.

[0122] Optionally, the training target of the MATD3 network and the TD3 network is system cost minimization, and the apparatus further includes:

[0123] a first calculation module configured to calculate a first total latency of tasks left by the target UAV for local processing in the current time slot, and calculate a second total latency of tasks unloaded by the target UAV to the target satellite for processing in the current time slot; wherein the target satellite represents any satellite in the space-based network.

[0124] a third determination module configured to take the first total latency and a maximum latency among all second total latencies as a processing latency of tasks generated by the target UAV in the current time slot.

[0125] a second calculation module configured to calculate an average value of processing latencies of tasks generated by all UAVs in the current time slot, and take the average value as a latency cost.

[0126] a third calculation module configured to calculate a first energy consumption of tasks left by the target UAV for local processing in the current time slot, and calculate a second energy consumption of tasks unloaded by the target UAV to the target satellite for processing in the current time slot.

[0127] a fourth calculation module configured to calculate an energy consumption cost based on the first energy consumption of all UAVs in the current time slot and the second energy consumption of all satellites.

[0128] a fifth calculation module configured to calculate a system cost of the current time slot based on the latency cost and the energy consumption cost.

[0129] Optionally, the first calculation module is specifically configured to:

[0130] take a processing time required by a task queue of the target UAV at a starting moment of the current time slot as a first waiting latency of tasks left by the target UAV for local processing in the current time slot.

[0131] determine an amount of tasks left by the target UAV for local processing in the current time slot based on a task offloading ratio of the target UAV in the current time slot and an amount of tasks generated by the target UAV in the current time slot.

[0132] determine a first calculation latency of tasks left by the target UAV for local processing in the current time slot based on the amount of tasks for local processing, a calculation resource required by the target UAV for processing 1-bit data, and a calculation capability of the target UAV.

[0133] determine a first total latency of the task left by the target UAV for local processing in the current time slot based on the first waiting latency and the first calculation latency.

[0134] Optionally, the first calculation module is specifically configured to:

[0135] calculate a processing latency of a current task queue of the target satellite; wherein the current task queue represents a task queue of the target satellite before receiving the task unloaded by the target UAV.

[0136] calculate a transmission latency of the task unloaded by the target UAV to the target satellite.

[0137] take the maximum latency of the processing latency of the current task queue and the transmission latency as a second waiting latency of the task unloaded by the target UAV to the target satellite for processing in the current time slot.

[0138] determine a second calculation latency of the task unloaded by the target UAV to the target satellite for processing in the current time slot based on the amount of the task unloaded by the target UAV to the target satellite for processing in the current time slot, the calculation resource required by the target satellite for processing 1-bit data, and the calculation capability of the target satellite.

[0139] determine a second total latency of the task unloaded by the target UAV to the target satellite for processing in the current time slot based on the second waiting latency and the second calculation latency.

[0140] Optionally, the calculation unit is specifically configured to:

[0141] calculate a first total cost of the task left by the target UAV for local processing in the current time slot based on the first total latency of the task left by the target UAV for local processing in the current time slot and the first energy consumption.

[0142] calculate a second total cost of the task unloaded by the target UAV to each satellite for processing in the current time slot based on the second total latency of the task unloaded by the target UAV to each satellite for processing in the current time slot and the second energy consumption.

[0143] calculate a first reward obtained after the action of the UAV agent corresponding to the target UAV in the current time slot based on the first total cost, the second total cost, the first waiting latency, and the system cost.

[0144] calculate a second reward obtained after the action of the satellite agent in the current time slot based on all the second total costs of the tasks unloaded by all the UAVs to the satellites for processing in the current time slot and the system cost.

[0145] Embodiment Three

[0146] Referring to Figure 5The electronic device provided by the embodiment of the present application comprises: a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is used for executing an executable module stored in the memory 61, for example, a computer program.

[0147] The memory 61 can comprise a high-speed random access memory (RAM) and can also comprise a non-volatile memory, for example, at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0148] The bus 62 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 5 Only one bidirectional arrow is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0149] The memory 61 is used for storing a program, and the processor 60 executes the program after receiving an execution instruction. The method executed by the device defined by the process disclosed in any of the foregoing embodiments of the present application can be applied to the processor 60 or realized by the processor 60.

[0150] The processor 60 can be an integrated circuit chip with signal processing capability. In implementation, each step of the above method can be completed by integrated logic circuit of hardware in the processor 60 or by instructions in the form of software. The processor 60 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61, and combines the hardware to complete the steps of the above method.

[0151] The computer program product of the space-air integration task processing method and device based on hierarchical reinforcement learning provided by the embodiments of the present application includes a computer readable storage medium storing non-volatile program codes executable by a processor. The instructions included in the program codes can be used to execute the method in the foregoing method embodiments. For specific implementation, please refer to the method embodiments, which will not be described here.

[0152] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0153] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0154] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0155] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is normally placed, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0156] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components must be absolutely horizontal or overhanging, but can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0157] In the description of the present application, it should also be noted that unless otherwise explicitly specified and limited, the terms "arrangement", "installation", "connection", "connection" should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0158] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing integrated air-space missions based on hierarchical reinforcement learning, characterized in that, include: Determine the status of all UAVs in the current time slot; wherein, the status of the target UAV includes: the distance between the mapping point of the central satellite on the UAV plane and the target UAV, the processing time required for the task queue of the target UAV at the start of the current time slot, the amount of tasks generated by the target UAV in the current time slot, the index of the target UAV and the time slot index; the target UAV refers to any UAV among all UAVs in the space-based network; the central satellite refers to the satellite with the least computing resources in the space-based network; The state of all UAVs is processed using a target multi-agent deep reinforcement learning model to obtain the task offloading ratio decision for each UAV. The target multi-agent deep reinforcement learning model includes multiple UAV agents, each UAV agent corresponds one-to-one with a UAV in the airborne network, the observation state of the UAV agent is the state of the corresponding UAV, and the action of the UAV agent is the task offloading ratio decision for the corresponding UAV. Determine the task queue processing time and computational resources required for each satellite in the space-based network at the start time of the current time slot; The status of the central satellite is constructed based on the task queue processing time of all satellites, the computing resources required by the task queue, the task offloading ratio decision of all UAVs, and the time slot index. The state of the central satellite is processed using a target single-agent deep reinforcement learning model to obtain the task allocation ratio decision of each UAV's offloaded tasks to the space-based network on each satellite; wherein, the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation ratio decision.

2. The air-space integrated mission processing method based on hierarchical reinforcement learning according to claim 1, characterized in that, The target multi-agent deep reinforcement learning model is multiple first performer networks in the MATD3 network, and the target single-agent deep reinforcement learning model is a second performer network in the TD3 network; Before determining the status of all drones in the current time slot, the following steps are also included: Initialize the network parameters of the MATD3 network and the TD3 network; The system environment state is initialized. Under the current network parameters and system environment state, the first experience data is continuously generated and stored in the UAV agent experience playback device, and the second experience data is continuously generated and stored in the satellite agent experience playback device. The first experience data includes: the observation state, actions, first reward and the observation state of the UAV agent in the current time slot and the next time slot. The second experience data includes: the observation state, actions, second reward and the observation state of the satellite agent in the current time slot and the next time slot. If the number of sets of first experience data in the UAV agent experience replay device reaches a multiple of the first preset update step number, a first number of first experience data is extracted from the UAV agent experience replay device to train the MATD3 network and update its network parameters. If the number of sets of second experience data in the satellite agent experience replay device reaches a multiple of the second preset update step number, a second number of second experience data are extracted from the satellite agent experience replay device to train the TD3 network and update its network parameters. If the network parameters of the MATD3 network or the TD3 network are updated, and the number of sets of the first experience data in the UAV agent experience replay device has not reached the target number, then the first experience data and the second experience data will continue to be generated under the updated network parameters; otherwise, the system environment state will be initialized again until the preset number of rounds is reached, and the target multi-agent deep reinforcement learning model and the target single-agent deep reinforcement learning model are obtained.

3. The air-space integrated mission processing method based on hierarchical reinforcement learning according to claim 2, characterized in that, Under the current network parameters and system environment, the system continuously generates first experience data and stores it in the UAV intelligent agent experience playback device, and continuously generates second experience data and stores it in the satellite intelligent agent experience playback device, including: Under the current network parameters and system environment, the observation status of each UAV agent in the MATD3 network in the current time slot is obtained, and the action of each UAV agent in the current time slot is determined based on the corresponding first performer network. The observation state of the satellite agent in the current time slot is constructed based on the actions of all UAV agents in the current time slot, and the observation state of the satellite agent in the current time slot is processed by the second performer network to obtain the actions of the satellite agent in the current time slot. Based on the observation state and actions of each UAV agent in the current time slot, the observation state of each UAV agent in the next time slot is determined, and the actions of each UAV agent in the next time slot are determined based on the corresponding first performer network. The observation state of the satellite agent in the next time slot is constructed based on the actions of all UAV agents in the next time slot; Calculate the first reward obtained by each of the UAV agents after performing the action in the current time slot and the second reward obtained by each of the satellite agents after performing the action in the current time slot; The observation status, actions, first reward, and observation status of all the UAV agents in the current time slot and the next time slot are stored as first experience data in the UAV agent experience playback device. The observation status, actions, second reward, and observation status of the satellite agent in the current time slot and the next time slot are stored as second experience data in the satellite agent experience playback device.

4. The air-space integrated mission processing method based on hierarchical reinforcement learning according to claim 3, characterized in that, The training objective of the MATD3 network and the TD3 network is to minimize the system cost. The calculation process for the system cost includes: Calculate the first total latency of the task that the target UAV remains to process locally in the current time slot, and calculate the second total latency of the task that the target UAV is offloaded to the target satellite for processing in the current time slot; wherein, the target satellite refers to any satellite in the space-based network; The maximum delay among the first total delay and all of the second total delays is taken as the processing delay of the task generated by the target UAV in the current time slot; Calculate the average processing latency of all tasks generated by all drones in the current time slot, and use the average value as the latency cost; Calculate the first energy consumption of the task that the target UAV remains to process locally in the current time slot, and calculate the second energy consumption of the task that the target UAV offloads to the target satellite for processing in the current time slot; Calculate the energy consumption cost based on the first energy consumption of all UAVs and the second energy consumption of all satellites in the current time slot; Calculate the system cost for the current time slot based on the latency cost and the energy consumption cost.

5. The air-space integrated mission processing method based on hierarchical reinforcement learning according to claim 4, characterized in that, Calculate the first total latency of the task that the target UAV leaves to process locally in the current time slot, including: The processing time required for the task queue of the target UAV at the start of the current time slot is taken as the first waiting delay for the task of the target UAV to be processed locally in the current time slot. Based on the task offloading ratio decision of the target UAV in the current time slot and the amount of tasks generated by the target UAV in the current time slot, determine the amount of tasks that the target UAV leaves to process locally in the current time slot. Based on the amount of locally processed tasks, the computing resources required for the target drone to process 1 bit of data, and the computing power of the target drone, the first computing delay of the task left for local processing by the target drone in the current time slot is determined. Based on the first waiting delay and the first calculation delay, the first total delay of the task that the target UAV leaves to process locally in the current time slot is determined.

6. The air-space integrated mission processing method based on hierarchical reinforcement learning according to claim 5, characterized in that, Calculating the second total delay of the task unloaded from the target UAV to the target satellite for processing in the current time slot includes: Calculate the processing latency of the target satellite's current task queue; wherein, the current task queue represents the task queue of the target satellite before receiving the tasks unloaded by the target UAV; Calculate the transmission delay for the target UAV to offload its mission to the target satellite; The maximum delay between the processing delay of the current task queue and the transmission delay is used as the second waiting delay for the task that the target UAV is unloaded to the target satellite for processing in the current time slot. Based on the amount of task that the target UAV is offloaded to the target satellite for processing in the current time slot, the computing resources required by the target satellite to process 1 bit of data, and the computing power of the target satellite, the second computing delay of the task that the target UAV is offloaded to the target satellite for processing in the current time slot is determined; Based on the second waiting delay and the second calculation delay, the second total delay of the task of unloading the target UAV to the target satellite for processing in the current time slot is determined.

7. The air-space integrated mission processing method based on hierarchical reinforcement learning according to claim 6, characterized in that, Calculate the first reward obtained by each of the UAV agents after performing the action in the current time slot and the second reward obtained by each satellite agent after performing the action in the current time slot, including: Based on the first total latency and first energy consumption of the task that the target UAV is processing locally in the current time slot, calculate the first total cost of the task that the target UAV is processing locally in the current time slot. Based on the second total latency and second energy consumption of the task of the target UAV being offloaded to each satellite for processing in the current time slot, calculate the second total cost of the task of the target UAV being offloaded to the satellite for processing in the current time slot. Based on the first total cost, the second total cost, the first waiting delay, and the system cost, calculate the first reward obtained by the drone agent corresponding to the target drone after performing the action in the current time slot; Based on the total second cost of all tasks unloaded from drones to satellite processing in the current time slot and the system cost, calculate the second reward obtained by the satellite agent after performing the action in the current time slot.

8. A space-air integrated mission processing device based on hierarchical reinforcement learning, characterized in that, include: The first determining module is used to determine the status of all UAVs in the current time slot; wherein, the status of the target UAV includes: the distance between the mapping point of the central satellite on the UAV plane and the target UAV, the computing resources required by the task queue of the target UAV at the start of the current time slot, the amount of tasks generated by the target UAV in the current time slot, the index of the target UAV and the time slot index; the target UAV represents any UAV among all UAVs in the space-based network; the central satellite represents the satellite with the least computing resources in the space-based network; The first processing module is used to process the state of all UAVs using a target multi-agent deep reinforcement learning model to obtain the task offloading ratio decision for each UAV; wherein, the target multi-agent deep reinforcement learning model includes multiple UAV agents, each UAV agent corresponds one-to-one with a UAV in the airborne network, the observation state of the UAV agent is the state of the corresponding UAV, and the action of the UAV agent is the task offloading ratio decision for the corresponding UAV. The second determining module is used to determine the task queue processing time and the computing resources required for the task queue of each satellite in the space-based network at the start time of the current time slot. A construction module is used to construct the status of the central satellite based on the task queue processing time of all satellites, the computing resources required by the task queue, the task offloading ratio decision of all UAVs, and the time slot index; The second processing module is used to process the state of the central satellite using a target single-agent deep reinforcement learning model to obtain the task allocation ratio decision of each UAV's offloaded tasks to the space-based network on each satellite; wherein, the satellite agent in the target single-agent deep reinforcement learning model corresponds to the central satellite, the observation state of the satellite agent is the state of the central satellite, and the action of the satellite agent is the task allocation ratio decision.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the air-space integrated mission processing method based on hierarchical reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the air-space integrated mission processing method based on hierarchical reinforcement learning as described in any one of claims 1 to 7.

Images