An unmanned cluster task offloading method based on a TRPO algorithm

Abstract

The present application relates to a kind of unmanned cluster task unloading method based on TRPO algorithm, step one: establish unmanned cluster three-layer computing framework, monitor the active state of ground equipment and receive the task that ground equipment periodically produces, according to workload and maximum tolerable delay, the information of task is expressed as set;Step two: establish Markov Poisson modulation's random process representation task arrival, quantitative analysis model in time-varying task service delay;Step three: build the communication model of system, delay model, energy consumption model;Step four: introduce multi-agent Markov decision process to describe the dynamic decision-making process of S2, information is shared between agent and makes real-time response;Step five: use TRPO algorithm to optimize delay, energy, train model to obtain optimal unloading decision, and adopt optimal decision to execute unloading.The present application effectively improves the fairness of unmanned cluster networking resource allocation, prolongs system life cycle.

Description

TECHNICAL FIELD

[0001] The application relates to an unmanned cluster task offloading method based on a TRPO algorithm and belongs to the technical field of wireless communication. BACKGROUND

[0002] Mobile edge computing (MEC) is a new computing architecture, which is applied to the fields of digital medical treatment, Internet of Vehicles, maritime communication and intelligent industry due to its support for a large number of node interconnections and extensive geographical distribution. MEC improves response delay and service performance by arranging a computing server in an edge network close to a terminal without transmitting a large amount of data back to a remote data center. However, it is a difficult task to erect a ground MEC base station in a region far from a core Internet, and the cost is high, so there is a problem of limited resources in the region in an emergency. An unmanned cluster system is a system composed of multiple unmanned devices, and common unmanned devices are unmanned aerial vehicles, unmanned vehicles and embodied robots, which can cooperate to complete a specific task. Since the unmanned devices are highly mobile and convenient to deploy, the unmanned cluster networking serving as an edge server is considered to be an effective solution to improve user experience under MEC, especially for application occasions with high real-time data requirements. In addition, an air communication relay operated by autonomous unmanned devices can quickly and safely transmit data collected by ground devices to a remote command and control center, and this technology expands the communication range and can provide higher freedom.

[0003] Compared with a traditional distributed hierarchical network, there are new problems in the cooperation process of unmanned cluster networking, and ignoring the energy bottleneck of individual unmanned devices can cause network failure nodes and bring disastrous disconnection and disconnection to the network. In addition to large-scale application scenarios, high-speed and frequent interaction also constantly impacts the performance bottleneck, and it is urgent to optimize the allocation of resources and provide low-latency services. In recent years, deep reinforcement learning methods have attracted more and more attention in the field of wireless communication. The core idea of deep reinforcement learning is to use a deep neural network to approximate a value function and a policy function, so as to realize learning and decision-making in a complex environment. MEC assisted by unmanned cluster networking has considerable complexity in transmission control and offloading decision, and the design of an offloading strategy needs to consider time-varying channel conditions, user mobility, energy supply, computing workload and cache capacity and other factors, so a method based on deep reinforcement learning becomes an effective candidate solution for joint optimization. SUMMARY

[0004] The application provides an unmanned cluster task offloading method based on a TRPO algorithm, so as to solve the problem of unbalanced utilization of cooperation resources of unmanned cluster networking, realize joint optimization of system delay, energy and service quality, and effectively improve the fairness of resource allocation of unmanned cluster networking.

[0005] A method for task offloading of unmanned cluster based on TRPO algorithm, comprising the following steps:

[0006] Step one: establish a three-layer computing framework of unmanned cluster, monitor the active state of ground equipment and receive the tasks generated periodically by ground equipment, and represent the information of tasks as a set according to the workload and maximum tolerable delay;

[0007] Step two: establish a Markov Poisson modulated stochastic process to represent the task arrival, and quantitatively analyze the service delay of time-varying tasks in the model;

[0008] Step three: build the communication model, delay model and energy consumption model of the system;

[0009] Step four: introduce a multi-agent Markov decision process to describe the dynamic decision-making process of S2, and the agents share information to make real-time response;

[0010] Step five: use TRPO algorithm to jointly optimize delay and energy, train the model to obtain the best offloading decision, and execute the offloading with the best decision.

[0011] The three-layer computing framework of unmanned cluster in step one includes task collection layer, relay transmission layer and information control layer from bottom to top, wherein the K small unmanned devices of the task collection layer are responsible for receiving the tasks generated by the cell, represented by the set K={1,2,…,K}, U k ∈K, to ensure that the tasks can be received and kept in a quasi-static state; the M unmanned devices of the relay transmission layer can act as relays to forward the tasks to the uppermost layer in multiple hops, or act as servers to perform local calculation to realize offloading, represented by the set M={1,2,…,M}, U m ∈M; the large unmanned device U h of the information control layer has strong computing efficiency and large capacity battery, and plans the task offloading process of the whole system as the leader.

[0012] The active state of the ground equipment in step one is represented by the state space k of the process A (t), and the constant element belongs to {0,1}, 0 represents low activity, and 1 represents high activity.

[0013] The task information set in step one covers three parameters, represented by where V j represents the amount of data to be processed, O j describes the total number of CPU cycles required to complete the task, represents the maximum delay that the task can tolerate, and if the task delivery fails; the unmanned device U kα j ∈{0,…,N} represents; when α j = n, it means that task j will be executed by the computing unit of unmanned device U n n, and join the unloading queue waiting; that is, when α j = 0, the task is locally calculated, j joins the queue follows the principle of first come first served.

[0014] The random process of task arrival in step two is subject to Poisson distribution, and the process A k (t) of task J arriving at any information collection layer unmanned device is modeled as a two-state Markov modulated Poisson process.

[0015] The communication model, delay model and energy consumption model of the system in step three specifically include:

[0016] If the task is executed locally, the computing delay of task j in unmanned device U k n is

[0017] If the small unmanned device chooses to unload, since the unmanned devices in the cell network are in fixed positions, the position of the small unmanned device is represented as (x k , y k , z k ), and the position of another unmanned device U m at time t is (x m (t), y m (t), z m (t)), the distance between them is

[0018] According to the free space propagation model, the unmanned cluster networking is modeled, and the channel gain is where d0 represents the reference distance; according to the Shannon formula, the maximum transmission rate can reach where W represents the channel bandwidth, P s represents the signal power of the small unmanned device, β0 represents the average channel power gain at the reference distance d0 = 1m, G km (t) represents the channel gain at time t, N0 represents the channel Gaussian white noise, and ψ represents the azimuth angle; the delay of task j in the one-to-one transmission process between U k and U m is

[0019] If the unloading object is not within the communication range, define the set N u to represent the associated unmanned devices on the unloading path, and when α j = n, n ∈ Nu And is the end of the offload, the transmission time is the sum of multi-hop transmission, the corresponding delay is:

[0020]

[0021] The total delay of the final task is:

[0022]

[0023] The method of establishing the energy consumption model of the system in step three comprises:

[0024] The energy loss includes the energy consumption cost of task j in the offload and execution phase, the energy of CPU executing the task and the energy consumption of offloading and forwarding the task;

[0025] If the task is executed locally, the energy consumed by task j in unmanned device U k The energy consumed by local computing is Where κ is the effective switching capacitance depending on the chip architecture; if the task is offloaded to the computing node U n , the corresponding energy consumption is The total energy loss of the final task is:

[0026]

[0027] The basic elements of Markov decision process in step four comprise:

[0028] State space: the state observed by a single agent includes the active state of the coverage area device, the congestion degree of the local queue, and the state of other unmanned devices in the environment, and the state set of all agents at time slot t is represented as S(t)={S A (t),S e (t),S o (t),S u (t)};

[0029] Action space: each unmanned device makes a decision on the offload object of task j according to the observation, α j ∈{0,…,N}, α j >0 represents the offload object of the server, and α j =0 represents local computing;

[0030] State transition: after the agent takes a certain action, the action interacts with the environment, and the state transition process of the whole system is Q Σ , whose generic element is defined as Q[s,s']=Pr(S Σ (t)=s'|S Σ (t-1)=s), where s and s' are two constant states of Markov chain;

[0031] Reward: The goal of the joint optimization is to minimize the latency and energy cost generated by each UAV, and the UAV will get a reward ω after interacting with the environment in each time slot k (t) represents the UAV U k The latency and energy cost of processing tasks in time slot t; Calculate the reward function ω = β * t total +(1-β)*E total , where β (0 < β < 1) is the weight of latency; The reward function is calculated by , where E{ω i (t)} = ∑ω*f k (ω), f k (ω) is the probability distribution function of the UAV cost.

[0032] The step five of training the improved multi-agent TRPO algorithm includes the following steps:

[0033] S5.1: Initialize state s1, reset the environment;

[0034] S5.2: Before meeting the maximum number of rounds, each agent obtains an observation o(t) from the environment, performs an action according to the policy π θ' (a t |o t ), gets a timely reward r t , observes the state s t+1 of the new environment, stores the trajectory Tr i (t) = {o i (t), a i (t), r i (t), s i (t+1)} into the buffer pool, and calculates the advantage function of each state;

[0035] S5.3: Estimate the sample policy model gradient, and calculate the step size using the conjugate gradient algorithm;

[0036] S5.4: Update the Actor network parameters θ under the condition of meeting the KL divergence;

[0037] S5.5: Optimize the value network parameters φ to minimize the mean square error between the return and the state value;

[0038] S5.6: Enter the decentralized execution phase, and independently execute the trained network on each UAV.

[0039] Compared with the prior art, the beneficial effects of the present application are:

[0040] The application designs a novel unmanned cluster networking computing framework, not only clearly defines the functions and roles of unmanned devices in the system, studies the operation of task computing offloading, but also focuses on the performance differences between unmanned device individuals in computing efficiency and communication distance, solves the limitations of traditional unmanned cluster networking flat role division; based on the improved multi-agent trust region optimization algorithm, the comprehensive optimal solution of unmanned device resource offloading is obtained, the system delay and energy are jointly optimized, the energy consumption of unmanned device is constrained, the heterogeneous network management is optimized, the fairness of unmanned cluster networking resource allocation is effectively improved, and the life cycle of the system is prolonged. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0042] Figure 1 The model training specific structure flowchart of the unmanned cluster task offloading method based on the TRPO algorithm of the present application is shown in the figure.

[0043] Figure 2 The three-layer computing framework schematic diagram of the unmanned cluster task offloading method based on the TRPO algorithm of the present application is shown in the figure.

[0044] Figure 3 The framework and structure diagram of the TRPO algorithm of the unmanned cluster task offloading method based on the TRPO algorithm of the present application is shown in the figure. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0046] Figure 1 The flowchart of the unmanned cluster task offloading method based on the TRPO algorithm of the present application is shown in the figure, and the method comprises the following steps:

[0047] Step one: establish a three-layer computing framework of unmanned cluster, monitor the active state of ground equipment and receive the tasks generated periodically by ground equipment, and represent the information of tasks as a set according to the workload and maximum tolerable delay;

[0048] Step two: Establish a Markov Poisson modulated random process to represent task arrival, and quantitatively analyze the service delay of time-varying tasks in the model;

[0049] Step three: Build a communication model, delay model, and energy consumption model of the system;

[0050] Step four: Introduce a multi-agent Markov decision process to describe the dynamic decision-making process of S2, and share information between agents to make real-time responses;

[0051] Step five: Jointly optimize delay and energy using the TRPO algorithm, train the model to obtain the best offloading decision, and execute offloading using the best decision.

[0052] Preferably, a three-layer computing framework of unmanned devices is established in step one, as shown in Figure 2 The framework includes a task collection layer, a relay transmission layer, and an information control layer from bottom to top, covering unmanned devices with different functions; K small unmanned devices are deployed in the task collection layer to collect tasks, represented by the set K = {1, 2, …, K}, U k ∈K; The K unmanned devices are divided into G cells, covering the divided ground area, and the unmanned devices in the cell can communicate with each other; the small unmanned devices in the relay transmission layer are represented by the set M = {1, 2, …, M}, U m ∈M; Large unmanned devices with strong computing efficiency and large capacity batteries are deployed in the uppermost layer, represented by U h ; The unmanned devices use binary offloading to allocate computing resources; based on the above process, the physical activity range constraints of the three types of unmanned devices are as follows: U k is deployed at a fixed position in low altitude, the relay unmanned device U m flies within a space width of for the area, and the height is limited below the information control layer, and the large unmanned device U h is in the cluster center;

[0053] The entire network serves as a MEC system, and the unmanned devices act as MEC servers; when the ground devices periodically generate tasks, the corresponding small unmanned devices U k in the task collection layer of the cell are dispatched to receive tasks; each unmanned device is equipped with a separate computing unit, referred to as CU, which is responsible for local computation; it is assumed that the CPU performance levels of the unmanned devices in the network, except for the large unmanned device, are consistent, i.e., the number of CPU cycles per second is f; U k There are two queues in the internal CU, let represent the set of task execution queues and offloading queues at time t, respectively, and the generated task j is represented as where V j represents the amount of data to be processed, and O j describes the total number of CPU cycles required to complete the task, represents the maximum delay that the task can tolerate, exceeding Task delivery fails; the task operates in binary offloading mode, and the UAV U k 's offloading decision is represented by α j ∈{0,…,N}; when α j =n, it means that task j is executed by the computing unit of the UAV U n , and j is added to the queue; that is, when α j =0, the task is locally computed, and j is added to the queue, which follows the first-come-first-served principle; the active state of the ground device is determined by the process A k (t) state space , whose constant elements belong to {0,1}, 0 represents low activity, and 1 represents high activity; τ seconds are divided into decision cycles, and it is assumed that the state of the entire system does not change during τ seconds; in this process, the leader UAV U h is responsible for multi-dimensional evaluation of the network state, global coordination, and efficient decision-making; the lower-layer UAVs transmit beacon messages containing their own positions and queue conditions to the leader UAV U h .

[0054] Preferably, step two has the following specific steps:

[0055] Assuming that the random process of task arrival obeys a Poisson distribution, the process A k (t) of task J arriving at any information collection layer UAV U k is modeled as a two-state Markov-modulated Poisson process, abbreviated as MMPP, and according to the above process, the state probability transition matrix Q k and the rate matrix Λ k are established:

[0056]

[0057] where λ1 is the job arrival rate under state 0, and λ2 is the job arrival rate under state 1; π represents the steady-state probability vector of the system, which satisfies the following equation at steady state:

[0058]

[0059] where e is the unit vector; after determining the steady-state probability, the average job arrival rate of the system at steady state is obtained:

[0060]

[0061] Let q e , q orespectively represent the maximum length of execution queue and offloading queue, define respectively represent the state of underlying Markov chain at time t, represent the number of tasks in execution queue and offloading queue at time t. k respectively represent the state of underlying Markov chain at time t, represent the number of tasks in execution queue and offloading queue at time t.

[0062] Preferably, the third step is specifically as follows:

[0063] The communication and latency model of the system is established:

[0064] If the task is executed locally, the task j is executed on the unmanned device U k The latency is calculated as

[0065] If the small unmanned device chooses to offload, since the unmanned devices in the cell network are in fixed positions, the position of the small unmanned device is represented as (x k ,y k ,z k ), and the position of another unmanned device U m at time t is (x m (t),y m (t),z m (t)), the distance between them is

[0066] The unmanned device cluster networking is modeled according to the free space propagation model, and the channel gain is where d0 represents a reference distance, equal to 1 m; according to the Shannon formula, the transmission rate can reach a maximum of where W represents the channel bandwidth, P s represents the signal power of the small unmanned device, β0 represents the average channel power gain at the reference distance d0=1 m, G km (t) represents the channel gain at time t, N0 represents the channel Gaussian white noise, and ψ represents the azimuth angle; according to the above analysis, the latency of the task j in the one-to-one transmission process between U k and U m is

[0067] If the offloaded object is not within the communication range, define the set N u to represent the associated unmanned devices on the offloading path, when α j =n, n∈N u and is the end point of offloading, at this time, the transmission time is the sum of the multi-hop transmission, and the corresponding latency is:

[0068]

[0069] The total latency of a final task is:

[0070]

[0071] Establish the energy consumption model of the system:

[0072] Energy consumption includes the energy cost of task j in the offloading and execution phase, the energy of CPU executing the task and the energy of offloading and forwarding the task;

[0073] If the task is executed locally, the energy consumed by task j in the unmanned device U k The energy consumed by local computing is Where κ is the effective switching capacitance depending on the chip architecture; if the task is offloaded to the computing node U n , the corresponding energy consumption is The total energy consumption of a task is finally:

[0074]

[0075] The basic elements of the Markov decision process established in step four are represented as follows:

[0076] State space: the states that a single agent can observe include the active state of the coverage area device, the congestion degree of the local queue, and the state of other unmanned devices in the environment, and the state set of all agents at time slot t is represented as S(t) = {S A (t), S e (t), S o (t), S u (t)}.

[0077] Action space: each unmanned device makes a decision on the offloading object of task j according to the observation, α j ∈{0,…,N}, α j >0 represents the offloading object of the server, and α j =0 indicates local computing.

[0078] State transition: after the agent takes a certain action, the action interacts with the environment, and the state transition process of the whole system is Q ∑ , whose generic element is defined as Q[s,s'] = Pr(S ∑ (t) = s'|S ∑ (t-1) = s), where s and s' are two state constants of the Markov chain.

[0079] Reward: the goal of joint optimization is to minimize the delay and energy consumption generated by each unmanned device, and the unmanned device will get a reward ω k (t) after interacting with the environment at each time slot, which represents the delay and energy consumption of the unmanned device U k in processing tasks at time slot t; the reward function ω = β*t total+ (1 - β) * E total where β (0 < β < 1) is the weight of the delay; the reward function is calculated by where E{ω i (t)} = ∑ω*f k (ω), f k (ω) is the probability distribution function of the unmanned device overhead.

[0080] Preferably, as shown in Figure 3 , the steps of the improved multi-agent TRPO algorithm training in step five are as follows:

[0081] S5.1: initialize state s1, reset the environment;

[0082] S5.2: before meeting the maximum number of rounds, each agent obtains an observation o(t) from the environment, performs an action according to the policy π θ' (a t |o t ), gets a timely reward r t , observes the state s t+1 of the new environment, stores the trajectory Tr i (t) = {o i (t), a i (t), r i (t), s i (t+1)} into the buffer pool, and calculates the advantage function of each state according to the following formula, i.e.:

[0083]

[0084] where, is the advantage function of the policy π, i.e. the advantage of action a relative to the average action under state s, Q represents the state-action value function, and V represents the value function when being in state s;

[0085] S5.3: estimate the sample policy model gradient, and calculate the step size using the CG conjugate gradient algorithm;

[0086] S5.4: update the Actor network parameters θ to maximize the difference between the cumulative reward value obtained using the new policy and the cumulative reward value obtained using the old policy under the condition of meeting the KL divergence, and the optimization goal is

[0087]

[0088] where π θ' is the new policy, π θ is the old policy, is the advantage estimate value of π θ ; D KL(·) is the KL divergence, the KL divergence avoids the policy to iterate drastically, limits the update step; δ is the threshold of the expectation of the KL divergence in the trust region, using the conjugate gradient method to solve this optimization problem, updating the policy parameters θ i ;

[0089] S5.5: Optimizing the value network parameters To minimize the mean squared error between the return and the state value, the loss function is defined as:

[0090]

[0091] S5.6: Entering the decentralized execution phase, independently executing the trained network on each unmanned device.

[0092] The embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of the embodiments can be made without departing from the principles and spirits of the present application, and still fall within the protection scope of the present application.

Claims

1. A TRPO algorithm-based task offloading method for unmanned clusters, characterized in that: The method comprises the following steps: Step one: establish a three-layer computing framework of unmanned cluster, monitor the active state of ground equipment and receive the tasks generated periodically by the ground equipment, and represent the information of the tasks as a set according to the workload and the maximum tolerable delay; Step two: establish a Markov Poisson modulated random process to represent the task arrival, and quantitatively analyze the service delay of the time-varying tasks in the model; Step three: build a communication model, a delay model and an energy consumption model of the system; Step four: introduce a multi-agent Markov decision process to describe the dynamic decision-making process of S2, and the agents share information to make real-time responses; Step five: jointly optimize the delay and energy consumption by using the TRPO algorithm, train the model to obtain the best unloading decision, and execute the unloading by using the best decision; The training of the improved multi-agent TRPO algorithm in step five comprises the following steps: S5.1: initialize the state s1 and reset the environment; S5.2: Before the maximum number of rounds is met, each agent obtains an observation o(t) from the environment, performs an action a(t) according to the policy p θ' (a t |o t ) to get the timely reward r t (t), and observes the state s t+1 (t+1) of the new environment, stores the trajectory Tr i (t) = {o i (t), a i (t), r i (t), s i (t+1)} into the buffer pool, and calculates the advantage function of each state, i.e. where A i π is the advantage function of the policy π, i.e., the advantage of action a relative to the average action at state s, Q denotes the state-action value function, and V denotes the value function of being in state s. S5.3: estimate the sample policy model gradient, and calculate the step length by using the conjugate gradient algorithm; S5.4: update the Actor network parameters θ under the condition of satisfying the KL divergence; S5.5: Optimize the value network parameters φ to minimize the mean squared error between the returns and the state values, the loss function defined as: S5.6: enter the decentralized execution phase, and independently execute the trained network on each unmanned equipment.

2. The method of claim 1, wherein the TRPO algorithm is based on: The three-layer computing framework of the unmanned cluster in step one includes a task collection layer, a relay transmission layer, and an information control layer from bottom to top. The K small unmanned devices of the task collection layer are responsible for receiving tasks generated by the cell, represented by the set K = {1, 2, …, K}, U k ∈K, ensuring that the tasks can be received and kept in a quasi-static state; the M unmanned devices of the relay transmission layer can act as relays to forward the tasks to the uppermost layer in multiple hops, and also act as servers to perform local computing to achieve offloading, represented by the set M = {1, 2, …, M}, U m ∈M; the large unmanned device of the information control layer U h has strong computing efficiency and large-capacity battery, and plans the process of task offloading of the entire system as the leader.

3. The method of claim 2, wherein: The active state of the ground equipment in step one is determined by process A k State space of (t) characterized by belongs to {0,1}, 0 means low activity, 1 means high activity.

4. The method of claim 3, wherein: The task information set in step one covers three parameters, denoted as where V j denotes the amount of data to be processed, O j describes the total number of CPU cycles required to complete the task, denotes the maximum latency that the task can tolerate, beyond which the task delivery fails; The tasks operate in a binary offloading mode, unmanned device U k The offloading decision is denoted by a j ∈{0,…,N} ; when a j =n, it means that task j will be executed by the computing unit of the unmanned device U n , joining the offloading queue waiting; that is, when a j =0, the task is locally computed, j joins the queue following the first come first served principle.

5. The method of claim 1, wherein: The random process of the task arrival in the step two is subject to a Poisson distribution, and a process A of the task J arriving at any information collection layer unmanned device k (t) modeled as a two-state Markov modulated Poisson process.

6. The method of claim 1, wherein: The communication model, the delay model and the energy consumption model of the system in step three specifically comprise: If the task is executed locally, the task j is executed on the unmanned device U k The computation latency is If the small unmanned device chooses to unload, since the unmanned device in the cell network is in a fixed position, the position of the small unmanned device is represented as (x k ,y k ,z k ), and the position of the other unmanned device U m at time t is (x m (t),y m (t),z m (t)), the distance between the two is According to the free space propagation model, the channel gain is where d0 represents a reference distance; according to the Shannon formula, the maximum transmission rate can reach where W represents a channel bandwidth, P s represents a signal power of a small unmanned device, β0 represents an average channel power gain at a reference distance d0 = 1 m, G km (t) represents a channel gain at time t, N0 represents a channel Gaussian white noise, and ψ represents an azimuth angle; the delay in a one-to-one transmission process between task j and U k and U m is If the object of unloading is not within the communication range, define the set N u representing the associated unmanned device on the unloading path, when α j = n, n ∈ N u and is the end point of unloading, the transmission time is the sum of multi-hop transmission, and the corresponding delay is: The total delay of a final task is:

7. The method of claim 1, wherein: The method for establishing the energy consumption model of the system in step three comprises: The energy loss includes the energy consumption cost of task j in the unloading and execution phases, the energy of CPU executing the task and the energy consumption of forwarding the task for unloading; If the task is executed locally, the task j at the unmanned device U k The energy consumed by the local computation is where κ is the effective switched capacitance depending on the chip architecture; if the task is offloaded to the computing node U n The corresponding energy consumption is The total energy loss of the final task is:

8. The method of claim 1, wherein: The basic elements of the Markov decision process in step four comprise: State space: The state observable by a single agent includes the active status of the coverage area devices, the congestion level of the local queue, and the status of other unmanned devices in the environment. The state set of all agents at time slot t is denoted as S(t) = {S A (t), S e (t), S o (t), S u (t)}; Action space: each drone makes a decision on the offloading object for task j based on observation, α j ∈{0,…,N}, α j >0 represents the server's offloading object, α j =0 indicates local computation; State transition: after the agent takes a certain action, the action interacts with the environment, and the process of the whole system state transition is Q Σ , whose generic element is defined as Q[s, s'] = Pr(S Σ (t) = s'| S Σ (t-1) = s), where s and s' are two constant states of the Markov chain; Reward: The goal of the joint optimization is to minimize the latency and energy cost generated by each agent, and the agent receives a reward, ω k (t) represents the agent U k The latency and energy cost of processing the task in time slot t; Calculate the reward function.

Images