Multi-objective optimized unmanned aerial vehicle auxiliary calculation unloading and resource allocation method

Abstract

The invention discloses a multi-objective optimized unmanned aerial vehicle auxiliary calculation unloading and resource allocation method. The method comprises the following steps: establishing a multi-unmanned aerial vehicle auxiliary calculation MEC system; taking the minimum system key information age and the energy consumption of a terminal user as optimization targets, and proposing a DIORA intelligent calculation unloading method based on DQN and DDPG algorithms; unmanned aerial vehicle position planning and terminal unloading decisions are determined through a DQN algorithm, and terminal transmitting power is determined through a DDPG algorithm. According to the method, the system key information age and the energy consumption of the terminal equipment can be effectively reduced by an intelligent unloading and resource allocation algorithm based on deep reinforcement learning according to the mobility of the terminal equipment and the dynamic requirements of a system environment.

Description

Technical Field

[0001] This invention relates to a computational offloading technology, particularly to unmanned aerial vehicle (UAV)-assisted computational offloading and resource allocation, and more specifically, to a multi-objective optimized UAV-assisted computational offloading and resource allocation method. Background Technology

[0002] With the rapid development of wireless communication technology and mobile computing, more and more applications are being installed on mobile smart devices. However, due to limitations in their own energy and computing power, smart mobile devices struggle to independently complete all computing tasks. Meanwhile, drone technology is rapidly advancing, possessing high flexibility and mobility, and can serve as an aerial edge server in mobile edge computing, providing computing services to ground-based mobile terminals. With the development of mobile communication and IoT technologies, there is a contradiction between the demand for high computing power and low latency from mobile applications and the limited energy supply and computing capabilities of end users. Mobile Edge Computing (MEC) systems mounted on drones have emerged as a potential technology for providing cloud computing services at the network edge. It can overcome the limitations of end users' capabilities and meet the needs of mobile applications. MEC systems place computing and storage resources at the network edge, providing low-latency computing services to mobile smart devices. Drones can be equipped with MEC systems to build a flexible mobile sensing platform. This configuration not only ensures a high-quality service experience for search and rescue equipment but also flexibly adapts to various practical application scenarios. In complex scenarios with multiple search and rescue devices, the task requirements of these devices change dynamically, making the allocation of computing resources and task offloading decisions particularly critical.

[0003] In multi-access edge computing systems, terminal devices often generate various tasks, each with a different criticality level. When the criticality level of a generated task changes, it is considered more important because it may affect certain key user decisions, and it is necessary to prevent users from making incorrect decisions due to missing updates to critical information. Simultaneously, when MEC servers may be unable to provide computing services due to a surge in business or ground-based failures, drones are needed to assist the edge computing system. Drones can flexibly adjust their positions to provide computing services to the terminal devices within the system. Terminal devices can choose to offload tasks to drones for processing or process them locally. The corresponding drone positioning optimization, terminal offloading decisions, and terminal power allocation issues directly impact system performance. Considering the mobility of terminal devices, the system environment in drone-assisted edge computing scenarios becomes more dynamic and complex, and traditional optimization methods cannot achieve optimal solutions. How to achieve intelligent offloading and resource allocation of drones for long-term, efficient computing services under the constraints of multiple mobile terminals and limited energy consumption remains a challenging research problem. Summary of the Invention

[0004] Purpose of the Invention: The purpose of this invention is to propose a multi-objective optimization method for UAV-assisted computation offloading and resource allocation. When ground-based mobile edge computing (MEC) servers are unable to provide services due to surges in business or failures, this method optimizes the age of critical system information and terminal energy consumption through dynamic deployment of UAVs and collaborative computation offloading. This invention aims to address the problem that traditional methods struggle to balance task real-time performance and energy efficiency in scenarios with dynamically changing terminal task criticality levels. It achieves joint optimization of UAV location planning, terminal offloading decisions, and power allocation through deep reinforcement learning, ensuring the freshness of critical tasks and reducing terminal energy consumption.

[0005] Technical solution: To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a centralized computing offloading and resource allocation method for multi-UAV assisted computing, comprising the following steps:

[0007] A centralized multi-UAV assisted computing offloading network architecture was constructed, consisting of a three-layer system comprised of terminal devices, multiple UAVs, and a central controller (Roadside Unit, RSU). UAVs are equipped with MEC servers and can move between fixed hovering points to provide computing services to the terminals. The RSU acts as the central controller, acquiring real-time terminal task status, UAV positions, and channel conditions, and centrally generating UAV position planning, terminal offloading decisions, and transmit power allocation strategies. The system's operating time is divided into multiple time slots. Terminals generate tasks with critical levels in each time slot, which can be processed locally or offloaded to the UAVs. The offloading process requires optimizing transmission power to balance energy consumption and information freshness.

[0008] A task generation and processing model is further established. Terminal-generated tasks include task size, computational requirements, and the maximum tolerable system key information age. Tasks can be processed locally or offloaded to the UAV. Local processing includes task computation, result uploading, and fusion; offloaded processing includes task transmission, UAV computation, and result feedback. A task processing latency and energy consumption model is defined, and the key information age is introduced to measure task timeliness.

[0009] Further optimization of the objective definition aims to minimize the weighted sum of the system's critical information age and terminal energy consumption, while also considering the maximum tolerable critical information age (constraining task processing timeout) and the maximum tolerable terminal power consumption (constraining transmit power threshold), thus constructing a multi-objective optimization problem:

[0010]

[0011] Where I(t) is the sum of the ages of all key task information in the system, it can be expressed as:

[0012]

[0013] e(t) represents the total energy consumption of all terminal devices within time slot t, which can be expressed as:

[0014]

[0015] Further development of the Intelligent Offloading Algorithm (DIORA) based on deep reinforcement learning aims to design an intelligent offloading and resource allocation algorithm (DIORA) based on deep reinforcement learning to achieve centralized decision-making. The state space encompasses dynamic information such as terminal task size, key information age, terminal and UAV positions, and channel status, providing real-time environmental feedback for decision-making. The action space includes discrete and continuous actions. Discrete actions (UAV position planning, terminal offloading decision) are output by a deep Q-network (DQN) through an ε-greedy strategy, while continuous actions (terminal transmit power) are generated by a deep deterministic policy gradient (DDPG) algorithm, with Gaussian noise introduced to balance exploration and utilization. The reward function is designed closely with key information age and terminal energy consumption, penalizing violations of constraints to guide the algorithm to prioritize task timeliness and terminal energy efficiency requirements. In terms of training mechanisms, experience replay and target network techniques are employed to promote collaborative updates between the DQN and DDPG networks. A soft update strategy enhances the stability of the training process. Finally, the central controller collects system states in real time and issues optimal decisions, forming a closed-loop optimized intelligent control system.

[0016] Furthermore, the joint optimization of the system's key information age and terminal energy consumption satisfies the following constraints: the age of the terminal's key information does not exceed the maximum tolerance value to ensure the freshness of information for time-sensitive services; the terminal's transmission power and total energy consumption do not exceed equipment limits to balance computational offloading efficiency and terminal endurance; and the UAV's position switching is within the preset hovering point range to ensure communication coverage and mobility feasibility.

[0017] Secondly, the present invention provides a centralized computing offloading and resource allocation system for multi-UAV assisted computing, comprising:

[0018] Scene building module: used to deploy drones, terminals and RSUs, and collect mission status and channel information in real time;

[0019] Optimization Modeling Module: Based on key system information such as age and energy consumption, a multi-objective optimization function is established and transformed into an MDP problem;

[0020] Reinforcement learning module: Integrates DQN and DDPG algorithms to train UAV position planning, offloading decision and power allocation strategies;

[0021] Dynamic execution module: Dynamically adjusts the drone's position and terminal resource allocation based on training results to ensure low-latency processing of critical tasks.

[0022] Beneficial Effects: This invention achieves joint optimization of UAV location planning, task offloading, and power allocation through deep reinforcement learning, significantly reducing terminal energy consumption while ensuring the real-time performance of critical tasks. Simulations show that compared to baseline algorithms (such as pure DQN and stochastic strategies), the DIORA algorithm improves performance by more than 25% on the weighted objective of system critical information age and terminal energy consumption, and exhibits strong robustness in multi-task load scenarios. Furthermore, the framework of centralized training and distributed execution balances global optimization and local adaptability, making it suitable for dynamic and complex disaster emergency scenarios.

[0023] In summary, this invention achieves joint optimization of a multi-UAV assisted computing offloading system by constructing a mathematical optimization model that incorporates key information age and terminal energy consumption, combined with a deep reinforcement learning algorithm. Simulation results show that, in the scenario of minimizing key information age, the algorithm's average total reward is improved by more than 25% compared to the baseline algorithm, and terminal energy consumption is reduced by 15%-20%. This provides a theoretically rigorous and engineeringly practical solution for ground MEC server failure scenarios, significantly improving the resource scheduling performance of multi-UAV assisted MEC systems in complex environments. Attached Figure Description

[0024] Figure 1 A schematic diagram of the MEC system for the multi-objective optimized UAV-assisted computational offloading and resource allocation method according to an embodiment of the present invention.

[0025] Figure 2 A detailed flowchart of the multi-objective optimized UAV-assisted computational unloading and resource allocation method according to an embodiment of the present invention.

[0026] Figure 3 This is a comparison chart showing the changes in the average total reward of the algorithm with the number of training rounds under different task arrival rates when α is set to 1 and δ is set to 0 in this embodiment of the invention.

[0027] Figure 4 This is a comparison chart showing the changes in the average total reward of the algorithm with the number of training rounds under different task arrival rates when α is set to 0 and δ is set to 1 in this embodiment of the invention.

[0028] Figure 5 This is a comparison of the average total reward of the algorithm with the number of training rounds under different task arrival rates when α is set to 0.5 and δ is set to 0.5 in this embodiment of the invention.

[0029] Figure 6 This is a comparison chart of the average key information of the system under different α and δ value settings in an embodiment of the present invention.

[0030] Figure 7This is a comparison chart of the average energy consumption of the terminal under different α and δ values ​​in the embodiments of the present invention.

[0031] Figure 8 This is a graph showing the comparison of the average total reward between the DIORA algorithm and the comparison algorithm under different task arrival rates when α is set to 1 and δ is set to 0 in this embodiment of the invention.

[0032] Figure 9 This is a graph showing the comparison of the average total reward between the DIORA algorithm and the comparison algorithm under different task arrival rates when α is set to 0 and δ is set to 1 in this embodiment of the invention. Detailed Implementation

[0033] To make the technical problems, technical solutions, and beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] exist Figure 1 The image shows a schematic diagram of a MEC system scenario describing a multi-objective optimization method for unmanned aerial vehicle (UAV)-assisted computation offloading and resource allocation. At an intersection, there are multiple moving end users, denoted by I = {1, 2, ..., i}, who generate tasks with different criticality levels. Combined with... Figure 1 This invention discloses a method for intelligent offloading and resource allocation of multi-UAV assisted computing for mobile terminals. First, a MEC scenario for UAV assisted computing on mobile terminals is constructed, in which j UAVs equipped with MEC servers and sensors (such as millimeter-wave radar and cameras) provide services within the system. Then, a task generation model is built based on the tasks generated on the terminal devices; a local processing model is built based on the local processing requirements of the terminal tasks; and an MEC server processing model is built based on the process of the terminal devices offloading tasks to the UAVs for processing. The optimization objective is clearly defined as the system's key information age and the terminal's energy consumption, and a trade-off between key information age and energy consumption is achieved by setting different α and δ values. Finally, an intelligent computing offloading algorithm model based on deep reinforcement learning is used, incorporating the MDP model and the DIORA algorithm for engineering training, effectively minimizing the key information age and terminal energy consumption.

[0035] exist Figure 2 The paper describes a detailed process of a multi-objective optimization method for unmanned aerial vehicle (UAV) assisted computation offloading and resource allocation. It uses Resource Unit (RSU) to formulate reasonable task offloading strategies and corresponding power allocation schemes for all terminals, and optimizes the position of UAVs as needed to improve the overall performance of the system.

[0036] The following is combined Figure 2A multi-objective optimization method for unmanned aerial vehicle (UAV)-assisted computation offloading and resource allocation is described in further detail. Specifically, the multi-objective optimization method for UAV-assisted computation offloading and resource allocation includes the following steps:

[0037] Step (1) establishes a multi-objective optimized UAV-assisted computing (MEC) scenario architecture. J UAVs equipped with MEC servers and sensors (such as millimeter-wave radar and cameras) provide services within the system, denoted as J = {1, 2, ..., j}. These UAVs can communicate with terminal devices at intersections and provide information fusion services to all terminals, such as optimized and collaborative driving, enhanced visibility, and accident warnings. The UAVs have several fixed hovering points and can switch between them as needed. All terminal devices can choose to offload generated tasks to the UAV for computation or perform local computation and then upload the results to the UAV. The RSU in the system is a central controller equipped with a computing unit. It can communicate with all devices within the system, obtain the current status of the devices, and centrally make decisions regarding terminal device offloading, determining transmission power, and planning UAV positions based on the current status. Since the positions of terminal devices and UAVs change dynamically over time, for ease of study, this chapter divides continuous time into time slots, denoted as T = {1, 2, ..., t}.

[0038] Step (2) establishes a task processing model including i end users, j UAVs equipped with MEC servers and sensors, specifically including

[0039] (2a) Establish a multi-UAV task generation model. The tasks generated on the terminal device can be represented as follows:

[0040] M(t) = {s(t), c(t), i(t)}

[0041] Where s(t) represents the total task size at time slot t, c(t) is the number of CPU cycles required to process 1 bit of task, and i(t) represents the maximum age of critical information allowed for the task.

[0042] The task generated by the i-th terminal in time slot t can be represented as follows:

[0043] M i (t)={s i (t), c i (t), a i (t)),

[0044] Where s i (t)=n i (t)k(t),n i (t) represents the number of tasks of the i-th terminal in time slot t, k i(t) represents the size of each task, and the size of the task follows a Poisson distribution with an arrival rate of λ. At the same time, each task generated in each time slot has an information category v(t).

[0045] Since resources on the terminal are limited, unloading operations need to be performed according to the needs of the task. The corresponding unloading decision can be represented as β. i (t). Where β i When (t) = 1, it means that the i-th terminal will unload the task to the nearest drone at time slot t. i When (t) = 0, it means that the task of the i-th terminal will be processed locally and the result will be uploaded to the nearest drone.

[0046] (2b) Establish a local task processing model for the terminal. The local task processing process of each terminal includes four stages: the terminal device processes the task on the local server; the result is uploaded to the MEC server, i.e., the drone; the uploaded result is processed on the MEC server and fused with the original information obtained by the sensor; and the result is downloaded from the MEC server.

[0047] Therefore, the execution time required by the local processing model is

[0048]

[0049] in This represents the time required for the i-th terminal to process the task locally in time slot t. This indicates the time required to upload the results to the MEC server. This refers to the time it takes for the MEC server to process the results. This represents the downlink transmission time from the MEC server to the terminal. Since both the results calculated locally at the terminal and the results processed by the MEC server are relatively small, this time is ignored. and Two items.

[0050] The time required for the terminal device to process the task locally can be calculated.

[0051]

[0052] in It is the CPU frequency of the local server of the terminal device.

[0053] Processing time on the MEC server It can be represented as

[0054]

[0055] Where f M(t) is the CPU frequency of the MEC server for processing tasks, x i (t) is the size of the local processing result of the i-th terminal in time slot t, x M (t) represents the size of the raw sensor information obtained by the UAV through the sensor.

[0056] Compared to x M (t), the result obtained from local processing at the terminal is small, so x can be ignored. i The term (t) can be rewritten as

[0057]

[0058] For the local processing model, the energy consumption of the terminal device in processing tasks locally can be expressed as:

[0059]

[0060] Among them κ i (t) is the effective energy coefficient, which depends on the chip structure on the terminal device.

[0061] (2c) Establishing a MEC server processing model, the process of the terminal device offloading the task to the UAV for processing is divided into three stages: sending the task to the UAV; the MEC server on the UAV processing the uploaded task and the raw perception information obtained by the sensors; and downloading the results from the MEC server. Therefore, the time required to upload the task to the MEC server for processing is...

[0062]

[0063] in It is the time required for the i-th terminal to upload the task to the MEC server in time slot t. This indicates the processing time of the uploaded task and the raw information obtained from the sensors on the MEC server. The downlink transmission time, representing the final result, is also negligible.

[0064] Use r ij (t) represents the uplink transmission rate between the i-th terminal device and the j-th drone in time slot t, then the task upload time of terminal i is... for

[0065]

[0066] The processing time for the task on the MEC server is

[0067]

[0068] The terminal device's power consumption at this time is mainly generated when uploading tasks, which can be expressed as:

[0069]

[0070] Where p i (t) is the transmit power of the terminal device.

[0071] Step (3) calculates the optimization objective. Since the focus is on the age of key system information and the energy consumption of terminal devices, each task generated by a terminal device has an information category. When the generated information category changes, the information is considered more important because it may indicate a change in some key information. The freshness of this key information can be measured by its age, defined as the relative age of key information that has a significant impact on user decisions. The age of the key information of the i-th terminal device at time slot t is expressed as...

[0072]

[0073] A younger critical information age means that terminal devices can capture critical-level changes as early as possible, enabling timely critical decisions. At the same time, with limited resources on terminal devices, reducing energy consumption is also crucial. Therefore, the optimization problem is formulated as a long-term multi-objective optimization problem.

[0074]

[0075] Where I(t) is the sum of the ages of all key task information in the system, which can be expressed as:

[0076] I(t)=∑ i∈I I i (t)

[0077] e(t) represents the total energy consumption of all terminal devices within time slot t, which can be expressed as:

[0078]

[0079] Furthermore, α and δ represent the weights of the key information's age and energy consumption, respectively, and α + δ = 1. A trade-off between the key information's age and energy consumption can be achieved by setting different values ​​for α and δ. max The most crucial piece of information is age, e max This represents the maximum energy consumption. Constraint C1 ensures that the critical information age of each terminal does not exceed the maximum tolerable age; C2 is a constraint on the terminal's energy consumption; C3 limits the transmit power of each terminal; and C4 represents the binary offload characteristic. Therefore, the above optimization problem is a nonlinear mixed-integer stochastic optimization problem. Furthermore, due to the transmit power p... i (t) is a continuous variable and the unloading decision β iThe location planning of (t) and the UAV is a discrete variable, and the problem is difficult to solve directly. Therefore, it is reformulated as an MDP and deep reinforcement learning is used to determine the appropriate offloading strategy and transmission power for each terminal, so as to minimize the age of key information and the energy consumption of terminal devices.

[0080] Step (4) takes into account load balancing among drones and drone energy consumption, establishes a system utility function, and establishes a deep reinforcement learning model with the goal of reducing the age of key system information and the energy consumption of terminal devices, including the following specific steps:

[0081] (4a) Constructs an MDP model for an intelligent computational offloading algorithm based on deep reinforcement learning, defined by the tuple {S, A, T, R}. In the multi-objective optimization problem studied, the transmit power p i (t) is a continuous variable, and the unloading decision β i Since the task offloading strategy (t) and the drone's position planning are discrete variables, a deep reinforcement learning-based intelligent offloading and resource allocation algorithm is proposed to determine the task offloading strategy, transmission power, and drone position for each terminal in the system. To better illustrate the proposed strategy, the state space S, action space A, and reward function R of the established system are first given. In each operation slot, the agent is in the current state s t Next, select a specific action a from action space A. t After performing this action, the agent receives a reward r. t As action a t The response, the current state s t It also becomes a new state s t+1 The specific state space, action space, and reward function are as follows:

[0082] State space S: The task size and key information age on each time slot terminal are variable, affecting action decisions. Simultaneously, due to the movement of the terminal and the drone, the distance between the drone and the terminal device is also dynamically changing. Therefore, the state space of time slot t can be represented as:

[0083] s t ={s t (t), I i (t), pos i (t), pos j (t)}

[0084] Action Space A: Each time slot UAV first plans its own position and makes an unloading decision β for the terminal equipment in the system. i (t) and transmission power P i (t) can be represented as:

[0085] at ={β i (t), p i (t), pos j (t)}

[0086] The actions here are output by two networks: the DQN network outputs the unloading decision of the terminal device and the UAV's position planning, and the DDPG network outputs the transmit power of the terminal device. The action space can be further divided into A = A N ∪A p and definition AN and AP are the action spaces for UAV position, unloading decision, and transmission power, respectively.

[0087] Reward R: Based on the definition of the optimization objective, the reward function in this chapter can be expressed as follows:

[0088]

[0089] Since α+δ=1, the reward ri ranges from [-1, 0]. When constraints C1 and C2 are not satisfied, the agent will receive a minimum reward of -1.

[0090] (4b), with state s t As input, appropriate actions are selected from the action space. The selected actions include UAV position planning, which is directly executed by the UAV; and all terminal unloading decisions and corresponding transmission power, which are transmitted to the terminals and executed by them. After all actions are executed, the agent receives the calculated reward as feedback. The DIORA algorithm then continues to update the action selection strategy based on the new state to obtain more rewards. The policy π is defined as a mapping from the state space to the action space. Under the guidance of policy π, the agent can select actions based on different states. t Choose the appropriate action to receive corresponding long-term cumulative discount rewards.

[0091] To obtain the optimal strategy π * Using the description of the current state s t The strategy is evaluated and improved using the Q-value function of the long-term cumulative discount reward Rt. The Q-value function is formulated as follows:

[0092] Q π (s t a t )E[R t |s t a t ]=E[r t +γQ π (s t+1 a t+1 )s t a t ]

[0093] For the entire action space and state space, the optimal policy π * It not only maximizes the long-term cumulative discount reward Rt, but also maximizes the Q-value function formula, strategy π. * The optimal Q-value function is expressed as follows:

[0094]

[0095] (4c) For DQN networks, they use an ε-greedy strategy to select actions. The core idea is to either randomly select an action with probability ε from the action space AN. Either choose the action with the highest Q value and probability of 1-ε, denoted as...

[0096]

[0097] Where θ represents the weights of the Q network.

[0098] After performing the action and receiving the corresponding reward, the interaction of each time slot is represented by a tuple (s). t, a t r t s t+1 The tuple is represented as () and stored in the experience buffer. When the buffer is full, the newest tuple replaces the oldest. An experience replay strategy is used to ensure that the samples for network training are of the same distribution and independent. In each training step, a batch of data of size N is randomly sampled from the experience buffer. For each tuple (s) randomly sampled from the experience buffer... t a t r t s t+1 The target Q-value generated by the target network can be expressed as:

[0099]

[0100] in The weights of the target network are represented by , therefore the Q-network can be trained by minimizing the following loss function.

[0101]

[0102] The weights of the target Q network are updated by periodically copying network parameters from the Q network.

[0103] (4d) In the DDPG algorithm, four networks are used. Two evaluation networks are the actor network for outputting actions and the critic network for evaluating the Q-values ​​of the output actions. The other two target networks are the target actor network and the target critic network for generating target values ​​for training the corresponding evaluation networks. The weights of the critic network and the target critic network are represented by η and η'. The weights of the actor network and the target actor network are represented by μ and μ. The actor network updates relevant parameters based on the deterministic gradient theorem to output appropriate actions. To balance the utilization of known actions and the exploration of new actions, random noise is added to the original output action, as shown below.

[0104]

[0105] in It is random noise that follows a normal distribution, and The value is restricted to the interval [0, p]. max ].

[0106] Since the action is determined by the deterministic strategy π(s) t The Q-value function can be expressed as μ) is generated.

[0107] Q π (s t a t )=E[r t +γQ π (s t+1 ,π(s t+1 ;μ))|s t, a t ]

[0108] The Q-value of the output action can be estimated by the critic network, and according to the deterministic policy gradient theorem, the weights μ in the actor network are updated in the direction of obtaining a larger cumulative discount reward, denoted as:

[0109]

[0110] The DDPG algorithm also employs an empirical replay strategy to reduce the correlation between samples. For N randomly sampled tuples from the empirical replay buffer, it can be approximated as:

[0111]

[0112] (4e) In each training step, the weights of the actor network are updated in the direction suggested by the critic network. Furthermore, the target Q-value can be generated by the target actor network and the target critic network, i.e.:

[0113]

[0114] (4f) The critic network is trained by minimizing the following loss function:

[0115]

[0116] A soft update method is used to slowly update the weights η and μ, thereby improving the stability of the training process.

[0117]

[0118] Where τ is a constant.

[0119] Step (5): Based on the deep reinforcement learning model, a simulation environment was built for simulation. A 400m×400m traffic intersection was established using appropriate simulation parameters. Two UAVs were set up in the system, flying at a fixed altitude and hovering at several fixed locations to provide computing services to the terminal devices in the system. The UAV altitude was set to 10m. The number of terminal users in the system was set to 4. Since it was a traffic intersection, the terminal users were modeled as vehicles with a maximum speed of 6m / s. The initial positions of all terminals were randomized, and their positions were updated at the maximum speed. Furthermore, the spatial positions of all terminals were considered quasi-static, meaning their positions did not change within each time slot, but were dynamic across multiple time slots. The task size generated by the terminal in each time slot followed a Poisson distribution with an arrival rate of λ. In the simulation, λ was used as an environmental variable to train and test the algorithm under different λ environments.

[0120] exist Figure 3 The document describes the comparison results of the average total reward of the algorithm with the number of training rounds under different task arrival rates when α is set to 1 and δ is set to 0 in an embodiment of the present invention. Since the reward depends entirely on the change of the key information age, the algorithm pays more attention to minimizing the key information age. Under different reward function settings, the algorithm achieves good convergence after 400 rounds of training. Moreover, the higher the task arrival rate, the lower the total reward obtained by the algorithm at convergence.

[0121] exist Figure 4 The document describes the comparison results of the average total reward of the algorithm with the number of training rounds under different task arrival rates when α is set to 0 and δ is set to 1 in an embodiment of the present invention. Since the reward depends entirely on the terminal's energy consumption, the algorithm focuses more on minimizing the terminal's energy consumption. Under different reward function settings, the algorithm achieved good convergence after 400 training rounds, and the higher the task arrival rate, the lower the total reward obtained by the algorithm at convergence.

[0122] exist Figure 5 The document describes the comparison of the average total reward of the algorithm with the number of training rounds under different task arrival rates when α and δ are set to 0.5 in an embodiment of the invention. In this case, the algorithm comprehensively considers the key information age of the task and the energy consumption of all terminals. Under different reward function settings, the algorithm achieved good convergence after 400 training rounds. Furthermore, the higher the task arrival rate, the lower the total reward obtained at convergence. This is because a higher task arrival rate requires processing a larger amount of tasks, leading to increased processing time and energy consumption, thus reducing the reward value.

[0123] exist Figure 6 This section describes the comparison results of the system's average critical information age under different α and δ values ​​in embodiments of the present invention. It can be seen that under different terminal task arrival rates, the system's average critical information age is lowest when α is set to 1 and δ to 0. This is because the algorithm's goal at this point is to minimize the critical information age. Under the constraint of satisfying the requirement, when the task volume is small or the wireless channel conditions are poor, the task will always be processed on the terminal, thus ensuring the minimization of the critical information age. When α is set to 0 and δ to 1, the system's average critical information age is highest. This is because the proposed algorithm at this point only focuses on minimizing terminal power consumption. Similarly, under the constraint of satisfying the requirement, the terminal will offload more tasks to the MEC server for processing, inevitably increasing the system's critical information age. It can also be seen that when α is set to 0.5 and δ is set to 0.5, the system's average information age is within a moderate range.

[0124] Figure 7 This diagram shows a comparison of the average terminal energy consumption under different α and δ values ​​in this embodiment of the invention. It can be seen that under different terminal task arrival rates, the system terminal average energy consumption is highest when α is set to 1 and δ to 0. This is because the algorithm's goal at this point is to minimize the critical information age. Under constraints, when the task volume is small or the wireless channel conditions are poor, tasks will always be processed on the terminal, resulting in higher terminal energy consumption. When α is set to 0 and δ to 1, the terminal average energy consumption is lowest. This is because the proposed algorithm at this point only focuses on minimizing terminal energy consumption. Similarly, under constraints, the terminal will offload more tasks to the MEC server for processing. It can also be seen that when α is set to 0.5 and δ is set to 0.5, the terminal average energy consumption is within a moderate range.

[0125] Figure 8The article describes the comparison of the average total reward between the DIORA algorithm and the comparison algorithm under different task arrival rates when α is set to 1 and δ is set to 0. The simulation compares four other algorithms, namely: (1) DQN: In the DQN strategy, the terminal's offloading decision, the determination of the transmission power, and the planning of the UAV's position are all output by the DQN algorithm. Since the DQN algorithm can only output discrete actions, while the terminal's transmission power is a continuous variable, the transmission power is uniformly quantized to 0 to p. max (2) Random + DDPG: The terminal randomly selects to unload the task to the UAV and outputs the transmission power at the time of unloading through the DDPG algorithm. (3) Unload + DDPG: The terminal always unloads the task to the UAV and outputs the transmission power at the time of unloading through the DDPG algorithm. (4) Local processing: All tasks are processed on the terminal.

[0126] The results show that the DIORA algorithm exhibits the best performance and adaptability, with an average total reward 18.6% higher than the random + DDPG strategy, 25.6% higher than DQN, and 45.5% higher than the offload + DDPG strategy. The reason for this is that in the offload + DDPG strategy, even with small tasks and poor channel conditions, tasks are uploaded to the UAV, resulting in longer task transmission times and consequently, higher system critical information age. In contrast, the local processing strategy always processes tasks on the terminal, avoiding high transmission latency under poor channel conditions. The algorithm proposed in this chapter, to meet the constraints, uploads a portion of the tasks to the UAV for processing based on task size and channel conditions; therefore, its average total reward is slightly lower than the local processing strategy.

[0127] Figure 9This paper describes the comparison of the average total reward of the DIORA algorithm and the contrasting algorithms under different task arrival rates, with α set to 0 and δ set to 1. Simulations were also conducted to compare four other algorithms: DQN, Random + DDPG, Offload + DDPG, and Local Processing. The results show that the average total reward of the proposed algorithm is slightly lower than that of the Offload + DDPG strategy. Similarly, when the terminal task arrival rate is 10%, the DIORA strategy has an average total reward 19.1% higher than DQN, 67.3% higher than the Random + DDPG strategy, and 79.6% higher than the Local Processing strategy. In the Offload + DDPG strategy, all tasks are offloaded to the UAV for processing, and the terminal's energy consumption only comes from sending tasks; therefore, it has the highest reward value when minimizing energy consumption. Under the Local Processing strategy, all tasks are processed locally, directly leading to the highest terminal energy consumption. The DIORA strategy, however, only processes tasks locally when channel conditions are poor or the task volume is small, thus achieving a higher average total reward. In summary, the DIORA algorithm performs well when different emphases are placed on the age of key information and the power consumption of the terminal. Furthermore, when the algorithm is tested under different environmental settings, its overall performance is the best, which further demonstrates the effectiveness of the algorithm.

[0128] Based on the description of the present invention, those skilled in the art should readily recognize that the present invention proposes a multi-objective optimized UAV-assisted computational unloading and resource allocation method, which can effectively reduce the age of critical system information and the energy consumption of terminal devices.

[0129] The program code used to implement the method of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the steps of the method of the present invention to be performed. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a standalone software package, or entirely on a remote machine or server. All aspects not detailed in this invention are well-known to those skilled in the art.

Claims

1. A multi-objective optimization method for unmanned aerial vehicle (UAV)-assisted computational unloading and resource allocation, characterized in that, Includes the following steps: Based on the age of key information generated by the system for each mobile terminal, the system determines the speed at which the system captures changes at the key level on the terminal device, thereby optimizing decision-making fairness. Based on the terminal device's resources, a trade-off between key information age and energy consumption is achieved by setting weights for these factors. This determines the task offloading strategy, transmission power, and drone location for each terminal, balancing improved operational efficiency with reduced drone energy consumption. A trial-and-error system function is established, and a deep reinforcement learning model is built for the system determining task offloading, aiming to maximize the utility function. Deep reinforcement learning is then used to determine appropriate offloading strategies and transmission power for each mobile terminal, minimizing both key information age and terminal device energy consumption.

2. The multi-objective optimization UAV-assisted computational unloading and resource allocation method according to claim 1, characterized in that, The task generated on the terminal device can be represented by M(t) = {s(t), c(t), i(t)}, where s(t) represents the total task size in time slot t, c(t) is the number of CPU cycles required to process 1 bit of task, and i(t) represents the maximum allowed critical information age of the task. The task generated by the i-th terminal in time slot t can be represented as follows: Where s i (t)=n i (t)k i Let (t) represent the number of tasks at time slot t for the i-th terminal, and let represent the size of each task. The task size follows a Poisson distribution with arrival rate λ. Each task generated in each time slot has an information category. Due to limited resources on the terminal, offloading operations are required based on task needs. The corresponding offloading decision can be represented as β. i (t), β i When (t) = 1, it means that the i-th terminal will unload the task to the nearest drone at time slot t. i When (t) = 0, it indicates that the task of the i-th terminal will be processed locally and the result will be uploaded to the nearest drone. The system includes multiple mobile terminals and multiple drones. The age of the key information of the terminal device may change each time it generates a task. In order to save the limited resources on the terminal device and reduce the energy consumption of the terminal, the weights of the key information age and energy consumption are adjusted. At this time, it is difficult to solve the optimization problem directly as a long-term multi-objective optimization problem. This invention reformulates it as an MDP and uses deep reinforcement learning to determine the appropriate offloading strategy and transmission power for each terminal to achieve the goal of minimizing the key information age and the energy consumption of the terminal device.

3. The multi-objective optimization method for unmanned aerial vehicle (UAV) assisted computational unloading and resource allocation according to claim 1, wherein the MDP model has a continuous variable for launch power and a discrete variable for unloading decision and UAV position planning. It comprehensively utilizes the DQN algorithm, actor-critic algorithm and DDPG algorithm, which solves the problems that can only solve discrete action spaces and involve two neural networks, and that the parameters are correlated before and after each parameter update process, leading to convergence difficulties.

4. According to claim 1, the proposed DIORA algorithm uses the state as input, selects appropriate actions in the action space, including UAV position planning, which is directly executed by the UAV. The unloading decisions and corresponding transmission powers of all terminals are transmitted to the terminals and then executed by the terminals. The calculated rewards after all actions are executed are used as feedback, and the action selection strategy is then updated to obtain more rewards. To obtain the optimal strategy, a Q-value function describing the long-term cumulative discount reward Rt in the current state is used to evaluate and improve the strategy. The Q-value function is formulated as follows: For the entire action space and state space, the optimal policy not only maximizes the long-term cumulative discounted reward Rt, but also maximizes equation (5.17). The optimal Q-value function under this policy is expressed as: The input to both networks is the state information s acquired by the UAV at the beginning of each time slot. t For DQN networks, an ε-greedy strategy is used to select actions. Its core idea is to either randomly select an action with probability ε from the action space AN. Either choose the action with the highest Q value and a probability of 1-ε.

5. According to claim 4, for the DQN network, which uses an ε-greedy strategy to select actions, the core idea is either to randomly select an action with probability ε from the action space AN, or to select the action with the highest Q value and probability 1-ε, expressed as: Where θ represents the weights of the Q network.

6. The multi-objective optimization UAV-assisted computational unloading and resource allocation method according to claim 1, characterized in that, The task size and key information age on each time slot terminal are variable, affecting action decisions. Furthermore, due to the movement of both the terminal and the drone, the distance between them is also dynamically changing. Therefore, the state space of time slot t can be represented as: s t ={s i (t),I i (t),pos i (t),pos j (t)} 7. The multi-objective optimization UAV-assisted computational unloading and resource allocation method according to claim 1, characterized in that, The action space A refers to the process where each time slot UAV first plans its own position and formulates an unloading decision β for the terminal equipment in the system. i (t) and transmission power p i (t) can be represented as: a i ={β i (t),p i (t),pos j (t)} 8. The multi-objective optimization method for UAV-assisted computational unloading and resource allocation according to claim 1, characterized in that, The key information age of the i-th terminal device at time slot t is represented as: A younger critical information age means that terminal devices can capture critical-level changes as early as possible, thereby making certain critical decisions in a timely manner. At the same time, terminal devices have limited resources, and reducing the terminal's energy consumption is also very important. Therefore, the optimization problem is formulated as a long-term multi-objective optimization problem: Where I(t) is the sum of the ages of all key task information in the system, it can be expressed as: e(t) represents the total energy consumption of all terminal devices within time slot t, which can be expressed as: Furthermore, α and δ represent the weights of the key information's age and energy consumption, respectively, and α + δ = 1. A trade-off between the key information's age and energy consumption can be achieved by setting different values ​​for α and δ. max The most crucial piece of information is age, e max The maximum energy consumption is given by constraint C1, which ensures that the age of the critical information of each terminal does not exceed the maximum tolerable age. C2 is a constraint on the terminal's energy consumption, C3 limits the transmission power of each terminal, and C4 represents the binary offload characteristic. Therefore, the above optimization problem is a nonlinear mixed-integer stochastic optimization problem. Furthermore, due to the transmission power p... i (t) is a continuous variable and the unloading decision β i (t) and the drone's location planning are discrete variables, so they are reformulated as MDPs. Deep reinforcement learning is used to determine the appropriate offloading strategy and transmission power for each terminal, so as to minimize the age of key information and the energy consumption of terminal devices.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of a multi-objective optimized UAV-assisted intelligent unloading and resource allocation method according to any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of a multi-objective optimized UAV-assisted intelligent unloading and resource allocation method according to any one of claims 1-5.

Images