A method and device for unmanned aerial vehicle task offloading based on deep reinforcement learning

Abstract

The application provides a kind of unmanned aerial vehicle task unloading method and device based on deep reinforcement learning, it is related to vehicle networking technical field.The method comprises: according to the computing capacity of vehicle, task is handled by vehicle or edge server;If handled by edge server, the vehicle position is obtained by constructing the Gaussian-Markov mobility model of vehicle, according to the computing resource of unmanned aerial vehicle and the distance between vehicle and intelligent roadside facility, it is judged by unmanned aerial vehicle or by unmanned aerial vehicle and intelligent roadside facility;If by unmanned aerial vehicle, the path planning result of unmanned aerial vehicle is output according to DGBCO model;If by intelligent roadside facility, task allocation is optimized according to computing reuse technology;Unmanned aerial vehicle energy constraint problem based on Lyapunov optimization is constructed;MADDPG algorithm is used to obtain task unloading strategy.The application optimizes UAV trajectory, designs a kind of joint optimization method in combination with the dynamic change of the mobility and computing resource of vehicle, improves the comprehensive performance of system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Vehicles, and particularly refers to a method and device for unmanned aerial vehicle task offloading based on deep reinforcement learning. BACKGROUND

[0002] With the rapid development of automatic driving, intelligent transportation and other technologies, the massive data generated in modern VANETs (Vehicular Ad-Hoc Networks) rapidly increases the demand for computing power and communication resources. This puts higher requirements on the existing network infrastructure, especially in multi-device, multi-task scenarios. How to efficiently process these tasks has become a problem to be solved. In order to cope with these challenges, the combination of MEC (Multi-Access Edge Computing) and UAV (Unmanned Aerial Vehicle) technology provides a new solution for vehicle task offloading.

[0003] MEC can effectively reduce the transmission delay of computing tasks and reduce the bandwidth burden of the core network by deploying computing resources at the edge of the network. UAVs further expand the application scenarios of MEC with their flexible mobility and wide coverage, especially in dynamic vehicle environments. The integration of UAVs and MEC provides a high probability of LoS (Line-of-Sight) communication links, which can significantly improve the communication coverage, capacity and reliability of the network. In addition, the flexibility of UAVs allows them to be quickly deployed in areas lacking ground infrastructure, providing on-demand computing resources for multi-device tasks in vehicle networks.

[0004] Although UAV-assisted MEC technology has many advantages, it still faces many challenges in practical applications: (1) In the vehicle environment, the tasks performed by different UDs (User Devices) are usually heterogeneous and time-varying. The demand for offloading services is often very strict. Due to the limited battery capacity of UAVs, their flight time is greatly limited. This not only requires optimizing the use of UAVs, but also balances energy consumption and task completion efficiency while ensuring task quality, so how to effectively allocate resources to meet different task demands under resource constraints is a key problem. (2) The offloading decisions of UDs have complex coupling. The offloading decision of each device not only affects its own computing task, but also is affected by the offloading decisions of other devices. This increases the computational complexity of the task offloading process. (3) The mobility of UAVs improves the flexibility of the system, but also brings complexity to its trajectory planning. In order to optimize task offloading, how to reasonably plan the flight path of the UAV becomes a challenge.

[0005] To address these issues, a UAV-cooperative V2X multi-device task offloading framework based on DRL (Deep Reinforcement Learning) is proposed. By jointly optimizing task offloading, resource allocation, and UAV trajectory planning, the framework aims to minimize the total time delay of the system and minimize the total energy consumption of the system.

[0006] In recent years, with the widespread application of UAVs and multi-access edge computing technology in V2X, researchers have proposed various optimization methods for task offloading and resource allocation, aiming to improve system performance and meet the growing demand for computing resources by vehicles.

[0007] Prior art

L. He et al., "An Online Joint Optimization Approach for QoE Maximization in UAV-Enabled Mobile Edge Computing," IEEE INFOCOM 2024 - IEEE Conference on Computer Communications, Vancouver, BC, Canada, 2024, pp. 101-110, doi: 10.1109 / INFOCOM52122.2024.10621306.

[0008] Prior art

X. Dai, Z. Xiao, H. Jiang and J. C. S. Lui, "UAV-Assisted Task Offloading in Vehicular Edge Computing Networks," in IEEE Transactions on Mobile Computing, vol. 23, no. 4, pp. 2520-2534, April 2024, doi: 10.1109 / TMC.2023.3259394.

[0009] Prior art

D. Yang, J. Wang, F. Wu, L. Xiao, Y. Xu and T. Zhang, "Energy Efficient Transmission Strategy for Mobile Edge Computing Network in UAV-Based Patrol Inspection System," in IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 5984-5998, May 2024, doi: 10.1109 / TMC.2023.3315477.

[0010] Prior art

P. A. Apostolopoulos, G. Fragkos, E. E. Tsiropoulou and S. Papavassiliou, "Data Offloading in UAV-Assisted Multi-Access Edge Computing Systems Under Resource Uncertainty," in IEEE Transactions on Mobile Computing, vol. 22, no. 1, pp. 175-190, 1 Jan. 2023, doi: 10.1109 / TMC.2021.3069911.

[0011] However, many edge computing scenarios are dynamically changing over time, such as real-time video analysis, which means that computing tasks are randomly arriving, computing demands of vehicles are time-varying, and vehicles are dynamically moving. Existing UAV MEC research mostly only studies UAV trajectory optimization problems without considering that vehicles are also dynamically moving. SUMMARY

[0012] To solve the technical problem that existing UAV MEC research mostly only studies UAV trajectory optimization problems without considering that vehicles are also dynamically moving, embodiments of the present application provide a UAV task offloading method and device based on deep reinforcement learning. The technical solution is as follows:

[0013] On the one hand, a UAV task offloading method based on deep reinforcement learning is provided, which is implemented by a UAV task offloading device, and the method comprises:

[0014] S1, a UAV MEC system model is constructed; wherein the UAV MEC system model comprises a plurality of dynamically moving vehicles and a plurality of edge servers, and the plurality of edge servers comprise a plurality of UAVs and a plurality of intelligent roadside facilities.

[0015] S2, any vehicle in the plurality of dynamically moving vehicles determines whether to process a generated task by the vehicle or offload the task to an edge server for processing by the edge server according to the computing capability of the vehicle after the task is generated.

[0016] S3, constructing a Gaussian-Markov mobility model of the vehicle when the task is processed by the edge server, obtaining a vehicle position according to the Gaussian-Markov mobility model of the vehicle, and obtaining a distance between the vehicle and the intelligent roadside facility according to the vehicle position.

[0017] determining whether the computing resource of the UAV is sufficient to complete the task.

[0018] If sufficient, the vehicle offloads the task to the UAV, the UAV executes the task, and the obtained swarm size of the UAV and the preset iteration number are input into the DGBCO model, and a path planning result of the UAV is output.

[0019] If insufficient, it is determined whether the distance between the vehicle and the intelligent roadside facility is less than the longest communication distance; if less, the vehicle offloads the task to the intelligent roadside facility, and the intelligent roadside facility executes the task; and if not less, the task is forwarded to the intelligent roadside facility for execution through the UAV as a relay node.

[0020] The task is forwarded to the intelligent roadside facility for execution through the UAV as a relay node, including:

[0021] S31, according to a calculation reuse technology, preferentially assigning the task to an edge server with a high request probability.

[0022] S32, constructing a UAV energy constraint problem based on Lyapunov optimization.

[0023] S33, using a MADDPG algorithm to solve the UAV energy constraint problem, obtaining a task offloading strategy, and forwarding the task to the intelligent roadside facility for execution through the UAV as a relay node according to the task offloading strategy; the global critic of the MADDPG algorithm uses a double-delay deterministic policy gradient.

[0024] Optionally, the Gaussian-Markov mobility model of the vehicle is constructed, the vehicle position is obtained according to the Gaussian-Markov mobility model of the vehicle, and the distance between the vehicle and the intelligent roadside facility is obtained according to the vehicle position, including:

[0025] The speed update of the vehicle at time slot t+1 is constructed, as shown in the following formula (1):

[0026]

[0027] In the formula, v v (t+1) represents the speed of the vehicle at time slot t+1, α represents a memory level, represents a speed vector at time slot t, represents the speed of the vehicle in the x-axis direction, represents the speed of the vehicle in the y-axis direction, W represents the asymptotic mean of the speed. v (t) represents an uncorrelated random Gaussian process.

[0028] The vehicle position update at time slot t+1 is constructed according to the speed update, as shown in the following formula (2):

[0029] P v [t+1]=P v [t]+v v [t] (2)

[0030] In the formula, P v [t+1] represents the vehicle position at time slot t+1, P v [t]=(x v [t],y v [t]) represents the position of vehicle v at time slot t, x v [t] represents the position of the vehicle in the x-axis direction, and y v [t] represents the position of the vehicle in the y-axis direction.

[0031] The distance between the vehicle and the intelligent roadside facility is constructed according to the vehicle position update, as shown in the following formula (3):

[0032]

[0033] In the formula, d v,r (t) represents the distance between the vehicle and the intelligent roadside facility at time slot t, x r represents the position of the intelligent roadside facility in the x-axis direction, and y r represents the position of the intelligent roadside facility in the y-axis direction.

[0034] Optionally, the calculation reuse technology in S31 comprises:

[0035] Obtain the comment of each vehicle on each task, transmit the comment to the intelligent roadside device, obtain the comment sentiment value of each task according to the comment and the BERT model, normalize the comment sentiment values of all tasks, and obtain the request probability matrix of the task.

[0036] Optionally, the construction of the unmanned aerial vehicle energy constraint problem based on Lyapunov optimization in S32 comprises:

[0037] Define a first virtual energy queue Q represents the calculation energy queue at time slot t, define a second virtual energy queue Q represents the propulsion energy queue at time slot t.

[0038] According to the first virtual energy queue Q and the second virtual energy queue Q Define the Lyapunov function L(Qu (t)), the Lyapunov function represents a scalar measure of the queue backlog.

[0039] According to the Lyapunov function, a conditional Lyapunov drift ΔL(Q u (t)) at time slot t is defined.

[0040] The drift is penalized, and an upper bound of the penalized drift is constructed.

[0041] According to the upper bound, a Lyapunov-based optimization problem of the UAV energy constraint is obtained, as shown in the following formula (4) :

[0042]

[0043] In the formula, A t represents a task offloading strategy at time slot t, F t represents a computing resource allocation at time slot t, W t represents a communication resource allocation at time slot t, D' u =D u represents the positions of the vehicle and the UAV at time slot t+1, D u represents the positions of the vehicle and the UAV at time slot t, represents the computing energy of the UAV at time slot t, represents the propulsion energy of the UAV at time slot t, H represents a parameter for balancing the total cost and the queue stability, V represents a set of all vehicles, and C(t) represents a task cost.

[0044] Optionally, the state of the vehicle in the MADDPG algorithm in S33 is as shown in the following formula (5) :

[0045]

[0046] In the formula, represents a task state, t represents a time slot, z v (t) represents a data size of the task, c v (t) represents a computing intensity of the task, τ v (t) represents a priority of the task, represents a normalized channel gain state, represents a power transmission budget, represents a maximum computing capability of the vehicle, represents a local resource allocation budget, represents a maximum computing resource of the vehicle.

[0047] Optionally, the behavior in the MADDPG algorithm in S33 includes a client agent behavior and a master agent behavior.

[0048] Where, the client agent behavior is as shown in the following formula (6):

[0049]

[0050] In the formula, A c (t) represents the client agent behavior, a v (t) represents the decision of the vehicle v to offload the task, p c,v (t) represents the client action that decides the transmission power, f c,v (t) represents the action that decides the local computing resource allocation, θ v represents the parameterized policy function of the client agent, S v (t) represents the state of the vehicle v, t represents the time slot, and V represents the set of all vehicles.

[0051] The master agent behavior is as shown in the following formula (7):

[0052]

[0053] In the formula, A m (t) represents the master agent behavior, x m,v (t) represents the decision of the master agent for the client agent at time slot t, represents the master policy, S represents the state set of all client agents, A represents the action set of all client agents, S c represents the state set of the client agent, A c represents the action set of the client agent.

[0054] Optionally, the reward function in the MADDPG algorithm in S33 is as shown in the following formula (8):

[0055]

[0056] In the formula, r(S(t),A(t)) represents the reward function, S(t) represents the state set at time slot t, A(t) represents the action set at time slot t, V represents the set of all vehicles, and P' represents the UAV energy constraint problem based on Lyapunov optimization.

[0057] In another aspect, a deep reinforcement learning-based UAV task offloading device is provided, which is applied to a deep reinforcement learning-based UAV task offloading method, and the device comprises:

[0058] A construction module is configured to construct a UAV MEC system model; wherein the UAV MEC system model comprises a plurality of dynamically moving vehicles and a plurality of edge servers, and the plurality of edge servers comprises a plurality of UAVs and a plurality of intelligent roadside facilities.

[0059] A judgment module is configured to judge, for any vehicle in the plurality of dynamically moving vehicles, whether a generated task is processed by the vehicle or unloaded to an edge server for processing by the edge server according to a computing capability of the vehicle after the task is generated.

[0060] An output module is configured to construct a Gaussian-Markov mobility model of the vehicle when the task is processed by the edge server, obtain a vehicle position according to the Gaussian-Markov mobility model of the vehicle, and obtain a distance between the vehicle and the intelligent roadside facility according to the vehicle position.

[0061] The computing resources of the UAV are judged to be sufficient to complete the task.

[0062] If the computing resources are sufficient, the vehicle unloads the task to the UAV, the UAV executes the task, and the obtained swarm size of the UAV and the preset iteration number are input to the DGBCO model to output a path planning result of the UAV.

[0063] If the computing resources are insufficient, it is judged whether the distance between the vehicle and the intelligent roadside facility is less than a longest communication distance; if the distance is less than the longest communication distance, the vehicle unloads the task to the intelligent roadside facility, and the intelligent roadside facility executes the task; and if the distance is not less than the longest communication distance, the task is forwarded to the intelligent roadside facility for execution by the UAV as a relay node.

[0064] The task is forwarded to the intelligent roadside facility for execution by the UAV as a relay node, and includes:

[0065] S31, according to a computing reuse technology, a task is preferentially assigned to an edge server with a high request probability.

[0066] S32, a UAV energy constraint problem based on Lyapunov optimization is constructed.

[0067] S33, a MADDPG algorithm is used to solve the UAV energy constraint problem to obtain a task offloading strategy, and the task is forwarded to the intelligent roadside facility for execution by the UAV as a relay node according to the task offloading strategy; a global critic of the MADDPG algorithm adopts a double-delay deterministic policy gradient.

[0068] In another aspect, a UAV task offloading device is provided, and the UAV task offloading device includes a processor and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement any one of the above-described deep reinforcement learning-based UAV task offloading methods.

[0069] In another aspect, a computer readable storage medium is provided, and the storage medium stores at least one instruction, the at least one instruction being loaded and executed by a processor to implement any one of the above-described deep reinforcement learning-based UAV task offloading methods.

[0070] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0071] In the present application, efficient task offloading: through the cooperative work of vehicles and drones, efficient task offloading is achieved, and the utilization of computing resources is optimized.

[0072] Reducing delay and energy consumption: introducing computing reuse and caching technology, reducing waste of computing resources, and optimizing task offloading strategy, computing resource and communication resource allocation through Lyapunov optimization method, significantly reducing task processing delay and system energy consumption.

[0073] Improving system throughput: multi-agent deep reinforcement learning algorithm (MADDPG) improves task completion rate and system throughput in complex vehicle networking environment.

[0074] Enhancing robustness: compared with traditional deep reinforcement learning algorithm, the algorithm of the present application shows better robustness in high load and dynamic scene.

[0075] Flexibility and efficiency: the double-stage group bee optimization algorithm (DGBCO) shows excellent flexibility and efficiency in the aspect of drone trajectory planning.

[0076] Theoretical basis and technical support: it provides a new solution for unmanned aerial vehicle assisted multi-access edge computing (MEC) of vehicle networking, and provides theoretical basis and technical support for resource optimization scheduling in future vehicle networking application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0078] Figure 1 is a flow chart of a drone task offloading method based on deep reinforcement learning provided by the embodiments of the present application;

[0079] Figure 2 is a system design diagram of a drone task offloading method based on deep reinforcement learning provided by the embodiments of the present application;

[0080] Figure 3 is a scene diagram of a drone MEC system provided by the embodiments of the present application;

[0081] Figure 4 is a block diagram of a drone task offloading device based on deep reinforcement learning provided by the embodiments of the present application;

[0082] Figure 5 Figure 1 is a structural schematic diagram of a UAV task offloading device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0083] The technical solutions in the present application will be described below with reference to the drawings.

[0084] In the embodiments of the present application, the words such as "example", "for example" and the like are used to represent an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0085] In the embodiments of the present application, "image" and "picture" can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent. "Of", "corresponding" and "corresponding" can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent.

[0086] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1. When the distinction is not emphasized, the meanings expressed are consistent.

[0087] In order to make the technical problems, technical solutions and advantages of the present application clearer, specific embodiments will be described in detail below with reference to the drawings.

[0088] The embodiments of the present application provide a UAV task offloading method based on deep reinforcement learning. The method can be implemented by a UAV task offloading device, which can be a terminal or a server. As shown in Figure 1 、 Figure 2 The flowchart of the UAV task offloading method based on deep reinforcement learning, the processing flow of the method can include the following steps:

[0089] S1, a UAV MEC system model is constructed; wherein the UAV MEC system model includes a plurality of dynamically moving vehicles and a plurality of edge servers, and the plurality of edge servers include a plurality of unmanned aerial vehicles and a plurality of intelligent roadside facilities.

[0090] In a feasible implementation manner, as shown in Figure 3As shown, the present application designs a random UAV MEC system model, which includes multiple UAVs and multiple dynamic moving ground vehicles. The UAVs serve as air edge servers to provide computing services for vehicles with time-varying computing demands. The system model considers the constraints of energy and resources.

[0091] Specifically, in the proposed vehicle networking multi-device task offloading framework, it is assumed that SRI (Smart Roadside Infrastructure) is deployed as an edge server beside the urban road, and the UAVs serve as edge servers or relay nodes in the air to provide services for vehicles. In the simulation environment, the present application assumes that the vehicle application selects a server within its communication range to communicate.

[0092] Further, each task is modeled as an independent and indivisible object, which can be completely processed by the vehicle terminal or the edge server. When the task is generated, the vehicle terminal decides whether the task is processed locally or offloaded to the edge server for processing according to the task offloading strategy. The present application designs and analyzes the communication and computing model in the vehicle networking-UAV-edge server environment, aiming to minimize the delay, energy consumption and total cost of the system.

[0093] S2, any of the plurality of dynamic moving vehicles determines whether the generated task is processed by the vehicle or offloaded to the edge server for processing by the edge server according to the computing capability of the vehicle after the task is generated.

[0094] S3, when the task is processed by the edge server, a Gaussian-Markov mobility model of the vehicle is constructed, the vehicle position is obtained according to the Gaussian-Markov mobility model of the vehicle, and the distance between the vehicle and the smart roadside infrastructure is obtained according to the vehicle position.

[0095] In a feasible implementation, the importance of the movement model needs to be carefully considered because they describe the way the mobile device moves, including direction, speed and acceleration, all of which are crucial for making appropriate scheduling decisions. In the context of the vehicle edge computing offloading model, in order to optimize the delay and energy consumption, the task is offloaded to the nearby edge server, which requires the use of accurate movement models to predict the position of the vehicle terminal and offload the task to the nearest edge server at the appropriate time.

[0096] Specifically, the mobility of the vehicle is modeled as a Gaussian-Markov mobility model, which is widely used in cellular communication networks. Specifically, the speed of the vehicle v at time slot t+1 is updated as:

[0097]

[0098] In the formula, v v(t+1) represents the speed of the vehicle at time slot t+1, a represents the memory level, reflecting the degree of time dependence, represents the speed vector at time slot t, represents the speed of the vehicle in the x-axis direction, represents the speed of the vehicle in the y-axis direction, represents the asymptotic mean of the speed, W v (t) represents an uncorrelated random Gaussian process where σ v represents the asymptotic standard deviation of the speed.

[0099] Similarly, the speed update of the UAV u at time slot t+1 is:

[0100]

[0101] Therefore, the vehicle position update at time slot t+1 is:

[0102] P v [t+1] = P v [t] + v v [t] (3)

[0103] In the formula, P v [t+1] represents the position of the vehicle at time slot t+1, P v [t] = (x v [t], y v [t]) represents the position of the vehicle v at time slot t, x v [t] represents the position of the vehicle in the x-axis direction, y v [t] represents the position of the vehicle in the y-axis direction, P v [1] represents the initial position of the vehicle v. At the beginning of time slot t, the present application assumes that the position feedback of the vehicle knows the current position P v [t], but the future trajectory is currently unknown.

[0104] Similarly, the position update of the UAV at time slot t+1 is:

[0105] P u [t+1] = P u [t] + v u [t] (4)

[0106] When the distance between the vehicle and the edge server is d v,r (t) is less than the maximum communication distance d max , the vehicle directly offloads the task to the edge server:

[0107]

[0108] where d v,r (t) represents the distance between the vehicle and the smart roadside facility at time slot t, x r represents the position of the smart roadside facility in the x-axis direction, y r represents the position of the smart roadside facility in the y-axis direction.

[0109] When greater than the longest communication distance, the trajectory optimization problem of the vehicle and the UAV is involved:

[0110]

[0111] The distance between the UAV and the smart roadside facility is similar:

[0112]

[0113] Further, it is judged whether the computing resources of the UAV are sufficient to complete the task.

[0114] If sufficient, the vehicle offloads the task to the UAV, the UAV performs the task, and the obtained UAV group size and the preset iteration number are input into the DGBCO model, and the path planning result of the UAV is output.

[0115] In a feasible implementation, the model proposed in the application combines different UAV path planning meta-heuristic algorithms. In the application, two stages of the DGBCO algorithm are discussed in detail, that is, the exploration stage for solving a complex path planning environment and the utilization stage. The particles are randomly assigned to the DGBCO in the entire search space. The equation used to specify the initial value is:

[0116] X(i)-Min X +rand*(Max X -Min X )(8)

[0117] where X(i) is the particle position, Min X is the minimum value, and Max X is the maximum value of the search space.

[0118] Further, since the DGBCO has a sub-group of exploration and development, which accounts for 70%-30% of the total participants. In the initial stage, a larger search space is needed to explore the global optimum, so the exploration group is 70% and the exploitation group is 30%. With iteration, the development group will increase to carry out more development.

[0119] The group plays an important role in the process of finding the global optimal solution in the entire search space. Therefore, the exploration stage of the algorithm needs to effectively solve engineering problems. In the DGBCO, the particle position of the exploration group is updated by two techniques.

[0120] The first is to search around the current position for the best solution using the following equation:

[0121]

[0122] where X(i) and X(i+1) are the current and updated position of the drone. r1 and r2 are randomly normally distributed numbers. d is the distance around the solution to search. The second method is mutation, which uses genetic operators to create and maintain diversity in the population.

[0123] In this swarm, the particles find the best solution of the swarm and converge towards the global optimum solution. Therefore, the exploitation phase also adopts two strategies to update the particle position. In the first strategy, the particles search towards the global optimum in a circular form. This technique can be used in DGBCO:

[0124]

[0125] where G(i) represents the best solution and r3 is a random number. The second method to update the particle position is to search around the global optimum solution.

[0126] This technique can be used:

[0127]

[0128] where r4 and r5 are randomly generated numbers and k is a value that exponentially decreases from 2 to 0.

[0129] Elitism is another property used by DGBCO, where the global best is found in each iteration and passed to the next iteration.

[0130] The DGBCO pseudo code is shown in Algorithm 1. The drone population and the number of iterations are taken as input parameters. First, the population is initialized and the fitness value of each drone is evaluated to obtain the best solution until the maximum iteration is reached. Next, if the best solution is not changed in the first two iterations, the position of the drone in the swarm is updated and the number of particles in the exploration group is increased. The present invention considers both the exploration and development groups to update the previous fitness value. For the exploration group, the parameters r1, r2 and P are updated, and if P > 0.5 the solution mutates, otherwise it searches around the current solution.

[0131] For the development group, the parameters r2, r3, r4 and P are updated, and if P > 0.5 it will identify the best solution, otherwise it will search for the best solution.

[0132]

[0133]

[0134] If not enough, it is judged whether the distance between the vehicle and the intelligent roadside facility is less than the longest communication distance; if less, the vehicle will unload the task to the intelligent roadside facility, and the intelligent roadside facility will execute the task; if not less, the task will be forwarded to the intelligent roadside facility for execution through the UAV as a relay node.

[0135] Wherein, the task is forwarded to the intelligent roadside facility for execution through the UAV as a relay node, comprising:

[0136] S31, according to the calculation reuse technology, the task is preferentially assigned to the edge server with high request probability.

[0137] Optionally, the calculation reuse technology in S31 comprises:

[0138] Obtain the comment of each vehicle on each task, transmit the comment to the intelligent roadside device, obtain the comment sentiment value of each task according to the comment and the BERT model, normalize the comment sentiment value of all tasks, and obtain the request probability matrix of the task.

[0139] In a feasible implementation, in the prior art, the conditions of vehicle task content are not carefully considered. Usually, it is assumed that the probability of vehicle task content follows a uniform distribution (P∝1 / N) or Zipf distribution (P∝1 / i γ ). However, the above distribution cannot represent the real situation of these tasks. Therefore, in the present application, the sentiment analysis of vehicle task feedback is used as a guide for obtaining the task request probability matrix.

[0140] As Figure 3 shown, the feedback of the vehicle will be collected and transmitted to the edge cloud server first, and in the edge server, the feedback of the vehicle will be fully analyzed by a deep learning neural network. Then, the result of sentiment analysis will be used as the basis for obtaining the task request probability matrix. Since it is assumed that the operation of task content caching can be completed in the idle time of the local device, the calculation speed of these neural networks does not need to be considered. The selection of deep learning neural network only depends on the accuracy of its sentiment analysis result.

[0141] Commonly used text analysis methods include TextCNN, TextRNN, FastText, Bert, etc. TextCNN is suitable for short text analysis, while TextRNN can further consider the correlation between words far apart in a sentence. Compared with the first two methods, Fasttext can achieve faster computing speed at the expense of accuracy. Bert enables each word to interact with other words by using the self-attention mechanism in Transformer, thereby capturing richer context information and achieving higher accuracy, although it may take more computing time. The further experimental results based on the IMDB dataset are shown in Table 1, where the sample numbers of the training set, the validation set and the test set are 70000, 15000 and 15000 respectively.

[0142] Table 1

[0143]

[0144] As mentioned above, the present application only considers the accuracy of sentiment analysis, and does not consider the computing time caused by the idle time transmission strategy. Through simple theoretical analysis and strict experiments on the above-mentioned methods, Bert is the most suitable method for the present application to perform sentiment analysis. Therefore, Bert is selected to obtain the request probability matrix of the user to solve the request probability sub-problem. The specific steps are as follows:

[0145] Firstly, the sentiment of each vehicle for each task is analyzed by using the Bert model for the collected comments of each vehicle for each task, and the comment sentiment value of the corresponding task is obtained; the sentiment value is between 0 and 1, the closer to 1, the more interested the vehicle is in the corresponding task content, and vice versa.

[0146] Subsequently, the sentiment value of each vehicle is normalized for all task content, so that their sum is equal to 1. After normalization, the sentiment value of each user for each content is taken as the request probability, and the request probability matrix is obtained. According to the request probability matrix, the tasks of the vehicle are preferentially allocated to the edge server with high request probability. In this way, it can be ensured that the edge server first processes the most interesting tasks, thereby improving the efficiency of task processing.

[0147] S32, construct a Lyapunov optimization-based unmanned aerial vehicle energy constraint problem.

[0148] In a feasible implementation manner, since the unmanned aerial vehicle energy constraint problem P depends on the future, an online method is needed to make real-time decisions without predicting the future. Similar to, the Lyapunov-based optimization framework is a commonly used online algorithm design method, which has the advantages of simplicity and effectiveness. Therefore, the present application first converts the problem P into a slot-by-slot real-time optimization problem based on the Lyapunov optimization framework.

[0149] Optionally, the step S32 can include steps S321-S326 as follows:

[0150] S321, to meet the energy constraints of the UAV, a first virtual energy queue is defined based on Lyapunov optimization technique represents the calculated energy queue of time slot t, a second virtual energy queue is defined represents the propulsion energy queue of time slot t. The present application assumes that the queue is in the initial time slot, i.e. Therefore, the virtual energy queue can be updated as:

[0151]

[0152] wherein and represent the calculation and propulsion energy budget of each slot,

[0153] S322, according to the first virtual energy queue and the second virtual energy queue a Lyapunov function L(Q u (t)) is defined, the Lyapunov function represents a scalar measure of the queue backlog:

[0154]

[0155] wherein is the vector of the current queue backlog.

[0156] S323, according to the Lyapunov function, a conditional Lyapunov drift ΔL(Q u (t)) of time slot t is defined:

[0157]

[0158] S324, a penalty is added to the drift:

[0159] D(Q u (t))=ΔL(Q u (t))+HE{C S (t)|Q u (t)} (15)

[0160] wherein is the total cost of all vehicles at time slot t, and H is a parameter that weighs the total cost and the stability of the queue.

[0161] S325, an upper bound of the drift plus penalty is constructed.

[0162] Theorem 1: for all t and all possible queue backlogs Q u(t), the upper bound of the drift plus penalty is:

[0163]

[0164] where is a finite constant.

[0165] According to the Lyapunov optimization framework, the right side of the inequality is minimized. Therefore, the problem P that depends on future information is converted into a real-time optimization problem P' that can be solved only by relying on current information, as follows:

[0166]

[0167] In the formula, A t denotes the task offloading strategy of the t time slot, F t denotes the computing resource allocation of the t time slot, W t denotes the communication resource allocation of the t time slot, D' u =D u (t+1) denotes the position of the vehicle and the unmanned aerial vehicle at the time slot t+1, D u (t+1) denotes the position of the vehicle and the unmanned aerial vehicle at the time slot t, denotes the computing energy of the unmanned aerial vehicle at the t time slot, denotes the propulsion energy of the unmanned aerial vehicle at the t time slot, H denotes a parameter for balancing the total cost and queue stability, V denotes a set of all vehicles, and C(t) denotes a task cost.

[0168]

[0169] S33, the MADDPG algorithm is used to solve the unmanned aerial vehicle energy constraint problem, and a task offloading strategy is obtained. According to the task offloading strategy, the task is forwarded to the intelligent roadside facility for execution through the unmanned aerial vehicle as a relay node; the global criticism of the MADDPG algorithm adopts a double-delay deterministic policy gradient.

[0170] In a feasible implementation manner, in order to solve the optimization problem of cost minimization in formula (17), the optimization problem is converted into a reward maximization problem, and LyBMADRL_MEC is applied. The expressions of the state, behavior and reward function are as follows:

[0171] S331, state: the state S(t) of the MEC environment at time t, which contains the state set of the vehicle v, and is described as Constant values such as the number of sub-channels K, the number of processing units Ue on the server, the processing capacity fe and the storage capacity ze of the server are not included in the state information. The state S v (t) is composed of four parts: the task state Normalized channel gain state Power budget Local resource allocation budget In the equation, define:

[0172]

[0173] where, denotes the task status, t denotes the time slot, z v (t) denotes the data size of the task, c v (t) denotes the computing intensity of the task, τ v (t) denotes the priority of the task, denotes the normalized channel gain status, determined by g n = h n = h n / σ 2 , g n denotes the normalized channel gain status, h n denotes the channel gain, σ 2 denotes the background noise variance, denotes the power budget, denotes the maximum computing capacity of the vehicle, denotes the local resource allocation budget, denotes the maximum computing resource of the vehicle.

[0174] S332, Action: At the beginning of each time step, the UDs terminal devices use the client agent to decide their resource allocation. Then, the SDN controller collects the status and action information of the UDs, and performs one of the following three processes through the master agent: 1) for UDs that decide to handle locally, the server does not intervene. 2) If the number of UDs that suggest offloading is greater than the number of sub-channels, or the sum of the sizes of the sub-channels is greater than the storage capacity of the server, the server will combine to decide which requests are approved and which requests are rejected. 3) If the proposed requests are less than the constraints, the server accepts all requests. Finally, the sub-channels are allocated to the accepted UDs, and then the task offloading and processing process begins.

[0175] Existing deep reinforcement learning based task offloading algorithms include the number of sub-channels in their state and action spaces. However, from the perspective of a UD, the transmission capacity of a sub-channel is equal. If a channel can only be used by one UD at a time, it does not matter which channel a UD uses. Therefore, including sub-channels in the state and action spaces creates a dimension problem but does not play any important role. The invention excludes channel information from the state and action spaces and uses it as a constraint for combined action selection. Note that by removing the constraint that a channel must be used by one UD at a time, a channel can be reused by multiple UDs one after another. In this case, only the storage capacity of the server becomes a constraint in the combined action selection.

[0176] The operations of the client agent and the master agent are as follows.

[0177] Client behavior. At each time step, each client agent generates three actions, which are continuous value actions in the range [0, 1]. The action space can be represented as:

[0178]

[0179] where S v (t) is the state of vehicle v in equation (18), A c (t) represents the client agent behavior, a v (t) represents the decision of vehicle v to offload tasks, p c,v (t) represents the client action that decides the transmission power, f c,v (t) represents the action that decides the allocation of local computing resources, θ v represents the parameterized policy function of the client agent, S v (t) represents the state of vehicle v, t represents the time slot, and V represents the set of all vehicles.

[0180] The client agent decides the actions of p v and f v as follows:

[0181]

[0182] Further, the master action: for the client actions, the master takes the combination of the client state and actions and provides a binary output, which is used to decide which should be allocated locally and which should be accepted for processing by the MEC server:

[0183]

[0184] where A m (t) represents the master agent behavior, x m,v(t) represents the master agent’s decision at time slot t for the client agent, represents the master policy, S represents the set of states of all client agents, A represents the set of actions of all client agents, S c represents the set of states of client agents, A c represents the set of actions of client agents.

[0185] S333, Reward function: Since the design of the cooperative learning formula of the present application is based on the cost minimization problem in equation (17), the system reward function of the present application is equal to the negative value of the system cost function. Therefore, the system reward function can be expressed as:

[0186]

[0187] where r(S(t), A(t)) represents the reward function, S(t) represents the set of states at time slot t, A(t) represents the set of actions at time slot t, V represents the set of all vehicles, and P’ represents the UAV energy constraint problem based on Lyapunov optimization.

[0188] The power and resource allocation decision made by vehicle v in the current step affects its operational lifetime in the next time step by affecting the energy consumption. Therefore, DRL must consider both the immediate reward and the long-term reward using the Bellman equation, as shown in the equation:

[0189]

[0190] where is the learning rate, γ is the discount factor, is the critic’s policy, is the target critic’s policy, S = {S1, …, S N} and S’ = {S1’, …, S’ N} are the current state and next state combinations, A = {A1, …, A N} and A’ = {A’1, …, A’ N} are the current and next step action combinations of the client agent, S’ N and A’ N are the corresponding state and action of the agent.

[0191] Furthermore, the multi-agent deep reinforcement learning algorithm learns how to maximize global and local rewards. It comprises two main critiques: a global critique, shared by all agents and taking into account the states and actions of all agents to maximize the global reward; and a local critique, which receives the states and actions of a specific agent and evaluates its reward. In addition, this invention considers the impact of policy approximation error and value updates on the global critique of the madpg algorithm. To improve the madpg algorithm, this invention employs a double-delay deterministic policy gradient instead of the global critique.

[0192] Specifically, this invention considers a vehicle environment with V vehicles (agents), where the strategies of all agents are π = {π1, π2, ..., π}. V The strategy π of vehicle v v Q function and the double-delay double-Q function are respectively represented by θ k , ψ1 and ψ2 are parameterized. For each agent, the modified policy gradient can be written as:

[0193]

[0194] Where s = (s1, s2, ..., s V ), a = (a1, a2, ..., a V ) represents the overall state and action vectors, D is the replay buffer, and a v =π v (s v Agent v acts according to its own strategy π v The chosen action. Therefore, the double-delay double-Q function is updated as follows:

[0195]

[0196] in

[0197] The detailed MARL algorithm is shown in Table 3.

[0198]

[0199]

[0200] In a feasible implementation, in the model of the application, after the task is generated, the vehicle selects an execution strategy according to its computing capability. If the local computing capability of the vehicle is sufficient, the task will be processed locally; if the computing resource is insufficient, the vehicle will offload the task to the edge server served by the unmanned aerial vehicle or the intelligent roadside facility. At this time, the communication and computing resource conditions between the vehicle and the edge server need to be considered. When the computing resource of the unmanned aerial vehicle is sufficient, the unmanned aerial vehicle is preferred to execute the task, because the unmanned aerial vehicle can dynamically adjust the position to keep synchronization with the vehicle. When the computing resource of the unmanned aerial vehicle is insufficient or the edge server is far away, the unmanned aerial vehicle will act as a relay node and forward the task to the SRI.

[0201] Further, to ensure the efficiency of task processing, the edge server will schedule the tasks based on the priority of the tasks when the tasks are queued. High-priority tasks are processed first, and when the priorities are the same, the first-in-first-out queue mechanism is adopted. In addition, the edge server can use the computing reuse technology to store the processed computing results in advance, and use the LSH (Locality Sensitive Hashing) algorithm to realize approximate matching and accelerate the processing process of similar tasks.

[0202] Specifically, first, the vehicle generates a task, when the vehicle has sufficient computing memory, the task is directly calculated locally in the vehicle; when the vehicle memory is insufficient, the vehicle will offload the task to the edge server served by the unmanned aerial vehicle or the intelligent roadside facility, at this time, the communication between the vehicle and the edge server and the computing resource of the unmanned aerial vehicle and the edge server need to be considered, when the computing resource of the unmanned aerial vehicle is sufficient, the unmanned aerial vehicle is preferred, because the unmanned aerial vehicle can be synchronized with the movement of the vehicle in real time, when the computing resource of the unmanned aerial vehicle is insufficient, the intelligent roadside facility is considered; when the computing resource of the unmanned aerial vehicle is insufficient and the edge server is too far away, the vehicle will offload the task to the edge server served by the intelligent roadside facility through the unmanned aerial vehicle as a relay node.

[0203] The application proposes a collaborative optimization algorithm based on deep reinforcement learning, which is used to solve the task offloading and resource allocation problem. By introducing the computing reuse and joint caching technology, and combining the Lyapunov optimization method, the long-term energy constraint problem is further solved.

[0204] Further, in the simulated dynamic environment of Internet of Vehicles, the effectiveness of the proposed method is verified through experiments. The results show that the method has significant advantages in reducing the total cost of the system and improving the efficiency of task processing, and exhibits good robustness in complex environments.

[0205] In the embodiment of the application, efficient task offloading: through the cooperative work of the vehicle and the unmanned aerial vehicle, efficient offloading of the task is realized, and the utilization of computing resources is optimized.

[0206] Reducing latency and energy consumption: The introduction of computational reuse and caching techniques reduces waste of computational resources and optimizes task offloading strategies, computational resource and communication resource allocation through Lyapunov optimization method, significantly reducing task processing latency and system energy consumption.

[0207] Improving system throughput: Multi-agent deep reinforcement learning algorithm (MADDPG) improves task completion rate and system throughput in complex vehicle networking environment.

[0208] Enhancing robustness: Compared with traditional deep reinforcement learning algorithms, the algorithm of the present invention shows better robustness in high load and dynamic scenarios.

[0209] Flexibility and efficiency: The two-stage group bee optimization algorithm (DGBCO) shows excellent flexibility and efficiency in unmanned aerial vehicle trajectory planning.

[0210] Theoretical basis and technical support: It provides a new solution for unmanned aerial vehicle assisted vehicle networking multi-access edge computing (MEC), and provides theoretical basis and technical support for resource optimization scheduling in future vehicle networking application scenarios.

[0211] Figure 4 is a deep reinforcement learning-based unmanned aerial vehicle task offloading device block diagram according to an exemplary embodiment, which is used for deep reinforcement learning-based unmanned aerial vehicle task offloading method. Referring to Figure 4 , the device includes a construction module 310, a judgment module 320 and an output module 330. Among them:

[0212] The construction module 310 is used to construct an unmanned aerial vehicle MEC system model; wherein the unmanned aerial vehicle MEC system model includes a plurality of dynamically moving vehicles and a plurality of edge servers, and the plurality of edge servers includes a plurality of unmanned aerial vehicles and a plurality of intelligent roadside facilities.

[0213] The judgment module 320 is used to determine whether the generated task is processed by the vehicle or offloaded to the edge server for processing by the edge server according to the computing ability of the vehicle after any vehicle in the plurality of dynamically moving vehicles generates a task.

[0214] The output module 330 is used to construct a Gaussian-Markov mobility model of the vehicle when the task is processed by the edge server, obtain the vehicle position according to the Gaussian-Markov mobility model of the vehicle, and obtain the distance between the vehicle and the intelligent roadside facility according to the vehicle position.

[0215] Determine whether the computing resources of the unmanned aerial vehicle are sufficient to complete the task.

[0216] If sufficient, the vehicle offloads the task to the UAV, the UAV performs the task, and the obtained UAV swarm size and preset iteration number are input into the DGBCO model, and a path planning result of the UAV is output.

[0217] If insufficient, it is judged whether the distance between the vehicle and the intelligent roadside facility is less than the longest communication distance; if less, the vehicle offloads the task to the intelligent roadside facility, and the intelligent roadside facility performs the task; if not less, the task is forwarded to the intelligent roadside facility for execution through the UAV as a relay node.

[0218] The task is forwarded to the intelligent roadside facility for execution through the UAV as a relay node, comprising:

[0219] S31, according to the calculation reuse technology, preferentially assigning the task to the edge server with high request probability.

[0220] S32, constructing a UAV energy constraint problem based on Lyapunov optimization.

[0221] S33, using the MADDPG algorithm to solve the UAV energy constraint problem, obtaining a task offloading strategy, and forwarding the task to the intelligent roadside facility for execution through the UAV as a relay node according to the task offloading strategy; the global critic of the MADDPG algorithm adopts a double-delay deterministic policy gradient.

[0222] In the embodiment of the application, efficient task offloading: through the cooperative work of the vehicle and the UAV, efficient offloading of the task is realized, and the utilization of the computing resources is optimized.

[0223] Reducing delay and energy consumption: introducing the calculation reuse and cache technology, reducing the waste of computing resources, and optimizing the task offloading strategy, the allocation of computing resources and communication resources through the Lyapunov optimization method, significantly reducing the task processing delay and system energy consumption.

[0224] Improving system throughput: the multi-agent deep reinforcement learning algorithm (MADDPG) improves the task completion rate and system throughput in the complex vehicle networking environment.

[0225] Enhancing robustness: compared with the traditional deep reinforcement learning algorithm, the algorithm of the application shows better robustness in high load and dynamic scenarios.

[0226] Flexibility and efficiency: the two-stage swarm bee optimization algorithm (DGBCO) shows excellent flexibility and efficiency in UAV trajectory planning.

[0227] Theoretical basis and technical support: a new solution is provided for unmanned aerial vehicle assisted multi-access edge computing (MEC) of Internet of Vehicles, and theoretical basis and technical support are provided for resource optimization scheduling in future Internet of Vehicles application scenarios.

[0228] Figure 5 is a structural schematic diagram of a UAV task offloading device provided by an embodiment of the application, as shown in Figure 5 The UAV task offloading device can include the UAV task offloading device based on deep reinforcement learning shown in Figure 4 Optionally, the UAV task offloading device 410 can include a first processor 2001.

[0229] Optionally, the UAV task offloading device 410 can further include a memory 2002 and a transceiver 2003.

[0230] The first processor 2001, the memory 2002 and the transceiver 2003 can be connected through a communication bus.

[0231] The above embodiments can be realized by software, hardware (such as circuitry), firmware or any combination thereof, in whole or in part. When realized by software, the above embodiments can be realized in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the flow or function described in the embodiments of the application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like containing one or more available medium sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD) or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0232] It should be understood that the term "and / or" in this document is merely used to describe associated objects, and can represent three conditions: A and / or B, which can mean that A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects, but can also represent an "and / or" relationship, which can be understood in combination with the context.

[0233] In this document, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of the items, including a single item or any combination of multiple items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be singular or plural.

[0234] It should be understood that the order of the above processes in various embodiments of the present application does not mean the order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0235] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0236] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0237] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the above-described device embodiments are only schematic, and the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0238] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0239] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0240] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0241] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for unmanned aerial vehicle task offloading based on deep reinforcement learning, characterized in that, The method comprises: S1, constructing a UAV MEC system model; wherein the UAV MEC system model comprises a plurality of dynamically moving vehicles and a plurality of edge servers, and the plurality of edge servers comprise a plurality of UAVs and a plurality of intelligent roadside facilities; S2, any vehicle in the plurality of dynamically moving vehicles judges whether a generated task is processed by the vehicle or unloaded to an edge server for processing according to the computing capability of the vehicle after the task is generated; S3, when the task is processed by the edge server, a Gaussian-Markov mobility model of the vehicle is constructed, a vehicle position is obtained according to the Gaussian-Markov mobility model of the vehicle, and a distance between the vehicle and the intelligent roadside facility is obtained according to the vehicle position; judging whether the computing resources of the UAV are sufficient to complete the task; if sufficient, the vehicle unloads the task to the UAV, the UAV executes the task, and the obtained UAV group size and the preset iteration number are input into a DGBCO model to output a path planning result of the UAV; if insufficient, judging whether the distance between the vehicle and the intelligent roadside facility is less than the longest communication distance; if less, the vehicle unloads the task to the intelligent roadside facility, and the intelligent roadside facility executes the task; if not less, the task is forwarded to the intelligent roadside facility for execution through the UAV as a relay node; wherein the task is forwarded to the intelligent roadside facility for execution through the UAV as a relay node, comprising: S31, according to a computing reuse technology, the task is preferentially assigned to an edge server with a high request probability; S32, constructing a UAV energy constraint problem based on Lyapunov optimization; S33, using a MADDPG algorithm to solve the UAV energy constraint problem to obtain a task offloading strategy, and forwarding the task to the intelligent roadside facility for execution through the UAV as a relay node according to the task offloading strategy; the global criticism of the MADDPG algorithm uses a double-delay deterministic policy gradient; the Gaussian-Markov mobility model of the vehicle is constructed, the vehicle position is obtained according to the Gaussian-Markov mobility model of the vehicle, and the distance between the vehicle and the intelligent roadside facility is obtained according to the vehicle position, comprising: constructing a speed update of the vehicle at time slot t+1, as shown in the following formula (1): where v v (t+1) denotes the speed of the vehicle at time slot t+1, a denotes the memory level, denotes the speed vector at time slot t, denotes the speed of the vehicle in the x-axis direction, denotes the speed of the vehicle in the y-axis direction, denotes the asymptotic mean of the speed, W v (t) denotes an uncorrelated random Gaussian process; constructing a vehicle position update at time slot t+1 according to the speed update, as shown in the following formula (2): P v [t+1] = P v [t] + v v [t] (2) where P v [t+1] represents the vehicle position at time slot t+1, P v [t] = (x v [t], y v [t]) represents the position of vehicle v at time slot t, x v [t] represents the position of the vehicle in the x-axis direction, y v [t] represents the position of the vehicle in the y-axis direction; constructing the distance between the vehicle and the intelligent roadside facility according to the vehicle position update, as shown in the following formula (3): where d v,r (t) denotes the distance of the vehicle from the smart roadside facility at time slot t, x r denotes the position of the smart roadside facility in the x-axis direction, y r denotes the position of the smart roadside facility in the y-axis direction. 2.The method of claim 1, wherein, the computing reuse technology in S31 comprises: obtaining a comment of each vehicle on each task, transmitting the comment to the intelligent roadside device, obtaining a comment sentiment value of each task according to the comment and a BERT model, normalizing the comment sentiment values of all tasks to obtain a request probability matrix of the task. 3.The method of claim 1, wherein, the UAV energy constraint problem based on Lyapunov optimization in S32 comprises: define a first virtual energy queue a calculated energy queue representing time slot t, define a second virtual energy queue a pushed energy queue representing time slot t; According to the first virtual energy queue and a second virtual energy queue defining a Lyapunov function L(Q u (t)) representing a scalar measure of queue backlog; According to the Lyapunov function, a conditional Lyapunov drift AL(Q u (t)) is defined for a time slot t - a Lyapunov function L(Q(t)) is defined for a time slot t adding a penalty to the drift, and constructing an upper bound of the drift plus the penalty; According to the upper bound, an unmanned aerial vehicle energy constraint problem based on Lyapunov optimization is obtained, as shown in the following formula (4): where A t denotes the task offloading strategy of t slot, F t denotes the computing resource allocation of t slot, W t denotes the communication resource allocation of t slot, D' u = D u (t+1) denotes the position of vehicle and UAV in time slot t+1, D u (t+1) denotes the position of vehicle and UAV in time slot t, denotes the computing energy of UAV in t slot, denotes the propulsion energy of UAV in t slot, H denotes the parameter of trade-off between total cost and queue stability, V denotes the set of all vehicles v, C(t) denotes the task cost. 4.The method of claim 1, wherein, The vehicle state in the MADDPG algorithm in S33 is as shown in the following formula (5): wherein, denotes the task state, t denotes the time slot, z v (t) denotes the data size of the task, c v (t) denotes the computation intensity of the task, τ v (t) denotes the priority of the task, denotes the normalized channel gain state, denotes the power delivery budget, denotes the vehicle maximum computation capability, denotes the local resource allocation budget, denotes the vehicle maximum computation resource. 5.The method of claim 1, wherein, The behavior in the MADDPG algorithm in S33 includes a client agent behavior and a master agent behavior; The client agent behavior is as shown in the following formula (6): where A c (t) represents the client agent behavior, a v (t) represents the decision of vehicle v to offload a task, p c,v (t) represents the client action that decides the transmission power, f c,v (t) represents the action that decides the local computing resource allocation, θ v represents the parameterized policy function of the client agent, S v (t) represents the state of vehicle v, t represents the time slot, V represents the set of all vehicles; The master agent behavior is as shown in the following formula (7): where A m (t) represents the master agent behavior, x m,v (t) represents the master agent's decision for the client agent at time slot t, represents the master policy, S represents the set of states of all client agents, A represents the set of actions of all client agents, S c represents the set of states of the client agent, A c represents the set of actions of the client agent. 6.The method of claim 1, wherein, The reward function in the MADDPG algorithm in S33 is as shown in the following formula (8): In the formula, r(S(t), A(t)) represents a reward function, S(t) represents a state set at a t time slot, A(t) represents an action set at the t time slot, V represents a set of all vehicles, and P' represents an unmanned aerial vehicle energy constraint problem based on Lyapunov optimization. 7.A deep reinforcement learning based UAV task offloading apparatus for implementing the deep reinforcement learning based UAV task offloading method according to any one of claims 1-6. The device comprises: A construction module configured to construct an unmanned aerial vehicle MEC system model, wherein the unmanned aerial vehicle MEC system model comprises a plurality of dynamically moving vehicles and a plurality of edge servers, and the plurality of edge servers comprise a plurality of unmanned aerial vehicles and a plurality of intelligent roadside facilities; A judgment module configured to, after a task is generated, determine, according to the computing capability of a vehicle, whether the generated task is processed by the vehicle or unloaded to an edge server for processing by the edge server, from among the plurality of dynamically moving vehicles; An output module configured to, when the task is processed by the edge server, construct a Gaussian-Markov mobility model of the vehicle, obtain a vehicle position according to the Gaussian-Markov mobility model of the vehicle, and obtain a distance between the vehicle and an intelligent roadside facility according to the vehicle position; Determine whether the computing resources of the unmanned aerial vehicle are sufficient to complete the task; If sufficient, the vehicle unloads the task to the unmanned aerial vehicle, the unmanned aerial vehicle executes the task, and the obtained swarm size of the unmanned aerial vehicle and a preset iteration number are input to a DGBCO model, and a path planning result of the unmanned aerial vehicle is output; If insufficient, it is determined whether the distance between the vehicle and the intelligent roadside facility is less than a longest communication distance; if less, the vehicle unloads the task to the intelligent roadside facility, and the intelligent roadside facility executes the task; if not less, the task is forwarded to the intelligent roadside facility for execution through the unmanned aerial vehicle as a relay node; The task is forwarded to the intelligent roadside facility for execution through the unmanned aerial vehicle as a relay node, comprising: S31, preferentially assigning the task to an edge server with a high request probability according to a computing reuse technology; S32, constructing an unmanned aerial vehicle energy constraint problem based on Lyapunov optimization; S33, solving the unmanned aerial vehicle energy constraint problem by using a MADDPG algorithm to obtain a task offloading strategy, and forwarding the task to the intelligent roadside facility for execution through the unmanned aerial vehicle as a relay node according to the task offloading strategy; a global criticism of the MADDPG algorithm adopts a double-delay deterministic policy gradient.

8. An unmanned aerial vehicle task offloading device, characterized by, The unmanned aerial vehicle task offloading device comprises: A processor; A memory having computer readable instructions stored thereon for causing a processor to implement the method of any one of claims 1 to 6.

9. A computer readable storage medium, characterized in that, A computer readable storage medium having program code stored thereon, the program code executable by a processor for causing the processor to implement the method of any one of claims 1 to 6.

Images