Resource allocation and cooperative unloading method based on federal deep reinforcement learning under multi-unmanned aerial vehicle assisted internet of vehicles

Abstract

The invention provides a resource allocation and cooperative unloading method based on federated deep reinforcement learning under a multi-unmanned aerial vehicle assisted Internet of Vehicles, and the method comprises the steps: firstly, decoupling a transmission power and calculation resource allocation sub-problem from an original problem, and introducing a convex optimization theory to solve the sub-problem; under a given unloading strategy, obtaining a near-optimal transmission power and computing resource allocation strategy by solving a KKT condition; then, designing a joint resource allocation and cooperative unloading strategy based on federal deep reinforcement learning (FDRL); specifically, each DRL agent making a decision for the UAV aggregates model parameters in a central server while interacting with an environment; the aggregated global model is distributed to the local model, and each agent can sense the overall state of the environment, so that a near-optimal cooperative unloading strategy is realized; by applying the technical scheme, a near-optimal resource allocation strategy can be obtained, and UAV cooperative unloading can be adaptively carried out to better balance the load of the UAV.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a resource allocation and collaborative unloading method based on federated deep reinforcement learning in a multi-drone-assisted vehicle network. Background Art

[0002] As a specific application of the Internet of Things (IoT) in the transportation field, the Internet of Vehicles (IoV) can effectively improve system performance and service quality in key areas such as real-time information sharing, autonomous driving control, and personalized services, thereby helping to build an emerging intelligent transportation system (ITS). In recent years, the number of vehicle terminals (VTs) has exploded, spawning a large amount of service data. At the same time, VTs' demand for real-time performance is also increasing. By sinking resources to the edge of the network, Mobile Edge Computing (MEC) can effectively reduce data transmission latency, thereby improving IoV system performance and user experience. With the help of MEC, VTs can alleviate load and power consumption pressure by requesting computing offload services deployed on road side units (RSUs).

[0003] Restricted by fixed network architecture and limited coverage, RSUs cannot continuously provide high-quality services to mobile VTs. As an aerial mobile device, Unmanned Aerial Vehicles (UAVs) have been applied in a variety of application scenarios. Thanks to the flexible mobility and continuous communication capabilities, deploying edge servers on UAVs can build a MEC aerial network and promote the development of the next generation of ITS. Considering the urgent need of ITS for real-time intelligent decision-making, distributed edge intelligence (DEI) is regarded as a promising solution by deploying computing power at the edge of the network to achieve intelligent data processing and real-time decision-making. In this distributed mode, offload requests in IoV can be distributed to multiple UAVs to achieve load balancing.

[0004] Some existing works use UAV to assist IoV to perform computational offloading to build ITS, and realize functions such as traffic flow monitoring, intelligent signal control, and real-time information release. However, these works ignore the resource limitations and energy consumption of UAVs. When in a long-term high-load working environment, these methods will lead to large load differences and inefficient resource utilization between UAVs. In order to improve load balancing and resource efficiency, some works introduce DEI in IoV to improve the overall performance and real-time decision-making of the system, but they ignore the effective allocation of edge node resources, resulting in excessive response delays. Although some works consider the joint optimization of computational offloading and resource allocation, most of them use heuristic or static methods, which cannot adapt to the dynamic distributed IoV environment and are prone to excessive computational overhead.

[0005] In a distributed scenario, data from multiple nodes are stored in a decentralized manner. The emerging Deep Reinforcement Learning (DRL) has attracted much attention due to its outstanding performance in dynamic time slot decision-making tasks. DRL usually requires a large amount of data for training, but this data is often distributed in different places. Considering privacy protection and data security, it is difficult to collect and process it centrally. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a resource allocation and collaborative unloading method based on federated deep reinforcement learning in a multi-UAV assisted vehicle network, which can achieve a near-optimal resource allocation strategy and adaptively perform UAV collaborative unloading to better balance its load.

[0007] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a resource allocation and collaborative offloading method based on federated deep reinforcement learning in a multi-UAV-assisted Internet of Vehicles. First, the transmission power and computing resource allocation sub-problems are decoupled from the original problem and the convex optimization theory is introduced to solve them; under a given offloading strategy, the KKT condition is solved to obtain a near-optimal transmission power and computing resource allocation strategy; then, a joint resource allocation and collaborative offloading strategy based on federated deep reinforcement learning (FDRL) is designed; specifically, each DRL agent that makes decisions for the UAV aggregates the model parameters on a central server while interacting with the environment; by distributing the aggregated global model to the local model, each agent can perceive the overall state of the environment, thereby achieving a near-optimal collaborative offloading strategy.

[0008] In a preferred embodiment, a resource allocation and collaborative offloading system for UAV-assisted IoV is included, and the resource allocation and collaborative offloading system for UAV-assisted IoV includes a system administrator, UAVs and VTs; UAVs are equipped with MEC servers to provide computing offloading services for VTs, and the system administrator is responsible for collecting network information, aggregating local models and controlling the operation of the system; the set of VTs in the system is represented as V = {v1,v2,...,v n}, the UAVs set is represented by UAV = {uav1,uav2,...,uav m}; When v i Enter UAV j When the coverage range of v i At most one computation offloading request is generated and a decision is made to execute it locally or offload it to the UAV connected to it; UAVs in a high-load state forward the received offloading task to other UAVs in a low-load state with sufficient energy.

[0009] In a preferred embodiment, at time slot t, the i The task is defined as a four-tuple <D i [t],S i [t],d i [t],c i [t]>, where D i [t] represents the amount of input data for the task, S i [t] represents the computational complexity of the task, d i [t] represents v in this time slot i With UAV j The distance between i [t] represents v i Connect to UAV; when v i Connect to UAV j When c i [t] = j; v i The unloading decision at time slot t is denoted as α i [t]∈{0,1}; when v i When no tasks are generated or when tasks are executed locally, α i [t] = 0; when v i When offloading the computation to the UAV, α i [t]=1; Specifically, the two calculation modes are as follows;

[0010] (1) Local computing mode

[0011] When the task is executed locally, the latency is:

[0012]

[0013] Among them, f i Indicates v i computing power;

[0014] (2) UAV computing model

[0015] When v i The task is offloaded to uav j When the input data is transmitted through the uplink first, the link between VTs and UAVs adopts orthogonal frequency division multiple access technology. According to Shannon's theorem, v i With UAV j The data transfer rate between is:

[0016]

[0017] Among them, B v Indicates v i With UAV j The channel bandwidth between v Defined as P i Represents v i The transmission power, ρ0 represents the channel gain per unit distance, and N0 represents the noise power spectral density;

[0018] Therefore, v i Offload tasks to uav j The transmission delay is:

[0019]

[0020] Next, we need to i The task specifies the target UAV and allocates computing resources to perform the task; when the UAV j When under low load, uav j Direct execution v i task, and assign it to the task f ji [t]; therefore, from v i The mission in UAV j The computational delay on is:

[0021]

[0022] Among them, f ji [t]∈(0,f max ], f max For UAV j The maximum available computing resources;

[0023] When UAV jWhen under high load, uav j The task will be forwarded to other UAVs for collaborative offloading; specifically, the uav j The cooperative offloading decision at time slot t is defined as β ijk [t]∈{0,1}; when uav j v i The task is forwarded to uav k When ijk [t]=1, otherwise, β ijk [t] = 0; uav j Forward v i The transmission power of the task process is P ji [t]∈(0,P max ], where P max represents the maximum transmission power achieved by the UAV; therefore, v i Mission from UAV j Forward to uav k The data transfer rate during the process is:

[0024]

[0025] Among them, B uav represents the channel bandwidth between UAVs, γ uav Defined as d jk [t] represents uav in time slot t j and UAV k The distance between i The mission from UAV j Forward to uav k The transmission delay is:

[0026]

[0027] When v i Task forwarding to uav k After that, the task is in UAV k The computational delay on is:

[0028]

[0029] Among them, f ki [t] indicates uav k The computing resources allocated to the task;

[0030] Execute v i The task delay is:

[0031]

[0032] Considering the above two calculation modes, processing vi The total delay of the task is:

[0033]

[0034] Therefore, in time slot t, the total delay for processing all tasks is:

[0035]

[0036] In a preferred embodiment, the uav j Execute v i The energy consumption of the task is:

[0037]

[0038] Where, κ represents the effective capacitance coefficient; uav j v i Task forwarding to uav k The energy consumption is:

[0039]

[0040] In addition, the energy consumption of each UAV to maintain the hovering state in each time slot is e;

[0041] When v i Task sent to uav j And in uav j When executed above, there is α i [t]=1,β ijk [t] = 0; therefore, uav j The energy consumption of executing all tasks in time slot t is:

[0042]

[0043] When v i The mission was uav j Forward to uav k When executing above, there is α i [t]=1,β ijk [t] = 1, therefore, uav j The energy consumption of transmitting all tasks in time slot t is:

[0044]

[0045] The total energy consumption of all UAVs in time slot t is:

[0046]

[0047] The initial energy of each UAV is b max, energy will be consumed continuously in the process of providing computing services; when the energy is lower than the threshold b min When the UAVs are in a state of emergency, they will temporarily stop providing computing offload services and return to their home base for battery recovery.

[0048] The optimization goal is to minimize the task processing delay; the offloading decision is recorded as a set α = {α i [t]|1≤i≤n,1≤t≤T}, the collaborative offloading decision is recorded as the set β={β ijk [t]|1≤i≤n,1≤j,k≤m,1≤t≤T}, the UAV computing resource allocation decision is recorded as the set in Indicates that in time slot t, uav j The set of task numbers calculated, the UAV transmission power allocation decision is recorded as the set in Indicates that in time slot t from uav j The set of task numbers forwarded; therefore, the optimization problem is formally defined as:

[0049]

[0050] Among them, b j [t] indicates uav j The remaining power at time slot t, b min represents the amount of power that UAVs need to reserve for returning; C1 and C2 represent the value range constraints for offloading decisions; C3 represents that a task can be forwarded at most once between UAVs; C4 and C5 represent the computing resource constraints that UAVs allocate to VTs tasks; C6 represents the transmission power constraint for collaboration between UAVs; C7 represents the minimum amount of power that UAVs need to reserve when providing computing offloading services. min Used for returning to the destination and continuing the battery life; in problem P1, α and β are integer variables, and F and P are continuous variables.

[0051] In a preferred embodiment, for resource allocation under a given offloading strategy, problem P1 is firstly analyzed and transformed to prove that it contains two convex programming problems, and finally the transmission power and computing resource allocation are solved by the KKT condition;

[0052] At time slot t, the transmission delay is decoupled from the computation delay, and P1 is transformed into:

[0053]

[0054] By decoupling F and P, P2 is decomposed into two sub-problems:

[0055]

[0056] Among them, P3 is the computational resource allocation sub-problem, and P4 is the collaborative transmission power allocation sub-problem;

[0057] For P3, when the computation offloading strategy α and the cooperative offloading strategy β are given, v i A task generated in time slot t is ultimately calculated on only one node, so the calculation delay is One of them; and for v i Local computing latency Since the computing resources allocated to tasks by VTs are fixed, Assuming it as a constant, after removing it from P3, only the computational delay on the UAV remains, and the problem can be reformulated as:

[0058]

[0059] in, Indicates that in time slot t, uav j A collection of task numbers to be calculated;

[0060] Substitute formula (4) into P5 and convert constraints C8 and C9 equivalently. Since the KKT condition is only applicable to equality constraints and non-strict inequality constraints, for the strict inequality constraints in P5 Introduce an arbitrarily small positive number ε to transform the constraint into a non-strict inequality constraint Therefore, P5 is equivalent to:

[0061]

[0062] Among them, C8 in P5 is converted into a quadratic constraint C10; through simple deduction, it can be obtained that C10 is a convex set, and constraint C11 is a linear constraint condition, which is also a convex set;

[0063] Next, the Hessian matrix of P6 is calculated by the second-order partial derivative to be a positive definite matrix; therefore, P6 is a convex programming problem and is solved using the Lagrange multiplier method; first, construct the Lagrangian function:

[0064]

[0065] in, and λ j is the Lagrange multiplier;

[0066] Next, the KKT condition is expressed as:

[0067]

[0068] According to convex optimization theory, if the original problem is a convex programming problem, then the KKT condition must have a solution, and the value solved by the KKT condition is the optimal value of the convex programming, so the near-optimal computing resource allocation of P3 is obtained. Similarly, it can also be proved that P4 is a convex programming problem.

[0069] In a preferred embodiment, a joint resource allocation and collaborative offloading strategy based on FDRL is proposed, in which the proposed multi-UAV-assisted IoV system is regarded as an environment, and multiple agents interact with the environment simultaneously and select corresponding offloading actions; each agent maintains an actor network for decision-making, and the system administrator maintains a critic network for calculating the advantage function, which can measure the advantage of the decision made by the agent in the current state compared with the average strategy; after receiving the reward signal fed back by the environment, the system administrator calculates the advantage function and distributes the value to each agent; then, the actor network of each agent is updated in a distributed manner to obtain the current local optimal model; after a certain number of update rounds, the actor network model parameters of each agent are sent to the system administrator for aggregation to obtain the current global optimal model; finally, the current global optimal model is distributed to each agent, so that each agent can better perceive the state of other agents, thereby improving the collaborative ability between UAVs.

[0070] In a preferred embodiment, the state space, action space and reward function are defined as follows:

[0071] State space: The state space includes task attributes, the computing power of VTs, and i , VTs transmission power P i , the remaining power of the UAV b[t]; therefore, at time slot t, the state observed by agent j (1≤j≤m) is expressed as:

[0072]

[0073] in, and Represent the vectors consisting of the data volume and computational complexity of all tasks respectively;

[0074] Action space: The action space contains the offloading decision α[t] of VTs and the collaborative offloading decision β[t] of UAVs. α[t] and β[t] are merged, and the merged offloading decision is recorded as η[t]∈{0,1,2,...,m}. Therefore, at time slot t, the action of agent j is expressed as:

[0075]

[0076] in, represents the offloading decision made by agent j for all tasks; η i[t]=0 means v i No task is generated or the agent decides to execute the task locally; η i [t]=k, k∈{1,2,...,m} indicates that the agent decides to offload or forward the task to the uav k Execute on;

[0077] Reward function: The optimization goal of P1 is to minimize the task processing delay; therefore, at time slot t, the reward function is expressed as:

[0078] r[t]=-T v_tot [t], (26)

[0079] All agents share the same reward and collaborate to maximize the reward; the system administrator obtains the task processing delay of each agent and adds up the negative of all delays to obtain the common reward for all agents.

[0080] In a preferred embodiment, each UAV adopts an actor-critic DRL framework, in which the critic network estimates the advantage function of the state-action pair using multi-step temporal difference, and the actor network updates the strategy according to the advantage function; first, the value network maintained by the system administrator and the strategy network of each agent are initialized; in each round of iteration, each agent obtains the initial state of the environment and initializes the round end mark; in each round, each agent selects a suitable unloading action according to its strategy network; at the same time, the task to be performed on each UAV is determined according to the unloading action; then, the agent executes the unloading action and resource allocation strategy and updates the reward, state and round end mark; when the power of all UAVs reaches b min The round ends when When , done = True; then, collect the interaction data of each agent and update its state; then, the system administrator first uses the value network to calculate the advantage function, and then sends the value to each agent; the advantage function is defined as:

[0081]

[0082] where γ is the discount factor, A hyperparameter λ∈[0,1] is introduced to control the step size of the recursive calculation in the advantage estimation; when λ=0, formula (27) is the advantage obtained by one step difference; when λ=1, formula (27) is the complete average of the advantages obtained by each step difference;

[0083] Next, the loss functions of the value network and the policy network of each agent are calculated, and the network parameters are iteratively updated. Specifically, policy clipping is used to limit the magnitude of each policy update. The loss function of the policy network of the jth agent is defined as:

[0084]

[0085] In the formula, ω θ' represents the importance weight, ε represents the cutoff range, which is defined as:

[0086]

[0087] in, represents the old policy from the previous iteration, It represents a new strategy that is continuously updated in iterations. The old strategy interacts with the environment to collect data, and then uses this data to train the new strategy. According to the principle of importance sampling, the data sampled by the old strategy can be used multiple times, and gradient ascent can be performed multiple times to update the strategy.

[0088] Then, when the training round reaches the predetermined aggregation frequency, the local model parameters of each agent are sent to the system administrator for aggregation to obtain the global model parameters; the aggregation method adopts the average federation aggregation method:

[0089]

[0090] Average federation obtains the global model parameters by calculating the average value of the local model parameters of all distributed nodes; finally, the global model parameters are distributed to each UAV, and each agent updates the parameters.

[0091] Compared with the prior art, the present invention has the following beneficial effects: the RACOMU proposed in the present invention achieves a more superior system delay and load balancing degree while maintaining high decision-making efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 It is a schematic diagram of a resource allocation and collaborative offloading system for UAV-assisted IoV according to a preferred embodiment of the present invention;

[0093] Figure 2 It is a schematic diagram of multi-UAV resource allocation and cooperative offloading based on FDRL in a preferred embodiment of the present invention;

[0094] Figure 3 A schematic diagram of average time slot rewards of different methods according to a preferred embodiment of the present invention;

[0095] Figure 4 A schematic diagram of average time slot delay of different methods according to a preferred embodiment of the present invention;

[0096] Figure 5 The preferred embodiments of the present invention are different from max Below is a diagram of the average time slot rewards for different methods;

[0097] Figure 6 A schematic diagram of the load balancing degree and total load of UAVs in different methods of a preferred embodiment of the present invention;

[0098] Figure 7 A schematic diagram of average time slot rewards for different federation aggregation frequencies according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0099] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0100] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0101] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0102] The present invention proposes a novel Resource Allocation and Collaborative Offloading method in Multi-UAV-assisted IoV (RACOMU) method. The RACOMU method can achieve a near-optimal resource allocation strategy and adaptively perform UAV collaborative offloading to better balance their loads.

[0103] System Model and Problem Definition

[0104] like Figure 1-7 As shown in Figure 1, the proposed resource allocation and collaborative offloading system for UAV-assisted IoV consists of a system administrator, UAVs, and VTs. UAVs are equipped with MEC servers to provide computing offloading services for VTs. The system administrator is responsible for collecting network information (e.g., channel conditions between UAVs and VTs, task traffic on UAVs, etc.), aggregating local models, and controlling the operation of the system. The set of VTs in the system is represented as V = {v1,v2,...,v n}, the UAVs set is represented by UAV = {uav1,uav2,...,uav m}. When v i Enter UAVj When the UAV is within the coverage range, a computation offloading request can be initiated to the UAV through the wireless link. i At most, one computation offloading request is generated and the decision is made to execute it locally or offload it to the connected UAV. Due to the mobility of VTs, the number of VTs that enter the coverage of different UAVs varies. In order to balance the load between UAVs, UAVs in a high-load state can forward the received offloading tasks to other UAVs in a low-load state with sufficient energy.

[0105] Computation and Communication Model

[0106] At time slot t, from v i The task is defined as a four-tuple <D i [t],S i [t],d i [t],c i [t]>, where D i [t] represents the amount of input data for the task, S i [t] represents the computational complexity of the task, d i [t] represents v in this time slot i With UAV j The distance between i [t] represents v i Connect to UAV. When v i Connect to UAV j When c i [t] = j. i The unloading decision at time slot t is denoted as α i [t]∈{0,1}. When v i When no tasks are generated or when tasks are executed locally, α i [t] = 0; when v i When offloading the computation to the UAV, α i [t] = 1. Specifically, the two calculation modes are as follows.

[0107] (1) Local computing mode

[0108] When the task is executed locally, the latency is:

[0109]

[0110] Among them, f i Indicates v i computing power.

[0111] (2) UAV computing model

[0112] When v iThe task is offloaded to uav j When the VTs and UAVs are connected, their input data is first transmitted through the uplink. The link between VTs and UAVs adopts orthogonal frequency division multiple access technology. According to Shannon's theorem, v i With UAV j The data transfer rate between is:

[0113]

[0114] Among them, B v Indicates v i With UAV j The channel bandwidth between v Defined as P i Represents v i The transmission power of ρ0 is the channel gain per unit distance, and N0 is the noise power spectral density.

[0115] Therefore, v i Offload tasks to uav j The transmission delay is:

[0116]

[0117] Next, we need to i The task specifies the target UAV and allocates computing resources to perform the task. j When under low load, uav j Direct execution v i task, and assign it to the task f ji [t] computing resources. Therefore, from v i The mission in UAV j The computational delay on is:

[0118]

[0119] Among them, f ji [t]∈(0,f max ], f max For UAV j The maximum available computing resources.

[0120] When UAV j When under high load, uav j The task will be forwarded to other UAVs for collaborative offloading. j The cooperative offloading decision at time slot t is defined as β ijk [t]∈{0,1}. When uav j v i The task is forwarded to uav k Whenijk [t]=1, otherwise, β ijk [t] = 0. uav j Forward v i The transmission power of the task process is P ji [t]∈(0,P max ], where P max represents the maximum transmission power achievable by the UAV. Therefore, v i Mission from UAV j Forward to uav k The data transfer rate during the process is:

[0121]

[0122] Among them, B uav represents the channel bandwidth between UAVs, γ uav Defined as d jk [t] represents uav in time slot t j and UAV k The distance between i The mission from UAV j Forward to uav k The transmission delay is:

[0123]

[0124] When v i Task forwarding to uav k After that, the task is in UAV k The computational delay on is:

[0125]

[0126] Among them, f ki [t] indicates uav k The computing resources allocated to the task.

[0127] Compared with the input of the task, the amount of data of the calculation result is usually small, so the delay of returning the result can be ignored. i The task delay is:

[0128]

[0129] Considering the above two calculation modes, processing v i The total delay of the task is:

[0130]

[0131] Therefore, in time slot t, the total delay for processing all tasks is:

[0132]

[0133] Energy consumption model

[0134] UAVs consume energy in the process of providing computing offloading services. Since UAVs have limited energy, the energy consumption of UAVs in each time slot needs to be considered to determine the return time. The energy consumption generated in the process of tasks being offloaded to UAVs for processing includes the energy consumption of task execution, the energy consumption of task forwarding, and the energy consumption of UAVs maintaining a hovering state.

[0135] uav j Execute v i The energy consumption of the task is:

[0136]

[0137] Where κ represents the effective capacitance coefficient. j v i Task forwarding to uav k The energy consumption is:

[0138]

[0139] In addition, the energy consumption of each UAV to maintain the hovering state in each time slot is e.

[0140] When v i Task sent to uav j And in uav j When executed above, there is α i [t]=1,β ijk [t] = 0. Therefore, uav j The energy consumption of executing all tasks in time slot t is:

[0141]

[0142] When v i The mission was uav j Forward to uav k When executing above, there is α i [t]=1,β ijk [t] = 1, therefore, uav j The energy consumption of transmitting all tasks in time slot t is:

[0143]

[0144] The total energy consumption of all UAVs in time slot t is:

[0145]

[0146] The initial energy of each UAV is b max , energy will be consumed continuously in the process of providing computing services. When the energy is lower than the threshold b min When the UAVs are in a state of emergency, they will temporarily stop providing computing offload services and return to their home base for battery recovery.

[0147] Based on the proposed system model, the optimization goal of the present invention is to minimize the task processing delay. The offloading decision is recorded as a set α = {α i [t]|1≤i≤n,1≤t≤T}, the collaborative offloading decision is recorded as the set β={β ijk [t]|1≤i≤n,1≤j,k≤m,1≤t≤T}, where Indicates that in time slot t, uav j The set of task numbers for calculation, and the UAV computing resource allocation decision is recorded as the set The UAV transmission power allocation decision is denoted as the set in Indicates that in time slot t from uav j The set of task numbers forwarded. Therefore, the optimization problem is formally defined as:

[0148]

[0149] Among them, b j [t] indicates uav j The remaining power at time slot t, b min Indicates the amount of power that UAVs need to reserve for returning. C1 and C2 indicate the value range constraints for offloading decisions. C3 indicates that a task can be forwarded at most once between UAVs. C4 and C5 indicate the computing resource constraints that UAVs allocate to VTs tasks. C6 indicates the transmission power constraints for collaboration between UAVs. C7 indicates the minimum amount of power that UAVs need to reserve when providing computing offloading services. min Used for returning to the destination and continuing the power supply.

[0150] In problem P1, α and β are integer variables, and F and P are continuous variables. Therefore, problem P1 is a mixed integer programming (MIP) problem, and it is difficult to find the optimal solution in polynomial time.

[0151] Proposed RACOMU method

[0152] In order to effectively solve P1, this paper proposes a novel resource allocation and collaborative offloading method in Multi-UAV-assisted IoV (RACOMU) method. First, the resource allocation is decoupled from the original problem and the convex optimization theory is introduced to solve it. Under a given offloading strategy, the KKT condition is solved to obtain a near-optimal resource allocation strategy. Then, a new FDRL-based joint resource allocation and collaborative offloading strategy is designed. The agents are distributedly trained. After a certain number of training rounds, the model parameters of each agent will be aggregated to protect data privacy.

[0153] Resource allocation under a given offloading strategy

[0154] In this section, the present invention first analyzes and transforms problem P1, proves that it contains two convex programming problems, and finally solves the transmission power and computing resource allocation through the KKT condition. Since the transmission power and computing resource allocation of each UAV are independent of each other, the original problem can be decoupled.

[0155] At time slot t, by decoupling the transmission delay from the computation delay, P1 can be transformed into:

[0156]

[0157] Furthermore, by decoupling F and P, P2 can be decomposed into two sub-problems:

[0158]

[0159] Among them, P3 is the computational resource allocation sub-problem, and P4 is the collaborative transmission power allocation sub-problem.

[0160] For P3, when the computation offloading strategy α and the cooperative offloading strategy β are given, v i A task generated in time slot t is ultimately calculated on only one node, so the calculation delay is One of them. And for v i The local computation delay T i v_c [t], since the computing resources allocated to tasks by VTs are fixed, T i v_c [t] can be regarded as a constant. After removing it from P3, only the computational delay on the UAV remains, and the problem can be reformulated as:

[0161]

[0162] in, Indicates that in time slot t, uav j A collection of task sequence numbers to be calculated.

[0163] Substitute formula (4) into P5 and convert constraints C8 and C9 to be equivalent. Since the KKT condition is only applicable to equality constraints and non-strict inequality constraints, for the strict inequality constraints in P5 Introduce an arbitrarily small positive number ε to transform the constraint into a non-strict inequality constraint Therefore, P5 can be equivalent to:

[0164]

[0165] Among them, C8 in P5 is converted into a quadratic constraint C10. Through simple deduction, it can be obtained that C10 is a convex set, and constraint C11 is a linear constraint condition, which is also a convex set.

[0166] Then, through the second-order partial derivative calculation, it can be obtained that the Hessian matrix of P6 is a positive definite matrix. Therefore, P6 is a convex programming problem and can be solved using the Lagrange multiplier method. First, construct the Lagrange function:

[0167]

[0168] Among them, μ ji1 and λ j is the Lagrange multiplier.

[0169] Next, the KKT condition is expressed as:

[0170]

[0171] According to convex optimization theory, if the original problem is a convex programming problem, then the KKT condition must have a solution, and the value solved by the KKT condition is the optimal value of the convex programming, so the near-optimal computing resource allocation of P3 can be obtained. Similarly, it can also be proved that P4 is a convex programming problem and the near-optimal transmission power allocation can be obtained by solving its KKT condition.

[0172] Joint resource allocation and cooperative offloading based on FDRL

[0173] In order to make real-time offloading decisions in a dynamic IoV environment, the present invention introduces DRL to continuously interact with it and adaptively learn the optimal collaborative offloading strategy. However, with the expansion of the IoV network and the increase in the number of VTs, centralized DRL is difficult to cope with the huge state and action space, resulting in dimensionality explosion and unacceptable training complexity. FL, as a distributed algorithm, provides a new idea for solving the above problems. FL can distribute training nodes and can collaboratively learn and improve models while protecting data privacy. A large number of offloading requests from VTs can be processed separately by multiple agents, which reduces the training complexity and speeds up decision-making to better meet the high real-time requirements of the IoV system. In addition, FL's model aggregation enables multiple agents to learn from a shared global perspective, and accelerates their learning process while improving model accuracy by sharing experience during training.

[0174] In view of the advantages of DRL and FL, this paper proposes a joint resource allocation and cooperative offloading strategy based on FDRL. Figure 2 As shown in the figure, the proposed multi-UAV-assisted IoV system is regarded as an environment, and multiple agents interact with the environment simultaneously and select corresponding unloading actions. Each agent maintains an actor network for decision-making, and the system administrator maintains a critic network for calculating the advantage function, which measures the advantage of the decision made by the agent in the current state compared with the average strategy. After receiving the reward signal fed back by the environment, the system administrator calculates the advantage function and distributes the value to each agent. Then, the actor network of each agent is updated in a distributed manner to obtain the current local optimal model. After a certain number of update rounds, the actor network model parameters of each agent are sent to the system administrator for aggregation to obtain the current global optimal model. Finally, the current global optimal model is distributed to each agent, so that each agent can better perceive the state of other agents, thereby improving the collaboration ability between UAVs. Accordingly, the state space, action space and reward function are defined as follows.

[0175] State space: The state space includes task attributes, the computing power of VTs, and i , VTs transmission power P i , the remaining power of the UAV b[t]. Therefore, at time slot t, the state observed by agent j (1≤j≤m) is expressed as:

[0176]

[0177] in, and Represent the vectors consisting of the data volume and computational complexity of all tasks respectively.

[0178] Action space: The action space contains the offloading decision α[t] of VTs and the collaborative offloading decision β[t] of UAVs. In order to improve the training efficiency, α[t] and β[t] are merged, and the merged offloading decision is recorded as η[t]∈{0,1,2,...,m}. Therefore, at time slot t, the action of agent j is expressed as:

[0179]

[0180] in, represents the offloading decision made by agent j for all tasks. i [t]=0 means v i No task is generated or the agent decides to perform the task locally. i [t]=k, k∈{1,2,...,m} indicates that the agent decides to offload or forward the task to the uav k Execute on.

[0181] Reward function: The optimization goal of P1 is to minimize the task processing delay. Therefore, at time slot t, the reward function is expressed as:

[0182] r[t]=-T v_tot [t], (26)

[0183] In order to further enhance the overall perception of the environment by each agent, all agents share the same reward and work together to maximize the reward. The system administrator obtains the task processing delay of each agent and adds up the negative of all delays to get the common reward for all agents.

[0184] The main steps of the proposed FDRL-based multi-UAV joint resource allocation and collaborative offloading are shown in Algorithm 1.

[0185]

[0186]

[0187] In Algorithm 1, each UAV adopts the actor-critic DRL framework, in which the critic network uses multi-step temporal difference to estimate the advantage function of the state-action pair, and the actor network updates the strategy according to the advantage function. First, initialize the value network maintained by the system administrator and the policy network of each agent (line 1). In each round of iteration, each agent obtains the initial state of the environment (line 3) and initializes the round end flag (line 4). In each round, each agent selects a suitable unloading action according to its policy network (line 6). At the same time, the tasks to be performed on each UAV are determined according to the unloading action, and the resource allocation decision is obtained using the convex optimization derivation in Section 4.1 (line 7). Next, the agent executes the unloading action and resource allocation strategy and updates the reward, state, and round end flag (line 8). When the battery power of all UAVs reaches b min The round ends when , done = True. Subsequently, the interaction data of each agent is collected and its state is updated (line 9). Next, the system administrator first calculates the advantage function using the value network and then sends the value to each agent (line 11). The advantage function is defined as:

[0188]

[0189] where γ is the discount factor, A hyperparameter λ∈[0,1] is introduced to control the step size of the recursive calculation in the advantage estimation. When λ=0, Equation (27) is the advantage obtained by one step difference; when λ=1, Equation (27) is the complete average of the advantages obtained by each step difference.

[0190] Next, the loss function of the value network and each agent’s policy network is calculated, and the network parameters are iteratively updated (lines 13-15). Specifically, policy clipping is used to limit the magnitude of each policy update to avoid instability that may be caused by excessive updates. The loss function of the policy network of the jth agent is defined as:

[0191]

[0192] In the formula, ω θ' represents the importance weight, ε represents the cutoff range, which is defined as:

[0193]

[0194] in, represents the old policy from the previous iteration, Represents a new strategy that is continuously updated in iterations. Unlike other DRL algorithms that discard previously sampled data after updating network parameters, the proposed method sets the old and new strategies and then transforms them into a training process based on different strategies. Specifically, the proposed method collects data through the interaction between the old strategy and the environment, and then uses these data to train the new strategy. According to the principle of importance sampling, the data sampled by the old strategy can be used multiple times, and gradient ascent can be performed multiple times to update the strategy.

[0195] Then, when the training round reaches the predetermined aggregation frequency, the local model parameters of each agent are sent to the system administrator for aggregation to obtain the global model parameters (lines 17-18). The aggregation method uses average federation aggregation:

[0196]

[0197] Average federation obtains the global model parameters by calculating the average of the local model parameters of all distributed nodes. Finally, the global model parameters are distributed to each UAV, and each agent updates the parameters (line 19).

[0198] Method evaluation

[0199] The proposed system model and RACOMU method are implemented based on Python, and the simulation experiment is run on a workstation equipped with Intel Xeon Silver 4208 CPU and NVIDIA Getforce GTX 3090 GPU. In order to simulate the dynamics of the IoV environment more realistically, the number of offloaded tasks and task attributes in the system are constantly changing between different time slots. The number of time slots in a round is determined by the time scale of all UAVs from full power to the minimum power threshold. The main parameter settings of the system model setting and the RACOMU method in the experiment are shown in Table 1.

[0200] Table 1 Parameter settings

[0201]

[0202]

[0203] In order to verify the superiority of the proposed RACOMU, the present invention compares RACOMU with the following methods:

[0204] (1) Local: VTs tasks are all executed locally.

[0205] (2) MEC: All VTs tasks are offloaded to UAVs for execution, and random collaboration is adopted between UAVs. The transmission power and computing resources adopt fixed value and average allocation strategies respectively.

[0206] (3) DDPG: DDPG algorithm is used to perform offloading decisions, computational resource allocation, and transmission power allocation.

[0207] (4) PPO: The PPO algorithm is used to execute offloading decisions, and the transmission power and computing resources adopt fixed value and average allocation strategies respectively.

[0208] (5) SA: The simulated annealing algorithm is used to make unloading decisions, and the transmission power and computing resources are allocated using fixed values ​​and average allocation strategies respectively.

[0209] First, the present invention compares the average time slot rewards of different methods during the convergence process. Figure 3 As shown in the figure, due to limited computing power, when all tasks are processed locally, the Local method achieves the lowest reward. When all tasks are offloaded to UAVs for execution and average resource allocation is adopted, the computing resources allocated to each task will be greatly reduced, resulting in a lower reward for the MEC method. Compared with the SA method, the RACOMU, DDPG and PPO methods achieved higher rewards. The SA method continuously iterates to find the optimal solution in a time slot, but because it cannot perceive environmental changes, this method is prone to fall into a local optimal solution. The PPO method requires a large number of training rounds to explore the environment in the early stage, but its rewards in the later stage increase rapidly. The DDPG method converges quickly, and the resource allocation strategy and offloading strategy learned by the agent further improve resource efficiency, so its rewards exceed the PPO method. Compared with the DDPG and PPO methods, the proposed RACOMU method can more effectively improve the average time slot reward and show the best performance among all methods. This is because the RACOMU method uses convex optimization to obtain the optimal resource allocation strategy, which greatly improves resource efficiency. In addition, the distributed execution method also makes the state and action space that each agent faces relatively small, which reduces the training pressure and shortens the early exploration process. In addition, the aggregation of model parameters also enables each agent to fully perceive the overall state of the environment, thereby better optimizing the agent's offloading decision.

[0210] Next, the present invention compares the time required for different methods to execute decisions. As shown in Table 2, since a large number of iterations are required to find the optimal solution, the SA method is very time-consuming to execute decisions, and its execution time is about 100 times that of other methods. In contrast, the RACOMU, DDPG and PPO methods can make decisions quickly based on the current environmental state, which can better meet the real-time performance of the IoV system. Compared with the PPO method, the RACOMU method executes decisions faster. This is because RACOMU adopts a distributed decision-making method, and the state space and action space that each agent needs to process are smaller than PPO, thus achieving a faster decision-making speed. Since the actor network directly outputs the strategy without the need to output the probability distribution of the strategy, DDPG further reduces the action space, so the decision-making speed is similar to that of the RACOMU method.

[0211] Table 2 Execution time of different methods

[0212]

[0213] Next, the present invention compares the delay distribution of different methods in 50 time slots. Figure 4 As shown in the figure, the median of the delay of each time slot under different methods is close to the center of the box, which shows that the delay distribution of different methods in each time slot is relatively symmetrical, and also reflects that these methods can adapt to the dynamically changing IoV environment better, thus showing good performance. Compared with the SA method, the RACOMU, DDPG and PPO methods have smaller differences in delays in each time slot and are more stable when making decisions. Compared with other methods, the median line of the box plot of the RACOMU method is slightly lower, and its average delay of 50 time slots is also the best among all methods, which means that the overall delay of the RACOMU method will be lower than that of other methods.

[0214] Next, the present invention compares different f max The average time slot rewards of different methods are as follows. Figure 5 As shown, the rewards of all methods increase with f max This is because the more computing resources the UAV has, the more computing resources are allocated to each task, which can reduce the computing delay of the task. max The increase of f is most obvious in the MEC method. This is because when the number of computing resources increases to a sufficient amount, for the same task computing requirements, f max is no longer a bottleneck limiting system performance, so the MEC method can allocate sufficient computing resources to each task. Compared with other methods, the proposed RACOMU method has maxThe highest reward can be obtained under all conditions, which shows that the proposed RACOMU method has superior performance in resource allocation and computation offloading. In addition, the present invention also compares the RACOMU method under f max The impact on the total number of time slots in a round. max As f increases, the total number of time slots implemented by the RACOMU method gradually decreases. max When it increases, RACOMU will offload more tasks to UAV and allocate more computing resources, resulting in an increase in UAV energy consumption and a reduction in service time.

[0215] Next, the present invention compares the average values ​​of the load balancing degrees of different methods under 10 time slots with the total load of UAVs (i.e., the total number of tasks of all UAVs), where the results are taken from the average values ​​of 10 time slots. Specifically, the load balancing degree (LBD) is defined as:

[0216]

[0217] Among them, ld j represents the load of the jth UAV (i.e., the number of tasks unloaded to and executed by the UAV), and ld represents the average load of all UAVs. The smaller LBD means the more balanced the load among UAVs.

[0218] like Figure 6 As shown in the figure, the total load of the MEC method is the highest. This is because this method uploads all tasks to UAVs for calculation, resulting in an overload on UAVs. At the same time, the load balancing degree of the MEC method is also the lowest. This is because this method adopts a random collaboration strategy and cannot effectively balance the load between UAVs. The load balancing degree of the DDPG method is slightly higher than that of the proposed RACOMUT method. However, the total load of UAVs in the DDPG method is too low, which will result in more idle computing resources that cannot be utilized. Compared with other methods, the proposed RACOMUT method can make better offloading decisions to make full use of the computing resources of UAVs, and can efficiently collaborate with multiple UAVs to achieve better load balancing effects.

[0219] Finally, the present invention compares the average time slot rewards of the proposed RACOMU method under different federation aggregation frequencies. Figure 7As shown in the figure, in the early stage of training, the local model rewards obtained by using 60-round and 80-round aggregation frequencies are slightly higher than those of other aggregation frequencies. In the later stage, the local model rewards are higher when the aggregation frequencies of 20-round and 40-round are used. This may be because when the local model accuracy is poor in the early stage of training, frequent model aggregation may make it difficult for the local model to converge and fail to improve the model accuracy. In the later stage of training, as the local model continues to improve, more frequent federated aggregation frequencies can enable the local model to absorb the experience of other models more quickly, thereby achieving higher rewards.

[0220] The process or method of using the product

[0221] (1) In each time slot, each client collects the task information generated by all VTs within the UAV coverage area, including the number of tasks and the computational complexity of the tasks.

[0222] (2) The agent on each client generates task offloading and forwarding decisions based on the task information generated by all VTs, the computing power of the corresponding VT, the transmission power of the corresponding VT, and the remaining power of the UAV.

[0223] (3) Based on the offloading and forwarding decisions, the task forwarding set and task execution set of the current UAV are obtained. Then, the convex optimization theory is used to obtain the optimal transmission power and computing resource allocation decision to forward or execute the task.

[0224] (4) During the computation offloading process, the agent on each client continuously records the state of each time slot, the actions taken, the rewards obtained, and the new state entered, and continuously optimizes its own performance based on the above information.

[0225] (5) After a certain number of updates, each client sends the local model parameters to the system administrator for aggregation to obtain the global model parameters, and then the system administrator distributes the global model parameters to each client.

Claims

1. A resource allocation and collaborative offloading method based on federated deep reinforcement learning in multi-UAV-assisted vehicle networking, characterized in that: Firstly, the transmission power and computing resource allocation sub-problems are decoupled from the original problem and convex optimization theory is introduced to solve them. Under a given offloading strategy, the KKT condition is solved to obtain a near-optimal transmission power and computing resource allocation strategy. Then, a joint resource allocation and collaborative offloading strategy based on federated deep reinforcement learning (FDRL) is designed. Specifically, each DRL agent that makes decisions for the UAV aggregates the model parameters on the central server while interacting with the environment. By distributing the aggregated global model to the local model, each agent can perceive the overall state of the environment, thereby achieving a near-optimal collaborative offloading strategy.

2. The resource allocation and collaborative offloading method based on federated deep reinforcement learning in multi-UAV-assisted vehicle networking according to claim 1 is characterized in that: The system comprises a resource allocation and collaborative offloading system for UAV-assisted IoV, wherein the resource allocation and collaborative offloading system for UAV-assisted IoV comprises a system administrator, UAVs and VTs; UAVs are equipped with MEC servers to provide computing offloading services for VTs, and the system administrator is responsible for collecting network information, aggregating local models and controlling the operation of the system; the set of VTs in the system is represented as V = {v1,v2,...,v n }, the UAVs set is represented by UAV = {uav1,uav2,...,uav m }; When v i Enter UAV j When the coverage range of v i At most one computation offloading request is generated and a decision is made to execute it locally or offload it to the UAV connected to it; UAVs in a high-load state forward the received offloading task to other UAVs in a low-load state with sufficient energy.

3. The resource allocation and collaborative unloading method based on federated deep reinforcement learning in multi-UAV-assisted vehicle networking according to claim 2 is characterized in that: At time slot t, from v i The task is defined as a four-tuple <D i [t],S i [t],d i [t],c i [t]>, where D i [t] represents the amount of input data for the task, S i [t] represents the computational complexity of the task, d i [t] represents v in this time slot i With UAV j The distance between i [t] represents v i Connect to UAV; when v i Connect to UAV j When c i [t] = j; v i The unloading decision at time slot t is denoted as α i [t]∈{0,1}; When v i When no tasks are generated or when tasks are executed locally, α i [t] = 0; when v i When offloading the computation to the UAV, α i [t]=1; Specifically, the two calculation modes are as follows; (1) Local computing mode When the task is executed locally, the latency is: Among them, f i Indicates v i computing power; (2) UAV computing model When v i The task is offloaded to uav j When the input data is transmitted through the uplink first, the link between VTs and UAVs adopts orthogonal frequency division multiple access technology. According to Shannon's theorem, v i With UAV j The data transfer rate between is: Among them, B v Indicates v i With UAV j The channel bandwidth between v Defined as P i Represents v i The transmission power, ρ0 represents the channel gain per unit distance, and N0 represents the noise power spectral density; Therefore, v i Offload tasks to uav j The transmission delay is: Next, we need to i The task specifies the target UAV and allocates computing resources to perform the task; when the UAV j When under low load, uav j Direct execution v i task, and assign it to the task f ji [t]; therefore, from v i The mission in UAV j The computational delay on is: Among them, f ji [t]∈(0,f max ], f max For UAV j The maximum available computing resources; When UAV j When under high load, uav j The task will be forwarded to other UAVs for collaborative offloading; specifically, the uav j The cooperative offloading decision at time slot t is defined as β ijk [t]∈{0,1}; when uav j v i The task is forwarded to uav k When ijk [t]=1, otherwise, β ijk [t] = 0; uav j Forward v i The transmission power of the task process is P ji [t]∈(0,P max ], where P max represents the maximum transmission power achieved by the UAV; therefore, v i Mission from UAV j Forward to uav k The data transfer rate during the process is: Among them, B uav represents the channel bandwidth between UAVs, γ uav Defined as d jk [t] indicates uav in time slot t j and UAV k The distance between i The mission from UAV j Forward to uav k The transmission delay is: When v i Task forwarding to uav k After that, the task is in UAV k The computational delay on is: Among them, f ki [t] indicates uav k The computing resources allocated to the task; Execute v i The task delay is: Considering the above two calculation modes, processing v i The total delay of the task is: T i total [t]=(1-α i [t])T i v_c [t]+α i [t](T i v_t [t]+T i uav_total [t]) (9) Therefore, in time slot t, the total delay for processing all tasks is:

4. The resource allocation and collaborative unloading method based on federated deep reinforcement learning in a multi-UAV-assisted vehicle network according to claim 3 is characterized in that: uav j Execute v i The energy consumption of the task is: Where, κ represents the effective capacitance coefficient; uav j v i Task forwarding to uav k The energy consumption is: In addition, the energy consumption of each UAV to maintain the hovering state in each time slot is e; When v i Task sent to uav j And in uav j When executed above, there is α i [t]=1,β ijk [t] = 0; therefore, uav j The energy consumption of executing all tasks in time slot t is: When v i The mission was uav j Forward to uav k When executing above, there is α i [t]=1,β ijk [t] = 1, therefore, uav j The energy consumption of transmitting all tasks in time slot t is: The total energy consumption of all UAVs in time slot t is: The initial energy of each UAV is b max , energy will be consumed continuously in the process of providing computing services; when the energy is lower than the threshold b min When the UAVs are in a state of emergency, they will temporarily stop providing computing offload services and return to their home base for battery recovery. The optimization goal is to minimize the task processing delay; the offloading decision is recorded as a set α = {α i [t]|1≤i≤n,1≤t≤T}, the collaborative offloading decision is recorded as the set β={β ijk [t]|1≤i≤n,1≤j,k≤m,1≤t≤T}, the UAV computing resource allocation decision is recorded as the set in Indicates that in time slot t, uav j The set of task numbers calculated, the UAV transmission power allocation decision is recorded as the set in Indicates that in time slot t from uav j The set of task numbers forwarded; therefore, the optimization problem is formally defined as: Among them, b j [t] indicates uav j The remaining power at time slot t, b min represents the amount of power that UAVs need to reserve for returning; C1 and C2 represent the value range constraints for offloading decisions; C3 represents that a task can be forwarded at most once between UAVs; C4 and C5 represent the computing resource constraints that UAVs allocate to VTs tasks; C6 represents the transmission power constraint for collaboration between UAVs; C7 represents the minimum amount of power that UAVs need to reserve when providing computing offloading services. min Used for returning to the destination and continuing the battery life; in problem P1, α and β are integer variables, and F and P are continuous variables.

5. The resource allocation and collaborative unloading method based on federated deep reinforcement learning in multi-UAV-assisted vehicle networking according to claim 4 is characterized in that: For resource allocation under a given offloading strategy, we first analyze and transform problem P1 to prove that it contains two convex programming problems. Finally, we solve the transmission power and computing resource allocation through the KKT condition. At time slot t, the transmission delay is decoupled from the computation delay, and P1 is transformed into: By decoupling F and P, P2 is decomposed into two sub-problems: Among them, P3 is the computational resource allocation sub-problem, and P4 is the collaborative transmission power allocation sub-problem; For P3, when the computation offloading strategy α and the cooperative offloading strategy β are given, v i A task generated at time slot t is ultimately calculated on only one node, so the calculation delay is T i v_c [t], One of them; and for v i The local computation delay T i v_c [t], since the computing resources allocated to tasks by VTs are fixed, T i v_c [t] is considered as a constant. After removing it from P3, only the computational delay on the UAV remains, and the problem can be reformulated as: in, Indicates that in time slot t, uav j A collection of task numbers to be calculated; Substitute formula (4) into P5 and convert constraints C8 and C9 equivalently. Since the KKT condition is only applicable to equality constraints and non-strict inequality constraints, for the strict inequality constraint f in P5 ji1 [t]>0, an arbitrarily small positive number ε is introduced to transform the constraint into a non-strict inequality constraint f ji1 [t]≥ε; therefore, P5 is equivalent to: Among them, C8 in P5 is converted into a quadratic constraint C10; through simple deduction, it can be obtained that C10 is a convex set, and constraint C11 is a linear constraint condition, which is also a convex set; Next, the Hessian matrix of P6 is calculated by the second-order partial derivative to be a positive definite matrix; therefore, P6 is a convex programming problem and is solved using the Lagrange multiplier method; first, construct the Lagrangian function: Among them, μ ji1 and λ j is the Lagrange multiplier; Next, the KKT condition is expressed as: According to convex optimization theory, if the original problem is a convex programming problem, the KKT condition must have a solution, and the value solved by the KKT condition is the optimal value of the convex programming, so a near-optimal computing resource allocation of P3 is obtained.

6. The resource allocation and collaborative offloading method based on federated deep reinforcement learning in multi-UAV-assisted vehicle networking according to claim 1 is characterized in that: A joint resource allocation and collaborative offloading strategy based on FDRL is proposed. The proposed multi-UAV-assisted IoV system is regarded as an environment. Multiple agents interact with the environment simultaneously and select corresponding offloading actions. Each agent maintains an actor network for decision-making, and the system administrator maintains a critic network for calculating the advantage function, which can measure the advantage of the decision made by the agent in the current state compared with the average strategy. After receiving the reward signal fed back by the environment, the system administrator calculates the advantage function and distributes the value to each agent. Then, the actor network of each agent is updated in a distributed manner to obtain the current local optimal model. After a certain number of update rounds, the actor network model parameters of each agent are sent to the system administrator for aggregation to obtain the current global optimal model. Finally, the current global optimal model is distributed to each agent, so that each agent can better perceive the state of other agents, thereby improving the collaborative ability between UAVs.

7. The resource allocation and collaborative unloading method based on federated deep reinforcement learning in a multi-UAV-assisted vehicle network according to claim 6 is characterized in that: The state space, action space, and reward function are defined as follows: State space: The state space includes task attributes, the computing power of VTs, and i , VTs transmission power P i , the remaining power of the UAV b[t]; therefore, at time slot t, the state observed by agent j (1≤j≤m) is expressed as: in, and Represent the vectors consisting of the data volume and computational complexity of all tasks respectively; Action space: The action space contains the offloading decision α[t] of VTs and the collaborative offloading decision β[t] of UAVs. α[t] and β[t] are merged, and the merged offloading decision is recorded as η[t]∈{0,1,2,...,m}. Therefore, at time slot t, the action of agent j is expressed as: in, represents the offloading decision made by agent j for all tasks; η i [t]=0 means v i No task is generated or the agent decides to execute the task locally; η i [t]=k, k∈{1,2,...,m} indicates that the agent decides to offload or forward the task to the uav k Execute on; Reward function: The optimization goal of P1 is to minimize the task processing delay; therefore, at time slot t, the reward function is expressed as: r[t]=-T v_tot [t], (26) All agents share the same reward and collaborate to maximize the reward; the system administrator obtains the task processing delay of each agent and adds up the negative of all delays to obtain the common reward for all agents.

8. The resource allocation and collaborative unloading method based on federated deep reinforcement learning in a multi-UAV-assisted vehicle network according to claim 7 is characterized in that: Each UAV adopts the actor-critic DRL framework, in which the critic network uses multi-step temporal difference to estimate the advantage function of the state-action pair, and the actor network updates the strategy according to the advantage function. First, the value network maintained by the system administrator and the strategy network of each agent are initialized. In each round of iteration, each agent obtains the initial state of the environment and initializes the round end mark. In each round, each agent selects the appropriate unloading action according to its strategy network. At the same time, the task to be performed on each UAV is determined according to the unloading action. Then, the agent executes the unloading action and resource allocation strategy and updates the reward, state and round end mark. When the battery power of all UAVs reaches b min The round ends when When , done = True; then, collect the interaction data of each agent and update its state; then, the system administrator first uses the value network to calculate the advantage function, and then sends the value to each agent; the advantage function is defined as: where γ is the discount factor, δ t+l =-V θ (s[t+l])+r[t+l]+γV θ (s[t+l+1]), and introduce a hyperparameter λ∈[0,1] to control the step size of the recursive calculation in the advantage estimation; when λ=0, formula (27) is the advantage obtained by one step difference; when λ=1, formula (27) is the complete average of the advantages obtained by each step difference; Next, the loss functions of the value network and the policy network of each agent are calculated, and the network parameters are iteratively updated. Specifically, policy clipping is used to limit the magnitude of each policy update. The loss function of the policy network of the jth agent is defined as: In the formula, ω θ ' represents the importance weight, ε represents the cutoff range, ω θ ' is defined as: in, represents the old policy from the previous iteration, It represents a new strategy that is continuously updated in iterations. The old strategy interacts with the environment to collect data, and then uses this data to train the new strategy. According to the principle of importance sampling, the data sampled by the old strategy can be used multiple times, and gradient ascent can be performed multiple times to update the strategy. Then, when the training round reaches the predetermined aggregation frequency, the local model parameters of each agent are sent to the system administrator for aggregation to obtain the global model parameters; the aggregation method adopts the average federation aggregation method: Average federation obtains the global model parameters by calculating the average value of the local model parameters of all distributed nodes; finally, the global model parameters are distributed to each UAV, and each agent updates the parameters.

Images