A joint dynamic decision-making method for cell association, trajectory planning and offloading scheduling
Through the joint dynamic decision-making method of cell association, trajectory planning and offload scheduling for multi-access edge computing, the problem of resource matching and communication computing requirements in multi-cell multi-user mobile edge computing network is solved, and dynamic resource optimization and efficient unmanned Machine track planning and user task unloading schedule.
Patent Information
- Application Number
- CN202310462583.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-04-26
AI Technical Summary
In multi-cell multi-user mobile edge computing networks, it is difficult for the agent to obtain interactive information efficiently. The dimensional disaster problem caused by massive devices limits the scale optimization and resource matching capabilities of the air-ground collaborative mobile edge computing network.
Adopting a joint dynamic decision-making method for cell association, trajectory planning and offload scheduling for multi-access edge computing, optimizes drone trajectory planning and user tasks through heterogeneous joint decision-making of hierarchical decision-making and average field theory, combining policy networks and value networks. Uninstall scheduling to achieve dynamic resource matching and optimization of communication computing requirements.
Effectively solve the communication/computing requirements for time/space changes in the highly dynamic air-to-ground integrated network, improve resource utilization and computing capabilities, and realize resource matching and offloading scheduling in network scenarios where large-scale drones and base stations are collaborated. .
Smart Images

Figure CN116669069B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of wireless communication technology, and in particular to a joint dynamic decision-making method for cell association, trajectory planning and offloading scheduling for multi-access edge computing. Background Art
[0002] In the face of the scenario of widespread deployment of IoT devices in the future, resources are more scarce, and the problem of massive data interaction and real-time computing processing needs to be solved urgently. Driven by the vision of industrial IoT and 6G communication, mobile computing has undergone a paradigm shift from centralized mobile cloud computing to mobile edge computing (MEC). However, the location of mobile edge computing servers is usually fixed, which is not flexible enough when facing the changes in real-time resource requirements of users in different regions, thus limiting the ability of mobile edge computing to a certain extent. In addition, the channel between ground base stations and user devices is affected by small-scale fading, and the transmission rate is limited. Using drones as edge computing nodes to assist user task offloading in mobile edge computing can not only alleviate the high latency problem caused by insufficient computing power of user devices, but also provide more dynamic network deployment and reduce resource unfairness between devices. This air-ground collaborative mobile edge computing network model can achieve low-cost and dynamic network deployment and improve the utilization of computing resources by reusing a large number of existing air-ground mobile computing resources. However, considering the large-scale optimization problem caused by the further increase in the scale of multi-cell and multi-user mobile edge computing networks in the future, the intelligent agent cannot efficiently obtain interactive information, and massive devices cause dimensionality disasters. This has become a difficulty in the theoretical and practical deployment of air-ground collaborative mobile edge computing networks.
[0003] The invention patent with publication number CN115134776A and titled “A method for unloading computing of drones for mobile edge computing” makes unloading decisions for terminal devices, drones, and base stations through a partial unloading method, and uses dual queues to prioritize tasks on drones according to deadlines, thereby reducing the total task execution time and improving the task completion rate. The invention patent with publication number CN112867065A and titled “A method for air-ground collaborative edge computing and its system” achieves the purpose of allocating system resources including communication resources and computing resources, as well as designing joint drone flight trajectories by balancing the relationship between the system’s computational workload and energy consumption overhead. None of the above patents take into account the large-scale optimization problems caused by the further increase in the scale of multi-cell, multi-user mobile edge computing networks in the future. Summary of the invention
[0004] The present invention discloses a joint dynamic decision method for cell association, trajectory planning and offloading scheduling for multi-access edge computing, which is oriented to the multi-access edge computing scenario assisted by drones, so as to solve the problem of matching the communication / computing requirements with multiple resources in time / space changes in highly dynamic air-ground integrated networks. The present invention can be used for base station cell association decision, drone online trajectory planning and user task offloading scheduling in a network scenario model where large-scale multi-level drones and base stations collaborate to provide offloading for ground user equipment.
[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:
[0006] A method for joint dynamic decision-making of cell association, trajectory planning and unloading scheduling for multi-access edge computing, the method comprising the following steps:
[0007] S1, divide the duration T into N equal time slots, and the state of each decision proxy base station, each decision proxy drone, and each decision proxy user equipment remains constant within a time slot;
[0008] S2, at the beginning of each time slot, based on the air-ground collaborative network framework, the joint decision-making problem in the air-ground collaborative network is transformed into a top-down three-layer hierarchical decision. The three decision-making tasks include cell association, trajectory planning, and unloading scheduling, which constitute the top, middle, and bottom layers of the hierarchical structure respectively; based on the observed positions of all drones and the average data backlog of drones, base stations, and all users in the cell, the base station decides on the cell association plan, the drone decides on the trajectory planning plan, and the user equipment decides on the unloading scheduling action;
[0009] Specifically, in each time slot t, the cell base station generates a cell association plan according to the policy network and transmits it to the UAV to be connected, and the UAV generates a UAV trajectory planning plan according to the policy network; the time slot interval Δ is defined, and the cell base station re-formulates the cell association scheduling every Δ time slots, and the user equipment performs unloading scheduling according to the latest cell association scheduling instruction issued by the base station in each time slot τ; when the trajectory planning of the UAV is determined, the flight direction remains unchanged in the subsequent Δ time slots until a new round of trajectory planning is generated;
[0010] S3 adopts heterogeneous joint decision-making based on mean field theory to perform Markov modeling on cell association, UAV trajectory planning and user offloading scheduling problems in dynamic environments. In cell association decision-making, the interaction between base station agents is equivalent to the interaction between the average effect of a single base station and a group of neighboring base stations based on mean field theory. Through information interaction with the environment, ground base stations, UAVs and users learn to obtain the joint optimal strategy.
[0011] The air-ground collaborative network includes a ground network consisting of J cells and a drone network consisting of K drones; each base station j is only connected to M j A fixed ground user equipment is associated with such a set of user equipment. And the user equipment sets of each cell do not have any intersection User equipment m j,i represents the i-th user in cell j. In cell j, user equipment m j,i satisfy Base station j and its associated user equipment set Together they constitute cell j. The user set of this multi-cell network is expressed as Each cell has at most one drone providing service. If cell j is associated with the kth drone, then user equipment m in cell j i When a request to offload to a drone is generated, it can only be offloaded to drone k, and the cell-associated scheduling is deployed at base station j of the corresponding cell.
[0012] Furthermore, the modeling process of hierarchical decision making includes the following steps:
[0013] In the cell association layer, the state observed by the base station is expressed as Where Q UAV,t-Δ Represents the position matrix composed of all drones; u BS,t and u UAV,t Represent the mission data backlog values of the base station and the drone respectively; represents the average data backlog of all users in cell j in time slot t, The cell association scheme of cell j is expressed as Each base station j decides which drone to associate with. , other drones cannot be selected as edge computing nodes by users in cell j, and the joint solution executed by all agents at the top level is expressed as
[0014] In the trajectory planning layer, the state observed by the drone Defined as Flight direction in represents the flight direction set; the joint trajectory planning scheme of K UAVs is expressed as
[0015] In the offloading scheduling layer, the state observed by the user in time slot t is Expressed as where q UAV,t and D -k,t Indicates the location and data accumulation information of the drone connected to cell j, and the decision Given; D j,t Indicates the task data backlog of the base station in cell j; u UE,j,t represents the data accumulation information of each user in cell j, v j,m,t represents the data size information of the head of the local FIFO task queue; the joint unloading scheduling action determined by each user equipment in cell j in time slot t is expressed as The elements The actions performed on behalf of each user include four operations: processing data locally, unloading a set of task data at the head of the team to the base station, unloading a set of task data at the head of the team to the drone, and unloading two sets of task data at the head of the team to the drone.
[0016] Furthermore, in step S3, the process of learning and obtaining a joint optimal strategy by the ground base station, the drone and the user through information interaction with the environment includes the following steps:
[0017] S31, obtaining the state, policy network parameters and value network parameters of the decision agent base station; obtaining the state, policy network parameters and value network parameters of each decision agent drone; obtaining the state, policy network parameters and value network parameters of each decision agent user device; initializing the dictionary of each network to an empty matrix;
[0018] S32, in time slot t, each cell base station j obtains state observations, including the position matrix composed of drones, the task data backlog values of base stations and drones, and the average data backlog of all users in cell j in time slot t, and generates a cell association plan based on the strategy network to calculate the new average action; specifically, considering the discrete action space, D represents the number of all possible actions, assuming that the action is represented by one-hot encoding as a j Each element in corresponds to an action; the action of the decision agent k is expressed as the average action of the neighborhood and the disturbance term λa j,k The sum of the neighborhood's average actions under the current game state is the empirical distribution of actions taken by the neighborhood of decision agent j, Each element represents the average action of the neighborhood of each action; the action based on the strategy decision of the agent in the neighborhood at the last moment is averaged. Where N j represents the number of users in cell j, action a k The choice of obedience strategy in is the average action used in the last decision, and s is the current state;
[0019] Each cell base station transmits the joint cell association plan to the drone to be connected; the decision agent drone obtains the state observation and generates the drone trajectory planning plan according to the policy network; in the next Δ time slots, each drone flies to the new position and transmits the position information and task queue information to the user equipment of the associated cell;
[0020] User device m obtains state observations and generates user offloading scheduling actions according to the policy network;
[0021] Among them, at the end of the time slot, the device calculates the time differential error of each decision agent according to the reward obtained, and updates the policy network and value network of the user equipment decision agent; after Δ time slots, the cell association plan and trajectory planning plan end; the base station calculates the time differential error of the base station decision agent according to the reward obtained, updates the policy network and value network of the base station decision agent, and generates a new cell association plan according to the state; the drone calculates the time differential error of the drone decision agent according to the reward obtained, updates the policy network and value network of the drone decision agent, and generates a new trajectory planning plan according to the state;
[0022] S33, repeat the above step S32 in each time slot until convergence, and output the action selection decision as the optimal decision.
[0023] Furthermore, the reward function of the decision agent j that performs cell association is expressed as:
[0024]
[0025] Where Y j,m,i is the reward of community user m, Y -k,i is the reward of drone k, Y j,i is the reward of base station j, Z t is the construction credit allocation term, is the sum of the differences in the data backlog of different cell tasks, expressed as
[0026] The reward function of the decision agent k that performs the UAV trajectory planning is expressed as:
[0027]
[0028] The reward function of agent m that performs user offloading scheduling is expressed as:
[0029]
[0030] In the formula, Z t (u UE,j,t ) is the sum of the differences in task data backlogs of different users in cell j, which can be expressed as Where α is the relevant constant of the user reward function, D j,m, t represents the local task queue observed at the end of time slot t, that is, the backlog of data, ΔD j,m,t This variable represents the amount of data backlog.
[0031] Further, in step S32, when base station j performs cell association decision, it uses the average solution of neighboring base stations The actor generates new solutions through the policy network The strategy representation method of kernel function output is Neighborhood averaging scheme As an extension of the state, they serve as the input of the policy network, w A,j is the weight vector of the actor, is the kernel vector of the actor dictionary under Gaussian kernel mapping, feature The kernel vector of the actor dictionary under Gaussian kernel mapping is:
[0032]
[0033] in represents the size of the actor dictionary of base station j, σ s,j and σ a,j They represent the state of base station j and the core parameters corresponding to the scheme respectively.
[0034] Furthermore, in step S32, the strategy of the kernel function output of drone k is expressed as feature Actor Core Vector The value of each element in is in, represents the size of the drone k-actor dictionary, σ s,k is the kernel parameter corresponding to the state of UAV k.
[0035] Furthermore, for the agent that unloads the scheduling layer, the actor and judge initialize their own feature dictionaries and The strategy output by the kernel function of user device m is expressed as For agent m that performs offloading scheduling, the characteristics of the actor Nuclear Vector The value of each element in is The judgement feature is Nuclear Vector The values of the elements in are:
[0036]
[0037] in and represents the size of the actor and judge dictionary of user device m, σ s,j,m and σ a,j,m They represent the state of user device m and the core parameters corresponding to the action respectively.
[0038] Furthermore, in step S32, updating the policy network and value network of the decision agent includes the following steps:
[0039] S321, at the end of time slot t, user device m obtains a reward, calculates the time difference error of the user device decision agent, updates the user device policy network parameters under multiple decision agents and the user device value network parameters under multiple decision agents, and performs feature similarity detection to determine whether to add new features to the dictionary of the decision agent user device m;
[0040] Specifically, the time difference error of the user device decision agent at time t is equal to the product of the reward of the user device decision agent at time t+1 plus the correlation coefficient and the state value function of the user device decision agent at time t+1 minus the action value function of the user device decision agent at time t, where the correlation coefficient is denoted as γ, expressed as:
[0041]
[0042] User equipment policy network parameters w under multiple decision agents A,j,m The updating method is the sum of the original policy network parameters and the correlation coefficient, the differential of the policy network parameters, the logarithm of the kernel function output strategy of the user device under multiple decision agents, and the time difference error of the user device decision agent. The correlation coefficient is denoted as α, which satisfies:
[0043]
[0044] The judge network parameter w of user equipment m C,j,m The update method is the sum of the original value network parameters and the correlation coefficient, the time difference error of the user equipment decision agent, and the product of the kernel vector. The correlation coefficient is denoted as α, which satisfies:
[0045]
[0046] S322, at the end of time slot t+Δ, base station j obtains the reward, calculates the time difference error of the base station decision agent, updates the base station strategy network parameters under multiple decision agents and the base station value network parameters under multiple decision agents, performs feature similarity detection, and determines whether to add the new feature to the dictionary of base station j;
[0047] The time difference error of the base station decision agent at time t is equal to the product of the reward of the base station decision agent at time t+Δ plus the correlation coefficient and the state value function of the base station decision agent at time t+Δ minus the action value function of the base station decision agent at time t. The correlation coefficient is denoted as γ, which is expressed as:
[0048]
[0049] Under the guidance of the time difference error of the base station decision agent, the update method of the base station strategy network parameters under multiple decision agents is the sum of the original strategy network parameters and the product of the correlation coefficient, the differential of the strategy network parameters, the logarithm of the kernel function output strategy of the base station under multiple decision agents, and the time difference error of the base station decision agent. The correlation coefficient is denoted as α, which satisfies:
[0050]
[0051] Under the guidance of the time difference error of the base station decision agent, the update method of the base station value network parameters under multiple decision agents is the sum of the original value network parameters and the product of the correlation coefficient, the time difference error of the base station decision agent and the kernel vector. The correlation coefficient is denoted as α, which satisfies:
[0052]
[0053] S323, drone k obtains a reward, calculates the time difference error of the drone decision agent, the drone strategy network parameters under multiple decision agents, and the drone value network parameters under multiple decision agents, performs feature similarity detection, and determines whether to add new features to the dictionary of drone k;
[0054] When each drone executes a joint plan After receiving the reward after Δ time slots, the time difference error of the value network update of the drone decision agent is equal to the product of the reward of the drone decision agent at time t+Δ plus the correlation coefficient and the state value function of the drone decision agent at time t+Δ minus the action value function of the drone decision agent at time t. The correlation coefficient is denoted as γ, which is expressed as:
[0055]
[0056] The update method of the UAV strategy network parameters in the trajectory planning layer under multiple decision agents is the sum of the original strategy network parameters and the correlation coefficient, the differential of the strategy network parameters, the logarithm of the kernel function output strategy of the UAV under multiple decision agents, and the time difference error of the UAV decision agent. The correlation coefficient is denoted as α, which satisfies:
[0057]
[0058] The updating method of the value network parameters of the trajectory planning layer of the drone under multiple decision agents is the sum of the original value network parameters and the product of the correlation coefficient, the time difference error of the drone decision agent and the kernel vector. The correlation coefficient is denoted as α, which satisfies:
[0059]
[0060] Furthermore, feature similarity detection refers to comparing the new feature with the original features in the dictionary one by one, calculating the difference between the two, defining the difference function threshold between the current new feature and the original feature in the dictionary, and if the difference is greater than the threshold, adding the current new feature to the dictionary set; if the difference is less than the threshold, maintaining the current dictionary set unchanged.
[0061] Compared with the prior art, the present invention has the following beneficial effects:
[0062] The cell-association trajectory planning and unloading scheduling joint dynamic decision-making method for air-ground collaborative multi-access edge computing of the present invention can solve the problem of matching the communication / computing requirements with multiple resources in the highly dynamic air-ground integrated network with time / space variations. The present invention can be used for base station cell association decision-making, online trajectory planning of drones, and user task unloading scheduling in a network scenario model where large-scale multi-level drones and base stations collaborate to provide unloading for ground user equipment. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 It is a flow chart of a joint dynamic decision-making method for cell-associated trajectory planning and offloading scheduling for air-ground coordinated multi-access edge computing according to an embodiment of the present invention;
[0064] Figure 2 A schematic diagram of a scenario of an embodiment of the present invention;
[0065] Figure 3 It is a flowchart of the heterogeneous joint dynamic decision-making method based on mean field theory;
[0066] Figure 4 It is a schematic diagram of the hierarchical joint dynamic decision-making structure based on the scheme;
[0067] Figure 5 Flowchart for updating the decision-making agent strategy network and value network based on feature similarity detection. DETAILED DESCRIPTION
[0068] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.
[0069] This embodiment discloses a method for joint dynamic decision-making of cell association, trajectory planning and offloading scheduling for multi-access edge computing, and the method for joint dynamic decision-making of cell association trajectory planning and offloading scheduling includes the following steps:
[0070] S1, divide the duration T into N equal time slots, and the state of each decision proxy base station, each decision proxy drone, and each decision proxy user equipment remains constant within a time slot;
[0071] S2, at the beginning of each time slot, based on the air-ground collaborative network framework, the joint dynamic decision-making problem in the air-ground collaborative network is transformed into a top-down three-layer hierarchical decision. The three decision-making tasks include cell association, trajectory planning, and unloading scheduling, which constitute the top, middle, and bottom layers of the hierarchical structure respectively; based on the observed positions of all drones and the average data backlog of drones, base stations, and all users in the cell, the base station decides on the cell association plan, the drone decides on the trajectory planning plan, and the user equipment decides on the unloading scheduling action;
[0072] Specifically, in each time slot t, the cell base station generates a cell association plan according to the policy network and transmits it to the UAV to be connected, and the UAV generates a UAV trajectory planning plan according to the policy network; the time slot interval Δ is defined, and the cell base station re-formulates the cell association scheduling every Δ time slots, and the user equipment performs unloading scheduling according to the latest cell association scheduling instruction issued by the base station in each time slot τ; when the trajectory planning of the UAV is determined, the flight direction remains unchanged in the subsequent Δ time slots until a new round of trajectory planning is generated;
[0073] S3 adopts heterogeneous joint decision-making based on mean field theory to perform Markov modeling on cell association, UAV trajectory planning and user offloading scheduling problems in dynamic environments. In cell association decision-making, the interaction between base station agents is equivalent to the interaction between the average effect of a single base station and a group of neighboring base stations based on mean field theory. Through information interaction with the environment, ground base stations, UAVs and users learn to obtain the joint optimal strategy.
[0074] like Figure 2 As shown in Figure 1, a multi-UAV scenario is introduced in the multi-cell network under study. The air-ground collaborative network includes a ground network consisting of J cells and a UAV network consisting of K UAVs. The ground network consists of J cells, where each base station j (satisfying j∈{1,…,J}) is connected only to M j A fixed ground user equipment is associated with such a set of user equipment. (satisfy ), and the user equipment sets of each cell do not have an intersection User equipment m j,i represents the i-th user in cell j. In cell j, user equipment m j,i satisfy Base station j and its associated user equipment set Together they constitute cell j. The user set of this multi-cell network can be expressed as In addition, it is assumed that each cell has at most one drone providing service. If cell j is associated with the kth drone, when the user equipment m in cell j generates a request to unload to the drone, it can only unload to drone k. This cell-associated scheduling is deployed at the base station j of the cell. The decision-making proxy base station can be expressed as decision-making proxy base station 1, decision-making proxy base station 2, ..., decision-making proxy base station J, that is, decision-making proxy base station 1, 2, ..., J, when describing a certain decision-making proxy base station, it is expressed as decision-making proxy base station j; the decision-making proxy drone can be expressed as decision-making proxy drone 1, decision-making proxy drone 2, ..., decision-making proxy drone K, that is, decision-making proxy drone 1, 2, ..., K, when describing a certain decision-making proxy drone, it is expressed as decision-making proxy drone k; the decision-making proxy user equipment can be expressed as decision-making proxy user equipment 1, decision-making proxy user equipment 2, ..., decision-making proxy user equipment M, that is, decision-making proxy user equipment 1, 2, ..., M, when describing a certain decision-making proxy user equipment, it is expressed as decision-making proxy user equipment m.
[0075] The duration of the entire method (denoted as T) is divided into N equal time slots (1, ..., N), with t representing the current time slot, t-1 representing the previous time slot, and t+1 representing the next time slot. In order to keep the state of each decision-making proxy base station, each decision-making proxy drone, and each decision-making proxy user equipment approximately constant within a time slot, the length of the time slot should be appropriately selected, i.e., δ t =T / N.
[0076] Due to the small-scale fading of the terrestrial non-line-of-sight channel, the channel gain is time-varying. Here, we consider the block fading channel model. That is, the terrestrial channel gain remains unchanged within a time slot, but varies between different time slots. In time slot t, a certain base station j and user equipment m (where ) can be expressed as h 0,j,m,t The corresponding channel power gain where β and d j,m Represent the path loss index and the distance between base station j and user equipment m respectively. In time slot t, given device m (where ) position q m (column vector) and the location q of the drone k serving the cell where the user is located UAV,t =[x t ,y t , H], which is the position matrix Q composed of all drones UAV,t-Δ Extract the distance d between the drone and the user device 0,m,t =|q UAV,t -q m |. And there is a line-of-sight channel between the drone and the user equipment, and the distance between the drone and the user equipment m (where ) is the channel power gain between where |h0| 2 Indicates the channel power gain when the reference distance is 1m.
[0077] In each time slot t, the cell base station generates a cell association plan based on the policy network and transmits it to the drone to be connected. Then the drone generates a drone trajectory planning plan based on the policy network. Define the time slot interval Δ. The cell base station re-formulates the cell association scheduling every Δ time slots, and the user equipment needs to perform unloading scheduling according to the latest cell association scheduling instruction issued by the base station in each time slot τ. Similarly, when the trajectory planning of the drone is determined, the flight direction remains unchanged in the subsequent Δ time slots until a new round of trajectory planning is generated. That is, in the next Δ time slots: each drone flies to a new position, that is, a new position matrix Q is generated. UAV,t-Δ The offloading scheduling options for each user device include performing calculations locally on the device and offloading task data to the associated drone or associated base station in the cell. The duration of offloading or (and) computing task data is specified as τ. Assuming that there are sufficient frequency domain channels, the offloading transmissions of user devices will not interfere with each other, and the calculation results can be returned to the user through a dedicated frequency domain channel. Therefore, the achievable offloading rate of base station j or drone k in time slot Δ is:
[0078]
[0079] Among them, B and P m and They represent the uplink channel bandwidth, the transmit power at user equipment m, and the average noise power, respectively. When the subscript X=0, it represents the unloading rate of the UAV; X=j∈{1,…,J} represents the unloading rate of the base station.
[0080] It is assumed that each user device continuously generates computing tasks in a time slot, and the statistical characteristics of the tasks generated are unknown to the network. The number of task bits generated during time slot Δ-1 can be expressed as L j,m,Δ-1 Due to the overhead of signaling and data preparation, these task data can only be locally calculated or offloaded in time slot Δ, and the decision is made in time slot Δ-1. However, due to the continuous generation of task data throughout the time slot, the offload scheduling decision in time slot Δ-1 cannot obtain L j,m,Δ-1 knowledge.
[0081] The CPU cycle frequency of each user device and the CPU cycle frequency of the multi-access edge computing server deployed on the base station / drone are defined as f local and f MECDefine the processing density c, which represents the number of CPU cycles required to process 1 bit of task data.
[0082] In addition, the user equipment, base station and drone are equipped with local task queues to cache unprocessed task data, which follows the first-in-first-out rule. j,m,Δ-1 The calculation cannot be completed in the time slot Δ, and the remaining unprocessed data L BL,j,m,Δ-1 The task queue will be stored there and will be calculated first in the next time slot. The following analyzes the L observed in time slot Δ in the local computing and offloading scenarios respectively. BL,j,m,Δ-1 And the change of task queue length:
[0083] In the case of local computation at user device m, define D j,m,Δ-1 is the task queue observed at the end of time slot Δ-1, which contains the amount of data generated but not processed before time slot Δ-1. In time slot Δ, due to the first-in first-out rule, these data D j,m,Δ-1 Will be processed first, and then the task data L j,m,Δ-1 Total processing time
[0084] Δt j,m,Δ =c(L j,m,Δ-1 +D j,m,Δ-1 ) / f local If Δt j,m,Δ <τ, then all task data can be processed before the end of the time slot, D j,m,Δ =0 and L BL,j,m,Δ-1 = 0. Conversely, when Δt j,m,Δ ≥τ, some task data cannot be processed completely, and the task queue length becomes D j,m,Δ =D j,m,Δ-1 +L j,m,Δ-1 -τf local / c. In this case, if D j,m,Δ-1 <τ flocal / c,L BL,j,m,Δ-1 =D j,m,Δ ; otherwise, there is no time to process L j,m,Δ-1 , so L BL,j,m,Δ-1 =L j,m,Δ-1 .
[0085] When there is an unloading operation in time slot t, L j,m,Δ-1 The mission data of bits can be transmitted to the multi-access edge computing server at the drone or the base station associated with the user equipment, where the transmission time is t rans =L j,m,Δ-1 / R X,m,Δ If t rans>τ, the multi-access edge computing server will ignore the received data and store it in the user's local task queue for processing. While receiving data, the multi-access edge computing server can process the data cached in its task queue. X,Δ-1 The time to process this part of the task data is defined as t pre According to t rans and t pre , analyze the relationship between L BL,j,m,Δ-1 There are two situations, such as Figure 1 shown.
[0086] In t rans >t pre In the scenario, all D x,Δ-1 All task data can be processed before the offload transmission ends. Therefore, the total processing time Δt MEC,m,Δ =t trans +L j,m,Δ-1 ·c / f MEC If Δt MEC,m,Δ <τ,D X,Δ =L BL,j,m,Δ-1 =0; otherwise, the task data L j,m,Δ-1 Cannot be processed, D X,Δ =L BL,j,m,Δ-1 =L j,m,Δ-1 -f MEC (τ-t trans ) / c. However, in t rans ≤t pre In the scenario of X,Δ-1 It has not been processed yet. Therefore, you need to wait for D X,Δ-1 After the data is processed, process L j,m,Δ-1 The total time can be expressed as Δt′ MEC,m,Δ =t pre +L j,m,Δ-1 ·c / f MEC If Δt′ MEC,m,Δ <τ,D X,Δ =L BL,j,m,Δ-1 = 0. On the contrary, for Δt′ MEC,m,Δ ≥τ,D X,Δ =L j,m,Δ-1 +D X,Δ-1 -τf MEC / c. If D X,Δ-1 <τf local / c,L BL,j,m,Δ-1 =D X,Δ Otherwise, L BL,j,m,Δ-1 =L j,m,Δ .
[0087] Since decision variables such as cell association and offloading scheduling have different time scales, that is, after cell association is performed in time slot t, the next round of decision will be made after Δ time slots, during which the offloading scheduling decision has been made many times, it is difficult to solve this problem using mutually coupled MDPs. Therefore, for the joint decision problem, different tasks can be modeled as SMDPs with different time scales, and a hierarchical structure is constructed through task relationships. SMDP is a special MDP suitable for modeling continuous-time discrete event systems. The decision scheme in SMDP can last for multiple time steps to simulate decision-making behaviors with multiple granularities. Hierarchy allows complex decision problems to be further subdivided into top-level decision modules and bottom-level decision modules for execution. Based on the scheme framework, the joint decision problem in the air-ground collaborative network is transformed into a top-down hierarchical decision with three layers. The three decision tasks include cell association, trajectory planning, and offloading scheduling, which constitute the top, middle, and bottom layers of the hierarchical structure respectively.
[0088] In addition, a reward function based on data backlog is designed. Define the change ΔD of the data backlog in the task queue of user m in time slot t j,m,t . Use ΔD j,m,t and D j,m,t Together, we judge the quality of user m’s unloading schedule and construct the reward Y as follows: On the one hand, the term log(1+X) ensures that the numerator is always greater than or equal to zero, and limits the j,m,t The continuous increase leads to the problem that the reward function value is too divergent; on the other hand, due to ΔD j,m,t It is possible to be equal to or close to 0, and the 2+tanh(-X) term limits the denominator to the range of [1, 3] to avoid a cliff-like increase or decrease in rewards. In this construction, a smaller D j,m,t and a larger ΔD j,m,t The action of is given a higher reward. Secondly, in order to more fairly process the task data between multiple cells and the task data of each user in a cell, the credit allocation term Z is constructed t as follows: The credit allocation item avoids favoritism towards a certain cell or a certain user device, which causes the decision-making agent's decision to fall into a local optimal solution. For example, the task data accumulation of the remaining cells has been effectively alleviated. There is a cell with a high amount of task data accumulation, but this problem cannot be reflected in the reward value of the decision-making agent. So far, this embodiment gives the construction method and meaning of the two terms contained in the reward function. Corresponding to the decision-making agent j that executes the cell association, its reward function can be expressed as: As the top-level decision-making agent, its reward function Contains Y of all users in the cell j,m,i 、Y of drone -k,iand the base station's Y j,i The reward function of the decision agent k that performs the UAV trajectory planning can be expressed as: The reward function of agent m that performs user offloading scheduling can be expressed as: Intuitively, cell association and trajectory planning require rewards to be formulated based on the task data backlog of the entire system. For offloading scheduling, the agent only needs to obtain the information of users in the cell and obtain the joint reward in the cell based on the execution of offloading scheduling by each user in the cell.
[0089] The specific modeling method of SMDP decision-making process is given below. Figure 4 , in the cell-associated SMDP, the state observed by the base station can be expressed as Where Q UAC,t-Δ Represents the position matrix of all drones. BS,t and u UAV,t Represent the mission data backlog values of the base station and the UAV respectively. represents the average data backlog of all users in cell j in time slot t, and The cell association scheme of cell j can be expressed as Each base station j decides which drone to associate with. , other drones cannot be selected as edge computing nodes by users in cell j, that is, a cell can only establish a connection with at most one drone in the same time slot. The joint solution executed with all top-level agents can be expressed as In trajectory planning SMDP, the state observed by the drone It can be defined as Each drone needs to decide the direction to fly The joint trajectory planning scheme for K UAVs is expressed as In the offloaded scheduled MDP, the state observed by the user in time slot t is It can be expressed as where q UAV,t and D -k,t Indicates the location and data accumulation information of the drone connected to cell j. This information is determined by the decision Given. UE,j,t represents the data accumulation information of each user in cell j, v j,m,t Represents the data size information at the head of the local FIF0 task queue.
[0090] In the above state definition of the decision agent, the trajectory planning layer, as the middle layer of multi-granularity decision-making, inherits the top-level solution information. This can be directly reflected in the state of the decision-making agent drone. The user unloading scheduling layer is the bottom layer of multi-granularity decision-making, which inherits the solution information of the top and middle layers. and For agent m, Information determines the drones connected to the cell, The information determines the real-time position of the drone, so it can also be expressed as The definition given above is a more intuitive representation. Then, the joint unloading scheduling action determined by each user equipment in cell j in time slot t can be expressed as The elements The actions performed by each user include processing data locally, unloading a set of task data at the head of the team to the base station, unloading a set of task data at the head of the team to the drone, and unloading two sets of task data at the head of the team to the drone. The application of the hierarchical structure not only greatly reduces the complexity of each sub-problem, but also improves the overall learning efficiency by reusing lower-level strategies.
[0091] See also Figure 3 In step S3, the process of learning the joint optimal strategy between the ground base station, the UAV and the user through information interaction with the environment includes the following steps:
[0092] S31, obtaining the state, policy network parameters and value network parameters of the decision agent base station; obtaining the state, policy network parameters and value network parameters of each decision agent drone; obtaining the state, policy network parameters and value network parameters of each decision agent user device; initializing the dictionary of each network to an empty matrix;
[0093] S32, in time slot t, each cell base station j obtains state observations, including the position matrix composed of drones, the task data backlog values of base stations and drones, and the average data backlog of all users in cell j in time slot t, and generates a cell association plan based on the strategy network to calculate a new average action;
[0094] Each cell base station transmits the joint cell association plan to the drone to be connected; the decision agent drone obtains the state observation and generates the drone trajectory planning plan according to the policy network; in the next Δ time slots, each drone flies to the new position and transmits the position information and task queue information to the user equipment of the associated cell;
[0095] User device m obtains state observations and generates user offloading scheduling actions according to the policy network;
[0096] Among them, at the end of the time slot, the device calculates the time differential error of each decision agent according to the reward obtained, and updates the policy network and value network of the user equipment decision agent; after Δ time slots, the cell association plan and trajectory planning plan end; the base station calculates the time differential error of the base station decision agent according to the reward obtained, updates the policy network and value network of the base station decision agent, and generates a new cell association plan according to the state; the drone calculates the time differential error of the drone decision agent according to the reward obtained, updates the policy network and value network of the drone decision agent, and generates a new trajectory planning plan according to the state;
[0097] S33, repeat the above step S32 in each time slot until convergence, and output the action selection decision as the optimal decision.
[0098] The influence of average action is considered in mean field reinforcement learning. Considering the discrete action space, D represents the number of all possible actions, assuming that the action is represented by one-hot encoding as a j Each element in corresponds to an action. Then the action of the decision agent k can be expressed as the average action of the neighborhood and a small disturbance term λa j,k The sum of: in, It can be interpreted as the empirical distribution of actions taken by the neighborhood of agent j. Solve the average action of the neighborhood in the current game state Take the average of the actions decided by the last strategy of the agents in the neighborhood Based on the obtained Agent j chooses a new action based on the strategy.
[0099] Taking the value-based method as an example, the update of the Q table of each base station decision agent j after the scheme is terminated can be expressed as:
[0100]
[0101] Where Δ is from the state To the terminal state The time step, is the cumulative discount reward during the execution of the cell association scheme. Similarly, the update of the Q table of each drone after the scheme is terminated can be expressed as:
[0102]
[0103] where r t+Δ Represents the cumulative discounted reward during the execution of the trajectory planning solution. and trajectory planning scheme Under the guidance, the offloading scheduling decision of the underlying MDP is based on the action value function to evaluate the optimal action: The Q-value update of trajectory planning can be expressed as:
[0104]
[0105] Kernel-based actor-critic algorithm:
[0106] Initialize the feature dictionaries for the actor network and the critic network and When base station j performs cell association decision, it uses the average solution of neighboring base stations. The actor generates new solutions through the policy network The policy representation of the kernel function output can be written as Neighborhood averaging scheme It can be regarded as an extension of the state and serves as the input of the actor network. A,j is the weight vector of the actor, is the kernel vector of the actor dictionary under Gaussian kernel mapping, corresponding to the feature The value of each element in the kernel vector is:
[0107]
[0108] in represents the size of the actor dictionary of base station j, σ s,j and σ a,j Represent the state of base station j and the kernel parameter corresponding to the scheme respectively. The action value function approximated by the kernel function can be expressed as w C,j is the weight vector of the judge, is the kernel vector of the judge dictionary under Gaussian kernel mapping, corresponding to the feature The value of each element in the kernel vector is:
[0109]
[0110] in Represents the size of the dictionary of the evaluator of base station j. At the end of time slot t+Δ, base station j obtains the reward, calculates the time differential error of the base station decision agent, updates the base station strategy network parameters under multiple decision agents and the base station value network parameters under multiple decision agents, performs feature similarity detection, and determines whether to add new features to the dictionary of base station j. The time differential error of the decision agent is equal to the difference between the true action value and the estimated action value. The time differential error of the base station decision agent at time t is equal to the product of the reward of the base station decision agent at time t+Δ plus the correlation coefficient and the state value function of the base station decision agent at time t+Δ minus the action value function of the base station decision agent at time t. The correlation coefficient is denoted as γ. When each cell implements the joint plan And get reward after Δ time slots After that, the value network updates the TD error:
[0111]
[0112] Subsequently, under the guidance of the time difference error of the base station decision agent, the update method of the base station strategy network parameters under multiple decision agents is the sum of the original strategy network parameters and the correlation coefficient, the differential of the strategy network parameters, the logarithm of the kernel function output strategy of the base station under multiple decision agents, and the time difference error of the base station decision agent. The update method of the base station value network parameters under multiple decision agents is the sum of the original value network parameters and the correlation coefficient, the time difference error of the base station decision agent, and the kernel vector. The correlation coefficient is denoted as α; that is, under the guidance of TD error, the strategy network parameter w A,j and value network parameter wC ,j The update method is as follows: Finally, the actor and discriminator of base station j use ALD test to update the dictionary respectively. For the decision agent of trajectory planning layer, the actor and discriminator initialize their own feature dictionaries and
[0113] The strategy of the kernel function output of drone k can be expressed as: Since the trajectory planning layer does not introduce neighborhood average information, the feature Actor Core Vector The values of the elements in are:
[0114]
[0115] in, represents the size of the drone k-actor dictionary, σ s,k is the kernel parameter corresponding to the state of drone k. The action value function approximated by the kernel function can be expressed as:
[0116]
[0117] Kernel vector of the judge The values of the elements in are:
[0118]
[0119] in represents the size of the drone k-judge dictionary, σ a,k is the kernel parameter corresponding to the solution of UAV k.
[0120] Drone k obtains rewards, calculates the time difference error of the drone decision agent, the drone strategy network parameters under multiple decision agents, and the drone value network parameters under multiple decision agents, performs feature similarity detection, and determines whether to add new features to the dictionary of drone k. The time difference error of the value network update of the drone decision agent at time t is equal to the product of the reward of the drone decision agent at time t+Δ plus the correlation coefficient and the state value function of the drone decision agent at time t+Δ minus the action value function of the drone decision agent at time t, and the correlation coefficient is denoted as γ. When each drone executes the joint plan And get reward after Δ time slots After that, the value network updates the TD error:
[0121]
[0122] The update method of the UAV strategy network parameters in the trajectory planning layer under multiple decision-making agents is the sum of the original strategy network parameters and the correlation coefficient, the differential of the strategy network parameters, the logarithm of the kernel function output strategy of the UAV under multiple decision-making agents, and the time difference error of the UAV decision-making agent. The update method of the UAV value network parameters in the trajectory planning layer under multiple decision-making agents is the sum of the original value network parameters and the correlation coefficient, the time difference error of the UAV decision-making agent, and the kernel vector. The correlation coefficient is denoted as α; the actor network parameters of the decision agent in the trajectory planning layer are w A,k and the judge network parameters w C,k The update method is as follows: Similarly, for the decision agent of the offloading scheduling layer, the actor and judge initialize their own feature dictionaries and The kernel function output strategy of user device m can be expressed as The action-value function is approximately:
[0123]
[0124] For the decision agent m that performs offloading scheduling, the characteristics of the actor Nuclear Vector The values of the elements in are:
[0125]
[0126] The judgement feature is Nuclear Vector The values of the elements in are:
[0127]
[0128] in and represents the size of the actor and judge dictionary of user device m, σ s,j,m and σ a,j,m Represent the kernel parameters corresponding to the state and action of user device m respectively. The evaluator evaluates the quality of the action taken by the actor by using the parameterized state value function V(s; w) or action value function Q(s, a; w), and uses the time difference error of the user device decision agent to guide the update of the policy network and its own value network parameters. At the end of time slot t, user device m obtains the reward, calculates the time difference error of the user device decision agent, updates the user device policy network parameters under multiple decision agents and the user device value network parameters under multiple decision agents, and performs feature similarity detection to determine whether to add new features to the dictionary of the decision agent user device m. The time difference error of the user device decision agent at time t is equal to the product of the reward of the user device decision agent at time t+1 plus the correlation coefficient and the state value function of the user device decision agent at time t+1 minus the action value function of the user device decision agent at time t, and the correlation coefficient is denoted as γ. The TD error can be expressed as:
[0129]
[0130] User equipment policy network parameters w under multiple decision agents A,j,m The update method is the sum of the original policy network parameters and the correlation coefficient, the differential of the policy network parameters, the logarithm of the kernel function output policy of the user device under multiple decision agents, and the time difference error of the user device decision agent. The correlation coefficient is denoted as α, and the actor network parameter w of user device m is A,j,m and the judge network parameters w C,j,m The update method is as follows:
[0131]
[0132] The judge network parameter w of user equipment m C,j,m The update method is the sum of the original value network parameters and the correlation coefficient, the time difference error of the user equipment decision agent, and the product of the kernel vector. The correlation coefficient is denoted as α. The above process also satisfies:
[0133]
[0134] like Figure 5 As shown in the figure, feature similarity detection means: comparing the new feature with the original features in the dictionary one by one, calculating the difference between the two, defining the difference function threshold between the current new feature and the original feature in the dictionary, and if the difference is greater than the threshold, adding the current new feature to the dictionary set; if the difference is less than the threshold, maintaining the current dictionary set unchanged.
[0135] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0139] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0140] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A joint decision-making method for cell association, trajectory planning and unloading scheduling, characterized in that: The cell-associated trajectory planning and unloading scheduling joint decision-making method is oriented to multi-access edge computing and includes the following steps: S1, divide the duration T into N equal time slots, and the state of each decision proxy base station, each decision proxy drone, and each decision proxy user equipment remains constant within a time slot; S2, at the beginning of each time slot, based on the air-ground collaborative network framework, the joint decision-making problem in the air-ground collaborative network is transformed into a top-down three-layer hierarchical decision. The three decision-making tasks include cell association, trajectory planning, and unloading scheduling, which constitute the top, middle, and bottom layers of the hierarchical structure respectively; based on the observed positions of all drones and the average data backlog of drones, base stations, and all users in the cell, the base station decides on the cell association plan, the drone decides on the trajectory planning plan, and the user equipment decides on the unloading scheduling action; Specifically, in each time slot t, the cell base station generates a cell association plan according to the policy network and transmits it to the UAV to be connected, and the UAV generates a UAV trajectory planning plan according to the policy network; the time slot interval Δ is defined, and the cell base station re-formulates the cell association scheduling every Δ time slots, and the user equipment performs unloading scheduling according to the latest cell association scheduling instruction issued by the base station in each time slot τ; when the trajectory planning of the UAV is determined, the flight direction remains unchanged in the subsequent Δ time slots until a new round of trajectory planning is generated; S3, adopts heterogeneous joint decision-making based on mean field theory to conduct Markov modeling of cell association, UAV trajectory planning and user offloading scheduling problems in dynamic environments. In cell association decision-making, the interaction between base station agents is equivalent to the interaction between the average effect of a single base station and a group of neighboring base stations based on mean field theory. Through information interaction with the environment, ground base stations, UAVs and users learn to obtain a joint optimal strategy. In step S3, the process of learning the joint optimal strategy by the ground base station, the UAV and the user through information interaction with the environment includes the following steps: S31, obtaining the state, policy network parameters and value network parameters of the decision agent base station; obtaining the state, policy network parameters and value network parameters of each decision agent drone; obtaining the state, policy network parameters and value network parameters of each decision agent user device; initializing the dictionary of each network to an empty matrix; S32, in time slot t, each cell base station j obtains state observations, including the position matrix composed of drones, the task data backlog values of base stations and drones, and the average data backlog of all users in cell j in time slot t, and generates a cell association plan based on the strategy network to calculate the new average action; specifically, considering the discrete action space, D represents the number of all possible actions, assuming that the action is represented by one-hot encoding as a j Each element in corresponds to an action; the action of the decision agent k is expressed as the average action of the neighborhood and the disturbance term λa j,k The sum of the neighborhood's average actions under the current game state is the empirical distribution of actions taken by the neighborhood of decision agent j, Each element represents the average action of the neighborhood of each action; the action based on the strategy decision of the agent in the neighborhood at the last moment is averaged. Where N j represents the number of users in cell j, action a k The choice of obedience strategy in is the average action used in the last decision, and s is the current state; Each cell base station transmits the joint cell association plan to the drone to be connected; the decision agent drone obtains the state observation and generates the drone trajectory planning plan according to the policy network; in the next Δ time slots, each drone flies to the new position and transmits the position information and task queue information to the user equipment of the associated cell; User device m obtains state observations and generates user offloading scheduling actions according to the policy network; Among them, at the end of the time slot, the device calculates the time differential error of each decision agent according to the reward obtained, and updates the policy network and value network of the user equipment decision agent; after Δ time slots, the cell association plan and trajectory planning plan end; the base station calculates the time differential error of the base station decision agent according to the reward obtained, updates the policy network and value network of the base station decision agent, and generates a new cell association plan according to the state; the drone calculates the time differential error of the drone decision agent according to the reward obtained, updates the policy network and value network of the drone decision agent, and generates a new trajectory planning plan according to the state; S33, repeat the above step S32 in each time slot until convergence, and output the action selection decision as the optimal decision.
2. The method for joint dynamic decision-making of cell association, trajectory planning and unloading scheduling according to claim 1, characterized in that: The air-ground collaborative network includes a ground network consisting of J cells and a drone network consisting of K drones; each base station j is only connected to M j A fixed ground user equipment is associated with such a set of user equipment. And the user equipment sets of each cell do not have any intersection j∈{1,…,J}; User equipment m j,i represents the i-th user in cell j. In cell j, user equipment m j,i satisfy Base station j and its associated user equipment set Together they constitute cell j. The user set of this multi-cell network is expressed as Each cell has at most one drone providing service. If cell j is associated with the kth drone, then user equipment m in cell j i When a request to offload to a drone is generated, it can only be offloaded to drone k, and the cell-associated scheduling is deployed at base station j of the corresponding cell.
3. The method for joint dynamic decision-making of cell association, trajectory planning and unloading scheduling according to claim 2 is characterized in that: The modeling process for hierarchical decision making includes the following steps: In the cell association layer, the state observed by the base station is expressed as Where Q UAV,t-Δ Represents the position matrix composed of all drones; u BS,t and u UAV,t Represent the mission data backlog values of the base station and the drone respectively; represents the average data backlog of all users in cell j in time slot t, The cell association scheme of cell j is expressed as Each base station j decides which drone to associate with. , other drones cannot be selected as edge computing nodes by users in cell j, and the joint solution executed by all agents at the top level is expressed as In the trajectory planning layer, the state observed by the drone Defined as Flight direction in represents the flight direction set; the joint trajectory planning scheme of K UAVs is expressed as In the offloading scheduling layer, the state observed by the user in time slot t is Expressed as where q UAV,t and D -k,t Indicates the location and data accumulation information of the drone connected to cell j, and the decision Given; D j,t Indicates the task data backlog of the base station in cell j; u UE,j,t represents the data accumulation information of each user in cell j, v j,m,t represents the data size information of the head of the local FIFO task queue; the joint unloading scheduling action determined by each user equipment in cell j in time slot t is expressed as The elements The actions performed on behalf of each user include four operations: processing data locally, unloading a set of task data at the head of the team to the base station, unloading a set of task data at the head of the team to the drone, and unloading two sets of task data at the head of the team to the drone.
4. The method for joint dynamic decision-making of cell association, trajectory planning and unloading scheduling according to claim 1, characterized in that: The reward function of the decision agent j that performs cell association is expressed as: Where Y j,m,i is the reward of community user m, Y -k,i is the reward of drone k, Y j,i is the reward of base station j, Z t is the construction credit allocation term, is the sum of the differences in the data backlog of different cell tasks, expressed as The reward function of the decision agent k that performs the UAV trajectory planning is expressed as: The reward function of agent m that performs user offloading scheduling is expressed as: In the formula, Z t (u UE,j,t ) is the sum of the differences in task data backlogs of different users in cell j, which can be expressed as Where α is the relevant constant of the user reward function, D j,m,t It is represented by the local task queue observed at the end of time slot t, that is, the backlog of data, ΔD j,m,t Indicates the change in the data backlog.
5. The method for joint dynamic decision-making of cell association, trajectory planning and unloading scheduling according to claim 1, characterized in that: In step S32, when base station j performs cell association decision, it uses the average solution of neighboring base stations The actor generates new solutions through the policy network The strategy representation method of kernel function output is Neighborhood averaging scheme As an extension of the state, they serve as the input of the policy network, w A,j is the weight vector of the actor, is the kernel vector of the actor dictionary under Gaussian kernel mapping, feature The kernel vector of the actor dictionary under Gaussian kernel mapping is: in represents the size of the actor dictionary of base station j, σ s,j and σ a,j They represent the state of base station j and the core parameters corresponding to the scheme respectively.
6. The method for joint dynamic decision-making of cell association, trajectory planning and unloading scheduling according to claim 1, characterized in that: In step S32, the strategy of the kernel function output of drone k is expressed as feature Actor Core Vector The value of each element in is in, represents the size of the drone k-actor dictionary, σ s,k is the kernel parameter corresponding to the state of UAV k.
7. The method for joint dynamic decision-making of cell association, trajectory planning and unloading scheduling according to claim 1, characterized in that: For the agent that unloads the scheduling layer, the actor and judge initialize their own feature dictionaries and The strategy output by the kernel function of user device m is expressed as For agent m that performs offloading scheduling, the characteristics of the actor Nuclear Vector The value of each element in is The judgement feature is Nuclear Vector The values of the elements in are: in and represents the size of the actor and judge dictionary of user device m, σ s,j,m and σ a,j,m They represent the state of user device m and the core parameters corresponding to the action respectively.
8. The method for joint dynamic decision-making of cell association, trajectory planning and unloading scheduling according to claim 1, characterized in that: In step S32, the updating of the policy network and the value network of the decision agent includes the following steps: S321, at the end of time slot t, user device m obtains a reward, calculates the time difference error of the user device decision agent, updates the user device policy network parameters under multiple decision agents and the user device value network parameters under multiple decision agents, and performs feature similarity detection to determine whether to add new features to the dictionary of the decision agent user device m; Specifically, the time difference error of the user device decision agent at time t is equal to the product of the reward of the user device decision agent at time t+1 plus the correlation coefficient and the state value function of the user device decision agent at time t+1 minus the action value function of the user device decision agent at time t, where the correlation coefficient is denoted as γ, expressed as: User equipment policy network parameters w under multiple decision agents A,j,m The updating method is the sum of the original policy network parameters and the correlation coefficient, the differential of the policy network parameters, the logarithm of the kernel function output strategy of the user device under multiple decision agents, and the time difference error of the user device decision agent. The correlation coefficient is denoted as α, which satisfies: The judge network parameter w of user equipment m C,j,m The update method is the sum of the original value network parameters and the correlation coefficient, the time difference error of the user equipment decision agent, and the product of the kernel vector. The correlation coefficient is denoted as α, which satisfies: S322, at the end of time slot t+Δ, base station j obtains the reward, calculates the time difference error of the base station decision agent, updates the base station strategy network parameters under multiple decision agents and the base station value network parameters under multiple decision agents, performs feature similarity detection, and determines whether to add the new feature to the dictionary of base station j; The time difference error of the base station decision agent at time t is equal to the product of the reward of the base station decision agent at time t+Δ plus the correlation coefficient and the state value function of the base station decision agent at time t+Δ minus the action value function of the base station decision agent at time t. The correlation coefficient is denoted as γ, which is expressed as: Under the guidance of the time difference error of the base station decision agent, the update method of the base station strategy network parameters under multiple decision agents is the sum of the original strategy network parameters and the product of the correlation coefficient, the differential of the strategy network parameters, the logarithm of the kernel function output strategy of the base station under multiple decision agents, and the time difference error of the base station decision agent. The correlation coefficient is denoted as α, which satisfies: Under the guidance of the time difference error of the base station decision agent, the update method of the base station value network parameters under multiple decision agents is the sum of the original value network parameters and the product of the correlation coefficient, the time difference error of the base station decision agent and the kernel vector. The correlation coefficient is denoted as α, which satisfies: S323, drone k obtains a reward, calculates the time difference error of the drone decision agent, the drone strategy network parameters under multiple decision agents, and the drone value network parameters under multiple decision agents, performs feature similarity detection, and determines whether to add new features to the dictionary of drone k; When each drone executes a joint plan After receiving the reward after Δ time slots, the time difference error of the value network update of the drone decision agent is equal to the product of the reward of the drone decision agent at time t+Δ plus the correlation coefficient and the state value function of the drone decision agent at time t+Δ minus the action value function of the drone decision agent at time t. The correlation coefficient is denoted as γ, which is expressed as: The update method of the UAV strategy network parameters in the trajectory planning layer under multiple decision agents is the sum of the original strategy network parameters and the correlation coefficient, the differential of the strategy network parameters, the logarithm of the kernel function output strategy of the UAV under multiple decision agents, and the time difference error of the UAV decision agent. The correlation coefficient is denoted as α, which satisfies: The updating method of the value network parameters of the trajectory planning layer of the drone under multiple decision agents is the sum of the original value network parameters and the product of the correlation coefficient, the time difference error of the drone decision agent and the kernel vector. The correlation coefficient is denoted as α, which satisfies:
9. The method for joint dynamic decision-making of cell association, trajectory planning and unloading scheduling according to claim 8, characterized in that: Feature similarity detection refers to comparing the new feature with the original features in the dictionary one by one, calculating the difference between the two, defining the difference function threshold between the current new feature and the original feature in the dictionary, and if the difference is greater than the threshold, the current new feature is added to the dictionary set; If the difference is less than the threshold, the current dictionary set is maintained unchanged.
