Air-ground collaborative method for unmanned aerial vehicle and base station to assist ground user equipment in offloading mission data

Abstract

The present invention discloses an air-ground collaboration method for unmanned aerial vehicles and base stations to assist ground user equipment in unloading task data. At the beginning of each time slot, the unmanned aerial vehicle flies a fixed distance in a certain direction. After the unmanned aerial vehicle reaches a new position, the base station allocates an unloading scheduling mode for users in the cell. A reinforcement learning method based on a deep neural network or a kernel function is used to perform Markov modeling on the unmanned aerial vehicle trajectory planning and user unloading scheduling problems in a dynamic environment. Through information interaction with the environment, the unmanned aerial vehicle online trajectory planning is realized and the optimal strategy for user task unloading scheduling is obtained, and the unmanned aerial vehicle and base station assist ground user equipment in unloading task data. The air-ground collaboration method can be used for unmanned aerial vehicle online trajectory planning and user task unloading scheduling in a network scenario model where unmanned aerial vehicles and base stations assist ground user equipment in unloading.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to an air-to-ground collaborative method in which an unmanned aerial vehicle (UAV) and a base station assist ground user equipment in unloading mission data. Background Art

[0002] With the widespread deployment of IoT devices, a large amount of task data generated by these devices will need to be offloaded to edge computing nodes (such as base stations) with more computing power for processing. However, ground channels are affected by small-scale fading, and the transmission rate is limited. There is often a line-of-sight channel between drones and ground user devices, which provides a more reliable connection for communication, multi-user scheduling, and resource allocation. In addition, the flexibility of drones can effectively cope with communication / computing needs that change in time / space.

[0003] In the context of Multi-Access Edge Computing (MEC) in air-ground integrated networks, the latest research on offloading strategy optimization mainly focuses on the scenario where ground user devices can decide to perform computing tasks locally or offload them to drones. In fact, without cooperating with the ground network to perform MEC, these studies cannot unleash the full potential of drones in improving the quality of service of ground networks. And considering the highly dynamic environment, the presence of time-varying and unpredictable computing requirements and channel propagation, deterministic optimization cannot solve such problems.

[0004] The invention patent with publication number CN113286314A and titled “A method for deploying drone base stations and user association based on Q learning algorithm” uses Q learning method to optimize the deployment of cellular network and drone base stations and user association problems in dynamic environment to maximize the sum of the transmission rates of users in the system. However, it does not consider the user's task model and time-varying computing requirements.

[0005] The invention patent with publication number CN110381445A and name “A resource allocation and flight trajectory optimization method based on UAV base station system” gives a limited backhaul link and divides ground users into delay-tolerant users and delay-sensitive users according to their sensitivity. The UAV, as an aerial base station, receives and forwards information sent by the source to the delay-tolerant and delay-sensitive users on the ground. The minimum throughput of delay-tolerant users is maximized by jointly optimizing bandwidth, power and trajectory, making the optimization of the system more targeted. The convex relaxation method is used to turn the original non-convex problem into a convex one, and the throughput of the system is increased by alternately optimizing bandwidth, power and trajectory. However, this method is not suitable for solving problems in highly dynamic scenarios. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide an air-ground collaborative method for unloading task data of unmanned aerial vehicles and base stations to assist ground user equipment in addressing the deficiencies of the above-mentioned prior art. At the beginning of each time slot, the unmanned aerial vehicle flies a fixed distance in a certain direction; after the unmanned aerial vehicle arrives at a new location, the base station allocates an unloading scheduling mode for users in the cell; through information interaction with the environment, the unmanned aerial vehicle and the base station learn the optimal strategy for joint unmanned aerial vehicle online trajectory planning and user task unloading scheduling, so as to solve the problem of matching the time / space-varying communication / computing requirements with distributed resources in a highly dynamic air-ground integrated network. It can be used for unmanned aerial vehicle online trajectory planning and user task unloading scheduling in a network scenario model where unmanned aerial vehicles and base stations assist ground user equipment in unloading.

[0007] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0008] The air-ground collaborative method for unloading mission data by UAV and base station-assisted ground user equipment includes:

[0009] At the beginning of each time slot, the UAV flies a fixed distance in a certain direction. After the UAV reaches the new position, the base station assigns an unloading scheduling mode to the users in the cell;

[0010] A reinforcement learning method based on deep neural networks or kernel functions is used to perform Markov modeling on the UAV trajectory planning and user offloading scheduling problems in a dynamic environment. Through information interaction with the environment, online UAV trajectory planning is achieved and the optimal strategy for user task offloading scheduling is obtained. Air-ground collaboration is carried out between UAVs and base stations to assist ground user equipment in offloading task data.

[0011] To optimize the above technical solutions, the specific measures taken also include:

[0012] When using a reinforcement learning method based on a deep neural network, the above methods include:

[0013] Step 1: Get the current state of the drone, including the location of the current time slot t, and the average task data generation of each cell user in the previous time slot t-1;

[0014] Step 2: The drone selects an action, chooses a direction from a preset set of directions and flies a fixed distance;

[0015] Step 3: After the drone arrives at the new location, it collects the status of the base station, including the location of the drone, the task queue length of each user in the cell at the end of time slot t-1, the task queue length of the base station and the drone, and the average task data generation of each cell user in the previous time slot t-1;

[0016] Step 4: The base station selects an action and allocates an offloading scheduling mode to each user in the cell. The offloading scheduling mode will be executed in the next time slot t+1.

[0017] Step 5: After the next time slot t+1 unloading scheduling is completed, the UAV and the base station obtain the corresponding reward value, store the experience in the experience pool, and extract a certain amount of experience to train the deep neural network;

[0018] Step 6: Repeat steps 1 to 5 in each time slot until convergence. At this time, the action selection decision is the optimal strategy for user task offloading scheduling.

[0019] In the above step 2, the ε greedy algorithm is used for action selection. 0，t Next actions The corresponding Q value is expressed as Q(s 0，t , a0; w0), where w0 is the neural network parameter of the UAV.

[0020] In the above step 4, an improved greedy algorithm is used to select actions, specifically:

[0021] First, quantize the obtained state and establish a matrix T to record whether the quantized state-action pair is accessed;

[0022] If the quantized state-action pair has been visited in the previous time slot, the corresponding position in the matrix is ​​set to 1; otherwise, it is set to 0;

[0023] In any time slot, the base station device selects the action with the largest Q value in the current state as the unloading scheduling mode for the cell users in this time slot according to the depth Q network with probability 1-ε;

[0024] In any time slot, the base station device randomly selects an unexplored action with probability ε, that is, the action set corresponding to the value of 0 in the matrix in the current quantization state of the T matrix;

[0025] In step 4, when the obtained state is quantized, the initial quantized state set is empty. For the state of time slot t, if the distance between it and each element of all quantized states in the set is less than the corresponding threshold, the existing quantized state is used to replace the state of the current time slot;

[0026] If the distance between it and an element of all quantized states in the set is greater than the threshold, it means that the set does not contain the state, so the state is added to the quantized state set, and the T matrix is ​​updated to add a new row of zero elements.

[0027] In step 5 above, the training of the neural network includes the following steps:

[0028] Step 5.1: Initialize the neural network at the beginning of training;

[0029] Step 5.2: After each agent receives a reward in each time slot, it stores the state and action of the previous time slot t-1, the reward received in the current time slot t, and the state as experience in the experience pool;

[0030] Step 5.3: Select a certain batch of experience and use the Adam algorithm to train the neural network with the goal of minimizing the sample mean square error.

[0031] When the above-mentioned reinforcement learning method based on kernel function is adopted, the method includes:

[0032] Step (1): The UAV obtains the position of the current time slot t and the average task data generated by users in each cell in the previous time slot t-1 as the current state of the UAV;

[0033] Step (2): The drone selects an action, chooses a direction from a preset direction set and flies a fixed distance;

[0034] Step (3): After the UAV arrives at the new location, the base station collects the location of the UAV, and the task queue length of each user in the cell at the end of time slot t-1, the task queue length of the base station and the UAV, and the average task data generation amount of each user in the cell in the previous time slot t-1 are used as the status of the base station;

[0035] Step (4): The base station performs action selection and allocates an unloading scheduling mode to each user in the cell, and the unloading scheduling mode will be executed in the next time slot t+1;

[0036] Step (5): When the unloading scheduling is completed at the next time slot t+1, the UAV and the base station obtain the corresponding reward value, train the kernel function, and update the corresponding dictionary and weight vector;

[0037] Step (6): Repeat steps (1) to (5) in each time slot until convergence. At this time, the action selection decision is the optimal strategy for user task offloading scheduling.

[0038] In the above step 2, the ε greedy algorithm is used for action selection. 0，t Next actions The corresponding Q value can be expressed as Q(s 0，t , a0; w0), where w0 is the kernel function weight vector of the drone, and the Gaussian kernel function is used.

[0039] In the above step 4, an improved greedy algorithm is used to select actions, specifically:

[0040] First, the state is quantized and a matrix T is established to record whether the quantized state-action pair has been accessed. If the quantized state-action pair has been accessed in the previous time slot, the corresponding position in the matrix is ​​set to 1; otherwise, it is set to 0;

[0041] In any time slot, the base station device selects the action with the largest Q value in the current state as the unloading scheduling mode for the cell users in this time slot with probability 1-ε according to the Q function approximated by the kernel function;

[0042] In any time slot, the base station device randomly selects an unexplored action with probability ε, that is, the action set whose value in the corresponding matrix is ​​0 under the current quantization state in the T matrix.

[0043] In the above step (5), the training of the kernel function includes the following steps:

[0044] Step (5.1): Create a dictionary set to store the acquired feature information, i.e., state-action pairs;

[0045] Step (5.2): solve the kernel function of the data sample feature of the current time slot with respect to each feature in the dictionary;

[0046] Step (5.3): Update the weight coefficient of each feature in the dictionary according to the state, action and instantaneous reward of the current time slot;

[0047] Step (5.4): Multiply the kernel function obtained in step (5.2) and the weight coefficient of the corresponding feature obtained in step (5.3) and sum them to obtain the Q function that approximates the kernel function of the current time slot;

[0048] Step (5.5): Update the dictionary using an online kernel sparse method based on ALD (Approximate Linear Dependence).

[0049] In the above step (5.2), the Gaussian kernel function is used to solve the kernel function of the data sample feature of the current time slot with respect to each feature in the dictionary;

[0050] In the step (5.5), the online kernel sparse method based on ALD determines the similarity between the current data sample and the elements in the dictionary by defining a difference function. When the value of the similarity function exceeds a certain threshold, it means that the current sample contains features that the feature elements in the dictionary do not have, and the sample needs to be added to the dictionary set. Step (5.5) specifically includes the following steps:

[0051] Step (5.5.1): define the difference function at any time slot as the difference between the feature vector of the current sample and the expectation of the feature vector of the sample in the current dictionary;

[0052] For the data sample of the current time slot, calculating the difference;

[0053] Step (5.5.2): define the difference function threshold μ between the current data sample and the elements in the dictionary;

[0054] Step (5.5.3): Determine the relationship between the difference calculated in step (5.5.1) and the threshold defined in step (5.5.2). If the difference is greater than the threshold, add the current sample to the dictionary set; if the difference is less than the threshold, maintain the current dictionary set unchanged.

[0055] The present invention has the following beneficial effects:

[0056] The present invention is aimed at the multi-access edge computing scenario assisted by drones, and can effectively solve the problem of matching the time / space-varying communication / computing requirements with distributed resources in the highly dynamic air-ground integrated network. Specifically:

[0057] The present invention proposes a method for unmanned aerial vehicles (UAVs) and base stations to collaboratively provide unloading services for ground user equipment. At the beginning of each time slot, the UAV flies a fixed distance in a certain direction. After the UAV arrives at a new location, the base station allocates an unloading scheduling mode for users in the cell.

[0058] In order to obtain the optimal strategy for UAV and base station learning to jointly plan online trajectory of UAV and schedule user task offloading, it is modeled as a mutually coupled Markov decision process. A joint trajectory planning and offloading scheduling method based on deep neural network is provided. This method uses a reinforcement learning method based on deep neural network to perform Markov modeling on UAV trajectory planning and user offloading scheduling problems in dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of the method of the present invention;

[0060] Figure 2 There are two situations when uninstalling data;

[0061] Figure 3 Schematic diagram of coupled Markov decision process;

[0062] Figure 4 It is the data interaction and execution module that exists when the drone is actually deployed;

[0063] Figure 5 It is the data interaction and execution module that exists in the actual deployment of the base station;

[0064] Figure 6 This is a flow chart of the joint trajectory planning and unloading scheduling based on deep neural network of the present invention;

[0065] Figure 7 This is a flow chart of the joint trajectory planning and unloading scheduling based on the kernel function of the present invention;

[0066] Figure 8 The present invention provides a flowchart for updating a dictionary using an online kernel sparse method based on ALD. DETAILED DESCRIPTION

[0067] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0068] See also Figure 1 , an air-ground collaborative method for unloading mission data by UAV and base station-assisted ground user equipment, including:

[0069] At the beginning of each time slot, the UAV flies a fixed distance in a certain direction. After the UAV reaches the new position, the base station assigns an unloading scheduling mode to the users in the cell;

[0070] A reinforcement learning method based on deep neural networks or kernel functions is used to perform Markov modeling on the UAV trajectory planning and user offloading scheduling problems in a dynamic environment. Through information interaction with the environment, online UAV trajectory planning is achieved and the optimal strategy for user task offloading scheduling is obtained. Air-ground collaboration is carried out between UAVs and base stations to assist ground user equipment in offloading task data.

[0071] In specific implementation, when using the reinforcement learning method based on deep neural network, such as Figure 6 As shown, the method includes:

[0072] Step 1: Get the current state of the drone, including the location of the current time slot t, and the average task data generation of each cell user in the previous time slot t-1;

[0073] Step 2: The drone selects an action, chooses a direction from a preset set of directions and flies a fixed distance;

[0074] Step 3: After the drone arrives at the new location, it collects the status of the base station, including the location of the drone, the task queue length of each user in the cell at the end of time slot t-1, the task queue length of the base station and the drone, and the average task data generation of each cell user in the previous time slot t-1;

[0075] Step 4: The base station selects an action and allocates an offloading scheduling mode to each user in the cell. The offloading scheduling mode will be executed in the next time slot t+1.

[0076] Step 5: After the next time slot t+1 unloading scheduling is completed, the UAV and the base station obtain the corresponding reward value, store the experience in the experience pool, and extract a certain amount of experience to train the deep neural network;

[0077] Step 6: Repeat steps 1 to 5 in each time slot until convergence. At this time, the action selection decision is the optimal strategy for user task offloading scheduling.

[0078] In the above step 2, the ε greedy algorithm is used for action selection. 0，t Next actions The corresponding Q value can be expressed as Q(s 0，t , a0; w0), where w0 is the neural network parameter of the UAV.

[0079] In the above step 4, an improved greedy algorithm is used to select actions, specifically:

[0080] First, quantize the obtained state and establish a matrix T to record whether the quantized state-action pair is accessed;

[0081] If the quantized state-action pair has been visited in the previous time slot, the corresponding position in the matrix is ​​set to 1; otherwise, it is set to 0;

[0082] In any time slot, the base station device selects the action with the largest Q value in the current state as the unloading scheduling mode for the cell users in this time slot according to the depth Q network with probability 1-ε;

[0083] In any time slot, the base station device randomly selects an unexplored action with probability ε, that is, the set of actions (columns) with a value of 0 in the corresponding matrix under the current quantization state (row) in the T matrix;

[0084] In step 4, when the obtained state is quantized, the initial quantized state set is empty. For the state of time slot t, if the distance between it and each element of all quantized states in the set is less than the corresponding threshold, the existing quantized state can be used to replace the state of the current time slot;

[0085] If the distance between it and an element of all quantized states in the set is greater than the threshold, it means that the state is not included in the set, so the state is added to the quantized state set, and the T matrix is ​​updated to add a new row of zero elements (indicating that the actions have not been explored).

[0086] In step 5 above, the training of the neural network includes the following steps:

[0087] Step 5.1: Initialize the neural network at the beginning of training;

[0088] Step 5.2: After each agent receives a reward in each time slot, it stores the state and action of the previous time slot t-1, the reward received in the current time slot t, and the state as experience in the experience pool;

[0089] Step 5.3: Select a certain batch of experience and use the Adam algorithm to train the neural network with the goal of minimizing the sample mean square error.

[0090] In specific implementation, when using the kernel function-based reinforcement learning method, such as Figure 7 As shown, the method includes:

[0091] Step (1): The UAV obtains the position of the current time slot t and the average task data generated by users in each cell in the previous time slot t-1 as the current state of the UAV;

[0092] Step (2): The drone selects an action, chooses a direction from a preset direction set and flies a fixed distance;

[0093] Step (3): After the UAV arrives at the new location, the base station collects the location of the UAV, and the task queue length of each user in the cell at the end of time slot t-1, the task queue length of the base station and the UAV, and the average task data generation amount of each user in the cell in the previous time slot t-1 are used as the status of the base station;

[0094] Step (4): The base station performs action selection and allocates an unloading scheduling mode to each user in the cell, and the unloading scheduling mode will be executed in the next time slot t+1;

[0095] Step (5): When the unloading scheduling is completed at the next time slot t+1, the UAV and the base station obtain the corresponding reward value, train the kernel function, and update the corresponding dictionary and weight vector;

[0096] Step (6): Repeat steps (1) to (5) in each time slot until convergence. At this time, the action selection decision is the optimal strategy for user task offloading scheduling.

[0097] In the above step 2, the ε greedy algorithm is used for action selection. 0，t Next actions The corresponding Q value can be expressed as Q(s 0，t , a0; w0), where w0 is the kernel function weight vector of the drone, and the Gaussian kernel function is used.

[0098] In the above step 4, an improved greedy algorithm is used to select actions, specifically:

[0099] First, the state is quantized and a matrix T is established to record whether the quantized state-action pair has been accessed. If the quantized state-action pair has been accessed in the previous time slot, the corresponding position in the matrix is ​​set to 1; otherwise, it is set to 0;

[0100] In any time slot, the base station device selects the action with the largest Q value in the current state as the unloading scheduling mode for the cell users in this time slot with probability 1-ε according to the Q function approximated by the kernel function;

[0101] In any time slot, the base station device randomly selects an unexplored action with probability ε, that is, the set of actions (columns) with a value of 0 in the corresponding matrix under the current quantization state (row) in the T matrix.

[0102] In the above step (5), the training of the kernel function includes the following steps:

[0103] Step (5.1): Create a dictionary set to store the acquired feature information, i.e., state-action pairs;

[0104] Step (5.2): solve the kernel function of the data sample feature of the current time slot with respect to each feature in the dictionary;

[0105] Step (5.3): Update the weight coefficient of each feature in the dictionary according to the state, action and instantaneous reward of the current time slot;

[0106] Step (5.4): Multiply the kernel function obtained in step (5.2) and the weight coefficient of the corresponding feature obtained in step (5.3) and sum them to obtain the Q function that approximates the kernel function of the current time slot;

[0107] Step (5.5): Update the dictionary using the ALD-based online kernel sparsification method.

[0108] In the above step (5.2), the Gaussian kernel function is used to solve the kernel function of the data sample feature of the current time slot with respect to each feature in the dictionary;

[0109] In the above step (5.5), the online kernel sparse method based on ALD determines the similarity between the current data sample and the elements in the dictionary by defining a difference function. When the value of the similarity function exceeds a certain threshold, it means that the current sample contains features that the feature elements in the dictionary do not have, and the sample needs to be added to the dictionary set, such as Figure 8 As shown, step (5.5) specifically includes the following steps:

[0110] Step (5.5.1): define the difference function at any time slot as the difference between the feature vector of the current sample and the expectation of the feature vector of the sample in the current dictionary;

[0111] For the data sample of the current time slot, the difference is calculated;

[0112] Step (5.5.2): define the difference function threshold μ between the current data sample and the elements in the dictionary;

[0113] Step (5.5.3): Determine the relationship between the difference calculated in step (5.5.1) and the threshold defined in step (5.5.2). If the difference is greater than the threshold, add the current sample to the dictionary set; if the difference is less than the threshold, maintain the current dictionary set unchanged.

[0114] The specific analysis and examples are as follows:

[0115] In the multi-cell network studied, the drone flies at a constant altitude H and acts as an edge computing node. The ground network consists of J cells, where each base station j (satisfying j∈{1,…,J}) only communicates with M j A fixed ground user equipment is associated with such a set of user equipment. (satisfy ), and the user equipment sets of each cell do not have an intersection

[0116] Base station j and its associated user equipment set Together they constitute cell j. The user set of this multi-cell network can be expressed as

[0117] Due to the small-scale fading of the terrestrial non-line-of-sight channel, the channel gain is time-varying. Here, a block fading channel model is considered. That is, the terrestrial channel gain remains unchanged within a time slot, but varies between different time slots.

[0118] In time slot t, a base station j and user equipment m (where ) can be expressed as h 0,j,m,t .

[0119] The corresponding channel power gain where β and d j,m represent the path loss index and the distance between base station j and user equipment m respectively.

[0120] In time slot t, given device m (where ) position q m (column vector) and the drone position q UAV,t =[x t ,y t ,H], the distance d between the drone and the user device 0,m,t =|q UAV,t -q m|. And there is a line-of-sight channel between the drone and the user equipment, and the distance between the drone and the user equipment m (where ) is the channel power gain between where |h0| 2 Indicates the channel power gain when the reference distance is 1m.

[0121] In each time slot t, the drone moves in the direction The flight distance is fixed v0, where Represents a set of flight directions;

[0122] Offloading scheduling options for each user device include performing computation locally on the device and offloading task data to the drone or associated base station.

[0123] The duration of offloading or (and) computing task data is designated as τ.

[0124] Assuming that there are sufficient frequency domain channels, the offload transmissions of user equipment will not interfere with each other, and the calculation results can be returned to the users through dedicated frequency domain channels.

[0125] Therefore, the achievable unloading rate of base station j or UAV in time slot t is:

[0126]

[0127] Among them, B and P m and represent the uplink channel bandwidth, the transmit power at user equipment m, and the average noise power respectively.

[0128] When the subscript X=0, it represents the unloading rate of the UAV; X=j∈{1,…,J} represents the unloading rate of the base station.

[0129] It is assumed that each user device continuously generates computing tasks within a time slot, and the statistical characteristics of the tasks generated are unknown to the network.

[0130] User Equipment The number of task bits generated during time slot t-1 can be expressed as L j，m，t-1 .

[0131] Due to the overhead of signaling and data preparation, these task data can only be locally computed or offloaded in time slot t, and this decision is made in time slot t-1.

[0132] However, due to the continuous task data generation in the entire time slot, the unloading scheduling decision in time slot t-1 cannot obtain L j，m，t-1 knowledge.

[0133] The CPU cycle frequency of each user device and the CPU cycle frequency of the multi-access edge computing server deployed on the base station / drone are defined as f local and f MEC .

[0134] Define the processing density c, which represents the number of CPU cycles required to process 1 bit of task data.

[0135] In addition, user devices, base stations, and drones are equipped with local task queues to cache unprocessed task data, which follows the first-in-first-out rule.

[0136] If the data L generated in time slot t-1 j，m，t-1 The calculation cannot be completed in time slot t, and the remaining unprocessed data L BL，j，m，t-1 It will be stored in the task queue there and calculated first in the next time slot.

[0137] Next, we analyze the L observed in time slot t in the local computing and offloading scenarios respectively. BL,j,m，t-1 And the change of task queue length:

[0138] In the case of local computation at user device m, define D j，m，t-1 is the task queue observed at the end of time slot t-1, which contains the amount of data generated but not processed before time slot t-1.

[0139] In time slot t, due to the first-in first-out rule, these data D j，m，t-1 Will be processed first, and then the task data L j，m，t-1 .

[0140] Total processing time Δt j，m，t =c(L j，m，t-1 +D j，m，t-1 ) / f local .

[0141] If Δt j，m，t <τ, then all task data can be processed before the end of the time slot, D j，m，t =0 and L BL，j，m，t-1 =0.

[0142] Conversely, when Δt j，m，t ≥τ, some task data cannot be processed completely, and the task queue length becomes D j，m，t =D j，m，t-1 +L j，m，t-1 -τf local / c.

[0143] In this case, if D j，m，t-1 <τf local / c,L BL，j，m，t-1 =Dj，m，t ;

[0144] Otherwise, there is no time to process L j，m，t-1 , so L BL，j，m，t-1 =L j，m，t-1 .

[0145] When there is an unloading operation in time slot t, L j，m，t-1 The mission data of bits can be transmitted to the multi-access edge computing server at the drone or the base station associated with the user equipment, where the transmission time is t trans =L j，m，t-1 / R X，m，t .

[0146] If t trans >τ, the multi-access edge computing server will ignore the received data and store it in the user's local task queue for processing.

[0147] While receiving data, the multi-access edge computing server can process the data cached in its task queue. X，t-1 .

[0148] The time to process this part of the task data is defined as t pre According to t trans and t pre , analyze the relationship between L BL，j，m，t-1 There are two situations, such as Figure 2 shown.

[0149] In t trans >t pre In the scenario, all D X，t-1 All task data can be processed before the offload transmission is completed.

[0150] Therefore, the total processing time Δt MEC，m，t =t trans +L j，m，t-1 ·c / f MEC .

[0151] If Δt MEC，m，t <τ,D X，t =L BL，j，m，t-1 =0;

[0152] Otherwise, the task data L j，m，t-1 Cannot be processed, D X，t =L BL，j，m，t-1 =L j，m，t-1 -f MEC (τ-t trans ) / c.

[0153] However, at t trans ≤t preIn the scenario of X，t-1 Not yet completed.

[0154] Therefore, we need to wait for D X，t-1 After the data is processed, process L j，m，t-1 .

[0155] The total time can be expressed as Δt′ MEC，m，t =t pre +L j，m，t-1 ·c / f MEC .

[0156] If Δt′ MEC，m，t <τ,D X，t =L BL，j，m，t-1 =0.

[0157] On the contrary, for Δt′ MEC，m，t ≥τ,D X，t =L j，m，t-1 +D X，t-1 -τf MEC / c. If D X，t-1 <τf local / c,L BL，j，m，t-1 =D X，t Otherwise, L BL，j，m，t-1 =L j，m，t .

[0158] The generation, caching, and execution of computational tasks are partially modeled to show that joint trajectory planning and user offloading scheduling is a sequential decision problem. However, formulating such a problem as a Markov decision process of a single agent suffers from the curse of dimensionality, where the joint state / action space grows exponentially with the total number of base stations, user devices, and drones.

[0159] When the trajectory planning and the offloading scheduling of user equipment in each cell are represented by a Markov decision process respectively, the formulated MDP-UAV and MDP-cell j are coupled with each other.

[0160] That is, in time slot t-1, the state s of MDP-cell j is j，t-1 Contains the position of the UAV and is therefore partially affected by the MDP-UAV execution action a 0，t (i.e., the movement of the UAV). Since the reward r of the MDP-UAV 0，t+1 Contains L of each cell BL,j,m,t-1 Data information is determined by the offloading scheduling of each cell, so MDP-UAV is also affected by the decision of MDP-cell j.

[0161] The specific modeling method of coupled Markov decision process is given below:

[0162] like Figure 3-5 As shown, in MDP-UAV, given the state space The state observed by the drone agent in time slot t can be defined as where u t-1 =[U 1,t-1 ,...,U J,t-1 ] T And each element is represented by Represents the average number of task bits generated in cell j in time slot t-1.

[0163] The direction selection action determined in time slot t can be defined as

[0164] The action a 0,t is executed immediately in time slot t, resulting in a new position that remains unchanged until a 0,t+1 is executed in time slot t+1 and is therefore designated as q UAV,t+1 The immediate reward is defined as It will be affected by the task offloading / execution in time period t+1.

[0165] In MDP-cell j, the state space and action space are specified as and

[0166] The state observed by agent j is defined as where q UAV,t+1 The results come from a 0,t ; Subscript m i Indicates the index of the user device, satisfying

[0167] Here u t-1 Introducing state s j,t This is to avoid the greedy strategy of always dispatching drones to provide services for only one cell.

[0168] Determine the offloading schedule action for each user device It can be written as α j,m,t ={α L ,α D ,α B}, where α L ,α D and α B They represent performing computation locally, offloading task data to UAV, and offloading to base station j, respectively.

[0169] Then the offloading scheduling action of MDP-cell j determined in time slot t can be obtained

[0170] a j,t Executed in time period t+1.

[0171] The immediate reward in time period t+1 is expressed as

[0172] Therefore, for MDP-UAV and The optimization problem can be expressed as:

[0173] and

[0174] The strategies are defined separately and The discount factor is γ∈(0,1).

[0175] In order to solve the above-mentioned joint trajectory planning and unloading scheduling problems, a deep neural network-based method and a kernel function-based method were proposed respectively.

[0176] Joint trajectory planning and unloading scheduling method based on deep neural network:

[0177] First, initialize the neural network parameters w0, and the experience pool of each agent.

[0178] In time slot t, in order to select action a 0,t , the drone agent uses the ε-greedy strategy:

[0179] Randomly select an action with probability ε that satisfies

[0180] Otherwise, a greedy approach is used to select the action with the largest Q value, i.e.

[0181] Then, the drone immediately moves in the specified direction a 0,t Fly and reach a new location q UAV,t+1 .

[0182] At the same time, each cell agent j∈{1,…,J} observes the state And get the corresponding quantization state

[0183] Here we define a matrix T j To record whether the state-action pair has been visited.

[0184] If the mth quantized state and nth action pair is visited, [T j ] m,n =1; otherwise, it is equal to 0. Given a threshold μ q , μd and μ u , for all The quantitative state If any of the following inequalities is satisfied: ‖q UAV,t+1 -q UAV ‖>μ q ,‖d j,t -d j ‖>μ d , ‖u t-1 -u‖>μ u , it is identified as a new quantization state, that is,

[0185] in this case, and

[0186] Then, given (Determine the row index m corresponding to the state), with probability ε, from [T j ] m,n Randomly select from the action set equal to 0 (unexplored);

[0187] In addition,

[0188] After the unloading scheduling action of the cell is selected, it will be executed in the t+1 time slot.

[0189] When time slot t+1 ends, the drone agent receives the reward r 0,t+1 And the experience (s 0,t ,a 0,t ,r 0,t+1 ,s 0,t+1 ) added to the experience pool;

[0190] Cell agent j receives reward r j,t+1 And the experience (s j,t ,a j,t ,r j,t+1 ,s j,t+1 ) is added to the experience pool. Then each agent extracts N sets of experience from the experience pool to train the neural network to minimize the mean square error as follows:

[0191]

[0192]

[0193] in,

[0194] and Respectively represent the corresponding Q(s0,a0;w0) and Q(s j ,a j ;wj ) and iterates along the time slots using the Adam algorithm.

[0195] Joint trajectory planning and unloading scheduling method based on kernel function:

[0196] First, initialize 3 empty dictionaries and weight vector w0,

[0197] In time slot t, in order to select action a 0,t , the drone agent uses the ε-greedy strategy: randomly selects an action with probability ε, satisfying

[0198] Otherwise, a greedy approach is used to select the action with the largest Q value, i.e.

[0199] Then, the drone immediately moves in the specified direction a 0,t Fly and reach a new location q UAV,t+1 .

[0200] At the same time, each cell agent j∈{1,…,J} observes the state And get the corresponding quantization state

[0201] Here we define a matrix T j To record whether the state-action pair has been visited.

[0202] If the mth quantized state and nth action pair is visited, [T j ] m,n =1;

[0203] Otherwise, it is equal to 0. Given a threshold μ q , μ d and μ u , for all The quantitative state If any of the following inequalities is satisfied: ‖q UAV,t+1 -q UAV ‖>μ q ,‖d j,t -d j ‖>μ d , ‖u t-1 -u‖>μ u , it is identified as a new quantization state, that is, in this case, and

[0204] Then, given (Determine the row index m corresponding to the state), with probability ε, from [Tj ] m,n Randomly select from the action set equal to 0 (unexplored);

[0205] In addition,

[0206] After the unloading scheduling action of the cell is selected, it will be executed in the t+1 time slot.

[0207] When time slot t+1 ends, the drone agent receives the reward r 0,t+1 , and update the weight vector as follows:

[0208]

[0209] Cell agent j receives reward r j,t+1 , and update the weight vector as follows:

[0210]

[0211] In the kernel function-based algorithm, the Q value is calculated by and To approximate.

[0212] f 0,t and f j,t They are respectively N 0,t and N j,t The kernel vector of the features.

[0213] The Gaussian kernel is used in this patent.

[0214] By specifying the feature space mapping as φ(·), then f(x,x′)=φ(x) T φ(x′).f 0,t It can be expressed as:

[0215] n=1,...,N 0,t

[0216] in and Represent the drone agent decision samples and features respectively.

[0217] Similarly, f j,t It can be expressed as:

[0218] n=1,...,N j,t ,

[0219] in and Represent the decision samples and features of the cell j agent respectively. All features x of the drone agent and base station agent 0,n and xj,n , collected in dictionaries and middle.

[0220] In addition to updating w0 and w j In addition, and Adding more features to the dictionary can further improve the accuracy of action value estimation.

[0221] By performing an ALD test, the samples in time slot t are Identify as a new feature.

[0222] Given a threshold μ, if δ 0,t ≤μ This means can be approximately linearly represented by the features in the dictionary There is no need to add features to the dictionary at this time On the contrary,

[0223] akin, Updates are also made this way.

[0224] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. An air-ground collaborative method for unloading mission data by a UAV and a base station-assisted ground user equipment, characterized in that: include: At the beginning of each time slot, the UAV flies a fixed distance in a certain direction. After the UAV reaches the new position, the base station assigns an unloading scheduling mode to the users in the cell; Using a reinforcement learning method based on a deep neural network or kernel function, Markov modeling is performed on the UAV trajectory planning and user offloading scheduling problems in a dynamic environment. Through information interaction with the environment, the online trajectory planning of the UAV is realized and the optimal strategy for user task offloading scheduling is obtained. The air-ground collaboration of the UAV and the base station-assisted ground user equipment to offload task data is carried out. When a reinforcement learning method based on a deep neural network is used, the method includes: Step 1: Get the current state of the drone, including the location of the current time slot t, and the average task data generation of each cell user in the previous time slot t-1; Step 2: The drone selects an action, chooses a direction from a preset set of directions and flies a fixed distance; Step 3: After the drone arrives at the new location, it collects the status of the base station, including the location of the drone, the task queue length of each user in the cell at the end of time slot t-1, the task queue length of the base station and the drone, and the average task data generation of each cell user in the previous time slot t-1; Step 4: The base station selects an action and allocates an offloading scheduling mode to each user in the cell. The offloading scheduling mode will be executed in the next time slot t+1. Step 5: After the next time slot t+1 unloading scheduling is completed, the UAV and the base station obtain the corresponding reward value, store the experience in the experience pool, and extract a certain amount of experience to train the deep neural network; Step 6: Repeat steps 1 to 5 in each time slot until convergence. At this time, the action selection decision is the optimal strategy for user task offloading scheduling; In step 2, the ε greedy algorithm is used to select actions. 0,t Next actions The corresponding Q value is expressed as Q(s 0,t ,a0;w0), where w0 is the neural network parameter of the drone; In step 4, an improved greedy algorithm is used to select actions, specifically: First, quantize the obtained state and establish a matrix T to record whether the quantized state-action pair is accessed; If the quantized state-action pair has been visited in the previous time slot, the corresponding position in the matrix is ​​set to 1; otherwise, it is set to 0; In any time slot, the base station device selects the action with the largest Q value in the current state as the unloading scheduling mode for the cell users in this time slot according to the depth Q network with probability 1-ε; In any time slot, the base station device randomly selects an unexplored action with probability ε, that is, the action set corresponding to the value of 0 in the matrix in the current quantization state of the T matrix; In step 4, when the obtained state is quantized, the initial quantized state set is empty. For the state of time slot t, if the distance between it and each element of all quantized states in the set is less than the corresponding threshold, the existing quantized state is used to replace the state of the current time slot; If the distance between it and an element of all quantized states in the set is greater than the threshold, it means that the set does not contain the state, so the state is added to the quantized state set, and the T matrix is ​​updated to add a new row of zero elements; In step 5, the training of the neural network includes the following steps: Step 5.1: Initialize the neural network at the beginning of training; Step 5.2: After each agent receives a reward in each time slot, it stores the state and action of the previous time slot t-1, the reward received in the current time slot t, and the state as experience in the experience pool; Step 5.3: Select a certain batch of experience and use the Adam algorithm to train the neural network with the goal of minimizing the sample mean square error; When a kernel function-based reinforcement learning method is used, the method includes: Step (1): The UAV obtains the position of the current time slot t and the average task data generated by users in each cell in the previous time slot t-1 as the current state of the UAV; Step (2): The drone selects an action, chooses a direction from a preset direction set and flies a fixed distance; Step (3): After the UAV arrives at the new location, the base station collects the location of the UAV, and the task queue length of each user in the cell at the end of time slot t-1, the task queue length of the base station and the UAV, and the average task data generation amount of each user in the cell in the previous time slot t-1 are used as the status of the base station; Step (4): The base station performs action selection and allocates an unloading scheduling mode to each user in the cell, and the unloading scheduling mode will be executed in the next time slot t+1; Step (5): When the unloading scheduling is completed at the next time slot t+1, the UAV and the base station obtain the corresponding reward value, train the kernel function, and update the corresponding dictionary and weight vector; Step (6): Repeat steps (1) to (5) in each time slot until convergence. At this time, the action selection decision is the optimal strategy for user task offloading scheduling; In step 2, the ε greedy algorithm is used to select actions. 0,t Next actions The corresponding Q value can be expressed as Q(s 0,t ,a0;w0), where w0 is the kernel function weight vector of the drone, and the Gaussian kernel function is used; In step 4, an improved greedy algorithm is used to select actions, specifically: First, the state is quantized and a matrix T is established to record whether the quantized state-action pair has been accessed. If the quantized state-action pair has been accessed in the previous time slot, the corresponding position in the matrix is ​​set to 1; otherwise, it is set to 0. In any time slot, the base station device selects the action with the largest Q value in the current state as the unloading scheduling mode for the cell users in this time slot with probability 1-ε according to the Q function approximated by the kernel function; In any time slot, the base station device randomly selects an unexplored action with probability ε, that is, the action set corresponding to the value of 0 in the matrix in the current quantization state of the T matrix; In step (5), the training of the kernel function includes the following steps: Step (5.1): Create a dictionary set to store the acquired feature information, i.e., state-action pairs; Step (5.2): solve the kernel function of the data sample feature of the current time slot with respect to each feature in the dictionary; Step (5.3): Update the weight coefficient of each feature in the dictionary according to the state, action and instantaneous reward of the current time slot; Step (5.4): Multiply the kernel function obtained in step (5.2) and the weight coefficient of the corresponding feature obtained in step (5.3) and sum them to obtain the Q function that approximates the kernel function of the current time slot; Step (5.5): Update the dictionary using the ALD-based online kernel sparse method; In the step (5.2), a Gaussian kernel function is used to solve the kernel function of the data sample feature of the current time slot with respect to each feature in the dictionary; In the step (5.5), the online kernel sparse method based on ALD determines the similarity between the current data sample and the elements in the dictionary by defining a difference function. When the value of the difference function exceeds a certain threshold, it means that the current sample contains features that the feature elements in the dictionary do not have, and the sample needs to be added to the dictionary set. Step (5.5) specifically includes the following steps: Step (5.5.1): define the difference function at any time slot as the difference between the feature vector of the current sample and the expectation of the feature vector of the sample in the current dictionary; For the data sample of the current time slot, calculating the difference; Step (5.5.2): define the difference function threshold μ between the current data sample and the elements in the dictionary; Step (5.5.3): Determine the relationship between the difference calculated in step (5.5.1) and the threshold defined in step (5.5.2). If the difference is greater than the threshold, add the current sample to the dictionary set; if the difference is less than the threshold, maintain the current dictionary set unchanged.

Images