Task unloading method and device based on space-air-ground integrated network

Abstract

The invention relates to the field of task unloading, in particular to a task unloading method and device based on an air-space-ground integrated network, computer equipment and a storage medium. The performance gain brought to the air-space-ground integrated network by the transmitting power of the Internet of Things equipment and the real-time change of the time slot duration allocated to the Internet of Things equipment by the unmanned aerial vehicle is fully considered, and the communication quality and the resource utilization rate are optimized by combining different transmitting powers required by the Internet of Things equipment at different positions and different distances. The power of the Internet of Things equipment can be adjusted according to actual requirements, the unmanned aerial vehicle can dynamically adjust the flight condition and adapt to the dynamically changing system environment, the performance of the space-air-ground integrated network is improved through efficient resource management, the energy loss of the Internet of Things equipment and the unmanned aerial vehicle is reduced, and the task unloading efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of task offloading, and in particular to a task offloading method, device, computer equipment and storage medium based on an air-ground integrated network. Background Art

[0002] With the rapid development and popularization of 5G technology, more and more computationally intensive and latency-sensitive applications are running on IoT devices. However, relying solely on ground wireless networks may not be able to meet the computing needs of IoT devices, especially in remote areas where communication facilities are lacking. Therefore, the integrated air-ground-space network is considered to be a key framework for providing computing offloading services for IoT devices. In this integrated network architecture, how to make optimal offloading decisions and efficiently allocate communication network resources to achieve the goal of reducing system latency and energy consumption is an important research direction in related fields.

[0003] In the existing research on offloading and resource allocation strategies for integrated air-ground-space networks, balancing the latency and energy consumption caused by computational offloading remains an important challenge. Over-reliance on offloading will increase communication latency, while too much local computation will increase the energy consumption of IoT devices. In addition, many studies on resource allocation methods for integrated air-ground-space networks usually assume that the transmit power of IoT devices and the duration of time slots allocated by drones to IoT devices are fixed values. However, fixed transmit power cannot adapt to the dynamically changing network environment, resulting in the inability of IoT devices to adjust power according to actual needs, thereby reducing the overall performance and efficiency of the system. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a task unloading method, device, computer equipment and storage medium based on an integrated air-space-ground network, which fully considers the performance gain brought to the integrated air-space-ground network by the real-time changes in the transmission power of the Internet of Things devices and the duration of the time slots allocated to the Internet of Things devices by the drones, and combines the different transmission powers required by Internet of Things devices at different locations and distances to optimize the communication quality and resource utilization, so that the Internet of Things devices can adjust the power according to actual needs, the drones can dynamically adjust the flight conditions, adapt to the dynamically changing system environment, improve the performance of the integrated air-space-ground network through efficient resource management, reduce the energy loss of Internet of Things devices and drones, and improve the efficiency of task unloading.

[0005] In a first aspect, an embodiment of the present application provides a task offloading method based on an air-space-ground integrated network, wherein the air-space-ground integrated network includes a plurality of Internet of Things devices, a drone, and a plurality of satellites; the method includes the following steps:

[0006] Obtaining the state space information of the current time slot of the air-ground integrated network and the preset task offloading model;

[0007] Inputting the state space information of the current time slot into the task offloading model to obtain the action space information of the current time slot, and controlling the air-ground integrated network to perform task offloading according to the action space information of the current time slot to obtain the state space information of the next time slot;

[0008] Perform reward calculation according to the state space information and action space information of the current time slot to obtain reward information of the current time slot; combine the state space information, action space information, reward information of the current time slot and the state space information of the next time slot to construct a training information combination of the current time slot;

[0009] Repeatedly construct training information combinations to obtain training information combinations of several time slots, update the model parameters of the task offloading model according to the training information combinations of several time slots, obtain the target task offloading model, and configure the target task offloading model in several IoT devices and drones to perform task offloading execution operations.

[0010] In a second aspect, an embodiment of the present application provides a task offloading device based on an air-ground integrated network, including:

[0011] A data acquisition module, used to obtain the state space information of the current time slot of the air-ground integrated network and a preset task offloading model;

[0012] A task offloading module is used to input the state space information of the current time slot into the task offloading model, obtain the action space information of the current time slot, control the air-ground integrated network to perform task offloading according to the action space information of the current time slot, and obtain the state space information of the next time slot;

[0013] A training information combination construction module is used to calculate the reward according to the state space information and action space information of the current time slot to obtain the reward information of the current time slot; the state space information, action space information, reward information of the current time slot and the state space information of the next time slot are combined to construct the training information combination of the current time slot;

[0014] The model parameter updating module is used to repeatedly execute the task offloading module and the training information combination building module to obtain the training information combination of several time slots, update the model parameters of the task offloading model according to the training information combination of several time slots, obtain the target task offloading model, and configure the target task offloading model in several IoT devices and drones to perform task offloading execution operations.

[0015] In a third aspect, an embodiment of the present application provides a computer device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the task offloading method based on the integrated air-space-ground network as described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the task offloading method based on the integrated air-ground-space network as described in the first aspect.

[0017] In an embodiment of the present application, a task offloading method, apparatus, computer equipment and storage medium based on an air-space-ground integrated network are provided, which fully considers the performance gain brought to the air-space-ground integrated network by the real-time changes in the transmission power of the Internet of Things devices and the duration of the time slots allocated to the Internet of Things devices by the drones, and combines the different transmission powers required by the Internet of Things devices at different locations and distances to optimize the communication quality and resource utilization, so that the Internet of Things devices can adjust the power according to actual needs, the drones can dynamically adjust the flight conditions, adapt to the dynamically changing system environment, improve the performance of the air-space-ground integrated network through efficient resource management, reduce the energy loss of the Internet of Things devices and drones, and improve the efficiency of task offloading.

[0018] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An application environment of a task offloading method based on an air-ground integrated network provided in one embodiment of the present application;

[0020] Figure 2 A flowchart of a task offloading method based on an air-ground integrated network provided by an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the process of S2 in a task offloading method based on an air-ground integrated network provided in one embodiment of the present application;

[0022] Figure 4 A schematic diagram of the process of S3 in a task offloading method based on an air-ground integrated network provided in one embodiment of the present application;

[0023] Figure 5 A schematic diagram of the process of S4 in a task offloading method based on an air-ground integrated network provided in one embodiment of the present application;

[0024] Figure 6A schematic diagram of the process of S42 in a task offloading method based on an air-ground integrated network provided in one embodiment of the present application;

[0025] Figure 7 A flowchart of S421 in a task offloading method based on an air-ground integrated network provided in one embodiment of the present application;

[0026] Figure 8 A schematic diagram of the structure of a task offloading device based on an air-ground integrated network provided by an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of the structure of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0029] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0030] It should be understood that, although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" / "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0031] The executor of the task offloading method based on the air-space-ground-integrated network is the offloading device of the task offloading method based on the air-space-ground-integrated network (hereinafter referred to as the offloading device). The offloading device can be implemented by software and / or hardware, and the task offloading method based on the air-space-ground-integrated network can be implemented by software and / or hardware. The offloading device can be composed of two or more physical entities, or it can be composed of one physical entity. The hardware pointed to by the offloading device essentially refers to computer equipment. For example, the offloading device can be a computer, a mobile phone, a tablet or an interactive tablet and other devices. In an optional embodiment, the offloading device can specifically be a server, or a server cluster composed of multiple computer devices.

[0032] See also Figure 1 , Figure 1 The application environment of the task offloading method based on the air-space-ground integrated network provided in one embodiment of the present application includes a plurality of IoT devices 6, a drone 2, and a plurality of satellites 1. In an area 8 lacking communication infrastructure, the IoT device 6 can establish communication with the drone 2 via Wi-Fi, and the drone 2 can act as an edge device or a wireless access point to provide edge computing services for the IoT devices 6 in the area. In addition, the satellite 1 is specifically a low-orbit satellite, which can provide cloud computing services for the IoT devices 6 in its coverage area via 5G / 4G / Wi-Fi communications.

[0033] Specifically, users move at a low speed in area 8, and each user carries an IoT device. Since these IoT devices have poor computing power and limited energy storage, the IoT devices need to choose to offload part of the generated computing tasks to drones or satellites. Drone 2 will fly to the sensing access point 5 and hover in each time slot, randomly dispatch an IoT device 6 located in its coverage range 7, and establish communication with the device through Wi-Fi channel 4. The scheduled IoT device 6 needs to adjust its own transmission power and the ratio of tasks offloaded to drones and low-orbit satellites according to the weighted sum of the system's latency and energy consumption, where IoT device 6 and satellite 1 establish communication through 5G\4G\Wi-Fi channel 3. After receiving and processing the computing tasks offloaded by IoT device 6, drone 2 will adjust the flight angle and speed, fly to the next sensing access point 5, and continue to provide computing offload services for other IoT devices 6.

[0034] See also Figure 2 , Figure 2 A flowchart of a task offloading method based on an air-ground integrated network provided in one embodiment of the present application is provided, wherein the method comprises the following steps:

[0035] S1: Obtain the state space information of the current time slot of the air-ground integrated network and the preset task offloading model.

[0036] In this embodiment, the unloading device communicates with the Internet of Things device, the drone and several satellites to obtain the state space information of the current time slot of the integrated air-space-ground network, wherein the state space information reflects the environmental information of the current time slot, including the location information of the Internet of Things device, the location information of the drone, the distance information between the drone and the Internet of Things device, the distance information between the satellite and the Internet of Things device, the amount of generated task data, the amount of remaining task data and the battery capacity information.

[0037] Specifically, the location information of the IoT device includes the location parameters of several IoT devices; the distance information between the drone and the IoT device includes the distance parameters between the drone and several IoT devices; the distance information between the satellite and the IoT device includes the distance parameters between several satellites and several IoT devices; the generated task data volume information is used to indicate the size of the task data volume generated by the IoT device in the current time slot, including the generated task data volume of several IoT devices; the remaining task data volume information is used to indicate the size of the task data volume that the allocated device needs to process in the current time slot; the battery capacity information is used to indicate the remaining battery capacity of the IoT device and the drone in each time slot, including the remaining battery capacity of several IoT devices and the drone.

[0038] The offloading device obtains a preset task offloading model, wherein the task offloading model includes a discrete action network and a continuous action network; and the action space information includes discrete action space information and continuous action space information.

[0039] S2: Input the state space information of the current time slot into the task offloading model to obtain the action space information of the current time slot, control the air-ground-space integrated network to perform task offloading according to the action space information of the current time slot, and obtain the state space information of the next time slot.

[0040] In this embodiment, the unloading device inputs the state space information of the current time slot into the task offloading model and adopts a hybrid action space deep reinforcement learning algorithm (HASDRL). HASDRL has significant advantages over traditional algorithms in solving complex optimization problems and can effectively handle systems such as the air-space-ground integrated network with large-scale state and action spaces to solve the approximate optimal solution of each parameter in the action space and obtain the action space information of the current time slot, wherein the action space information includes discrete action space information and continuous action space information, wherein the discrete action space information includes Internet of Things device scheduling information, and the Internet of Things device scheduling information is used to indicate a certain Internet of Things device called by the time slot; the continuous action space information includes Internet of Things device transmission power information, drone flight information and task offloading ratio information; the Internet of Things device transmission power information includes the transmission power of several Internet of Things devices; the drone flight information includes drone flight speed, drone flight angle and drone flight time; the task offloading ratio information includes drone task offloading ratios and satellite task offloading ratios corresponding to several Internet of Things devices.

[0041] The unloading device controls the air-ground integrated network to perform task unloading according to the action space information of the current time slot, and obtains the state space information of the next time slot.

[0042] See also Figure 3 , Figure 3 The flowchart of S2 in the task offloading method based on the air-ground integrated network provided in one embodiment of the present application includes steps S21 to S22, which are as follows:

[0043] S21: Input the IoT device position information and the drone position information in the state space information into the evaluation network in the discrete action generation network, and use a greedy algorithm to obtain the discrete action space information.

[0044] In this embodiment, the IoT device location information and the drone location information in the state space information of the unloading device are input into the evaluation network in the discrete action generation network, and the discrete action space information is obtained by using a greedy algorithm, wherein the discrete action space information includes the IoT device scheduling information, and the discrete action space information is:

[0045]

[0046] Where a is the discrete action space information, Z() is the evaluation network function, and s i is the state space information of the i-th time slot, a i is the discrete action space information of the i-th time slot, θ Z The network parameters for the evaluation network.

[0047] S22: The distance information between the UAV and the IoT device, the distance information between the satellite and the IoT device, the generated mission data volume information, the remaining mission data volume information and the battery capacity information in the state space information are input into the strategy network in the continuous action generation network to make action decisions, and obtain the continuous action space information.

[0048] The distance between the drone and each IoT device is an important basis for the drone to decide to adjust its own flight speed, flight angle and flight time. In order to reduce the transmission delay and energy consumption of the system, the drone will dispatch the IoT device closest to itself and adjust its own flight speed, flight angle and flight time according to the location parameters of the IoT device to obtain the drone flight information, where the flight time is the time it takes for the drone to fly to the corresponding sensing access point.

[0049] The remaining task data volume information in the state space information is an important basis for judging whether the task is completed. When the remaining task size is 0, it means that the drone or satellite has processed all the computing tasks generated by the IoT devices.

[0050] In this embodiment, the unloading device inputs the distance information between the drone and the IoT device, the distance information between the satellite and the IoT device, the generated task data volume information, the remaining task data volume information and the battery capacity information in the state space information into the strategy network in the continuous action generation network to make action decisions and obtain the continuous action space information, wherein the continuous action space information includes the IoT device transmission power information, the drone flight information and the task offloading ratio information.

[0051] Specifically, the unloading device uses the remaining task data volume information and battery capacity information to construct corresponding constraints, and uses the IoT device location information, drone location information, and the distance information between the satellite and the IoT device in the state space information to calculate the channel gain between the drone and the satellite and the IoT device, and then derive the data transmission rate between the two and the IoT device. The transmission power of the IoT device will affect the data transmission rate, and the size of the data transmission rate will affect the transmission delay of the system. By optimizing the transmission power of the IoT device, the transmission delay of the system can be indirectly reduced, and the transmission power information of the IoT device can be obtained.

[0052] The information on the amount of task data generated in the state space information is an important basis for optimizing the task offloading ratios of drones and satellites. The offloading device uses the amount of task data generated by the IoT device in the information on the amount of task data generated; the amount of task data unloaded from the IoT device to the drone and satellite is obtained, thereby calculating the time delay and energy consumption generated by the task offloading to the drone and satellite. Based on the calculated time delay and energy consumption, the task offloading ratio is optimized, the drone task offloading ratio and the satellite task offloading ratio corresponding to the IoT device are obtained, and the task offloading ratio information is obtained.

[0053] S3: Calculate the reward based on the state space information and action space information of the current time slot to obtain the reward information of the current time slot; combine the state space information, action space information, reward information of the current time slot and the state space information of the next time slot to construct a training information combination of the current time slot.

[0054] In this embodiment, the unloading device calculates the reward according to the state space information and the action space information of the current time slot to obtain the reward information of the current time slot.

[0055] The unloading device combines the state space information, action space information, reward information of the current time slot and the state space information of the next time slot to construct a training information combination of the current time slot. In an optional embodiment, in order to prevent the algorithm from falling into a local optimal solution during training and enable drones and IoT devices to better explore non-stationary environments, the unloading device adds Gaussian noise to all continuous actions in the action space information.

[0056] See also Figure 4 , Figure 4 The flowchart of S3 in the task offloading method based on the air-ground integrated network provided in one embodiment of the present application includes step S31, which is as follows:

[0057] S31: Obtain reward information of the current time slot according to the state space information, action space information and a preset reward calculation algorithm of the current time slot.

[0058] The reward calculation algorithm is:

[0059]

[0060] In the formula, r iis the reward information of the ith time slot, min is the minimum value function, max{} is the maximum value function, n={n(i)}, N is the number of IoT devices, n represents the nth IoT device, n(i) is the IoT device called in the ith time slot, U={θ(i),v(i)}, U is the UAV flight information, θ(i) is the UAV flight angle of the ith time slot, v(i) is the UAV flight speed of the ith time slot, R is the task offloading ratio information, is the drone task offloading ratio corresponding to the nth IoT device in the i-th time slot, is the satellite task offloading ratio corresponding to the nth IoT device in the ith time slot, P = {P n (i)}, P is the transmission power information of the IoT device, P n (i) is the transmission power of the nth IoT device in the ith time slot, T = {T fly (i)}, T fly (i) is the flight time of the UAV in the i-th time slot, δ is the weight parameter, which represents the proportion of delay and energy consumption. The latency for the task of the nth IoT device to be processed locally, The transmission delay and processing delay of offloading the task of the nth IoT device to the drone, The transmission delay and processing delay of offloading the tasks of the nth IoT device to the low-orbit satellite, The energy consumption generated by the local processing of tasks for the nth IoT device, The transmission energy and processing energy consumed by offloading the tasks of the nth IoT device to the drone, Offloading the transmission and processing energy consumption of the n-th IoT device to low-orbit satellites.

[0061] Since the duration of the time slot allocated by the drone to the IoT device consists of two parts, namely where t n(i) is the time slot duration allocated by the drone to the nth IoT device. In this embodiment, the unloading device obtains the reward information of the current time slot by jointly optimizing the user scheduling, the drone flight trajectory, the task offloading ratio, the IoT device transmission power and the flight time of the drone in each time slot according to the state space information, action space information and preset reward calculation algorithm of the current time slot to minimize the weighted sum of delay and energy consumption. The performance gain brought to the air-ground integrated network by the real-time changes of the IoT device transmission power and the time slot duration allocated by the drone to the IoT device is fully considered, and the communication quality and resource utilization are optimized by combining the different transmission powers required by IoT devices at different locations and distances, so that the IoT device can adjust the power according to actual needs, thereby improving the efficiency of task offloading.

[0062] S4: Repeatedly construct training information combinations to obtain training information combinations of several time slots, update the model parameters of the task offloading model according to the training information combinations of several time slots, obtain the target task offloading model, and configure the target task offloading model in several IoT devices and drones to perform task offloading execution operations.

[0063] In this embodiment, the offloading device repeatedly constructs a training information combination to obtain a training information combination of several time slots, and updates the model parameters of the task offloading model according to the training information combination of the several time slots to obtain a target task offloading model.

[0064] In order to improve the stability and convergence of the model training process, specifically, the unloading device adopts an experience replay mechanism, repeats steps S2 to S3, constructs initial training information combinations for several time slots, and stores them in a preset database. The unloading device obtains initial training information combinations for several time slots by randomly sampling the initial training information combinations of the minimum batch. The unloading device updates the model parameters of the task offloading model according to the training information combinations of several time slots to obtain the target task offloading model so as to execute it in the environment and obtain the maximized long-term return.

[0065] The unloading device configures the target task unloading model in several IoT devices and drones. As intelligent entities, the IoT devices and drones can obtain corresponding action space information based on the corresponding information in the state space information of the current time slot to perform task unloading execution operations. The performance gain brought to the air-space-ground integrated network by the real-time changes in the transmission power of the IoT devices and the time slot duration allocated to the IoT devices by the drones is fully considered. In addition, the IoT devices at different locations and distances require different transmission powers to optimize the communication quality and resource utilization, so that the IoT devices can adjust the power according to actual needs, and the drones will dynamically adjust the flight time to the next perception access point to adapt to the dynamically changing system environment. Through efficient resource management, the performance of the air-space-ground integrated network is improved, the energy loss of IoT devices and drones is reduced, and the efficiency of task unloading is improved.

[0066] The discrete action network consists of a value network and a target network. Figure 5 , Figure 5 The flowchart of S4 in the task offloading method based on the air-ground integrated network provided in one embodiment of the present application includes steps S411 to S413, which are specifically as follows:

[0067] S411: Obtain discrete evaluation value parameters of the current time slot according to the state space information, discrete action space information and evaluation network of the current time slot in the training information combination; obtain discrete action space information of the next time slot according to the state space information of the next time slot in the training information combination.

[0068] In this embodiment, the unloading device obtains the discrete evaluation value parameter of the current time slot according to the state space information, discrete action space information and evaluation network of the current time slot in the training information combination, wherein the discrete evaluation value parameter is: Z(s i ,a i |θ Z ).

[0069] The unloading device obtains the discrete action space information of the next time slot according to the state space information of the next time slot in the training information combination. For the specific implementation, please refer to steps S21 to S22, which will not be described in detail here.

[0070] S412: Input the reward information of the current time slot, the state space information of the next time slot, and the discrete action space information in the training information combination into the target network, and obtain the discrete target value parameters of the current time slot according to a preset discrete target value calculation algorithm.

[0071] The discrete target value calculation algorithm is:

[0072]

[0073] In the formula, z′ i is the discrete target value parameter of the i-th time slot, r i is the reward information of the i-th time slot, γ1 is the first discount factor, Z′() is the target network function, Z() is the evaluation network function, To find the maximum value function, used to select actions, s i+1 is the state space information of the i+1th time slot, a i+1 is the discrete action space information of the i+1th time slot, θ Z To evaluate the network parameters of the network, θ Z′ are the network parameters of the target network.

[0074] In this embodiment, the unloading device inputs the reward information of the current time slot, the state space information of the next time slot, and the discrete action space information in the training information combination into the target network, and obtains the discrete target value parameters of the current time slot according to a preset discrete target value calculation algorithm.

[0075] S413: According to the discrete evaluation value parameters and discrete target value parameters of the current time slot, according to a preset expected loss algorithm, an expected loss value is obtained, according to the expected loss value, the parameters of the evaluation network are updated to obtain an updated evaluation network, and a soft update strategy is adopted to update the parameters of the updated evaluation network to the target network to obtain an updated target network.

[0076] The expected loss algorithm is:

[0077] L1=E[(z′ i -z i 2 ]

[0078] In the formula, L1 is the expected loss value, E() is the expected function, z i is the discrete evaluation value parameter for the i-th time slot.

[0079] In this embodiment, the unloading device obtains an expected loss value according to the discrete evaluation value parameters and discrete target value parameters of the current time slot and a preset expected loss algorithm, updates the parameters of the evaluation network according to the expected loss value to obtain an updated evaluation network, and adopts a soft update strategy to update the parameters of the updated evaluation network to the target network to obtain an updated target network.

[0080] The continuous action network includes a value network, and the value network includes a predicted value network and a target value network, wherein the predicted value network is used to evaluate the action value of the current strategy in real time, and the target value network is used to calculate the loss of the predicted value network; please refer to Figure 6 , Figure 6 The flowchart of S4 in the task offloading method based on the air-ground integrated network provided in one embodiment of the present application includes steps S421 to S423, which are as follows:

[0081] S421: Input the state space information and continuous action space information of the current time slot in the training information combination into the prediction value network, obtain the prediction value parameters of the current time slot according to the preset prediction value calculation algorithm, and update the parameters of the policy network according to the prediction value parameters, state space information and continuous action space information of the current time slot using the policy gradient training method to obtain an updated policy network.

[0082] In this embodiment, the unloading device inputs the state space information and the continuous action space information of the current time slot in the training information combination into the prediction value network, and obtains the prediction value parameter of the current time slot according to the preset prediction value calculation algorithm, wherein the prediction value calculation algorithm is:

[0083] y i =Q(s i ,a′ i |θ Q )

[0084] In the formula, y i is the prediction value parameter of the i-th time slot, Q() is the prediction value network function, s i is the state space information of the i-th time slot, a′ i is the continuous action space information of the i-th time slot, θ Q are the network parameters of the predicted value network.

[0085] The unloading device updates the parameters of the policy network using a policy gradient training method according to the predicted value parameters of the current time slot, the state space information, and the continuous action space information to obtain an updated policy network.

[0086] See also Figure 7 , Figure 7 The flowchart of S421 in the task offloading method based on the air-ground integrated network provided in one embodiment of the present application includes step S4211, which is as follows:

[0087] S4211: Obtain a policy gradient loss value based on the predicted value parameters of the current time slot, state space information, continuous action space information, and a preset policy gradient loss algorithm; and update the parameters of the policy network based on the policy gradient loss value to obtain an updated policy network.

[0088] The policy gradient loss algorithm is:

[0089]

[0090] In the formula, is the policy gradient loss value, E s～D [] is the policy gradient function, μ() is the action generation function of the policy network, θ μ are the network parameters of the policy network, is the gradient of the continuous action space information, is the gradient of the network parameters of the policy network.

[0091] In this embodiment, the unloading device obtains a policy gradient loss value based on the predicted value parameters of the current time slot, state space information, continuous action space information and a preset policy gradient loss algorithm, and updates the parameters of the policy network based on the policy gradient loss value to obtain an updated policy network.

[0092] S422: According to the state space information of the next time slot in the training information combination and the updated strategy network, the continuous action space information of the next time slot is obtained, the state space information of the next time slot, the continuous action space information and the reward information of the current time slot are input into the target value network, and the continuous target value parameters of the current time slot are obtained according to the preset continuous target value calculation algorithm.

[0093] The continuous target value calculation algorithm is:

[0094] y′ i =r i +γ1Q′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ )

[0095] In the formula, y′ i is the continuous target value parameter of the i-th time slot, r i is the reward information of the i-th time slot, γ2 is the second discount factor, Q′() is the target value network function, s i+1 is the state space information of the i+1th time slot, μ′() is the action generation function of the updated policy network, θ μ′ is the network parameter of the updated policy network, θQ′ is the network parameter of the target value network;

[0096] In this embodiment, the unloading device obtains the continuous action space information of the next time slot based on the state space information of the next time slot in the training information combination and the updated policy network, inputs the state space information of the next time slot, the continuous action space information and the reward information of the current time slot into the target value network, obtains the continuous target value parameters of the current time slot according to the preset continuous target value calculation algorithm, updates the parameters of the policy network, and adjusts its strategy to generate the optimal action under the current state, thereby maximizing the long-term cumulative return and ensuring the stability of the strategy.

[0097] S423: According to the target value parameters and prediction value parameters of the current time slot, according to the preset mean square error loss algorithm, a mean square error loss value is obtained, and according to the mean square error loss value, the parameters of the prediction value network are updated to obtain an updated prediction value network; a soft update strategy is adopted to update the parameters of the updated prediction value network to the target value network to obtain an updated target value network.

[0098] In this embodiment, the unloading device obtains a mean square error loss value according to the target value parameter and the predicted value parameter of the current time slot according to a preset mean square error loss algorithm, and updates the parameters of the predicted value network according to the mean square error loss value to obtain an updated predicted value network, wherein the mean square error loss algorithm is:

[0099]

[0100] Where L2 is the mean square error loss value, and I is the total number of training information combinations.

[0101] The unloading device adopts a soft update strategy to update the parameters of the updated prediction value network to the target value network to obtain an updated target value network. Through the above steps, the prediction value network continuously improves its estimate of the Q value, thereby providing more accurate feedback to the prediction value network and helping the prediction value network optimize its strategy.

[0102] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of a task offloading device based on an air-ground integrated network provided by an embodiment of the present application, the device can implement all or part of the task offloading device based on the air-ground integrated network through software, hardware or a combination of both, the device 8 includes:

[0103] The data acquisition module 81 is used to obtain the state space information of the current time slot of the air-ground integrated network and the preset task offloading model;

[0104] The task offloading module 82 is used to input the state space information of the current time slot into the task offloading model, obtain the action space information of the current time slot, control the air-ground integrated network to perform task offloading according to the action space information of the current time slot, and obtain the state space information of the next time slot;

[0105] The training information combination construction module 83 is used to calculate the reward according to the state space information and action space information of the current time slot to obtain the reward information of the current time slot; combine the state space information, action space information, reward information of the current time slot and the state space information of the next time slot to construct the training information combination of the current time slot;

[0106] The model parameter updating module 84 is used to repeatedly execute the task offloading module and the training information combination building module to obtain the training information combination of several time slots, update the model parameters of the task offloading model according to the training information combination of several time slots, obtain the target task offloading model, and configure the target task offloading model in several IoT devices and drones to perform task offloading execution operations.

[0107] In an embodiment of the present application, the state space information of the current time slot of the air-space-ground integrated network and a preset task offloading model are obtained through a data acquisition module; the state space information of the current time slot is input into the task offloading model through a task offloading module to obtain the action space information of the current time slot, and the air-space-ground integrated network is controlled to perform task offloading according to the action space information of the current time slot to obtain the state space information of the next time slot; a reward calculation is performed according to the state space information and action space information of the current time slot through a training information combination construction module to obtain the reward information of the current time slot; the state space information, action space information, reward information of the current time slot and the state space information of the next time slot are combined to construct a training information combination of the current time slot; the task offloading module and the training information combination construction module are repeatedly executed through a model parameter updating module to obtain training information combinations of several time slots, and the model parameters of the task offloading model are updated according to the training information combinations of several time slots to obtain a target task offloading model, and the target task offloading model is configured in several IoT devices and drones to perform task offloading execution operations. The performance gain brought to the integrated air-space-ground network by the real-time changes in the transmission power of IoT devices and the duration of time slots allocated to IoT devices by drones is fully considered, and the different transmission powers required by IoT devices at different locations and distances are used to optimize communication quality and resource utilization. This allows IoT devices to adjust power according to actual needs, and drones to dynamically adjust flight conditions to adapt to the dynamically changing system environment. Through efficient resource management, the performance of the integrated air-space-ground network is improved, the energy loss of IoT devices and drones is reduced, and the efficiency of task offloading is improved.

[0108] Please refer to Fig. 9 , Fig. 9 The computer device 9 is a schematic diagram of a structure of a computer device provided in an embodiment of the present application. The computer device 9 includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91; the computer device may store a plurality of instructions, which are suitable for being loaded and executed by the processor 91 Figures 1 to 7 The method steps shown in the figure can be found in the specific execution process. Figures 1 to 7 The specific description shown will not be repeated here.

[0109] Among them, the processor 91 may include one or more processing cores. The processor 91 uses various interfaces and lines to connect various parts in the server, and executes various functions and processes data of the task unloading device 8 based on the integrated air-space-ground network by running or executing instructions, programs, code sets or instruction sets stored in the memory 92, and calling the data in the memory 92. Optionally, the processor 91 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programble Logic Array, PLA). The processor 91 can integrate one or more combinations of a central processing unit 91 (Central Processing Unit, CPU), an image processor 91 (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 91, and it can be implemented by a single chip.

[0110] Among them, the memory 92 may include a random access memory 92 (Random Access Memory, RAM), and may also include a read-only memory 92 (Read-Only Memory). Optionally, the memory 92 includes a non-transitory computer-readable storage medium. The memory 92 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 92 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 92 may also be optionally at least one storage device located away from the aforementioned processor 91.

[0111] The present application also provides a storage medium that can store multiple instructions, which are suitable for the processor to load and execute the above-mentioned Figures 1 to 7 The method steps shown in the figure can be found in the specific execution process. Figures 1 to 7 The specific description shown will not be repeated here.

[0112] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0113] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0114] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraint algorithm of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0115] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0116] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0118] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0119] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications to the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims and equivalent technologies of the present invention, the present invention is also intended to include these changes and modifications.

Claims

1. A task offloading method based on an air-ground integrated network, wherein the air-ground integrated network includes a plurality of Internet of Things devices, a drone and a plurality of satellites; characterized in that: The method comprises the following steps: Obtaining the state space information of the current time slot of the air-ground integrated network and the preset task offloading model; Inputting the state space information of the current time slot into the task offloading model to obtain the action space information of the current time slot, and controlling the air-ground integrated network to perform task offloading according to the action space information of the current time slot to obtain the state space information of the next time slot; Perform reward calculation according to the state space information and action space information of the current time slot to obtain reward information of the current time slot; combine the state space information, action space information, reward information of the current time slot and the state space information of the next time slot to construct a training information combination of the current time slot; Repeatedly construct training information combinations to obtain training information combinations of several time slots, update the model parameters of the task offloading model according to the training information combinations of several time slots, obtain the target task offloading model, and configure the target task offloading model in several IoT devices and drones to perform task offloading execution operations.

2. The task offloading method based on the air-ground integrated network according to claim 1 is characterized in that: The state space information includes the location information of the IoT device, the location information of the drone, the distance information between the drone and the IoT device, the distance information between the satellite and the IoT device, the amount of task data generated, the amount of remaining task data, and the battery capacity information; The task offloading model includes a discrete action network and a continuous action network; the action space information includes discrete action space information and continuous action space information; The state space information of the current time slot is input into the task offloading model to obtain the action space information of the current time slot, including the following steps: Inputting the IoT device location information and the drone location information in the state space information into the evaluation network in the discrete action generation network, and using a greedy algorithm to obtain the discrete action space information, wherein the discrete action space information includes IoT device scheduling information; The distance information between the UAV and the IoT device, the distance information between the satellite and the IoT device, the generated task data volume information, the remaining task data volume information and the battery capacity information in the state space information are input into the strategy network in the continuous action generation network to make action decisions, and obtain the continuous action space information, wherein the continuous action space information includes the IoT device transmission power information, the UAV flight information and the task offloading ratio information.

3. The task offloading method based on the air-ground integrated network according to claim 2 is characterized in that: The IoT device location information includes location parameters of several IoT devices; the distance information between the satellite and the IoT device includes distance parameters between several satellites and several IoT devices; the generated task data volume information is used to indicate the size of the task data volume generated by the IoT device in the current time slot, including the generated task data volume of several IoT devices; the remaining task data volume information is used to indicate the size of the task data volume that the unloading device needs to process in the current time slot; the battery capacity information is used to indicate the remaining battery capacity of the IoT device and the drone in each time slot, including the remaining battery capacity of several IoT devices and the drone; The IoT device scheduling information is used to indicate a certain IoT device called by a time slot; the IoT device transmission power information includes the transmission powers of several IoT devices; the UAV flight information includes the UAV flight speed, UAV flight angle and UAV flight time; the task offloading ratio information includes the UAV task offloading ratio and the satellite task offloading ratio corresponding to several IoT devices.

4. The task offloading method based on the air-ground integrated network according to claim 3 is characterized in that: The step of performing reward calculation according to the state space information and action space information of the current time slot to obtain reward information of the current time slot includes the following steps: According to the state space information, action space information and a preset reward calculation algorithm of the current time slot, the reward information of the current time slot is obtained, wherein the reward calculation algorithm is: In the formula, r i is the reward information of the ith time slot, min is the minimum value function, max{} is the maximum value function, n={n(i)}, N is the number of IoT devices, n represents the nth IoT device, n(i) is the IoT device called in the ith time slot, U={θ(i),v(i)}, U is the UAV flight information, θ(i) is the UAV flight angle of the ith time slot, v(i) is the UAV flight speed of the ith time slot, R is the task offloading ratio information, is the drone task offloading ratio corresponding to the nth IoT device in the i-th time slot, is the satellite task offloading ratio corresponding to the nth IoT device in the ith time slot, P = {P n (i)}, P is the transmission power information of the IoT device, P n (i) is the transmission power of the nth IoT device in the ith time slot, T = {T fly (i)}, T fly (i) is the flight time of the UAV in the i-th time slot, δ is the weight parameter, The latency for the task of the nth IoT device to be processed locally, The transmission delay and processing delay of offloading the task of the nth IoT device to the drone, The transmission delay and processing delay of offloading the tasks of the nth IoT device to the low-orbit satellite, The energy consumption generated by the local processing of tasks for the nth IoT device, The transmission energy and processing energy consumed by offloading the tasks of the nth IoT device to the drone, Offloading the transmission and processing energy consumption of the n-th IoT device to low-orbit satellites.

5. The task offloading method based on the air-ground integrated network according to any one of claims 2 to 4, characterized in that: The discrete action network includes a target network; The method of updating the model parameters of the task offloading model according to the combination of training information of a plurality of time slots to obtain a target task offloading model comprises the following steps: Obtaining a discrete evaluation value parameter of the current time slot according to the state space information, discrete action space information and evaluation network of the current time slot in the training information combination; obtaining discrete action space information of the next time slot according to the state space information of the next time slot in the training information combination; The reward information of the current time slot, the state space information of the next time slot, and the discrete action space information in the training information combination are input into the target network, and the discrete target value parameter of the current time slot is obtained according to the preset discrete target value calculation algorithm, wherein the discrete target value calculation algorithm is: In the formula, z′ i is the discrete target value parameter of the i-th time slot, r i is the reward information of the ith time slot, γ1 is the first discount factor, Z′( ) is the target network function, Z( ) is the evaluation network function, To find the maximum function, s i+1 is the state space information of the i+1th time slot, a i+1 is the discrete action space information of the i+1th time slot, θ Z To evaluate the network parameters of the network, θ Z′ are the network parameters of the target network. According to the discrete evaluation value parameter and the discrete target value parameter of the current time slot, according to the preset expected loss algorithm, the expected loss value is obtained, according to the expected loss value, the parameters of the evaluation network are updated to obtain an updated evaluation network, and the parameters of the updated evaluation network are updated to the target network using a soft update strategy to obtain an updated target network, wherein the expected loss algorithm is: L1=E[(z′ i -with i 2 ] In the formula, L1 is the expected loss value, E( ) is the expected function, z i is the discrete evaluation value parameter for the i-th time slot.

6. The task offloading method based on the air-ground integrated network according to any one of claims 2 to 4, characterized in that: The continuous action network includes a value network, and the value network includes a predicted value network and a target value network, wherein the predicted value network is used to evaluate the action value of the current strategy in real time, and the target value network is used to calculate the loss of the predicted value network; The method of updating the model parameters of the task offloading model according to the combination of training information of a plurality of time slots to obtain a target task offloading model comprises the following steps: The state space information and continuous action space information of the current time slot in the training information combination are input into the prediction value network, and the prediction value parameters of the current time slot are obtained according to the preset prediction value calculation algorithm. According to the prediction value parameters of the current time slot, the state space information and the continuous action space information, the policy gradient training method is adopted to update the parameters of the policy network to obtain an updated policy network, wherein the prediction value calculation algorithm is: y i =Q(s i ,a′ i |θ Q ) In the formula, y i is the prediction value parameter of the i-th time slot, Q() is the prediction value network function, s i is the state space information of the i-th time slot, a′ i is the continuous action space information of the i-th time slot, θ Q To predict the network parameters of the value network; According to the state space information of the next time slot in the training information combination and the updated policy network, the continuous action space information of the next time slot is obtained, the state space information of the next time slot, the continuous action space information and the reward information of the current time slot are input into the target value network, and the continuous target value parameters of the current time slot are obtained according to the preset continuous target value calculation algorithm, wherein the continuous target value calculation algorithm is: y′ i =r i +γ1Q′(s i+1 ,μ′(s i+1 |θ μ′ )[θ Q′ ) In the formula, y′ i is the continuous target value parameter of the i-th time slot, r i is the reward information of the ith time slot, γ2 is the second discount factor, Q′( ) is the target value network function, s i+1 is the state space information of the i+1th time slot, μ′( ) is the action generation function of the updated policy network, θ μ′ is the network parameter of the updated policy network, θ Q′ is the network parameter of the target value network; According to the target value parameter and the predicted value parameter of the current time slot, a mean square error loss value is obtained according to a preset mean square error loss algorithm, and according to the mean square error loss value, the parameters of the predicted value network are updated to obtain an updated predicted value network; a soft update strategy is adopted to update the parameters of the updated predicted value network to the target value network to obtain an updated target value network, wherein the mean square error loss algorithm is: Where L2 is the mean square error loss value, and I is the total number of training information combinations.

7. The task offloading method based on the air-ground integrated network according to claim 6 is characterized in that: The method of updating the parameters of the policy network by using a policy gradient training method according to the predicted value parameters of the current time slot, the state space information and the continuous action space information to obtain an updated policy network comprises the following steps: According to the predicted value parameter of the current time slot, the state space information, the continuous action space information and the preset policy gradient loss algorithm, a policy gradient loss value is obtained, and according to the policy gradient loss value, the parameters of the policy network are updated to obtain an updated policy network, wherein the policy gradient loss algorithm is: In the formula, is the policy gradient loss value, E s～D [ ] is the policy gradient function, μ( ) is the action generation function of the policy network, θ μ are the network parameters of the policy network, is the gradient of the continuous action space information, is the gradient of the network parameters of the policy network.

8. A task offloading device based on an air-ground integrated network, wherein the air-ground integrated network includes a plurality of Internet of Things devices, drones, and a plurality of satellites, characterized in that: include: A data acquisition module, used to obtain the state space information of the current time slot of the air-ground integrated network and a preset task offloading model; A task offloading module is used to input the state space information of the current time slot into the task offloading model, obtain the action space information of the current time slot, control the air-ground integrated network to perform task offloading according to the action space information of the current time slot, and obtain the state space information of the next time slot; A training information combination construction module is used to calculate the reward according to the state space information and action space information of the current time slot to obtain the reward information of the current time slot; the state space information, action space information, reward information of the current time slot and the state space information of the next time slot are combined to construct the training information combination of the current time slot; The model parameter updating module is used to repeatedly execute the task offloading module and the training information combination building module to obtain the training information combination of several time slots, update the model parameters of the task offloading model according to the training information combination of several time slots, obtain the target task offloading model, and configure the target task offloading model in several IoT devices and drones to perform task offloading execution operations.

9. A computer device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the task offloading method based on an air-space-ground integrated network as described in any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the task offloading method based on the air-ground integrated network as described in any one of claims 1 to 7 are implemented.

Images