Joint optimization method, device and equipment for network task unloading strategy and resource scheduling

Abstract

The invention provides a network task unloading strategy and resource scheduling joint optimization method, device and equipment. The method comprises the following steps: determining a node path of a data stream and a weight of an unloading strategy according to position information of nodes in the unmanned aerial vehicle multi-hop networking and the unloading strategy of the data stream; according to the initial joint optimization model, reward information and a target state value of a target unloading strategy and a dominant function are determined, and state prediction information and a prediction state value of the state prediction information are determined; adjusting the initial joint optimization model according to the predicted state value, the target state value and the dominant function to obtain a target joint optimization model; and determining a target data sending action according to the target joint optimization model and the target unloading strategy, and generating a resource scheduling scheme according to the target data sending action. According to the method provided by the invention, optimization of an unloading strategy and resource scheduling is realized, the data transmission service completion time is shortened, and efficient data transmission is realized.

Description

Technical Field

[0001] The present application relates to the field of wireless communication network technology, and in particular to a network task offloading strategy and resource scheduling joint optimization method, device and equipment. Background Art

[0002] With the increasing complexity of application scenarios and demands, traditional single drones are increasingly unable to meet user needs due to power and size limitations. Drone self-organizing network technology has emerged as the times require, effectively improving the application scope and effectiveness of drones by expanding the perception capabilities of drones and the flexibility and efficiency of data transmission. The application of millimeter wave technology further enhances the ability of drone self-organizing networks, and uses its ultra-large bandwidth and directional beam characteristics to achieve efficient spatial multiplexing and data transmission.

[0003] As an extension of drone self-organizing network, air-ground multi-hop networking combines drones and ground mobile networks and uses mesh multi-hop connection to enable drones to be both data transmission users and routing forwarding nodes, transmitting end-to-end messages to ground servers for business processing, solving the problem of limited computing power of drones, improving computing speed, and saving energy consumption. Existing end-to-end performance optimization research mainly focuses on communication resource allocation problems in static scenarios, or converting multi-slot resource allocation problems into slot-by-slot resource allocation problems and using heuristic algorithms to obtain the optimal solution for each slot.

[0004] However, when studying the resource allocation problem in air-ground multi-hop networking, traditional heuristic algorithms have limitations in dealing with multi-slot joint resource allocation problems, making it difficult to find the optimal solution and resulting in poor resource allocation results. Summary of the invention

[0005] The embodiments of the present application provide a network task offloading strategy and resource scheduling joint optimization method, device and equipment to solve the problem of poor resource allocation effect when the existing method studies the resource allocation problem in air-ground multi-hop networking, thereby optimizing the offloading strategy and resource scheduling scheme to achieve efficient data transmission.

[0006] In a first aspect, an embodiment of the present application provides a network task offloading strategy and a resource scheduling joint optimization method, including:

[0007] Determine the drone multi-hop network, the location information of the nodes in the drone multi-hop network, and the offloading strategy when the data flow is transmitted from the starting node to the end node in the node;

[0008] According to the location information of the nodes in the multi-hop network of drones, determine the node path of the data flow from the starting node to the end node, the value weight of the node path, and the weight of the unloading strategy;

[0009] According to the first neural network model in the initial joint optimization model, the reward information and the target state value, and the advantage function of the target unloading strategy are determined, wherein the target unloading strategy is an unloading strategy whose weights satisfy preset weight conditions, the reward information of the target unloading strategy is determined according to the node state information of the target node in the target unloading strategy and the corresponding data sending action, the node state information of the target node and the corresponding data sending action are determined by simulating resource allocation for the target unloading strategy, the target state value is determined according to the target node state information of the receiving target node, the receiving target node is a node that receives data after the target node transmits data according to the data sending action, and the advantage function is determined according to the reward information and the target state value;

[0010] According to the second neural network model of the initial joint optimization model, the state of the data sending action is predicted to obtain the state prediction information of the receiving target node and the predicted state value corresponding to the state prediction information;

[0011] According to the predicted state value, the target state value, and the advantage function, the initial joint optimization model is adjusted to obtain the target joint optimization model;

[0012] According to the target joint optimization model and the target offloading strategy, a target data sending action and a resource scheduling scheme generated according to the target data sending action are determined.

[0013] In a possible implementation, determining a drone multi-hop network, location information of nodes in the drone multi-hop network, and an unloading strategy when a data flow is transmitted from a starting node in the node to an end node includes:

[0014] Determine the drone multi-hop networking, the location information of the nodes in the drone multi-hop networking, and the data flow offloading task in the drone multi-hop networking;

[0015] According to the data flow offloading task, determine the starting node of the data flow offloading task in the node and the end node in the multi-hop network of the drone, and the end node is the node that receives the data and performs data offloading processing;

[0016] According to the location information of the starting node, the end node and the nodes in the multi-hop network of drones, the offloading strategy is determined when the data flow is transmitted from the starting node to the end node in the node.

[0017] In a possible implementation, according to the location information of the nodes in the multi-hop network of the drone, determining the node path of the data flow from the starting node to the end node, the value weight of the node path, and the weight of the unloading strategy includes:

[0018] According to the location information of the nodes in the multi-hop network of drones, a node path set for transmitting the data stream from the starting node to the end node is determined;

[0019] Determine the node path for transmitting the data stream from the starting node to the end node and the number of routing hops in the node path according to the path distance of each node path in the node path set;

[0020] According to the node path of the data flow transmitted from the starting node to the end node and the number of routing hops in the node path, the value weight of the node path and the weight of the unloading strategy are obtained, wherein the smaller the number of routing hops in the node path, the greater the value weight.

[0021] In a possible implementation, according to the number of routing hops in the node path of the data flow transmitted from the starting node to the end node, the value weight of the node path and the weight of the offloading strategy are obtained, including:

[0022] According to the number of starting nodes and the number of end nodes in the multi-hop network of the UAV, an unloading sub-strategy for transmitting the data flow from the starting node to the end node in the unloading strategy and a node path corresponding to the unloading sub-strategy are determined;

[0023] According to the number of routing hops of the node path corresponding to the unloading sub-strategy, the value weight of the node path corresponding to the unloading sub-strategy is obtained;

[0024] The weight of the uninstallation strategy is obtained according to the value weight of the uninstallation sub-strategy and the node path corresponding to the uninstallation sub-strategy.

[0025] In a possible implementation, before determining the reward information and target state value of the target unloading strategy and the advantage function according to the first neural network model in the initial joint optimization model, the method further includes:

[0026] According to the state of the amount of data to be sent and the node label of each node in the multi-hop network of drones, the node status information is obtained;

[0027] According to the data transmission power, azimuth angle and pitch angle of each node in the multi-hop network of the drone, the data sending action is obtained;

[0028] Based on the service completion time of the data stream offloading task, a reward information function corresponding to the offloading strategy is constructed;

[0029] An initial joint optimization model is constructed based on node status information, data sending actions, reward information functions, and preset neural networks.

[0030] In a possible implementation, determining the reward information and target state value and advantage function of the target unloading strategy according to the first neural network model in the initial joint optimization model includes:

[0031] Determine a target uninstallation strategy in the uninstallation strategy according to the weight of the uninstallation strategy and a preset weight condition;

[0032] According to the first neural network model in the initial joint optimization model, a simulation process of resource allocation is performed on the target offloading strategy to determine the node state information of the target node in the target offloading strategy and the data sending action corresponding to the node state information;

[0033] Determine the target node state information of the receiving target node according to the environment information of the first neural network model and the data sending action corresponding to the node state information;

[0034] Determine reward information according to the node status information of the target node and the target node status information of the receiving target node;

[0035] According to the first neural network model, the state value of the target node state information of the receiving target node is estimated to determine the state value of the receiving target node and the target state value of the target unloading strategy;

[0036] According to the reward information and the target state value, the advantage function is obtained.

[0037] In a possible implementation, determining reward information according to the node status information of the target node and the target node status information of the receiving target node includes:

[0038] Determine the initial reward information and the transmission processing status of the data flow offloading task corresponding to the target offloading strategy according to the reward information function of the first neural network model, the node state information of the target node and the target node state information of the receiving target node;

[0039] If the data flow offloading task corresponding to the target offloading strategy has unfinished transmission processing, the node status information of the target node, the data sending action, the initial reward information, the target node status information of the receiving target node, and the transmission processing status are stored in the initial joint optimization model, and the receiving target node is used as the target node. The simulation processing of resource allocation for the target offloading strategy is repeated according to the first neural network model in the initial joint optimization model, and the steps of determining the node status information of the target node in the target offloading strategy and the data sending action corresponding to the node status information are determined until the transmission processing of the data flow offloading task corresponding to the target offloading strategy is completed and the reward information is obtained.

[0040] In a possible implementation, when there are multiple target uninstallation strategies,

[0041] According to the target joint optimization model and the target offloading strategy, a target data sending action and a resource scheduling scheme generated according to the target data sending action are determined, including:

[0042] Compare the reward information of each target unloading strategy to determine the optimal target unloading strategy among the target unloading strategies;

[0043] Determine a joint optimization model corresponding to an optimal target unloading strategy in the target joint optimization model;

[0044] According to the optimal target unloading strategy and the target number of simulations of the optimal target unloading strategy by the preset joint optimization model, the joint optimization model is iterated to obtain the optimal joint optimization model;

[0045] According to the optimal joint optimization model, resource allocation processing is performed on the optimal target offloading strategy to obtain the target data sending action corresponding to each node in the optimal target offloading strategy;

[0046] According to the target data sending action of each node in the optimal target offloading strategy, a resource scheduling plan is generated.

[0047] In a second aspect, an embodiment of the present application provides a network task offloading strategy and a resource scheduling joint optimization device, including:

[0048] A first determination module is used to determine the drone multi-hop network, the location information of the nodes in the drone multi-hop network, and the unloading strategy when the data flow is transmitted from the starting node to the end node in the node;

[0049] The second determination module is used to determine the node path of the data flow from the starting node to the end node, the value weight of the node path, and the weight of the unloading strategy according to the location information of the nodes in the multi-hop network of the drone;

[0050] A first analysis module is used to determine the reward information and target state value of the target unloading strategy, and the advantage function according to the first neural network model in the initial joint optimization model, wherein the target unloading strategy is an unloading strategy whose weight satisfies a preset weight condition, the reward information of the target unloading strategy is determined according to the node state information of the target node in the target unloading strategy and the corresponding data sending action, the node state information of the target node and the corresponding data sending action are determined by simulating resource allocation for the target unloading strategy, the target state value is determined according to the target node state information of the receiving target node, the receiving target node is a node that receives data after the target node transmits data according to the data sending action, and the advantage function is determined according to the reward information and the target state value;

[0051] A second analysis module is used to perform state prediction on the data sending action according to the second neural network model of the initial joint optimization model, and obtain state prediction information of the receiving target node and a predicted state value corresponding to the state prediction information;

[0052] An optimization module, used for adjusting the initial joint optimization model according to the predicted state value, the target state value, and the advantage function to obtain a target joint optimization model;

[0053] The third determination module is used to determine the target data sending action and the resource scheduling solution generated according to the target data sending action according to the target joint optimization model and the target offloading strategy.

[0054] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0055] Memory stores computer-executable instructions;

[0056] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0058] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0059] The network task offloading strategy and resource scheduling joint optimization method, device and equipment provided by the present application determine the weight of the transmission path of the data flow from the starting node to the end node through the node location information of the multi-hop network of the drone and the offloading strategy of the data flow; use the first neural network model to calculate the reward information and the target state value according to the state information and data sending action of the target node in the target offloading strategy, and then determine the advantage function, which can provide a key feedback signal for strategy optimization; then predict the state of the data sending action according to the second neural network model to obtain the state prediction information and predicted state value of the receiving target node, and then adjust the model in combination with the predicted state value, the target state value and the advantage function to form a target joint optimization model, so that the model can guide the data flow transmission node to make a better resource scheduling decision in a complex network environment, that is, dynamically allocate data sending actions. Therefore, by preliminarily screening the offloading strategy through weights, and then using the simulation processing of the first neural network model and the prediction of the second neural network model to further adjust the plan, the optimized offloading strategy and resource scheduling plan can be obtained, which reduces the delay of data transmission, reduces the completion time of data transmission service, and realizes efficient data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0061] Figure 1 A schematic diagram of a scenario of a network task offloading strategy and a resource scheduling joint optimization method provided in this application;

[0062] Figure 2 A flowchart of a network task offloading strategy and resource scheduling joint optimization method provided in this application;

[0063] Figure 3 A flowchart of another network task offloading strategy and resource scheduling joint optimization method provided by this application;

[0064] Figure 4 A schematic diagram of the structure of the network task offloading strategy and resource scheduling joint optimization device provided in this application;

[0065] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0066] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0068] In the prior art, for the communication resource allocation problem in static scenarios, optimization algorithms, such as linear programming, dynamic programming or heuristic algorithms, are usually used to study how to find the optimal or approximately optimal resource allocation scheme to efficiently allocate limited communication resources, such as spectrum, power and time, to meet the needs of different users and applications. For multi-time slot resource allocation, the multi-time slot resource allocation problem is decomposed into multiple single-time slot resource allocation problems, and a heuristic algorithm is used to find the optimal solution for resource allocation for each time slot, and then the optimal resource allocation schemes of each single time slot are merged to form a multi-time slot resource allocation scheme. However, decomposing the multi-time slot problem into a single time slot problem may lead to the loss of the overall optimal solution, and the optimal solution of a single time slot may not be optimal in multiple time slots. In addition, in the multi-stream service multi-hop communication scenario, the dimensions of the decision variables (such as routing, time slot, beam, power) are very high, and the heuristic algorithm has low computational efficiency, making it difficult to solve the resource allocation problem of multiple time slots jointly considered.

[0069] In order to solve the above problems, the embodiments of the present application provide a network task offloading strategy and resource scheduling joint optimization method, device and equipment, which can use the topological structure and node location information of the multi-hop network of drones to determine the weight of each node path in the data flow offloading strategy and the weight of the offloading strategy, so as to screen out the target offloading strategy according to the weight of the offloading strategy, exclude the offloading strategies that do not meet the requirements, reduce the number of offloading schemes, and improve the computing efficiency. The first neural network model is used to calculate the reward information and the target state value according to the state information and data sending action of the target node in the target offloading strategy, and then determine the advantage function, which can provide key feedback for strategy optimization. The feedback signal is then used to predict the state of the data sending action according to the second neural network model to obtain the state prediction information and predicted state value of the receiving target node. Then, the initial joint optimization model is adjusted in combination with the predicted state value, the target state value and the advantage function to form a target joint optimization model, so that the model can guide the data stream transmission nodes to make better resource scheduling decisions in a complex network environment, and dynamically allocate the transmission power, millimeter wave beam azimuth and elevation angle of each data stream routing link in multiple time slots, so as to obtain the optimized offloading strategy and resource scheduling scheme, reduce the delay of data transmission, shorten the completion time of data transmission service, and realize efficient data transmission.

[0070] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0071] Figure 1 A scenario diagram of a network task offloading strategy and resource scheduling joint optimization method provided in this application, such as Figure 1 As shown, the specific application scenario of the present application can be a multi-stream service directional multi-hop communication scenario. In this scenario, it includes air nodes composed of a large number of drone nodes and ground nodes composed of a small number of edge servers (ESs). ES provides cloud computing functions to process and calculate drone mission data. The network in this scenario can refer to a drone multi-hop network (a network composed of ground nodes and air nodes connected in a mesh multi-hop manner), which includes a set of data streams corresponding to drone missions. ES collection is Each data stream contains a set of nodes The amount of data that needs to be unloaded for each data flow W f , Data flow offloading strategy The data offloading strategy for each data flow is x f,n ∈{0,1}. During the data transmission process, the drone first needs to select the edge server to unload the data. After determining the unloading plan, the drone needs to generate a directional beam through the millimeter wave array antenna, transmit data in a directional manner, and establish an end-to-end multi-hop link so that the drone can complete the data transmission unloading task according to the node's transmission power and beam direction. Among them, the ground node can refer to a server node that is different from the drone data transmission. The ground node is used to receive and process data. For example, the ground node can be multiple ground base stations, multiple air base stations, and multiple offshore base stations.

[0072] The embodiment of the present application does not impose any special restrictions on the selection type of drone multi-hop networking, as long as the drone multi-hop networking, the location information of the nodes in the drone multi-hop networking, and the unloading strategy when the data flow is transmitted from the starting node to the end node in the node can be determined; based on the location information of the nodes in the drone multi-hop networking, the node path of the data flow from the starting node to the end node, the value weight of the node path, and the weight of the unloading strategy are determined; based on the first neural network model in the initial joint optimization model, the reward information and target state value of the target unloading strategy, as well as the advantage function are determined, wherein the target unloading strategy is an unloading strategy whose weight meets the preset weight condition, the reward information of the target unloading strategy is determined according to the node state information of the target node in the target unloading strategy and the corresponding data sending action, and the node state information of the target node and the corresponding The data sending action is determined by simulating the resource allocation for the target offloading strategy, the target state value is determined according to the target node state information of the receiving target node, the receiving target node is the node that receives the data after the target node transmits the data according to the data sending action, and the advantage function is determined according to the reward information and the target state value; according to the second neural network model of the initial joint optimization model, the state of the data sending action is predicted to obtain the state prediction information of the receiving target node and the predicted state value corresponding to the state prediction information; according to the predicted state value, the target state value, and the advantage function, the initial joint optimization model is adjusted to obtain the target joint optimization model; according to the target joint optimization model and the target offloading strategy, the target data sending action and the resource scheduling scheme generated according to the target data sending action are determined.

[0073] Among them, data flow can refer to the sequence of data transmitted through the network, which can flow between aerial nodes (such as drones) and ground nodes (such as edge servers or base stations). Multi-hop can refer to multi-hop data transmission, that is, data may pass through multiple intermediate nodes during the process of transmitting from the starting node (source node) to the end node (such as edge server or user equipment).

[0074] The data flow offloading strategy can refer to the strategy for allocating and transmitting data flows in the network (such as the selection of transmission paths and link transmission data) to optimize network performance and resource utilization. The resource scheduling scheme can refer to the scheme for allocating and using the antenna beam resources in each time slot (such as the allocation of beam type, beam direction, and beam power) to optimize the efficiency and performance of data transmission.

[0075] Figure 2 The flowchart of a network task offloading strategy and resource scheduling joint optimization method provided by this application is shown in FIG. The execution subject of this method can be a server or other server storing the network task offloading strategy and resource scheduling joint optimization method. This embodiment is not particularly limited here. Figure 2As shown, the method may include:

[0076] S201, determining a drone multi-hop network, location information of nodes in the drone multi-hop network, and an unloading strategy when a data flow is transmitted from a starting node to an end node among the nodes.

[0077] Among them, the drone multi-hop networking can be the networking after the end-to-end multi-hop link is built. Further, the data transmission rate of the link is modeled to determine the key factors affecting the transmission rate and calculate the service completion time of the end-to-end data flow offloading task, including: defining the transmission power set of the nodes of each data flow path in different time slots The pointing angle set of each data flow node antenna in each time slot in, Contains the azimuth and elevation information of the node; definition is the channel gain between millimeter wave directional links. According to the 3D directional antenna model in 3GPP, the channel gain of the millimeter wave directional antenna is related to the transmission power p of the node. f [t], antenna pointing angle ψ f [t] is related; then the transmission rate of the link in each time slot can satisfy:

[0078]

[0079] Among them, σ 2 is the noise power.

[0080] At the same time, the data flow offloading strategy will affect the number of routing nodes in the link and the distance between nodes, which will affect the transmission rate. Therefore, the transmission power, the antenna pointing angle and the data offloading strategy will affect the data transmission rate, which will affect the service completion time of the data flow offloading task.

[0081] The service completion time can refer to the time from the start of the timing of the data flow offloading task to the completion of the service, that is, after the timing starts, in each time slot, the nodes in the data flow transmit data according to the respective resource scheduling schemes of each time slot, and transmit the data to the next node. At the same time, the drone node and the edge server adopt full-duplex working mode to receive the transmission data of the previous node, temporarily store it in the cache area and transmit or process this data in the next time slot. When all data streams are unloaded to ES and all are processed by ES, it is considered that all data transmission services in the scene are completed.

[0082] Among them, in the embodiment of the present application, the method for determining the drone multi-hop networking, the location information of the nodes in the drone multi-hop networking, and the unloading strategy when the data flow is transmitted from the starting node in the node to the end node may include:

[0083] Determine the drone multi-hop networking, the location information of the nodes in the drone multi-hop networking, and the data flow offloading task in the drone multi-hop networking;

[0084] According to the data flow offloading task, determine the starting node of the data flow offloading task in the node and the end node in the multi-hop network of the drone, and the end node is the node that receives the data and performs data offloading processing;

[0085] According to the location information of the starting node, the end node and the nodes in the multi-hop network of drones, the offloading strategy is determined when the data flow is transmitted from the starting node to the end node in the node.

[0086] The data flow unloading task may include the number of data flows to be unloaded and the amount of data to be unloaded for each data flow. Since the source nodes of different data flows may be the same node or different nodes in the multi-hop network of drones, the starting node of the data flow may be a source node storing identification information of different data flows, and the terminal node in the multi-hop network of drones may include multiple nodes. The unloading strategy may refer to a strategy set when the data flow is transmitted from the starting node to each different terminal node.

[0087] In one example, there are three data flows f1, f2, f3 and two offloading servers n1 and n2 in the network. The offloading strategy of the data flow can be Furthermore, the unloading strategy may be x={1,0; 1,1; 0,1}, x={0,1; 0,1; 1,0}, where x f,n =1 indicates that the data flow f is unloaded to servers n, x f,n =0 indicates that the data flow f is not offloaded to the server n.

[0088] S202: Determine the node path of the data flow from the starting node to the end node, the value weight of the node path, and the weight of the offloading strategy according to the location information of the nodes in the multi-hop network of the drone.

[0089] Among them, before determining the node path for transmitting the data flow from the starting node to the end node, it is also necessary to set constraints on the offloading strategy to exclude strategies that do not meet the actual strategy allocation situation. The constraints can be set according to the factors that affect the data transmission rate, and the influencing factors can be used as optimization variables to constrain the allocation of strategies.

[0090] In one example, the constraints may be:

[0091]

[0092] Among them, T service The service completion time for offloading tasks for end-to-end data flows; M nC is the maximum number of drones that can be served simultaneously by each ES; min is the minimum throughput requirement in the system; P max is the maximum transmission power of the UAV in each time slot; φ 3dB and θ 3dB Represents the main lobe width of the directional beam; constraints C1 and C2 are data flow unloading constraints, ensuring that each data flow can be unloaded to at most one ES; C3 is the maximum service number constraint of ES; C4 is the throughput constraint; C5 is the power constraint; C6 and C7 are antenna pointing angle constraints.

[0093] In this step, since there may be multiple paths for the node path transmitted from the starting node to the end node in the multi-hop network of drones, the minimum distance routing algorithm can be used to calculate the path length from the starting node to the end node of each data flow f based on the node location information to obtain the node path corresponding to the shortest path length, and according to the number of routing hops of the node path, the node path (i.e., the unloading strategy selected by the data flow) is assigned a value weight, and then the value weight of the node path of each data flow transmitted from the starting node to the end node is weighted and summed with the unloading strategy of each data flow to obtain the weight of the unloading strategy corresponding to the whole data flow unloading task. The target unloading strategy can be screened out by the weight of the unloading strategy, which reduces the complexity of the subsequent unloading strategy and resource scheduling joint optimization calculation.

[0094] The hop count refers to the number of nodes that a data packet passes through when it starts from a source node, passes through multiple intermediate nodes in the network, and finally reaches the destination. The hop count of the data packet increases by one each time it passes through a node.

[0095] Among them, in the embodiment of the present application, according to the location information of the nodes in the multi-hop network of the drone, the method for determining the node path of the data flow transmitted from the starting node to the end node, the value weight of the node path, and the weight of the unloading strategy may include:

[0096] According to the location information of the nodes in the multi-hop network of drones, a node path set for transmitting the data stream from the starting node to the end node is determined;

[0097] Determine the node path for transmitting the data stream from the starting node to the end node and the number of routing hops in the node path according to the path distance of each node path in the node path set;

[0098] According to the node path of the data flow transmitted from the starting node to the end node and the number of routing hops in the node path, the value weight of the node path and the weight of the unloading strategy are obtained, wherein the smaller the number of routing hops in the node path, the greater the value weight.

[0099] Optionally, the value weight of the node path may be obtained by assigning value weights of N, N-1, ..., 1 in order from small to large according to the number of routing hops.

[0100] In one example, the offloading strategy corresponding to the entire data flow offloading task is X match =(x f,n ), whose size is F×N, where N represents the number of ESs, F represents the number of data flows, and x of each data flow offloading strategy f,n The value satisfies the constraints C1-C4. At the same time, define a value matrix V = (v f,n ), size is F×N, where v f,n ∈{1,2,…,N} represents the value weight corresponding to the node path of each data flow from the starting node to the end node. Multiplying the unloading strategy of each data flow with the corresponding value weight, the value matrix of the entire unloading strategy is obtained. Then, the value weight of the node path of each data flow from the starting node to the end node is summed with the offloading strategy of each data flow, and the weight of the offloading strategy corresponding to the entire data flow offloading task is obtained as Among them, due to so Where min(NM n ,F) is the number of data flows participating in data transmission in the entire data flow offloading task.

[0101] In the embodiment of the present application, according to the number of routing hops in the node path of the data flow transmitted from the starting node to the end node, the method of obtaining the value weight of the node path and the weight of the offloading strategy may include:

[0102] According to the number of starting nodes and the number of end nodes in the multi-hop network of the UAV, an unloading sub-strategy for transmitting the data flow from the starting node to the end node in the unloading strategy and a node path corresponding to the unloading sub-strategy are determined;

[0103] According to the number of routing hops of the node path corresponding to the unloading sub-strategy, the value weight of the node path corresponding to the unloading sub-strategy is obtained;

[0104] The weight of the uninstallation strategy is obtained according to the value weight of the uninstallation sub-strategy and the node path corresponding to the uninstallation sub-strategy.

[0105] In one example, there are three data flows f1, f2, f3 and two offloading servers n1 and n2 in the network. The offloading strategy X is known. match = {1,0; 0,1; 1,0}, the value weight of the node path of each data flow obtained according to the routing hop count is V = {3,1; 2,4; 1,2}. Then the value matrix V of the entire offloading strategy ismatch ={3,0; 0,4; 1,0}, the weight V(X match )=3+4+1=8.

[0106] S203. Determine the reward information and target state value, and advantage function of the target unloading strategy according to the first neural network model in the initial joint optimization model, wherein the target unloading strategy is an unloading strategy whose weights satisfy preset weight conditions, the reward information of the target unloading strategy is determined according to the node state information of the target node in the target unloading strategy and the corresponding data sending action, the node state information of the target node and the corresponding data sending action are determined by simulating resource allocation for the target unloading strategy, the target state value is determined according to the target node state information of the receiving target node, the receiving target node is a node that receives data after the target node transmits data according to the data sending action, and the advantage function is determined according to the reward information and the target state value.

[0107] The preset weight condition is used to screen out the offloading strategies with too low weights and those that do not meet the constraint conditions, thereby narrowing the scope of offloading strategy optimization and improving calculation efficiency.

[0108] In one example, the preset weight condition may satisfy:

[0109]

[0110] in, is the set of all uninstallation policies to be screened, V min is the minimum acceptable value.

[0111] In this step, the initial joint optimization model can be composed of two neural network models, wherein the first neural network model can be used to generate the actual actions taken and store the generated data in the experience replay pool; the second neural network model can learn directly from the experience replay pool, update the action allocation strategy and model parameters, and then synchronize the updated strategy and parameters to the first neural network model to ensure that the strategies of the two networks remain consistent, thereby ensuring the stability of the learning process, improving learning efficiency, and allowing the network to find a balance between actual action exploration and using data to train model parameters. Optionally, the initial joint optimization model can use PPO combined with GAE's deep reinforcement learning algorithm to train two actor-critic (AC) neural networks. In reinforcement learning, the AC network is an algorithm commonly used for policy optimization. It includes two main components: actor and critic. The actor is responsible for generating actions, while the critic is responsible for evaluating the value of these actions.

[0112] In the embodiment of the present application, before determining the reward information and target state value of the target unloading strategy and the advantage function according to the first neural network model in the initial joint optimization model, the method may further include:

[0113] According to the state of the amount of data to be sent and the node label of each node in the multi-hop network of drones, the node status information is obtained;

[0114] According to the data transmission power, azimuth angle and pitch angle of each node in the multi-hop network of the drone, the data sending action is obtained;

[0115] Based on the service completion time of the data stream offloading task, a reward information function corresponding to the offloading strategy is constructed;

[0116] An initial joint optimization model is constructed based on node status information, data sending actions, reward information functions, and preset neural networks.

[0117] Since the target offloading strategy is already an offloading strategy that satisfies some constraints (such as C1-C4), when constructing the initial joint optimization model, only other constraints (such as C5-C7) need to be considered, that is, the impact of the distribution of transmit power and antenna pointing angle on the offloading strategy and resource scheduling, which reduces the complexity of the joint optimization problem of offloading strategy and resource scheduling.

[0118] In one example, the offloading strategy and resource scheduling problem within a period of time can be modeled as a Markov decision process. A node is selected in the network as a centralized controller. The centralized controller has the state information of all nodes and makes task offloading, power allocation, and beam scheduling decisions based on the state information and location information of all nodes in each time slot to achieve the optimization goal of minimizing the service completion time. In the Markov decision process, the problem is equivalently represented as a five-tuple: (S, A, R, P, γ), where S represents the state space, A represents the action space, R represents the reward function, P represents the state transition probability, and γ represents the discount factor.

[0119] Since each data flow in each offloading strategy selects different terminal nodes to offload data, the relay nodes and number of nodes that the data flow passes through are different, and the link beam direction, data transmission rate and interference to other links of each node path are also different. It can be seen that for each different offloading strategy, the characteristics of the link are dynamically changing. Therefore, in this step, for the design of (S, A, R), the node state information s can be divided into two parts, s1 is used to represent the data volume state of each node in the data flow, and s2 represents the node label of each node; the transmission power, azimuth and pitch angle of each node are used as the data sending action a of the node in the Markov decision process; the service completion time of the data flow offloading task is used as the objective function (that is, the purpose of offloading strategy and resource scheduling optimization), and a specific single-step reward information function is given. After all the data flows in the offloading strategy are transmitted and processed, the single-step reward information function is summed to obtain the reward information r of the offloading strategy. By continuously using the model to train the agent (air node) and let the reward information constantly tend to the maximum value, the optimization goal can be achieved - minimizing the end-to-end service completion time.

[0120] In one example, among all the offloading strategies, the number of nodes that the longest data flow passes through is L, and the number of data flows that need to be offloaded is F, then the state s∈S can be expressed as:

[0121]

[0122] in, It represents the amount of data to be transmitted by the i-th node of the current f-th data stream in time slot t. In order to fix the dimension of the state space, 0 is added after the node of each data stream to ensure that the dimension of each data link in the state space remains L. At this time, for all offloading strategies, the dimension of state s1 can be kept as FL, that is, for all strategies, the number of nodes considered is FL. The node label that represents the i-th node of the current f-th data stream in the actual scene is used to distinguish the nodes that the data stream passes through. The size of s1 is the same as s2, and the dimension is FL. Therefore, the dimension of the state space S is 2FL.

[0123] The data sending action of each node can be expressed as:

[0124]

[0125] Combining the actions of all nodes, the action set of the unloading strategy can be expressed as:

[0126]

[0127] The single-step reward information function can be expressed as:

[0128]

[0129] The reward information function of the uninstallation strategy can be expressed as:

[0130]

[0131] Among them, t max It represents the maximum time to complete data transmission and processing in all data streams in the last time slot. As long as there are still data streams that have not been transmitted, the reward information r will continue to accumulate negative values, and after the data transmission is completed, a larger positive reward will be given, which can encourage the agent to learn in the direction of completing the task as quickly as possible.

[0132] Among them, in the embodiment of the present application, according to the first neural network model in the initial joint optimization model, the method for determining the reward information and target state value of the target unloading strategy and the advantage function may include:

[0133] Determine a target uninstallation strategy in the uninstallation strategy according to the weight of the uninstallation strategy and a preset weight condition;

[0134] According to the first neural network model in the initial joint optimization model, a simulation process of resource allocation is performed on the target offloading strategy to determine the node state information of the target node in the target offloading strategy and the data sending action corresponding to the node state information;

[0135] Determine the target node state information of the receiving target node according to the environment information of the first neural network model and the data sending action corresponding to the node state information;

[0136] Determine reward information according to the node status information of the target node and the target node status information of the receiving target node;

[0137] According to the first neural network model, the state value of the target node state information of the receiving target node is estimated to determine the state value of the receiving target node and the target state value of the target unloading strategy;

[0138] According to the reward information and the target state value, the advantage function is obtained.

[0139] Among them, the target unloading strategy obtained by screening according to the preset weight condition may include multiple strategies. When there are multiple target unloading strategies, for each target unloading strategy, neural network parameter training is performed separately according to the initial joint optimization model to obtain the optimized neural network parameters corresponding to each target unloading strategy, that is, different target unloading strategies have different reward information and target joint optimization models obtained through the initial joint optimization model training.

[0140] The first neural network model can obtain the node status information s of the target node in the current time slot network through the selected centralized controller node, wherein the target node can be a node in a data waiting to be sent state when the state of the node corresponding to the unloading strategy in the current time slot is obtained. Based on the node state information s, the actor of the first neural network model outputs a data sending action a corresponding to the node state information, and the probability p2(a|s) of taking the action, so that the target node interacts with the environment according to the data transmission power, azimuth and pitch angle in the action, and transmits the data to the receiving target node or unloads the data. In the next time slot, the environment updates the state according to the action execution result, obtains the target node state information s_ of the receiving target node, and judges the data transmission processing situation according to the node state information, obtains the reward information r based on the reward information function, and stores the node state information of the target node generated from the current time slot to the next time slot (the target node transmits data to the receiving target node), the data sending action, the target node state information of the receiving target node, the reward information, and the data transmission processing judgment result done (whether all data streams in the unloading strategy have been transmitted and processed) in the experience replay pool of the first neural network model with a preset structure (s, a, r, s_, done).

[0141] According to the critic and GAE methods of the first neural network model, the state value of (s,a,r,s_,done) is estimated to obtain the target state value and advantage function.

[0142] Among them, in the embodiment of the present application, according to the node status information of the target node and the target node status information of the receiving target node, the method for determining the reward information may include:

[0143] Determine the initial reward information and the transmission processing status of the data flow offloading task corresponding to the target offloading strategy according to the reward information function of the first neural network model, the node state information of the target node and the target node state information of the receiving target node;

[0144] If the data flow offloading task corresponding to the target offloading strategy has unfinished transmission processing, the node status information of the target node, the data sending action, the initial reward information, the target node status information of the receiving target node, and the transmission processing status are stored in the initial joint optimization model, and the receiving target node is used as the target node. The simulation processing of resource allocation for the target offloading strategy is repeated according to the first neural network model in the initial joint optimization model, and the steps of determining the node status information of the target node in the target offloading strategy and the data sending action corresponding to the node status information are determined until the transmission processing of the data flow offloading task corresponding to the target offloading strategy is completed and the reward information is obtained.

[0145] Among them, when the result of the data transmission processing judgment shows that there is a data flow that has not completed the transmission processing, (s, a, r, s_, done) is stored in the experience replay pool of the first neural network model, and then the receiving target node is taken as the target node s = s_, and then the node state information is obtained, according to the actor output data sending action of the first neural network model, and the probability of taking the action, the data sending action is used to interact with the environment to obtain the target node state information s_ and reward information r of the receiving target node, and store them in the experience replay pool of the first neural network model in a preset structure, and this cycle is repeated, that is, Buffer = Buffer∪(s, a, r, s_, done), s = s_, until the data flow offloading task transmission processing corresponding to the target offloading strategy is completed, and the various reward information is accumulated to obtain the final reward information.

[0146] S204. Perform state prediction on the data sending action according to the second neural network model of the initial joint optimization model to obtain state prediction information of the receiving target node and a predicted state value corresponding to the state prediction information.

[0147] Among them, the probability p1(a|s) of the data sending action can be obtained by using the stored information in the experience replay pool and the actor of the second neural network model, and the predicted state value can be obtained by using the critic.

[0148] S205. According to the predicted state value, the target state value, and the advantage function, the initial joint optimization model is adjusted to obtain the target joint optimization model.

[0149] Among them, based on the defined loss function, the gradient descent method is used to update the model parameters of the initial joint optimization model. Furthermore, the second neural network model is used to obtain the predicted state value and calculate the loss function. The loss function Loss can satisfy:

[0150]

[0151] Among them, ε is a small positive number; v1 is the predicted state value; v2 is the target state value.

[0152] The parameters of the second neural network model are adjusted based on the loss function and the gradient descent method. The number of training times for adjusting the parameters of the second neural network model can be determined according to the amount of information stored in the experience replay pool, and then the adjusted model parameters are synchronously transmitted to the first neural network model to obtain the target joint optimization model.

[0153] In one example, the number of simulation training times of the initial joint optimization model can also be limited by setting a maximum number of simulations m1. When the second neural network model synchronously transmits the adjusted model parameters to the first neural network model to obtain the target joint optimization model, it is determined whether the current number of simulations is less than m1. If so, steps S203 to S205 are repeated until the current number of simulations is greater than or equal to m1, and step S206 is entered.

[0154] S206: Determine a target data sending action and a resource scheduling solution generated according to the target data sending action according to the target joint optimization model and the target offloading strategy.

[0155] Among them, the current network state s is determined according to the target joint optimization model, and then the target data sending action is the resource scheduling sub-plan under the current state (time slot). The target data sending action is used to interact with the environment to obtain the target node state information s_(next time slot), reward information r, and data transmission processing judgment result done. Then, according to the above method (s→a→s_, r, done), it continues to interact with the environment until all data streams in the target offloading strategy are transmitted and processed. Done = True, the resource scheduling sub-plans in each time slot are merged, which is the optimal resource scheduling plan. The time spent from the beginning to the end is the minimum service completion time T best .

[0156] Among them, in the embodiment of the present application, when there are multiple target uninstallation strategies,

[0157] According to the target joint optimization model and the target offloading strategy, a method for determining a target data sending action and a resource scheduling scheme generated according to the target data sending action may include:

[0158] Compare the reward information of each target unloading strategy to determine the optimal target unloading strategy among the target unloading strategies;

[0159] Determine a joint optimization model corresponding to an optimal target unloading strategy in the target joint optimization model;

[0160] According to the optimal target unloading strategy and the target number of simulations of the optimal target unloading strategy by the preset joint optimization model, the joint optimization model is iterated to obtain the optimal joint optimization model;

[0161] According to the optimal joint optimization model, resource allocation processing is performed on the optimal target offloading strategy to obtain the target data sending action corresponding to each node in the optimal target offloading strategy;

[0162] According to the target data sending action of each node in the optimal target offloading strategy, a resource scheduling plan is generated.

[0163] Among them, when there are multiple target offloading strategies, if all target offloading strategies are trained until the algorithm fully converges before determining the optimal offloading strategy among the target offloading strategies, it will take too long, and the algorithm will improve slowly most of the time. Therefore, the training method can be divided into two stages: preliminary training and detailed training, which avoids a large amount of training for all target offloading strategies, reduces the algorithm calculation time, and improves the calculation efficiency.

[0164] In the preliminary training stage, the neural network parameters of different target unloading strategies are trained in a small amount, and the optimal unloading strategy is found through the reward function value after the preliminary training; in the detailed training stage, the neural network parameters of the optimal unloading strategy are trained in detail and in large quantities to obtain the optimized neural network parameters to generate the optimized resource scheduling plan. Further, after the completion of the preliminary training stage, comparing the reward information of each target unloading strategy can refer to determining the minimum service completion time between different target unloading strategies according to the size of the reward information, and determining the target unloading strategy with the least time consumption as the optimal target unloading strategy. Since the model parameters of different target unloading strategies are different in the preliminary training stage, it is necessary to determine the joint optimization model corresponding to the optimal target unloading strategy. Then, according to the preset target simulation times m2 (m2>>m1), the joint optimization model is iteratively processed to obtain the optimal joint optimization model, and then the resource scheduling plan is generated according to the optimal joint optimization model.

[0165] The network task offloading strategy and resource scheduling joint optimization method provided in the embodiment of the present application utilize the topological structure and node location information of the multi-hop networking of the drone to determine the weight of each node path in the data flow offloading strategy and the weight of the offloading strategy, thereby screening out the target offloading strategy according to the weight of the offloading strategy, excluding the offloading strategy that does not meet the requirements, and reducing the number of offloading schemes to improve the calculation efficiency. The first neural network model is used to calculate the reward information and the target state value according to the state information and data sending action of the target node in the target offloading strategy, and then the advantage function is determined, which can provide a key feedback signal for the strategy optimization. The state of the data sending action is predicted according to the second neural network model to obtain the state prediction information and the predicted state value of the receiving target node. Then, the initial joint optimization model is adjusted in combination with the predicted state value, the target state value and the advantage function to form a target joint optimization model, so that the model can guide the data flow transmission node to make a better resource scheduling decision in a complex network environment, and dynamically allocate the transmission power, millimeter wave beam azimuth and pitch angle of each data flow routing link in multiple time slots, thereby obtaining the optimized offloading strategy and resource scheduling scheme, reducing the delay of data transmission, reducing the completion time of data transmission service, and realizing efficient data transmission.

[0166] Figure 3A flowchart of another network task offloading strategy and resource scheduling joint optimization method provided by this application is as follows: Figure 3 As shown, the method may include:

[0167] S301. Construct a model for multi-source data multi-hop transmission problem in an air-to-ground multi-hop network scenario.

[0168] In the air-ground multi-hop network scenario, the aerial nodes are composed of a large number of drone nodes, and the ground nodes are composed of a small number of edge servers ES. ES provides cloud computing functions to process and calculate drone mission data. During the data transmission process of the network, the drone first needs to select an edge server to unload the data. After determining the unloading plan, the drone needs to generate a directional beam through the millimeter wave array antenna, transmit data in a direction, and establish an end-to-end multi-hop link. After the link is established, the size of the node transmission power and the direction of the beam will affect the data transmission quality. The data flow unloading strategy will also affect the number of routing nodes in the link and the distance between nodes, thereby affecting the size of the transmission rate. Therefore, based on the data flow unloading strategy, the time from the start of data transmission to the completion of the service where all data flows are unloaded can be used as the objective function to optimize the multi-source data multi-hop transmission problem. The data flow unloading strategy, the transmission power of each node in different time slots, and the antenna pointing angle can be used as optimization variables to optimize the unloading strategy and resource scheduling plan through deep reinforcement learning. Among them, the unloading plan can refer to Figure 1 The overall offloading strategy of the data flow offloading task in the data flow offloading task includes the offloading strategy of each data flow in the data flow offloading task.

[0169] S302: Based on the air-ground multi-hop network topology structure, screen the offloading solutions to obtain an offloading solution set.

[0170] Among them, since the number of offloading strategies increases sharply with the increase in the number of edge servers and the number of data flows, it is obviously unrealistic to traverse all offloading strategy solutions in the massive drone scenario. However, among all the solutions, there are many solutions, such as the offloading solution that all data flows choose the ES with the largest number of hops and the longest distance to offload data, which is obviously not the optimal solution. Therefore, these solutions can be excluded first to reduce the number of offloading solutions and improve computational efficiency. Furthermore, the constraints of each offloading solution are set, and according to the topological structure of the air-ground multi-hop network, the minimum distance routing algorithm is used to calculate the number of routing hops from its initial node to different ES in the offloading solution, and sort them. For the number of hops, the value weight values ​​of N, N-1, ..., 1 are assigned in order from small to large, and the offloading solutions that meet the requirements are screened out according to the value weight values ​​and constraints of different solutions.

[0171] S303, preliminary training stage: first determine the unloading plan to be trained, and then use the deep reinforcement learning algorithm to obtain the reward function and neural network parameters of the unloading plan.

[0172] Among them, the state s is divided into two parts. One part is used to characterize the state of the data to be sent by each node in the data stream, and the second part is the label of the routing node in the task offloading scheme; the transmission power, azimuth and pitch angle of each node are used as the action a of the node in the Markov decision process; the reward function is the key to the reinforcement learning method. We combine the objective function of the research problem to give a specific single-step reward function. After a round, the single-step reward function is summed up to get the cumulative reward value r of a round. We only need to continuously use reinforcement learning to train the intelligent agent and let r continue to tend to the maximum value to achieve our optimization goal - minimizing the end-to-end service completion time.

[0173] In one example, the state s1 of each node in the data stream to be sent and the label s2 of the routing node in the task offloading scheme can be defined as:

[0174]

[0175] Unloading plan for all participants in training Perform the following steps:

[0176] (1) Import the neural network parameters of two actor-critic (AC) networks AC1 and AC2. match Initialize the parameters of the reinforcement learning environment env, initialize the number of simulations ep=0, the maximum number of simulations m1, and clear the experience replay pool Buffer.

[0177] (2) Reset user status s.

[0178] (3) Input the current state s into the actor network of the neural network AC2 to obtain the action a and the probability of the action p2(a|s).

[0179] (4) Action a interacts with env to obtain the next state s_, reward value r, and whether it is done.

[0180] (5) Store (s,a,r,s_,done) into the experience replay pool, Buffer = Buffer∪(s,a,r,s_,done), s = s_.

[0181] (6) Repeat steps (2)-(5) until done = True. Get a cumulative reward value reward_ep of ep, ep = ep+1.

[0182] (7) When the number of elements in the Buffer reaches a certain amount, use (s, a, r, s_, done) to generate the advantage function A and the cumulative discounted reward value v2 using the GAE method.

[0183] (8) Using the actor network of AC1, we get the probability of action a, p1(a|s), and using the critic network of AC1, we get the state value, v1.

[0184] (9) Obtain Loss function And use gradient descent to update the actor and critic network parameters in AC1.

[0185] (10) Repeat steps (8)-(9) k times and pass the AC1 network parameters to the AC2 network.

[0186] (11) Repeat steps (2)-(10) until ep = m1, and obtain and save the maximum reward function reward (X match ) and the actor parameters and critic parameters of the neural network AC2. And the corresponding unloading scheme X is obtained through the maximum reward function match Service completion time T service (X match ) = positive constant - reward(X match ).

[0187] S304, detailed training phase: Compare rewards of different solutions to obtain the optimal unloading solution X best , and then use deep reinforcement learning to train in detail to get the optimal reward function value reward best , optimal neural network parameters actor best 、critic best .

[0188] Among them, the detailed training stage may include:

[0189] (1) By comparing the minimum service completion time between different offloading schemes, the optimal offloading scheme with the shortest service time is obtained.

[0190] (2) Optimal unloading solution X best , expand the number of simulations to m2>>m1, import the neural network parameters saved in the corresponding scheme in step S303 (11), and repeat steps S303 (1) to (11) to obtain the optimal neural network parameters actor after detailed training best and critic best .

[0191] S305: Input the user's initial state into the optimal neural network, and continuously interact with the environment to output a multi-slot beam scheduling strategy and a minimum service completion time.

[0192] In this step, get the actor best and critic best After that, the state s is input into the neural network, and the obtained action a is the optimal beam scheduling solution under this state (this time slot). By using action a to interact with the environment, we can get the state s_, reward function r, and whether the next time slot is done. We continue to interact with the environment in the way of s→a→s_,r,done until done=True. Combining the actions of each step is the optimal multi-time slot beam scheduling solution. The time spent from the beginning to the end is the minimum service completion time T best .

[0193] The network task offloading strategy and resource scheduling joint optimization method provided in the embodiment of the present application establishes a mathematical model of joint offloading scheme, power allocation and beam pointing angle under joint multi-time slots to minimize network flow delay, and then sets the scheme weight based on the air-ground multi-hop networking topology structure to screen the offloading scheme, thereby reducing the complexity of the problem; and optimizes the offloading decision based on the deep reinforcement learning algorithm, screens the multi-domain routing scheme, and dynamically allocates the transmission power, millimeter wave beam azimuth and pitch angle of each data flow routing link in multi-time slots, so as to realize low-complexity cross-domain network multi-flow service efficient data transmission and minimize the service completion delay of the offloading decision.

[0194] Figure 4 A schematic diagram of the structure of a network task offloading strategy and resource scheduling joint optimization device provided in this application, such as Figure 4 As shown, the network task offloading strategy and resource scheduling joint optimization device 40 provided in this embodiment includes:

[0195] The first determination module 401 is used to determine the drone multi-hop network, the location information of the nodes in the drone multi-hop network, and the unloading strategy when the data flow is transmitted from the starting node to the end node in the node;

[0196] The second determination module 402 is used to determine the node path of the data flow from the starting node to the end node, the value weight of the node path, and the weight of the unloading strategy according to the location information of the nodes in the multi-hop network of the drone;

[0197] The first analysis module 403 is used to determine the reward information and target state value of the target unloading strategy, and the advantage function according to the first neural network model in the initial joint optimization model, wherein the target unloading strategy is an unloading strategy whose weight satisfies a preset weight condition, the reward information of the target unloading strategy is determined according to the node state information of the target node in the target unloading strategy and the corresponding data sending action, the node state information of the target node and the corresponding data sending action are determined by the simulation processing of resource allocation for the target unloading strategy, the target state value is determined according to the target node state information of the receiving target node, the receiving target node is a node that receives data after the target node transmits data according to the data sending action, and the advantage function is determined according to the reward information and the target state value;

[0198] The second analysis module 404 is used to perform state prediction on the data sending action according to the second neural network model of the initial joint optimization model, and obtain state prediction information of the receiving target node and a predicted state value corresponding to the state prediction information;

[0199] The optimization module 405 is used to adjust the initial joint optimization model according to the predicted state value, the target state value, and the advantage function to obtain the target joint optimization model;

[0200] The third determination module 406 is used to determine the target data sending action and the resource scheduling solution generated according to the target data sending action according to the target joint optimization model and the target offloading strategy.

[0201] In a possible implementation, the first determining module 401 may also be used to:

[0202] Determine the drone multi-hop networking, the location information of the nodes in the drone multi-hop networking, and the data flow offloading task in the drone multi-hop networking;

[0203] According to the data flow offloading task, determine the starting node of the data flow offloading task in the node and the end node in the multi-hop network of the drone, and the end node is the node that receives the data and performs data offloading processing;

[0204] According to the location information of the starting node, the end node and the nodes in the multi-hop network of drones, the offloading strategy is determined when the data flow is transmitted from the starting node to the end node in the node.

[0205] In a possible implementation, the second determining module 402 may also be used to:

[0206] According to the location information of the nodes in the multi-hop network of drones, a node path set for transmitting the data stream from the starting node to the end node is determined;

[0207] Determine the node path for transmitting the data stream from the starting node to the end node and the number of routing hops in the node path according to the path distance of each node path in the node path set;

[0208] According to the node path of the data flow transmitted from the starting node to the end node and the number of routing hops in the node path, the value weight of the node path and the weight of the unloading strategy are obtained, wherein the smaller the number of routing hops in the node path, the greater the value weight.

[0209] In a possible implementation, the second determining module 402 may also be used to:

[0210] According to the number of starting nodes and the number of end nodes in the multi-hop network of the UAV, an unloading sub-strategy for transmitting the data flow from the starting node to the end node in the unloading strategy and a node path corresponding to the unloading sub-strategy are determined;

[0211] According to the number of routing hops of the node path corresponding to the unloading sub-strategy, the value weight of the node path corresponding to the unloading sub-strategy is obtained;

[0212] The weight of the uninstallation strategy is obtained according to the value weight of the uninstallation sub-strategy and the node path corresponding to the uninstallation sub-strategy.

[0213] In a possible implementation, the first analysis module 403 may also be used to:

[0214] According to the state of the amount of data to be sent and the node label of each node in the multi-hop network of drones, the node status information is obtained;

[0215] According to the data transmission power, azimuth angle and pitch angle of each node in the multi-hop network of the drone, the data sending action is obtained;

[0216] Based on the service completion time of the data stream offloading task, a reward information function corresponding to the offloading strategy is constructed;

[0217] An initial joint optimization model is constructed based on node status information, data sending actions, reward information functions, and preset neural networks.

[0218] In a possible implementation, the first analysis module 403 may also be used to:

[0219] Determine a target uninstallation strategy in the uninstallation strategy according to the weight of the uninstallation strategy and a preset weight condition;

[0220] According to the first neural network model in the initial joint optimization model, a simulation process of resource allocation is performed on the target offloading strategy to determine the node state information of the target node in the target offloading strategy and the data sending action corresponding to the node state information;

[0221] Determine the target node state information of the receiving target node according to the environment information of the first neural network model and the data sending action corresponding to the node state information;

[0222] Determine reward information according to the node status information of the target node and the target node status information of the receiving target node;

[0223] According to the first neural network model, the state value of the target node state information of the receiving target node is estimated to determine the state value of the receiving target node and the target state value of the target unloading strategy;

[0224] According to the reward information and the target state value, the advantage function is obtained.

[0225] In a possible implementation, the first analysis module 403 may also be used to:

[0226] Determine the initial reward information and the transmission processing status of the data flow offloading task corresponding to the target offloading strategy according to the reward information function of the first neural network model, the node state information of the target node and the target node state information of the receiving target node;

[0227] If the data flow offloading task corresponding to the target offloading strategy has unfinished transmission processing, the node status information of the target node, the data sending action, the initial reward information, the target node status information of the receiving target node, and the transmission processing status are stored in the initial joint optimization model, and the receiving target node is used as the target node. The simulation processing of resource allocation for the target offloading strategy is repeated according to the first neural network model in the initial joint optimization model, and the steps of determining the node status information of the target node in the target offloading strategy and the data sending action corresponding to the node status information are determined until the transmission processing of the data flow offloading task corresponding to the target offloading strategy is completed and the reward information is obtained.

[0228] In a possible implementation, the third determining module 406 may also be used to:

[0229] Compare the reward information of each target unloading strategy to determine the optimal target unloading strategy among the target unloading strategies;

[0230] Determine a joint optimization model corresponding to an optimal target unloading strategy in the target joint optimization model;

[0231] According to the optimal target unloading strategy and the target number of simulations of the optimal target unloading strategy by the preset joint optimization model, the joint optimization model is iterated to obtain the optimal joint optimization model;

[0232] According to the optimal joint optimization model, resource allocation processing is performed on the optimal target offloading strategy to obtain the target data sending action corresponding to each node in the optimal target offloading strategy;

[0233] According to the target data sending action of each node in the optimal target offloading strategy, a resource scheduling plan is generated.

[0234] The network task offloading strategy and resource scheduling joint optimization device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0235] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0236] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0237] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0238] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0239] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0240] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0241] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0242] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0243] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0244] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0245] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0246] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0247] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0248] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0249] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0250] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A network task offloading strategy and resource scheduling joint optimization method, characterized in that: include: Determine a drone multi-hop network, location information of nodes in the drone multi-hop network, and an unloading strategy when a data flow is transmitted from a starting node in the nodes to a terminal node; Determine, according to the location information of the nodes in the multi-hop network of the drone, a node path for transmitting the data stream from the starting node to the end node, a value weight of the node path, and a weight of the unloading strategy; According to the first neural network model in the initial joint optimization model, the reward information and the target state value, and the advantage function of the target unloading strategy are determined, wherein the target unloading strategy is an unloading strategy whose weight satisfies a preset weight condition, the reward information of the target unloading strategy is determined according to the node state information of the target node in the target unloading strategy and the corresponding data sending action, the node state information of the target node and the corresponding data sending action are determined by a simulation process of resource allocation for the target unloading strategy, the target state value is determined according to the target node state information of the receiving target node, the receiving target node is a node that receives data after the target node transmits data according to the data sending action, and the advantage function is determined according to the reward information and the target state value; According to the second neural network model of the initial joint optimization model, the state of the data sending action is predicted to obtain the state prediction information of the receiving target node and the predicted state value corresponding to the state prediction information; According to the predicted state value, the target state value, and the advantage function, the initial joint optimization model is adjusted to obtain a target joint optimization model; According to the target joint optimization model and the target offloading strategy, a target data sending action and a resource scheduling scheme generated according to the target data sending action are determined.

2. The method according to claim 1, characterized in that: The determining of the drone multi-hop network, the location information of the nodes in the drone multi-hop network, and the unloading strategy when the data flow is transmitted from the starting node to the end node in the nodes includes: Determine a drone multi-hop network, location information of nodes in the drone multi-hop network, and a data flow offloading task in the drone multi-hop network; According to the data flow offloading task, determining a starting node of the data flow offloading task and an end node in the multi-hop network of the drone among the nodes, wherein the end node is a node that receives data and performs data offloading processing; According to the location information of the starting node, the end node and the nodes in the multi-hop network of the drone, an unloading strategy is determined when the data flow is transmitted from the starting node among the nodes to the end node.

3. The method according to claim 1, characterized in that: The determining, according to the location information of the nodes in the multi-hop network of the drone, the node path of the data stream transmitted from the starting node to the end node, the value weight of the node path, and the weight of the unloading strategy includes: Determine, according to the location information of the nodes in the multi-hop network of the drone, a set of node paths for transmitting the data stream from the starting node to the end node; Determine, according to the path distance of each node path in the node path set, a node path for transmitting the data stream from the starting node to the end node, and the number of routing hops in the node path; According to the node path of the data flow transmitted from the starting node to the end node and the number of routing hops in the node path, the value weight of the node path and the weight of the unloading strategy are obtained, wherein the smaller the number of routing hops in the node path, the greater the value weight.

4. The method according to claim 3, characterized in that The step of obtaining the value weight of the node path and the weight of the offloading strategy according to the number of routing hops in the node path of the data flow transmitted from the starting node to the end node includes: According to the number of starting nodes and the number of end nodes in the multi-hop network of the drone, determine an unloading sub-strategy in the unloading strategy for transmitting the data flow from the starting node to the end node, and a node path corresponding to the unloading sub-strategy; Obtaining a value weight of the node path corresponding to the unloading sub-strategy according to the number of routing hops of the node path corresponding to the unloading sub-strategy; The weight of the uninstallation strategy is obtained according to the uninstallation sub-strategy and the value weight of the node path corresponding to the uninstallation sub-strategy.

5. The method according to claim 1, characterized in that Before determining the reward information and target state value and advantage function of the target unloading strategy according to the first neural network model in the initial joint optimization model, the method further includes: Obtaining node status information according to the state of the amount of data to be sent and the node label of each node in the multi-hop network of the drone; According to the data transmission power, azimuth angle and pitch angle of each node in the multi-hop network of the drone, a data sending action is obtained; Based on the service completion time of the data stream offloading task, a reward information function corresponding to the offloading strategy is constructed; An initial joint optimization model is constructed according to the node status information, the data sending action, the reward information function, and a preset neural network.

6. The method according to claim 1, characterized in that Determining the reward information and target state value and advantage function of the target unloading strategy according to the first neural network model in the initial joint optimization model includes: Determining a target uninstallation strategy in the uninstallation strategy according to the weight of the uninstallation strategy and a preset weight condition; According to the first neural network model in the initial joint optimization model, a simulation process of resource allocation is performed on the target offloading strategy to determine the node state information of the target node in the target offloading strategy and the data sending action corresponding to the node state information; Determine the target node state information of the receiving target node according to the environment information of the first neural network model and the data sending action corresponding to the node state information; Determining reward information according to the node status information of the target node and the target node status information of the receiving target node; According to the first neural network model, a state value estimation is performed on the target node state information of the receiving target node to determine the state value of the receiving target node and the target state value of the target unloading strategy; An advantage function is obtained according to the reward information and the target state value.

7. The method according to claim 6, characterized in that The step of determining reward information according to the node status information of the target node and the target node status information of the receiving target node includes: Determine initial reward information and a transmission processing status of the data stream offloading task corresponding to the target offloading strategy according to the reward information function of the first neural network model, the node state information of the target node and the target node state information of the receiving target node; If the data flow unloading task corresponding to the target unloading strategy has incomplete transmission processing, the node status information of the target node, the data sending action, the initial reward information, the target node status information of the receiving target node, and the transmission processing status are stored in the initial joint optimization model, and the receiving target node is used as the target node, and the simulation processing of resource allocation for the target unloading strategy is repeated according to the first neural network model in the initial joint optimization model, and the node status information of the target node in the target unloading strategy and the data sending action corresponding to the node status information are determined until the transmission processing of the data flow unloading task corresponding to the target unloading strategy is completed and the reward information is obtained.

8. The method according to claim 1, characterized in that: When there are multiple target uninstallation policies, The determining, according to the target joint optimization model and the target offloading strategy, a target data sending action, and a resource scheduling scheme generated according to the target data sending action, comprises: Compare the reward information of each target unloading strategy to determine the optimal target unloading strategy among the target unloading strategies; Determine a joint optimization model in the target joint optimization model corresponding to the optimal target offloading strategy; According to the optimal target unloading strategy and the target number of simulations of the optimal target unloading strategy by the preset joint optimization model, the joint optimization model is iterated to obtain the optimal joint optimization model; According to the optimal joint optimization model, resource allocation processing is performed on the optimal target offloading strategy to obtain target data sending actions corresponding to each node in the optimal target offloading strategy; A resource scheduling scheme is generated according to the target data sending action of each node in the optimal target offloading strategy.

9. A network task offloading strategy and resource scheduling joint optimization device, characterized in that: include: A first determination module is used to determine the drone multi-hop network, the location information of the nodes in the drone multi-hop network, and the unloading strategy when the data flow is transmitted from the starting node to the end node in the nodes; A second determination module is used to determine the node path of the data flow transmitted from the starting node to the end node, the value weight of the node path, and the weight of the unloading strategy according to the location information of the nodes in the multi-hop network of the drone; A first analysis module is used to determine the reward information and target state value, and the advantage function of the target unloading strategy according to the first neural network model in the initial joint optimization model, wherein the target unloading strategy is an unloading strategy whose weight satisfies a preset weight condition, the reward information of the target unloading strategy is determined according to the node state information of the target node in the target unloading strategy and the corresponding data sending action, the node state information of the target node and the corresponding data sending action are determined by a simulation process of resource allocation for the target unloading strategy, the target state value is determined according to the target node state information of the receiving target node, the receiving target node is a node that receives data after the target node transmits data according to the data sending action, and the advantage function is determined according to the reward information and the target state value; A second analysis module is used to perform state prediction on the data sending action according to the second neural network model of the initial joint optimization model, and obtain state prediction information of the receiving target node and a predicted state value corresponding to the state prediction information; an optimization module, configured to adjust the initial joint optimization model according to the predicted state value, the target state value, and the advantage function to obtain a target joint optimization model; The third determination module is used to determine the target data sending action and the resource scheduling solution generated according to the target data sending action according to the target joint optimization model and the target offloading strategy.

10. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8.

Images