Edge computing enabled air-to-ground network service deployment and resource allocation method and system

Abstract

The invention belongs to but is not limited to the technical field of wireless communication, and discloses an edge computing enabled air-ground network service deployment and resource allocation method and system, which is characterized in that application programs are strategically deployed and replaced on a UAV (Unmanned Aerial Vehicle), computing resources and power are reasonably allocated, and on the premise that the task execution efficiency and accuracy are ensured, the resource allocation efficiency is improved. The task execution success rate of the IoT equipment in the intelligent railway system is maximized; the mobile edge computing is introduced into the air-space-ground integrated network, the characteristics of flexible deployment and small transmission delay of the UAV in the air-based network are utilized, and the complexity and dynamism of a traffic system are effectively dealt with and the resource allocation and power control are optimized through collaborative optimization of task placement, cache replacement, a resource allocation strategy and the transmission power of IoT (Internet of Things) equipment. According to the method, a deep reinforcement learning framework is adopted, and an algorithm based on a multi-agent depth deterministic strategy gradient is provided to realize joint optimization of task placement, cache replacement, resource allocation and IoT equipment transmission power.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of wireless communication technology, and in particular relates to an edge computing-enabled air-to-ground network service deployment and resource allocation method and system. Background Art

[0002] MEC is a powerful supplement to cloud computing. By deploying computing and caching resources at the edge of the network, IoT devices can execute computing-intensive and resource-intensive applications on edge nodes, significantly reducing the latency and energy consumption of task processing. At the same time, the integrated air-ground-air network achieves seamless coverage, high-speed data transmission and multi-dimensional communication services around the world by organically combining ground-based networks, space-based networks and air-based networks. This network architecture not only solves the problem of insufficient coverage of traditional networks, but also provides effective support for communication needs in various scenarios, including remote areas, oceans, aviation and Internet of Vehicles. Under this architecture, UAVs can quickly respond to high bandwidth requirements in local areas, while satellites make up for the lack of UAV coverage, ensuring stable network services even in the most remote areas. In addition, the introduction of edge computing further improves the efficiency of data processing, enabling applications such as Internet of Vehicles to achieve real-time data analysis and decision-making, thereby bringing users a safer, more convenient and personalized experience.

[0003] However, there are some challenges that need to be addressed when combining the Internet of Vehicles with UAVs for IoT devices.

[0004] (1) As the resources of the antenna and air layers are very limited, it is crucial to rationally allocate bandwidth resources to drones and satellites. Therefore, it is necessary to effectively save bandwidth resources while ensuring the reduction of system energy consumption to support the data transmission needs of Internet of Vehicles and IoT devices.

[0005] (2) Since the high dynamics of drones require the algorithm to be able to quickly adapt to their position changes, it is necessary to rationally plan the movement paths of drones to maximize coverage and service quality.

[0006] (3) In UAV and IoV-based scenarios, the environment is usually highly dynamic and the goals are usually defined as long-term objectives. In this case, we need to design higher-performance algorithms to achieve goal optimization with low complexity and good performance.

[0007] Through the above analysis, the problems and defects of the existing technology are as follows: the existing literature does not take into account the joint optimization of drone cache placement and edge computing capacity allocation in IRS, how to reasonably utilize the rapid movement of drones, and drones and other equipment in air-ground integrated scenarios are often highly dynamic, which puts higher performance requirements on the algorithm and is therefore difficult to solve. Summary of the invention

[0008] In view of the problems existing in the prior art, the joint optimization of the air-ground network integration system enabled by edge computing provided by the present invention specifically involves the joint optimization of collaborative task deployment, offloading decision, power control and resource allocation methods.

[0009] The present invention is implemented as follows: an edge computing-enabled air-to-ground network service deployment and resource allocation method comprises the following steps:

[0010] Step 1: Establish a training network model. The training network of each intelligent agent includes the online network and the target network, and establish an air-ground network model enabled by edge computing. The online network includes the online Actor network and the online Critic network, and the target network includes the target strategy network Actor and the target value network Critic.

[0011] Step 2: Initialize the parameters in the network, including the parameters θ of the online Actor and Critic networks Q and θ μ , the parameters θ of the target Actor and Critic network Q′ and θ μ′ , the maximum number of training steps T, the number of samples extracted from the experience replay buffer mini-batch, the reward discount factor γ γ , the maximum number of rounds Epi, the noise ψ used for action exploration, the learning rates α and β used by the Actor and Critic networks, the soft update parameter τ, etc.;

[0012] Step 3: reset each agent in the environment to obtain the initial system state and summarize it into a state set S; set the time of each round t = t0;

[0013] Step 4: The agent passes through the Actor network Get action A k (t), and add noise ψ; and reconstruct the action, and transform the action A into k (t) Reconstructed into A k (t)=(A k (t)+1) / 2;

[0014] Step 5: Get the global action A(t) and execute it to get the reward and the state S(t+1) at the next moment; store the current experience tuple <S(t), A(t), r(t), S(t+1)> in the replay buffer pool;

[0015] Step 6: Determine whether the replay buffer is full. If so, randomly select mini-batch samples from the buffer to train the network. Otherwise, determine whether t is equal to T. If so, start again with step 4.

[0016] Step 7: Calculate the gradient of the current Critic network of the agent and update the current Actor network through gradient ascent; use the current Critic network to calculate the value of the action at time t, and use the target network to calculate the value of the action at time t+1, and then update the current Critic network through gradient descent;

[0017] Step 8: Use soft update technology to update the Actor network and Critic network in the target network;

[0018] Step nine, determine whether it converges, if so, get the final optimal solution, otherwise start from step four again.

[0019] Furthermore, the state of the agent and the global state in step 3 are respectively expressed as:

[0020] S k (t) = {r k (t),U k (t-1),E k (t),L m,k (t),PL k (t),A k (t)},

[0021] S(t)={S1(t),S2(t),...,S K (t)}.

[0022] where r k (t) = {r m,k (t)}, represents the task request sent by the IoT device at the beginning of each time slot; U k (t-1) = {n m,k (t-1)}, represents the set of existing contents on UAV k in time slot t-1; It represents the path loss coefficient under line-of-sight and non-line-of-sight transmission; L m,k (t), represents the Euclidean distance matrix between IoT device m and UAV k; PL k (t), A represents the additional path loss matrix between IoT device m and UAV k; k (t), Indicates the access situation in the system, S k (t), is the state of UAV k.

[0023] Further, in step 4, the obtained action A k(t) is expressed as follows:

[0024]

[0025] Among them, Δ k (t) = {δ n,k (t)}, is the mission placement strategy for UAV k; is the task replacement strategy of UAV k; P k (t) = {p m,k (t)}, is the transmission power between UAV k and associated IoT device m; F k (t) = {f m,k (t)}, is the computing resource allocation strategy of UAV k.

[0026] Furthermore, in step 4, the process of reconstructing the action is expressed as follows:

[0027] (1) For binary task placement decision Δ k (t) and replacement decision Use the rounding method to obtain the corresponding binary discrete action;

[0028] (2) For integer transmit power P k (t) and computing resources F k (t), and compare it with the maximum processing power p of IoT devices m and UAV maximum processing capacity f m Multiply them to obtain the actual power matrix and the computing resource matrix allocated to the UAV respectively;

[0029] (3) Through simple mathematical transformation A k (t)=(A k (t)+1) / 2, changing the range of the intrinsic action from [-1,1] to [0,1].

[0030] Furthermore, in step 5, the calculation formula of reward is as follows:

[0031]

[0032] Among them, r suc (t) represents the task execution success rate of the system, that is, the objective function.

[0033] Furthermore, in step seven, the updating process of the policy network is as follows:

[0034] The current gradient of the Critic network is as follows

[0035]

[0036] The average gradient is given by

[0037]

[0038] Update the current Actor network through gradient ascent as follows

[0039]

[0040] Furthermore, in step 7, the updating process of the current value network is as follows:

[0041] The target TD is as follows

[0042]

[0043] The TD error is as follows:

[0044]

[0045] Perform gradient descent to update the parameters of the current Critic network as follows

[0046]

[0047] Further, in step eight, the soft update formula of the target network is as follows:

[0048]

[0049] in Represents the parameters of the current Actor network, Represents the parameters of the target Actor network, Represents the parameters of the current Critic network, Represents the parameters of the target Critic network, τ∈[0,1].

[0050] Another object of the present invention is to provide an edge computing-enabled air-to-ground network service deployment and resource allocation method, an edge computing-enabled air-to-ground network service deployment and resource allocation system, comprising:

[0051] The network model building module builds a training network model. The training network of each intelligent agent includes the online network and the target network, and builds an air-ground network model enabled by edge computing. The online network includes the online Actor network and the online Critic network, and the target network includes the target strategy network Actor and the target value network Critic.

[0052] Parameter initialization module, which initializes the parameters in the network, including the parameters θ of the online Actor and Critic networksQ and θ μ , the parameters θ of the target Actor and Critic network Q′ and θ μ′ , the maximum number of training steps T, the number of samples extracted from the experience replay buffer mini-batch, the reward discount factor γ γ , the maximum number of rounds Epi, the noise ψ used for action exploration, the learning rates α and β used by the Actor and Critic networks, the soft update parameter τ, etc.;

[0053] The state reset module resets each agent in the environment to obtain the initial system state and summarizes it into a state set S; sets the time of each round t = t0;

[0054] Action reconstruction module, the agent passes through the Actor network Get action A k (t), and add noise ψ; and reconstruct the action, and transform the action A into k (t) Reconstructed into A k (t)=(A k (t)+1) / 2;

[0055] The cache pool module obtains the global action A(t) and executes it to obtain the reward and the state S(t+1) at the next moment; the current experience tuple <S(t), A(t), r(t), S(t+1)> is stored in the replay cache pool;

[0056] The judgment module determines whether the replay buffer is full. If so, randomly select mini-batch samples from the buffer to train the network. Otherwise, it determines whether t is equal to T. If so, it starts executing step 4.

[0057] The network update module calculates the gradient of the current Critic network of the agent and updates the current Actor network through gradient ascent; uses the current Critic network to calculate the value of the action at time t, and uses the target network to calculate the value of the action at time t+1, and then updates the current Critic network through gradient descent;

[0058] Use soft update technology to update the Actor network and Critic network in the target network;

[0059] The convergence loop module determines whether it has converged. If so, the final optimal solution is obtained. Otherwise, it starts from step 4 again.

[0060] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0061] First, the present invention implements a joint optimization method of an integrated air-space-ground network combined with an edge computing system in a vehicle networking scenario, and studies the access optimization strategy of IoT devices, as well as application placement, application replacement, allocation of edge server computing resources, and transmission power control of IoT devices. Since continuous and discrete variables are involved, DRL is applied to this complex optimization problem by improving the input of the deep neural network. Ultimately, support for ground IoT device task requests is achieved, maximizing the success rate of IoT device task execution. This method has made significant progress in the following key aspects, which are described in detail as follows:

[0062] 1) Reasonable task request processing strategy:

[0063] By reasonably dividing the tasks sent by IoT devices, the system can intelligently adjust the application services of single or multiple devices deployed on the UAV to adapt to dynamically changing network conditions, which enhances the network stability and user experience.

[0064] 2) Efficient task placement and replacement strategy:

[0065] This method significantly enhances resource utilization and overall system performance in edge computing environments by adopting an efficient task allocation and replacement mechanism. Driven by the deep MADDPG algorithm, the task allocation and replacement strategy can fully consider task requirements, resource conditions, and real-time changes in the network. Through continuous learning and optimization, the MADDPG agent can accurately evaluate the potential value of different tasks in different execution locations, and make reasonable task placement and replacement decisions accordingly.

[0066] 3) Task execution success rate:

[0067] The reward mechanism included in the method focuses on maximizing the success rate of task execution, which can not only effectively reduce latency, but also improve data processing capabilities to support decision-making, thereby effectively reducing the maintenance cost of the system and improving user experience.

[0068] 4) Integration of reinforcement learning:

[0069] Combining the DRL algorithm with the integrated air-ground-space network supported by MEC enables the connected vehicle system to self-learn and adapt based on real-time data, making optimal decisions without explicit instructions. This adaptive capability is essential for handling complex and dynamically changing environments.

[0070] 5) Intelligent power optimization strategy:

[0071] By optimizing the transmission power during UAV mission processing, this method significantly reduces the energy consumption cost of edge devices and extends the service life of UAVs, thereby enabling edge computing devices to adapt to more diverse scenarios and dynamically adjust their performance based on the current status.

[0072] 6) Improvement of system stability and reliability:

[0073] By accurately calculating and optimizing the immediate rewards after action execution and state transitions, this approach enhances the stability and reliability of the system, especially when processing large amounts of data and high concurrent requests.

[0074] 7) Network’s autonomous learning and optimization capabilities:

[0075] This method enables the system to continuously optimize its decision-making process through continuous iterative training and experience-based network updates, thereby improving overall performance.

[0076] These technological innovations promote each other, so that in the scenario of Internet of Vehicles, the integrated air-space-ground network system supported by MEC not only has improved performance, but also has significant improvements in energy consumption, stability and adaptability, which is crucial for modern complex computing needs.

[0077] In addition, the edge computing system optimization method based on the combination of air-space-ground integrated network and deep reinforcement learning provided by the present invention is based on the use of mathematical models to guide the behavior and learning process of the system. The technical effects brought by these mathematical models can be discussed based on their characteristics:

[0078] 1) Calculation of instant rewards

[0079] The calculation of instant rewards is used to evaluate the task execution efficiency of all devices in the system. By combining rewards with the task execution success rate, the system is motivated to seek strategies to improve the task success rate to improve energy efficiency. In the long run, in edge computing environments with high energy consumption, improving the task success rate can help reduce operating costs.

[0080] 2) Updates to the Actor and Critic networks

[0081] The Actor network and the Critic network are updated using randomly sampled small batches of empirical data through the gradient ascent method. This updating method enables the system to continuously adjust its decision parameters, learn and adopt more effective strategies. At the same time, the use of small batches of data improves the system's response speed and enhances its adaptability to environmental changes.

[0082] 3) Loss function of TD target and Critic network

[0083] The system uses the TD target and the Q value predicted by the Critic network to construct the loss function and optimizes it through the gradient descent method. This TD target-based method helps stabilize the learning process and avoid instability caused by prediction errors. Through accurate loss function calculation, the system can improve the accuracy and efficiency of decision-making.

[0084] 4) Parameter update strategy

[0085] The system achieves a smooth transition between the Actor network and the Critic network by gradually updating the target network parameters. This update strategy avoids performance fluctuations caused by sudden changes in strategies, while ensuring that the system can continue to learn and adapt to long-term changes in the environment.

[0086] In summary, the advantages and positive effects of the present invention are as follows: the present invention optimizes the entire process in task offloading by jointly optimizing task deployment, offloading decisions, power control, and resource allocation, maximizing the success rate of task execution of all IoT devices while ensuring task processing constraints. Specifically, we adopt a method that integrates caching and computing: the UAV is responsible for caching the applications required to process specific tasks and using its processing power to execute these tasks. By strategically deploying and replacing applications on the UAV to select the appropriate UAV to process the request of each IoT device, the resource allocation and power control of the UAV are optimized. Considering the complexity of the problem and the dynamics of the network, we reformulate the problem as a Markov Decision Process (MDP) and propose a joint optimization algorithm based on MADDPG to achieve low complexity and real-time adaptive decision making. Since MADDPG is designed to solve problems with continuous action spaces, and our problem contains integer, binary, and continuous variables, MADDPG cannot work directly. To this end, we take the following measures: first normalize the continuous variables and then reformulate the continuous output of MADDPG into discrete and binary variables while ensuring the coupling constraints between different variables.

[0087] Second, the technical solution of the present invention solves a technical problem that people have been eager to solve but have never succeeded in solving: In the air-ground network system enabled by edge computing, the present invention achieves support for task requests of ground IoT devices while maximizing the success rate of IoT device task execution by coordinating access optimization strategies of IoT devices, as well as application placement, application replacement, allocation of edge server computing resources, and transmission power control of IoT devices. The present invention introduces air-ground integrated networks and mobile edge computing into the Internet of Things. Since this optimization problem contains continuous variables and discrete variables, the input of the deep neural network is improved and the DRL algorithm is adapted to the optimization problem. The solution proposed by the present invention can be applied to a variety of Internet of Things scenarios that require support for task requests of ground IoT devices.

[0088] Third, the technical problems existing in the industrial application of the prior art solved by the present invention are mainly reflected in the following aspects: First, the traditional air-ground network service deployment and resource allocation methods are often difficult to adapt to the dynamically changing environment, and cannot achieve efficient resource utilization and task execution. Secondly, these methods usually ignore the cooperation and competition relationship between intelligent agents, resulting in limited overall system performance. Furthermore, when dealing with large-scale air-ground networks, traditional methods have high computational complexity and are difficult to meet real-time requirements. Finally, the existing technology often lacks an effective mechanism to balance the relationship between task execution success rate and resource consumption.

[0089] In view of the above problems, the present invention proposes an edge computing-enabled air-to-ground network service deployment and resource allocation method. This method realizes the intelligent deployment and dynamic resource allocation of air-to-ground network services by introducing edge computing technology. Specifically, the present invention uses a deep reinforcement learning framework to establish a training network model including online and target networks, and optimizes task placement, replacement strategies, and transmission power and computing resource allocation through collaboration and competition between intelligent agents.

[0090] The application of the present invention has achieved significant technical progress. On the one hand, through the learning and decision-making of the intelligent agent, the method can adapt to the dynamically changing environment and achieve efficient resource utilization and task execution. On the other hand, the present invention improves the stability and convergence speed of learning by introducing the experience replay mechanism and the gradient ascent and descent update strategy, thereby further improving the overall performance of the system. In addition, the method exhibits lower computational complexity when processing large-scale air-ground networks, meeting the real-time requirements. Finally, by optimizing the relationship between the success rate of task execution and resource consumption, the present invention achieves an effective balance between air-ground network service deployment and resource allocation.

[0091] In summary, the present invention solves the problems existing in the traditional air-ground network service deployment and resource allocation methods by introducing edge computing and deep reinforcement learning technology, and realizes the intelligent, efficient and dynamic management of air-ground network services. The application of this method not only improves resource utilization and task execution success rate, but also reduces computational complexity and meets real-time requirements, bringing significant technological progress and application value to the development of the air-ground network industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 It is a flow chart of the edge computing-enabled air-to-ground network service deployment and resource allocation method provided in an embodiment of the present invention.

[0093] Figure 2 It is an applicable scene graph provided by an embodiment of the present invention.

[0094] Figure 3 It is a flow chart for implementing the joint optimization method provided by an embodiment of the present invention.

[0095] Figure 4 It is a structural diagram of the air-to-ground network service deployment and resource allocation system enabled by edge computing provided in an embodiment of the present invention.

[0096] Figure 5 It is a comparison chart of rewards obtained for different maximum tolerable task processing delays provided by an embodiment of the present invention and existing collaborative content caching and replacement decisions, computational offloading decisions, user resource allocation methods and transmission power control.

[0097] Figure 6 It is a comparison diagram of rewards obtained for different replacement application delays provided by the present invention and the existing collaborative content caching and replacement decision, computation offloading decision, user resource allocation method and transmission power control provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0099] In order to effectively solve the dynamic characteristics of the network and the uncertainty of information acquisition, the problem becomes quite difficult. The present invention reconstructs the problem into MDP. Since the variables involved are continuous, the MADDPG algorithm is used, which is a DRL algorithm designed specifically for solving problems with continuous variables and can support real-time online decision-making.

[0100] As an emerging technology, edge computing has been widely used in many industrial fields. The following are two specific industrial application examples:

[0101] Example 1: Intelligent traffic management system

[0102] The intelligent traffic management system uses edge computing technology to deploy edge computing nodes at key locations such as urban roads and transportation hubs to achieve real-time collection, processing and analysis of traffic data to improve traffic efficiency and safety.

[0103] 1. Data collection: Use cameras, sensors and other equipment deployed on the roadside to collect key data such as traffic flow, speed, and vehicle type in real time.

[0104] 2. Edge processing: The collected data is directly transmitted to a nearby edge computing node for processing, rather than being transmitted back to the cloud. The edge computing node uses built-in intelligent algorithms to analyze traffic data in real time and identify abnormal situations such as traffic congestion and accidents.

[0105] 3. Decision-making and response: Based on the analysis results, edge computing nodes can make autonomous decisions and trigger corresponding response measures, such as adjusting the timing of traffic lights and sending alarm information to the traffic management center.

[0106] 4. Collaborative optimization: Multiple edge computing nodes can collaborate with each other, share traffic data and analysis results, and further optimize the overall traffic flow.

[0107] Real-time: Through edge computing, the system can realize real-time processing and analysis of traffic data and quickly respond to traffic changes.

[0108] Low latency: Data does not need to be transmitted to the cloud, which reduces network latency and improves system response speed.

[0109] Privacy protection: Edge computing enhances data privacy protection capabilities, and sensitive data is processed locally, reducing the risk of leakage.

[0110] Example 2: Intelligent manufacturing production line

[0111] In the field of intelligent manufacturing, edge computing technology is widely used in production line monitoring, quality control, and predictive maintenance to improve production efficiency and product quality.

[0112] 1. Production line monitoring: Deploy edge computing nodes at key locations on the production line to collect real-time information such as production equipment operation data and product quality parameters.

[0113] 2. Data analysis and prediction: Edge computing nodes use built-in AI algorithms to analyze and process collected data, identify anomalies in the production process, and predict potential equipment failures or product quality issues.

[0114] 3. Autonomous decision-making and response: Based on the analysis results, edge computing nodes can make autonomous decisions and take corresponding measures, such as adjusting production parameters, triggering maintenance tasks, etc., to avoid production interruptions or product quality problems.

[0115] 4. Remote collaboration: Edge computing nodes can also work in collaboration with the cloud, uploading key data to the cloud for further analysis or backup, while receiving instructions or model updates from the cloud.

[0116] Efficiency: Edge computing reduces the amount of data transmission, reduces network bandwidth requirements, and improves data processing efficiency.

[0117] Flexibility: Edge computing nodes can be flexibly deployed and configured according to the actual needs of the production line to meet application requirements in different scenarios.

[0118] Intelligence: Combined with AI algorithms, edge computing realizes intelligent monitoring and predictive maintenance of the production process, improving the overall intelligence level of the production line.

[0119] These two examples demonstrate the specific application of edge computing in the fields of intelligent transportation and intelligent manufacturing, as well as the technical advantages and value it brings. As technology continues to develop and mature, edge computing will be widely used and promoted in more industrial fields.

[0120] like Figure 1 The embodiment of the present invention provides an edge computing-enabled air-to-ground network service deployment and resource allocation method, comprising the following steps:

[0121] Step 1: Establish a training network model. The training network of each intelligent agent includes the online network and the target network, and establish an air-ground network model enabled by edge computing. The online network includes the online Actor network and the online Critic network, and the target network includes the target strategy network Actor and the target value network Critic.

[0122] Step 2: Initialize the parameters in the network, including the parameters θ of the online Actor and Critic networks Q and θ μ , the parameters θ of the target Actor and Critic network Q′ and θ μ′ , the maximum number of training steps T, the number of samples extracted from the experience replay buffer mini-batch, the reward discount factor γ γ , the maximum number of rounds Epi, the noise ψ used for action exploration, the learning rates α and β used by the Actor and Critic networks, the soft update parameter τ, etc.;

[0123] Step 3: reset each agent in the environment to obtain the initial system state and summarize it into a state set S; set the time of each round t = t0;

[0124] Step 4: The agent passes through the Actor network Get action A k (t), and add noise ψ; and reconstruct the action, and transform the action A into k (t) Reconstructed into A k (t)=(A k (t)+1) / 2;

[0125] Step 5: Get the global action A(t) and execute it to get the reward and the state S(t+1) at the next moment; store the current experience tuple <S(t), A(t), r(t), S(t+1)> in the replay buffer pool;

[0126] Step 6: Determine whether the replay buffer is full. If so, randomly select mini-batch samples from the buffer to train the network. Otherwise, determine whether t is equal to T. If so, start again with step 4.

[0127] Step 7: Calculate the gradient of the current Critic network of the agent and update the current Actor network through gradient ascent; use the current Critic network to calculate the value of the action at time t, and use the target network to calculate the value of the action at time t+1, and then update the current Critic network through gradient descent;

[0128] Step 8: Use soft update technology to update the Actor network and Critic network in the target network;

[0129] Step nine, determine whether it converges, if so, get the final optimal solution, otherwise start from step four again.

[0130] Figure 2 is a scenario diagram to which the method of the present invention can be applied. The system consists of multiple UAVs forming an aerial infrastructure network, and each UAV is deployed with an edge cache server as an edge node to provide services for multiple IoT devices on the IRS. The set and number of drones are represented as and K, the set and number of IoT devices are and M. The present invention divides time into multiple time slots and regards each time slot as a discrete time unit for executing tasks and placing applications. Before each application is placed, each IoT device generates a random task that needs to be processed. However, since the processing power of these IoT devices is very limited, the generated tasks need to be offloaded to the aerial infrastructure network and executed by the corresponding applications deployed on the UAV. In order to maximize the success rate of task execution, the present invention considers installing the required applications on the edge server at the beginning of each time slot. In addition, deleting idle existing applications can free up space for deploying other applications, especially those that are frequently requested. At the same time, task processing is also achieved by deploying corresponding applications on the UAV.

[0131] In the relevant scenario, there are N types of tasks on the UAV corresponding to different applications, where the task set is composed of Indicates that each task n is described as Among them, S n is the amount of input data transferred by the task; n is the density of task processing, which represents the CPU cycles required to process each bit of data transmitted; n is the size of the application corresponding to the task; is the maximum tolerable delay in completing the task. n and λ n , the computational effort after the application requests upload can be expressed as C n =S n λ n .

[0132] like Figure 3 As shown, the method of collaborative task deployment, offloading decision, power control and resource allocation of the present invention includes the following steps:

[0133] Phase 1: The actual online network and the corresponding target network constitute the training model of each agent. The online network consists of the Actor network that executes the current strategy and the Critic network that evaluates the current state; the target network contains the corresponding target Actor network and target Critic network. Establish all the above network models;

[0134] Phase 2: Initialize the parameters in the model, including the parameters θ of the online Actor and Critic networks Q and θ μ , the parameters θ of the target Actor and Critic network Q′ and θ μ′, the maximum number of training steps T, the number of samples extracted from the experience replay buffer mini-batch, the reward discount factor γ, the maximum number of rounds Epi, the noise ψ used for action exploration, the learning rates α and β used by the Actor and Critic networks, the soft update parameter τ, etc.;

[0135] Phase 3: Reset each agent in the environment to obtain the initial system state and summarize it into a state set S.

[0136] Phase 4: Set the initial time of each round t=t0.

[0137] Stage 5: Agent passes through the policy network Get action A k (t), and promote the exploration performance by adding noise ψ.

[0138] Phase 6: Reconstruct the action, transform action A into k (t) Reconstructed into A k (t)=(A k (t)+1) / 2.

[0139] Stage 7: Determine whether k is equal to K. If not, return to stage 5, otherwise proceed to stage 8.

[0140] Stage 8: Get the global action A(t).

[0141] Stage 9: Execute action A(t) and obtain reward r(t) and the state S(t+1) at the next moment.

[0142] Stage 10: Store the current experience tuple <S(t),A(t),r(t),S(t+1)> into the playback buffer.

[0143] Stage 11: Determine whether the playback buffer is full. If so, randomly select mini-batch samples from the buffer to train the network. Otherwise, determine whether t is equal to T. If so, return to stage 3. Otherwise, execute stage 12.

[0144] Stage 12: Calculate the gradient corresponding to the current Critic network of the agent, and use the gradient ascent method to update the current Actor network.

[0145] Phase 13: The agent's action value at time t is evaluated by the current critic network, while the action value at time t+1 is predicted by the target network. After that, the current critic network is updated by executing the gradient descent algorithm.

[0146] Phase 14: Use soft update technology to update the Actor network and Critic network in the target network.

[0147] Stage 15: Determine whether it converges. If it converges, the final optimal solution is obtained, otherwise jump back to stage 3.

[0148] In a preferred embodiment of the present invention, the state of the agent and the global state in the third stage are respectively expressed as:

[0149] S k (t) = {r k (t),U k (t-1),E k (t),L m,k (t),PL k (t),A k (t)},

[0150] S(t)={S1(t),S2(t),...,S K (t)},

[0151] where r k (t) = {r n,k (t)}, represents the task request sent by the IoT device at the beginning of each time slot; U k (t-1) = {n n,k (t-1)}, represents the set of existing contents on UAV k in time slot t-1; It represents the path loss coefficient under line-of-sight and non-line-of-sight transmission; L m,k (t), represents the Euclidean distance matrix between IoT device m and UAV k; PL k (t), A represents the additional path loss matrix between IoT device m and UAVk; k (t), Indicates the access situation in the system, S k (t), is the state of UAV k.

[0152] In a preferred embodiment of the present invention, in the stage 5, the obtained action A k (t) is expressed as follows:

[0153]

[0154] Among them, Δ k (t) = {δ n,k (t)}, is the mission placement strategy for UAV k; is the replacement strategy of UAV k; P k (t) = {pm,k (t)}, is the transmission power between UAV k and the associated IoT device; F k (t) = {f m,k (t)}, is the computing resource allocation strategy of UAV k.

[0155] In a preferred embodiment of the present invention, in the sixth stage, the process of the reconstruction action is expressed as follows:

[0156] (1) For binary task placement decision Δ k (t) and replacement decision We will simply use rounding to get the corresponding binary discrete actions.

[0157] (2) For integer transmit power P k (t) and computing resources F k (t), we compare it with the maximum processing power p of the IoT device m and UAV maximum processing capacity f m Multiply them to get the actual power matrix and the computing resource matrix allocated to the UAV respectively.

[0158] (3) Through simple mathematical transformation A k (t)=(A k (t)+1) / 2, changing the range of the intrinsic action from [-1,1] to [0,1].

[0159] In a preferred embodiment of the present invention, in the above-mentioned stage, the calculation formula of reward is as follows:

[0160]

[0161] Among them, r suc (t) represents the system's task execution success rate, which is our objective function.

[0162] In a preferred embodiment of the present invention, in the stage 12, the updating process of the Actor network is as follows:

[0163] The current gradient of the Critic network is as follows

[0164]

[0165] The average gradient is given by

[0166]

[0167] Update the current Actor network through gradient ascent as follows

[0168]

[0169] In a preferred embodiment of the present invention, in the stage thirteen, the updating process of the current Critic network is as follows:

[0170] The target TD can be given by

[0171]

[0172] Then, the TD error can be given by

[0173]

[0174] Then perform gradient descent to update the parameters of the current Critic network as follows

[0175]

[0176] In a preferred embodiment of the present invention, in the stage 14, the soft update formula of the target network is as follows:

[0177]

[0178] in Represents the parameters of the current Actor network, Represents the parameters of the target Actor network, Represents the parameters of the current Critic network, Represents the parameters of the target Critic network, τ∈[0,1].

[0179] like Figure 4 As shown, an edge computing-enabled air-to-ground network service deployment and resource allocation method provided by an embodiment of the present invention includes:

[0180] The network model building module builds a training network model. The training network of each intelligent agent includes the online network and the target network, and builds an air-ground network model enabled by edge computing. The online network includes the online Actor network and the online Critic network, and the target network includes the target strategy network Actor and the target value network Critic.

[0181] Parameter initialization module, which initializes the parameters in the network, including the parameters θ of the online Actor and Critic networks Q and θ μ , the parameters θ of the target Actor and Critic network Q′ and θ μ′, the maximum number of training steps T, the number of samples extracted from the experience replay buffer mini-batch, the reward discount factor γ γ , the maximum number of rounds Epi, the noise ψ used for action exploration, the learning rates α and β used by the Actor and Critic networks, the soft update parameter τ, etc.;

[0182] The state reset module resets each agent in the environment to obtain the initial system state and summarizes it into a state set S; sets the time of each round t = t0;

[0183] Action reconstruction module, the agent passes through the Actor network Get action A k (t), and add noise ψ; and reconstruct the action, and transform the action A into k (t) Reconstructed into A k (t)=(A k (t)+1) / 2;

[0184] The cache pool module obtains the global action A(t) and executes it to obtain the reward and the state S(t+1) at the next moment; the current experience tuple <S(t), A(t), r(t), S(t+1)> is stored in the replay cache pool;

[0185] The judgment module determines whether the replay buffer is full. If so, randomly select mini-batch samples from the buffer to train the network. Otherwise, it determines whether t is equal to T. If so, it starts executing step 4.

[0186] The network update module calculates the gradient of the current Critic network of the agent and updates the current Actor network through gradient ascent; uses the current Critic network to calculate the value of the action at time t, and uses the target network to calculate the value of the action at time t+1, and then updates the current Critic network through gradient descent;

[0187] Use soft update technology to update the Actor network and Critic network in the target network;

[0188] The convergence loop module determines whether it has converged. If so, the final optimal solution is obtained. Otherwise, it starts from step 4 again.

[0189] 1. Specific application fields or related products of the present invention.

[0190] In order to elaborate on the joint optimization of computing resources and power of UAV when the Internet of Vehicles is combined with UAV and applied on IoT devices, the present invention provides two specific application embodiments, including key details of the implementation scheme.

[0191] Application Example 1: Traffic Monitoring and Management

[0192] 1) System initialization:

[0193] In the traffic monitoring and management system, the Actor and Critic networks are first deployed. The network parameters are initialized, including learning rate, discount factor, experience replay buffer size, etc. At the same time, UAVs are deployed at key traffic nodes to monitor traffic flow and vehicle behavior in real time.

[0194] 2) MADDPG agent interacts with the environment:

[0195] The intelligent agent consists of multiple agents, each responsible for different tasks, such as monitoring area allocation, flight path planning, energy management, and task scheduling. The intelligent agent generates coordinated actions based on real-time traffic data, UAV status, and monitoring requirements to optimize the coverage and efficiency of traffic monitoring.

[0196] 3) Action execution and reward acquisition:

[0197] After executing the actions generated by the agent, the system obtains immediate rewards based on the effectiveness of traffic monitoring. The reward signal is calculated based on indicators such as monitoring coverage, accuracy of data collection, and response time. The system moves from the current state to the next state, involving changes in traffic flow, updates to monitoring requirements, and new traffic events.

[0198] 4) Optimization and update:

[0199] Based on the collected experience data (state, action, reward and next state), the agent updates its Actor network through the MADDPG algorithm. The agent learns the optimization strategy that adapts to the needs of traffic monitoring, while focusing on improving monitoring efficiency and reducing the energy consumption of UAV.

[0200] 5) Performance evaluation and tuning:

[0201] Evaluate the performance of the MADDPG algorithm through simulation experiments or field tests. Analyze key indicators such as traffic monitoring coverage, data accuracy, and response speed, and compare with traditional methods. Based on the evaluation results, adjust and optimize the algorithm parameters to further improve the performance and efficiency of traffic monitoring. The MADDPG algorithm allows multiple agents to learn optimal strategies while considering the behavior of other agents. This collaborative optimization method enables UAVs to achieve more efficient and intelligent monitoring tasks in complex traffic monitoring environments.

[0202] Application Example 2: Intelligent Logistics Distribution

[0203] 1) System initialization:

[0204] In the intelligent logistics distribution system, the Actor and Critic networks are first deployed. The network parameters are initialized, including the learning rate, discount factor, experience replay buffer size, etc. At the same time, UAVs are deployed at distribution centers and key distribution nodes to perform the last mile delivery task.

[0205] 2) MADDPG agent interacts with the environment:

[0206] The intelligent agent is composed of multiple agents, each of which is responsible for different tasks, such as path planning, energy management, real-time task scheduling, etc. The intelligent agent generates coordinated actions based on information such as delivery demand, traffic conditions, and UAV status to optimize delivery efficiency and response speed.

[0207] 3) Action execution and reward acquisition:

[0208] After executing the actions generated by the agent, the system obtains immediate rewards based on the completion of the delivery task. The reward signal is calculated based on indicators such as delivery success rate, punctuality rate, energy consumption, etc. The system moves from the current state to the next state, involving the update of the delivery task, the change of UAV status, and new delivery requests.

[0209] 4) Optimization and update:

[0210] Based on the collected experience data (state, action, reward and next state), the agent updates its Actor network through the MADDPG algorithm. The agent learns the optimization strategy that adapts to the needs of intelligent logistics distribution, while focusing on improving distribution efficiency and reducing energy consumption.

[0211] 5) Performance evaluation and tuning:

[0212] Evaluate the performance of the MADDPG algorithm through simulation experiments or field tests. Analyze key indicators such as the completion rate, punctuality rate, and energy efficiency of delivery tasks and compare them with traditional methods. According to the evaluation results, adjust and optimize the algorithm parameters to further improve the performance and customer satisfaction of intelligent logistics distribution. The MADDPG algorithm allows multiple agents to learn optimal strategies while considering the behavior of other agents. This collaborative optimization method enables UAVs to achieve more efficient and intelligent delivery tasks in complex logistics and distribution environments.

[0213] Figure 5The relationship between the maximum tolerable task processing delay and the reward is described. When the maximum tolerable task processing delay is set to 0.01s, no matter how the three algorithms optimize the parameters, they cannot complete the task within the given delay and can only obtain the minimum reward that does not meet the maximum tolerable task processing delay. When the maximum tolerable task processing delay setting gradually increases, the system does not need to perform precise optimization to meet the delay requirements, so the reward values ​​of each algorithm show an upward trend. The proposed algorithm has the best performance.

[0214] Figure 6 The relationship between the application replacement delay and the reward is explained. When the application replacement delay is a minimum value, for the entire system, there is a certain room for optimizing the task transmission delay and task processing delay. At the same time, the success rate of task execution has been improved to a certain extent compared with the default parameters. When the application replacement delay gradually increases, the room left by the system for optimization decisions becomes smaller, so the reward value of each algorithm decreases. When the application replacement delay is set to 0.4s, the replacement delay is too long, so no matter how the optimization is performed, the maximum tolerable task processing delay cannot be met. The performance of the proposed algorithm is the best compared with the baseline algorithm.

[0215] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0216] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. An edge computing-enabled air-to-ground network service deployment and resource allocation method, characterized in that: Establish a training network model including online and target networks, initialize network parameters, obtain and reconstruct actions through the Actor network, execute actions and store experience, train the network based on the experience replay mechanism, use gradient ascent and descent to update the Actor and Critic networks, use soft update technology to update the target network, and iterate until convergence to obtain the optimal solution.

2. The edge computing-enabled air-to-ground network service deployment and resource allocation method as described in claim 1, characterized in that: The following steps are involved: Step 1: Establish a training network model. The training network of each intelligent agent includes the online network and the target network, and establish an air-ground network model enabled by edge computing. The online network includes the online Actor network and the online Critic network, and the target network includes the target strategy network Actor and the target value network Critic. Step 2: Initialize the parameters in the network, including the parameters θ of the online Actor and Critic networks Q and θ μ , the parameters θ of the target Actor and Critic network Q′ and θ μ′ , the maximum number of training steps T, the number of samples extracted from the experience replay buffer mini-batch, the reward discount factor γ γ , the maximum number of rounds Epi, the noise ψ used for action exploration, the learning rates α and β used by the Actor and Critic networks, the soft update parameter τ, etc.; Step 3: reset each agent in the environment to obtain the initial system state and summarize it into a state set S; set the time of each round t = t0; Step 4: The agent passes through the Actor network Get action A k (t), and add noise ψ; and reconstruct the action, and transform the action A into k (t) Reconstructed into A k (t)=(A k (t)+1) / 2; Step 5: Get the global action A(t) and execute it to get the reward and the state S(t+1) at the next moment; store the current experience tuple <S(t), A(t), r(t), S(t+1)> in the replay buffer pool; Step 6: Determine whether the replay buffer is full. If so, randomly select mini-batch samples from the buffer to train the network. Otherwise, determine whether t is equal to T. If so, start again with step 4. Step 7: Calculate the gradient of the current Critic network of the agent and update the current Actor network through gradient ascent; use the current Critic network to calculate the value of the action at time t, and use the target network to calculate the value of the action at time t+1, and then update the current Critic network through gradient descent; Step 8: Use soft update technology to update the Actor network and Critic network in the target network; Step nine: determine whether it has converged. If so, the final optimal solution is obtained. Otherwise, start again from step four.

3. The edge computing-enabled air-to-ground network service deployment and resource allocation method as described in claim 2, characterized in that: The state of the agent and the global state in step 3 are respectively expressed as: S k (t)={r k (t),U k (t-1),E k (t),L m,k (t),PL k (t),A k (t)}, S(t)={S1(t),S2(t),...,S K (t)}. in Represents the task request sent by the IoT device at the beginning of each time slot; represents the set of existing contents on UAV k in time slot t-1; Represents the path loss under line-of-sight and non-line-of-sight transmission; represents the Euclidean distance matrix between IoT device m and UAV k; represents the additional path loss matrix between IoT device m and UAV k; Indicates the access status in the system.

4. The edge computing-enabled air-to-ground network service deployment and resource allocation method as described in claim 2, characterized in that: In step 4, the obtained action A k (t) is expressed as follows: in, is the mission placement strategy for UAV k; is the task replacement strategy of UAV k; is the transmission power between UAV k and the associated IoT device m; is the computing resource allocation strategy of UAV k.

5. The edge computing-enabled air-to-ground network service deployment and resource allocation method as described in claim 2, characterized in that: In step 4, the process of reconstructing the action is as follows: (1) For binary task placement decision Δ k (t) and replacement decision Use the rounding method to get the corresponding binary discrete action; (2) For integer transmit power P k (t) and computing resources F k (t), and compare it with the maximum processing power p of IoT devices m and UAV maximum processing capacity f m Multiply them to obtain the actual power matrix and the computing resource matrix allocated to the UAV respectively; (3) Through simple mathematical transformation A k (t)=(A k (t)+1) / 2, changing the range of the intrinsic action from [-1,1] to [0,1].

6. The edge computing-enabled air-to-ground network service deployment and resource allocation method as described in claim 2, characterized in that: In step 5, the calculation formula of reward is as follows: Among them, r suc (t) represents the task execution success rate of the system, that is, the objective function.

7. The edge computing-enabled air-to-ground network service deployment and resource allocation method as described in claim 1, characterized in that: In step 7, the updating process of the policy network is as follows: The current gradient of the Critic network is as follows The average gradient is given by Update the current Actor network through gradient ascent as follows 8. The edge computing-enabled air-to-ground network service deployment and resource allocation method as described in claim 2, characterized in that: In step 7, the updating process of the current value network is as follows: The target TD is as follows The TD error is as follows: Perform gradient descent to update the parameters of the current Critic network as follows 9. The edge computing-enabled air-to-ground network service deployment and resource allocation method as described in claim 1, characterized in that: In step eight, the soft update formula of the target network is as follows: in, Represents the parameters of the current Actor network, Represents the parameters of the target Actor network, Represents the parameters of the current Critic network, Represents the parameters of the target Critic network, τ∈[0,1].

10. An edge computing-enabled air-to-ground network service deployment and resource allocation system according to the edge computing-enabled air-to-ground network service deployment and resource allocation method according to any one of claims 1 to 9, characterized in that: include: The network model building module builds a training network model. The training network of each intelligent agent includes the online network and the target network, and builds an air-ground network model enabled by edge computing. The online network includes the online Actor network and the online Critic network, and the target network includes the target strategy network Actor and the target value network Critic; Parameter initialization module, which initializes the parameters in the network, including the parameters θ of the online Actor and Critic networks Q and θ μ , the parameters θ of the target Actor and Critic network Q′ and θ μ′ , the maximum number of training steps T, the number of samples extracted from the experience replay buffer mini-batch, the reward discount factor γ γ , the maximum number of rounds Epi, the noise ψ used for action exploration, the learning rates α and β used by the Actor and Critic networks, the soft update parameter τ, etc.; The state reset module resets each agent in the environment to obtain the initial system state and summarizes it into a state set S; sets the time of each round t = t0; Action reconstruction module, the agent passes through the Actor network Get action A k (t), and add noise ψ; and reconstruct the action, and transform the action A into k (t) Reconstructed into A k (t)=(A k (t)+1) / 2; The cache pool module obtains the global action A(t) and executes it to obtain the reward and the state S(t+1) at the next moment; the current experience tuple <S(t), A(t), r(t), S(t+1)> is stored in the replay cache pool; The judgment module determines whether the replay buffer is full. If so, randomly select mini-batch samples from the buffer to train the network. Otherwise, it determines whether t is equal to T. If so, it starts with step 4. The network update module calculates the gradient of the current Critic network of the agent and updates the current Actor network through gradient ascent; uses the current Critic network to calculate the value of the action at time t, and uses the target network to calculate the value of the action at time t+1, and then updates the current Critic network through gradient descent; Use soft update technology to update the Actor network and Critic network in the target network; The convergence loop module determines whether it has converged. If so, the final optimal solution is obtained. Otherwise, execution starts from step 4 again.

Images