Air-ground network optimization method and system based on MEC and digital twinning

Abstract

The invention belongs to but not limited to the technical field of communication, and particularly relates to an air-to-ground network optimization method and system based on MEC and digital twinning, and the method comprises the steps: carrying out the learning and decision-making of an intelligent agent through employing a main strategy network and a commentator network of deep reinforcement learning; initializing parameters of a strategy network and an evaluation network, and setting training related hyper-parameters; the intelligent agent interacts with the environment according to the current strategy, executes an action and updates a state; an experience playback buffer area is used for storing experience, and updating is carried out when the capacity is reached so as to keep the latest learning experience; performing parameter updating on the main strategy network and the commentator network by adopting loss function calculation and a gradient descent method; soft updating of strategy network and value network parameters in the target network is carried out regularly to ensure learning stability; training is continued until the strategy converges, and then task allocation and trajectory planning are completed by applying the strategy.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of communication technology, and in particular relates to an air-ground network optimization method and system based on MEC and digital twins. Background Art

[0002] In recent years, with the surge in the number of IoT devices, traditional ground networks are increasingly unable to meet a wide range of needs and are also facing many challenges: (1) Delay sensitivity: Many smart IoT applications have strict requirements on service response time, that is, low latency requirements. However, due to the limitations of device size and capabilities, terminal devices often find it difficult to meet these stringent latency requirements. (2) Energy-intensive applications: Smart applications are usually energy-intensive or computationally intensive and require a lot of energy. This leads to a significant reduction in the battery life of user terminals or IoT devices. (3) Scarcity of wireless resources: With the access of a large number of smart terminals and IoT devices, limited wireless spectrum resources are becoming increasingly tight. Current networks face tremendous pressure in accommodating these rapidly increasing terminal devices. Against this background, the concept of air-ground integrated networks has emerged. By combining different layers to provide greater coverage, higher throughput, flexible deployment, and strong resilience. Among them, the integration of drones brings unique value to air-ground integrated networks. As a typical airborne device, drones have become a key tool for performing various intelligent tasks, such as real-time image processing, environmental monitoring, and emergency response. However, in order for the air-ground integrated network to operate efficiently, the large amount of data generated by drones during missions must be processed and analyzed. Driven by this demand, MEC technology has been introduced into air-ground integrated networks. MEC can significantly reduce data processing latency and improve system response speed by deploying computing power at the edge of the network closer to the data source. Therefore, deploying MEC on drones can bring data processing closer to the data source, thereby reducing data transmission latency and bandwidth requirements. This deployment method allows drones to perform real-time data analysis and decision-making during mission execution.

[0003] There are still some challenges that need to be addressed when combining MEC and drones into an integrated air-ground network architecture.

[0004] 1) Energy consumption and equipment life issues

[0005] When MEC and drones are combined, drones need to perform intensive computing and data processing during missions. Especially during task offloading and resource allocation, the high consumption of computing resources increases significantly. The hardware resources and expected service life of drones are limited, and frequent computing and data storage activities will accelerate the wear of hardware and shorten its life. The large amount of data writing and storage operations involved in mission execution not only poses a challenge to the immediate performance of drones, but may also cause their performance to degrade or even fail. Therefore, frequent task execution and data storage activities in MEC systems limit the reliability and durability of drones.

[0006] 2) Processing capacity issues

[0007] During the execution of UAV missions, intensive real-time data stream processing and decision making are required, which poses significant challenges to its computing power. Limited by the hardware resources of UAVs such as central processing unit (CPU) and random access memory (RAM), its ability to handle complex tasks is challenged. Since the energy supply of UAVs mainly relies on batteries, its ability to continuously operate in high-load computing tasks is limited. In addition, frequent task offloading and resource allocation decisions increase the burden on computing resources, resulting in delays in data processing and reduced system performance.

[0008] 3) Real-time data update problem

[0009] Drones need to frequently acquire and process data during dynamic mission execution, which places high demands on the real-time data processing capabilities of the MEC system. Data transmission delays, data consistency issues in distributed MEC environments, and limited communication bandwidth limit the data transmission rate, which in turn affects the real-time update of data. At the same time, frequent data transmission and processing increase the energy consumption of drones and may shorten their battery life. In addition, the reliability of the system in the face of environmental interference is also a key challenge to ensure the continuity and accuracy of data updates.

[0010] 4) Dynamic optimization problem

[0011] In drone-based and MEC-based scenarios, the environment is often highly dynamic and the goals are often defined as long-term objectives. In this case, we need to design more performant algorithms that achieve goal optimization with low complexity and good performance.

[0012] In view of the above analysis, the technical problems that need to be solved urgently in the existing technology are: the existing literature does not consider the resource allocation problem caused by the limited local storage resources of drones, nor does it deeply explore the optimization of UEs and drone association strategies, offloading mode selection and task offloading ratio. Although some literature considers data allocation and offloading optimization problems, it ignores the economic expenses incurred in the process. In the scenario of air-ground integrated network enabling MEC, other devices are often highly dynamic, which requires higher performance of the algorithm and is therefore difficult to solve. Summary of the invention

[0013] In response to the problems existing in the prior art, the present invention provides an air-ground network optimization method and system based on MEC and digital twins.

[0014] The present invention is implemented as follows: an air-ground network optimization method based on MEC and digital twins, using a main policy network and a critic network of deep reinforcement learning (DRL) to learn and make decisions for the intelligent agent; initializing the parameters of the policy network and the evaluation network, and setting training-related hyperparameters; the intelligent agent interacts with the environment according to the current strategy, performs actions and updates the state; uses an experience replay buffer to store experience, and updates it when the capacity is reached to keep the latest learning experience; uses loss function calculation and gradient descent method to update the parameters of the main policy network and the critic network; regularly performs soft updates on the parameters of the policy network and the value network in the target network to ensure learning stability; continues training until the strategy converges, and then applies the strategy to complete task allocation and trajectory planning.

[0015] Furthermore, the air-ground network optimization method based on MEC and digital twins specifically includes:

[0016] S101. Initialize the parameters θ of the agent’s main policy network and critic network j and ω j , the parameters of the target policy network and the critic network and Number of rounds MP, maximum number of training steps T max , initialize the learning rates α and β corresponding to the critic network and the policy network, the discount factor γ, initialize the playback buffer size W, the mini-batch size N, and the noise Ψ(t) used for action exploration; initialize the network layout parameters, such as the number of UEs I, the number of drones J and other parameters.

[0017] S102: Initialize the agent state, the agent interacts with the environment, and the main strategy network generates actions according to the current strategy.

[0018] S103. The agent executes the actions generated by the main strategy network, obtains rewards, and updates the status.

[0019] S104: Store the experience tuple into the experience playback buffer. When the buffer reaches the upper limit of the capacity, the latest experience is introduced by overwriting the oldest experience data.

[0020] S105. Update the parameters of the main strategy network and the critic network.

[0021] S106. Calculate the loss function of the critic network based on the TD target and the value function predicted by the critic network, extract samples from the experience replay buffer, and use the gradient descent method to update the target policy network and the evaluation network.

[0022] S107. Update the parameters of the main policy network and the critic network using small batch experience samples.

[0023] S108. Adopting a soft update mechanism to update the parameters of the policy network and the value network in the target network.

[0024] S109, iterate the training until the algorithm converges stably, and apply the strategy to the intelligent agent to achieve optimal task allocation and trajectory planning.

[0025] Further, in S102, the state of the agent is initialized, the agent interacts with the environment, and the main strategy network generates actions based on the current strategy. The state of the agent is expressed as:

[0026]

[0027] where u j (t-1) = (X j (t-1),Y j (t-1),H j (t-1)), represents the three-dimensional coordinates of drone j at the end of time slot t-1, corresponding to the starting point of time slot t; D(t) = {D i (t)}, represents the task input data size of the UE in time slot t; λ(t)={λ i (t)}, represents the task processing density of UE in time slot t; represents the computational resource allocation within time slot t; represents the remaining energy of UAV j at the end of time slot t; It represents the time life index of each UAV, and stipulates that the initial life of each UAV is 0.

[0028] Further, in S103, the agent executes the action generated by the main strategy network, obtains a reward, and updates the state. The calculation formula of the reward in the state update is as follows:

[0029]

[0030] In the above formula Represents the immediate reward obtained by each agent in the system after performing an action.

[0031] Further, the S105: updating the main strategy network and the critic network parameters; updating the current strategy network by gradient ascent is as follows:

[0032]

[0033] Further, the S106: calculate the loss function of the critic network according to the TD target and the value function predicted by the critic network, extract samples from the experience replay buffer, and use the gradient descent method to update the target policy network and the evaluation network. The TD target can be given by the following formula:

[0034]

[0035] The loss function can be given by:

[0036]

[0037] Then update the parameters w of the current value network of the main value network through gradient descent j as follows:

[0038]

[0039] Further, in S108: a soft update mechanism is used to update the parameters of the policy network and the value network in the target network; the soft update formula is as follows:

[0040]

[0041]

[0042] where θ j Represents the parameters of the current policy network, represents the parameters of the target policy network, w j Represents the parameters of the current value network, Represents the parameters of the target value network, χ∈[0,1].

[0043] Another object of the present invention is to provide an air-ground network optimization system based on MEC and digital twins to implement the air-ground network optimization method based on MEC and digital twins, comprising:

[0044] System initialization module, used to initialize the parameters of the deep deterministic policy gradient algorithm, including setting the number of rounds MP and the maximum number of training steps T max, initialize the learning rates α and β corresponding to the critic network and the policy network, the discount factor γ, initialize the playback buffer size W, and the mini-batch size N;

[0045] Network construction module, used for motion exploration noise Ψ(t); initialization of network layout parameters, such as the number of UEs I, the number of drones J and other network layout parameters;

[0046] The agent module is used to generate actions based on the current network state at the beginning of each cycle, with the ability to add exploratory noise to these behaviors in order to introduce a certain amount of randomness in the execution process;

[0047] Action execution module, used to execute resource allocation and task offloading strategies;

[0048] The reward acquisition module is used to perform actions and calculate immediate rewards, evaluate rewards based on the long-term average utility of all devices in the system, and transfer the system from the current state to the next state;

[0049] The experience replay module is used to store each system state, action executed, reward obtained, and experience tuple of the next state;

[0050] The data sampling module is used to extract mini-batch experiences from the stored experience playback module for learning;

[0051] The network update module is used to update the main policy network and the main value network according to the data of the experience playback module, including a parameter optimization unit, which uses the gradient ascent method and the gradient descent method to adjust the network parameters;

[0052] The parameter update module is used to synchronize the parameter updates of the main network to the target strategy network and the target value network. The soft update strategy is adopted so that the parameters of the target network are the weighted average of the parameters of the main network, and this synchronization process is realized through the parameter synchronization unit.

[0053] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the air-ground network optimization method based on MEC and digital twins.

[0054] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the air-ground network optimization method based on MEC and digital twins.

[0055] Another object of the present invention is to provide an information data processing terminal, which includes the air-ground network optimization system based on MEC and digital twins.

[0056] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0057] First, the present invention improves network coverage and service quality. In the air-ground integrated network architecture, drones, as key aerial nodes, can provide wider network coverage for remote areas. Through the drone-assisted edge caching system, we can efficiently process computing tasks, reasonably allocate computing resources and task division, thereby making up for the lack of ground base station coverage. The integration of the present invention is of great significance for enhancing network coverage and service quality in an air-ground integrated environment.

[0058] The present invention combines MEC with an air-ground integrated network assisted by drones, providing a more efficient and scalable solution for network services. The powerful network coverage of drones can improve the limitations of processing tasks only through ground base stations. By combining the MEC system with the air-ground integrated network assisted by drones, faster task processing can be achieved and task processing latency can be reduced. This combination provides greater room for the scalability of network services, can better cope with the growing demand for data processing, and can meet the mission requirements of remote areas, making the entire system more sustainable.

[0059] The present invention realizes real-time monitoring and dynamic mapping of the physical network environment by adopting digital twin technology, thereby improving the efficiency and response speed of network management. By mapping the physical network to the digital network, comprehensive monitoring and real-time mapping of the network status are realized, thereby optimizing network performance and resource allocation. The application of digital twin technology not only improves the intelligence level of the network, but also provides strong support for predictive maintenance and fault diagnosis of the network.

[0060] The present invention provides an optimization strategy for the collaborative work between drones and ground base stations based on the actual dynamic environment. By comparing with experimental data, it is confirmed that the dynamic resource allocation method is more accurate than the traditional static resource allocation method, can better simulate the actual environment, is easy to operate, has higher real-time performance, is closer to the real scene, and is conducive to network optimization and system performance improvement.

[0061] Second, the significant technical progress achieved by the present invention is to realize a joint optimization method of an air-ground integrated network combined with an edge computing system and a digital twin assisted by a drone, which has made significant progress in the following key aspects:

[0062] 1) Efficient resource allocation:

[0063] This method significantly improves resource utilization in drone-assisted air-ground integrated networks by optimizing resource allocation. This includes more efficient allocation of bandwidth resources to drones and base stations to achieve efficient collaborative processing of tasks. This optimization not only improves system performance but also reduces energy consumption.

[0064] 2) Reasonable task division strategy:

[0065] By properly distributing the input data, an intelligent task division strategy is implemented. The system can dynamically adjust task processing according to the capabilities of the drone and network conditions, intelligently adjusting whether the processing is done by the drone or the base station, thereby adapting to the dynamic changes of the network. This enhances the stability of the network and the user experience.

[0066] 3) Integration of reinforcement learning:

[0067] By combining the DRL algorithm with MEC, the drone-assisted air-ground integrated network can analyze data in real time and autonomously adapt to environmental changes, thereby making optimal decisions under complex and dynamic conditions.

[0068] 4) System utility optimization:

[0069] The reward mechanism included in the method focuses on reducing the total system utility of drones and ground equipment, optimizing energy consumption through the reward mechanism, improving energy efficiency, and being environmentally friendly.

[0070] 5) Improvement of system stability and reliability:

[0071] By accurately calculating the immediate feedback after the drone performs a mission and optimizing its state transition process, this method significantly improves the stability and reliability of the system when facing large-scale data processing and high-density requests.

[0072] 6) Network’s autonomous learning and optimization capabilities:

[0073] This method enables the system to continuously optimize its decision-making process through continuous iterative training and experience-based network updates, thereby improving overall performance.

[0074] 7) Application of digital twin technology:

[0075] This method uses digital twin technology to build a highly accurate virtual network mapping system, which realizes real-time monitoring and dynamic mapping of the physical network environment, and improves the efficiency and response speed of network management.

[0076] The combined application of these technologies has significantly improved the performance of drone-assisted MEC network systems in many aspects, including performance, energy efficiency, stability, and adaptability, effectively supporting the complex needs of modern computing.

[0077] Third, the edge computing system optimization method based on the combination of air-ground integrated network and deep reinforcement learning provided by the present invention is based on the use of mathematical models to guide the behavior and learning process of the system. The technical effects brought by these mathematical models can be discussed based on their characteristics:

[0078] 1) Calculation of instant rewards

[0079] This method focuses on improving the overall utility of all devices in the system through the calculation of instant rewards.

[0080] Energy efficiency optimization: By directly linking the reward mechanism to system utility, the system is encouraged to explore strategies to reduce energy consumption and achieve higher energy efficiency.

[0081] Cost savings: As system utility continues to be optimized, operating costs can be significantly reduced in the long run, especially in edge computing environments with high energy consumption.

[0082] 2) Update of main policy network and critic network

[0083] Using randomly sampled small batches of empirical data, the current policy network is updated using the gradient ascent method.

[0084] Strategy optimization: The system learns and adopts more efficient decision-making strategies by continuously adjusting the parameters of the policy network.

[0085] Improved responsiveness: By using a small batch data update strategy, the network can quickly respond to environmental changes, improving the system's ability to adjust in real time.

[0086] 3) Loss function of TD target and critic network

[0087] The loss function is calculated based on the temporal difference (TD) target and the Q-value predicted by the critic network and updated by gradient descent.

[0088] Learning stability: Using the TD target as the basis for learning helps to balance the learning process and reduce learning fluctuations caused by large prediction errors.

[0089] Performance optimization: By accurately calculating the loss function and updating the network parameters in a timely manner, the system can improve the accuracy of decision-making and execution efficiency.

[0090] 4) Parameter update formula

[0091] Describes how the main policy network and the critic network update the parameters of the target network, which involves the synchronization of the current network and the target network parameters.

[0092] Strategy gradually approaches: By gradually adjusting the target network parameters, the system can smoothly adopt new strategies and avoid performance fluctuations caused by sudden changes in strategies.

[0093] Continuous learning and adaptation: This continuous parameter update mechanism enables the system to adapt to environmental changes in the long term and maintain the continuity of learning and adaptation.

[0094] The application of the mathematical model provided by the present invention not only improves the operating efficiency and decision-making quality of the system, but also enhances the system's adaptability to environmental changes and long-term stability, and realizes real-time monitoring and mapping of network status by integrating 3D trajectory optimization, offloading and caching strategy optimization, and digital twin technology. These technical effects are crucial for processing large amounts of data and high-frequency interactions in modern edge computing environments.

[0095] Fourth, the air-ground network optimization method and system based on MEC and digital twins provided by the present invention adopt deep reinforcement learning technology to optimize the performance of the network through the interaction between the intelligent agent and the environment.

[0096] Initialize the state of the agent, which includes multiple variables, such as the amount of task input data, task processing density, task processing deadline, estimated computing resource allocation, initial remaining energy of each drone, horizontal distance between IoT device and drone, flight altitude of drone, environment-related parameters, and path loss parameters. These variables together define the environment state of the agent at a specific moment, which in turn affects the decision-making of the agent.

[0097] The agent performs tasks according to the actions determined by the main strategy network. After performing the corresponding actions, the agent obtains corresponding rewards according to the results obtained, and updates and transfers the state according to the rewards and execution results. The calculation of rewards is based on the long-term average utility of all devices in the system, which is the core goal of system design, and the purpose is to minimize the total utility of the entire system.

[0098] Adjusting the parameters of the policy network through the gradient ascent method helps the policy network generate better actions. By optimizing the policy network, the performance of the entire system can be further optimized.

[0099] Using the temporal difference (TD) target and the value function predicted by the critic network to calculate the loss function and updating the parameters of the target policy network and the critic network by gradient descent is a key step in updating the value function in deep reinforcement learning. This process guides the continuous improvement of the strategy by improving the accuracy of the prediction, thereby optimizing the operating performance of the entire system.

[0100] The soft update method updates the target network. The soft update method is a progressive parameter update technology that smoothly adjusts the target network by calculating the weighted average of the target network parameters and the current network parameters. This method helps to reduce the drastic fluctuations that may occur during the learning process and ensures the smoothness and stability of the entire learning process.

[0101] These steps and the application of mathematical models have led to significant technological advances:

[0102] Policy Optimization: Deep reinforcement learning enables the system to autonomously learn and optimize policies to adapt to dynamic changes in the drone-assisted air-ground integrated network environment.

[0103] Efficient resource utilization: By optimizing task offloading and resource allocation, it ensures that the limited resources in the drone network are efficiently utilized, especially in 3D trajectory optimization and data offloading and caching strategies.

[0104] Minimizing system utility: The system supports green communications and reduces environmental impact by optimizing long-term average utility, which is particularly important for energy management of drones and energy efficiency optimization of MEC.

[0105] System stability: The soft update method ensures a smooth learning process and reduces the risk of system instability caused by rapid parameter updates, which is critical for maintaining the accuracy of digital twin mapping and network status monitoring.

[0106] Real-time monitoring of network status: The application of digital twin technology realizes real-time mapping and monitoring of the physical network status, providing the system with in-depth insights into the network operation status, so that policy optimization and resource allocation can be based on the most accurate network information.

[0107] These advances demonstrate the potential of deep reinforcement learning in building intelligent and efficient air-ground integrated communication networks, especially in complex network systems integrated with drone assistance, MEC support and digital twin technology.

[0108] Fifth, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0109] 1. Filling the technical gap:

[0110] The technical solution of the present invention provides a new solution for the field of air-ground network optimization by integrating MEC and digital twin technology. This fusion method utilizes the low latency and high bandwidth characteristics of MEC and the mapping and simulation capabilities of digital twins in virtual space to the physical world, solving the limitations of the single network architecture in the existing technology.

[0111] 2. Solve technical problems:

[0112] This technical solution proposes an effective solution strategy for challenges such as high latency sensitivity, energy-intensive applications, and scarce wireless resources in air-ground networks. It uses the edge computing capabilities of MEC to achieve localized processing of services, reduce latency, and use digital twin technology to monitor and optimize the network in real time, improving resource utilization and system performance.

[0113] 3. Overcoming Technological Bias:

[0114] The technical solution of the present invention overcomes the reliance on centralized processing in traditional network optimization methods and realizes decentralized data processing and decision-making through MEC. At the same time, it also uses digital twin technology to provide a new means of network status evaluation and prediction, overcomes the bias of previous technologies for single-dimensional optimization, and realizes comprehensive optimization of network performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1 It is an air-ground network optimization method and system flow chart based on MEC and digital twin provided by an embodiment of the present invention;

[0116] Figure 2 is an applicable scene graph provided by an embodiment of the present invention;

[0117] Figure 3 This is a structural diagram of the air-ground network optimization system based on MEC and digital twins provided in an embodiment of the present invention.

[0118] Figure 4 It is a convergence performance comparison diagram provided by an embodiment of the present invention.

[0119] Figure 5 This is a comparison diagram of network life and drone remaining energy threshold provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0120] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0121] The technical solution provided by the present invention proposes an air-ground network optimization method and system based on MEC and digital twin, mainly using the DRL framework. The solution mainly includes:

[0122] 1) Deep reinforcement learning framework: A DRL framework is adopted, which includes the agent’s main policy network and critic network to perform learning and decision-making tasks.

[0123] 2) Network initialization: Before training begins, the parameters of the main policy network, target policy network, and critic network are initialized, and the hyperparameters related to training are configured.

[0124] 3) Environmental interaction and state management: The agent interacts with the environment according to the current strategy, performs corresponding actions and realizes state transfer.

[0125] 4) Experience replay: Use the replay buffer to store the experience tuples of the agent’s interaction with the environment. When the buffer reaches its capacity limit, it is updated to ensure that the latest experience data is used during the learning process.

[0126] 5) Network parameter update: Update the parameters of the main policy network and the critic network by calculating the loss function and applying the gradient descent algorithm to optimize the network performance.

[0127] 6) Target network soft update: Regularly adopt soft update methods to update the policy and value network parameters in the target network, which helps to maintain the stability of the learning process.

[0128] 7) Strategy convergence and implementation: The training process is repeated until the strategy converges, and then the trained strategy is applied to task segmentation and resource allocation to achieve efficient decision-making.

[0129] The following are two specific embodiments and their implementation schemes provided by the present invention:

[0130] Example 1: Smart City Management

[0131] Use drones for smart city management, collect data and process it.

[0132] 1) Network parameter initialization: According to the requirements of urban environment monitoring, key network parameters such as the number of drones and IoT devices are set.

[0133] 2) Strategy and environment interaction: Drones perform urban environment monitoring tasks according to the instructions generated by the strategy network to achieve automated data collection.

[0134] 3) Data collection and processing: The collected environmental data is transmitted to the MEC server in real time and processed quickly using edge computing capabilities.

[0135] 4) Task and resource allocation: Apply DRL strategies to intelligently optimize data management and resource allocation to improve urban management efficiency.

[0136] 5) Network update and optimization: Based on monitoring results and resource usage, strategies are continuously updated and optimized to adapt to the dynamic changes in the urban environment.

[0137] Example 2: Disaster emergency response

[0138] Rapidly deploy drones and IoT devices for rescue missions after natural disasters.

[0139] 1) Network parameter initialization: Configure drones and IoT devices and set necessary network parameters to support disaster emergency response.

[0140] 2) Mission execution: The drone automatically performs monitoring and assessment tasks in the disaster area based on the preset strategy network.

[0141] 3) Data transmission and processing: Drones collect key data in the disaster area, such as damage and personnel distribution, and transmit it back to the MEC server.

[0142] 4) Dynamic resource allocation: Use the DRL strategy to dynamically optimize the allocation of emergency resources, such as the delivery of relief supplies and personnel evacuation guidance.

[0143] 5) Strategy iteration and update: Based on real-time monitoring data and resource scheduling effects, continuously adjust and optimize emergency response strategies to improve rescue efficiency and effectiveness.

[0144] Example 3

[0145] In view of the problems existing in the prior art, this paper provides an air-ground network optimization method and system based on MEC and digital twin. Figure 2 is a scenario diagram to which the method of the present invention can be applied. At the physical entity layer, there are I UEs, J drones supported by MEC, and a macro base station (MBS) as the cloud center. I = {1, 2, ..., I} and J = {1, 2, ..., J} represent the UE set and the UAV set, respectively. We assume that the BS and UE are managed by the same entity, while the UAVs are managed by different entities. We consider a three-dimensional Cartesian coordinate system operating on a time slot basis, whose time span T = {1, 2, ..., T} is divided into a total of T time slots with a duration of Δt. Since the duration Δt is small enough, this division ensures that the network state within each time slot remains static. The UE is randomly dispersed within the considered range and remains stationary after dispersion, q i =(x i ,y i ,0),i∈I,(x BS ,y BS ,0) and u j (t) = (X j (t),Y j (t),H j (t)) denote the coordinates of UE i, BS and UAVj, respectively.

[0146] In order to achieve instant reflection of the actual network status, a DT layer is deployed on the BS. The actual device needs to transmit real-time status data to the DT layer, and use the transmitted data to build a virtual twin in the DT layer to monitor the network status of the actual device, and finally complete the mapping and offloading decisions through these virtual twins. This project involves two types of entities, namely UE and UAV. On the one hand, the location and task information of each UEi in each time interval t needs to be recorded by its virtual twin, defined as DT i (t)=(q i ,Υ i (t)), where q i =(x i ,y i ,0),i∈I represents the location information of each UE i, i (t)=(D i (t),λ i (t),T max ) represents the task information, where D i (t) represents the amount of input data, in bits; λ i (t) represents the number of CPU cycles required to process 1 bit of data; T max represents the maximum delay tolerance of task processing. On the other hand, the service status of each UAVj also needs to be recorded by the twin, which is defined as in represents the estimated computing resources allocated by UAV j to UE i, Represents the remaining energy of UAVj.

[0147] For this project, the duration of the time slot τ is expected to be greater than or equal to the maximum allowed delay of all jobs to avoid cross-period optimization caused by different task delay requirements. In addition, in this project, the UE has no processing power in the above scenario. With the support of MEC, drones and BS will provide communication and computing support for nearby UEs. Drones can communicate with BS through wireless links, while drones cannot communicate with each other.

[0148] Secondly, the present invention provides a method and system for efficiently integrating edge computing systems and air-ground integrated networks. The system includes a physical entity layer and a digital twin layer. The physical entity layer is composed of I UEs, J drones supported by MECs, and MBS as a cloud center. The digital twin layer includes two digital twins that record relevant information of UE i and UAVj respectively. In each fixed time slot, the UE generates task-related information, and the task must be processed jointly by the drone or base station within the limited time slot. By reasonably dividing each task during execution and reasonably allocating drone and base station resources, the drone network life is maximized and the additional economic expenses in the process are minimized, thereby minimizing the total utility of the entire system.

[0149] Example 4

[0150] Considering the constant changes in the network and the uncertainty of information acquisition, the problem is more complicated to deal with. In order to deal with it effectively, we use Markov Decision Process (MDP) to reframe the problem. Given that the variables involved in the problem are continuous, we chose the MADDPG algorithm, which is a deep reinforcement learning algorithm specifically designed to handle problems with continuous action spaces and support real-time decision making.

[0151] like Figure 1 As shown, the air-ground network optimization method and system based on MEC and digital twin provided in this embodiment are characterized by comprising the following steps:

[0152] S101. Initialize the parameters θ of the agent’s main policy network and critic network j and ω j , the parameters of the target policy network and the critic network and Number of rounds MP, maximum number of training steps T max , initialize the learning rates α and β corresponding to the critic network and the policy network, the discount factor γ, initialize the playback buffer size W, the mini-batch size N and the noise Ψ(t) used for action exploration; initialize the network layout parameters, such as the number of UEs I, the number of drones J and other parameters.

[0153] S102: Initialize the agent state, the agent interacts with the environment, and the main strategy network generates actions according to the current strategy.

[0154] S103. The agent executes the actions generated by the main strategy network, obtains rewards, and updates the status.

[0155] S104: Store the experience tuple into the experience playback buffer. When the buffer reaches the upper limit of the capacity, the latest experience is introduced by overwriting the oldest experience data.

[0156] S105. Update the parameters of the main strategy network and the critic network.

[0157] S106. Calculate the loss function of the critic network according to the value function based on the temporal difference (TD) target and the critic network prediction, extract samples from the experience replay buffer, and use the gradient descent method to update the target policy network and the evaluation network.

[0158] S107. Update the parameters of the main policy network and the critic network through small batch experience samples.

[0159] S108. Adopting a soft update mechanism to update the parameters of the policy network and the value network in the target network.

[0160] S109, iterate the training until the algorithm converges stably, and apply the strategy to the intelligent agent to achieve optimal task offloading and resource allocation.

[0161] In step S102, at the beginning of each cycle, the state of the agent is initialized, the agent interacts with the environment, and the main policy network generates actions based on the current policy. The state of the agent is expressed as:

[0162]

[0163] where u j (t-1) = (X j (t-1),Y j (t-1),H j (t-1)), represents the three-dimensional coordinates of drone j at the end of time slot t-1, corresponding to the starting point of time slot t; D(t) = {D i (t)}, represents the task input data size of the UE in time slot t; λ(t)={λ i (t)}, represents the task processing density of UE in time slot t; represents the computational resource allocation within time slot t; represents the remaining energy of UAV j at the end of time slot t; It represents the time life index of each UAV, and stipulates that the initial life of each UAV is 0.

[0164] In step S103, the agent executes the action generated by the main strategy network, obtains a reward, and updates the state. The calculation formula for the reward in the state update is as follows:

[0165]

[0166] In the above formula Represents the immediate reward obtained by each agent in the system after performing an action.

[0167] In step S105, the main policy network and critic network parameters are updated; the current policy network is updated by gradient ascent as follows:

[0168]

[0169] in represents the parameters θ of the jth actor network j The gradient with respect to the objective function.

[0170] In step S106: the loss function of the critic network is calculated based on the TD target and the value function predicted by the critic network, samples are extracted from the experience replay buffer, and the target policy network and the evaluation network are updated using the gradient descent method. Representing each critic network that obtains the corresponding state and action, the TD target can be given by:

[0171]

[0172] The loss function can be given by:

[0173]

[0174] Then update the parameters w of the current value network of the main value network through gradient descent j as follows:

[0175]

[0176] In the formula Indicates the time difference error.

[0177] In step S108, a soft update mechanism is used to update the parameters of the strategy network and the value network in the target network; the soft update formula is as follows:

[0178]

[0179]

[0180] where θ j Represents the parameters of the current policy network, represents the parameters of the target policy network, w j Represents the parameters of the current value network, Represents the parameters of the target value network, χ∈[0,1].

[0181] In order to elaborate on the air-ground network optimization method and system based on MEC and digital twin, the present invention provides two specific application embodiments, including key details of the implementation scheme.

[0182] In order to prove the creativity and technical value of the technical solution of the present invention, this section provides application examples of the claimed technical solution on specific products or related technologies.

[0183] Application Example 1: Smart City Management

[0184] 1) Network parameter initialization: Deploy the main policy network and the critic network in the city's environmental monitoring center. Initialize the network parameters, including the learning rates α and β corresponding to the critic network and the policy network, the discount factor γ, the initial playback buffer size W, the mini-batch size N, and the noise Ψ(t) for action exploration; deploy necessary IoT devices and drones to achieve environmental monitoring and data collection.

[0185] 2) Interaction between strategy and environment: The drone performs urban monitoring tasks based on the actions generated by the strategy network, and noise is incorporated into the strategy to explore a better monitoring path.

[0186] 3) Data collection and processing: The environmental data collected by the drone is transmitted to the MEC server for rapid data processing.

[0187] 4) Task and resource allocation: Based on the DRL strategy, gradually optimize data processing and resource allocation, such as the allocation of bandwidth resources to different devices.

[0188] 5) Network update and optimization: Update strategies regularly based on monitoring results and resource usage to ensure that the monitoring system continues to manage and transmit data efficiently.

[0189] Application Example 2: Disaster Emergency Response

[0190] 1) Network parameter initialization: Rapidly deploy the main policy network and critic network in the disaster area and initialize key network parameters. Use drone equipment to provide real-time aerial images and videos to support rapid response to disasters and rescue operations.

[0191] 2) Mission execution: The drone performs search and rescue missions according to the strategy. After each rescue action is executed, it obtains immediate feedback based on the rescue effect and updates the system status accordingly.

[0192] 3) Data transmission and processing: The rescue information collected by the drone is quickly transmitted to the MEC system for processing to ensure timely delivery of information.

[0193] 4) Dynamic resource allocation: Through the DRL strategy, the allocation of rescue resources is dynamically adjusted, such as the coordination between drones and ground rescue teams, to optimize rescue efficiency.

[0194] 5) Strategy iteration and update: Based on the actual rescue progress and feedback, continuously adjust and optimize the strategy to improve response speed and rescue effect, and ensure that the rescue operation is more accurate and efficient.

[0195] In these two application examples, an efficient, reliable, and energy-saving joint optimization method is demonstrated for air-ground network optimization method and system based on MEC and digital twin. This method is not only applicable to smart city management, but also to various scenarios such as disaster emergency response, which once again proves its wide application potential and technical advantages.

[0196] An application embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of an air-ground network optimization method based on MEC and digital twins.

[0197] An application embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of an air-ground network optimization method based on MEC and digital twins.

[0198] An application embodiment of the present invention provides an information data processing terminal, which includes an air-ground network optimization system based on MEC and digital twins.

[0199] To compare the invention with the current industry standard algorithms, we conducted standardized performance benchmarks. This analysis revealed significant advantages of the invention on key performance indicators and provided deep insights into potential performance bottlenecks.

[0200] 1. The present invention: represents the proposed air-ground network optimization method based on MEC and digital twin.

[0201] 2. Random Movement: In this algorithm, the drone moves randomly. We allow the device to move randomly within the range and change its speed while letting the drone move randomly. In addition, the strategies including association strategy, caching decision, mode selection and task offloading are still optimized using our proposed algorithm.

[0202] 3. Access Cache: In this algorithm, only the association strategy, mode selection, and task offloading ratio are optimized, while the input data caching decision is randomly generated.

[0203] 4. Access offloading: In this algorithm, all input data are not cached in the cache memory of the drone and are completely calculated using the offloading model. While the other three strategies are optimized using the proposed algorithm.

[0204] 5. Random offloading: In this algorithm, mode selection and task offloading are determined by random values, and the offloading ratio of tasks is also given randomly. Access control and cache decisions are optimized using the proposed algorithm.

[0205] from Figure 4 It can be seen that all five algorithms show good convergence. However, compared with the other four methods, our proposed method outperforms them in terms of convergence speed and rewards obtained. In terms of convergence speed, our proposed algorithm achieves convergence at a slightly faster speed than other algorithms. In terms of reward performance, the algorithm of the present invention maintains a relatively stable reward throughout the convergence process, while some algorithms will experience certain fluctuations after convergence, or perform worse than the algorithm of the present invention in terms of the final reward value.

[0206] Figure 5 The relationship between the network lifetime and the residual energy threshold of the drone is shown in a bar graph. The residual energy of the drone depends on the energy consumption of the drone during flight and the energy consumption of the drone when processing UE tasks, the latter of which is affected by the mode selection. Therefore, different mode selections lead to obvious changes in the residual energy, which in turn affects the network lifetime. As can be seen from the figure, as the residual energy threshold increases, the network lifetime exhibited by the five algorithms shows a downward trend. Despite this, the algorithm proposed by us always maintains a relatively high network lifetime at different energy thresholds.

[0207] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0208] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for optimizing air-ground networks based on MEC and digital twins, characterized in that: Use the main policy network and critic network of deep reinforcement learning to learn and make decisions for the agent; initialize the parameters of the policy network and evaluation network, and set the training-related hyperparameters; the agent interacts with the environment according to the current strategy, performs actions and updates the state; uses the experience replay buffer to store experience and updates it when it reaches capacity to keep the latest learning experience; uses loss function calculation and gradient descent method to update the parameters of the main policy network and critic network; regularly soft-updates the parameters of the policy network and value network in the target network to ensure learning stability; Continue training until the strategy converges, and then apply the strategy to complete task allocation and trajectory planning.

2. The air-ground network optimization method based on MEC and digital twin according to claim 1, characterized in that: The air-ground network optimization method based on MEC and digital twins specifically includes: S101. Initialize the parameters θ of the agent’s main policy network and critic network j and ω j , the parameters of the target policy network and the critic network and Number of rounds MP, maximum number of training steps T max , initialize the learning rates α and β corresponding to the critic network and the policy network, the discount factor γ, initialize the playback buffer size W, the mini-batch size N, and the noise Ψ(t) used for action exploration; initialize the network layout parameters, such as the number of UEs I and the number of drones J; S102, initializing the agent state, the agent interacts with the environment, and the main strategy network generates actions according to the current strategy; S103, the agent executes the action generated by the main strategy network, obtains rewards, and updates the state; S104, storing the experience tuple into the experience playback buffer; when the buffer reaches the upper limit of the capacity, introducing the latest acquired experience by overwriting the oldest experience data; S105, updating the parameters of the main strategy network and the critic network; S106, calculating the loss function of the critic network according to the TD target and the value function predicted by the critic network, extracting samples from the experience replay buffer, and updating the target policy network and the evaluation network using the gradient descent method; S107, update the parameters of the main strategy network and the critic network through small batch experience samples; S108, using a soft update mechanism to update the parameters of the policy network and the value network in the target network; S109, iterate the training until the algorithm converges stably, and apply the strategy to the intelligent agent to achieve optimal task allocation and trajectory planning.

3. The air-ground network optimization method based on MEC and digital twins according to claim 2 is characterized in that: S102, initializing the state of the intelligent agent, the intelligent agent interacts with the environment, the main strategy network generates actions based on the current strategy, and the state of the intelligent agent is expressed as: in represents the three-dimensional coordinates of UAV j at the end of time slot t-1, corresponding to the starting point of time slot t; represents the task input data size of UE in time slot t; represents the task processing density of UE in time slot t; represents the computational resource allocation within time slot t; represents the remaining energy of UAV j at the end of time slot t; It represents the time life index of each UAV, and stipulates that the initial life of each UAV is 0.

4. The air-ground network optimization method based on MEC and digital twins according to claim 2, characterized in that: S103: The agent executes the action generated by the main strategy network, obtains a reward, and updates the state. The calculation formula of the reward in the state update is as follows: In the above formula Represents the immediate reward obtained by each agent in the system after performing an action.

5. The air-ground network optimization method based on MEC and digital twins according to claim 2, characterized in that: S105: Update the parameters of the main strategy network and the critic network; update the current strategy network by gradient ascent as follows: S106: Calculate the loss function of the critic network according to the TD target and the value function predicted by the critic network, extract samples from the experience replay buffer, and use the gradient descent method to update the target policy network and the evaluation network; the TD target can be given by the following formula: The loss function can be given by: Then update the parameters w of the current value network of the main value network through gradient descent j as follows:

6. The air-ground network optimization method based on MEC and digital twins according to claim 2, characterized in that: S108: adopting a soft update mechanism to update the parameters of the policy network and the value network in the target network; the soft update formula is as follows: where θ j Represents the parameters of the current policy network, represents the parameters of the target policy network, w j Represents the parameters of the current value network, Represents the parameters of the target value network, χ∈[0,1].

7. An air-ground network optimization system based on MEC and digital twins that implements the air-ground network optimization method based on MEC and digital twins as described in any one of claims 1 to 6, characterized in that: include: System initialization module, used to initialize the parameters of the deep deterministic policy gradient algorithm, including setting the number of rounds MP and the maximum number of training steps T max , initialize the learning rates α and β corresponding to the critic network and the policy network, the discount factor γ, initialize the playback buffer size W, and the mini-batch size N; Network construction module, used for motion exploration noise Ψ(t); initialization of network layout parameters, such as the number of UEs I, the number of drones J and other network layout parameters; The agent module is used to generate actions based on the current network state at the beginning of each cycle, with the ability to add exploratory noise to these behaviors in order to introduce a certain amount of randomness in the execution process; Action execution module, used to execute resource allocation and task offloading strategies; The reward acquisition module is used to perform actions and calculate immediate rewards, evaluate rewards based on the long-term average utility of all devices in the system, and transfer the system from the current state to the next state; The experience replay module is used to store each system state, action executed, reward obtained, and experience tuple of the next state; The data sampling module is used to extract mini-batch experiences from the stored experience playback module for learning; The network update module is used to update the main policy network and the main value network according to the data of the experience playback module, including a parameter optimization unit, which uses the gradient ascent method and the gradient descent method to adjust the network parameters; The parameter update module is used to synchronize the parameter updates of the main network to the target strategy network and the target value network. The soft update strategy is adopted so that the parameters of the target network are the weighted average of the parameters of the main network, and this synchronization process is realized through the parameter synchronization unit.

8. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the air-ground network optimization method based on MEC and digital twins as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the air-ground network optimization method based on MEC and digital twins as described in any one of claims 1 to 6.

10. An information data processing terminal, comprising the air-ground network optimization system based on MEC and digital twin as described in claim 7.

Images