An air-ground network optimization method and system based on MEC and digital twinning

Abstract

The application belongs to but is not limited to the field of communication technology, and particularly relates to a kind of air-ground network optimization method and system based on MEC and digital twinning, the main strategy network and critic network of deep reinforcement learning are used to learn and decision of agent;Initialize the parameters of strategy network and evaluation network, and set the training related hyperparameters;Agent interacts with the environment according to the current strategy, executes action and updates state;Experience replay buffer is used to store experience, and is updated when the capacity is reached to keep the latest learning experience;Loss function calculation and gradient descent method are used to update the parameters of the main strategy network and critic network;Soft update of policy network and value network parameters in target network is carried out regularly to ensure learning stability;Continue training until the policy converges, then apply the policy to complete task allocation and trajectory planning.

Description

TECHNICAL FIELD

[0001] The present application belongs to but is not limited to the field of communication technology, and particularly relates to a method and system for optimizing air-ground network based on MEC and digital twinning. BACKGROUND

[0002] In recent years, with the proliferation of Internet of Things devices, traditional ground networks are increasingly unable to meet the extensive demands and are also facing many challenges: (1) Time sensitivity: Many intelligent Internet of Things applications have strict requirements for service response time, i.e. low latency requirements. However, due to the limitations of device size and capability, terminal devices often have difficulty meeting these stringent latency requirements. (2) Energy-intensive applications: Intelligent applications are often energy-intensive or computationally-intensive, requiring a large amount of energy consumption. This results in a significant reduction in the battery life of user terminals or Internet of Things devices. (3) Wireless resource scarcity: With the access of a large number of intelligent terminals and Internet of Things devices, limited wireless spectrum resources become increasingly scarce. The current network is under great pressure in accommodating these rapidly increasing terminal devices. Against this background, the concept of air-ground integrated network has emerged. By combining different layers to provide greater coverage, higher throughput, flexible deployment and strong resilience. Among them, the integration of unmanned aerial vehicles brings unique value to air-ground integrated networks. As a typical airborne device, unmanned aerial vehicles have become a key tool for performing various intelligent tasks, such as real-time image processing, environmental monitoring and emergency response. However, in order to make the air-ground integrated network operate efficiently, a large amount of data generated by the unmanned aerial vehicle during the execution of the task must be processed and analyzed. Under the impetus of this demand, MEC technology is introduced into the air-ground integrated network. MEC can significantly reduce data processing delay and improve system response speed by deploying computing power closer to the data source at the network edge. Therefore, deploying MEC on unmanned aerial vehicles can make data processing closer to the data source, thereby reducing data transmission delay and bandwidth requirements. This deployment method allows unmanned aerial vehicles to perform real-time data analysis and decision-making during task execution.

[0003] When MEC and unmanned aerial vehicles are combined into the air-ground integrated network architecture, some challenges need to be addressed.

[0004] 1) Energy consumption and device life problems

[0005] When MEC and drones are combined, drones need to perform intensive computation and data processing during task execution. In particular, the high consumption of computing resources significantly increases during task offloading and resource allocation. The hardware resources of drones are limited and the expected service life is limited, and frequent computation and data storage activities can accelerate the wear and tear of hardware and shorten its service life. During task execution, a large number of data write and storage operations are involved, which not only challenges the immediate performance of the drone, but also can cause its performance to decline or even fail. Therefore, frequent task execution and data storage activities in the MEC system limit the reliability and durability of the drone.

[0006] 2) Processing power issues

[0007] During the execution of tasks by drones, intensive real-time data stream processing and decision making are required, which poses significant challenges to their computing power. Limited by the hardware resources of drones such as central processing units (CPUs) and random access memory (RAM), their ability to handle complex tasks is challenged. Since the energy supply of drones mainly depends on batteries, their continuous operation ability in high-load computing tasks is limited. In addition, frequent task offloading and resource allocation decisions increase the burden on computing resources, resulting in delays in data processing and reduced system performance.

[0008] 3) Real-time data update issues

[0009] Drones need to frequently acquire and process data during dynamic task execution, which poses high requirements on the immediate data processing capability of the MEC system. The delay of data transmission, the consistency of data in a distributed MEC environment, and the limited communication bandwidth limit the data transmission rate, which in turn affects the real-time update of data. At the same time, frequent data transmission and processing increase the energy consumption of drones, which may shorten their battery life. In addition, the reliability of the system in the face of environmental interference is also a key challenge to ensure the continuity and accuracy of data update.

[0010] 4) Dynamic optimization issues

[0011] In the scenario of drones and MEC, the environment is usually highly dynamic, and the target is usually defined as a long-term target. In this case, we need to design more efficient algorithms to achieve target optimization while maintaining low complexity and good performance.

[0012] In view of the above analysis, the technical problems existing in the prior art that need to be solved urgently are that the existing literatures do not consider the resource allocation problem caused by the limited local storage resources of the unmanned aerial vehicle, do not deeply explore the optimization problems of the UE and unmanned aerial vehicle association strategy, unloading mode selection and task unloading ratio, and although part of the literatures consider the data allocation and unloading optimization problem, the economic overhead generated in the process is ignored. Other devices in the air-ground integrated network enabled MEC scene often have high dynamics, which puts forward higher performance to the algorithm, and thus it is difficult to solve. SUMMARY

[0013] In view of the problems existing in the prior art, the present application provides an air-ground network optimization method and system based on MEC and digital twinning.

[0014] The present application is implemented in the following way: a kind of air-ground network optimization method based on MEC and digital twinning, the main strategy network and critic network of Deep Reinforcement Learning (DRL) are used to learn and decision-making of agent;The parameters of policy network and evaluation network are initialized, and the training related hyperparameters are set;Agent interacts with the environment according to the current strategy, executes action and updates state;Experience replay buffer is used to store experience, and is updated to keep the latest learning experience when the capacity is reached;Loss function calculation and gradient descent method are used to update the parameters of the main policy network and critic network;The policy network and value network parameters in the target network are updated regularly to ensure the stability of learning;Continuous training until policy convergence, then apply the strategy to complete task allocation and trajectory planning.

[0015] Further, the air-ground network optimization method based on MEC and digital twinning specifically includes:

[0016] S101, the parameters θ of the main policy network and critic network of the agent are initialized j and ω j , the parameters of the target policy network and critic network and number of rounds MP, maximum training step T max , the learning rate α and β corresponding to the critic network and policy network, the discount factor γ are initialized, the replay buffer size W, the size N of mini-batch and the noise Ψ (t) used for action exploration are initialized;Initialize network layout parameters, such as UE number I, unmanned aerial vehicle number J and other parameters.

[0017] S102, the state of the agent is initialized, the agent interacts with the environment, and the main policy network generates action according to the current strategy.

[0018] S103, the agent executes the action generated by the main policy network, obtains the reward, and updates the state.

[0019] S104. Store the experience tuple into the experience replay buffer. When the buffer reaches its capacity limit, introduce the latest acquired experience by overwriting the oldest experience data.

[0020] S105, Update the parameters of the main strategy network and the commentator network.

[0021] S106. Calculate the loss function of the critic network based on the TD objective and the value function predicted by the critic network. Draw samples from the empirical replay buffer and update the objective policy network and evaluation network using gradient descent.

[0022] S107. Update the parameters of the main policy network and the critic network using small batches of empirical samples.

[0023] S108. Use a soft update mechanism to update the parameters of the policy network and value network in the target network.

[0024] S109. Iterate the training until the algorithm converges stably. Apply this strategy to the agent to achieve optimal task allocation and trajectory planning.

[0025] Furthermore, in S102, the agent's state is initialized, the agent interacts with the environment, and the main policy network generates actions based on the current policy. The agent's state is represented as follows:

[0026]

[0027] Where u j (t-1)=(X j (t-1),Y j (t-1),H j (t-1)), Let D(t) represent the three-dimensional coordinates of UAV j at the end of time slot t-1, corresponding to the starting point of time slot t; D(t) = {D i (t)}, λ(t) represents the size of the task input data of the UE in time slot t; λ(t) = {λ i (t)}, This represents the task processing density of the UE within time slot t; This indicates the allocation of computing resources within time slot t; This represents the remaining energy of drone j at the end of time slot t; This represents the time-based lifespan index of each drone, and specifies that the initial lifespan of each drone is 0.

[0028] Furthermore, in S103: the agent executes the action generated by the main policy network, obtains a reward, and updates its state. The calculation formula for the reward during the state update is as follows:

[0029]

[0030] In the above formula represents the immediate reward obtained after each agent in the system carries out an action.

[0031] Further, the S105: update the main policy network and critic network parameters; update the current policy network as follows through gradient ascent:

[0032]

[0033] Further, the S106: calculate the loss function of the critic network according to the TD target and the value function predicted by the critic network, extract samples from the experience replay buffer, and update the target policy network and the evaluation network using the gradient descent method. The TD target can be given by the following formula:

[0034]

[0035] The loss function can be given by the following formula:

[0036]

[0037] Then update the parameters w of the current value network of the main value network through gradient descent j As follows:

[0038]

[0039] Further, the S108: update the parameters of the policy network and the value network in the target network using a soft update mechanism; the soft update formula is as follows:

[0040]

[0041]

[0042] Where θ j represents the parameters of the current policy network, represents the parameters of the target policy network, w j represents the parameters of the current value network, represents the parameters of the target value network, χ∈[0,1].

[0043] Another object of the present application is to provide a MEC and digital twin based air-ground network optimization system for realizing the MEC and digital twin based air-ground network optimization method, comprising:

[0044] The system initialization module is used for initializing the deep deterministic policy gradient algorithm parameters, including setting the number of rounds MP, the maximum training step T max, initialize learning rates a and b corresponding to critic network and policy network, discount factor g, initialize replay buffer size W, size N of mini-batch;

[0045] A network construction module is configured to initialize network layout parameters such as the number of UEs I, the number of UAVs J, and the like.

[0046] An agent module is configured to generate actions based on the current network state at the beginning of each period, and has a function of adding exploratory noise in these actions so as to introduce a certain randomness in the execution process.

[0047] An action execution module is configured to execute resource allocation and task offloading strategies.

[0048] A reward acquisition module is configured to execute actions and calculate immediate rewards, evaluate rewards according to long-term average utility of all devices in the system, and a state transition module is configured to transfer the system from the current state to the next state.

[0049] An experience replay module is configured to store experience tuples of each system state, executed action, obtained reward and next state.

[0050] A data sampling module is configured to sample mini-batch experiences from the stored experience replay module for learning.

[0051] A network updating module is configured to update the main policy network and the main value network according to the data of the experience replay module, and includes a parameter optimization unit configured to adjust network parameters using gradient ascent and gradient descent.

[0052] A parameter updating module is configured to update the parameters of the main network to the target policy network and the target value network, and adopts a soft update strategy so that the parameters of the target network are the weighted average of the parameters of the main network, and the synchronization process is realized through a parameter synchronization unit.

[0053] Another object of the present application is to provide a computer device comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to enable the processor to perform the steps of the MEC and digital twin based air-ground network optimization method.

[0054] Another object of the present application is to provide a computer readable storage medium storing a computer program, the computer program being executed by the processor to enable the processor to perform the steps of the MEC and digital twin based air-ground network optimization method.

[0055] Another object of the present application is to provide an information data processing terminal comprising the air-ground network optimization system based on MEC and digital twinning.

[0056] In combination with the above technical solutions and the technical problems solved, the technical solutions to be protected by the present application have the following advantages and positive effects:

[0057] Firstly, the present application improves network coverage and service quality. In the air-ground integrated network architecture, unmanned aerial vehicles serve as key aerial nodes, providing broader network coverage for remote areas. Through the edge caching system assisted by unmanned aerial vehicles, we can efficiently handle computing tasks, reasonably allocate computing resources and task division, thereby making up for the lack of ground base station coverage. The integration of the present application is of great significance to strengthening the coverage and service quality of the network in the air-ground integrated environment

[0058] The present application combines MEC with unmanned aerial vehicle-assisted air-ground integrated network to provide a more efficient and scalable solution for network services. The strong network coverage of unmanned aerial vehicles can improve the limitations of task processing only through ground base stations. By combining the MEC system with the unmanned aerial vehicle-assisted air-ground integrated network, faster task processing can be achieved, and task processing latency can be reduced. This combination provides more space for the scalability of network services, better meets the growing data processing needs, and meets the task needs of remote areas, making the entire system more sustainable.

[0059] The present application realizes real-time monitoring and dynamic mapping of the physical network environment by adopting digital twinning technology, thereby improving the efficiency and response speed of network management. By mapping the physical network to the digital network, comprehensive monitoring and real-time mapping of the network state are realized, thereby optimizing network performance and resource allocation. The application of digital twinning technology not only improves the intelligent level of the network, but also provides strong support for predictive maintenance and fault diagnosis of the network.

[0060] The present application provides an optimization strategy for the cooperative work between unmanned aerial vehicles and ground base stations according to the actual dynamic environment. Compared with experimental data, it is confirmed that the dynamic resource allocation method is more accurate than the traditional static resource allocation method, can better simulate the actual environment, is simple to operate, has higher real-time performance, is closer to the real scene, and is conducive to network optimization and system performance improvement.

[0061] Secondly, the present application has achieved significant technical progress in that it realizes a joint optimization method combining an unmanned aerial vehicle-assisted air-ground integrated network with an edge computing system and digital twinning. The method has made significant progress in the following key aspects:

[0062] 1) Efficient Resource Allocation:

[0063] This method significantly improves the resource utilization in the UAV-assisted air-ground integrated network by optimizing resource allocation. This includes more efficient allocation of bandwidth resources to UAVs and base stations to achieve efficient collaborative task processing. This optimization not only improves system performance but also reduces energy consumption.

[0064] 2) Reasonable Task Allocation Strategy:

[0065] By reasonably allocating input data, an intelligent task allocation strategy is achieved. The system can dynamically adjust task processing according to the capabilities of UAVs and network conditions, intelligently adjusting whether UAVs or base stations process, thereby adapting to the dynamic changes of the network. This enhances the stability of the network and user experience.

[0066] 3) Integration of Reinforcement Learning:

[0067] By combining DRL algorithms with MEC, the UAV-assisted air-ground integrated network can analyze data in real time and adapt to environmental changes autonomously, making optimal decisions in complex and dynamic conditions.

[0068] 4) Optimization of System Utility:

[0069] The reward mechanism included in the method focuses on reducing the total utility of the system of UAVs and ground equipment, optimizing energy consumption through the reward mechanism, improving energy efficiency, and being environmentally friendly.

[0070] 5) Improvement of System Stability and Reliability:

[0071] By accurately calculating the immediate feedback of UAVs after executing tasks and optimizing their state transition process, this method significantly improves the stability and reliability of the system when facing large-scale data processing and high-density requests.

[0072] 6) Autonomous Learning and Optimization Capability of the Network:

[0073] This method enables the system to continuously optimize its decision-making process through iterative training and experience-based network updates, thereby improving overall performance.

[0074] 7) Application of Digital Twin Technology:

[0075] This method uses digital twin technology to build a highly accurate virtual network mapping system, achieving real-time monitoring and dynamic mapping of the physical network environment, improving the efficiency and response speed of network management.

[0076] The integrated application of these technologies significantly improves the performance of the UAV-assisted MEC network system in multiple aspects, including performance, energy efficiency, stability, and adaptability, effectively supporting the complex demands of modern computing. The system is capable of learning and adapting to changes in the environment, ensuring optimal performance and energy efficiency.

[0077] Thirdly, the edge computing system optimization method based on the integration of air-ground networks and deep reinforcement learning provided by the present application uses mathematical models to guide the behavior and learning process of the system. The technical effects brought by these mathematical models can be explored based on their characteristics:

[0078] 1) Calculation of immediate rewards

[0079] This method focuses on improving the overall utility of all devices in the system through the calculation of immediate rewards.

[0080] Energy efficiency optimization: By directly linking the reward mechanism to the system utility, the system is encouraged to explore strategies that reduce energy consumption, achieving higher energy efficiency.

[0081] Cost savings: With the continuous optimization of system utility, long-term operation costs can be significantly reduced, especially in edge computing environments with high energy consumption.

[0082] 2) Updating of the main policy network and critic network

[0083] Using small batches of randomly sampled experience data, the current policy network is updated through gradient ascent.

[0084] Policy optimization: The system learns and adopts more efficient decision-making strategies by continuously adjusting the parameters of the policy network.

[0085] Improved responsiveness: Updating the policy using small batches of data allows the network to quickly respond to environmental changes, improving the system's ability to make immediate adjustments.

[0086] 3) TD target and critic network loss function

[0087] Based on the Q value predicted by the time difference (TD) target and critic network, the loss function is calculated and updated through gradient descent.

[0088] Learning stability: Using the TD target as the basis for learning helps to balance the learning process and reduce learning fluctuations caused by excessive prediction errors.

[0089] Performance optimization: By accurately calculating the loss function and updating the network parameters in a timely manner, the system can improve the accuracy and efficiency of decision-making.

[0090] 4) Parameter update formula

[0091] It describes how the main policy network and critic network update the parameters of the target network, involving the synchronization of current network and target network parameters.

[0092] Policy gradual approximation: By gradually adjusting the target network parameters, the system can smoothly adopt new strategies, avoiding performance fluctuations caused by sudden changes in strategy.

[0093] Continuous learning and adaptation: This continuous parameter update mechanism enables the system to adapt to environmental changes over the long term, maintaining the continuity of learning and adaptation.

[0094] The application of the mathematical model provided by the present application not only improves the operation efficiency and decision-making quality of the system, but also enhances the adaptability and long-term stability of the system to environmental changes, and realizes real-time monitoring and mapping of network status through the integration of 3D trajectory optimization, unloading and caching strategy optimization, and digital twin technology. These technical effects are crucial for modern edge computing environments that handle large amounts of data and high-frequency interactions.

[0095] Fourth, the MEC and digital twin-based air-ground network optimization method and system provided by the present application adopts deep reinforcement learning technology to optimize the performance of the network through the interaction between the agent and the environment.

[0096] Initialize the state of the agent, which includes multiple variables such as task input data volume, task processing density, task processing deadline, estimated computing resource allocation, initial remaining energy of each UAV, horizontal distance between IoT devices and UAVs, flight altitude of UAVs, environment-related parameters, and path loss parameters. These variables collectively define the environmental state of the agent at a specific time, which in turn affects the agent's decision-making.

[0097] The agent performs tasks according to the actions determined by the main policy network, and after performing the corresponding actions, the agent obtains the corresponding rewards based on the results obtained, and updates and transfers the state according to the rewards and execution results. The calculation of rewards is based on the long-term average utility of all devices in the system, which is the core goal of system design, aiming to minimize the total utility of the entire system.

[0098] Adjusting the parameters of the policy network through the gradient ascent method helps the policy network to generate more optimal actions, and by optimizing the policy network, further optimization of the overall system performance is achieved.

[0099] Using the time difference (TD) target and critic network predicted value function to calculate the loss function, and updating the parameters of the target policy network and critic network through gradient descent method is a key step in the value function update in deep reinforcement learning. This process guides the continuous improvement of the policy by improving the accuracy of the prediction, thereby optimizing the overall system performance.

[0100] The target network is updated by a soft update method, which is a gradual parameter update technique that adjusts the target network by calculating the weighted average of the target network parameters and the current network parameters. This method helps to reduce the sharp fluctuations that may occur during the learning process, ensuring the stability and smoothness of the entire learning process.

[0101] The application of these steps and mathematical models brings significant technical progress:

[0102] Policy optimization: Deep reinforcement learning enables the system to autonomously learn and optimize policies to adapt to dynamic changes in the UAV-assisted air-ground integrated network environment.

[0103] Efficient use of resources: By optimizing task offloading and resource allocation, it ensures efficient use of limited resources in the UAV network, especially in 3D trajectory optimization and data offloading, caching strategies.

[0104] System utility minimization: The system optimizes the long-term average utility to support green communication and reduce environmental impact, which is particularly important for energy management of UAVs and energy efficiency optimization of MEC.

[0105] System stability: The use of soft update method ensures the smoothness of the learning process and reduces the risk of system instability caused by rapid parameter updates, which is crucial for maintaining the accuracy of digital twin mapping and network state monitoring.

[0106] Real-time network state monitoring: The application of digital twin technology realizes real-time mapping and monitoring of the physical network state, providing in-depth insights into the network operating conditions, enabling policy optimization and resource allocation based on the most accurate network information.

[0107] These advances reflect the potential of deep reinforcement learning in building intelligent and efficient air-ground integrated communication networks, especially in complex network systems with UAV assistance, MEC support and digital twin technology integration.

[0108] Fifth, as a creative auxiliary evidence of the invention's claims, it is also reflected in the following important aspects:

[0109] 1. Fill the technical gap:

[0110] The technical solution of the present invention integrates MEC and digital twin technology to provide a new solution for air-ground network optimization. This integrated approach takes advantage of the low latency and high bandwidth characteristics of MEC, as well as the mapping and simulation capabilities of digital twins in virtual space to solve the limitations of single network architecture in existing technologies.

[0111] 2. Solve technical problems:

[0112] The technical solution proposes an effective solution strategy for the challenges of high latency sensitivity, energy-intensive applications, and wireless resource scarcity in air-ground networks. It realizes the localization of services through the edge computing capability of MEC, reduces the latency, and improves the resource utilization and system performance through real-time monitoring and optimization of the network by digital twin technology.

[0113] 3. Overcome technical bias:

[0114] The technical solution of the present application overcomes the dependence on centralized processing in traditional network optimization methods, and realizes decentralized data processing and decision-making through MEC. At the same time, it also provides a new means of network state evaluation and prediction by using digital twin technology, which overcomes the bias of single-dimensional optimization in previous technologies and realizes the overall optimization of network performance. BRIEF DESCRIPTION OF DRAWINGS

[0115] Figure 1 is the flow chart of the air-ground network optimization method and system based on MEC and digital twin provided by the embodiment of the present application;

[0116] Figure 2 is a scene diagram that can be applied according to the embodiment of the present application;

[0117] Figure 3 is the structure diagram of the air-ground network optimization system based on MEC and digital twin provided by the embodiment of the present application.

[0118] Figure 4 is the convergence performance comparison diagram provided by the embodiment of the present application.

[0119] Figure 5 is the network life and residual energy threshold comparison diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0120] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0121] The technical solution provided by the present application proposes an air-ground network optimization method and system based on MEC and digital twin, mainly using the framework of DRL. The scheme mainly includes:

[0122] 1) Deep reinforcement learning framework: using DRL framework, which includes the main policy network and critic network of the agent, for learning and decision-making tasks.

[0123] 2) Network Initialization: Before training begins, the parameters of the main policy network, target policy network, and critic network are initialized, and hyperparameters related to training are configured.

[0124] 3) Environment Interaction and State Management: The agent interacts with the environment according to the current policy, performs corresponding actions, and realizes state transition.

[0125] 4) Experience Replay: The experience tuples of agent-environment interaction are stored in the replay buffer, which is updated when the buffer reaches its capacity limit, ensuring that the latest experience data is used in the learning process.

[0126] 5) Network Parameter Update: The parameters of the main policy network and critic network are updated by calculating the loss function and applying the gradient descent algorithm to optimize network performance.

[0127] 6) Target Network Soft Update: Regularly update the policy and value network parameters in the target network using the soft update method, which helps maintain the stability of the learning process.

[0128] 7) Policy Convergence and Implementation: Repeat the training process until the policy converges, then apply the trained policy to task segmentation and resource allocation to make efficient decisions.

[0129] The following are two specific embodiments provided by the invention and their implementation schemes:

[0130] Example 1: Intelligent City Management

[0131] Intelligent city management using drones, collecting and processing data.

[0132] 1) Network Parameter Initialization: According to the city environment monitoring requirements, set the number of drones and key network parameters such as IoT devices.

[0133] 2) Policy and Environment Interaction: Drones execute city environment monitoring tasks according to the instructions generated by the policy network, achieving automated data collection.

[0134] 3) Data Collection and Processing: Collected environmental data is transmitted to the MEC server in real time, and edge computing capabilities are used for rapid processing.

[0135] 4) Task and Resource Allocation: Apply DRL policy to intelligently optimize data management and resource allocation, improving city management efficiency.

[0136] 5) Network Update and Optimization: Based on monitoring results and resource usage, continuously update and optimize the policy to adapt to dynamic changes in the city environment.

[0137] Example 2: Disaster Emergency Response

[0138] After natural disasters occur, drones and IoT devices are quickly deployed for rescue missions.

[0139] 1) Network parameter initialization: configure drones and IoT devices, set necessary network parameters to support disaster emergency response.

[0140] 2) Task execution: drones automatically perform monitoring and evaluation tasks in disaster areas according to preset strategy networks.

[0141] 3) Data transmission and processing: drones collect key data such as damage, personnel distribution, etc. in disaster areas and return to MEC servers.

[0142] 4) Dynamic resource allocation: dynamically optimize the allocation of emergency resources such as rescue supplies and personnel evacuation guidance using DRL strategy.

[0143] 5) Strategy iteration and update: continuously adjust and optimize emergency response strategies based on real-time monitoring data and resource scheduling effects to improve rescue efficiency and effectiveness.

[0144] Embodiment 3

[0145] To solve the problems in the prior art, the embodiment provides an air-ground network optimization method and system based on MEC and digital twinning, Figure 2 is a scene diagram in which the method of the present application can be applied. At the physical entity layer, there are I UEs, J MEC-supported drones, and a macro base station (MBS) as a cloud center. I = {1, 2,..., I} and J = {1, 2,..., J} represent the UE set and the UAV set, respectively. We assume that the BS and the UE are managed by the same entity, while the UAV is managed by a different entity. We consider a three-dimensional Cartesian coordinate system running on a time slot basis, whose time span T = {1, 2,..., T} is divided into T time slots with a duration of Δt. Since the duration Δt is small enough, this division ensures that the network state remains static within each time slot. The UEs are randomly dispersed within the considered range and remain stationary after dispersion, q i = (x i ,y i ,0), i ∈ I, (x BS ,y BS ,0) and u j (t) = (X j (t), Y j (t), H j (t)) represent the coordinates of UE i, BS and UAV j, respectively.

[0146] To achieve the instant reflection of the actual network status, a DT layer is deployed on the BS. Real devices need to transmit real-time status data to the DT layer and use the transmitted data to build virtual twins in the DT layer to monitor the network status of real devices, and finally complete mapping and offloading decisions through these virtual twins. This project involves two types of entities, i.e., UE and UAV. On the one hand, the location and task information of each UEi in each time interval t needs to be recorded by its virtual twin, defined as DT i (t) = (q i , Y i (t)), where q i = (x i , y i , 0), i∈I represents the location information of each UE i, Y i (t) = (D i (t), λ i (t), T max ) represents the task information, wherein D i (t) represents the input data volume, in bits; λ i (t) represents the number of CPU cycles required to process 1 bit of data; T max represents the maximum delay tolerance of task processing. On the other hand, the service status of each UAVj also needs to be recorded by the twin, defined as where represents the predicted computing resources allocated to UE i by UAV j, represents the remaining energy of UAVj.

[0147] For this project, the duration of the predicted time slot τ is greater than or equal to the maximum allowed delay of all jobs to avoid cross-period optimization due to different task delay requirements. In addition, in this project, the UE has no processing capacity in the above scenario. With the support of MEC, the UAV and the BS will provide communication and computing support for nearby UEs. The UAV can communicate with the BS through a wireless link, while the UAVs cannot communicate with each other.

[0148] Secondly, the application provides a method and system for efficiently integrating edge computing systems and air-ground integrated networks. The system includes a physical entity layer and a digital twin layer. The physical entity layer is composed of I UEs, J MEC-supported drones, and MBS as a cloud center. The digital twin layer includes two digital twins that record information related to UE i and UAV j, respectively. In each fixed time slot, UEs generate task-related information that needs to be processed by drones or base stations within a limited time slot. By reasonably dividing each task during execution and reasonably allocating resources of drones and base stations, the life of the drone network is maximized, the additional economic overhead is minimized, and the overall utility of the system is minimized.

[0149] Embodiment 4

[0150] Considering the constant changes of the network and the uncertainty of information acquisition, the problem is relatively complex to deal with. To effectively respond, we use Markov Decision Process (MDP) to reconstruct the problem. Since the variables involved in the problem are continuous, we use the MADDPG algorithm, which is a deep reinforcement learning algorithm specifically designed to handle problems with continuous action spaces and supports real-time decision-making

[0151] As Figure 1 shown, the air-ground network optimization method and system based on MEC and digital twin provided by the embodiment include the following steps:

[0152] S101, initialize the parameters θ of the main policy network and critic network of the agent j and ω j , the parameters of the target policy network and critic network and the number of rounds MP, the maximum training step T max , initialize the learning rates α and β of the critic network and policy network, the discount factor γ, the size of the replay buffer W, the size of the mini-batch N, and the noise Ψ(t) for action exploration; initialize the network layout parameters, such as the number of UEs I, the number of drones J, etc.

[0153] S102, initialize the state of the agent, and the agent interacts with the environment. The main policy network generates actions according to the current policy.

[0154] S103, the agent executes the action generated by the main policy network, obtains the reward, and updates the state.

[0155] S104, store the experience tuple in the experience replay buffer. When the buffer reaches the upper limit of the capacity, introduce the latest acquired experience by covering the oldest experience data.

[0156] S105, updating the main policy network and the critic network parameters.

[0157] S106, calculating the loss function of the critic network according to the time difference (TD) target and the value function predicted by the critic network, extracting samples from the experience replay buffer, and updating the target policy network and the evaluation network by using the gradient descent method.

[0158] S107, updating the parameters of the main policy network and the critic network by small batch experience samples.

[0159] S108, updating the parameters of the policy network and the value network in the target network by using a soft update mechanism.

[0160] S109, iteratively training until the algorithm converges stably, applying the strategy to the agent to achieve optimal task offloading and resource allocation.

[0161] In the S102 step, at the beginning of each cycle, the state of the agent is initialized, the agent interacts with the environment, the main policy network generates actions based on the current policy, and the state of the agent is represented as:

[0162]

[0163] Where u j (t-1) = (X j (t-1), Y j (t-1), H j (t-1)), represents the three-dimensional coordinates of the UAV j at the end of time slot t-1, corresponding to the starting point of time slot t; D(t) = {D i (t)}, represents the task input data size of the UE in time slot t; λ(t) = {λ i (t)}, represents the task processing density of the UE in time slot t; represents the computing resource allocation in time slot t; represents the remaining energy of the UAV j at the end of time slot t; represents the time life index of each UAV, and the initial life of each UAV is set to 0.

[0164] In the S103 step, the agent performs the action generated by the main policy network, obtains the reward, and updates the state. The calculation formula of the reward reward in the state update is as follows:

[0165]

[0166] In the above formula, The immediate reward obtained after each agent in the system carries out an action.

[0167] In step S105, the main policy network and critic network parameters are updated; the current policy network is updated by gradient ascent as follows:

[0168]

[0169] Wherein θj represents the parameters of the jth actor network j The gradient related to the objective function.

[0170] In step S106: the loss function of the critic network is calculated according to the TD target and the critic network predicted value function, samples are extracted from the experience replay buffer, and the target policy network and evaluation network are updated by gradient descent. The loss function is represented by Each critic network that obtains the corresponding state and action is represented by

[0171]

[0172] The loss function can be given by the following formula:

[0173]

[0174] Then the parameters w of the current value network of the main value network are updated by gradient descent j As follows:

[0175]

[0176] In the formula TD error represents the time difference error.

[0177] In step S108, a soft update mechanism is used to update the parameters of the policy network and the value network in the target network; the soft update formula is as follows:

[0178]

[0179]

[0180] Wherein θ j θ represents the parameters of the current policy network, θ represents the parameters of the target policy network, w j w represents the parameters of the current value network, w represents the parameters of the target value network, χ∈[0,1].

[0181] In order to elaborate the air-ground network optimization method and system based on MEC and digital twinning, the present application provides two specific application embodiments, including the key details of the implementation scheme.

[0182] In order to prove the creativity and technical value of the technical solutions of the present application, this part is the application examples of the technical solutions of the claims on specific products or related technologies.

[0183] Application Example 1: Intelligent City Management

[0184] 1) Network parameter initialization: Deploy the main policy network and critic network in the environmental monitoring center of the city. Initialize the network parameters, including the learning rates α and β of the critic network and the policy network, the discount factor γ, the initialization of the replay buffer size W, the size of the mini-batch N, and the noise Ψ(t) for action exploration; deploy the necessary IoT devices and drones to realize environmental monitoring and data collection.

[0185] 2) Policy and environment interaction: The drone executes the city monitoring task according to the action generated by the policy network, and the noise is integrated into the policy to explore a better monitoring path.

[0186] 3) Data collection and processing: The environmental data collected by the drone is transmitted to the MEC server for rapid data processing.

[0187] 4) Task and resource allocation: Based on the DRL policy, gradually optimize the data processing and resource allocation, such as the allocation of bandwidth resources in different devices.

[0188] 5) Network update and optimization: Regularly update the policy according to the monitoring effect and resource usage, to ensure that the monitoring system continues to efficiently manage and transmit data.

[0189] Application Example 2: Disaster Emergency Response

[0190] 1) Network parameter initialization: Quickly deploy the main policy network and critic network in the disaster area, and initialize the key network parameters. Use the drone device to provide real-time aerial images and videos to support rapid response to disasters and rescue operations.

[0191] 2) Task execution: The drone executes search and rescue tasks according to the policy. After each rescue action is executed, immediate feedback is obtained according to the rescue effect, and the system state is updated accordingly.

[0192] 3) Data transmission and processing: The rescue information collected by the drone is quickly transmitted to the MEC system for processing to ensure timely delivery of information.

[0193] 4) Dynamic resource allocation: Through the DRL policy, dynamically adjust the allocation of rescue resources, such as the coordination of drones and ground rescue teams, to optimize rescue efficiency.

[0194] 5) Strategy iteration and update: Based on the actual rescue progress and feedback, continuously adjust and optimize the strategy to improve the response speed and rescue effect, and ensure that the rescue operation is more accurate and efficient.

[0195] In these two application instances, an efficient, reliable, and energy-saving joint optimization method is demonstrated for the MEC and digital twin-based air-ground network optimization method and system. This method not only applies to smart city management, but also to disaster emergency response and other scenarios, further demonstrating its broad application potential and technical advantages.

[0196] The application embodiment of the present application provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the MEC and digital twin-based air-ground network optimization method.

[0197] The application embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the MEC and digital twin-based air-ground network optimization method.

[0198] The application embodiment of the present application provides an information data processing terminal, which comprises an MEC and digital twin-based air-ground network optimization system.

[0199] In order to compare and analyze the present application with the current industry standard algorithm, we conducted a standardized performance benchmark test. This analysis reveals the significant advantages of the present application in key performance indicators and provides deep insights into potential performance bottlenecks.

[0200] 1、The present application: represents the proposed MEC and digital twin-based air-ground network optimization method.

[0201] 2、Random movement: In this algorithm, the UAV moves randomly. We allow the device to move randomly within the range and change its speed while letting the UAV move randomly. In addition, the strategies including association strategy, cache decision, mode selection and task offloading are still optimized using the proposed algorithm.

[0202] 3、Access cache: In this algorithm, only the association strategy, mode selection and task offloading ratio are optimized, and the input data cache decision is randomly generated.

[0203] 4、Access offloading: In this algorithm, all input data is not cached in the UAV's cache memory, and is completely calculated using the offloading model. The remaining three strategies are optimized using the proposed algorithm.

[0204] 5. Random offloading: In this algorithm, the mode selection and task offloading are determined by random values, and the offloading proportion of tasks is also given randomly. The access control and cache decision are optimized by the proposed algorithm.

[0205] From Figure 4 It can be seen from the above that the five algorithms all show good convergence. However, compared with the other four methods, the proposed method is superior to them in terms of convergence speed and reward obtained. In terms of convergence speed, the proposed algorithm converges slightly faster than the other algorithms. In terms of reward performance, the algorithm of the present application maintains a relatively stable reward throughout the convergence process, while some algorithms will have certain fluctuations after convergence, or the final reward value is not as good as the algorithm of the present application.

[0206] Figure 5 The relationship between network lifetime and UAV residual energy threshold is represented by a bar chart. The residual energy of the UAV depends on the energy consumption of the UAV when flying and the energy consumption of the UAV when processing UE tasks, which is affected by mode selection. Therefore, different mode selection leads to significant changes in residual energy, thereby affecting network lifetime. As can be seen from the figure, with the increase of the residual energy threshold, the network lifetime shown by the five algorithms all shows a downward trend. Nevertheless, the proposed algorithm always maintains a relatively high network lifetime

[0207] It should be noted that the embodiments of the present application can be realized by hardware, software or a combination of software and hardware. The hardware part can be realized by using special logic; the software part can be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or a specially designed hardware. Those skilled in the art can understand that the above-mentioned devices and methods can be realized by using computer executable instructions and / or included in processor control code, such as provided on a carrier medium, such as a magnetic disk, CD or DVD-ROM, a programmable memory, such as a read-only memory (firmware), or a data carrier, such as an optical or electronic signal carrier. The devices of the present application and their modules can be realized by hardware circuit, such as ultra-large scale integrated circuit or gate array, semiconductor, such as logic chip, transistor, etc., or programmable hardware device, such as field programmable gate array, programmable logic device, etc., or by software executed by various types of processors, or by a combination of the above-mentioned hardware circuit and software, such as firmware.

[0208] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any modification, equivalent replacement and improvement made by those skilled in the art within the technical scope disclosed by the present application, as long as it is within the spirit and principle of the present application, should be covered within the protection scope of the present application.

Claims

1. A method for air-ground network optimization based on MEC and digital twinning, characterized in that, The agent is learned and decided by the main policy network and critic network of deep reinforcement learning; the parameters of the main policy network and critic network are initialized, and the training related parameters are set; the agent interacts with the environment according to the current policy, executes the action and updates the state; the experience replay buffer is used to store the experience, and the latest learning experience is updated when the capacity is reached; the loss function is calculated and the gradient descent method is used to update the parameters of the main policy network and critic network; the main policy network and critic network parameters in the target network are updated regularly to ensure the stability of learning; The training is continuously performed until the policy converges, and then the policy is applied to complete task allocation and trajectory planning; The air-ground network optimization method based on MEC and digital twinning further comprises: S101, initialize the parameters θ of the main policy network and the critic network of the intelligent agent j and ω j , the parameters of the target main policy network and the critic network and the number of rounds MP, the maximum training step T max , initialize the learning rates α and β corresponding to the critic network and the main policy network, the discount factor γ, initialize the replay buffer size W, the size N of the mini-batch, and the noise Ψ(t) for action exploration; initialize the network layout parameters, including the number of UEs I, the number of drones J; S102, initialize the state of the agent, the agent interacts with the environment, and the main policy network generates an action based on the current policy; S103, the agent executes the action generated by the main policy network, obtains a reward, and updates the state; S104, store the experience tuple into the experience replay buffer; when the buffer reaches the capacity limit, the latest acquired experience is introduced by covering the oldest experience data; S105, update the parameters of the main policy network and critic network; S106, calculate the loss function of the critic network according to the TD target and the value function predicted by the critic network, extract samples from the experience replay buffer, and update the target main policy network and critic network by using the gradient descent method; S107, update the parameters of the main policy network and critic network through batch experience samples; S108, update the parameters of the main policy network and critic network in the target network by using a soft update mechanism; S109, iteratively train until the algorithm converges stably, and apply the policy to the agent to achieve optimal task allocation and trajectory planning.

2. The MEC and digital twin based air-ground network optimization method of claim 1, wherein, In the S102, the state of the agent is initialized, the agent interacts with the environment, and the main policy network generates an action based on the current policy, and the state of the agent is represented as: wherein represents the three-dimensional coordinates of the UAV j at the end of time slot t-1, corresponding to the starting point of time slot t; represents the task input data size of the UE within time slot t; represents the task processing density of the UE within time slot t; represents the computation resource allocation within time slot t; represents the residual energy of the UAV j at the end of time slot t; represents the time life index of each UAV, and the initial life of each UAV is defined as 0. 3.The MEC and digital twinning based air-ground network optimization method of claim 1, wherein, In the S103, the agent executes the action generated by the main policy network, obtains a reward, and updates the state, and the calculation formula of the reward reward in the state update is as follows: In the above formula represents the immediate reward obtained by each agent in the system after performing an action. 4.The MEC and digital twinning based air-ground network optimization method of claim 1, wherein, In the S105, the parameters of the main policy network and critic network are updated; the current main policy network is updated by gradient ascent as follows: wherein, denotes the parameters θ of the j-th master policy network j a gradient related to the objective function, In the S106, the loss function of the critic network is calculated according to the TD target and the value function predicted by the critic network, samples are extracted from the experience replay buffer, and the target main policy network and critic network are updated by using the gradient descent method; the TD target can be given by the following formula: The loss function can be given by the following formula: The parameters w of the critic network are then updated by gradient descent j As follows: 5.The MEC and digital twinning based air-ground network optimization method of claim 1, wherein, In the S108, the parameters of the main policy network and critic network in the target network are updated by using a soft update mechanism; the soft update formula is as follows: where θ j represent the parameters of the current main policy network, represent the parameters of the target main policy network, w j represent the parameters of the critic network, represent the parameters of the target critic network, χ ∈ [0, 1]. 6.A MEC and digital twin based space-air network optimization system for implementing the MEC and digital twin based space-air network optimization method according to any one of claims 1 to 5. It comprises: The system initialization module is configured to initialize parameters of the deep deterministic policy gradient algorithm, including setting the number of rounds MP, the maximum number of training steps T max , initializing learning rates α and β corresponding to the critic network and the main policy network, a discount factor γ, and initializing the size W of the replay buffer and the size N of the mini-batch. A network construction module is used for noise exploration of actions; network layout parameters are initialized, including the number of UEs I and the number of drones J; An agent module is used for generating actions based on the current network state at the beginning of each period, and has the function of adding exploratory noise in these behaviors, so as to introduce a certain randomness in the execution process; An action execution module for executing the resource allocation and task offloading strategy; A reward obtaining module for executing the action and calculating the immediate reward, a state transition module for evaluating the reward according to the long-term average utility of all devices of the system and transferring the system from the current state to the next state; An experience replay module for storing the experience tuple of each time of system state, executed action, obtained reward and next state; A data sampling module for sampling mini-batch experience from the stored experience replay module for learning; A network updating module for updating the main policy network and critic network according to the data of the experience replay module, including a parameter optimization unit for adjusting the network parameters using the gradient ascent method and the gradient descent method; A parameter updating module for updating the parameters of the main network to the target main policy network and the target critic network, adopting a soft update strategy so that the parameters of the target network are the weighted average of the parameters of the main network, and realizing this synchronization process through a parameter synchronization unit. 7.A computer device, comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to make the processor execute the steps of the method for optimizing the air-ground network based on MEC and digital twinning according to any one of claims 1-5. 8.A computer readable storage medium, storing a computer program, the computer program being executed by a processor to make the processor execute the steps of the method for optimizing the air-ground network based on MEC and digital twinning according to any one of claims 1-5. 9.An information data processing terminal, comprising the system for optimizing the air-ground network based on MEC and digital twinning according to claim 6.

Images