Active reasoning-based optimization methods in MEC-enabled low-altitude metaverse systems

Abstract

This invention belongs to the field of wireless communication technology and discloses an optimization method based on active reasoning in a MEC-enabled low-altitude meta-universe system. By jointly optimizing the flight trajectory and resource allocation strategy of UAVs, this invention achieves low power consumption and low latency for IoT devices while ensuring user service quality, and also provides auxiliary means to improve the economic benefits for operators. Addressing the dynamic nature of networks, the uncertainty of information acquisition, and the incompleteness of data usage, this invention transforms the optimization problem into a Markov decision process and employs an improved deep learning algorithm based on active reasoning. This algorithm considers factors other than the reward dimension in addition to the dynamic learning guidance, exploring personalized strategies based on agent preferences.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, and particularly relates to an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system. Background Technology

[0002] Multi-access edge computing is a distributed computing architecture that moves data processing and computing tasks to edge devices closer to the data source, rather than centralizing them in a remote data center. This approach reduces data transmission latency and improves real-time performance and processing efficiency. Edge computing is particularly suitable for applications requiring instant response, such as smart cities, industrial automation, and the Internet of Things (IoT). By deploying computing resources at the network edge, edge computing can also reduce the burden on central servers, improving the overall performance and reliability of the system. Air-Ground Integrated Network (AGIN) is a communication network system that integrates air and ground network resources. Its core concept is to achieve seamless expansion of network coverage and efficient resource utilization by tightly integrating airborne communication facilities (such as low-Earth orbit satellites and drones) with ground communication infrastructure (such as base stations and fiber optic cables). This network structure can effectively solve the problem of insufficient coverage of traditional ground networks in remote areas or disaster relief, improving network reliability and flexibility. At the same time, AGIN supports higher data transmission rates and lower latency, helping to enhance the performance of applications such as mobile communications, IoT, and remote control. By dynamically allocating network resources in the air and on the ground, the integrated air-ground network can not only optimize network performance but also provide broader service coverage, meeting the modern society's demand for efficient and reliable communication.

[0003] There are some challenges to overcome when combining MEC and drones into an integrated air-ground network architecture.

[0004] (1) Resource Management and Scheduling: The allocation and scheduling of resources for edge computing and drones require precise coordination to avoid resource conflicts and improve the overall efficiency of the system. This requires efficient resource management in dynamic environments.

[0005] (2) Data processing and latency: Although edge computing can reduce latency, the computing power of drones is limited, and the processing of large amounts of data may still face latency issues, especially under high data traffic conditions.

[0006] (3) Energy Management: Drones have limited battery life, and edge computing nodes also require a stable power supply. Optimizing energy use and extending drone runtime is a key issue.

[0007] (4) Computing power and storage requirements: Drones typically have limited computing and storage capabilities. How to effectively utilize these limited resources in an edge computing environment and ensure the efficiency of data processing is a challenge.

[0008] Secondly, Deep Reinforcement Learning (DRL) is an effective approach for optimizing MEC-enabled low-altitude metaverse systems. DRL combines deep learning and reinforcement learning to solve complex decision-making and control problems. In DRL, agents learn through interaction with the environment and optimize their behavioral policies based on the reward signals received. Deep learning is used in DRL to extract important features from high-dimensional input data (such as images), while reinforcement learning drives the agent to continuously improve its decision-making policies through reward mechanisms, thereby maximizing long-term rewards. DRL can handle complex problems that traditional methods struggle with, such as playing video games, autonomous driving, and robot control, when dealing with tasks in both continuous and discrete action spaces. Many classic DRL algorithms exist, among which Deep Q-Network (DQN) is a significant milestone. It overcomes the limitations of traditional Q-learning in high-dimensional state spaces by introducing deep neural networks to approximate the Q-value function. Another important algorithm is the Policy Gradient method, which directly optimizes the policy function by calculating the gradient of the policy to optimize the decision. A relatively novel algorithm is Proximal Policy Optimization (PPO), which maintains stability and effectiveness by limiting the magnitude of policy updates. These algorithms each have their advantages and disadvantages in solving practical problems and can cope with complex decision-making challenges in different environments. However, these algorithms still have some shortcomings that need improvement.

[0009] (1) On the one hand, in complex environments, DRL algorithms generally rely on fixed reward functions. When applied to agents with different preferences or characteristics, the training effect may be very poor due to the weak generalization ability of the algorithm.

[0010] (2) On the other hand, traditional DRL struggles to strike a balance between exploration (discovering new strategies) and exploitation (utilizing known features and states). This limitation often prevents agents from effectively using prior preferences to explore unknown states.

[0011] The main technical problems existing in the industrial application of current technologies are reflected in the following aspects:

[0012] 1. Weak generalization ability:

[0013] Classical deep reinforcement learning (DRL) algorithms often rely on fixed reward functions, which leads to insufficient generalization ability when facing agents with different preferences or characteristics, and the training effect may be greatly reduced.

[0014] 2. Imbalance between exploration and utilization:

[0015] Traditional DRL algorithms struggle to strike a balance between exploring new strategies and leveraging known features and states. This imbalance hinders agents from effectively utilizing prior preferences to explore unknown states, thus impacting the overall performance and efficiency of the algorithm.

[0016] 3. Unreasonable allocation of resources:

[0017] In ground-based computing scenarios, especially when resources such as bandwidth are extremely scarce, existing literature and technologies have not adequately considered how to allocate resources more rationally. This may lead to underutilization of resources, thereby affecting the overall performance and stability of the system.

[0018] 4. Drone path planning problem:

[0019] In air-ground integrated scenarios, drone path planning is a critical issue. Existing technologies may not fully consider the drone's perception coverage problem, that is, how to maximize its environmental perception and coverage capabilities while ensuring that the drone can effectively perform its mission.

[0020] 5. Insufficient algorithm adaptability:

[0021] Drones and other IoT devices in air-to-ground integrated scenarios are often highly dynamic, requiring algorithms to be more adaptable. However, most classic DRL algorithms lack this characteristic and struggle to adapt to complex and diverse changing needs.

[0022] In summary, the main technical challenges faced by existing technologies in industrial applications are weak generalization ability, an imbalance between exploration and utilization, unreasonable resource allocation, drone path planning problems, and insufficient algorithm adaptability. These issues limit the effectiveness and scope of existing technologies in practical applications, necessitating the introduction of new mechanisms and methods for improvement and optimization. Summary of the Invention

[0023] To address the problems existing in the prior art, this invention provides an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system.

[0024] This invention is implemented as follows: an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system includes:

[0025] S101. Initialize the global network parameters of the agent;

[0026] Initialize the agent's global network θ, number of rounds M, maximum number of training steps T, number of optimization iterations per step I, number of candidate policies J, number of optimal candidate policies k, initialize the global network's learning rate α, discount factor γ, initialize the replay buffer size D, and initialize the policy distribution. Initialize the transition probability distribution Initialize hyperparameters, such as carrier frequency f c Time slot length Δt, etc.;

[0027] S102. At the start of each round, the initial state s t The policy distribution is set, and J alternative policies are obtained by random sampling.

[0028] S103. Randomly sample J actions from these strategies, then obtain J conditional transition probability distributions based on the J alternative strategies, and calculate the corresponding current reward from the corresponding J actions.

[0029] S104. Based on active reasoning and the principle of free energy, the free energy of each alternative strategy is obtained using cumulative reward and conditional transition probability distribution.

[0030] S105. Calculate the average of the first k smallest free energies and obtain the current policy distribution based on the average value;

[0031] S106. Based on the obtained policy distribution, sample out the policy, then sample out the action as the current agent's behavior, and interact with the environment to obtain the next state.

[0032] S107. Store the experience tuple into the replay buffer; if the replay buffer is full, delete the oldest experience and store the latest experience.

[0033] S108. Calculate the predicted value of the global network output free energy, find the error function between the predicted and actual free energy, and update the global network parameters using backpropagation and gradient descent.

[0034] S109. Repeat the training until the algorithm converges, and finally obtain the policy distribution. Thus, the policy for each step can be randomly selected from the distribution to control the agent's actions and obtain the optimal joint UAV trajectory planning and resource offloading allocation.

[0035] Furthermore, in S102, at the beginning of each round, an initial state is set; J candidate policies are obtained by random sampling of the policy distribution, and J actions are then obtained by random sampling from each of the J policies; the agent's state is represented as follows:

[0036] S(t)={λ(t),W(t),u(t-1),d(t),h Rice (t),h(t),f(t),fVSP (t),f E (t),R V-E (t),R UE (t)},

[0037] Where λ(t)={λ m (t)}, λ m W(t) represents the sensing rate of UAV m in time slot t; W(t) = {W m (t)}, W m (t) represents the bandwidth resource occupied by UAV m in time slot t; u(t-1) = {u m (t-1)}, Where u m (t-1)={X m (t-1),Y m (t-1),H m (t-1)} is the coordinate of UAV m at the end of time slot t-1; d(t)={d m (t)}, d m (t) represents the distance between the UAV m and the ground base station in time slot t; Let m be the Rice value of the UAV in time slot t; h(t) = {h m (t)}, h m f(t) represents the information gain between the UAV m and the ground base station in time slot t; f(t) = {f m (t)}, f m (t) represents the available computing resources of UAV m in time slot t; f VSP (t) represents the computing resources available to the virtual service operator in time slot t, f E (t) represents the available computing resources of the edge service platform in time slot t, R V-E (t) represents the data transmission rate from the virtual service operator to the edge service platform in time slot t, R UE (t) is the data rate at which the operator transmits the processed data back to the user equipment in the metaverse during time slot t; self-policy π t The action of sampling generation is represented as:

[0038] a(t)={t sens (t),Δ(t),t fly (t),κ(t),ι(t)},

[0039] in, The duration of the drone's perception is represented by Δ(t) = {Δ m (t)}={ΔXm (t),ΔY m (t),ΔH m (t)}, Indicates the drone's trajectory. Let κ(t) represent the flight time of the drone. m (t)}, ι(t) represents the data offloading ratio of drones, and ι(t) represents the data offloading ratio of virtual service operators.

[0040] Furthermore, in S103: J conditional transition probability distributions are obtained based on J alternative strategies, and the corresponding current reward is calculated from the corresponding J actions. The formula for calculating the immediate reward is as follows:

[0041]

[0042] In the above formula It represents the reciprocal of the weighted sum of energy consumption, latency, and virtual service operator costs of all devices in the system during time slot t. In other words, the denominator is the objective function we are looking for.

[0043] Furthermore, in S104: based on active reasoning and the principle of free energy, cumulative rewards are utilized. and conditional transition probability distribution The free energy of the J alternative strategies is obtained, where the formula for calculating the inverse of the free energy is:

[0044]

[0045] S105: Averaging the first k smallest free energies and obtaining the current policy distribution based on the average. This process is equivalent to taking the first k largest values ​​of the inverse free energy, sorting the values ​​from largest to smallest, and then calculating the average.

[0046]

[0047] And obtain the policy distribution:

[0048]

[0049] Where σ(·) represents a continuous distribution that is exponentially related to the natural number e, such as the exponential distribution and the gamma distribution;

[0050] S106: Based on the obtained strategy distribution Sampling Strategy π t Then sample action a t The current agent's behavior is used to interact with the environment and obtain the next state; the extraction process is summarized as follows: π t ~q(π), a t ~πt .

[0051] Furthermore, in S108: the predicted value of the global network output free energy Q(s) t ,a t ;θ), find the result with respect to the objective (i.e., the actual free energy). The error function L(θ) is used to update the global network parameters using backpropagation and gradient descent; the loss function is given by the following equation:

[0052]

[0053] The gradient of the loss function is given by the following equation:

[0054]

[0055] Then, the parameters of the global network are updated using gradient descent, as follows:

[0056] θ←θ-α·▽ θ L(θ),

[0057] Where θ is the internal parameter of the global model network, and α is the learning rate.

[0058] Another objective of this invention is to provide an optimization system based on active reasoning in a MEC-enabled low-altitude metaverse system, comprising:

[0059] The system initialization module is used to initialize the parameters of the deep deterministic policy gradient algorithm. It includes a configuration module for setting the network learning rate, discount factor and replay buffer size, and a network construction module for defining network layout parameters such as the number of drones and the location of ground base stations.

[0060] The agent module is used to generate actions based on the current network state at the beginning of each cycle. This agent module uses an active reasoning mechanism and the free energy principle to fit the distribution of the policy by selecting the mean of the largest k minimum free energies.

[0061] The action execution module is used to execute UAV path planning and task unloading strategies.

[0062] The reward acquisition module is used to perform actions and calculate instant rewards. The reward acquisition module calculates rewards based on the reciprocal of the weighted average of latency, energy consumption and operator costs of all devices in the system. It also includes a state transition module that transfers the system state from the current state to the next state.

[0063] The experience replay module is used to store the experience tuples for each system state, action, reward, cumulative reward, and next state.

[0064] The sampling module is used to sample data from the distribution of the policy. The module for sampling from the middle and the module for sampling from strategy π;

[0065] The global network update module is used for global network updates using the backpropagation algorithm. This module includes a parameter optimization unit that uses gradient descent to adjust network parameters.

[0066] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform steps of an optimization method based on active reasoning in the MEC and UAV-enabled metaverse system.

[0067] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of an optimization method based on active reasoning in the MEC and UAV-enabled metaverse system.

[0068] Another objective of this invention is to provide an information data processing terminal for implementing an optimization system based on active reasoning in the MEC and UAV-enabled metaverse system.

[0069] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0070] First, this invention significantly improves the overall efficiency and flexibility of the system. In this innovative architecture, the drone not only acts as a mobile computing and communication node but also compensates for the insufficient coverage of ground-based IoT devices, especially in remote or hard-to-access areas. The drone can provide immediate computing and communication support in these areas, collaborating with ground-based IoT devices to achieve data collection, preliminary processing, and rapid transmission. This integrated air-ground solution expands network coverage, enabling efficient fulfillment of various computing needs even in areas with limited network coverage, optimizing data processing efficiency, and improving the overall performance of the system. The dynamic flexibility of the drone allows it to adjust the deployment of computing resources according to real-time needs, achieving efficient data forwarding and task processing, reducing data transmission latency, and effectively alleviating bandwidth bottlenecks. Simultaneously, MEC technology brings computing resources closer to the data source, effectively reducing the time for data transmission to the remote cloud, thereby significantly improving data processing speed and system responsiveness. This near-source computing approach not only enhances the system's real-time processing capabilities but also strengthens its adaptability to different environmental conditions, enabling the system to maintain high stability and reliability in dynamically changing network demands and environments. By combining the maneuverability of drones with the efficient data processing capabilities of MEC technology, air-to-ground integrated networks can achieve more efficient data processing, lower latency, and better resource utilization, providing more reliable, flexible, and efficient solutions for various complex and dynamic application scenarios. This combination not only drives the technological advancement of intelligent networks but also opens up new possibilities for future network applications.

[0071] The significant technological advancement achieved by this invention lies in realizing an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system. This method has made significant progress in the following key aspects:

[0072] 1) Efficient task unloading and allocation strategy:

[0073] This method significantly improves resource utilization in the air-ground integrated metaverse network by optimizing the resource allocation ratio from UAV offloading to ground base stations and from base stations to edge service platforms. This optimization not only improves the overall system performance but also effectively reduces energy consumption and latency.

[0074] 2) Reasonable drone path planning strategy:

[0075] This method optimizes the system by analyzing the UAV's three-dimensional coordinates, enabling the system to automatically adjust the UAV's flight path to ensure comprehensive perception coverage of the target area and adapt to constantly changing network conditions. This optimization process effectively improves the user experience by adaptively adjusting performance parameters such as the UAV's flight time and perception time.

[0076] 3) Integration of reinforcement learning:

[0077] This invention combines Deep Reinforcement Learning (DRL) algorithms with proactive reasoning methods, enabling the system to make decisions that rely not only on a single reward but also on a comprehensive utilization of additional environmental information. By autonomously learning and adapting based on real-time data, the system can make optimal decisions even in the absence of explicit instructions. This adaptive capability is extremely important in dealing with complex and dynamically changing environments.

[0078] 4) Optimization of energy consumption, latency, and operator costs:

[0079] The incentive mechanism incorporated in this method focuses specifically on reducing system energy consumption, latency, and operating costs, which not only improves energy efficiency but is also environmentally friendly. This is particularly crucial given rising energy costs and increasing emphasis on environmental protection.

[0080] 5) Improved system reliability and adaptability:

[0081] By filtering multiple alternative strategies and iterating at each step, the predicted free energy gradually approaches the true value. Therefore, the scheme of using a global network to approximate this value enhances the system's reliability. Furthermore, by focusing on factors other than system rewards, rather than relying on a deterministic reward function, the agent's exploration space is expanded, indirectly improving the system's adaptability.

[0082] 6) The network's self-learning and optimization capabilities:

[0083] This method improves overall performance by continuously optimizing the system's decision-making process through iterative training and experience-based network updates.

[0084] The combined effect of these technological advancements has led to significant performance improvements in MEC-enabled low-altitude metaverse systems, as well as enhancements in energy efficiency, stability, and adaptability. These improvements are crucial for meeting the complex computing demands of the big data era.

[0085] The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system provided by this invention lies in using mathematical models to guide the system's behavior and learning process. The technical effects brought about by these mathematical models can be explored based on their characteristics:

[0086] 1) Calculation of free energy

[0087] The calculation of free energy not only focuses on system energy consumption, latency, and operator costs, but also includes a factor related to environmental information, called information gain. This factor additionally considers agent preferences and increases the agent's subjective initiative.

[0088] Energy efficiency optimization and cost savings: By directly linking rewards to a weighted sum of system energy consumption, latency, and operator costs, this approach encourages agents to explore environments that minimize the weighted sum, thereby achieving a win-win situation of improved user service quality and operator cost savings.

[0089] 2) Global network loss function and update

[0090] By approximating the calculated free energy, the global network updates itself using backpropagation and gradient descent.

[0091] Strategy optimization: By continuously adjusting the global network parameters, the system can learn and adopt more effective decision-making strategy distributions, thereby sampling specific strategies.

[0092] Learning stability: Using the target calculated based on the active reasoning mechanism as a basis can balance the learning process and avoid instability caused by excessive prediction error.

[0093] Performance optimization: By accurately calculating the loss function and updating the network, the accuracy and efficiency of system decision-making are improved.

[0094] 3) Parameter update formula

[0095] The method for updating parameters of the global network is described.

[0096] Gradual strategy approximation: By gradually updating the global model network parameters, the system can smoothly transition to a new strategy, preventing performance fluctuations caused by drastic changes.

[0097] Continuous learning and adaptation: This continuous parameter update mechanism ensures that the system can adapt to long-term environmental changes.

[0098] The application of the mathematical model provided by this invention not only improves the operational efficiency and decision-making quality of MEC-enabled low-altitude metaverse systems, but also enhances their adaptability to environmental changes and long-term stability. These technical effects are crucial for modern edge computing environments that handle massive amounts of data and high-frequency interactions.

[0099] The present invention provides an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system, which optimizes network performance through the interaction between the agent and the environment.

[0100] The agent's state is initialized, and this state includes multiple variables, such as: the drone's perception rate, the bandwidth occupied by the drone, the time-varying channel gain, the distance between the ground base station and the drone, the available computing resources of the drone, the ground base station, and the edge computing platform, the drone's three-dimensional coordinates, and the transmission power. These variables collectively define the agent's environmental state at a specific moment, thereby influencing the agent's decision-making.

[0101] The agent initially simulates actions and receives immediate rewards, then simulates and generates actions to obtain rewards. After a series of tedious selections, the optimal action is determined, at which point the agent executes the actual action and transitions to a new state. The reward calculation formula considers the long-term energy consumption, long-term latency, and operator costs of all devices in the system. This is a comprehensive goal of the system design, aiming to reduce operator costs while improving user experience.

[0102] Calculating the loss function and updating the network, along with using gradient descent to update the current global network, helps improve the global network's prediction accuracy of free energy, thereby optimizing system performance. This part of the operation is analogous to the value function update in deep reinforcement learning, with the key being to improve prediction accuracy to guide policy improvement.

[0103] The application of these steps and mathematical models has led to significant technological advancements:

[0104] Policy optimization: Through deep reinforcement learning, the system can learn and optimize its policies to adapt to the ever-changing network environment.

[0105] Efficient resource utilization: Optimized task offloading and allocation ensures that all computing power devices are utilized efficiently, especially drones with multiple functions.

[0106] Minimize energy consumption: By optimizing long-term energy consumption, it is possible to achieve green communication and reduce environmental impact.

[0107] System stability and adaptability: By employing a unique active reasoning mechanism that pays attention to additional information besides rewards, the system increases the agent's focus on its own preferences, thereby enhancing stability and universality.

[0108] These advancements demonstrate the potential of deep reinforcement learning based on active reasoning in complex network systems, particularly in enabling intelligent and efficient integrated air-ground communication networks.

[0109] Secondly, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0110] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows:

[0111] This invention deeply considers the operating costs of service providers in the metaverse network, effectively reducing energy consumption through innovative technical means while ensuring superior performance in meeting users' stringent latency requirements. Specifically, this invention adopts a series of optimization strategies that not only improve system efficiency at the technical implementation level but also minimize operator costs from an economic perspective. This comprehensive improvement provides a valuable reference for subsequent patent deployment and practical business applications, indicating that operators can achieve higher economic benefits while improving user experience.

[0112] (2) Whether the technical solution of the present invention solves the technical problem that people have long wanted to solve but have never been able to solve successfully:

[0113] Classical DRL algorithms largely adhere to a few fixed reward function design paradigms, making them generally applicable only to specific scenarios. Once the requirements or scenario change, the training results often deviate significantly from expectations, indicating poor generalization performance. While many studies have incorporated and integrated theories into DRL, such as Lyapunov optimization theory and attention mechanisms, they have consistently failed to overcome the curse of weak generalization. This invention combines active reasoning mechanisms from neuroscience with DRL, using free energy to guide the algorithm. By combining research from multiple scholars, the free energy is ultimately converted into a sum of cumulative reward and an indeterminate term. This approach retains the reward element of traditional DRL while incorporating the preferences or characteristics of different agents (abstracted from specific requirements) within the indeterminate term, significantly improving the algorithm's generalization performance.

[0114] Third, this invention proposes an innovative solution to several key problems existing in the industrial application of existing technologies, and achieves significant technological progress.

[0115] First, addressing the weakness of generalization ability in existing DRL algorithms, this invention enhances the algorithm's adaptability to different environments and preferences by introducing active reasoning and the free energy principle. This enables the algorithm to generalize better when facing agents with different characteristics, thereby improving training effectiveness.

[0116] Secondly, this invention addresses the imbalance between exploration and exploitation in traditional DRL algorithms. By optimizing the free energy of alternative strategies and selecting actions based on this free energy, the algorithm achieves a better balance between exploring new strategies and utilizing known states, thus improving overall performance and efficiency.

[0117] Furthermore, this invention also addresses the issue of unreasonable resource allocation in air-to-ground layer computing. By introducing policy distribution and conditional transition probability distribution, the algorithm can allocate resources for drones and other IoT devices more rationally, improving resource utilization and system stability.

[0118] Finally, this invention has achieved significant technological advancements in industrial applications. By optimizing UAV trajectory planning and resource offloading allocation, the algorithm reduces system energy consumption, latency, and cost, thereby improving overall performance. This provides strong technical support for the development of MEC and UAV-enabled metaverse systems, promoting the progress and development of related industries. Attached Figure Description

[0119] Figure 1 This is a flowchart of an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system provided in this embodiment of the invention.

[0120] Figure 2 This is an applicable scenario diagram provided by an embodiment of the present invention.

[0121] Figure 3 This is a flowchart illustrating the implementation of an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system provided in this embodiment of the invention.

[0122] Figure 4 This is a diagram provided by an embodiment of the present invention, which simulates the relationship between the cumulative system reward and operating cost after the algorithm converges and the edge service pricing standard.

[0123] Figure 5 This is a comparison chart of the convergence performance of the present invention with several baseline algorithms.

[0124] Figure 6 This is a block diagram of an optimized system structure based on active reasoning in a MEC-enabled low-altitude metaverse system provided in an embodiment of the present invention. Detailed Implementation

[0125] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0126] The technical solution provided by this invention proposes an optimization method based on active reasoning in a low-altitude meta-universe system enabled by multi-access edge computing (MEC), mainly employing a deep reinforcement learning framework based on active reasoning. This solution primarily includes:

[0127] 1) Deep Reinforcement Learning Framework: It applies a deep neural network to represent the global model and uses Deep Reinforcement Learning (DRL) based on active reasoning for learning and decision-making.

[0128] 2) Network initialization: Initialize the parameters of the global model network and set training-related hyperparameters.

[0129] 3) Environmental interaction and state management: The agent performs virtual interaction with the environment based on several alternative strategies, selects the one with the lowest free energy, calculates the average to obtain the action to be actually executed, and then performs state transition.

[0130] 4) Experience Replay: Experience tuples are stored in a replay buffer. Instead of immediate rewards, cumulative rewards are added to the tuples. The buffer is updated when it overflows to maintain the latest learning experience.

[0131] 5) Network parameter update: The parameters of the entire model network are updated by calculating the loss function and gradient descent.

[0132] 7) Strategy convergence and implementation: Repeat training until the strategy converges, and use the trained strategy for UAV path planning and task offloading assignment.

[0133] The following are two specific embodiments and their implementation schemes provided by the present invention:

[0134] Example 1: Real-time Urban Traffic Monitoring

[0135] Drones are used to monitor urban transportation facilities, collect data, and process it.

[0136] 1) Network parameter initialization: Set network parameters such as the number of drones and the coordinates of the nearest base station.

[0137] 2) Strategy and environment interaction: The computer or intelligent device generates motion simulation monitoring tasks based on alternative strategies, selects several owners of the smallest free energy and calculates the average to obtain the real motion, and the drone performs the motion to complete the displacement and monitoring tasks.

[0138] 3) Data collection and processing: Collect environmental data and transmit it to the monitoring terminal.

[0139] 4) Task and resource allocation: Optimize resource allocation and task offloading allocation based on DRL strategy.

[0140] 5) Network updates and optimizations: Update strategies based on monitoring results and resource utilization.

[0141] Example 2: Real-view Navigation

[0142] When traveling, it is necessary to know the road conditions and the tour routes in natural scenic areas. Deploying drones can help avoid peak congestion and provide reconnaissance and analysis of complex road sections.

[0143] 1) Network parameter initialization: Set the drone deployment plan for the scenic area.

[0144] 2) Mission execution: The drone provides real-time feedback to the user on the road conditions of the scenic spots and the actual view of the internal routes according to the strategy.

[0145] 3) Data transmission and processing: The collected information is transmitted to the base station and MEC for processing to obtain services such as route recommendations.

[0146] 4) Dynamic resource allocation: The DRL strategy dynamically adjusts resources to optimize transmission efficiency.

[0147] 5) Strategy iteration and update: Adjust strategies according to user needs to improve feedback speed.

[0148] Example 3

[0149] To address the problems existing in the prior art, this embodiment provides an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system. Figure 2 This is a scenario diagram illustrating the application of the method of this invention. The Air-Ground Integrated Network (AGIN) supported by the MEC of interest consists of two parts: an air-based layer and a ground-based layer. In this scenario, the virtual service operator deploys a group of drones... The virtual service provider (V2V) offers corresponding virtual services to users within its coverage area. UAVs in the airborne layer can sense the physical environment and collect its attributes or status. The ground-based network mainly consists of ground base stations, edge computing platforms, and metaverse user equipment. Ground base stations are owned by the V2V and possess powerful computing capabilities and a stable power supply. Edge computing platforms perform services within the coverage area of ​​ground base stations; operators can provide economic incentives to edge computing platforms to obtain additional computing resources. UAVs can transmit their sensor data to ground base stations via wireless communication links. After receiving data from UAVs, to reduce their own processing load, ground base stations can also offload some data to edge computing platforms to seek greater computing power and lower latency. A set representing time slots.

[0150] In the relevant scenario, at the beginning of each time slot, users located in various locations submit virtual service requests to the operator. These requests are forwarded to the drone via the base station. To complete the perception task and achieve large-area coverage, the drone needs to perform the action a = {t}. sens ,Δ,t fly ,κ}. This includes the drone's perception time t. sens Flight time t flyThe data includes the displacement Δ in three-dimensional coordinates and the task offloading ratio κ to the ground base station. Ultimately, all sensed data is divided into three parts, processed separately by the UAV, the ground base station, and the edge computing platform. Note that these processes are parallel and simultaneous. Furthermore, in this paper, communication between the UAV and the edge computing platform is blocked; the UAV can only offload tasks to the ground base station, and the edge computing platform only provides direct services to the ground base station.

[0151] Secondly, this invention provides a MEC-enabled low-altitude metaverse system. This system comprises an airborne base layer and a ground-based base layer. The ground-based base layer consists of an edge computing platform, ground base stations, and metaverse user equipment. The airborne base layer is a network of drone swarms responsible for sensing the ground environment. In each fixed time slot, the metaverse user equipment generates tasks, which must be processed jointly by the drones, ground base stations, and the edge computing platform within the specified time slot. By rationally offloading each task during execution and planning the routes of each drone in the airborne base layer, the entire system minimizes energy consumption and latency while saving operator costs.

[0152] Example 4

[0153] The dynamic nature of networks and the uncertainty of information acquisition make the problem quite challenging. To address this effectively, the problem is reconstructed as a Markov Decision Process (MDP). Since agents have different preferences, an improved reinforcement learning algorithm based on active reasoning is used. This algorithm supports both continuous and discrete action spaces, enabling real-time online decision-making.

[0154] like Figure 1 As shown, the optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system provided in this embodiment is characterized by the following steps:

[0155] S101. Initialize the global network parameters of the agent;

[0156] Initialize the agent's global network θ, number of rounds M, maximum number of training steps T, number of optimization iterations per step I, number of candidate policies J, number of optimal candidate policies k, initialize the global network's learning rate α, discount factor γ, initialize the replay buffer size D, and initialize the policy distribution. Initialize the transition probability distribution Initialize hyperparameters, such as carrier frequency f c Time slot length Δt, etc.;

[0157] S102. At the start of each round, the initial state s t The policy distribution is set, and J candidate policies are obtained by random sampling. Then, J actions are obtained by random sampling from each of these policies.

[0158] S103. Randomly sample J actions from these strategies, then obtain J conditional transition probability distributions based on the J alternative strategies, and calculate the corresponding current reward from the corresponding J actions.

[0159] S104. Based on active reasoning and the principle of free energy, the free energy of each alternative strategy is obtained using cumulative reward and conditional transition probability distribution.

[0160] S105. Calculate the average of the first k smallest free energies and obtain the current policy distribution based on the average value;

[0161] S106. Based on the obtained policy distribution, sample out the policy, then sample out the action as the current agent's behavior, and interact with the environment to obtain the next state.

[0162] S107. Store the experience tuple in the replay buffer. If the replay buffer is full, delete the oldest experience and store the latest experience;

[0163] S108. Calculate the predicted value of the global network output free energy, find the error function between the predicted and actual free energy, and update the global network parameters using backpropagation and gradient descent.

[0164] S109. Repeat the training until the algorithm converges, and finally obtain the policy distribution. Thus, the policy for each step can be randomly selected from the distribution to control the agent's actions and obtain the optimal joint UAV trajectory planning and resource offloading allocation.

[0165] In step S102: At the start of each round, the initial state is set; J candidate policies are randomly sampled from the policy distribution, and J actions are then randomly sampled from each of the J policies. The agent's state is represented as follows:

[0166] S(t)={λ(t),W(t),u(t-1),d(t),h Rice (t),h(t),f(t),f VSP (t),f E (t),R V-E (t),R UE (t)},

[0167] Where λ(t)={λ m (t)}, λ m W(t) represents the sensing rate of UAV m in time slot t; W(t) = {W m (t)}, W m (t) represents the bandwidth resource occupied by UAV m in time slot t; u(t-1) = {u m(t-1)}, Where u m (t-1)={X m (t-1),Y m (t-1),H m (t-1)} is the coordinate of UAV m at the end of time slot t-1; d(t)={d m (t)}, d m (t) represents the distance between the UAV m and the ground base station in time slot t; Let m be the Rice value of the UAV in time slot t; h(t) = {h m (t)}, h m f(t) represents the information gain between the UAV m and the ground base station in time slot t; f(t) = {f m (t)}, f m (t) represents the available computing resources of UAV m in time slot t; f VSP (t) represents the computing resources available to the virtual service operator in time slot t, f E (t) represents the available computing resources of the edge service platform in time slot t, R V-E (t) represents the data transmission rate from the virtual service operator to the edge service platform in time slot t, R UE (t) is the data rate at which the operator transmits the processed data back to the user equipment in the metaverse during time slot t. (Self-policy π) t The action of sampling generation is represented as:

[0168] a(t)={t sens (t),Δ(t),t fly (t),κ(t),ι(t)},

[0169] in, The duration of the drone's perception is represented by Δ(t) = {Δ m (t)}={ΔX m (t),ΔY m (t),ΔH m (t)}, Indicates the drone's trajectory. Let κ(t) represent the flight time of the drone. m (t)}, ι(t) represents the data offloading ratio of drones, and ι(t) represents the data offloading ratio of virtual service operators.

[0170] In step S103: Based on the J alternative strategies, J conditional transition probability distributions are obtained, and the corresponding current reward is calculated from the corresponding J actions. The formula for calculating the immediate reward is as follows:

[0171]

[0172] In the formula The denominator is the reciprocal of the weighted sum of energy consumption, latency, and virtual service operator costs of all devices in time slot t, which is our objective function.

[0173] In step S104: Based on active reasoning and the free energy principle, cumulative rewards are utilized. and conditional transition probability distribution The free energy of the J alternative strategies is obtained, where the formula for calculating the inverse of the free energy is:

[0174]

[0175] Furthermore, in S105: averaging the first k smallest free energies and obtaining the current policy distribution based on the average, this process is equivalent to taking the first k largest values ​​of the inverse free energy, sorting the values ​​from largest to smallest, and then calculating the average.

[0176]

[0177] And obtain the policy distribution:

[0178]

[0179] Where σ(·) represents a continuous distribution that is exponentially related to the natural number e, such as the exponential distribution and the gamma distribution.

[0180] In step S106: based on the obtained policy distribution Sampling Strategy π t Then sample action a t The current agent's behavior is used to interact with the environment and obtain the next state; the extraction process is summarized as follows:

[0181]

[0182] In step S108: the global network outputs the predicted value of the free energy Q(s). t ,a t ;θ), find the result with respect to the objective (i.e., the actual free energy). The error function L(θ) is used to update the global network parameters using backpropagation and gradient descent; the loss function is given by the following equation:

[0183]

[0184] The gradient of the loss function is given by the following equation:

[0185]

[0186] Then, the parameters of the global network are updated using gradient descent, as follows:

[0187] θ←θ-α·▽ θ L(θ).

[0188] Where θ is the internal parameter of the global model network, and α is the learning rate.

[0189] To elaborate on the active reasoning-based optimization method in the MEC-enabled low-altitude metaverse system, this invention provides two specific application examples, including key details of the implementation scheme.

[0190] To demonstrate the inventiveness and technical value of the technical solution of this invention, this section provides specific product or related technology application examples of the technical solution claimed.

[0191] Application Example 1: Real-time Urban Traffic Monitoring

[0192] 1) Network parameter initialization: Deploy the global model network in the monitoring department of the city's traffic management center. Initialize network parameters, including the learning rate α of the commentator network, the initial replay buffer size D, the time slot length Δt, and other hyperparameters; deploy multiple drones to monitor traffic facilities and real-time conditions, and coordinate nearby base stations and edge service platforms to assist in task processing.

[0193] 2) Strategy and Environment Interaction: Drones perform monitoring tasks and movement through actions generated based on proactive reasoning mechanisms. Multi-threaded parallelism is beneficial to the computational efficiency of proactive reasoning, therefore the department's equipment needs to support parallel computing and possess superior computing power.

[0194] Data collection and processing: Collect urban traffic data and transmit it to nearby base stations.

[0195] 4) Task offloading and allocation: Optimize data processing and task offloading and allocation based on the DRL strategy. Use reinforcement learning algorithms to gradually improve task allocation.

[0196] 5) Network Updates and Optimization: Update strategies based on monitoring results and resource utilization to ensure the monitoring system maintains efficient data processing and transmission at all times.

[0197] Application Example 2: Reality Navigation

[0198] Network parameter initialization: Deploy a global model network in the scenic area and initialize relevant network parameters. Configure multiple drone devices; the drones can provide real-time aerial observation 3D models, facilitating the intuitive presentation of hotspot information such as road conditions, complex terrain, and population density within the scenic area.

[0199] Mission execution: The drone executes various hot tasks according to the strategy, and performs actions to enter the next state to continue to complete the next round of tasks without interruption.

[0200] Data transmission and processing: The sensed data information is transmitted to the ground base station for processing.

[0201] Dynamic resource allocation: The DRL strategy dynamically allocates resources to optimize task execution efficiency.

[0202] Strategy iteration and updates: Adjust strategies based on user feedback to improve response speed and further enhance the efficiency and accuracy of various tasks.

[0203] In these two embodiments, the active reasoning-based optimization method in the metaverse system that enables MEC and drones provides an efficient, reliable and energy-saving solution suitable for different application scenarios, from real-time urban traffic monitoring to augmented reality navigation, demonstrating its broad application potential and technical advantages.

[0204] To more clearly demonstrate the positive effects achieved by the embodiments of the present invention during the research process, the advantages of the embodiments of the present invention in research and development compared with the prior art will be explained below.

[0205] exist Figure 4 In this paper, the relationship between the average system reward after algorithm convergence and the edge server charging standard is simulated. As the edge service platform charging standard increases, the rewards of all algorithms gradually decrease. This is because when the edge service platform charging standard increases, to reduce costs, tasks will be biased towards local processing on drones and base station processing, thus reducing the task processing volume of the edge service platform to some extent. Simultaneously, for the entire system, lower latency is better, so in practice, to ensure low latency, tasks will still be offloaded to the edge computing platform, so the operating cost does not decrease significantly with the increase in charging standard. It can be seen that the proposed algorithm is superior to the other three algorithms in increasing the reward function. Compared with the other three algorithms, the proposed algorithm can show higher rewards and lower operating costs. This demonstrates that the embodiments of this invention have achieved some positive effects in simulated use and indeed have significant advantages compared with existing technologies.

[0206] exist Figure 5The table shows the convergence performance of all algorithms with reward as the performance metric under default parameter settings. Our algorithm shows a significantly faster convergence speed and exhibits initial signs of stabilization around round 75. Furthermore, it performs well in terms of reward acquisition and post-convergence volatility. This is due to the varying degrees of constraint relaxation in the baseline algorithms, leading to excessive randomness in the agent's exploration along specific directions. Therefore, the process of finding the optimal solution is prolonged, and the convergence volatility remains high. Simultaneously, the increased difficulty in obtaining large immediate rewards results in a characteristic of smaller cumulative rewards. Specifically, compared to several benchmark algorithms, our algorithm consistently achieves higher performance metrics and exhibits less volatility. It is worth noting that some of the comparison algorithms simply modify the agent without imposing strict decision (i.e., action) constraints from different perspectives to cater to unique preferences. Figure 5 The stable convergence of several algorithms observed in the study underscores the ability of our algorithm to adapt to agents with different preferences and highlights its superior generalization ability.

[0207] like Figure 6 As shown, another objective of this invention is to provide an optimization system based on active reasoning in a MEC-enabled low-altitude metaverse system, comprising:

[0208] The system initialization module is used to initialize the parameters of the deep deterministic policy gradient algorithm. It includes a configuration module for setting the network learning rate, discount factor and replay buffer size, and a network construction module for defining network layout parameters such as the number of drones and the location of ground base stations.

[0209] The agent module is used to generate actions based on the current network state at the beginning of each cycle. This agent module uses an active reasoning mechanism and the free energy principle to fit the distribution of the policy by selecting the mean of the largest previous minimum free energy.

[0210] The action execution module is used to execute UAV path planning and task unloading strategies.

[0211] The reward acquisition module is used to perform actions and calculate instant rewards. The reward acquisition module calculates rewards based on the reciprocal of the weighted average of latency, energy consumption and operator costs of all devices in the system. It also includes a state transition module that transfers the system state from the current state to the next state.

[0212] The experience replay module is used to store the experience tuples for each system state, action, reward, cumulative reward, and next state.

[0213] The sampling module is used to sample from the distribution of the strategy and to sample from the strategy.

[0214] The global network update module is used for global network updates using the backpropagation algorithm. This module includes a parameter optimization unit that uses gradient descent to adjust network parameters.

[0215] Another object of the present invention is to provide a computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform steps of an optimization method based on active reasoning in the MEC-enabled low-altitude metacosm system.

[0216] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of an optimization method based on active reasoning in the MEC-enabled low-altitude metacosm system.

[0217] Another objective of this invention is to provide an information data processing terminal for implementing an optimization system based on active reasoning in the MEC-enabled low-altitude meta-universe system.

[0218] Application Example 1: Real-time Urban Traffic Monitoring

[0219] 1) Network parameter initialization: Deploy a global model network in the city's traffic management center's monitoring department. Initialize network parameters, including the learning rate of the commentator network, the initial replay buffer size, time slot length, and other hyperparameters; deploy multiple drones to monitor traffic facilities and real-time conditions, and coordinate nearby base stations and edge service platforms to assist in task processing.

[0220] 2) Strategy and Environment Interaction: Drones perform monitoring tasks and movement through actions generated based on proactive reasoning mechanisms. Multi-threaded parallelism is beneficial to the computational efficiency of proactive reasoning, therefore the department's equipment needs to support parallel computing and possess superior computing power.

[0221] Data collection and processing: Collect urban traffic data and transmit it to nearby base stations.

[0222] 4) Task offloading and allocation: Optimize data processing and task offloading and allocation based on the DRL strategy. Use reinforcement learning algorithms to gradually improve task allocation.

[0223] 5) Network Updates and Optimization: Update strategies based on monitoring results and resource utilization to ensure the monitoring system maintains efficient data processing and transmission at all times.

[0224] Application Example 2: Reality Navigation

[0225] Network parameter initialization: Deploy a global model network in the scenic area and initialize relevant network parameters. Configure multiple drone devices; the drones can provide real-time aerial observation 3D models, facilitating the intuitive presentation of hotspot information such as road conditions, complex terrain, and population density within the scenic area.

[0226] Mission execution: The drone executes various hot tasks according to the strategy, and performs actions to enter the next state to continue to complete the next round of tasks without interruption.

[0227] Data transmission and processing: The sensed data information is transmitted to the ground base station for processing.

[0228] Dynamic resource allocation: The DRL strategy dynamically allocates resources to optimize task execution efficiency.

[0229] Strategy iteration and updates: Adjust strategies based on user feedback to improve response speed and further enhance the efficiency and accuracy of various tasks.

[0230] In these two embodiments, the active reasoning-based optimization method for MEC-enabled low-altitude metaverse systems provides an efficient, reliable, and energy-saving solution suitable for various application scenarios, from real-time urban traffic monitoring to augmented reality navigation, demonstrating its broad application potential and technological advantages.

[0231] exist Figure 4 In this paper, the relationship between the average system reward after algorithm convergence and the edge server charging standard is simulated. As the edge service platform charging standard increases, the rewards of all algorithms gradually decrease. This is because when the edge service platform charging standard increases, to reduce costs, tasks will be biased towards local processing on drones and base station processing, thus reducing the task processing volume of the edge service platform to some extent. Simultaneously, for the entire system, lower latency is better, so in practice, to ensure low latency, tasks will still be offloaded to the edge computing platform, so the operating cost does not decrease significantly with the increase in charging standard. It can be seen that the proposed algorithm is superior to the other three algorithms in increasing the reward function. Compared with the other three algorithms, the proposed algorithm can show higher rewards and lower operating costs. This demonstrates that the embodiments of this invention have achieved some positive effects in simulated use and indeed have significant advantages compared with existing technologies.

[0232] exist Figure 5The table shows the convergence performance of all algorithms with reward as the performance metric under default parameter settings. Our algorithm shows a significantly faster convergence speed and exhibits initial signs of stabilization around round 75. Furthermore, it performs well in terms of reward acquisition and post-convergence volatility. This is due to the varying degrees of constraint relaxation in the baseline algorithms, leading to excessive randomness in the agent's exploration along specific directions. Therefore, the process of finding the optimal solution is prolonged, and the convergence volatility remains high. Simultaneously, the increased difficulty in obtaining large immediate rewards results in a characteristic of smaller cumulative rewards. Specifically, compared to several benchmark algorithms, our algorithm consistently achieves higher performance metrics and exhibits less volatility. It is worth noting that some of the comparison algorithms simply modify the agent without imposing strict decision (i.e., action) constraints from different perspectives to cater to unique preferences. Figure 5 The stable convergence of several algorithms observed in the study underscores the ability of our algorithm to adapt to agents with different preferences and highlights its superior generalization ability.

[0233] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0234] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system, characterized in that, By initializing the global network parameters and hyperparameters of the agent, random sampling is performed using policy distribution and transition probability distribution. The cumulative reward is calculated based on active reasoning and the free energy principle. The free energy is obtained and averaged to generate a new policy distribution. The global network parameters are updated using the backpropagation algorithm and gradient descent. Through repeated training until the algorithm converges, the optimal joint UAV trajectory planning and resource offloading allocation of the agent in the metaverse environment are finally achieved. The method specifically includes the following steps: S101. Initialize the global network parameters of the agent; Initialize the agent's global network θ, number of rounds M, maximum number of training steps T, number of optimization iterations per step I, number of candidate policies J, number of optimal candidate policies k, initialize the global network learning rate α, discount factor γ, initialize the replay buffer size D, initialize the policy distribution q(π), and initialize the transition probability distribution p(s). t |s t-1 Initialize hyperparameters (θ, π). S102. At the start of each round, the initial state s t The policy distribution is set, and J alternative policies are obtained by random sampling. S103. Randomly sample J actions from these strategies, then obtain J conditional transition probability distributions based on the J alternative strategies, and calculate the corresponding current reward from the corresponding J actions. S104. Based on active reasoning and the principle of free energy, the free energy of each alternative strategy is obtained using cumulative reward and conditional transition probability distribution. S105. Calculate the average of the first k smallest free energies and obtain the current policy distribution based on the average value; S106. Based on the obtained policy distribution, sample out the policy, then sample out the action as the current agent's behavior, and interact with the environment to obtain the next state. S107. Store the experience tuple into the replay buffer; if the replay buffer is full, delete the oldest experience and store the latest experience. S108. Calculate the predicted value of the global network output free energy, find the error function between the predicted and actual free energy, and update the global network parameters using backpropagation and gradient descent. S109. Repeat the training until the algorithm converges and finally obtains the policy distribution. Thus, the policy for each step can be randomly selected from the distribution to control the agent's actions and obtain the optimal joint UAV trajectory planning and resource offloading allocation. The air-ground integrated network AGIN supported by MEC consists of two parts: the air base layer and the ground base layer. The ground base layer network consists of ground base stations, edge computing platforms, and metaverse user equipment.

2. The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system as described in claim 1, characterized in that, In step S102, at the beginning of each round, an initial state is set; J candidate policies are obtained by random sampling of the policy distribution, and J actions are then obtained by random sampling from each of the J policies; the agent's state is represented as follows: Where λ(t)={λ m (t)}, λ m W(t) represents the sensing rate of UAV m in time slot t; W(t) = {W m (t)}, W m (t) represents the bandwidth resource occupied by UAV m in time slot t; u(t-1) = {u m (t-1)}, Where u m (t-1)={X m (t-1),Y m (t-1),H m (t-1)} is the coordinate of UAV m at the end of time slot t-1; d(t)={d m (t)}, d m (t) represents the distance between the UAV m and the ground base station in time slot t; Let m be the Rice value of the UAV in time slot t; h(t) = {h m (t)}, h m f(t) represents the information gain between the UAV m and the ground base station in time slot t; f(t) = {f m (t)}, f m (t) represents the available computing resources of UAV m in time slot t; f VSP (t) represents the computing resources available to the virtual service operator in time slot t, f E (t) represents the available computing resources of the edge service platform in time slot t, R V-E (t) represents the data transmission rate from the virtual service operator to the edge service platform in time slot t, R UE (t) is the data rate at which the operator transmits the processed data back to the user equipment in the metaverse during time slot t; self-policy π t The action of sampling generation is represented as: a(t)={t sens (t),Δ(t),t fly (t),κ(t),ι(t)}, in, The duration of the drone's perception is represented by Δ(t) = {Δ m (t)}={ΔX m (t),ΔY m (t),ΔH m (t)}, Indicates the drone's trajectory. Let κ(t) represent the flight time of the drone. m (t)}, ι(t) represents the data offloading ratio of drones, and ι(t) represents the data offloading ratio of virtual service operators.

3. The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system as described in claim 1, characterized in that, S103: Based on J alternative strategies, obtain J conditional transition probability distributions, and calculate the corresponding current reward from the corresponding J actions. The formula for calculating the immediate reward is as follows: In the above formula It represents the reciprocal of the weighted sum of energy consumption, latency, and virtual service operator costs of all devices in the system during time slot t. In other words, the denominator is the objective function we are looking for.

4. The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system as described in claim 3, characterized in that, S104: Based on active reasoning and the principle of free energy, utilizing cumulative rewards and conditional transition probability distribution The free energy of the J alternative strategies is obtained, and the formula for calculating the inverse of the free energy is: S105: Averaging the first k smallest free energies and obtaining the current policy distribution based on the average. This process is equivalent to taking the first k largest values ​​of the inverse free energy, sorting the values ​​from largest to smallest, and then calculating the average. And obtain the policy distribution: Where σ(·) represents a continuous distribution that is exponentially related to the natural number e, including the exponential distribution and the gamma distribution; S106: Based on the obtained strategy distribution Sampling Strategy π t Then sample action a t The current agent's behavior is used to interact with the environment and obtain the next state; the extraction process is summarized as follows: p t ~q(π), a t ~π t 。 5. The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system as described in claim 4, characterized in that, S108: The predicted value of the global network output free energy Q(s) t ,a t ;θ), find the result with respect to the target The error function L(θ) is used to update the global network parameters using backpropagation and gradient descent. The loss function is given by the following formula: The gradient of the loss function is given by the following equation: Then, the parameters of the global network are updated using gradient descent, as follows: θ←θ-α·▽ θ L(θ), Where θ is the internal parameter of the global model network, and α is the learning rate.

6. An optimization system based on active reasoning in a MEC-enabled low-altitude metaverse system, implementing the active reasoning-based optimization method as described in any one of claims 1-5, characterized in that, The MEC-enabled low-altitude metaverse system based on active reasoning includes: The system initialization module is used to initialize the parameters of the deep deterministic policy gradient algorithm. It includes a configuration module for setting the network learning rate, discount factor and replay buffer size, and a network construction module for defining network layout parameters such as the number of drones and the location of ground base stations. The agent module is used to generate actions based on the current network state at the beginning of each cycle. This agent module uses an active reasoning mechanism and the free energy principle to fit the distribution of the policy by selecting the mean of the largest k minimum free energies. The action execution module is used to execute UAV path planning and task unloading strategies. The reward acquisition module is used to perform actions and calculate instant rewards. The reward acquisition module calculates rewards based on the reciprocal of the weighted average of latency, energy consumption and operator costs of all devices in the system. It also includes a state transition module that transfers the system state from the current state to the next state. The experience replay module is used to store the experience tuples for each system state, action, reward, cumulative reward, and next state. The sampling module is used to sample data from the distribution of the policy. The module for sampling from the middle and the module for sampling from strategy π; The global network update module is used for global network updates using the backpropagation algorithm. This module includes a parameter optimization unit that uses gradient descent to adjust network parameters.

7. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the active reasoning-based optimization method in the MEC-enabled low-altitude metacosm system as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of an optimization method based on active reasoning in a MEC-enabled low-altitude metacosm system as described in any one of claims 1-5.

9. An information data processing terminal, characterized in that, The information data processing terminal is used to implement an optimization system based on active reasoning in the MEC-enabled low-altitude metaverse system as described in claim 6.

Images