Optimization method based on active reasoning in MEC enabled low-altitude element cosmic system

Abstract

The invention belongs to the technical field of wireless communication, and discloses an optimization method based on active reasoning in an MEC-enabled low-altitude element cosmic system, which realizes low energy consumption and low delay of Internet of Things equipment on the premise of ensuring user service quality through joint optimization of a flight path and a resource allocation strategy of an unmanned aerial vehicle, and improves user experience. And meanwhile, an auxiliary means is provided for improving the economic benefits of operators. Aiming at the dynamic nature of the network, the uncertainty of information acquisition and the incompleteness of data use, an optimization problem is converted into a Markov decision process, an improved deep learning algorithm based on active reasoning is adopted, factors except the reward dimension are additionally considered on the guidance of dynamic learning, and the dynamic learning algorithm is optimized. And a personalized strategy is explored according to the preference of the intelligent agent.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology, and in particular to an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system. Background Art

[0002] Multi-access edge computing is a distributed computing architecture that moves data processing and computing tasks to edge devices closer to the data source rather than concentrating them in remote data centers. This approach reduces data transmission latency and improves real-time performance and processing efficiency. Edge computing is particularly suitable for applications that require instant response, such as smart cities, industrial automation, and the Internet of Things (IoT). By deploying computing resources at the edge of the network, edge computing can also reduce the burden on central servers and improve the overall performance and reliability of the system. Air-Ground Integrated Network (AGIN) is a communication network system that integrates air and ground network resources. Its core concept is to achieve seamless expansion of network coverage and efficient use of resources by closely integrating air communication facilities (such as low-orbit satellites and drones) with ground communication infrastructure (such as base stations and optical cables). This network structure can effectively solve the problem of insufficient coverage of traditional ground networks in remote areas or post-disaster rescue, and improve the reliability and flexibility of the network. At the same time, the air-ground integrated network supports higher data transmission rates and lower latency, which helps to enhance the performance of application scenarios such as mobile communications, the Internet of Things, and remote control. By dynamically allocating network resources in the air and on the ground, the air-ground integrated network can not only optimize network performance, but also provide a wider range of service coverage to meet the needs of modern society for efficient and reliable communications.

[0003] When combining MEC and drones into an air-ground integrated network architecture, some challenges need to be addressed.

[0004] (1) Resource management and scheduling: Resource allocation and scheduling of edge computing and drones need to be precisely coordinated to avoid resource conflicts and improve the overall efficiency of the system. This requires efficient resource management in a dynamic environment.

[0005] (2) Data processing and latency: Although edge computing can reduce latency, the computing power of drones is limited, and the processing of large amounts of data may still face latency issues, especially under high data traffic conditions.

[0006] (3) Energy management: The battery life of drones is limited, and edge computing nodes also require a stable power supply. Optimizing energy usage and extending the operating time of drones is a key issue.

[0007] (4) Computing power and storage requirements: Drones usually have limited computing and storage capabilities. How to effectively utilize these limited resources in an edge computing environment and ensure efficient data processing is a challenge.

[0008] Secondly, deep reinforcement learning (DRL) is an effective solution for optimizing the low-altitude metaverse system enabled by MEC. DRL is a technology that combines deep learning and reinforcement learning to solve complex decision-making and control problems. In DRL, the agent learns through interaction with the environment and optimizes its behavior strategy based on the reward signal obtained. Deep learning technology is used in DRL to extract important features from high-dimensional input data (such as images), while reinforcement learning drives the agent to continuously improve its decision-making strategy through a reward mechanism to achieve the goal of maximizing long-term returns. When dealing with tasks in continuous and discrete action spaces, DRL can handle complex problems that traditional methods find difficult to deal with, such as playing video games, autonomous driving, and robot control. There are many classic DRL algorithms, among which Deep Q-Network (DQN) is an important milestone. It overcomes the limitations of traditional Q-learning in high-dimensional state space by introducing deep neural networks to approximate the Q-value function. Another important algorithm is the Policy Gradient method, which directly optimizes the policy function and optimizes the decision by calculating the gradient of the policy. A relatively new algorithm is Proximal Policy Optimization (PPO), which maintains the stability and effectiveness of the algorithm by limiting the range of policy updates. These algorithms have their own advantages and disadvantages in solving practical problems and can cope with complex decision-making challenges in different environments. However, these algorithms still have some shortcomings that need to be improved.

[0009] (1) On the one hand, in complex environments, DRL algorithms generally rely on a fixed reward function. When they are applied to agents with different preferences or characteristics, the training effect may be very poor due to the weak generalization ability of the algorithm.

[0010] (2) On the other hand, traditional DRL has difficulty striking a balance between exploration (discovering new strategies) and exploitation (leveraging known features and states). This limitation often prevents the agent from effectively using prior preferences to explore unknown states.

[0011] The technical problems existing in the industrial application of existing technologies are mainly reflected in the following aspects:

[0012] 1. Weak generalization ability:

[0013] Classic deep reinforcement learning (DRL) algorithms often rely on a fixed reward function, which results in insufficient generalization ability of the algorithm when faced with intelligent agents with different preferences or characteristics, and the training effect may be greatly reduced.

[0014] 2. Exploration and Exploitation Imbalance:

[0015] Traditional DRL algorithms have difficulty striking a balance between exploring new strategies and exploiting known features and states. This imbalance prevents the agent from effectively using prior preferences to explore unknown states, thus affecting the overall performance and efficiency of the algorithm.

[0016] 3. Unreasonable resource allocation:

[0017] In the air-ground computing scenario, especially when resources such as bandwidth are very scarce, existing literature and technology do not fully consider how to allocate resources more reasonably. This may lead to insufficient resource utilization, thus affecting the overall performance and stability of the system.

[0018] 4. UAV path planning problem:

[0019] In the air-ground integration scenario, the path planning of drones is a key issue. Existing technologies may not fully consider the perception and coverage of drones, that is, how to maximize their perception and coverage capabilities of the environment while ensuring that drones can effectively perform their tasks.

[0020] 5. Insufficient algorithm adaptability:

[0021] Drones and other IoT devices in air-ground integrated scenarios are often highly dynamic, which requires algorithms to be more adaptable. However, most classic DRL algorithms do not have this feature and are difficult to adapt to complex and diverse demand changes.

[0022] In summary, the main technical problems faced by existing technologies in industrial applications are weak generalization ability, imbalance between exploration and utilization, unreasonable resource allocation, UAV path planning problems, and insufficient algorithm adaptability. These problems limit the effectiveness and scope of existing technologies in practical applications, and new mechanisms and methods need to be introduced for improvement and optimization. Summary of the invention

[0023] In response to the problems existing in the prior art, the present invention provides an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system.

[0024] The present invention is implemented as follows: an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system includes:

[0025] S101, initializing the agent global network parameters;

[0026] Initialize the agent's global network θ, number of rounds M, maximum number of training steps T, number of optimization iterations per step I, number of alternative strategies J, number of optimal candidate strategies k, initialize the learning rate α of the global network, discount factor γ, initialize the replay buffer size D, and initialize the strategy distribution Initialize the transition probability distribution Initialize hyperparameters, such as carrier frequency f c , time slot length Δt, etc.

[0027] S102, at the beginning of each round, the initial state s t is set, and the strategy distribution is randomly sampled to obtain J alternative strategies;

[0028] S103, randomly sampling J actions from these strategies respectively, and then obtaining J conditional transition probability distributions based on the J alternative strategies, and calculating the corresponding current rewards from the corresponding J actions;

[0029] S104. According to active reasoning and free energy principle, the free energy of each alternative strategy is obtained using cumulative rewards and conditional transition probability distribution;

[0030] S105, averaging the first k smallest free energies, and obtaining the current strategy distribution based on the average value;

[0031] S106, sampling a strategy based on the obtained strategy distribution, and then sampling an action as the current agent's behavior, and interacting with the environment to obtain the next state;

[0032] S107, storing the experience tuple in the replay buffer; if the replay buffer is full, deleting the oldest experience to store the latest experience;

[0033] S108, the predicted value of the global network output free energy is calculated, the error function between the predicted value and the actual free energy is calculated, and the global network parameters are updated using the back propagation algorithm and gradient descent;

[0034] S109. Repeat the training until the algorithm converges and finally obtains the strategy distribution, so that the strategy of each step can be randomly extracted from the distribution to control the action of the intelligent agent and obtain the optimal joint UAV trajectory planning and resource unloading allocation.

[0035] Further, in S102, at the beginning of each round, the initial state is set; J candidate strategies are randomly sampled from the strategy distribution, and then J actions are randomly sampled from each of the J strategies; the state of the agent is represented as:

[0036] S(t)={λ(t),W(t),u(t-1),d(t),h Rice (t),h(t),f(t),fVSP (t),f E (t),R V-E (t),R UE (t)},

[0037] Where λ(t)={λ m (t)}, λ m (t) is the sensing rate of drone m in time slot t; W(t) = {W m (t)}, W m (t) is the bandwidth resource occupied by UAV m in time slot t; u(t-1) = {u m (t-1)}, where u m (t-1) = {X m (t-1),Y m (t-1),H m (t-1)} is the coordinate of drone m at the end of time slot t-1; d(t) = {d m (t)}, d m (t) is the distance between UAV m and the ground base station at time slot t; is the Rice value of drone m at time slot t; h(t) = {h m (t)}, h m (t) is the information gain between UAV m and the ground base station at time slot t; f(t) = {f m (t)}, f m (t) is the available computing resources of UAV m in time slot t; f VSP (t) is the computing resources available to the virtual service operator in time slot t, f E (t) is the available computing resources of the edge service platform in time slot t, R V-E (t) is the data transmission rate from the virtual service operator to the edge service platform in time slot t, R UE (t) is the data rate at which the operator transmits processed data back to the Metaverse user device in time slot t; self-strategy π t The actions generated by sampling are expressed as:

[0038] a(t)={t sens (t),Δ(t),t fly (t),κ(t),ι(t)},

[0039] in, represents the duration of drone perception, Δ(t) = {Δ m (t)}={ΔXm (t),ΔY m (t),ΔH m (t)}, represents the trajectory of the drone, represents the flight time of the UAV, κ(t)={κ m (t)}, is the data offloading ratio of the drone, and ι(t) is the data offloading ratio of the virtual service operator.

[0040] Further, in S103: J conditional transition probability distributions are obtained based on the J candidate strategies, and the corresponding current rewards are calculated from the corresponding J actions. The calculation formula of the immediate reward is as follows:

[0041]

[0042] In the above formula It represents the inverse of the weighted sum of the energy consumption, delay and virtual service operator cost of all devices in the system at time slot t. In other words, the denominator is the required objective function.

[0043] Further, the S104: according to active reasoning and free energy principle, using cumulative rewards And the conditional transition probability distribution The free energy of J alternative strategies is obtained, where the calculation formula for the opposite number of free energy is:

[0044]

[0045] S105: averaging the first k smallest free energies, and obtaining the current strategy distribution based on the average value, this process is equivalent to taking the first k maximum values ​​of the opposite number of the free energy, sorting the values ​​from large to small, and obtaining the average value:

[0046]

[0047] And get the policy distribution:

[0048]

[0049] Here, σ(·) represents a continuous distribution related to the exponential of the natural number e, such as the exponential distribution and the gamma distribution;

[0050] S106: Based on the obtained strategy distribution Sampling strategy π t , and then sample action a t As the current agent's behavior, it interacts with the environment to get the next state; the extraction process is summarized as follows: t ~q(π),a t ~πt .

[0051] Further, S108: the predicted value Q(s) of the global network output free energy t ,a t ; θ), and the target (i.e., actual free energy) The error function L(θ) is used to update the global network parameters using the back propagation algorithm and gradient descent; the loss function is given by:

[0052]

[0053] The gradient of the loss function is given by:

[0054]

[0055] Then update the parameters of the global network through gradient descent as follows:

[0056] θ←θ-α·▽ θ L(θ),

[0057] Among them, θ is the internal parameter of the global model network, and α is the learning rate.

[0058] Another object of the present invention is to provide an optimization system based on active reasoning in a MEC-enabled low-altitude metaverse system, including:

[0059] System initialization module, which is used to initialize the parameters of the deep deterministic policy gradient algorithm. It includes a configuration module for setting the network learning rate, discount factor, and playback buffer size, and a network construction module for defining network layout parameters such as the number of drones and the location of ground base stations;

[0060] The agent module is used to generate actions based on the current network state at the beginning of each cycle. The agent module uses the active reasoning mechanism and the free energy principle to fit the distribution of strategies by selecting the mean of the largest top k free energy minima;

[0061] Action execution module, which is used to execute the path planning and task offloading strategies of UAV;

[0062] A reward acquisition module for executing actions and calculating instant rewards, which calculates rewards based on the inverse of the weighted average of latency, energy consumption, and operator costs of all devices in the system, and a state transition module for transferring the system state from the current state to the next state;

[0063] An experience replay module is used to store each system state, action, reward, cumulative reward and experience tuple of the next state;

[0064] Sampling module for sampling from the distribution of strategies Modules sampled from and modules sampled from strategy π;

[0065] A global network update module, a global network update module for using a back-propagation algorithm, the module comprising a parameter optimization unit for adjusting network parameters using a gradient descent method.

[0066] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the optimization method based on active reasoning in the MEC and drone-enabled metaverse system.

[0067] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the optimization method based on active reasoning in the MEC and drone-enabled metaverse system.

[0068] Another object of the present invention is to provide an information data processing terminal, which is used to implement an optimization system based on active reasoning in the metaverse system enabled by MEC and drones.

[0069] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0070] First, the present invention significantly improves the overall performance and flexibility of the system. In this innovative architecture, drones not only act as mobile computing and communication nodes, but also make up for the lack of coverage of ground IoT devices, especially in remote or difficult-to-access areas. Drones can provide instant computing and communication support in these areas, and work together with ground IoT devices to achieve data collection, preliminary processing and rapid transmission. This air-ground integration solution expands the coverage of the network, so that even in places with limited network coverage, it can efficiently meet various computing needs, optimize data processing efficiency, and improve the overall performance of the system. The dynamic flexibility of drones enables them to adjust the deployment of computing resources according to real-time needs, achieve efficient data forwarding and task processing, reduce data transmission delays, and effectively alleviate bandwidth bottlenecks. At the same time, MEC technology brings computing resources closer to the data source, effectively reducing the time for data transmission to the remote cloud, thereby significantly improving data processing speed and system responsiveness. This close-to-source computing method not only improves the system's real-time processing capability, but also enhances its adaptability to different environmental conditions, allowing the system to maintain a high degree of stability and reliability in dynamically changing network requirements and environments. By combining the mobility of drones with the efficient data processing capabilities of MEC technology, the air-ground integrated network can achieve more efficient data processing, lower latency and better resource utilization, providing more reliable, flexible and efficient solutions for various complex and dynamic application scenarios. This combination not only promotes the technological advancement of intelligent networks, but also opens up new possibilities for future network applications.

[0071] The significant technical progress achieved by the present invention is to realize an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system, which has made significant progress in the following key aspects:

[0072] 1) Efficient task offloading allocation strategy:

[0073] This method significantly improves the resource utilization rate in the air-ground integrated metaverse network by optimizing the resource allocation ratio of unloading from drones to ground base stations and the resource allocation ratio of unloading from base stations to edge service platforms. This optimization measure not only improves the overall performance of the system, but also effectively reduces energy consumption and latency.

[0074] 2) Reasonable UAV path planning strategy:

[0075] This method optimizes by analyzing the three-dimensional coordinates of the drone, allowing the system to automatically adjust the drone's flight path to ensure full perception coverage of the target area and adapt to changing network conditions. This optimization process effectively improves the user experience by adaptively adjusting the drone's performance parameters such as flight time and perception time.

[0076] 3) Integration of reinforcement learning:

[0077] The present invention combines the Deep Reinforcement Learning (DRL) algorithm with the active reasoning method, so that the system does not rely on a single reward in the decision-making process, but makes comprehensive use of additional information from the environment. The system can make optimal decisions in the absence of explicit instructions by autonomously learning and adapting based on real-time data. This adaptive ability plays an extremely important role in dealing with complex and dynamically changing environments.

[0078] 4) Optimization of energy consumption, latency and operator costs:

[0079] The incentive mechanism included in this approach focuses specifically on reducing the system's energy consumption, latency, and operating costs, which is not only energy efficient but also environmentally friendly, which is particularly critical in the context of rising energy costs and increasing attention to environmental protection.

[0080] 5) Improvement of system reliability and adaptability:

[0081] By screening multiple candidate strategies and performing multiple iterations at each step, the predicted free energy value gradually approaches the true value. Therefore, the solution of using a global network to approximate this value enhances the reliability of the system. In addition, by focusing on factors other than system rewards instead of relying on a deterministic reward function, the agent's exploration space is expanded, which indirectly improves the adaptability of the system.

[0082] 6) Network’s autonomous learning and optimization capabilities:

[0083] This method enables the system to continuously optimize its decision-making process through continuous iterative training and experience-based network updates, thereby improving overall performance.

[0084] The combined effect of these technological advances has enabled the MEC-enabled low-altitude metaverse system to achieve significant performance improvements, while also improving energy efficiency, stability, and adaptability. These improvements are crucial to meeting the complex computing needs of the big data era.

[0085] The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system provided by the present invention is centered on using mathematical models to guide the behavior and learning process of the system. The technical effects brought by these mathematical models can be discussed based on their characteristics:

[0086] 1) Calculation of free energy

[0087] The calculation of free energy not only focuses on system energy consumption, latency, and operator costs, but also includes a factor about environmental information, called information gain, which pays additional attention to the agent's preferences and increases the agent's subjective initiative.

[0088] Energy efficiency optimization and cost savings: By directly linking rewards to the weighted sum of system energy consumption, latency, and operator costs, this approach encourages the agent to explore environments where the weighted sum tends to be minimized, thereby achieving a win-win situation of improved user service quality and operator cost savings.

[0089] 2) Global network loss function and update

[0090] By approximating the calculated free energy approximation, the global network is updated using back-propagation and gradient descent methods.

[0091] Policy optimization: By continuously adjusting global network parameters, the system is able to learn and adopt more effective decision-making policy distributions to sample specific policies.

[0092] Learning stability: Using the goals inferred from the active inference mechanism as a basis can balance the learning process and avoid instability caused by large prediction errors.

[0093] Performance optimization: By accurately calculating the loss function and updating the network, the accuracy and efficiency of system decision-making are improved.

[0094] 3) Parameter update formula

[0095] A parameter update method for the global network is described.

[0096] Policy gradually approaches: By gradually updating the global model network parameters, the system can smoothly transition to the new policy and prevent performance fluctuations due to drastic changes.

[0097] Continuous learning and adaptation: This continuous parameter update mechanism ensures that the system can adapt to long-term environmental changes.

[0098] The application of the mathematical model provided by the present invention not only improves the operating efficiency and decision-making quality of the low-altitude metaverse system enabled by MEC, but also enhances its adaptability to environmental changes and long-term stability. These technical effects are crucial for modern edge computing environments that process large amounts of data and high-frequency interactions.

[0099] The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system provided by the present invention optimizes the performance of the network through the interaction between the intelligent agent and the environment.

[0100] Initialize the state of the agent, which includes multiple variables, such as the drone’s perception rate, the bandwidth occupied by the drone, the time-varying channel gain, the distance between the ground base station and the drone, the available computing resources of the drone, the ground base station, and the edge computing platform, the drone’s three-dimensional coordinates, the transmission power, etc. These variables together define the agent’s environmental state at a specific moment, which in turn affects the agent’s decision.

[0101] The agent initially simulates executing actions and obtains immediate rewards, simulates generating actions, and obtains rewards, and then obtains the optimal action through a series of tedious screening. Only then does the agent perform real actions and transfer states. The reward calculation formula takes into account the long-term energy consumption, long-term delays, and operator costs of all devices in the system. This is the comprehensive goal of system design, which aims to reduce operator costs while improving user experience.

[0102] Calculate the loss function and update the network, and use the gradient descent method to update the current global network, which can help improve the global network's prediction accuracy for free energy, thereby optimizing system performance. This part of the operation can be analogous to the value function update in deep reinforcement learning. The key is to improve the prediction accuracy to guide strategy improvement.

[0103] These steps and the application of mathematical models have led to significant technological advances:

[0104] Policy optimization: Through deep reinforcement learning, the system can self-learn and optimize strategies to adapt to the ever-changing network environment.

[0105] Efficient resource utilization: Optimization of task offloading allocation ensures that all computing devices are efficiently utilized, especially drones carrying multiple functions.

[0106] Minimize energy consumption: By optimizing long-term energy consumption, it helps achieve green communications and reduce the impact on the environment.

[0107] System stability and adaptability: It adopts a unique active reasoning mechanism to pay attention to additional information other than rewards, which increases the agent's attention to its own preferences, thereby enhancing stability and universality.

[0108] These advances demonstrate the potential of deep reinforcement learning based on active inference in complex network systems, especially in realizing intelligent and efficient integrated air-ground communication networks.

[0109] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0110] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0111] The present invention deeply considers the operating costs of service providers in the Metaverse network, effectively reduces energy consumption through innovative technical means, and ensures superior performance in meeting users' strict requirements for latency. Specifically, the present invention adopts a series of optimization strategies, which not only improves the efficiency of the system at the technical implementation level, but also minimizes the cost of operators from an economic perspective. This comprehensive improvement plan provides a valuable reference for subsequent patent production and actual business applications, indicating that while improving user experience, operators can achieve higher economic benefits.

[0112] (2) Whether the technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:

[0113] Most of the classic DRL algorithms are inseparable from the design paradigm of several fixed reward functions, which makes these algorithms generally applicable only to specific scenarios. Once the demand or scenario changes, the training results will be far from the expectations, that is, the generalization performance is poor. Of course, many studies have also introduced and integrated some theories in DRL, such as Lyapunov optimization theory and attention mechanism, but they have never been able to get rid of the curse of weak generalization ability. The present invention combines the active reasoning mechanism in brain neuroscience with DRL, and guides the algorithm through free energy. Combined with the research of many scholars, the free energy is finally converted into the form of the sum of cumulative rewards and an indefinite term. The rewards in traditional DRL are not abandoned, and the preferences or characteristics of different intelligent agents (abstracted from specific needs) are reflected in the indefinite term, so that the generalization performance of the algorithm is significantly improved.

[0114] Third, the present invention proposes an innovative solution to several key problems existing in the prior art in industrial applications and achieves significant technological progress.

[0115] First, in order to solve the problem of weak generalization ability of DRL algorithm in the prior art, the present invention enhances the adaptability of the algorithm in different environments and preferences by introducing active reasoning and free energy principle. This enables the algorithm to generalize better and improve the training effect when facing agents with different characteristics.

[0116] Secondly, the present invention solves the problem of imbalance between exploration and utilization in traditional DRL algorithms. By optimizing the free energy of alternative strategies and selecting actions based on free energy, the algorithm can find a better balance between exploring new strategies and utilizing known states, improving overall performance and efficiency.

[0117] In addition, the present invention also considers the problem of unreasonable resource allocation in air-ground basic computing. By introducing strategy distribution and conditional transition probability distribution, the algorithm can more reasonably allocate resources of drones and other IoT devices, improving resource utilization and system stability.

[0118] Finally, the present invention has achieved significant technical progress in industrial applications. By optimizing the trajectory planning and resource unloading allocation of drones, the algorithm reduces system energy consumption, latency and cost, and improves overall performance. This provides strong technical support for the development of MEC and drone-enabled metaverse systems, and promotes the progress and development of related industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 It is a flow chart of an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system provided in an embodiment of the present invention.

[0120] Figure 2 It is an applicable scenario diagram provided by an embodiment of the present invention.

[0121] Figure 3 It is a flow chart for implementing an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system provided in an embodiment of the present invention.

[0122] Figure 4 It is a relationship diagram between the accumulated system rewards and operating costs after the algorithm convergence and the edge service pricing standard, provided by an embodiment of the present invention.

[0123] Figure 5 It is a comparison diagram of the convergence performance of the embodiment of the present invention and several baseline algorithms.

[0124] Figure 6 It is a structural block diagram of an optimization system based on active reasoning in a MEC-enabled low-altitude metaverse system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0125] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0126] The technical solution provided by the present invention proposes an optimization method based on active reasoning in a low-altitude metaverse system enabled by multi-access edge computing (MEC), which mainly adopts a framework of deep reinforcement learning based on active reasoning. The solution mainly includes:

[0127] 1) Deep reinforcement learning framework: A deep neural network is used to represent the global model, and the deep reinforcement learning (DRL) method based on active inference is used for learning and decision-making.

[0128] 2) Network initialization: Initialize the parameters of the global model network and set training-related hyperparameters.

[0129] 3) Environmental interaction and state management: The agent conducts virtual interactions with the environment based on several current alternative strategies, selects the one with the smallest free energy and averages it to obtain the actual action to be executed, and then performs state transfer.

[0130] 4) Experience replay: A replay buffer is used to store experience tuples, and the tuples no longer contain immediate rewards but accumulated rewards. When the buffer overflows, it is updated to maintain the latest learning experience.

[0131] 5) Network parameter update: Update the parameters of the full model network by calculating the loss function and gradient descent.

[0132] 7) Strategy convergence and implementation: Repeat the training until the strategy converges, and use the trained strategy for UAV path planning and task offloading allocation.

[0133] The following are two specific embodiments and their implementation schemes provided by the present invention:

[0134] Example 1: Real-time urban traffic condition monitoring

[0135] Use drones to monitor traffic facilities in various cities, collect data and process it.

[0136] 1) Network parameter initialization: set the number of drones, determine the coordinates of the nearest base station and other network parameters.

[0137] 2) Strategy and environment interaction: The computer or intelligent device generates actions based on alternative strategies to simulate the monitoring task, selects several minimum free energy owners and averages them to obtain the real action, and the drone executes the action to complete the displacement and monitoring tasks.

[0138] 3) Data collection and processing: Collect environmental data and transmit it to the monitoring terminal.

[0139] 4) Task and resource allocation: Optimize resource allocation and task offloading allocation based on DRL strategy.

[0140] 5) Network update and optimization: Update strategies based on monitoring results and resource utilization.

[0141] Example 2: Real-life navigation

[0142] When traveling, you need to know the road conditions and tourist routes in natural scenic spots, and deploy drones to help avoid peak congestion and conduct reconnaissance and analysis of complex road sections.

[0143] 1) Network parameter initialization: Set the drone deployment plan for the scenic area.

[0144] 2) Mission execution: The drone provides real-time feedback on the road conditions of scenic spots and internal routes required by users based on the strategy.

[0145] 3) Data transmission and processing: The collected information is transmitted to the base station and MEC for processing to obtain services such as route recommendations.

[0146] 4) Dynamic resource allocation: The DRL strategy dynamically adjusts resources to optimize transmission efficiency.

[0147] 5) Strategy iteration and update: Adjust strategies according to user needs and improve feedback speed.

[0148] Example 3

[0149] In view of the problems existing in the prior art, this embodiment provides an optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system. Figure 2 This is a scenario diagram to which the method of the present invention can be applied. The air-ground integrated network (AGIN) supported by MEC is composed of two parts: the air base and the ground base. In this scenario, the virtual service operator deploys a group of drones to Provide corresponding virtual services to users within its own radiation area. Drones in the air-based layer can sense the physical environment and collect its attributes or status. The ground-based network is mainly composed of ground base stations, edge computing platforms, and metaverse user devices. The ground base station is owned by the virtual operator and has powerful computing power and stable energy supply. The edge computing platform performs services within the radiation range of the ground base station, and the operator can provide certain economic incentives to the edge computing platform in order to obtain additional computing power resources. Drones can transmit their sensor data to the ground base station through wireless communication links. After receiving data from the drone, in order to reduce its own processing pressure, the ground base station can also offload part of the data to the edge computing platform in order to seek greater computing power and lower latency. Symbol Represents a collection of time slots.

[0150] In the relevant scenario, at the beginning of each time slot, users everywhere make virtual service requests to the operator, and the request is forwarded to the drone through the base station. In order to complete the perception task and cover a large area, the drone needs to perform action a = {t sens ,Δ,t fly ,κ}. Among them, it includes the perception time t of the drone sens , flight time t fly, as well as the displacement Δ in the three-dimensional coordinates and the task offloading ratio κ to the ground base station. Finally, all the sensed data are divided into three parts, which are processed by the UAV, the ground base station and the edge computing platform respectively. Note that these processes are parallel and simultaneous. In addition, in this paper, the communication between the UAV and the edge computing platform is blocked, and the task can only be offloaded to the ground base station. At the same time, the edge computing platform only provides direct services to the ground base station.

[0151] Secondly, the present invention provides a low-altitude metaverse system enabled by MEC. The system includes an air-based layer and a ground-based layer. The ground-based layer is composed of an edge computing platform, a ground base station, and a metaverse user device. The air-based layer is networked by a drone cluster responsible for sensing the ground environment. In each fixed time slot, the metaverse user device generates a task, which must be processed jointly by the drone, the ground base station, and the edge computing platform within the limited time slot. By reasonably unloading each task during execution and planning the routes of each drone in the air-based layer, the entire system can minimize energy consumption and latency while saving operator costs.

[0152] Example 4

[0153] Considering the dynamic characteristics of the network and the uncertainty of information acquisition, the problem becomes quite difficult. In order to solve it effectively, the problem is reconstructed as a Markov decision process (MDP). Since the preferences of the agents are different, an improved reinforcement learning algorithm based on active inference is used, which supports both continuous action space and discrete action space, and can support real-time online decision-making.

[0154] like Figure 1 As shown, the optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system provided in this embodiment is characterized by comprising the following steps:

[0155] S101, initializing the agent global network parameters;

[0156] Initialize the agent's global network θ, number of rounds M, maximum number of training steps T, number of optimization iterations per step I, number of alternative strategies J, number of optimal candidate strategies k, initialize the learning rate α of the global network, discount factor γ, initialize the replay buffer size D, and initialize the strategy distribution Initialize the transition probability distribution Initialize hyperparameters, such as carrier frequency f c , time slot length Δt, etc.

[0157] S102, at the beginning of each round, the initial state s t is set, the strategy distribution is randomly sampled to obtain J candidate strategies, and then these strategies are randomly sampled to obtain J actions;

[0158] S103, randomly sampling J actions from these strategies respectively, and then obtaining J conditional transition probability distributions based on the J alternative strategies, and calculating the corresponding current rewards from the corresponding J actions;

[0159] S104. According to active reasoning and free energy principle, the free energy of each alternative strategy is obtained using cumulative rewards and conditional transition probability distribution;

[0160] S105, averaging the first k smallest free energies, and obtaining the current strategy distribution based on the average value;

[0161] S106, sampling a strategy based on the obtained strategy distribution, and then sampling an action as the current agent's behavior, and interacting with the environment to obtain the next state;

[0162] S107, storing the experience tuple in the replay buffer. If the replay buffer is full, deleting the oldest experience and storing the latest experience;

[0163] S108, the predicted value of the global network output free energy is calculated, the error function between the predicted value and the actual free energy is calculated, and the global network parameters are updated using the back propagation algorithm and gradient descent;

[0164] S109. Repeat the training until the algorithm converges and finally obtains the strategy distribution, so that the strategy of each step can be randomly extracted from the distribution to control the action of the intelligent agent and obtain the optimal joint UAV trajectory planning and resource unloading allocation.

[0165] In step S102: at the beginning of each round, the initial state is set; the strategy distribution is randomly sampled to obtain J candidate strategies, and then the J strategies are randomly sampled to obtain J actions. The state of the agent is represented as:

[0166] S(t)={λ(t),W(t),u(t-1),d(t),h Rice (t),h(t),f(t),f VSP (t),f E (t),R V-E (t),R UE (t)},

[0167] Where λ(t)={λ m (t)}, λ m (t) is the sensing rate of drone m in time slot t; W(t) = {W m (t)}, W m (t) is the bandwidth resource occupied by UAV m in time slot t; u(t-1) = {u m(t-1)}, where u m (t-1) = {X m (t-1),Y m (t-1),H m (t-1)} is the coordinate of drone m at the end of time slot t-1; d(t) = {d m (t)}, d m (t) is the distance between UAV m and the ground base station at time slot t; is the Rice value of drone m at time slot t; h(t) = {h m (t)}, h m (t) is the information gain between UAV m and the ground base station at time slot t; f(t) = {f m (t)}, f m (t) is the available computing resources of UAV m in time slot t; f VSP (t) is the computing resources available to the virtual service operator in time slot t, f E (t) is the available computing resources of the edge service platform in time slot t, R V-E (t) is the data transmission rate from the virtual service operator to the edge service platform in time slot t, R UE (t) is the data rate at which the operator transmits processed data back to the Metaverse user device in time slot t. t The sampled actions are represented as:

[0168] a(t)={t sens (t),Δ(t),t fly (t),κ(t),ι(t)},

[0169] in, represents the duration of drone perception, Δ(t) = {Δ m (t)}={ΔX m (t),ΔY m (t),ΔH m (t)}, represents the trajectory of the drone, represents the flight time of the UAV, κ(t)={κ m (t)}, is the data offloading ratio of the drone, and ι(t) is the data offloading ratio of the virtual service operator.

[0170] In step S103: J conditional transition probability distributions are obtained based on the J candidate strategies, and the corresponding current rewards are calculated from the corresponding J actions. The calculation formula of the immediate reward is as follows:

[0171]

[0172] In the above formula It represents the inverse of the weighted sum of energy consumption, delay and virtual service operator cost of all devices in the system at time slot t, that is, the denominator is our objective function.

[0173] In step S104: Based on active inference and free energy principle, using cumulative rewards And the conditional transition probability distribution The free energy of J alternative strategies is obtained, where the calculation formula for the opposite number of free energy is:

[0174]

[0175] Furthermore, the S105: averaging the first k smallest free energies and obtaining the current strategy distribution based on the average value is equivalent to taking the first k maximum values ​​of the opposite number of the free energy, sorting the values ​​from large to small and obtaining the average value:

[0176]

[0177] And get the policy distribution:

[0178]

[0179] Here, σ(·) represents a continuous distribution related to the exponential of the natural number e, such as the exponential distribution and the gamma distribution.

[0180] In step S106: based on the obtained strategy distribution Sampling strategy π t , and then sample action a t As the current agent's behavior, it interacts with the environment to get the next state; the extraction process is summarized as follows:

[0181]

[0182] In step S108: the global network outputs the predicted value of free energy Q(s t ,a t ; θ), and the target (i.e., actual free energy) The error function L(θ) is used to update the global network parameters using the back propagation algorithm and gradient descent; the loss function is given by:

[0183]

[0184] The gradient of the loss function is given by:

[0185]

[0186] Then update the parameters of the global network through gradient descent as follows:

[0187] θ←θ-α·▽ θ L(θ).

[0188] Among them, θ is the internal parameter of the global model network, and α is the learning rate.

[0189] In order to elaborate on the optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system, the present invention provides two specific application embodiments, including key details of the implementation scheme.

[0190] In order to prove the creativity and technical value of the technical solution of the present invention, this section provides application examples of the claimed technical solution on specific products or related technologies.

[0191] Application Example 1: Real-time Urban Traffic Condition Monitoring

[0192] 1) Network parameter initialization: Deploy a global model network in the monitoring department of the city's traffic management center. Initialize network parameters, including the critic network's learning rate α, the initial playback buffer size D, the time slot length Δt and other hyperparameters; deploy multiple drones to monitor traffic facilities and conditions, and deploy nearby base stations and edge service platforms for auxiliary task processing.

[0193] 2) Strategy and environment interaction: The drone performs monitoring tasks and displacements through actions generated based on active reasoning mechanisms. Multithreaded parallelism is beneficial to the computational efficiency of active reasoning, so the equipment in this department needs to support parallel computing and have super computing power.

[0194] Data collection and processing: Collect urban traffic data and transmit it to nearby base stations.

[0195] 4) Task offloading allocation: Optimize data processing and task offloading allocation based on DRL strategy. Use reinforcement learning algorithm to gradually improve task allocation.

[0196] 5) Network update and optimization: Update strategies based on monitoring results and resource utilization, so that the monitoring system can always maintain efficient data processing and transmission.

[0197] Application Example 2: Real Scene Navigation

[0198] Network parameter initialization: deploy a global model network in the scenic area and initialize relevant network parameters. Configure multiple drone devices, which can provide real-time aerial observation 3D models, making it easy to intuitively present hot information such as road conditions, complex terrain, and population density in the scenic area.

[0199] Mission execution: The drone executes various hot tasks according to the strategy, and executes actions to enter the next state to continue to complete the next round of tasks without interruption.

[0200] Data transmission and processing: The sensed data information is transmitted to the ground base station for processing.

[0201] Dynamic resource allocation: DRL strategy dynamically allocates resources to optimize task execution efficiency.

[0202] Strategy iteration and update: Adjust strategies based on user feedback, improve response speed, and further improve the efficiency and accuracy of various tasks.

[0203] In these two embodiments, the optimization method based on active reasoning in the metaverse system empowering MEC and drones provides an efficient, reliable and energy-saving solution, which is suitable for different application scenarios, from real-time urban traffic condition monitoring to real-scene navigation, demonstrating its wide application potential and technical advantages.

[0204] In order to more clearly demonstrate the positive effects achieved by the embodiments of the present invention during the research process, the advantages of the embodiments of the present invention in research and development compared with the prior art are described below.

[0205] exist Figure 4 In the present invention, the relationship between the average system reward after the algorithm converges and the edge server charging standard is simulated. As the charging standard of the edge service platform increases, the rewards of all algorithms will gradually decrease. This is because when the charges of the edge service platform increase, in order to reduce costs, the tasks will tend to be processed locally by the drone and the base station, and the task processing volume of the edge service platform will be reduced to a certain extent. At the same time, for the entire system, the smaller the delay, the better, so in fact, in order to ensure low delay, the tasks will still be unloaded to the edge computing platform, so the operating cost will not be significantly reduced with the increase in charges. It can be seen that the proposed algorithm is superior to the other three algorithms in increasing the reward function. Compared with the other three algorithms, the proposed algorithm can show higher rewards and lower operating costs. It shows that the embodiments of the present invention have achieved some positive effects in the simulated use process, and it does have great advantages compared with the prior art.

[0206] exist Figure 5In , the convergence performance of all algorithms with reward as the performance metric is shown under the default parameter settings. Our algorithm shows a significantly faster convergence speed, showing initial signs of stability around the 75th round. In addition, it performs well in terms of both reward acquisition and post-convergence volatility. This is due to the different degrees of constraint relaxation in the baseline algorithms, which leads to excessive randomness in the agent's exploration along specific directions. Therefore, the process of finding the optimal solution is prolonged and the volatility of convergence remains large. At the same time, the difficulty of obtaining large immediate rewards increases, resulting in the characteristic of smaller cumulative rewards. Specifically, compared with several baseline algorithms, our algorithm consistently achieves higher performance indicators and exhibits less volatility. It is worth noting that some of the comparison algorithms simply modify the agent without imposing strict decision-making (i.e., action) constraints from different perspectives to cater to unique preferences. In Figure 5 The stable convergence of several algorithms observed in underscores the ability of our algorithm to adapt to agents with different preferences and highlights its superior generalization ability.

[0207] like Figure 6 As shown, another object of the present invention is to provide an optimization system based on active reasoning in a MEC-enabled low-altitude metaverse system, including:

[0208] System initialization module, which is used to initialize the parameters of the deep deterministic policy gradient algorithm. It includes a configuration module for setting the network learning rate, discount factor, and playback buffer size, and a network construction module for defining network layout parameters such as the number of drones and the location of ground base stations;

[0209] The agent module is used to generate actions based on the current network state at the beginning of each cycle. The agent module uses the active reasoning mechanism and the free energy principle to fit the distribution of strategies by selecting the mean of the largest previous free energy minimum;

[0210] Action execution module, which is used to execute the path planning and task offloading strategies of UAV;

[0211] A reward acquisition module for executing actions and calculating instant rewards, which calculates rewards based on the inverse of the weighted average of latency, energy consumption, and operator costs of all devices in the system, and a state transition module for transferring the system state from the current state to the next state;

[0212] An experience replay module is used to store each system state, action, reward, cumulative reward and experience tuple of the next state;

[0213] Sampling modules, modules for sampling from the distribution of policies and modules for sampling from policies;

[0214] A global network update module, a global network update module for using a back-propagation algorithm, the module comprising a parameter optimization unit for adjusting network parameters using a gradient descent method.

[0215] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the active reasoning-based optimization method in the MEC-enabled low-altitude metaverse system.

[0216] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the active reasoning-based optimization method in the MEC-enabled low-altitude metaverse system.

[0217] Another object of the present invention is to provide an information data processing terminal, which is used to implement an optimization system based on active reasoning in the MEC-enabled low-altitude metaverse system.

[0218] Application Example 1: Real-time Urban Traffic Condition Monitoring

[0219] 1) Network parameter initialization: Deploy a global model network in the monitoring department of the city's traffic management center. Initialize network parameters, including the critic network's learning rate, initial playback buffer size, time slot length and other hyperparameters; deploy multiple drones to monitor traffic facilities and conditions, and deploy nearby base stations and edge service platforms for auxiliary task processing.

[0220] 2) Strategy and environment interaction: The drone performs monitoring tasks and displacements through actions generated based on active reasoning mechanisms. Multithreaded parallelism is beneficial to the computational efficiency of active reasoning, so the equipment in this department needs to support parallel computing and have super computing power.

[0221] Data collection and processing: Collect urban traffic data and transmit it to nearby base stations.

[0222] 4) Task offloading allocation: Optimize data processing and task offloading allocation based on DRL strategy. Use reinforcement learning algorithm to gradually improve task allocation.

[0223] 5) Network update and optimization: Update strategies based on monitoring results and resource utilization, so that the monitoring system can always maintain efficient data processing and transmission.

[0224] Application Example 2: Real Scene Navigation

[0225] Network parameter initialization: deploy a global model network in the scenic area and initialize relevant network parameters. Configure multiple drone devices, which can provide real-time aerial observation 3D models, making it easy to intuitively present hot information such as road conditions, complex terrain, and population density in the scenic area.

[0226] Mission execution: The drone executes various hot tasks according to the strategy, and executes actions to enter the next state to continue to complete the next round of tasks without interruption.

[0227] Data transmission and processing: The sensed data information is transmitted to the ground base station for processing.

[0228] Dynamic resource allocation: DRL strategy dynamically allocates resources to optimize task execution efficiency.

[0229] Strategy iteration and update: Adjust strategies based on user feedback, improve response speed, and further improve the efficiency and accuracy of various tasks.

[0230] In these two embodiments, the active inference-based optimization method in the MEC-enabled low-altitude metaverse system provides an efficient, reliable and energy-saving solution, which is suitable for different application scenarios, from real-time urban traffic condition monitoring to real-scene navigation, demonstrating its wide application potential and technical advantages.

[0231] exist Figure 4 In the present invention, the relationship between the average system reward after the algorithm converges and the edge server charging standard is simulated. As the charging standard of the edge service platform increases, the rewards of all algorithms will gradually decrease. This is because when the charges of the edge service platform increase, in order to reduce costs, the tasks will tend to be processed locally by the drone and the base station, and the task processing volume of the edge service platform will be reduced to a certain extent. At the same time, for the entire system, the smaller the delay, the better, so in fact, in order to ensure low delay, the tasks will still be unloaded to the edge computing platform, so the operating cost will not be significantly reduced with the increase in charges. It can be seen that the proposed algorithm is superior to the other three algorithms in increasing the reward function. Compared with the other three algorithms, the proposed algorithm can show higher rewards and lower operating costs. It shows that the embodiments of the present invention have achieved some positive effects in the simulated use process, and it does have great advantages compared with the prior art.

[0232] exist Figure 5In , the convergence performance of all algorithms with reward as the performance metric is shown under the default parameter settings. Our algorithm shows a significantly faster convergence speed, showing initial signs of stability around the 75th round. In addition, it performs well in terms of both reward acquisition and post-convergence volatility. This is due to the different degrees of constraint relaxation in the baseline algorithms, which leads to excessive randomness in the agent's exploration along specific directions. Therefore, the process of finding the optimal solution is prolonged and the volatility of convergence remains large. At the same time, the difficulty of obtaining large immediate rewards increases, resulting in the characteristic of smaller cumulative rewards. Specifically, compared with several baseline algorithms, our algorithm consistently achieves higher performance indicators and exhibits less volatility. It is worth noting that some of the comparison algorithms simply modify the agent without imposing strict decision-making (i.e., action) constraints from different perspectives to cater to unique preferences. In Figure 5 The stable convergence of several algorithms observed in underscores the ability of our algorithm to adapt to agents with different preferences and highlights its superior generalization ability.

[0233] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0234] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. An optimization method based on active reasoning in a MEC-enabled low-altitude metaverse system, characterized in that: By initializing the global network parameters and hyperparameters of the agent, using the policy distribution and transition probability distribution for random sampling, calculating the cumulative reward based on active reasoning and the free energy principle, obtaining the free energy and averaging it, generating a new policy distribution, and updating the global network parameters with the back-propagation algorithm and gradient descent, and through repeated training until the algorithm converges, the optimal joint UAV trajectory planning and resource unloading allocation of the agent in the metaverse environment are finally achieved.

2. The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system as claimed in claim 1, characterized in that: The following steps are involved: S101, initializing agent global network parameters; Initialize the agent's global network θ, number of rounds M, maximum number of training steps T, number of optimization iterations for each step I, number of alternative strategies J, number of optimal candidate strategies k, initialize the learning rate α of the global network, discount factor γ, initialize the replay buffer size D, initialize the strategy distribution q(π), and initialize the transition probability distribution Initialize hyperparameters; S102, at the beginning of each round, the initial state s t is set, and J alternative strategies are obtained by random sampling of the strategy distribution; S103, randomly sampling J actions from these strategies respectively, and then obtaining J conditional transition probability distributions based on the J alternative strategies, and calculating the corresponding current rewards from the corresponding J actions; S104. According to active reasoning and free energy principle, the free energy of each alternative strategy is obtained using cumulative rewards and conditional transition probability distribution; S105, averaging the first k smallest free energies, and obtaining the current strategy distribution based on the average value; S106, sampling a strategy based on the obtained strategy distribution, and then sampling an action as the current agent's behavior, and interacting with the environment to obtain the next state; S107, storing the experience tuple in the replay buffer; If the replay buffer is full, the oldest experience is deleted to store the latest experience; S108, the predicted value of the global network output free energy is calculated, the error function between the predicted value and the actual free energy is calculated, and the global network parameters are updated using the back propagation algorithm and gradient descent; S109. Repeat the training until the algorithm converges and finally obtains the strategy distribution, so that the strategy of each step can be randomly extracted from the distribution to control the action of the intelligent agent and obtain the optimal joint UAV trajectory planning and resource unloading allocation.

3. The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system as claimed in claim 2, characterized in that: In S102, at the beginning of each round, the initial state is set; J candidate strategies are randomly sampled from the strategy distribution, and then J actions are randomly sampled from each of the J strategies; the state of the agent is represented as: S(t)={λ(t),W(t),u(t-1),d(t),h Rice (t),h(t),f(t),f VSP (t),f E (t),R V-E (t),R UE (t)}, in, is the sensing rate of UAV m in time slot t; is the bandwidth resource occupied by UAV m in time slot t; where u m (t-1) = {X m (t-1),Y m (t-1),H m (t-1)} is the coordinate of UAV m at the end of time slot t-1; is the distance between UAV m and the ground base station at time slot t; is the Ricean value of UAV m at time slot t; is the information gain between UAV m and the ground base station at time slot t; is the available computing resources of UAV m in time slot t; f VSP (t) is the computing resources available to the virtual service operator in time slot t, f E (t) is the available computing resources of the edge service platform in time slot t, R V-E (t) is the data transmission rate from the virtual service operator to the edge service platform in time slot t, R UE (t) is the data rate at which the operator transmits processed data back to the Metaverse user device in time slot t; self-strategy π t The actions generated by sampling are expressed as: a(t)={t sens (t),Δ(t),t fly (t),κ(t),ι(t)}, in, Indicates the drone’s perception duration, represents the trajectory of the drone, Indicates the flight time of the drone. is the data offloading ratio of the drone, and ι(t) is the data offloading ratio of the virtual service operator.

4. The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system as claimed in claim 2, characterized in that: S103: Based on the J candidate strategies, J conditional transition probability distributions are obtained, and the corresponding current rewards are calculated from the corresponding J actions. The calculation formula of the immediate reward is as follows: In the above formula It represents the inverse of the weighted sum of the energy consumption, delay and virtual service operator cost of all devices in the system at time slot t. In other words, the denominator is the required objective function.

5. The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system as claimed in claim 2, characterized in that: S104: Based on active reasoning and free energy principle, using cumulative rewards And the conditional transition probability distribution The free energy of J alternative strategies is obtained, where the calculation formula for the opposite number of free energy is: S105: averaging the first k smallest free energies, and obtaining the current strategy distribution based on the average value, this process is equivalent to taking the first k maximum values ​​of the opposite number of the free energy, sorting the values ​​from large to small, and obtaining the average value: And get the policy distribution: Here, σ(·) represents a continuous distribution related to the exponential of the natural number e, such as the exponential distribution and the gamma distribution; S106: Based on the obtained strategy distribution Sampling strategy π t , and then sample action a t As the current agent's behavior, it interacts with the environment to get the next state; the extraction process is summarized as follows: p t ~q(π),a t ~π t 。 6. The optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system as claimed in claim 2, characterized in that: S108: predicted value Q(s) of global network output free energy t ,a t ;θ), and the target The error function L(θ) is used to update the global network parameters using the back propagation algorithm and gradient descent; the loss function is given by: The gradient of the loss function is given by: Then update the parameters of the global network through gradient descent as follows: Among them, θ is the internal parameter of the global model network, and α is the learning rate.

7. An optimization system based on active reasoning in a MEC-enabled low-altitude Metaverse system implementing the optimization method based on active reasoning in a MEC-enabled low-altitude Metaverse system as described in any one of claims 1 to 6, characterized in that: The optimization system based on active reasoning in the MEC-enabled low-altitude metaverse system includes: System initialization module, which is used to initialize the parameters of the deep deterministic policy gradient algorithm. It includes a configuration module for setting the network learning rate, discount factor, and playback buffer size, and a network construction module for defining network layout parameters such as the number of drones and the location of ground base stations; The agent module is used to generate actions based on the current network state at the beginning of each cycle. The agent module uses the active reasoning mechanism and the free energy principle to fit the distribution of strategies by selecting the mean of the largest top k free energy minima; Action execution module, which is used to execute the path planning and task offloading strategies of UAV; A reward acquisition module for executing actions and calculating instant rewards, which calculates rewards based on the inverse of the weighted average of latency, energy consumption, and operator costs of all devices in the system, and a state transition module for transferring the system state from the current state to the next state; An experience replay module is used to store each system state, action, reward, cumulative reward and experience tuple of the next state; Sampling module for sampling from the distribution of strategies Modules sampled from and modules sampled from strategy π; A global network update module, a global network update module for using a back-propagation algorithm, the module comprising a parameter optimization unit for adjusting network parameters using a gradient descent method.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the optimization method based on active reasoning in the MEC-enabled low-altitude metaverse system as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the active reasoning-based optimization method in the MEC-enabled low-altitude metaverse system as described in any one of claims 1-6.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the optimization system based on active reasoning in the MEC-enabled low-altitude metaverse system as described in claim 7.

Images