A metaverse task offloading and resource scheduling method based on deep reinforcement learning

Abstract

The present invention discloses a metaverse task offloading and resource scheduling method based on deep reinforcement learning, which relates to the field of wireless communication technology, including: step 1, placing a drone cluster in a user-dense area, and a fixed edge server obtaining the position coordinates of itself, each user and each drone; step 2, the user sends a rendering request to the fixed edge server through the drone cluster; step 3, constructing an optimization target, and converting the optimization target into a Markov decision process, solving the optimization target through the PPO algorithm, and obtaining the horizontal flight rate, flight direction, vertical direction rate and computing resource set available to the drone; step 4, the fixed edge server sends the decision result to the user and the drone, and the user and the drone execute the decision; step 5, after the drone completes the rendering task, the result is transmitted back to the user. The present invention has the characteristics of multi-node collaborative optimization, optimized energy consumption and resource utilization.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and more specifically, to a metaverse task offloading and resource scheduling method based on deep reinforcement learning. Background Art

[0002] With the growing demand for the metaverse and immersive experience, users' demand for low-latency, high-bandwidth computing resources has increased significantly, especially for users wearing head-mounted displays. When interacting in virtual reality or augmented reality environments, real-time rendering and response speed are critical to the user experience. However, the computing power and battery capacity of terminal devices (such as head-mounted displays) are limited, making it difficult to support high-intensity computing tasks. Therefore, the use of edge computing to offload computing tasks to nearby drone computing nodes is considered to solve this problem. In recent years, "drone-assisted networks" in mobile edge computing have emerged, and drones have shown significant advantages in providing edge computing services, expanding network coverage, and improving computing power.

[0003] Drone-assisted mobile edge computing networks not only offer flexible mobility, enabling them to provide computing resources anytime and anywhere, but also collaborate with other edge servers or fixed base stations through self-organizing networks to optimize resource allocation and computation offloading. Furthermore, deep reinforcement learning, as an intelligent decision-making algorithm, enables adaptive task offloading and resource allocation strategy optimization in dynamic environments, effectively addressing dynamic resource changes, user location uncertainty, and network fluctuations.

[0004] Currently, there are some related technical solutions in the fields of metaverse, edge computing, and deep learning, involving resource allocation and offloading strategies in computing:

[0005] Chinese patent document CN114745736A discloses a metaverse service processing method based on base station edge computing nodes. This method deploys edge computing nodes at the base station and processes metaverse service requests through the nodes, reducing computing delays and data transmission delays.

[0006] Chinese patent document CN115204505A discloses an optimal resource allocation method for an on-board edge metaverse system, which provides computing support for AR vehicles through an on-board edge computing (VEC) server. This method uses a deep learning model to optimize computing resources and transmission power, but it focuses on fixed on-board edge server application scenarios.

[0007] Chinese patent document CN117255418A discloses a fog computing resource allocation method based on deep reinforcement learning. Through the proximal policy optimization algorithm of deep reinforcement learning, resource allocation and computation offloading of mobile devices are realized in a fog computing environment. This solution has strong adaptability in dynamic resource allocation and optimizes latency and energy consumption.

[0008] Although the above research has enhanced edge computing, resource allocation, and deep learning algorithms, it still has the following obvious shortcomings when applied to dynamic scenarios in the metaverse:

[0009] 1. Existing edge computing systems usually rely on fixed base stations or ground nodes (such as vehicle-mounted edge servers and fixed edge computing nodes). They lack flexibility and cannot dynamically adjust the deployment of computing resources according to user locations. There is a problem of dynamic allocation of computing resources caused by mobility. This fixed node architecture is difficult to meet the low latency and high real-time requirements of mobile users in the metaverse, especially when users wear head-mounted displays and move frequently.

[0010] 2. Most current technologies allocate resources at a single node, and few solutions involve the collaborative optimization of multiple mobile nodes (such as drones). Even with on-board edge computing networks, they mainly focus on local single-node optimization and fail to effectively achieve resource sharing and intelligent scheduling among multiple nodes, resulting in uneven resource utilization and affecting the overall performance of the system.

[0011] 3. Traditional edge computing offloading strategies and resource allocation methods are difficult to adapt to the complex and changing user mobility, network status and computing needs. Many existing solutions use allocation strategies based on static rules or traditional optimization algorithms, which lack the ability to adapt to highly dynamic environments. Therefore, it is difficult to achieve efficient task offloading and resource allocation.

[0012] 4. Fixed base stations and ground servers usually have high power consumption, while drones have strong mobility but limited battery life. Therefore, continuing to operate in unnecessary scenarios may result in energy waste. Existing technologies do not fully utilize intelligent algorithms to balance computing resources and energy consumption to adapt to the high-intensity and long-term resource demands in the metaverse environment. Summary of the Invention

[0013] The purpose of this invention is to design and develop a metaverse task offloading and resource scheduling method based on deep reinforcement learning. Through multi-node collaboration combined with the PPO algorithm, intelligent task offloading and dynamic resource scheduling are realized. While meeting the high-quality service requirements of the metaverse, it effectively balances resource utilization and energy consumption, extends the flight time of drones, and improves the overall resource efficiency of the system.

[0014] The technical solution provided by the present invention is:

[0015] A metaverse task offloading and resource scheduling method based on deep reinforcement learning includes the following steps:

[0016] Step 1: Deploy a drone cluster in a user-dense area. Based on the Cartesian coordinate system, the fixed edge server obtains the location coordinates of itself, each user, and each drone.

[0017] Step 2: The user sends a rendering request to the fixed edge server through the drone cluster;

[0018] Step 3: Construct an optimization objective and convert it into a Markov decision process. Solve the optimization objective using the PPO algorithm to obtain the UAV's horizontal flight speed, flight direction, vertical speed, and the set of computing resources available to the edge device.

[0019] The optimization goal is:

[0020] ;

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] Where, is the frame delay weight, For users Frame delay, is the weight of energy consumption, For users The system energy consumption, For edge devices Assign to user The proportion of computing resources, For edge devices and users The distribution coefficient between For users or edge devices and user or edge device The distance between For users or edge devices exist The position coordinates at the moment, The boundary of the movable area for users and drones, For user collection, is an edge device set, which includes a drone set and a fixed edge server;

[0026] Step 4: The fixed edge server sends the decision result to the user and drone, and the user and drone execute the decision;

[0027] Step 5: After the drone completes the rendering task, it transmits the result back to the user.

[0028] Preferably, the user's location coordinate update satisfies:

[0029] ;

[0030] Where, For drones exist The position coordinates at the moment, For drones exist The horizontal rate at time, For drones exist The direction of time, , For drones exist The vertical velocity at the moment, For Time has come The length of the time period, For drones exist The position coordinates at the moment.

[0031] Preferably, the user The frame delay satisfies:

[0032] ;

[0033] Where, For rendering delay, For transmission delay.

[0034] Preferably, the rendering delay satisfies:

[0035] ;

[0036] Where, The computing resources required for edge devices to process unit data, is a user Offload to edge devices The frame size, For edge devices Available computing resources.

[0037] Preferably, the transmission delay satisfies:

[0038] ;

[0039] Where, For edge devices and users Communication rate between

[0040] The edge device and users The communication rate between them satisfies:

[0041] ;

[0042] Where, For edge devices bandwidth, For edge devices The power, is the channel gain coefficient, For edge devices and users The distance between is the noise power.

[0043] Preferably, the user The system energy consumption meets the following requirements:

[0044] ;

[0045] Where, To transmit energy, For rendering energy.

[0046] Preferably, the transmission energy satisfies:

[0047] ;

[0048] Where, For edge devices transmission power.

[0049] Preferably, the rendering energy satisfies:

[0050] ;

[0051] Where, is the power factor of the edge device.

[0052] Preferably, converting the optimization objective into a Markov decision process specifically includes:

[0053] State Space : ;

[0054] Action Space : ;

[0055] Reward Function : ;

[0056] in, for The state space at time, for The set of user and drone locations at each moment, The power of the user and the drone is collected. is the user's task data set, A collection of computing resources available to edge devices. for The action space at each moment, for A collection of drone flight actions at every moment, is the set of user task offloading decisions, for The reward function at that moment.

[0057] Preferably, the PPO algorithm specifically includes the following steps:

[0058] Step 1: Initialize the network parameters of the policy network , network parameters of the value network and experience pool ;

[0059] Step 2: Fixed edge server passing current policy Interact with the environment and record trajectory data And stored in the experience pool middle;

[0060] in, For the The policy function of the iteration, Indicates that the parameter is The policy network outputs the policy, is the iteration round, For The state of the environment at any moment;

[0061] Step 3: For each trajectory in the experience pool , calculate the cumulative return of actual observations:

[0062] ;

[0063] Where, is the single sampling time step, for Step discount factor Power, For the moment Instant rewards received;

[0064] Step 4: Use the advantage estimation method to directly predict the state value function based on the value network Compute the advantage function:

[0065] ;

[0066] Where, is an action-value function that satisfies:

[0067] ;

[0068] Where, is the discount factor, is a hyperparameter, , used to control the bias-variance trade-off, For the moment The timing difference error of Indicates the current moment, Indicates the step offset from the current moment;

[0069] Step 5: Use the gradient descent method to adjust the network parameters of the policy network Perform updates with the goal of maximizing the objective function;

[0070] The objective function is:

[0071] ;

[0072] Where, is the objective function, For expectations, is the probability ratio of the current strategy to the old strategy, is the advantage function, is the clipping function, is the cutting range;

[0073] Network parameters of the policy network Updates meet:

[0074] ;

[0075] Where, For the The size of the experience pool at step time;

[0076] Step 6: Update the value function parameters using gradient descent :

[0077] ;

[0078] Where, is the loss function of the value network, For the The updated value network parameters after iterations.

[0079] The beneficial effects of the present invention are:

[0080] (1) The present invention designs and develops a metaverse task offloading and resource scheduling method based on deep reinforcement learning, which uses drones as mobile edge computing nodes to provide metaverse services for users wearing head-mounted displays. This drone-assisted architecture makes up for the coverage limitations of fixed base stations or edge servers, enabling computing resources to be dynamically adjusted as users move, ensuring that users can obtain low-latency services at any location. Compared with traditional fixed nodes, the present invention has more advantages in service coverage and flexibility.

[0081] (2) The present invention designs and develops a metaverse task offloading and resource scheduling method based on deep reinforcement learning. Through the multi-node collaborative computing and resource sharing mechanism between drones, fixed edge servers and user terminals, combined with deep reinforcement learning for task scheduling and load balancing, it can intelligently allocate tasks among multiple nodes, avoid single node overload or resource idleness, and achieve efficient resource utilization. This multi-node collaborative optimization mechanism is more systematic than existing single-node or local optimization solutions. It can still provide high-quality metaverse services in real time when network conditions, computing resources and user needs change, thereby improving the overall service quality.

[0082] (3) The present invention designs and develops a metaverse task offloading and resource scheduling method based on deep reinforcement learning. By introducing the deep reinforcement learning (DRL) algorithm, the system can intelligently adapt to changes in computing nodes (such as drones and edge servers) and user locations, optimize task offloading and real-time allocation of computing resources, and achieve efficient task offloading and rapid transmission, improve dynamic resource allocation capabilities and real-time decision-making capabilities, significantly reduce user-side computing delays, and enhance the real-time experience of metaverse services.

[0083] (4) The metaverse task offloading and resource scheduling method based on deep reinforcement learning designed and developed by the present invention intelligently analyzes the resource utilization and energy consumption status of different nodes, selects the optimal offloading path and computing node, reduces unnecessary energy consumption, extends the drone's flight time, and ensures service quality. In a drone network with limited resources, this energy consumption management method effectively reduces unnecessary energy consumption, improves the sustainability and efficiency of the system, and is superior to traditional high-energy-consuming edge computing solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 This is a flow chart of the metaverse task offloading and resource scheduling method based on deep reinforcement learning described in the present invention.

[0085] Figure 2 Schematic diagram of the total reward simulation curve of the algorithm described in the present invention and the greedy algorithm.

[0086] Figure 3 Schematic diagram of the frame reward simulation curve of the algorithm described in the present invention and the greedy algorithm.

[0087] Figure 4 Schematic diagram of energy consumption reward simulation curve of the algorithm described in the present invention and the greedy algorithm. DETAILED DESCRIPTION

[0088] The present invention will be further described below in detail with reference to the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0089] like Figure 1 As shown, the present invention provides a metaverse task offloading and resource scheduling method based on deep reinforcement learning, which is implemented through a system consisting of three key components: a user terminal, a drone and a fixed edge server. The user terminal is equipped with a head-mounted display for receiving and presenting metaverse content, and the user terminal accesses edge computing resources through a wireless network; the drone is a drone carrying a computing module, which serves as a mobile edge computing node and is responsible for providing computing resources near the user, and can be dynamically scheduled according to the user's location and computing task requirements; the fixed edge server is a fixed computing node deployed on the ground, which forms a collaborative network with the drone to provide support for greater computing needs.

[0090] Therefore, the metaverse task offloading and resource scheduling method based on deep reinforcement learning described in the present invention specifically includes the following steps:

[0091] Step 1: Deploy a drone cluster in a user-dense area. Based on the Cartesian coordinate system, the fixed edge server obtains the location coordinates of itself, each user, and each drone.

[0092] Among them, user-dense areas are areas with more than 0.05 users / m²;

[0093] The user set is , the drone collection is ,Edge devices include a set of drones and fixed edge servers, and the set is defined as ;

[0094] user exist The position coordinates at the moment are , the drone exist The position coordinates at the moment are , fixed edge server The coordinates are ;

[0095] User in The position at a moment can be expressed as:

[0096] ;

[0097] Where, For drones exist The position coordinates at the moment, For drones exist The horizontal rate at time, For drones exist The direction of time, , For drones exist The vertical velocity at the moment, For Time has come The length of the time period, For drones exist Position coordinates at the moment

[0098] A safe distance should be maintained between drones and between drones and users:

[0099] ;

[0100] ;

[0101] Where, For drones and drones The distance between For users and drones The distance between is the minimum safe distance;

[0102] The drone and drones The distance between them satisfies:

[0103] ;

[0104] ;

[0105] Where, For drones exist The position coordinates at the moment, For drones exist The position coordinates at the moment.

[0106] Step 2: The user sends a rendering request to the fixed edge server through the drone cluster;

[0107] Step 3: Construct an optimization objective and convert it into a Markov decision process. Solve the optimization objective using the PPO algorithm to obtain the UAV's horizontal flight speed, flight direction, vertical speed, and the set of computing resources available to the edge device.

[0108] Wherein, the optimization goal is:

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] Where, is the frame delay weight, For users Frame delay, is the weight of energy consumption, For users The system energy consumption, For edge devices Assign to user The proportion of computing resources, For edge devices and users The distribution coefficient between For users or edge devices and user or edge devices The distance between For users or edge devices exist The position coordinates at the moment, The boundary of the movable area for users and drones, For user collection, is an edge device set, which includes a drone set and a fixed edge server;

[0115] The user The frame delay of is composed of two parts: rendering delay and transmission delay, namely:

[0116] ;

[0117] Where, For rendering delay, is the transmission delay;

[0118] The rendering delay satisfies:

[0119] ;

[0120] The transmission delay satisfies:

[0121] ;

[0122] Where, The computing resources required for edge devices to process unit data, is a user Offload to edge devices The frame size, For edge devices Available computing resources, For edge devices and users The communication rate between them meets the following requirements:

[0123] ;

[0124] Where, For edge devices bandwidth, For edge devices The power, is the channel gain coefficient, For edge devices and users The distance between is the noise power.

[0125] The user The system energy consumption meets the following requirements:

[0126] ;

[0127] Where, To transmit energy, To render energy;

[0128] The transmission energy satisfies:

[0129] ;

[0130] The rendering energy satisfies:

[0131] ;

[0132] Where, For edge devices The transmission power, is the power factor of the edge device.

[0133] Proximal Policy Optimization (PPO) is a reinforcement learning algorithm. The present invention optimizes task offloading and resource scheduling strategies through the PPO algorithm to achieve efficient real-time computing allocation in a dynamic environment.

[0134] In the PPO algorithm, the task offloading problem is modeled as a Markov decision process (MDP) to express the decision process in a formal way. The MDP consists of five tuples: They represent the state space, observation space, action space, reward function and state transfer equation respectively.

[0135] Among them, the state space : ;

[0136] Where, for The state space at time, yes The set of user and drone locations at each moment, The power of the user and the drone is collected. is the user's task data set, A collection of computing resources available to edge devices.

[0137] Action Space : ;

[0138] Where, for The action space at each moment, express The collection of drone flight actions at any moment, including the horizontal direction, horizontal speed and vertical speed of flight. is the set of user task offloading decisions, which is a set of binary variables. At this time, if the user task is offloaded to the edge device ,but =1; if the user task is not offloaded to the edge device ,but =0.

[0139] Reward Function : ;

[0140] Where, for The reward function at that moment.

[0141] The PPO algorithm mainly includes two networks, the policy network and value network ,in, are the network parameters of the policy network, is the network parameter of the value network. The specific steps are as follows:

[0142] Step 1: Initialize the network parameters of the policy network , network parameters of the value network and experience pool ;

[0143] Step 2: Fixed edge server passing current policy Interact with the environment, collect and record information such as the state, action, reward, and next state related to each decision to form trajectory data Stored in experience pool middle;

[0144] in, For the The policy function of the iteration, Indicates that the parameter is The policy network outputs the policy, is the iteration round, For The state of the environment at any moment;

[0145] Step 3: For each trajectory in the experience pool , calculate the cumulative return of actual observations:

[0146] ;

[0147] Where, is the single sampling time step, for Step discount factor Power, For Instant rewards at every moment;

[0148] Step 4: Use the advantage estimation method to directly predict the state value function based on the value network Compute the advantage function:

[0149] ;

[0150] Where, is the action value function, which is estimated using the generalized advantage estimation method, namely:

[0151] ;

[0152] Where, is the discount factor, is a hyperparameter, , used to control the bias-variance trade-off, for The time series difference error at time , where Indicates the current moment, Indicates the step offset from the current moment;

[0153] The time series difference error at the moment satisfies:

[0154] ;

[0155] Where, For the current moment The timing difference error, Status The true value function value of

[0156] The advantage function is used to measure the superiority of the current action relative to the baseline strategy and guide the optimization direction of the strategy.

[0157] Step 5: Use the gradient descent method to adjust the network parameters of the policy network Perform updates with the goal of maximizing the objective function;

[0158] The objective function is:

[0159] ;

[0160] Where, is the objective function, For expectations, is the probability ratio of the current strategy to the old strategy, is the advantage function, is the clipping function, The clipping range is usually set to 0.1 to limit the policy update amplitude and avoid excessive policy updates;

[0161] The probability ratio of the current strategy and the old strategy satisfies:

[0162] ;

[0163] Where, The current policy network is in state Select Action The probability of For the old policy network in state Next select action probability;

[0164] The clipping function limits the update amplitude to arrive , to ensure that the policy changes are controlled, thereby avoiding excessive policy fluctuations. Through this process, the policy is continuously improved in multiple iterations, achieving stable improvement of the task offloading policy and dynamic adaptation to the environment. Therefore, the clipping function satisfies:

[0165] ;

[0166] Network parameters of the policy network Updates meet:

[0167] ;

[0168] Step 6. Calculation , use gradient descent to update the value function parameters :

[0169] ;

[0170] ;

[0171] Where, For the The size of the experience pool at step time;

[0172] Where, is the loss function of the value network, For the The updated value network parameters after iterations;

[0173] Step 4: The fixed edge server sends the decision result to the user and drone. The user and drone execute the decision. The drone adjusts its trajectory based on the decision result. The user offloads the rendering task to the user or the corresponding edge device (drone or edge server) based on the decision result.

[0174] Step 5: After the drone completes the rendering task, it transmits the result back to the user.

[0175] In this embodiment, the , other parameter values are shown in Table 1:

[0176] Table 1 Parameters

[0177]

[0178] like Figure 2 As shown, the algorithm described in the present invention improves the total system reward by 66.7% compared with the traditional greedy algorithm, indicating that this DRL-based dynamic decision-making capability enables the system to more effectively respond to real-time requirements in complex scenarios in the metaverse, which is significantly better than traditional solutions that rely on fixed strategies or simple optimization algorithms.

[0179] like Figure 3 As shown in the simulation result diagram, the algorithm described in the present invention reduces energy consumption and increases the number of frames by 58.4% compared with the traditional greedy algorithm. It can be seen that the algorithm described in the present invention performs task scheduling and load balancing through deep reinforcement learning, and can intelligently allocate tasks among multiple nodes to avoid overload of a single node or idle resources, thereby achieving efficient utilization of resources. This multi-node collaborative optimization mechanism is more systematic than the existing single-node or local optimization scheme, and improves the overall service quality.

[0180] like Figure 4 As shown in the simulation result diagram, the algorithm described in the present invention reduces the energy consumption reward by 7.2% compared with the traditional greedy algorithm. It can be seen that in the resource-limited drone network, this energy consumption management method effectively reduces unnecessary energy consumption, improves the sustainability and efficiency of the system, and is better than the traditional high-energy-consuming edge computing solution.

[0181] The present invention designs and develops a metaverse task offloading and resource scheduling method based on deep reinforcement learning. It adopts the DRL algorithm to comprehensively optimize the offloading path and resource allocation according to the resource availability, user location and network status of the drone and edge server. According to the decision result, the drone dynamically adjusts the movement trajectory and can be flexibly deployed according to the user's location, providing low-latency and high-bandwidth computing support for users wearing head-mounted displays, making up for the limitations of traditional fixed edge servers, and is particularly suitable for scenarios where users move frequently in the metaverse. At the same time, the user computing task is offloaded to the selected edge node. For example, if a drone node is selected, the task will be transmitted to a nearby drone for processing. The drone quickly completes data processing through local computing and returns the processing results to the user terminal. The system The optimal configuration of computing resources is achieved through a multi-node collaborative mechanism to improve the system's resource utilization, latency performance, and reliability. Tasks are dynamically allocated between multiple drone nodes and edge servers to effectively balance the load and avoid resource bottlenecks at a single node. The system continuously optimizes and adjusts the strategy of the DRL decision module by monitoring user experience indicators (latency) and energy consumption feedback. The system continuously updates its strategy and selects the optimal computing offloading strategy in real time, which not only achieves dynamic offloading but also optimizes the energy consumption of drones. By balancing energy consumption and computing resources, the system provides users with efficient computing while maximizing the drone's flight time, thereby adapting to the needs of long-term and frequent computing tasks, adapting to the dynamic changes in the environment and task requirements, and achieving efficient metaverse services.

[0182] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and embodiments shown and described herein.

Claims

1. A metaverse task offloading and resource scheduling method based on deep reinforcement learning, characterized in that: The steps include: Step 1: Deploy a drone cluster in a user-dense area. Based on the Cartesian coordinate system, the fixed edge server obtains the location coordinates of itself, each user, and each drone. Step 2: The user sends a rendering request to the fixed edge server through the drone cluster; Step 3: Construct an optimization objective and convert it into a Markov decision process. Solve the optimization objective using the PPO algorithm to obtain the UAV's horizontal flight speed, flight direction, vertical speed, and the set of computing resources available to the edge device. The optimization goal is: ； ； ； ； ； Where, is the frame delay weight, For users Frame delay, is the number of users, is the weight of energy consumption, For users The system energy consumption, For edge devices Assign to user The proportion of computing resources, For edge devices and users The distribution coefficient between For users or edge devices and user or edge devices The distance between For users or edge devices exist The position coordinates at the moment, The boundary of the movable area for users and drones, For user collection, is an edge device set, which includes a drone set and a fixed edge server; Step 4: The fixed edge server sends the decision result to the user and drone, and the user and drone execute the decision; Step 5: After the drone completes the rendering task, it transmits the result back to the user.

2. The method for metaverse task offloading and resource scheduling based on deep reinforcement learning according to claim 1, characterized in that: The user's location coordinate update satisfies: ； Where, For drones exist The position coordinates at the moment, For drones exist The horizontal rate at time, For drones exist The direction of time, , For drones exist The vertical velocity at the moment, For Time has come The length of the time period, For drones exist The position coordinates at the moment.

3. The method for metaverse task offloading and resource scheduling based on deep reinforcement learning according to claim 2, characterized in that: The user The frame delay satisfies: ； Where, For rendering delay, For transmission delay.

4. The method for metaverse task offloading and resource scheduling based on deep reinforcement learning according to claim 3, characterized in that: The rendering delay satisfies: ； Where, The computing resources required for edge devices to process unit data, is a user Offload to edge devices The frame size, For edge devices Available computing resources.

5. The method for metaverse task offloading and resource scheduling based on deep reinforcement learning according to claim 4, characterized in that: The transmission delay satisfies: ； Where, For edge devices and users Communication rate between The edge device and users The communication rate between them satisfies: ； Where, For edge devices bandwidth, For edge devices The power, is the channel gain coefficient, For edge devices and users The distance between is the noise power.

6. The method for metaverse task offloading and resource scheduling based on deep reinforcement learning according to claim 5, characterized in that: The user The system energy consumption meets the following requirements: ； Where, To transmit energy, For rendering energy.

7. The method for metaverse task offloading and resource scheduling based on deep reinforcement learning according to claim 6, characterized in that: The transmission energy satisfies: ； Where, For edge devices transmission power.

8. The method for metaverse task offloading and resource scheduling based on deep reinforcement learning according to claim 7, characterized in that: The rendering energy satisfies: ； Where, is the power factor of the edge device.

9. The method for metaverse task offloading and resource scheduling based on deep reinforcement learning according to claim 8, characterized in that: Converting the optimization objective into a Markov decision process specifically includes: State Space : ; Action Space : ; Reward Function : ; in, for The state space at time, for The set of user and drone locations at each moment, The power of the user and the drone is collected. is the user's task data set, A collection of computing resources available to edge devices. for The action space at each moment, for A collection of drone flight actions at every moment, is the set of user task offloading decisions, for The reward function at that moment.

10. The method for metaverse task offloading and resource scheduling based on deep reinforcement learning according to claim 9, characterized in that: The PPO algorithm specifically includes the following steps: Step 1: Initialize the network parameters of the policy network , network parameters of the value network and experience pool ; Step 2: Fixed edge server passing current policy Interact with the environment and record trajectory data And stored in the experience pool middle; in, For the The policy function of the iteration, Indicates that the parameter is The policy network outputs the policy, is the iteration round, For The state of the environment at any moment; Step 3: For each trajectory in the experience pool , calculate the cumulative return of actual observations: ； Where, is the single sampling time step, for Step discount factor Power, For the moment Instant rewards received; Step 4: Use the advantage estimation method to directly predict the state value function based on the value network Compute the advantage function: ； Where, is an action-value function that satisfies: ； Where, is the discount factor, is a hyperparameter, , used to control the bias-variance trade-off, For the moment The timing difference error of Indicates the current moment, Indicates the step offset from the current moment; Step 5: Use the gradient descent method to adjust the network parameters of the policy network Perform updates with the goal of maximizing the objective function; The objective function is: ； Where, is the objective function, For expectations, is the probability ratio of the current strategy to the old strategy, is the advantage function, is the clipping function, is the cutting range; Network parameters of the policy network Updates meet: ； Where, For the The size of the experience pool at step time; Step 6: Update the value function parameters using gradient descent : ； Where, is the loss function of the value network, For the The updated value network parameters after iterations.

Images