A reinforcement learning-based unmanned aerial vehicle task offloading fast adaptation method

Abstract

The application provides a kind of unmanned aerial vehicle task unloading fast adaptation method based on reinforcement learning, it is related to unmanned aerial vehicle flight control technical field.The application learns from the offline experience collected in the environment with different states, executes online policy adaptation in the environment with new state dynamics, is applicable to the real scene in which online interaction cost is higher, and can realize generalization based on offline experience data.The application establishes a reinforcement learning model based on decoupling policy and environment representation and a policy adaptation algorithm based on gradient ascent to obtain the action of unmanned aerial vehicle under the policy that can achieve the best performance of unmanned aerial vehicle, so that unmanned aerial vehicle and base station jointly provide edge computing services for the computing tasks generated by ground terminal equipment.The application only needs a small amount of online interaction between unmanned aerial vehicle and current environment in online adaptation test environment, compared with a large number of online interaction samples required by other mainstream methods, can greatly reduce the training cost of model in actual environment.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) flight control technology, and in particular to a rapid adaptation method for UAV mission offloading based on reinforcement learning. Background Technology

[0002] With the rapid development of IoT technology, the traffic demands of various services are experiencing explosive growth. For example, the widespread application of big data and cloud computing has made it difficult for traditional terrestrial wireless networks to meet the massive traffic requirements of these applications. Furthermore, due to limitations in network capacity and coverage, terrestrial base stations cannot be quickly deployed to provide network services during disasters such as forest fires. Therefore, there is an urgent need for a network architecture that is easy to deploy and highly scalable to cope with complex and harsh environments and meet the traffic demands of numerous mobile users.

[0003] Given the advantages of drones—low cost, easy scalability, and rapid deployment—their application as aerial base stations has garnered widespread attention from academia and industry, becoming a potentially effective solution. However, unlike traditional static ground base stations, drones have limited energy storage and transmission power, making long-term deployment difficult, and their service range is time-limited. Furthermore, user locations often change dynamically, leading to fluctuations in network demand across different areas. In traditional drone networks, because a single drone can only perceive user information within its own service range and lacks cooperation with other drones, multiple drones often concentrate in the same area, resulting in low network coverage efficiency and wasted drone resources. Therefore, designing reasonable flight paths for drones to maximize service coverage within limited flight time is crucial for improving the efficiency and quality of drone network services.

[0004] Deep reinforcement learning (DRL) has become an important tool for solving complex decision-making problems. The core objective of reinforcement learning is to learn a policy through interaction with the environment to maximize cumulative rewards. Recently, DRL has achieved significant success in some high-dimensional state observation and planning problems, such as video games, Go, and robot control. Inspired by this, the DRL framework has gradually become a popular solution for solving traditional optimization problems in unmanned aerial vehicle (UAV) networks. However, the generalization problem of reinforcement learning is a major challenge in current artificial intelligence research. Generalization aims to enable reinforcement learning agents to perform well in unknown environments beyond the training environment. Traditional reinforcement learning methods typically rely on repeated interaction and optimization with a specific environment. However, this approach has the limitation that the trained policy is often only effective in the specific training environment. When the dynamics, observations, and rewards of the environment change, the performance of the policy may drop sharply. This lack of generalization reflects two key challenges of reinforcement learning in the real world: first, the uncertainty of the environment. Real-world environments are usually diverse and complex, and cannot be fully simulated by a limited training environment. For example, a robot may need to cope with different ground materials, and the friction and elasticity of each material will affect the effectiveness of its actions. Secondly, the training process is costly. In many reinforcement learning applications (such as medical diagnosis or industrial control), conducting extensive trials directly in real-world environments may be impractical due to the high cost and risks involved. Therefore, generalization must be achieved based on limited training data or offline data.

[0005] Numerous methods have been proposed in the literature on reinforcement learning generalization. For example, Yarts et al. proposed a data augmentation method in "Deep variational information bottleneck," and Peng et al. proposed a domain randomization method in "Learning dexterous in-hand manipulation" to increase the similarity between the training and testing environments (i.e., the generalized or adapted environments). These methods typically require knowledge of environmental changes and the ability to generate the environment. In contrast, other works aim to achieve rapid adaptation to the testing environment without requiring such access and knowledge. For instance, Finn et al. proposed gradient-based meta-RL in "Outperforming the atari human benchmark," aiming to learn a meta-policy that can adapt to the testing environment within a few policy gradient steps. Also in this branch, Rakelly et al. proposed context-based meta-RL in "A contrastive log-ratio upper bound of mutual information," utilizing a context-conditional policy that allows adaptation through generalization across contexts. In their paper "Fast reinforcement learning via slow reinforcement learning," Fu et al. proposed learning useful contextual representations in the training environment to capture changes; this allows inference of the test environment's context from limited probing interactions. Current solutions to this problem improve generalization to some extent, but they typically rely on knowledge of environmental changes and require manually designed randomization strategies, which are costly. Meta-reinforcement learning solutions allow agents to interact arbitrarily with the training environment within the interaction budget. However, in real-world problems, online interactions are often expensive, while offline experiences are often available and relatively plentiful. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a rapid adaptation method for unmanned aerial vehicle (UAV) task offloading based on reinforcement learning. This method learns from offline experience collected in environments with different states and performs online policy adaptation in environments with new dynamic states. It is suitable for real-world scenarios where online interaction is costly and can achieve generalization based on offline experience data, thus reducing the generalization cost.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] This invention provides a rapid adaptation method for unmanned aerial vehicle (UAV) task offloading based on reinforcement learning, involving UAVs, base stations, and ground terminal equipment. The UAV collects computing tasks generated by the ground terminal equipment, maximizes the number of collected computing tasks by optimizing the UAV's flight trajectory, and maximizes the computing service efficiency jointly provided by the UAV and base station for the computing tasks generated by the ground terminal equipment by optimizing the UAV's task offloading decision. The method includes the following steps:

[0009] Step 1: Establish a reinforcement learning model based on decoupling strategy and environment representation, and initialize the parameters of the reinforcement learning model based on decoupling strategy and environment representation;

[0010] The reinforcement learning model based on decoupling strategy and environment representation includes several UAVs and an environment. The UAVs interact with the environment and iteratively learn to optimize the actions of the UAVs in the environment. The environment includes several ground terminal devices and several base stations. The actions of the UAVs in the environment include collecting computing tasks generated by the ground terminal devices in the environment, making task offloading decisions, and jointly providing computing services to the ground terminal devices in the environment with the help of the base stations in the environment. The task offloading decision is the decision to offload the computing tasks collected by the UAVs to the base stations in the environment.

[0011] Initialize the parameters of the reinforcement learning model based on the decoupling policy and environment representation, including the state space S of the environment, the reward function R, and the action space A of the UAV. The specific method is as follows:

[0012] The environment is divided into a training environment and a testing environment, and the training environment set is defined as follows. The test environment is set as The training strategy set is ,in, For the number of training environments, For the number of test environments, To increase the number of training policies, an offline experience buffer is established. As training samples, among which In the first Training environment Use the first Training strategies Generated offline experience;

[0013] Assuming the drone starts from an initial state and continuously interacts with several ground terminal devices and several base stations in the environment, the resulting drone trajectory is as follows: It is used to record the state transitions of the drone and the rewards obtained during the state transition process. For environmental conditions, For the actions of drones, Rewards earned by the drone; set in the training environment A set of drone trajectories generated in For the context information of the environment and set context information Environmental status and the actions of drones For the behavior of drones ;

[0014] Step 2: Train a reinforcement learning model based on decoupled policy and environment representation using training samples, different training environments and different training strategies. Obtain environment embedding vectors for different training environments and policy embedding vectors for different training strategies. Predict the value prediction network reward value obtained by using different training strategies in different training environments.

[0015] Step 2.1: Build the context encoder By comparing the contextual information of the drone in different training environments, the environment embedding vectors of different training environments are extracted;

[0016] For any training environment with a training environment set Using from the training environment Offline experience Contextual information of mid-sampling As anchor sample and positive samples From another training environment Offline experience Contextual information from mid-samples is used as negative samples Based on anchor samples and positive samples Generate similar sample pairs based on anchor samples and negative samples Generate different sample pairs;

[0017] Build a context encoder Used to train the environment Offline experience Contextual information of mid-sampling Mapped to environment embedding vector Context encoder loss function for: (1)

[0018] in, For context encoder Updated parameter matrix, For mathematical expectation, For anchor samples The environment embedding vector, , Each is a context encoder Extracted positive samples negative samples The environment embedding vector, For anchor samples Context embedding vector The transpose of ;

[0019] Use training samples to train the context encoder Training is performed to obtain a trained context encoder. and based on this context encoder Extract training environment Contextual information Context embedding vector ;

[0020] Step 2.2: Construct the policy encoder and policy decoder Through the policy encoder Extracting training strategies strategy embedding vector Then through the policy decoder Predictive training strategy The action of disembarking the drone;

[0021] Build a policy encoder For training strategies Chinese drone behavior Using from the training environment Offline experience The behavior of drones sampled in the middle Embedded vectors as training strategy strategy embedding vector ;

[0022] Build a policy decoder , environmental status and training strategies strategy embedding vector As a policy decoder Input, predict training strategy The action of disembarking the drone;

[0023] By minimizing the L2 loss function Update policy encoder and policy decoder As shown in the formula below:

[0024] (2)

[0025] Obtain the trained policy encoder and policy decoder Through a well-trained policy encoder Extracting training strategies strategy embedding vector Then, through the trained policy decoder Predictive training strategy The action of disembarking the drone;

[0026] Step 2.3, Construction parameters are: Value Prediction Network Using a trained value prediction network Predict the training environment The following training strategies The value that can be obtained is the network reward value for prediction. ;

[0027] Value Prediction Network It is a multi-layer nonlinear neural network, for offline experience Based on environmental conditions Training strategies strategy embedding vector Training environment Context embedding vector As a value prediction network The input is used to train a value prediction network using the Monte Carlo method. Value prediction network loss function As shown in the formula below:

[0028] (3)

[0029] in, , For long-term reward benefits, γ As a discount factor, For time step t The reward function, where T is the total number of time steps, is given by... This is the Monte Carlo Return. This represents the initial environmental state.

[0030] Use a trained value prediction network Predict the training environment The following training strategies The value that can be obtained is the network reward value for prediction. ;

[0031] Step 3: Based on the reinforcement learning model of decoupled strategy and environment representation, establish a gradient ascent-based strategy adaptation algorithm to optimize the policy embedding vector of the UAV adopting different strategies in the test environment, obtain the UAV's actions under the best-performing strategy, and interact with the current environment.

[0032] Step 3.1: From the training policy set Midsampling any training strategy The drone uses this training strategy With the test environment Interact with the test environment and collect test environment data. Context information ;

[0033] Step 3.2: Predict the network reward value based on the value. A gradient ascent-based policy adaptation algorithm is established, and the test environment is optimized using this algorithm. Training strategies for drones The optimized training policy is obtained by using the policy embedding vector. strategy embedding vector ;

[0034] The gradient-ascent-based policy adaptation algorithm sets the number of iterations. Each iteration uses the training strategy Sample a drone trajectory , For testing environment The environmental conditions, For training strategy The action of disembarking the drone Perform actions for drones The new environment that followed The environmental conditions will determine the drone's trajectory. As an environmental encoder The input yields the test environment. Context embedding vector ; track the drone As a policy encoder The input is used to obtain the training strategy. strategy embedding vector ; along the value prediction network using the gradient ascent algorithm Predicted value, predicted network reward value With learning rate η Increased direction to optimize training strategy strategy embedding vector As shown in the formula below:

[0035] (4)

[0036] Until completion This process continues until the predetermined number of iterations is reached. Then, the optimized training strategy is obtained. strategy embedding vector ;

[0037] Step 3.3: Use the policy decoder Based on the optimized training strategy strategy embedding vector To obtain the drone's actions and interactions with the current environment under the strategy that optimizes drone performance;

[0038] Step 4: Implement drone-assisted mobile edge computing based on the drone's actions under the strategy of optimal drone performance, so that the drone and base station jointly provide edge computing services for the computing tasks generated by ground terminal equipment;

[0039] Step 4.1: Generate optimized drone flight trajectory and mission offloading decisions based on the drone's actions under the strategy that optimizes drone performance;

[0040] Step 4.2: The UAV collects and stores the computing tasks of the ground terminal equipment. Based on the optimized UAV flight trajectory, the number of tasks generated by the ground terminal equipment collected by the UAV is maximized and stored in the UAV's task queue, thus completing the collection of computing tasks of the ground terminal equipment.

[0041] Step 4.3: Determine the allocation of computing tasks for ground terminal equipment based on the UAV's task offloading decision, including the number of computing tasks allocated to the UAV and the number of computing tasks offloaded to the base station.

[0042] Step 4.4: Execute the calculation tasks according to the calculation task allocation results; the UAV executes the calculations of the tasks that do not need to be unloaded, and transmits the tasks that need to be unloaded from the UAV to the base station for calculation on the base station;

[0043] Step 4.5: After the task calculations in the base station and the UAV are completed, the task calculation results in the base station and the UAV are returned to the ground terminal equipment.

[0044] The beneficial effects of adopting the above technical solution are as follows: This invention provides a rapid adaptation method for UAV task offloading based on reinforcement learning. It separates the policy representation from the dynamic feature representation of the environment, trains the two representation spaces separately through a contrastive learning mechanism, and utilizes mutual information optimization to ensure decoupling while maintaining the integrity of the representation. A value prediction network is introduced to evaluate the value of different policy and environment dynamic combinations. Offline experience learning is used to infer the environment representation through a small amount of online data in a new environment, achieving rapid policy adjustment. A gradient-ascent-based policy adaptation algorithm is established to optimize the policy embedding vectors of different policies adopted by the UAV in the test environment. The environment and policy representations are trained on offline experience data with various dynamic features, and the policy is efficiently optimized in the test environment, avoiding retraining the entire model and significantly improving adaptation efficiency. The training data of this invention mainly consists of offline samples, requiring only a small amount of online interaction between the UAV and the current environment in the test environment. Compared with other mainstream methods that require a large number of online interaction samples, this significantly reduces the training cost of the model in real-world environments. Attached Figure Description

[0045] Figure 1 A schematic diagram of the reinforcement learning framework provided in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a gradient-ascent-based strategy adaptation algorithm provided in an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the experimental environment for the reinforcement learning fast adaptation method based on decoupling strategy and environment representation provided in the embodiments of the present invention;

[0048] Figure 4 The comparative test results provided in this embodiment of the invention are shown in the following figures: (a) is a schematic diagram of the UAV trajectory without generalization, (b) is a schematic diagram of the UAV trajectory after generalization, and (c) is a graph showing the experimental results of the reward value comparison between this embodiment and other methods. Detailed Implementation

[0049] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0050] This embodiment presents a rapid adaptation method for drone task offloading based on reinforcement learning, involving a drone, a base station, and ground terminal equipment. The drone collects computational tasks generated by the ground terminal equipment. By optimizing the drone's flight trajectory, the number of collected computational tasks is maximized. By optimizing the drone's task offloading decisions, the efficiency of the computational services jointly provided by the drone and the base station for the computational tasks generated by the ground terminal equipment is maximized. Figure 1 As shown, it includes the following steps:

[0051] Step 1: Establish a reinforcement learning model based on decoupling strategy and environment representation, and initialize the parameters of the reinforcement learning model based on decoupling strategy and environment representation;

[0052] The reinforcement learning model based on decoupling strategies and environment representation includes several UAVs and an environment. The UAVs interact with the environment and iteratively learn to optimize the actions of the UAVs in the environment. The environment includes several ground terminal devices and several base stations. The actions of the UAVs in the environment include collecting computing tasks generated by the ground terminal devices in the environment, making task offloading decisions, and jointly providing computing services to the ground terminal devices in the environment with the help of the base stations in the environment. The task offloading decision is the decision to offload the computing tasks collected by the UAVs to the base stations in the environment.

[0053] In reinforcement learning models based on decoupled strategies and environment representations, the interaction between the environment and the drone is modeled as a Markov Decision Process (MDP) to describe the state transitions, reward mechanisms, and drone action choices during the interaction process. This includes: a state space S, an action space A, transition probabilities P, a reward function R, and a discount factor γ. The state space S is the set of all possible environment states, describing the specific state of the environment at each time point. The action space A is the set of all possible drone actions; each drone action affects the state of the environment. The transition probability P is the probability that the current environment state will transition to another environment state after the drone takes an action in a given environment state; P is a probability model based on environment dynamics. The reward function R is the immediate reward obtained by the drone after performing an action in a given environment state; the reward signal is the basis for evaluating the quality of the drone's behavior. The discount factor γ is a discount coefficient used to calculate future rewards, reflecting the drone's emphasis on future rewards. The value of the discount factor γ is between 0 and 1; the closer the value of the discount factor γ is to 1, the more the drone values ​​future rewards.

[0054] Initialize the parameters of the reinforcement learning model based on the decoupling policy and environment representation, including the state space S of the environment, the reward function R, and the action space A of the UAV. The specific method is as follows:

[0055] The state space S of the environment is initialized based on the observations of the UAV, which include: the position of the UAV and the positions of the three nearest ground terminal devices; the action space A of the UAV is initialized, which includes: the UAV's direction of movement, speed of movement, and task unloading decision; the reward function R is initialized, which is set as the sum of three objective reward functions: the total energy consumption of the UAV Etotal, the task delay Ttotal, and the number of tasks collected Ntotal.

[0056] The environment is divided into a training environment and a testing environment, and the training environment set is defined as follows. The test environment is set as The training strategy set is ,in, For the number of training environments, For the number of test environments, To increase the number of training policies, an offline experience buffer is established. As training samples, among which In the first Training environment Use the first Training strategies Generated offline experience;

[0057] Assuming the drone starts from an initial state and continuously interacts with several ground terminal devices and several base stations in the environment, the resulting drone trajectory is as follows: It is used to record the state transitions of the drone and the rewards obtained during the state transition process. For environmental conditions, For the actions of drones, Rewards earned by the drone; set in the training environment A set of drone trajectories generated in For the context information of the environment and set context information Environmental status and the actions of drones For the behavior of drones .

[0058] Step 2: Train a reinforcement learning model based on decoupled policy and environment representation using training samples and different training environments. Obtain environment embedding vectors for different training environments and policy embedding vectors for different training policies. Predict the value of the network reward that can be obtained by using different training policies in different training environments.

[0059] Step 2.1: Build the context encoder By comparing the contextual information of the drone in different training environments, the environment embedding vectors of different training environments are extracted;

[0060] For any training environment with a training environment set Using from the training environment Offline experience Contextual information of mid-sampling As anchor sample and positive samples From another training environment Offline experience Contextual information from mid-samples is used as negative samples Based on anchor samples and positive samples Generate similar sample pairs based on anchor samples and negative samples Generate different sample pairs;

[0061] Build a context encoder Used to train the environment Offline experience Contextual information of mid-sampling Mapped to environment embedding vector Context encoder loss function for: (1)

[0062] in, For context encoder Updated parameter matrix, For mathematical expectation, For anchor samples The environment embedding vector, , Each is a context encoder Extracted positive samples negative samples The environment embedding vector, For anchor samples Context embedding vector The transpose of ;

[0063] Use training samples to train the context encoder Training is performed to obtain a trained context encoder. and based on this context encoder Extract training environment Contextual information Context embedding vector ;

[0064] Step 2.2: Construct the policy encoder and policy decoder Through the policy encoder Extracting training strategies strategy embedding vector Then through the policy decoder Predictive training strategy The action of disembarking the drone;

[0065] Build a policy encoder For training strategies Chinese drone behavior Using from the training environment Offline experience The behavior of drones sampled in the middle Embedded vectors as training strategy strategy embedding vector ;

[0066] Build a policy decoder , environmental status and training strategies strategy embedding vector As a policy decoder Input, predict training strategy The action of disembarking the drone;

[0067] By minimizing the L2 loss function Update policy encoder and policy decoder As shown in the formula below:

[0068] (2)

[0069] Obtain the trained policy encoder and policy decoder Through a well-trained policy encoder Extracting training strategies strategy embedding vector Then, through the trained policy decoder Predictive training strategy The action of disembarking the drone;

[0070] Step 2.3, Construction parameters are: Value Prediction Network Using a trained value prediction network Predict the training environment The following training strategies The value that can be obtained is the network reward value for prediction. ;

[0071] Value Prediction Network It is a multi-layer nonlinear neural network, for offline experience Based on environmental conditions Training strategies strategy embedding vector Training environment Context embedding vector As a value prediction network The input is used to train a value prediction network using the Monte Carlo method. Value prediction network loss function As shown in the formula below:

[0072] (3)

[0073] in, , For long-term reward benefits, γ As a discount factor, For time step t The reward function, where T is the total number of time steps, is given by... This is the Monte Carlo Return. This represents the initial environmental state.

[0074] Use a trained value prediction network Predict the training environment The following training strategies The value that can be obtained is the network reward value for prediction. The value prediction network algorithm is shown in Table 1.

[0075] Table 1 Value Prediction Network Algorithm:

[0076]

[0077] Step 3: Based on the reinforcement learning model of decoupled strategy and environment representation, establish a gradient ascent-based strategy adaptation algorithm to optimize the policy embedding vector of the UAV adopting different strategies in the test environment, obtain the UAV's actions under the best-performing strategy, and interact with the current environment.

[0078] Step 3.1: From the training policy set Midsampling any training strategy The drone uses this training strategy With the test environment Interact with the test environment and collect test environment data. Context information ;

[0079] Step 3.2: Predict the network reward value based on the value. A gradient ascent-based policy adaptation algorithm is established, and the test environment is optimized using this algorithm. Training strategies for drones The optimized training policy is obtained by using the policy embedding vector. strategy embedding vector ;

[0080] Setting the number of iterations in a gradient-ascent-based policy adaptation algorithm Each iteration uses the training strategy Sample a drone trajectory , For testing environment The environmental conditions, For training strategy The action of disembarking the drone Perform actions for drones The new environment that followed The environmental conditions will determine the drone's trajectory. As an environmental encoder The input yields the test environment. Context embedding vector ; track the drone As a policy encoder The input is used to obtain the training strategy. strategy embedding vector ; along the value prediction network using the gradient ascent algorithm Predicted value, predicted network reward value With learning rate η Increased direction to optimize training strategy strategy embedding vector As shown in the formula below:

[0081] (4)

[0082] Until completion This process continues until the predetermined number of iterations is reached. Then, the optimized training strategy is obtained. strategy embedding vector The policy adaptation algorithm based on gradient ascent is shown in Table 2.

[0083] Table 2 shows the policy adaptation algorithm based on gradient ascent:

[0084]

[0085] Use a policy decoder Based on the optimized training strategy strategy embedding vector To obtain the drone's actions and interactions with the current environment under the strategy that optimizes drone performance;

[0086] In this embodiment, the best-performing policy representation encountered during training is used as the adjusted policy. The policy and the current environment are used as input to the policy decoder to obtain the action the UAV should take, such as... Figure 2 As shown.

[0087] Step 4: Implement drone-assisted mobile edge computing based on the drone's actions under the strategy of optimal drone performance, so that the drone and base station jointly provide edge computing services for the computing tasks generated by ground terminal equipment;

[0088] Based on steps 2 and 3, the UAV-assisted mobile edge computing model is established by training a generalization model on the collected offline dataset. This allows the UAV to quickly adapt to the real environment. By optimizing the UAV's trajectory and task offloading decisions, the UAV and the edge cloud jointly provide edge computing services for computing tasks generated by ground terminal devices. UAV-assisted mobile edge computing includes:

[0089] Step 4.1: Generate optimized drone flight trajectory and mission offloading decisions based on the drone's actions under the strategy that optimizes drone performance;

[0090] Step 4.2: The UAV collects and stores the computing tasks of the ground terminal equipment. Based on the optimized UAV flight trajectory, the number of tasks generated by the ground terminal equipment collected by the UAV is maximized and stored in the UAV's task queue, thus completing the collection of computing tasks of the ground terminal equipment.

[0091] Step 4.3: Determine the allocation of computing tasks for ground terminal equipment based on the UAV's task offloading decision, including the number of computing tasks allocated to the UAV and the number of computing tasks offloaded to the base station.

[0092] Step 4.4: Execute the calculation tasks according to the calculation task allocation results; the UAV executes the calculations of the tasks that do not need to be unloaded, and transmits the tasks that need to be unloaded from the UAV to the base station for calculation on the base station;

[0093] Step 4.5: After the task calculations in the base station and the drone are completed, the task calculation results in the base station and the drone are returned to the terminal device.

[0094] In this embodiment, an experimental environment is built for a reinforcement learning-based rapid adaptation method based on decoupling strategies and environment representations, such as... Figure 3 As shown, including N One drone, MThere are one ground terminal device and one base station, and their locations are represented by coordinates. The ground terminal devices lack computing power, while the UAVs provide limited computing and communication resources. The base station, unlike the UAVs, consists of edge servers, which are small-scale cloud data centers providing real-time data processing, analysis, and decision-making. Each ground terminal device needs to handle computationally intensive tasks, and the UAVs provide edge computing services to the ground terminal devices. Since the ground terminal devices lack computing power, the computing tasks are completely offloaded to the UAVs. However, the UAVs have limited computing power; therefore, this embodiment uses a joint UAV and edge cloud approach to provide offloading services for the tasks generated by the ground terminal devices. In this embodiment, the UAV trajectory is planned to save energy, reduce latency, collect as many tasks as possible, and avoid collisions between UAVs.

[0095] To demonstrate the effectiveness of this invention, this embodiment creates many different environments. Then, the environment set is split into training and testing subsets so that during testing, the agent must find a policy that performs well in unseen environments. Finally, the average reward value obtained by the proposed generalization method in the test set is calculated and compared with the average reward obtained by the model trained separately in each test environment using the PPO algorithm, and the average reward obtained by the model trained in each training set in each test set. It is also compared with two current mainstream reinforcement learning generalization algorithms, PDVF and MAML. Experimental results are as follows: Figure 4 As shown, (a) is a schematic diagram of the drone trajectory without generalization, (b) is a schematic diagram of the drone trajectory after generalization, and (c) is a diagram of the experimental results comparing the reward value of this embodiment with other methods.

[0096] Experimental results show that the average reward obtained by the proposed method (ThisMethod) is significantly higher than the average reward obtained by the PPO model without generalization (PPOInTrain). This is also evident from the trajectories of the four drones, which demonstrate a significant reduction in drone collisions. Furthermore, the performance of this method is higher than MAML and PDVF, but slightly lower than the PPO model trained in the test environment (PPOInTest). These experimental results fully demonstrate the effectiveness of the generalization method proposed in this invention.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A rapid adaptation method for unmanned aerial vehicle (UAV) task offloading based on reinforcement learning, involving UAVs, base stations, and ground terminal equipment, characterized in that: The drone collects computing tasks generated by ground terminal equipment, maximizes the number of collected computing tasks by optimizing the drone's flight trajectory, and maximizes the computing service efficiency jointly provided by the drone and base station for the computing tasks generated by the ground terminal equipment by optimizing the drone's task offloading decision. This includes the following steps: Step 1: Establish a reinforcement learning model based on decoupling strategy and environment representation, and initialize the parameters of the reinforcement learning model based on decoupling strategy and environment representation; Step 2: Train a reinforcement learning model based on decoupled policy and environment representation using training samples, different training environments and different training strategies. Obtain environment embedding vectors for different training environments and policy embedding vectors for different training strategies. Predict the value prediction network reward value obtained by using different training strategies in different training environments. Step 3: Based on the reinforcement learning model of decoupled strategy and environment representation, establish a gradient ascent-based strategy adaptation algorithm to optimize the policy embedding vector of the UAV adopting different strategies in the test environment, obtain the UAV's actions under the best-performing strategy, and interact with the current environment. Step 4: Implement drone-assisted mobile edge computing based on the drone's actions under the strategy of optimal drone performance, so that the drone and base station jointly provide edge computing services for the computing tasks generated by ground terminal equipment; The reinforcement learning model based on decoupling strategy and environment representation described in step 1 includes several UAVs and an environment. The UAVs interact with the environment and iteratively learn to optimize the actions of the UAVs in the environment. The environment includes several ground terminal devices and several base stations. The actions of the UAVs in the environment include collecting computing tasks generated by the ground terminal devices in the environment, making task offloading decisions, and jointly providing computing services to the ground terminal devices in the environment with the help of the base stations in the environment. The task offloading decision is the decision to offload the computing tasks collected by the UAVs to the base stations in the environment. Step 1 involves initializing the reinforcement learning model parameters based on the decoupling strategy and environment representation. The specific method is as follows: The state space S of the environment is initialized based on the observations of the UAV, which include: the position of the UAV and the positions of the three nearest ground terminal devices; the action space A of the UAV is initialized, which includes: the UAV's direction of movement, movement speed, and task offloading decision; the reward function R is initialized as the sum of three objective reward functions: the total energy consumption Etotal of the UAV, the task delay Ttotal, and the number of tasks collected Ntotal. The environment is divided into a training environment and a testing environment, and the training environment set is defined as follows. The test environment is set as The training strategy set is ,in, For the number of training environments, For the number of test environments, To increase the number of training policies, an offline experience buffer is established. As training samples, among which In the first Training environment Use the first Training strategies Generated offline experience; Assuming the drone starts from an initial state and continuously interacts with several ground terminal devices and several base stations in the environment, the resulting drone trajectory is as follows: It is used to record the state transitions of the drone and the rewards obtained during the state transition process. For environmental conditions, For the actions of drones, Rewards earned by the drone; set in the training environment A set of drone trajectories generated in For the context information of the environment and set context information Environmental status and the actions of drones For the behavior of drones .

2. The rapid adaptation method for unmanned aerial vehicle (UAV) mission offloading based on reinforcement learning according to claim 1, characterized in that: Step 2 includes: Step 2.1: Build the context encoder By comparing the contextual information of the drone in different training environments, the environment embedding vectors of different training environments are extracted; Step 2.2: Construct the policy encoder and policy decoder Through the policy encoder Extracting training strategies strategy embedding vector Then through the policy decoder Predictive training strategy The action of disembarking the drone; Step 2.3, Construction parameters are: Value Prediction Network Using a trained value prediction network Predict the training environment The following training strategies The value that can be obtained is the network reward value for prediction. .

3. The rapid adaptation method for unmanned aerial vehicle (UAV) mission offloading based on reinforcement learning according to claim 2, characterized in that: The specific method for step 2.1 is as follows: For any training environment with a training environment set Using from the training environment Offline experience Contextual information of mid-sampling As anchor sample and positive samples From another training environment Offline experience Contextual information from mid-samples is used as negative samples Based on anchor samples and positive samples Generate similar sample pairs based on anchor samples and negative samples Generate different sample pairs; Build a context encoder Used to train the environment Offline experience Contextual information of mid-sampling Mapped to environment embedding vector Context encoder loss function for: (1) in, For context encoder Updated parameter matrix, For mathematical expectation, For anchor samples The environment embedding vector, , Each is a context encoder Extracted context positive samples negative samples The environment embedding vector, For anchor samples Context embedding vector The transpose of ; Use training samples to train the context encoder Training is performed to obtain a trained context encoder. and based on this context encoder Extract training environment Contextual information Context embedding vector .

4. The rapid adaptation method for unmanned aerial vehicle (UAV) mission offloading based on reinforcement learning according to claim 3, characterized in that: The specific method for step 2.2 is as follows: Build a policy encoder For training strategies Chinese drone behavior Using from the training environment Offline experience The behavior of drones sampled in the middle Embedded vectors as training strategy strategy embedding vector ; Build a policy decoder , environmental status and training strategies strategy embedding vector As a policy decoder Input, predict training strategy The action of disembarking the drone; By minimizing the L2 loss function Update policy encoder and policy decoder The trained policy encoder is obtained. and policy decoder Through a well-trained policy encoder Extracting training strategies strategy embedding vector Then, through the trained policy decoder Predictive training strategy The action of disembarking the drone; l2 loss function As shown in the formula below: (2) 。 5. The rapid adaptation method for unmanned aerial vehicle (UAV) mission offloading based on reinforcement learning according to claim 4, characterized in that: The specific method for step 2.3 is as follows: Value Prediction Network It is a multi-layer nonlinear neural network, for offline experience Based on environmental conditions Training strategies strategy embedding vector Training environment Context embedding vector As a value prediction network The input is used to train a value prediction network using the Monte Carlo method. Value prediction network loss function As shown in the formula below: (3) in, , For long-term reward returns, γ is the discount factor. Let be the reward function for time step t, where T is the total number of time steps, and let This is the Monte Carlo Return. This represents the initial environmental state. Use a trained value prediction network Predict the training environment The following training strategies The value that can be obtained is the network reward value for prediction. .

6. The rapid adaptation method for unmanned aerial vehicle (UAV) mission offloading based on reinforcement learning according to claim 5, characterized in that: Step 3 includes: Step 3.1: From the training policy set Midsampling any training strategy The drone uses this training strategy With the test environment Interact with the test environment and collect test environment data. Context information ; Step 3.2: Predict the network reward value based on the value. A gradient ascent-based policy adaptation algorithm is established, and the test environment is optimized using this algorithm. Training strategies for drones The optimized training policy is obtained by using the policy embedding vector. strategy embedding vector ; Step 3.3: Use the policy decoder Based on the optimized training strategy strategy embedding vector To obtain the optimal drone performance strategy, we need to determine the drone's actions and how it interacts with the current environment.

7. A rapid adaptation method for unmanned aerial vehicle (UAV) mission offloading based on reinforcement learning according to claim 6, characterized in that: The specific method for step 3.2 is as follows: The gradient-ascent-based policy adaptation algorithm sets the number of iterations. Each iteration uses the training strategy Sample a drone trajectory , For testing environment The environmental conditions, For training strategy The action of disembarking the drone Perform actions for drones The new environment that followed The environmental conditions will determine the drone's trajectory. As an environmental encoder The input yields the test environment. Context embedding vector ; track the drone As a policy encoder The input is used to obtain the training strategy. strategy embedding vector ; along the value prediction network using the gradient ascent algorithm Predicted value, predicted network reward value Optimize training strategy in the direction of increasing learning rate η The policy embedding vector is shown in the following formula: (4) Until completion This process continues until the predetermined number of iterations is reached. Then, the optimized training strategy is obtained. strategy embedding vector .

8. A rapid adaptation method for unmanned aerial vehicle (UAV) mission offloading based on reinforcement learning according to claim 7, characterized in that: Step 4 includes: Step 4.1: Generate optimized drone flight trajectory and mission offloading decisions based on the drone's actions under the strategy that optimizes drone performance; Step 4.2: The UAV collects and stores the computing tasks of the ground terminal equipment. Based on the optimized UAV flight trajectory, the number of tasks generated by the ground terminal equipment collected by the UAV is maximized and stored in the UAV's task queue, thus completing the collection of computing tasks of the ground terminal equipment. Step 4.3: Determine the allocation of computing tasks for ground terminal equipment based on the UAV's task offloading decision, including the number of computing tasks allocated to the UAV and the number of computing tasks offloaded to the base station. Step 4.4: Execute the calculation tasks according to the calculation task allocation results; the UAV executes the calculations of the tasks that do not need to be unloaded, and transmits the tasks that need to be unloaded from the UAV to the base station for calculation on the base station; Step 4.5: After the task calculations in the base station and the UAV are completed, the task calculation results in the base station and the UAV are returned to the ground terminal equipment.

Images