Unmanned aerial vehicle mobile edge network optimization method driven by evolutionary multi-objective deep reinforcement learning

Abstract

The invention provides an evolutionary multi-objective deep reinforcement learning-driven unmanned aerial vehicle mobile edge network optimization method. The method comprises the following steps of 1, establishing a multi-objective optimization model of an unmanned aerial vehicle mobile edge network; and 2, solving the multi-objective optimization model established in the step 1 by adopting an evolutionary multi-objective deep reinforcement learning driven unmanned aerial vehicle mobile edge network optimization method. According to the method, the Chebyshev decomposition strategy is fused into the deep reinforcement learning framework, and the purpose of evaluating multiple objective functions at the same time is achieved. According to the method, the population is intelligently guided to continuously evolve towards a direction with better distribution and better convergence by utilizing the powerful decision-making capability of the PPO algorithm and the good diversity maintenance capability of the TD strategy, and the evolution efficiency and diversity distribution of the population are remarkably improved. Finally, by implementing the method provided by the invention, the performance of the mobile edge network of the unmanned aerial vehicle on user rate and system energy consumption can be greatly improved.

Description

Technical Field

[0001] The present invention relates to an unmanned aerial vehicle mobile edge network optimization method driven by evolutionary multi-objective deep reinforcement learning. Background Art

[0002] With the rapid development of mobile communication technology and the surge in IoT devices, drone mobile edge networks, as an emerging network architecture, are gradually becoming a key technology to meet users' demand for low latency and high bandwidth. In this context, drones, as mobile base stations or data processing centers, can dynamically adjust their locations to optimize network coverage and performance, providing users with more flexible and efficient network services. However, the optimization of drone mobile edge networks faces a series of challenges, such as multi-objective optimization problems such as resource allocation and path planning, which often have multiple conflicting optimization objectives, such as increasing user data rates, reducing system energy consumption, and reducing task processing delays. Summary of the invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide an optimization method for UAV mobile edge network driven by evolutionary multi-objective deep reinforcement learning in view of the shortcomings of the prior art. The core of this method is to build a multi-objective optimization model for UAV mobile edge network that comprehensively considers key performance indicators such as total user rate and total system energy consumption, and takes into account constraints such as computing resource allocation and task processing delay.

[0004] The method of the present invention comprises the following steps:

[0005] Step 1, establish a multi-objective optimization model for the UAV mobile edge network;

[0006] In step 2, an evolutionary multi-objective deep reinforcement learning-driven UAV mobile edge network optimization method is used to solve the multi-objective optimization model established in step 1.

[0007] Step 1 includes:

[0008] Step 1.1, establish the objective function of the total user rate;

[0009] Step 1.2, establish the objective function of user delay;

[0010] Step 1.3, establish the objective function of the total energy consumption of the system;

[0011] Step 1.4, establish a multi-objective optimization model for UAV mobile edge network.

[0012] Step 1.1 includes: the signal transmission between the drone and the user is divided into two categories: line-of-sight transmission and non-line-of-sight transmission, and the average path loss between user i and drone j is established. for:

[0013]

[0014] P LoS =1-P NLoS (2)

[0015] Among them, P LoS is the probability of line-of-sight transmission, P NLoS is the probability of non-line-of-sight transmission, PL LoS is the line-of-sight transmission loss, PL NLoS It is the non-line-of-sight transmission loss;

[0016] SINR when user i communicates with drone i Modeled as:

[0017]

[0018] in, It means that user i has established communication with drone j, and the transmission power of drone j that has established communication with user i is the largest. represents the power of other users within the communication range of the UAV, σ 2 is the noise power, M is the number of users; is the power of the kth user, is the average road strength loss;

[0019] The data transmission rate V of user i i for:

[0020] V i =Blog(1+SINR i ) (4)

[0021] Where B is the communication bandwidth;

[0022] The total user rate V of the entire drone mobile edge network system total for:

[0023]

[0024] Step 1.2 includes: Due to limited local computing resources, consider offloading local user data to the drone for processing. The delay of user i's data offloading to the drone for processing includes offloading delay and computing delay, where the offloading delay refers to the transmission time when user i offloads data to the drone after establishing communication with the drone. The calculation formula is:

[0025]

[0026] Among them, D i is the total data volume of user i;

[0027] Computational latency refers to the time required for the drone to calculate the data of user i The calculation formula is:

[0028]

[0029] Among them, C unit is the number of CPU cycles required for unit data, f i The calculation frequency assigned to the drone for user i;

[0030] Finally, the delay function of user i is established for:

[0031]

[0032] Step 1.3 includes: the energy consumption of unloading user data to the drone includes the drone communication energy consumption and the drone computing energy consumption. Among them, the communication energy consumption refers to the energy consumption of the drone receiving the unloaded data of user i The calculation formula is:

[0033]

[0034] in, is the unloading power of user i;

[0035] The computing energy consumption of drones refers to the energy consumption generated by drones to calculate the unloaded data of user i. The calculation formula is:

[0036]

[0037] Where k is the energy consumption factor;

[0038] Drone flight energy consumption Hover energy consumption They are:

[0039]

[0040] Where ρ is the air density, C is the drag coefficient of the drone, A is the cross-sectional area of ​​the drone, v is the flight speed of the drone, and d i is the distance from the drone to user i, W represents the weight of the drone, and η represents the hovering efficiency;

[0041] The total energy consumption of the final system is E total for:

[0042]

[0043] In step 1.4, the multi-objective optimization model of the UAV mobile edge network is:

[0044] min-V total ,E total

[0045]

[0046] Where st means restricted by, condition C1 means the user task must be completed within T time, and condition C2 means the computing frequency assigned by the drone to the user cannot exceed the maximum computing frequency of the drone

[0047] Step 2 includes:

[0048] Step 2.1, design the proximal policy optimization (PPO) algorithm, which specifically includes the following steps:

[0049] Step 2.1.1, Initialization: Initialize the sampling strategy network π θ' (a t |s t ) and the target policy network π θ (a t |s t ), sampling strategy network π θ' (a t |s t ) and the target policy network π θ (a t |s t ) have the same initial parameters, s t represents the state at time t, a t Represents the action at time t; initializes the value network Used to evaluate the value of a state or state-action pair; set hyperparameters, including learning rate η, batch size Batch, and update interval GT;

[0050] Step 2.1.2, state observation: observe the current state s from the environment t ;

[0051] Step 2.1.3, action selection: Set state s t Input into the sampling strategy network to get action a t The mean μ θ (s t ) and variance σ θ (s t ), thus determining the normal distribution, and sampling from the normal distribution to obtain the specific action a t ;

[0052] Step 2.1.4, perform actions and obtain feedback: perform action at , and get the next state s from the environment t+1 and the reward r at time t t ; will (s t , a t , r t ,s t+1 ) are stored as a sample pair;

[0053] Step 2.1.5, Storing samples: Store more than two (s t , a t , r t ,s t+1 ) The dataset consisting of sample pairs is stored in the trajectory set;

[0054] Step 2.1.6, update the network: extract a batch of sample pairs from the trajectory set, and then use these sample pairs to calculate the advantage function A t :

[0055]

[0056] Where T is the total number of steps, is the state value at time t+k, r t+k is the reward at time t+k; γ∈[0,1] is the discount factor used to measure the importance of future rewards; λ∈[0,1] is the smoothing parameter used to control the balance between bias and variance;

[0057] At the same time, the Clip strategy gradient method L clip (θ) to update the target policy network, the formula is:

[0058]

[0059] in represents the expectation for all moments t, ε is a parameter used to control the clipping amplitude, represents the first-order partial derivative of θ; L clip (θ) represents the Clip strategy item;

[0060] Then, the state s in the sample t Input into the value network to obtain the value prediction of the corresponding state of each time slot, and use the reward r t After calculating the target value, the value network is updated and defined as:

[0061]

[0062] That is the state value at time t+1; L V (θ) is the loss function of the target policy network, is the state value predicted at time t;

[0063] Step 2.1.7, parameter synchronization: At a certain update interval GT (generally 20), copy the parameters of the target strategy network to the sampling strategy network to keep the parameters of the two networks synchronized;

[0064] Step 2.2, design the Chebycheff Decomposition (TD) strategy.

[0065] Step 2.2 specifically includes the following steps:

[0066] Step 2.2.1, initialize N learning tasks to form the initial population P, each learning task includes the target strategy network parameter π θ , sampling strategy network parameter π θ' and value network parameters Initialize N uniformly distributed weight vectors W = [w1, w2, ... w N ], the jth weight vector w j =[w j,1 ,w j,2 ,…,w j,m ], m is the number of objective functions, here it is 2, j is 1 to N, w j,m Represents the m-th dimension component of the j-th weight vector; initialize the reference point Z ref =[z1,z2,…,z m ],z m Indicates the reference point Z ref The m-th dimension component; Set the maximum number of evolutions G max ;

[0067] Step 2.2.2, calculate the target vector F(π θ ):

[0068]

[0069] Step 2.2.3, calculate the target value F(π θ ) and the reference point Z ref The deviation F t :

[0070] F t =F(π θ )-Z ref ;

[0071] Step 2.2.4, use Chebyshev decomposition function to find the deviation F t The maximum index p of the dot product with the weight vector W:

[0072]

[0073] Use Chebyshev decomposition function to match each learning task with a weight vector, that is, each learning task corresponds to a weight vector;

[0074] Step 2.3: Determine whether the maximum evolutionary generation G has been reached max If it is reached, it ends; otherwise, go to step 2.1.2 to step 2.1.7.

[0075] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0076] The present invention also provides a storage medium storing a computer program or instruction. When the computer program or instruction is run on a computer, the steps of the method described are executed.

[0077] Beneficial effects: The present invention designs an innovative evolutionary multi-objective deep reinforcement learning algorithm. The algorithm uses the proximal policy optimization algorithm PPO as the main framework, and cleverly integrates the Chebyshev decomposition strategy TD, so that multiple objective functions can be evaluated at the same time. By making full use of the powerful decision-making ability of the PPO algorithm and the excellent diversity maintenance ability of the TD strategy, the population can be intelligently guided to evolve continuously in the direction of better distribution and better convergence. This method significantly improves the evolutionary efficiency and diversity distribution of the population, thereby effectively optimizing the performance of the drone mobile edge network in terms of user rate and system energy consumption. By implementing the method proposed in the present invention, the overall performance of the drone mobile edge network will be greatly improved, providing strong technical support for future network optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0079] Figure 1 It is a flow chart of the method of the present invention.

[0080] Figure 2 It is a schematic diagram of a drone's moving trajectory on a map that is 400 meters long and 400 meters wide.

[0081] Figure 3 This is another schematic diagram of the drone's movement trajectory on a map that is 400 meters long and 400 meters wide. DETAILED DESCRIPTION

[0082] like Figure 1As shown, an embodiment of the present invention provides an unmanned aerial vehicle mobile edge network optimization method driven by evolutionary multi-objective deep reinforcement learning, comprising the following steps:

[0083] Step 1, establish a multi-objective optimization model for the UAV mobile edge network;

[0084] In step 2, an evolutionary multi-objective deep reinforcement learning-driven UAV mobile edge network optimization method is used to solve the multi-objective optimization model established in step 1.

[0085] Step 1 includes:

[0086] Step 1.1, establish the objective function of the total user rate;

[0087] Step 1.2, establish the objective function of user delay;

[0088] Step 1.3, establish the objective function of the total energy consumption of the system;

[0089] Step 1.4, establish a multi-objective optimization model for UAV mobile edge network.

[0090] Step 1.1 includes: the signal transmission between the drone and the user is divided into two categories: line-of-sight transmission and non-line-of-sight transmission, and the average path loss between user i and drone j is established. for:

[0091]

[0092] P LoS =1-P NLoS (2)

[0093] Among them, P LoS is the probability of line-of-sight transmission, P NLoS is the probability of non-line-of-sight transmission, PL LoS is the line-of-sight transmission loss, PL NLoS It is the non-line-of-sight transmission loss;

[0094] SINR when user i communicates with drone i Modeled as:

[0095]

[0096] in, It means that user i has established communication with drone j, and the transmission power of drone j that has established communication with user i is the largest. represents the power of other users within the communication range of the UAV, σ 2 is the noise power, M is the number of users; is the power of the kth user, is the average road strength loss;

[0097] The data transmission rate V of user i i for:

[0098] V i =Blog(1+SINR i ) (4)

[0099] Where B is the communication bandwidth;

[0100] The total user rate V of the entire drone mobile edge network system total for:

[0101]

[0102] Step 1.2 includes: Due to limited local computing resources, consider offloading local user data to the drone for processing. The delay of user i's data offloading to the drone for processing includes offloading delay and computing delay, where the offloading delay refers to the transmission time when user i offloads data to the drone after establishing communication with the drone. The calculation formula is:

[0103]

[0104] Among them, D i is the total data volume of user i;

[0105] Computational latency refers to the time required for the drone to calculate the data of user i The calculation formula is:

[0106]

[0107] Among them, C unit is the number of CPU cycles required for unit data, f i The calculation frequency assigned to the drone for user i;

[0108] Finally, the delay function of user i is established for:

[0109]

[0110] Step 1.3 includes: the energy consumption of unloading user data to the drone includes the drone communication energy consumption and the drone computing energy consumption. Among them, the communication energy consumption refers to the energy consumption of the drone receiving the unloaded data of user i The calculation formula is:

[0111]

[0112] in, is the unloading power of user i;

[0113] The computing energy consumption of drones refers to the energy consumption generated by drones to calculate the unloaded data of user i. The calculation formula is:

[0114]

[0115] Where k is the energy consumption factor;

[0116] Drone flight energy consumption Hover energy consumption They are:

[0117]

[0118] Where ρ is the air density, C is the drag coefficient of the drone, A is the cross-sectional area of ​​the drone, v is the flight speed of the drone, and d i is the distance from the drone to user i, W represents the weight of the drone, and η represents the hovering efficiency;

[0119] The total energy consumption of the final system is E total for:

[0120]

[0121] In step 1.4, the multi-objective optimization model of the UAV mobile edge network is:

[0122] min-V total ,E total

[0123]

[0124] Where st means restricted by, condition C1 means the user task must be completed within T time, and condition C2 means the computing frequency assigned by the drone to the user cannot exceed the maximum computing frequency of the drone

[0125] Step 2 includes:

[0126] Step 2.1, design the proximal policy optimization (PPO) algorithm, which specifically includes the following steps:

[0127] Step 2.1.1, Initialization: Initialize the sampling strategy network π θ' (a t |s t ) and the target policy network π θ (a t |s t ), sampling strategy network π θ' (a t |st ) and the target policy network πθ(a t |s t ) have the same initial parameters, s t represents the state at time t, a t Represents the action at time t; initializes the value network Used to evaluate the value of a state or state-action pair; set hyperparameters, including learning rate η, batch size Batch, and update interval GT;

[0128] Step 2.1.2, state observation: observe the current state s from the environment t ;

[0129] Step 2.1.3, action selection: Set state s t Input into the sampling strategy network to get action a t The mean μ θ (s t ) and variance σ θ (s t ), thus determining the normal distribution, and sampling from the normal distribution to obtain the specific action a t ;

[0130] Step 2.1.4, perform actions and obtain feedback: perform action a t , and get the next state s from the environment t+1 and the reward r at time t t ; will (s t , a t , r t ,s t+1 ) are stored as a sample pair;

[0131] Step 2.1.5, Storing samples: Store more than two (s t , a t , r t ,s t+1 ) The dataset consisting of sample pairs is stored in the trajectory set;

[0132] Step 2.1.6, update the network: extract a batch of sample pairs from the trajectory set, and then use these sample pairs to calculate the advantage function A t :

[0133]

[0134] Where T is the total number of steps, is the state value at time t+k, r t+kis the reward at time t+k; γ∈[0,1] is the discount factor used to measure the importance of future rewards; λ∈[0,1] is the smoothing parameter used to control the balance between bias and variance;

[0135] At the same time, the Clip strategy gradient method L clip (θ) to update the target policy network, the formula is:

[0136]

[0137] in represents the expectation for all moments t, ε is a parameter used to control the clipping amplitude, represents the first-order partial derivative of θ; L clip (θ) represents the Clip strategy item;

[0138] Then, the state s in the sample t Input into the value network to obtain the value prediction of the corresponding state of each time slot, and use the reward r t After calculating the target value, the value network is updated and defined as:

[0139]

[0140] That is the state value at time t+1; L V (θ) is the loss function of the target policy network, is the state value predicted at time t;

[0141] Step 2.1.7, parameter synchronization: At a certain update interval GT (generally 20), copy the parameters of the target strategy network to the sampling strategy network to keep the parameters of the two networks synchronized;

[0142] Step 2.2, design the Chebycheff Decomposition (TD) strategy.

[0143] Step 2.2 specifically includes the following steps:

[0144] Step 2.2.1, initialize N learning tasks to form the initial population P, each learning task includes the target strategy network parameter π θ , sampling strategy network parameter π θ ' and value network parameters Initialize N uniformly distributed weight vectors W = [w1, w2, ... w N ], the jth weight vector w j =[w j,1 ,w j,2 ,…,w j,m], m is the number of objective functions, here it is 2, j is 1 to N, w j,m Represents the m-th dimension component of the j-th weight vector; initialize the reference point Z ref =[z1,z2,…,z m ],z m Indicates the reference point Z ref The m-th dimension component; Set the maximum number of evolutions G max ;

[0145] Step 2.2.2, calculate the target vector F(π θ ):

[0146]

[0147] Step 2.2.3, calculate the target value F(π θ ) and the reference point Z ref The deviation F t :

[0148] F t =F(π θ )-Z ref ;

[0149] Step 2.2.4, use Chebyshev decomposition function to find the deviation F t The maximum index p of the dot product with the weight vector W:

[0150]

[0151] Use Chebyshev decomposition function to match each learning task with a weight vector, that is, each learning task corresponds to a weight vector;

[0152] Step 2.3: Determine whether the maximum evolutionary generation G has been reached max If it is reached, it ends; otherwise, go to step 2.1.2 to step 2.1.7.

[0153] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0154] The present invention also provides a storage medium storing a computer program or instruction. When the computer program or instruction is run on a computer, the steps of the method described are executed.

[0155] In one embodiment of the present invention, a method for optimizing a UAV mobile edge network driven by evolutionary multi-objective deep reinforcement learning is provided, comprising:

[0156] 1. Establish a multi-objective optimization model for the UAV mobile edge network, including the total user rate and the total system energy consumption. The parameters are set as follows: the number of users M = 10, the number of UAVs is 1, and the noise power σ 2 =-102dBm, communication bandwidth B = 1MHz, user data volume D = 20kbit, unloading power P uload =0.2W, energy consumption factor k=10 -27 .

[0157] 2. Design an evolutionary multi-objective deep reinforcement learning-driven UAV mobile edge network optimization method, including a proximal policy optimization algorithm and a Chebyshev decomposition strategy. The parameters are set as: learning rate η = 0.0001, batch size Batch = 64, and update interval GT = 10, population size N = 15.

[0158] 3. Optimize the UAV mobile edge network using evolutionary multi-objective deep reinforcement learning. The optimization results are compared as follows: the total user rate before optimization is 17.97Mbit, and the total system energy consumption is 0.05kJ; the total user rate after optimization is 28.13Mbit, and the total system energy consumption is 0.03kJ. After optimization, the total user rate is increased by 56.5%, and the system energy consumption is reduced by 40.0%.

[0159] This invention solves the multi-objective joint optimization problem in the optimization of UAV mobile edge networks. Traditional methods are inefficient in dealing with such high-dimensional, non-convex optimization problems, and even fail to find the optimal solution. However, this invention combines deep reinforcement learning and evolutionary algorithms to achieve joint optimization of multiple objectives, effectively improving the overall performance of the system.

[0160] The present invention greatly improves the total user rate and enhances the communication performance of the system by optimizing the drone trajectory and computing resource allocation. The present invention reduces the total system energy consumption and extends the drone's flight time by comprehensively considering the flight energy consumption and computing energy consumption. The method of the present invention is applicable to different drone mobile edge network scenarios and mission requirements, and has strong versatility and adaptability. Figure 2 , Figure 3 As shown in the figure (UAV Environment Simulation represents UAV environment simulation), the starting point of the UAV is randomly initialized on a map with a length of 400 meters and a width of 400. Figure 2 and Figure 3In the figure, the starting point of the drone and the position of the user are randomly generated, the number of users is 60, and the number of drones is 1. The horizontal and vertical axes represent the horizontal and vertical axes of the drone position, respectively. The black asterisks represent the users for whom the drone provides unloading services, and the red asterisks represent the users for whom the drone does not provide unloading services. The yellow circle represents the starting point of the drone, and the blue diamond is the base station. At a fixed number of moments, the drone provides communication services to as many users as possible by moving, while ensuring the optimization of total energy consumption and delay. From the drone trajectory in the figure, it can be seen that the number of users provided is large, and the efficiency of energy consumption and delay is also high.

[0161] The present invention provides an optimization method for a UAV mobile edge network driven by evolutionary multi-objective deep reinforcement learning. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. An evolutionary multi-objective deep reinforcement learning-driven UAV mobile edge network optimization method, characterized in that: The following steps are involved: Step 1, establish a multi-objective optimization model for the UAV mobile edge network; In step 2, an evolutionary multi-objective deep reinforcement learning-driven UAV mobile edge network optimization method is used to solve the multi-objective optimization model established in step 1.

2. The method according to claim 1, characterized in that: Step 1 includes: Step 1.1, establish the objective function of the total user rate; Step 1.2, establish the objective function of user delay; Step 1.3, establish the objective function of the total energy consumption of the system; Step 1.4, establish a multi-objective optimization model for UAV mobile edge network.

3. The method according to claim 2, characterized in that Step 1.1 includes: the signal transmission between the drone and the user is divided into two categories: line-of-sight transmission and non-line-of-sight transmission, and the average path loss between user i and drone j is established. for: P LoS =1-P NLoS (2) Among them, P LoS is the probability of line-of-sight transmission, P NLoS is the probability of non-line-of-sight transmission, PL LoS is the line-of-sight transmission loss, PL NLoS It is the non-line-of-sight transmission loss; SINR when user i communicates with drone i Modeled as: in, It means that user i has established communication with drone j, and the transmission power of drone j that has established communication with user i is the largest. represents the power of other users within the communication range of the UAV, σ 2 is the noise power, M is the number of users; is the power of the kth user, is the average road strength loss; The data transmission rate V of user i i for: V i =Blog(1+SINR i ) (4) Where B is the communication bandwidth; The total user rate V of the entire drone mobile edge network system total for:

4. The method according to claim 3, characterized in that Step 1.2 includes: the delay of user i's data unloading to the drone for processing includes unloading delay and calculation delay, where unloading delay refers to the transmission time when user i unloads data to the drone after user i establishes communication with the drone The calculation formula is: Among them, D i is the total data volume of user i; Computational latency refers to the time required for the drone to calculate the data of user i The calculation formula is: Among them, C unit is the number of CPU cycles required for unit data, f i The calculation frequency assigned to the drone for user i; Finally, the delay function of user i is established for:

5. The method according to claim 4, characterized in that Step 1.3 includes: the energy consumption of unloading user data to the drone includes the drone communication energy consumption and the drone computing energy consumption, where the communication energy consumption refers to the energy consumption of the drone receiving the unloaded data from user i The calculation formula is: in, is the unloading power of user i; The computing energy consumption of drones refers to the energy consumption generated by drones to calculate the unloaded data of user i. The calculation formula is: Where k is the energy consumption factor; Drone flight energy consumption Hover energy consumption They are: Where ρ is the air density, C is the drag coefficient of the drone, A is the cross-sectional area of ​​the drone, v is the flight speed of the drone, and d i is the distance from the drone to user i, W represents the weight of the drone, and η represents the hovering efficiency; The total energy consumption of the final system is E total for:

6. The method according to claim 5, characterized in that In step 1.4, the multi-objective optimization model of the UAV mobile edge network is: Where st means restricted by, condition C1 means the user task must be completed within T time, and condition C2 means the computing frequency assigned by the drone to the user cannot exceed the maximum computing frequency of the drone 7. The method according to claim 6, characterized in that Step 2 includes: Step 2.1, design a proximal strategy optimization algorithm, which specifically includes the following steps: Step 2.1.1, Initialization: Initialize the sampling strategy network π θ' (a t |s t ) and the target policy network π θ (a t |s t ), sampling strategy network π θ' (a t |s t ) and the target policy network π θ (a t |s t ) have the same initial parameters, s t represents the state at time t, a t Represents the action at time t; initializes the value network Used to evaluate the value of a state or state-action pair; set hyperparameters, including learning rate η, batch size Batch, and update interval GT; Step 2.1.2, state observation: observe the current state s from the environment t ; Step 2.1.3, action selection: Set state s t Input into the sampling strategy network to get action a t The mean μ θ (s t ) and variance σ θ (s t ), thus determining the normal distribution, and sampling from the normal distribution to obtain the specific action a t ; Step 2.1.4, perform actions and obtain feedback: perform action a t , and get the next state s from the environment t+1 and the reward r at time t t ; will (s t , a t , r t ,s t+1 ) are stored as a sample pair; Step 2.1.5, Storing samples: Store more than two (s t , a t , r t ,s t+1 ) The dataset consisting of sample pairs is stored in the trajectory set; Step 2.1.6, update the network: extract a batch of sample pairs from the trajectory set, and then use the sample pairs to calculate the advantage function A t : Where T is the total number of steps, is the state value at time t+k, r t+k is the reward at time t+k; γ∈[0,1] is the discount factor; λ∈[0,1] is the smoothing parameter; At the same time, the Clip strategy gradient method L clip (θ) to update the target policy network, the formula is: in represents the expectation for all moments t, ε is a parameter used to control the clipping amplitude, represents the first-order partial derivative of θ; L clip (θ) represents the Clip strategy item; Then, the state s in the sample t Input into the value network to obtain the value prediction of the corresponding state of each time slot, and use the reward r t After calculating the target value, the value network is updated and defined as: That is the state value at time t+1; L V (θ) is the loss function of the target policy network, is the state value predicted at time t; Step 2.1.7, parameter synchronization: At a certain update interval GT, the parameters of the target policy network are copied to the sampling policy network to keep the parameters of the two networks synchronized; Step 2.2, design a Chebyshev decomposition strategy.

8. The method according to claim 7, characterized in that Step 2.2 specifically includes the following steps: Step 2.2.1, initialize N learning tasks to form the initial population P, each learning task includes the target strategy network parameter π θ , sampling strategy network parameter π θ' and value network parameters Initialize N uniformly distributed weight vectors W = [w1, w2, ... w N ], the jth weight vector w j =[w j,1 ,w j,2 ,…,w j,m ], m is the number of objective functions, j takes values ​​from 1 to N, w j,m Represents the m-th dimension component of the j-th weight vector; initialize the reference point Z ref =[z1,z2,…,z m ],z m Indicates the reference point Z ref The m-th dimension component of ; Set the maximum number of evolutions G max ; Step 2.2.2, calculate the target vector F(π θ ): Step 2.2.3, calculate the target value F(π θ ) and the reference point Z ref The deviation F t : F t =F(π θ )-Z ref ; Step 2.2.4, use Chebyshev decomposition function to find the deviation F t The maximum index p of the dot product with the weight vector W: Use Chebyshev decomposition function to match each learning task with a weight vector, that is, each learning task corresponds to a weight vector; Step 2.3: Determine whether the maximum evolutionary generation G has been reached max If it is reached, it ends; otherwise, go to step 2.1.2 to step 2.1.

7.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 8 are executed.

Images