Emergency UAV group trajectory control method and related equipment

Abstract

The present application provides a method for controlling the trajectory of an emergency drone group and related equipment, which are applied to an emergency drone communication network system. The emergency drone communication network system includes multiple drones with base stations and multiple user terminals. Each base station is configured with a federated reinforcement learning agent and an experience replay sample pool, and each federated reinforcement learning agent includes a deep neural network. The method includes: constructing an emergency drone communication network model; obtaining the state information corresponding to the base station, and inputting the state information into the federated reinforcement learning agent, wherein the drone with the closest base station is the target drone; training the deep neural network by retrieving the sample data stored in the experience replay sample pool corresponding to the base station, outputting the action information of the target drone at the current moment, and adjusting the flight trajectory of the target drone according to the action information, and entering the next moment to adjust the flight trajectory. It can interact with the environment in real time and adapt to dynamic environmental changes.

Description

Technical Field

[0001] The present application relates to the field of drone control technology, and in particular to an emergency drone group trajectory control method and related equipment. Background Art

[0002] In the emergency communication system, drone aerial base stations are quickly deployed as relay nodes between users in the disaster area and the external network, efficiently restoring communications in the disaster area through air-to-ground communication. Since the dynamics of drone flight and the relative positions between drone swarms will greatly affect the communication performance of the drone emergency network, the flight trajectory of the emergency drone swarm needs to be reasonably regulated.

[0003] Based on the above situation, the dynamic nature of the emergency communication network makes it impossible for traditional optimization techniques to obtain accurate information about the environment at future moments to accurately control the trajectory of the drone swarm. Reinforcement learning methods can use a large amount of flight data for self-learning, fit unknown environments and cope with their dynamics. However, existing reinforcement learning methods still face problems such as dimensionality explosion and slow convergence. In addition, due to communication overhead and security reasons, it is difficult to centrally control the flight trajectory of drone swarms in emergency networks. Summary of the invention

[0004] In view of this, the purpose of this application is to propose an emergency drone swarm trajectory control method and related equipment to solve or partially solve the above-mentioned technical problems.

[0005] Based on the above purpose, the first aspect of the present application provides an emergency drone group trajectory control method, which is applied to an emergency drone communication network system, wherein the emergency drone communication network system includes multiple drones equipped with base stations and multiple user terminals, each of the base stations is configured with a federated reinforcement learning agent and an experience replay sample pool, and each of the federated reinforcement learning agents includes a deep neural network; the method includes:

[0006] Constructing an emergency drone communication network model corresponding to the emergency drone communication network system according to a plurality of the drones provided with base stations and a plurality of the user terminals;

[0007] In response to determining that any user terminal is activated, the activated user terminal is connected to the nearest base station, and state information corresponding to the nearest base station at the current moment is obtained based on the emergency drone communication network model, and the state information is input into the federated reinforcement learning agent corresponding to the nearest base station, wherein the drone with the nearest base station is the target drone;

[0008] The federated reinforcement learning agent trains the deep neural network based on the received state information and the emergency drone communication network model by retrieving sample data stored in the experience playback sample pool corresponding to the nearest base station, outputs the action information of the target drone at the current moment through the trained deep neural network, adjusts the flight trajectory of the target drone according to the action information at the current moment, and re-optimizes the flight trajectory of the target drone at the next moment;

[0009] In response to determining that all the user terminals have completed the task of restoring the communication service, the optimization and adjustment of the flight trajectory of each of the UAVs is stopped.

[0010] From the above, it can be seen that the emergency drone group trajectory control method and related equipment provided by this application store sample data through the experience regression sampling pool configured by the base station of each drone, optimize the training of the federated reinforcement learning agent by retrieving the sample data, and learn and train the federated reinforcement learning agent through the experience playback sample pool, so that it has the ability to learn autonomously from the accumulated flight data, thereby realizing distributed learning and distributed execution, and can solve the dimensionality explosion crisis caused by the increase in the number of agents. In addition, the federated reinforcement learning algorithm applied by the federated reinforcement learning agent integrates the federated learning algorithm and the multi-agent deep reinforcement learning algorithm, breaks the data islands between drones, improves the performance of the algorithm and accelerates the convergence speed of the algorithm. The optimal flight trajectory adjustment can be obtained in a dynamic real-life scenario, with strong environmental adaptability, generalization ability and faster algorithm convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 This is a flow chart of the trajectory control method of the emergency drone group according to an embodiment of the present application;

[0013] Figure 2 A schematic diagram of an emergency drone communication network system according to an embodiment of the present application;

[0014] Figure 3 This is a structural block diagram of the trajectory control method of the emergency drone group according to an embodiment of the present application;

[0015] Figure 4 This is a schematic diagram of the structure of the trajectory control device for emergency drone swarms according to an embodiment of the present application;

[0016] Figure 5 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0018] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0019] When optimizing drone trajectories in related technologies, it is necessary to establish a target equation for the optimization target problem, bring related variables such as drone trajectory and network environment into the target equation as determined parameters, and optimize the drone trajectory variables in the equation to maximize the target. However, the form of the target equation in actual scenarios is very complex, and requires network environment information at all times from the beginning to the end of the mission. Due to the dynamic nature of the communication network environment, it is difficult to accurately obtain all the parameters in the target equation in actual scenarios. On the other hand, when the communication demand changes, the drone trajectory obtained by the original optimization algorithm cannot be applied and needs to be retrained, which is computationally complex.

[0020] In addition, the drone trajectory control of the deep reinforcement learning algorithm used in related technologies is difficult to eliminate interference between drones in the complex communication network environment in reality, and the reward function cannot decouple the multiple agents in the drone network from each other. The multi-agent reinforcement learning algorithm uses a centralized learning framework, which requires all drones to send back information such as status, action, and reward to the centralized learning center to train the global state value function. This process will generate huge communication overhead that is difficult for the emergency communication network to handle, and there are information security risks. In addition, the input dimension of the state value function will increase in proportion to the number of drones, and there is a potential risk of dimensionality explosion. The convergence speed and effect of the algorithm will also slow down as the number of drones increases.

[0021] The embodiments of the present application provide a method for trajectory control of an emergency drone swarm. By using a federated reinforcement learning agent to autonomously learn from the accumulated flight data, the flight speed and direction can be intelligently and jointly controlled by a distributed algorithm in a dynamic emergency communication scenario to summarize a trajectory with optimal spectrum efficiency. This method can break down data silos between drones, improve algorithm performance and accelerate algorithm convergence speed. At the same time, it can also solve the dimensionality explosion crisis caused by the increase in the number of federated reinforcement learning agents, significantly reduce communication overhead, and improve the feasibility of trajectory control of drone swarms.

[0022] like Figure 1 As shown, the method of this embodiment is applied to an emergency drone communication network system, wherein the emergency drone communication network system includes a plurality of drones equipped with base stations and a plurality of user terminals, each of the base stations is configured with a federated reinforcement learning agent and an experience playback sample pool, and each of the federated reinforcement learning agents includes a deep neural network; the method includes:

[0023] Step 101: construct an emergency drone communication network model corresponding to the emergency drone communication network system based on multiple drones equipped with base stations and multiple user terminals.

[0024] In this step, if Figure 2 As shown, the emergency drone communication network system is a realistic scenario in which multiple drone base stations restore communication services for disaster-stricken users on the ground. In this realistic scenario, there are many disaster-stricken users, drones, and base stations carried by drones. By constructing an emergency drone communication network model corresponding to this dynamic realistic scenario, the actual problem can be abstracted and simplified to solve the actual problem.

[0025] Step 102, in response to determining that any user terminal is activated, the activated user terminal is connected to the nearest base station, and based on the emergency drone communication network model, the status information corresponding to the nearest base station at the current moment is obtained, and the status information is input into the federated reinforcement learning agent corresponding to the nearest base station, wherein the drone with the nearest base station is the target drone.

[0026] In this step, if the user is in an active state, the nearest drone base station will be selected to establish a connection, where users are randomly distributed and their positions move dynamically.

[0027] Step 101: construct an emergency drone communication network model corresponding to the emergency drone communication network system according to the plurality of drones, the plurality of base stations and the plurality of users.

[0028] In this step, the emergency drone communication network system is a realistic scenario in which multiple drone base stations restore communication services for disaster-stricken users on the ground. In this realistic scenario, there are many disaster-stricken users, drones, and base stations carried by drones. By constructing an emergency drone communication network model corresponding to this dynamic realistic scenario, the actual problem is abstracted and simplified to solve the actual problem.

[0029] Step 102, in response to determining that any user terminal is activated, the activated user terminal is connected to the nearest base station, and based on the emergency drone communication network model, the status information corresponding to the nearest base station at the current moment is obtained, and the status information is input into the federated reinforcement learning agent corresponding to the nearest base station, wherein the drone with the nearest base station is the target drone.

[0030] In this step, if the user terminal is in an activated state, it will select the nearest drone base station to establish a connection, where users are randomly distributed and their positions move dynamically.

[0031] If the maximum number of user terminals that each drone can connect to is N max , when the number of user terminal connections established by the drone is more than N max , then N which is closer to the drone base station will be selected max The communication recovery service is provided to the user terminals, and the user terminals farther away are marked as waiting state and are not provided with services at the current moment.

[0032] Step 103: The federated reinforcement learning agent trains the deep neural network according to the received state information and based on the emergency UAV communication network model by retrieving sample data stored in the experience replay sample pool corresponding to the nearest base station, outputs the action information of the target UAV at the current moment through the trained deep neural network, adjusts the flight trajectory of the target UAV according to the action information at the current moment, and enters the next moment to re-optimize the flight trajectory of the target UAV.

[0033] In this step, after entering the next moment, each drone in the emergency drone communication network system will be used as a target drone, and the flight trajectory of each target drone will be optimized and adjusted again.

[0034] Among them, the federated reinforcement learning agent is trained through the experience playback sample pool, so that it has the ability to learn autonomously from the accumulated flight data, so as to achieve distributed learning and distributed execution, which can solve the dimensionality explosion crisis caused by the increase in the number of agents. Then, the federated reinforcement learning agent outputs optimized action information, and the aerial flight trajectory of the drone is accurately controlled in real time according to the action information. The federated reinforcement learning algorithm applied by the federated reinforcement learning agent combines the federated learning algorithm and the multi-agent deep reinforcement learning algorithm, breaking the data islands between drones, improving the performance of the algorithm and accelerating the convergence speed of the algorithm. In dynamic real-world scenarios, the optimal flight trajectory adjustment can be obtained, with strong environmental adaptability, generalization ability and faster algorithm convergence speed.

[0035] The federated reinforcement learning algorithm applied by the federated reinforcement learning agent combines the federated learning algorithm and the multi-agent deep reinforcement learning algorithm, breaking the data silos between drones, improving the performance of the algorithm and accelerating the convergence of the algorithm. It also optimizes parameters during the learning process instead of optimizing trajectory variables based on fixed parameter target equations. Therefore, it has greater flexibility and can quickly respond to changes in scenarios and environments.

[0036] Step 104, in response to determining that all the user terminals have completed the task of restoring the communication service, stop optimizing and adjusting the flight trajectory of each of the drones.

[0037] In this step, all affected users have completed communication restoration and the flight trajectory regulation is completed.

[0038] In the above scheme, the federated reinforcement learning agent is trained through the experience playback sample pool, so that it has the ability to learn autonomously from the accumulated flight data, so as to achieve distributed learning and distributed execution, which can solve the dimensionality explosion crisis caused by the increase in the number of agents. Then, the federated reinforcement learning agent outputs optimized action information, and the aerial flight trajectory of the drone is accurately controlled in real time according to the action information. The federated reinforcement learning algorithm applied by the federated reinforcement learning agent integrates the federated learning algorithm and the multi-agent deep reinforcement learning algorithm, breaks the data islands between drones, improves the performance of the algorithm and accelerates the convergence speed of the algorithm. In dynamic real-world scenarios, the optimal flight trajectory adjustment can be obtained, with strong environmental adaptability, generalization ability and faster algorithm convergence speed.

[0039] In some embodiments, in step 103, the storing of the sample data in the experience playback sample pool corresponding to the nearest base station specifically includes:

[0040] Step A1, based on the emergency drone communication network model and the current status information of the base station provided to the target drone, the current action information of the target drone and the action information of the neighboring drones of the target drone, obtain reward information.

[0041] Step A2: Use the action information at the current moment as shared communication overhead and transmit it to the neighboring drone.

[0042] Step A3, in response to determining that the neighboring drone receives the shared communication overhead, takes the status information of the nearest base station at the current moment, the action information of the target drone at the current moment, the action information of the neighboring drone at the current moment and the return reward information at the current moment as the sample data, and stores them in the experience replay sample pool corresponding to the base station provided for the target drone.

[0043] In the above scheme, the reward information is the weighted spectrum utilization efficiency of the nearest base station k and its K neighboring drone base stations.

[0044] Among them, the spectrum utilization efficiency of the nearest base station at each moment is related to the location of the nearest base station, the locations of all drones adjacent to the target drone, the locations of all ground user terminals, and the activation status.

[0045] The motion information is used to control the flight trajectory of the target UAV. Since the target UAV can move in the x-axis, y-axis, and z-axis directions in three-dimensional space, the motion information includes the speed of any UAV in the x-axis, y-axis, and z-axis directions.

[0046] In addition, if the current moment is moment t, the method of obtaining sample data includes: after the target drone adjusts the flight trajectory according to the action information of the previous moment, obtains the network status at moment t, and shares the action information taken with the base stations of all neighboring drones, that is, shares the communication overhead. After receiving K action information from the base stations of all neighboring drones, the target drone k in the status information, action schedule, and reward information is exchanged with any drone among all neighboring drones, and rearranged to obtain (K+1) pieces of sample data. in, Represents the importance factor, with a value range of And meet

[0047] In some embodiments, the user terminal includes a periodically activated user terminal and / or a randomly activated user terminal;

[0048] The emergency drone communication network model includes at least one of the following: an average path loss model, a signal to interference plus noise ratio model, a spectrum utilization efficiency model, a state information model, an action information model, a reward model, a strategy model, a sample data model and an activation state model of the randomly activated user terminal.

[0049] In the above scheme, if K drones serve N mobile user terminals affected by disasters on the ground, the affected user terminals are randomly distributed and their positions are dynamically moved, and their activation states are periodic or random, that is, the activation states of N1 user terminals in the network are periodic, and the activation states of N2 user terminals are random, where N = N1 + N2. Each user terminal with a periodic activation state will continue to be activated for τ seconds every T seconds, while the activation state of each user terminal with a random activation state follows a beta distribution within time t∈[0, T].

[0050] In some embodiments, the deep neural network includes a first deep neural network, a second deep neural network, a third deep neural network and a fourth deep neural network, and the first deep neural network, the second deep neural network, the third deep neural network and the fourth deep neural network are obtained by learning and training a neural network in an initial state;

[0051] In step 103, the deep neural network is trained by retrieving sample data stored in the experience playback sample pool corresponding to the nearest base station, which specifically includes:

[0052] Step B1, retrieving the sample data from the experience playback sample pool;

[0053] Step B2, inputting the sample data into the first deep neural network, outputting the action information, and obtaining a state behavior value according to the action information;

[0054] Step B3, inputting the sample data into the second deep neural network, fitting the state behavior value through the second deep neural network, and obtaining a first state behavior estimation value;

[0055] Step B4, obtaining parameters of the third deep neural network and parameters of the fourth deep neural network, and performing function processing based on the parameters of the third deep neural network and the parameters of the fourth deep neural network to obtain a time series difference estimation value of the state behavior value;

[0056] Step B5, obtaining a timing difference error function based on the timing difference estimation value, and performing minimization calculation on the timing difference error function to obtain optimized parameters of the second deep neural network, and obtaining an optimized second deep neural network based on the optimized parameters of the second deep neural network;

[0057] Step B6, optimizing the first deep neural network by outputting the first state behavior estimation value of the optimized second deep neural network to obtain optimized parameters of the first deep neural network, and obtaining an optimized first deep neural network based on the optimized parameters of the first deep neural network;

[0058] Step B7, obtaining optimized parameters of the deep neural network based on the optimized parameters of the first deep neural network and the optimized parameters of the second deep neural network, and obtaining the optimized deep neural network based on the optimized parameters of the deep neural network.

[0059] In the above scheme, E sample data are selected from all sample data and stored in the experience replay sample pool for learning and training the federated reinforcement learning agent, namely the first deep neural network, the second deep neural network, the third deep neural network and the fourth deep neural network in the federated reinforcement learning agent. Among them, E is the federated system data, and the trained federated reinforcement learning agent outputs optimized action information, and adjusts the flight trajectory according to the action information, so as to realize distributed learning and distributed execution, which can solve the dimensionality explosion crisis caused by the increase in the number of agents, and greatly reduce the communication overhead, and improve the feasibility of trajectory control of drone groups.

[0060] Among them, Figure 3As shown in the figure, the first deep neural network is the actor network, the second deep neural network is the critic network, the third deep neural network is the target actor network, and the fourth deep neural network is the target critic network. The algorithm used by the federated reinforcement learning agent uses the actor-critic framework, which combines the value function-based and policy gradient-based reinforcement learning methods, inherits the advantages of the policy gradient algorithm in making action decisions in the continuous action space, and improves its low-efficiency learning characteristics based on round updates. In addition, the critic network uses multi-agent reinforcement learning technology to fit the value function with global information to prevent vicious games and performance shocks caused by non-stationary learning environments. The input of the first deep neural network is state information, and the output is action information. The input of the third deep neural network is also state information, and the output is action information. The input of the second deep neural network is state information, action information and action information of neighboring drones, and the output is state behavior estimation. The input of the fourth neural network is also state information, action information and action information of neighboring drones, and the output is state behavior estimation.

[0061] By introducing the third deep neural network and the fourth deep neural network, the training of the deep neural network is made more stable, and the convergence speed and stability of the algorithm are improved.

[0062] In some embodiments, the activation state model:

[0063]

[0064]

[0065] Wherein, κ1 and κ2 represent parameters of the activation state model; T represents the activation period of each of the periodically activated user terminals; t represents any time belonging to [0, T];

[0066] The average path loss model:

[0067]

[0068] Among them, f c It is expressed as the center frequency; It is represented as the position of the target UAV k at time t; Represented as the position of the user terminal u at time t; is represented by the Euclidean distance between the target UAV k and the user terminal u at time t; c is represented by the speed of light; η LoS It is expressed as the additional spatial propagation loss of the line-of-sight wireless transmission link;

[0069] The signal to interference plus noise ratio model is obtained based on the average path loss model, wherein the signal to interference plus noise ratio model is specifically:

[0070]

[0071] Among them, P k Indicates the transmission power of the downlink communication link of the base station provided for the target UAV to the user terminal; It is represented as the gain of the channel between the base station provided on the target UAV and the user terminal u at time t; N0 is represented as the noise power;

[0072] The spectrum utilization efficiency model:

[0073]

[0074] in, Expressed as spectrum utilization efficiency; Indicates whether the user terminal u is in an active state. If it is in an active state and not in a waiting service state, then 1, other states is 0; Indicates whether the base station k provided for the target drone provides services to the user terminal u. If so, If the service is not provided, is 0;

[0075] The state model:

[0076] Get the three-dimensional position of the base station k set by the target drone at time t

[0077] Obtain the three-dimensional positions of the base stations of K neighboring UAVs that are provided with the base station k provided with the target UAV;

[0078] Acquire information of the user terminals with the maximum number of services provided by the base station k provided to the target drone by the neighboring K drones,

[0079] The information of the user terminal includes: two-dimensional location, activation type and continuous activation time;

[0080] The action model:

[0081]

[0082] in, Represented as the action information; It is represented by the speed of any UAV in the x-axis direction; It is represented as the speed of any UAV in the y-axis direction; Expressed as the velocity in the z-axis direction;

[0083] The return reward model:

[0084]

[0085] in, Represented as the return reward information: Represented as the state information of the base station k provided to any UAV at time t; α r It is represented as the weighted discount factor of the neighboring UAV; ξ is represented as the additional safety penalty; Represents the spectrum utilization efficiency of the base station k provided for any of the drones at time t; is the spectrum utilization efficiency of K base stations adjacent to the base station k provided to any of the drones;

[0086] The policy model:

[0087]

[0088] in, It is expressed as the probability of adopting the action information described in the action information set A in any state in the state information set S, also known as strategy information; θ k ∈R n denoted as the parameters of the first deep neural network that need to be optimized;

[0089] Sample data model:

[0090]

[0091] in, Represented as the sample data; Expressed as an importance factor, the value range is and meet

[0092] In the above scheme, a signal to interference plus noise ratio model is obtained based on the average path loss model, a spectrum utilization efficiency model is obtained based on the signal to interference plus noise ratio model, and finally a reward reward model is obtained based on the spectrum utilization efficiency model.

[0093] The state information model is extracted and calculated from the wireless communication network environment. Any UAV base station k and the N neighboring UAV base stations serve maxThe information of each user includes: 2D location, activation type, and continuous activation time. If the user is periodically activated, the activation type is recorded as 0, and if it is randomly activated, it is recorded as 1. The continuous activation time indicates the time the user has been continuously activated. If it is in an inactive state, it is recorded as 0. If the number of users served by any drone base station is less than N max , then fill it with any two-dimensional position, activation type 2, any continuous activation time, and status information The dimension is D(s t )=3+3*K+(2+1+1)*N max .

[0094] In some embodiments, step B6 specifically includes:

[0095] Step B61, obtaining a first gradient function based on the first state behavior estimation value.

[0096] Step B62, by minimizing the first gradient function so as to fit the state behavior value, thereby obtaining optimized parameters of the first deep neural network.

[0097] Step B63, obtaining an optimized first deep neural network based on the optimized parameters of the first deep neural network,

[0098] Wherein, the state behavior value is expressed as:

[0099]

[0100] Represented as the federated reinforcement learning agent in the state information Take the action information below The value of is also called the state behavior value; Indicates the action information selected for the neighboring drone base station; It is represented as the strategy information; γ is represented as a discount factor, and its value range is γ∈[0,1]; Represented as the return reward information;

[0101] The first gradient function is expressed as:

[0102]

[0103] In the above scheme, the state behavior value reflects the parameter θ of the first deep neural network k The quality of the optimization effect.

[0104] Among them, the first deep neural network, the actor network, represents the flight strategy information of the base station set by the target drone, and the federated reinforcement learning agent k in the state information Policy Information The state value equation under is the long-term cumulative reward. When the state information is The policy information is When , its value equation is:

[0105]

[0106] γ is the discount factor, and its value range is γ∈[0,1]. Similarly, we get Next, the state behavior value of action information a.

[0107] Since the reward information after time t cannot be obtained at time t, the unbiased true value of the state behavior value in the optimization target cannot be obtained. In the federated reinforcement learning algorithm applied by the federated reinforcement learning agent, a second deep neural network is used to fit the state behavior value, that is, the state information obtained by the federated reinforcement learning agent k Selected action information Actions selected by the base stations of neighboring drones Input the second deep neural network and output the first state behavior estimate of this state behavior value ω k is the parameter of the second deep neural network. Substitute the estimated value of the first state behavior output by the second deep neural network into the target equation The first gradient function is minimized and the parameters of the first neural network are optimized using the gradient descent method.

[0108] In some embodiments, the timing difference estimate is expressed as:

[0109]

[0110] in, The state information is represented as the state information after the base station position of the drone itself is replaced by the base station position of any of the neighboring drones; Represented as optimized strategy information; Represented as optimized state behavior values; are parameters of the third deep neural network; Represented as parameters of the fourth deep neural network.

[0111] In the above scheme, by introducing such a third deep neural network and a fourth deep neural network, the deep neural network training can be made more stable, and the convergence speed and stability of the algorithm can be improved. and They are the parameters of the third deep neural network and the fourth deep neural network, which are independent of the parameters of the first deep neural network and the second deep neural network, and can further reduce the correlation of the data, thereby reducing the variance when updating the policy gradient parameters, reducing the computational complexity during iterative updates, and improving learning efficiency.

[0112] In some embodiments, the timing difference error function is expressed as:

[0113]

[0114] Wherein, B represents the number of sample data selected from the experience playback sample pool; The time series difference estimate value of the state behavior value.

[0115] In the above scheme, the temporal difference error function reflects the effect of parameter optimization of the second deep neural network.

[0116] In some embodiments, step B5 specifically includes:

[0117] Step B51, obtaining a second gradient function based on the time difference error function.

[0118] Step B52, by minimizing the second gradient function to minimize the temporal difference error function, obtaining optimized parameters of the second deep neural network,

[0119] Wherein, the second gradient function is expressed as:

[0120]

[0121] In the above scheme, the time difference error function is minimized by minimizing the calculation of the second gradient function, so that the optimization of the parameters of the second deep neural network is best.

[0122] In some embodiments, step B7 specifically includes:

[0123] Step B71, updating the parameters of the first deep neural network and the parameters of the second deep neural network based on the second gradient function to obtain updated parameters of the first deep neural network and the parameters of the second deep neural network,

[0124] The updated parameters of the first deep neural network and the updated parameters of the second deep neural network are expressed as:

[0125]

[0126]

[0127] α θ and α ω They are respectively represented as the update step size.

[0128] Step B72: Obtain updated parameters of the third deep neural network and updated parameters of the fourth deep neural network based on the updated parameters of the first deep neural network and the updated parameters of the second deep neural network.

[0129] Step B73, obtaining optimized parameters of the deep neural network based on the updated parameters of the first deep neural network, the updated parameters of the second deep neural network, the updated parameters of the third deep neural network and the updated parameters of the fourth deep neural network.

[0130] Step B74, obtaining the optimized deep neural network based on the optimized parameters of the deep neural network,

[0131] The updated parameters of the third deep neural network and the updated parameters of the fourth deep neural network are expressed as:

[0132]

[0133]

[0134] σ<<1 represents the update step size of the deep neural network.

[0135] In the above scheme, the deep neural network is slowly updated to stabilize the training process.

[0136] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0137] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0138] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an emergency drone group trajectory control device.

[0139] refer to Figure 4 , the emergency drone group trajectory control device comprises:

[0140] A model building module 401 is configured to build an emergency drone communication network model corresponding to the emergency drone communication network system according to a plurality of the drones provided with base stations and a plurality of the user terminals;

[0141] The processing module 402 is configured to, in response to determining that any user terminal is activated, connect the activated user terminal to the nearest base station, obtain state information corresponding to the nearest base station at the current moment based on the emergency drone communication network model, and input the state information to the federated reinforcement learning agent corresponding to the nearest base station, wherein the drone with the nearest base station is the target drone;

[0142] The trajectory adjustment module 403 is configured such that the federated reinforcement learning agent trains the deep neural network based on the emergency drone communication network model by retrieving sample data stored in the experience playback sample pool corresponding to the nearest base station according to the received state information, outputs the action information of the target drone at the current moment through the trained deep neural network, adjusts the flight trajectory of the target drone according to the action information at the current moment, and re-optimizes the flight trajectory of the target drone at the next moment;

[0143] The optimization termination module 404 is configured to stop optimizing and adjusting the flight trajectory of each of the UAVs in response to determining that all of the user terminals have completed the task of restoring the communication service.

[0144] In some embodiments, the trajectory adjustment module 403 is specifically configured to:

[0145] Based on the emergency drone communication network model and the current state information of the base station provided to the target drone, the current action information of the target drone and the action information of the neighboring drones of the target drone, reward information is obtained;

[0146] The action information at the current moment is used as a shared communication overhead and transmitted to the neighboring drone;

[0147] In response to determining that the neighboring drone receives the shared communication overhead, the status information of the nearest base station at the current moment, the action information of the target drone at the current moment, the action information of the neighboring drone at the current moment and the return reward information at the current moment are used as the sample data and stored in the experience replay sample pool corresponding to the base station provided for the target drone.

[0148] In some embodiments, the user terminal includes a periodically activated user terminal and / or a randomly activated user terminal;

[0149] The emergency drone communication network model includes at least one of the following: an average path loss model, a signal to interference plus noise ratio model, a spectrum utilization efficiency model, a state information model, an action information model, a reward model, a strategy model, a sample data model and an activation state model of the randomly activated user terminal.

[0150] In some embodiments, the deep neural network includes a first deep neural network, a second deep neural network, a third deep neural network and a fourth deep neural network, and the first deep neural network, the second deep neural network, the third deep neural network and the fourth deep neural network are obtained by learning and training a neural network in an initial state;

[0151] The trajectory adjustment module 403 is further specifically configured as follows:

[0152] Retrieving the sample data from the experience playback sample pool;

[0153] Inputting the sample data into the first deep neural network, outputting the action information, and obtaining a state behavior value according to the action information;

[0154] Inputting the sample data into the second deep neural network, fitting the state behavior value through the second deep neural network, and obtaining a first state behavior estimation value;

[0155] Acquire parameters of the third deep neural network and parameters of the fourth deep neural network, and perform function processing based on the parameters of the third deep neural network and the parameters of the fourth deep neural network to obtain a time series difference estimation value of the state behavior value;

[0156] Obtaining a time difference error function based on the time difference estimation value, and performing minimization calculation on the time difference error function to obtain optimized parameters of the second deep neural network, and obtaining an optimized second deep neural network based on the optimized parameters of the second deep neural network;

[0157] Optimizing the first deep neural network by outputting a first state behavior estimation value through the optimized second deep neural network to obtain optimized parameters of the first deep neural network, and obtaining an optimized first deep neural network based on the optimized parameters of the first deep neural network;

[0158] Based on the optimized parameters of the first deep neural network and the optimized parameters of the second deep neural network, the optimized parameters of the deep neural network are obtained, and based on the optimized parameters of the deep neural network, the optimized deep neural network is obtained.

[0159] In some embodiments, the activation state model:

[0160]

[0161]

[0162] Wherein, k1 and k2 represent parameters of the activation state model; T represents the activation period of each of the periodically activated user terminals; t represents any time belonging to [0, T];

[0163] The average path loss model:

[0164]

[0165] Among them, f c It is expressed as the center frequency; It is represented as the position of the target UAV k at time t; Represented as the position of the user terminal u at time t; is represented by the Euclidean distance between the target UAV k and the user terminal u at time t; c is represented by the speed of light; η LoS It is expressed as the additional spatial propagation loss of the line-of-sight wireless transmission link;

[0166] The signal to interference plus noise ratio model is obtained based on the average path loss model, wherein the signal to interference plus noise ratio model is specifically:

[0167]

[0168] Among them, P k Indicates the transmission power of the downlink communication link of the base station provided for the target UAV to the user terminal; It is represented as the gain of the channel between the base station provided on the target UAV and the user terminal u at time t; N0 is represented as the noise power;

[0169] The spectrum utilization efficiency model:

[0170]

[0171] in, Expressed as spectrum utilization efficiency; Indicates whether the user terminal u is in an active state. If it is in an active state and not in a waiting service state, then 1, other states is 0; Indicates whether the base station k provided for the target drone provides services to the user terminal u. If so, If the service is not provided, is 0;

[0172] The state model:

[0173] Get the three-dimensional position of the base station k set by the target drone at time t

[0174] Obtain the three-dimensional positions of the base stations of K neighboring UAVs that are provided with the base station k provided with the target UAV;

[0175] Acquire information of the user terminals with the maximum number of services provided by the base station k provided to the target drone by the neighboring K drones,

[0176] The information of the user terminal includes: two-dimensional location, activation type and continuous activation time;

[0177] The action model:

[0178]

[0179] in, Represented as the action information; It is represented by the speed of any UAV in the x-axis direction; It is represented as the speed of any UAV in the y-axis direction; Expressed as the velocity in the z-axis direction;

[0180] The return reward model:

[0181]

[0182] in, Represented as the return reward information; Represented as the state information of the base station k provided to any UAV at time t; α r It is represented as the weighted discount factor of the neighboring UAV; ξ is represented as the additional safety penalty; Represents the spectrum utilization efficiency of the base station k provided for any of the drones at time t; is the spectrum utilization efficiency of K base stations adjacent to the base station k provided to any of the drones;

[0183] The policy model:

[0184]

[0185] in, It is expressed as the probability of adopting the action information described in the action information set A in any state in the state information set S, also known as strategy information; θ k ∈R n denoted as the parameters of the first deep neural network that need to be optimized;

[0186] Sample data model:

[0187]

[0188] in, Represented as the sample data; Expressed as an importance factor, the value range is and meet

[0189] In some embodiments, the first deep neural network is optimized by outputting the first state behavior estimate value through the optimized second deep neural network to obtain optimized parameters of the first deep neural network, and the optimized first deep neural network is obtained based on the optimized parameters of the first deep neural network, specifically including:

[0190] Obtaining a first gradient function based on the first state behavior estimation value;

[0191] By minimizing the first gradient function, the state behavior value is fitted to obtain optimized parameters of the first deep neural network;

[0192] Based on the optimized parameters of the first deep neural network, an optimized first deep neural network is obtained.

[0193] Wherein, the state behavior value is expressed as:

[0194]

[0195] Represented as the federated reinforcement learning agent in the state information Take the action information below The value of is also called the state behavior value; Indicates the action information selected for the neighboring drone base station; It is represented as the strategy information; γ is represented as a discount factor, and its value range is γ∈[0,1]; Represented as the return reward information;

[0196] The first gradient function is expressed as:

[0197]

[0198] In some embodiments, the timing difference estimate is expressed as:

[0199]

[0200] in, It is represented as the state information after replacing the base station position of the drone itself with the base station position of any of the neighboring drones: Represented as optimized strategy information; Represented as optimized state behavior values; are parameters of the third deep neural network; Represented as parameters of the fourth deep neural network.

[0201] In some embodiments, the timing difference error function is expressed as:

[0202]

[0203] Wherein, B represents the number of sample data selected from the experience playback sample pool; The time series difference estimate value of the state behavior value.

[0204] In some embodiments, the minimizing the temporal difference error function to obtain optimized parameters of the second deep neural network specifically includes:

[0205] Obtaining a second gradient function based on the temporal difference error function;

[0206] By minimizing the second gradient function to minimize the temporal difference error function, the optimized parameters of the second deep neural network are obtained.

[0207] Wherein, the second gradient function is expressed as:

[0208]

[0209] In some embodiments, the step of obtaining the optimized parameters of the deep neural network based on the optimized parameters of the first deep neural network and the optimized parameters of the second deep neural network, and obtaining the optimized deep neural network based on the optimized parameters of the deep neural network, specifically includes:

[0210] updating the parameters of the first deep neural network and the parameters of the second deep neural network based on the second gradient function to obtain updated parameters of the first deep neural network and the parameters of the second deep neural network,

[0211] The updated parameters of the first deep neural network and the updated parameters of the second deep neural network are expressed as:

[0212]

[0213]

[0214] α θ and α ω They are respectively represented as update step lengths;

[0215] Obtaining updated parameters of the third deep neural network and updated parameters of the fourth deep neural network based on the updated parameters of the first deep neural network and the updated parameters of the second deep neural network;

[0216] Obtaining optimized parameters of the deep neural network based on the updated parameters of the first deep neural network, the updated parameters of the second deep neural network, the updated parameters of the third deep neural network, and the updated parameters of the fourth deep neural network;

[0217] obtaining an optimized deep neural network based on the optimized parameters of the deep neural network,

[0218] The updated parameters of the third deep neural network and the updated parameters of the fourth deep neural network are expressed as:

[0219]

[0220]

[0221] σ<<1 represents the update step size of the deep neural network.

[0222] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0223] The device of the above embodiment is used to implement the corresponding emergency drone group trajectory control method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0224] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the emergency drone group trajectory control method described in any of the above embodiments is implemented.

[0225] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 501, a memory 502, an input / output interface 503, a communication interface 504, and a bus 505. The processor 501, the memory 502, the input / output interface 503, and the communication interface 504 are connected to each other through the bus 505 in the device.

[0226] The processor 501 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0227] The memory 502 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 502 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 502 and called and executed by the processor 502.

[0228] The input / output interface 503 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0229] The communication interface 504 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0230] The bus 505 includes a path for transmitting information between various components of the device (eg, the processor 501 , the memory 502 , the input / output interface 503 , and the communication interface 504 ).

[0231] It should be noted that, although the above device only shows a processor 501, a memory 502, an input / output interface 503, a communication interface 504, and a bus 505, in a specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0232] The electronic device of the above embodiment is used to implement the corresponding emergency drone group trajectory control method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0233] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the emergency drone swarm trajectory control method as described in any of the above embodiments.

[0234] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0235] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the emergency drone swarm trajectory control method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0236] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0237] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0238] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0239] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for controlling the trajectory of an emergency drone group, characterized in that: Applied to an emergency drone communication network system, the emergency drone communication network system includes multiple drones with base stations and multiple user terminals, each of the base stations is configured with a federated reinforcement learning agent and an experience replay sample pool, and each of the federated reinforcement learning agents includes a deep neural network; the method includes: Constructing an emergency drone communication network model corresponding to the emergency drone communication network system according to a plurality of the drones provided with base stations and a plurality of the user terminals; In response to determining that any user terminal is activated, the activated user terminal is connected to the nearest base station, and state information corresponding to the nearest base station at the current moment is obtained based on the emergency drone communication network model, and the state information is input into the federated reinforcement learning agent corresponding to the nearest base station, wherein the drone with the nearest base station is the target drone; The federated reinforcement learning agent trains the deep neural network based on the received state information and the emergency drone communication network model by retrieving sample data stored in the experience playback sample pool corresponding to the nearest base station, outputs the action information of the target drone at the current moment through the trained deep neural network, adjusts the flight trajectory of the target drone according to the action information at the current moment, and re-optimizes the flight trajectory of the target drone at the next moment; In response to determining that all the user terminals have completed the task of restoring the communication service, stopping optimizing and adjusting the flight trajectory of each of the drones; The sample data stored in the experience playback sample pool corresponding to the nearest base station specifically includes: Based on the emergency drone communication network model and the current state information of the base station provided to the target drone, the current action information of the target drone and the action information of the neighboring drones of the target drone, reward information is obtained; The action information at the current moment is used as a shared communication overhead and transmitted to the neighboring drone; In response to determining that the neighboring drone receives the shared communication overhead, the state information of the nearest base station at the current moment, the action information of the target drone at the current moment, the action information of the neighboring drone at the current moment, and the return reward information at the current moment are used as the sample data, and stored in the experience playback sample pool corresponding to the base station provided for the target drone; The deep neural network includes a first deep neural network, a second deep neural network, a third deep neural network and a fourth deep neural network, wherein the first deep neural network, the second deep neural network, the third deep neural network and the fourth deep neural network are obtained by learning and training the neural network in the initial state; The training of the deep neural network by retrieving sample data stored in the experience playback sample pool corresponding to the nearest base station specifically includes: Retrieving the sample data from the experience playback sample pool; Inputting the state information in the sample data into the first deep neural network, outputting the action information, and obtaining a state behavior value according to the action information; Inputting the state information, the action information, and the action information of the neighboring drones in the sample data into the second deep neural network, and fitting the state behavior value through the second deep neural network to obtain a first state behavior estimation value; Acquire parameters of the third deep neural network and parameters of the fourth deep neural network, and perform function processing based on the parameters of the third deep neural network and the parameters of the fourth deep neural network to obtain a time series difference estimation value of the state behavior value; Obtaining a time difference error function based on the time difference estimation value, and performing minimization calculation on the time difference error function to obtain optimized parameters of the second deep neural network, and obtaining an optimized second deep neural network based on the optimized parameters of the second deep neural network; Optimizing the first deep neural network by outputting a first state behavior estimation value through the optimized second deep neural network to obtain optimized parameters of the first deep neural network, and obtaining an optimized first deep neural network based on the optimized parameters of the first deep neural network; Based on the optimized parameters of the first deep neural network and the optimized parameters of the second deep neural network, the optimized parameters of the deep neural network are obtained, and based on the optimized parameters of the deep neural network, the optimized deep neural network is obtained.

2. The method according to claim 1, characterized in that The user terminal includes a periodically activated user terminal and / or a randomly activated user terminal; The emergency drone communication network model includes at least one of the following: an average path loss model, a signal to interference plus noise ratio model, a spectrum utilization efficiency model, a state information model, an action information model, a reward model, a strategy model, a sample data model and an activation state model of the randomly activated user terminal.

3. The method according to claim 2, characterized in that The activation state model: in, and are represented as parameters of the activation state model; Indicates the activation period time of each periodically activated user terminal; Indicated as belonging to any time; The average path loss model: in, It is expressed as the center frequency; Expressed as The target drone at the moment location; Expressed as The user terminal at the time location; Expressed as The target drone at the moment With the user terminal The Euclidean distance of Expressed as the speed of light; It is expressed as the additional spatial propagation loss of the line-of-sight wireless transmission link; The signal to interference plus noise ratio model is obtained based on the average path loss model, wherein the signal to interference plus noise ratio model is specifically: in, Indicates the transmission power of the downlink communication link of the base station provided for the target UAV to the user terminal; Expressed as The base station and the user terminal provided by the target UAV at the moment The gain of the channel between Expressed as noise power; The spectrum utilization efficiency model: in, Expressed as spectrum utilization efficiency; Represented as the user terminal Is it in active state? If it is in active state and not in waiting for service state, then 1, other states is 0; Indicates the base station provided for the target drone Is it the user terminal? Provide services, and if services are provided, If the service is not provided, is 0; The state model: Get The base station provided by the target drone at the moment The three-dimensional position of ; Obtain the base station provided by the target drone Neighborhood The three-dimensional position of the base station provided on each of the drones; Obtain the base station provided by the target drone Neighborhood information of the maximum number of user terminals served by the base station provided to the drone, The information of the user terminal includes: two-dimensional location, activation type and continuous activation time; Action Model: in, Represented as the action information; It is expressed as the speed of any UAV in the x-axis direction; It is expressed as the speed of any UAV in the y-axis direction; Expressed as the velocity in the z-axis direction; The return reward model: in, Represented as the return reward information; Expressed as The base station set by any drone at any time The status information of denoted as a weighted discount factor for the neighboring UAV; Expressed as an additional safety penalty; It represents the spectrum utilization efficiency of the base station k provided for any UAV at time t; is the spectrum utilization efficiency of K base stations adjacent to the base station k provided to any drone; The policy model: in, It is expressed as the probability of adopting the action information described in the action information set A in any state in the state information set S, also known as strategy information; denoted as the parameters of the first deep neural network that need to be optimized; Sample data model: in, Represented as the sample data; Expressed as an importance factor, the value range is , and meet .

4. The method according to claim 3, characterized in that The step of optimizing the first deep neural network by outputting the first state behavior estimation value through the optimized second deep neural network to obtain optimized parameters of the first deep neural network, and obtaining the optimized first deep neural network based on the optimized parameters of the first deep neural network, specifically includes: Obtaining a first gradient function based on the first state behavior estimation value; By minimizing the first gradient function, the state behavior value is fitted to obtain optimized parameters of the first deep neural network; Based on the optimized parameters of the first deep neural network, an optimized first deep neural network is obtained, Wherein, the state behavior value is expressed as: Represented as the federated reinforcement learning agent in the state information Take the action information below The value of is also called the state behavior value; Indicates the action information selected for the neighboring drone base station; Represented as the strategy information; Expressed as a discount factor, the value range is ; Represented as the return reward information; The first gradient function is expressed as: 。 5. The method according to claim 3, characterized in that: The time series difference estimation value is expressed as: in, The state information is represented as the state information after the base station position of the drone itself is replaced by the base station position of any of the neighboring drones; Represented as optimized strategy information; Represented as optimized state behavior values; are parameters of the third deep neural network; Represented as parameters of the fourth deep neural network.

6. The method according to claim 3, characterized in that The time difference error function is expressed as: Wherein, B represents the number of sample data selected from the experience playback sample pool; The time series difference estimate value of the state behavior value.

7. The method according to claim 6, characterized in that The step of performing minimization calculation on the temporal difference error function to obtain optimized parameters of the second deep neural network specifically includes: Obtaining a second gradient function based on the temporal difference error function; By minimizing the second gradient function to minimize the temporal difference error function, the optimized parameters of the second deep neural network are obtained. Wherein, the second gradient function is expressed as: 。 8. The method according to claim 7, characterized in that The step of obtaining the optimized parameters of the deep neural network based on the optimized parameters of the first deep neural network and the optimized parameters of the second deep neural network, and obtaining the optimized deep neural network based on the optimized parameters of the deep neural network, specifically includes: updating the parameters of the first deep neural network and the parameters of the second deep neural network based on the second gradient function to obtain updated parameters of the first deep neural network and the parameters of the second deep neural network, The updated parameters of the first deep neural network and the updated parameters of the second deep neural network are expressed as: and They are respectively represented as update step lengths; Obtaining updated parameters of the third deep neural network and updated parameters of the fourth deep neural network based on the updated parameters of the first deep neural network and the updated parameters of the second deep neural network; Obtaining optimized parameters of the deep neural network based on the updated parameters of the first deep neural network, the updated parameters of the second deep neural network, the updated parameters of the third deep neural network, and the updated parameters of the fourth deep neural network; obtaining an optimized deep neural network based on the optimized parameters of the deep neural network, The updated parameters of the third deep neural network and the updated parameters of the fourth deep neural network are expressed as: Represents the update step size of the deep neural network.

Images