Optimization method, system and device for cooperative communication of multiple unmanned aerial vehicles

Abstract

The present invention belongs to the field of drone communication, and specifically relates to an optimization method, system and device for cooperative communication of multiple drones. The method comprises: specifying a host and a slave among multiple communicating drones; optimizing the flight trajectories of the host and the slave through a first decision network and a second decision network pre-trained based on a DDQN algorithm. Among them, the first decision network takes the status information of the host and the user as input; the second decision network takes the status information of the host, the slave and the user as input. The second decision module can also generate two different types of actions according to different working modes. The slave determines whether support is needed in combination with the temporary assistance domain of the host: if so, the slave is driven to the temporary assistance domain, and then the action of the auxiliary mode is executed; otherwise, the action of the active mode is executed. The present invention solves the problems of high difficulty in scheduling drones, great difficulty in improving communication coverage, and poor user experience in the existing multi-drone cooperative communication system.

Description

Technical Field

[0001] The present invention belongs to the field of UAV communication, and specifically relates to an optimization method for multi-UAV cooperative communication, a multi-UAV cooperative communication system, and an optimization device for multi-UAV cooperative communication. Background Art

[0002] In recent years, with the rapid development of aviation technology and automation technology, drones have been widely used in military, agriculture, transportation, public management and other fields due to their superior mobility, low cost and easy operation. By installing small communication equipment on drones, they can act as relay nodes to provide communication services for ground users. Especially in the field of emergency communications, drones are used as mobile base stations to provide temporary communications for ground users and provide real-time information on the spot for remote command. These applications make aerial drone communications of great research significance. However, in the face of widely distributed user needs, a single drone is often difficult to effectively meet the communication needs of ground users. Therefore, it is particularly important to carry out research on collaborative communications among multiple drones.

[0003] The existing literature "Path Planning for UAV-Mounted Mobile Edge Computing With Deep Reinforcement Learning" uses artificial neural networks to explore the relationship between scene information such as drone position, energy state and throughput; while the literature "Deep Reinforcement Learning for UAV Navigation Through Massive MIMO Technique" uses a convolutional neural network to model scene information as a tensor composed of multiple channels to make the scene feature hierarchy clearer; R. Zhong et al. proposed a multi-agent MDQN algorithm in "Multi-agent reinforcement learning in noma-aided UAV networks for cellular offloading" to learn such a strategy.

[0004] However, based on existing research, there are still some challenges: first, there is a lot of redundant exploration in the early stage of drone training, which causes waste of resources and time; second, the existing auxiliary behaviors of multiple drones are vaguely defined and need to rely on decision-making algorithms to explore on their own, which is more restricted and often difficult to achieve in open scenarios. Summary of the invention

[0005] In order to solve the problem of difficulty in drone scheduling in a multi-drone cooperative communication system in the prior art, the present invention provides an optimization method for multi-drone cooperative communication, a multi-drone cooperative communication system, and an optimization device for multi-drone cooperative communication.

[0006] The technical solution provided by the present invention is:

[0007] An optimization method for cooperative communication of multiple unmanned aerial vehicles, comprising:

[0008] The host and slave are specified in multiple communication drones. The flight trajectory of the host in the service range is optimized through a first decision network pre-trained based on the DDQN algorithm. The first decision network takes the status information of itself and the users in the service scene as input, combined with the preset total reward objective function R total , generates actions for controlling the flight of the host, which are executed by the host. The flight trajectory of the secondary aircraft in the service range is optimized through a second decision network pre-trained based on the DDQN algorithm. The second decision network takes the status information of the host, secondary aircraft and users in the service scene as input, combined with the preset collaborative reward function R 1 Generate a type of action for controlling the flight of the auxiliary aircraft in active mode; and combine it with the preset auxiliary reward function R 2 Generates the second type of action used to control the auxiliary aircraft flight in auxiliary mode.

[0009] The slave receives the first and second actions and makes the following decisions: (1) Generates a temporary assistance domain related to the real-time position of the host; (2) Determines whether assistance is required based on the preset criteria: If yes, drives the slave to reach the temporary assistance domain and then performs the second action; otherwise, performs the first action.

[0010] As a further improvement of the present invention, the scene modeling of the first decision network and the second decision network in the training phase includes:

[0011] (1) The service range is gridded and the location information of the service range is encoded into a matrix. The spatial locations of the host, slave, and user are represented by a two-dimensional position vector (W, H).

[0012] (2) The actions of the communication drone and the user are represented by motion vectors associated with nine directions: east, south, west, north, southeast, southwest, northeast, northwest, and stationary. u 1~ u 9 for characterization.

[0013] (3) The circular area with the communication drone as the center and the communication coverage distance D as the radius is regarded as the receptive field of the communication drone, and the communication demand of the users contained in the receptive field is regarded as the communication reward value of the communication drone. V0.

[0014] In addition, in the present invention, scene modeling may also include the following contents:

[0015] (4) Define the communication reward value generated by any user to decay over time to meet the timeliness constraint.

[0016] (5) Define the hovering time and motion power consumption of communication drones E 1 and communication power consumption E 2, satisfying the energy constraint.

[0017] As a further improvement of the present invention, the temporary assistance domain is a circular area with the host position as the center and the distance moved by the host performing a single action for a specified number of time steps as the radius.

[0018] As a further improvement of the present invention, the criterion for defining whether execution support is required is:

[0019] ,

[0020] in, W It represents the communication reward value of the overlapping part of its own receptive field and the temporary assistance field when the auxiliary machine calculates the estimated arrival time according to the direction vector; Refers to the average expected value of the communication reward calculated by the host; T A constant representing the preset condition trigger threshold.

[0021] As a further improvement of the present invention, the expression of the timeliness constraint is:

[0022] ,

[0023] In the above formula, V 0 represents the initial value of the communication reward corresponding to the user's communication demand. V t Indicates that the communication reward value corresponding to the user's communication demand is t The remaining value after the period; Represents the attenuation factor.

[0024] As a further improvement of the present invention, the training process of the first decision network is:

[0025] S1: Define the host's action space as A: A=( u 1, u 2,…, u 9) Establish the final total reward objective function within the task cycle R total :

[0026] ,

[0027] In the above formula, Indicates the end time of the task. Represents the discount factor used to balance the proportion of future rewards in the current accumulated rewards. R t represents the immediate reward function.

[0028] S2: Generate the host's state based on the host's receptive field s t , and input it into the decision network of the DDQN algorithm. The decision network generates the value matrix Q of all possible actions, and the host selects an action to execute based on the state transition probability P.

[0029] S3: Decision network combines the actions performed a t Calculating Reward Value r t , determine the new state s t+1 , and add a new experience Δ to the experience replay pool: Δ=( s t , a t , r t , s t+1 );

[0030] S4: Randomly sample multiple experiences from the experience replay pool for offline learning, and use the gradient descent method to update the weights of the decision network , and after every preset number of actions, update the parameters of the target network in the DDQN algorithm to finally obtain the optimal strategy.

[0031] As a further improvement of the present invention, in step S2, the calculation formula of the state transition probability is as follows:

[0032] ,

[0033] In the above formula, Represents the greedy coefficient in the action selection process; Q A value matrix representing all possible actions;

[0034] As a further improvement of the present invention, the loss function used in the offline learning process is for:

[0035] ,

[0036] The weight update formulas for the decision network and the target network are as follows:

[0037] ,

[0038] In the above formula, represents the function for obtaining the expectation; represents the discount factor of the reward; represents the learning rate; To accumulate rewards; Indicates execution of an action a t The next new state reached later; Indicates that in the next state Next, use the target network to select the network with the largest Q Actions of values; Represents the parameters of the target network; Represents the parameters of the decision network; the parameters of the target network are periodically copied from the parameters of the online network to calculate the target value, thereby avoiding Q Overestimation of value.

[0039] As a further improvement of the present invention, the collaborative reward function R 1 Using the total reward objective function R total .

[0040] Alternatively, a reward mechanism based on information entropy is introduced in the part of the second decision network used to generate a class of actions in the active mode. In this case, the collaborative reward function R 1 satisfies the following formula:

[0041] R 1= R total + R H ,

[0042] In the above formula, R H represents the auxiliary reward provided by the reward mechanism based on information entropy, which satisfies:

[0043] ,

[0044] In the above formula, H initial represents the initial target entropy value; P ij Indicates the service scope i Line j exploration probability of the grid of columns; N Represents the number of elements in the information entropy matrix; and Respectively R H The trade-off factor between the overall entropy difference term and the local entropy difference term contained in .

[0045] As a further improvement of the present invention, the auxiliary reward function R The expression of 2 is:

[0046] ,

[0047] In the above formula, t 1 indicates the estimated time for the host to traverse the temporary assistance domain; t 2 indicates the shortest time for the secondary aircraft to reach the temporary assistance area; Indicates the communication reward value of the host; Indicates the communication reward value of the slave machine; and are the collaborative weight factors of the host and slave respectively.

[0048] The present invention also includes a multi-UAV cooperative communication system, which includes: multiple communication UAVs, a first decision module and a second decision module.

[0049] Among them, a certain communication drone is designated as the host among multiple communication drones, and the others are slaves; each communication drone cruises in the service area according to the received motion control instructions, and then provides communication services to users within its own communication coverage distance.

[0050] The first decision module runs a first decision network pre-trained in the aforementioned optimization method for multi-UAV cooperative communication. The first decision module is used to obtain status information of the host and users in the service scene, and then generate instructions for controlling the flight of the host.

[0051] The second decision module runs a second decision network pre-trained in the aforementioned optimization method for cooperative communication of multiple UAVs. The second decision module is used to obtain the status information of the auxiliary aircraft, the host and the user in the service scene, and then generate the first action and the second action for controlling the flight of the auxiliary aircraft. The second decision module also includes a regional update unit and a state judgment unit. Among them, the regional update unit is used to generate the location information of the current temporary assistance domain according to the real-time location of the host. The state judgment unit is used to determine whether support is required according to a preset criterion. The first decision module is used to execute the following decision: when support is required, a control instruction to reach the temporary assistance domain is first sent to the auxiliary aircraft, and then a control instruction to execute the second type of action is sent. When no support is required, a control instruction to execute the first type of action is sent.

[0052] The present invention also includes an optimization device for cooperative communication of multiple UAVs, which includes a memory, a processor, and a computer program stored in the memory and running in the processor. When the processor executes the computer program, the optimization method for cooperative communication of multiple UAVs as described above is implemented to optimize the flight trajectory of each communicating UAV in the service range.

[0053] The technical solution provided by the present invention has the following beneficial effects:

[0054] The present invention provides a new strategy for scheduling communication drones in multi-drone collaboration scenarios. The new strategy of the present invention divides communication drones into two categories: host drones and slave drones. The host drone uses the DDQN algorithm to perform learnable flight actions in combination with actual user communication needs within the service scope. The slave drone plays different roles based on different situations in which the host drone needs support, and uses the DDQN algorithm to perform differentiated flight actions. When support is needed, the slave drone acts as the host drone's "wingman" to expand the host drone's communication service scope. When support is not needed, the slave drone and the host drone "work side by side", and the two together provide better communication services.

[0055] The present invention also models the tasks for the new communication drone scheduling strategy, and designs and trains two decision networks for executing the corresponding scheduling strategy through the DDQN algorithm, so as to realize the automatic coordinated control of each communication drone, thereby greatly improving the communication experience of users in the drone communication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a schematic diagram of the optimization method for multi-UAV cooperative communication provided in Example 1 of the present invention.

[0057] Figure 2 This is a topology diagram of a multi-UAV cooperative communication system using embedded modules in Example 2 of the present invention.

[0058] Figure 3 This is a topology diagram of a multi-UAV cooperative communication system using a cloud server in Example 2 of the present invention.

[0059] Figure 4 This is a topology diagram of a multi-UAV collaborative communication system using an edge server in Example 2 of the present invention.

[0060] Figure 5 This is the curve showing the total reward obtained by the drone communication system in the test experiment as the number of training rounds changes.

[0061] Figure 6 This is the curve showing how the average number of flight steps required for the secondary UAV to complete the task of collecting the same amount of communication information changes with the number of training rounds in the test experiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] Example 1

[0064] This embodiment provides an optimization method for multi-UAV cooperative communication. In the solution of this embodiment, Figure 1 As shown in the figure, communication drones are divided into two categories: one is the main communication drone (hereinafter referred to as the host), and the other is the auxiliary communication drone (hereinafter referred to as the auxiliary drone). The role of the host is to cruise within the specified service range according to a learnable strategy to meet the dynamic changing needs of the users below and provide corresponding communication services. Among them, the flight path of the host is only affected by its own and the user's communication needs, so as to maximize the satisfaction of user needs.

[0065] The slave has two working modes, one can be called active mode and the other can be called auxiliary mode. In the active mode, the slave and the host have the same functions, and the slave also cruises within the specified service range according to a learnable strategy to meet the dynamic changing needs of the users below and provide corresponding communication services. However, unlike the host, in the learnable strategy of the slave, the slave is not only affected by its own and the user's communication needs, but also by the host running at the same time. That is: the flight path and communication coverage of the slave should try to avoid overlapping with the host to achieve a higher communication coverage rate. Therefore, considering the influence of the host on the slave, if the host is defined as a fully autonomous (not interfered with by the operation of other communication drones) communication drone, then in the active mode, the slave is a semi-autonomous (affected by the host) communication drone.

[0066] When the communication coverage provided by the host is insufficient to meet the user's needs, the slave in this embodiment enters the auxiliary mode. In the auxiliary mode, the host no longer works as an independent communication drone, but as the host's "wingman". At this time, the slave will fly with the host to extend the coverage of the communication service provided by the host; thereby achieving full coverage of the user's communication needs.

[0067] Specifically, the optimization method for multi-UAV cooperative communication provided in this embodiment is implemented in the following manner:

[0068] 1. Role Allocation

[0069] The solution of this embodiment first needs to specify the host and slave among multiple communication drones. In practical applications, the solution can be applied to scenarios with only two communication drones at the minimum. In this scenario, one of the communication drones is used as the host and the other is used as the slave. When the solution is applied to a scenario involving two communication drones, the allocation method of the host and slave can be flexibly designed according to actual scenario elements such as service scope and demand distribution. For example, one of the communication drones is selected as the host, and the rest of the communication drones are all used as slaves. When the host needs communication assistance, a relatively close communication drone is selected from the many slaves to perform the accompanying mission to provide auxiliary communication. Alternatively, the entire communication service range is divided into sections, and then two communication drones are configured for each section. The two communication drones only provide services in the sections they are responsible for. In their respective sections, one of the two communication drones is used as the host and the other as the slave. The two work according to the dual drone collaborative communication strategy introduced above to provide communication services to users in the corresponding sections.

[0070] In addition, it should be emphasized that in the actual application of the scheme of this embodiment, each communication drone can flexibly switch between working as a host or a slave according to the task requirements; the identity of the host and the slave is not an inherent attribute of any communication drone. For example, in a mission cycle, drone No. 1 can be used as the host and drone No. 2 can be used as the slave. In the next mission cycle, drone No. 2 can be used as the slave and drone No. 1 can be used as the host.

[0071] 2. Scheduling and Execution

[0072] The above describes the role allocation of each drone in the multi-communication drone scenario and the task content designed for each role of the communication drone in this embodiment. In order to enable each drone to perform the corresponding task according to the designated role, thereby providing better communication coverage and services; this embodiment also designs a corresponding scheduling system for various communication drones in the multi-drone cooperative communication system. The scheduling system includes two parts, one for controlling the host communication drone and the other for controlling the auxiliary communication drone.

[0073] 2.1 Host Scheduling Strategy

[0074] The host's task is to cruise within a specified range according to the real-time user communication needs to ensure that the customer's communication needs can be met. Therefore, the scheduling of the host is essentially to adjust the flight path of the host in combination with the predicted future communication needs of the user. For this problem, although the user's communication needs are unpredictable, in the larger time and space dimensions, the user's needs actually have certain statistical laws. In view of this feature, this embodiment uses the dual deep reinforcement learning model (DDQN) as a tool, based on historical status data such as user communication needs and the host's operation process in the service area, to train a decision network that can output the host's ideal flight action based on user needs and the host's real-time status.

[0075] In the technical solution provided in this embodiment, the flight trajectory of the host in the service range is optimized by a first decision network pre-trained based on the DDQN algorithm. The first decision network takes the status information of itself and the users in the service scene as input, combined with the preset total reward objective function R total , generates actions for controlling the host's flight and is executed by the host.

[0076] In order to make the DDQN algorithm applicable to processing the scheduling problem of the host, this embodiment performs scenario modeling on the scheduling problem targeted by the first decision network in the training phase, including:

[0077] (1) The service range is gridded and the location information of the service range is encoded into a matrix. The spatial locations of the host, slave, and user are represented by a two-dimensional position vector (W, H).

[0078] In this embodiment, the drone flies at a fixed altitude within the service range. h The communication drones cruise in the sky, while users with communication needs move on the ground. Therefore, in practical applications, the communication drones and users will move on horizontal planes at different heights. In order to more concisely represent the spatial positions of the communication drones and users, this embodiment grids the service range (corresponding to the working scene of the communication drone). Combined with the large number of equidistant grids divided, the location information of the service range can be encoded in the form of a matrix. In this state, the communication drone flies between the equidistant grids of W*H. Ground mobile users are randomly distributed at the center of each grid.

[0079] (2) The actions of the communication drone and the user are represented by motion vectors associated with nine directions: east, south, west, north, southeast, southwest, northeast, northwest, and stationary. u 1~ u 9 for characterization.

[0080] In this embodiment, the communication drone and the user have two different states, namely, motion and stillness. When modeling the scene in this embodiment, it is defined that the communication drone and the user both move at a constant speed according to their respective typical speeds. In their respective planes, the movement directions of the user and the communication drone can be divided into 8 types, namely, due east, due south, due west, due north, southeast, southwest, northeast, and northwest. In combination with different states and movement directions, this embodiment characterizes the movement trends of users and communication drones through 9 motion vectors. Among them, the nine motion vectors of due east, due south, due west, due north, southeast, southwest, northeast, northwest and stillness are recorded as: u 1~ u 9. On this basis, the optimization of the flight trajectory of the communication UAV is essentially to generate corresponding motion vectors for controlling the flight posture of the communication UAV at different times.

[0081] (3) The communication demand of users included in the receptive field is used as the communication reward value of the communication drone V 0.

[0082] The idea of ​​the DDQN algorithm is to evaluate the benefits generated by the selected action through the reward values ​​of different actions, and then drive the target to execute the action with the highest benefit. In the scenario of drone communication in this embodiment, the essence of the optimization problem is to meet the communication needs of users. Therefore, in this embodiment, the communication needs of users satisfied by the communication services provided by drones are used as the communication reward value in the DDQN algorithm. On this basis, considering that the communication coverage range that each communication drone can provide is limited, this embodiment regards the circular area with the communication coverage distance D as the radius, centered on the communication drone, as the receptive field of the communication drone; the drone can provide communication services to ground mobile users within the receptive field. When a user who proposes a communication demand is located in the current receptive field of any communication drone, it means that the communication demand of the user at the current moment is met. On the contrary, when a user who proposes a communication demand is outside the current receptive field of all communication drones, it means that the communication demand of the user at the current moment is not met.

[0083] In addition, in order to achieve a more detailed mathematical description of the optimization problem, the scene modeling in this embodiment may also include the following contents:

[0084] (4) Define the communication reward value generated by any user V 0 decays over time, satisfying the timeliness constraint.

[0085] The timeliness constraint describes the changes in the user experience of the user's communication demand over time. As we all know, any communication demand issued by a user should be met immediately, and in the worst case, it should be met after a specified delay after the user's communication demand is issued. When the user waits too long, the user's communication demand may disappear, and the user's experience will be the worst. Therefore, in this embodiment, the timeliness constraint of the communication reward value corresponding to the user's communication demand in each grid is represented by the following expression:

[0086] ,

[0087] In the above formula, V 0 represents the initial value of the communication reward corresponding to the user's communication demand. V t Indicates that the communication reward value corresponding to the user's communication demand is t The remaining value after the period; Represents the attenuation factor. Combining the above expressions, we can see that considering the timeliness of information, as the time length increases, when the receptive field of the communication drone covers the user, the communication reward value it obtains is gradually decreasing.

[0088] (5) Define the hovering time and motion power consumption of communication drones E 1 and communication power consumption E 2, satisfying the energy constraint.

[0089] When a communication drone flies in the sky to provide communication services to users on the ground, its lag time is not unlimited. Generally speaking, the hovering time of a communication drone is restricted by two factors: its own energy supply and comprehensive energy consumption. Among them, considering that the energy supply (battery or fuel tank, etc.) carried by each communication drone is fixed, the hovering time of the communication drone is only restricted by the comprehensive energy consumption. In this embodiment, the comprehensive energy consumption is further divided into two aspects: one is motion power consumption. E 1. The other is communication power consumption E 2. Among them, communication drones usually cruise at a specified altitude, so the motion power consumption E 1 usually remains unchanged and can be characterized by a fixed value. The communication power consumption is closely related to the communication needs of ground users. The correlation between the two is as follows:

[0090] ,

[0091] In the above formula, V r Indicates the rate at which the communication drone transmits signals during the communication process; e tr Indicates the transmission efficiency of the communication process; cIndicates the bandwidth limitation during the communication between the communication UAV and the ground user; d It represents the communication distance between the UAV and the ground user, and its calculation formula is as follows:

[0092] ,

[0093] In the above formula, h Indicates the height of the drone from the horizontal ground. express t The horizontal coordinate projection of the drone in the scene at this moment, express t The horizontal coordinate of the ground user in the scene at that moment, express t The vertical coordinate projection of the drone in the scene at this moment, express t The vertical coordinate of the ground user in the scene at this moment.

[0094] In detail, the dual deep reinforcement learning model is a classic reinforcement learning algorithm, in which two identical networks are introduced, one is the action selection network (decision network) and the other is the target network. The drone uses the action selection network to predict the Q value of each possible action in the scene corresponding to different times, and selects the action with the highest Q value. After executing the action, the drone will observe new scene information and reward values. The target network is then used to evaluate the Q value of each action in this new scene state, and the largest Q value is selected as the best estimate of this action. Then, this largest Q value is used to update the parameters of the action selection network, but the parameters of the target network are not updated at each step. Instead, after multiple actions of a specified duration, the parameters of the action selection network are copied to the target network to make the estimate of the target network more stable.

[0095] Specifically, the process of obtaining the required first decision network by training the DDQN algorithm in this embodiment is roughly as follows:

[0096] S1: Define the host's action space as A:

[0097] A=( u 1, u 2,…, u 9);

[0098] in, u 1~ u 9 correspond to 9 different actions of the drone flying towards the east, south, west, north, southeast, southwest, northeast, northwest, and remaining stationary.

[0099] Establish the final total reward objective function within the task cycle R total :

[0100] ,

[0101] In the above formula, Indicates the end time of the task. Represents the discount factor used to balance the proportion of future rewards in the current accumulated rewards. The discount factor is used to adjust the weight of future rewards in the current accumulated rewards. R t Indicates an immediate reward. In the algorithm of this embodiment, after the communication drone performs an action, it will obtain a new state and a corresponding reward value.

[0102] Among them, instant rewards R t express t The drone and its sensing range d The total transmission rate between ground users is The difference is expressed as follows:

[0103] ,

[0104] In the above formula, Indicates the communication transmission rate between the drone and the user, The power consumption of the UAV in period t is expressed as follows:

[0105] ,

[0106] In the above formula, p f Indicates the flight power of the drone; v uav Indicates the flight speed of the drone; express t The projection coordinates of the drone on the ground at this moment; express t -1 The projection coordinates of the drone on the ground at time

[0107] in, The expression is as follows:

[0108] ,

[0109] Where: W e It represents the bandwidth allocated by the UAV to each ground communication user; SNR is the signal-to-noise ratio.

[0110] S2: Generate the host's state based on the host's receptive field s t, and input it into the decision network of the DDQN algorithm. The decision network generates the value matrix Q of all possible actions, and the host selects an action to execute based on the state transition probability P.

[0111] Specifically, the status information of the service scenarios including the user is collected, and the status of the host is obtained by combining the host receptive field. s t , s t Indicates the current t The scene observation information at the moment. The scene information and the host receptive field are used as the input of the decision network. The decision network outputs the value matrix Q of all possible actions of the drone, and selects the action to be executed according to the state transition probability P. Its mathematical expression is:

[0112] ,

[0113] In the above formula, Represents the greedy coefficient in the action selection process; s t+1 Indicates that the drone is performing an action a t The scene observation information of the next moment after ; a t Represents the action taken by the drone at time t.

[0114] The host transfers according to the state probability P After the action is selected, the scene information and reward value change, and the optimal strategy is updated accordingly. , its mathematical expression is as follows:

[0115] ,

[0116] S3: Decision network combines the actions performed a t Calculating Reward Value r t , determine the new state s t+1 , and add a new experience Δ to the experience replay pool: Δ=( s t , a t , r t , s t+1 ).

[0117] The DQNN algorithm used in this embodiment establishes an experience playback pool to store the experience value of each state change of the communication drone. The data in each experience value includes: the current state information of the host s t, the selected action a t , the reward value of the currently executed action r t , the new state information obtained after executing the action s t+1 .

[0118] S4: Randomly sample multiple (corresponding to the number of batches) experiences from the experience replay pool for offline learning, and use the gradient descent method to update the weights of the decision network , and after every preset number of actions, update the parameters of the target network in the DDQN algorithm to finally obtain the optimal strategy.

[0119] In this embodiment, the loss function used in the offline learning process is for:

[0120] ,

[0121] The weight update formula for the decision network and target network in the DQN algorithm is as follows:

[0122] ,

[0123] In the above formula, represents the function for obtaining the expectation; represents the discount factor of the reward; represents the learning rate; To accumulate rewards; Indicates execution of an action a t The next new state reached later; Indicates that in the next state Next, use the target network to select the network with the largest Q Actions of values; Represents the parameters of the target network; Represents the parameters of the decision network; the parameters of the target network are periodically copied from the parameters of the online network to calculate the target value, thereby avoiding Q Overestimation of value.

[0124] The above content introduces in detail the key information such as the action, state, reward, loss function, weight update formula, etc. defined when the DDQN algorithm is used to train the first decision network in this embodiment. For other non-core technical details included in the DDQN algorithm in the training stage, this embodiment will not go into details.

[0125] 2.2. Scheduling strategy of auxiliary aircraft

[0126] The previous article introduced the content of the first decision network obtained by pre-training the DDQN network, which can output the flight actions that the host should take in the future according to the status of the user and the host. In this implementation, based on the same idea. The flight trajectory of the auxiliary aircraft in the service range can also be optimized through a second decision network pre-trained based on the DDQN algorithm. Among them, unlike the host, which can be completely autonomous when making decisions on flight actions, the flight action instructions generated by the auxiliary aircraft using the second decision network need to take into account the status of the host that already exists in the scene, thereby avoiding collision or competition between the two. Therefore, the second decision network in this embodiment takes the status information of the host, auxiliary aircraft and users in the service scene as input. In addition, considering that the auxiliary aircraft in this embodiment needs to choose different working modes according to the actual situation during actual operation. Therefore, the second decision unit in the auxiliary aircraft should be combined with the preset collaborative reward function during the training and application stages. R 1 Generate a type of action for controlling the flight of the auxiliary aircraft in active mode; and combine it with the preset auxiliary reward function R 2 Generates the second type of action used to control the auxiliary aircraft flight in auxiliary mode.

[0127] In summary, the second decision network in this embodiment actually uses the same historical state data to train the DDQN algorithm through different methods to obtain two network units. One of them is designed and trained according to the working scenarios and requirements of the active mode, and it can generate the flight actions that the auxiliary aircraft should adopt in the active mode. The other is designed and trained according to the working scenarios and requirements in the auxiliary mode, and it can generate the flight actions that the auxiliary model auxiliary aircraft should adopt. The auxiliary aircraft can select and execute the corresponding flight actions according to its own working mode. That is:

[0128] The slave receives the first and second actions and makes the following decisions: (1) Generates a temporary assistance domain related to the real-time position of the host; (2) Determines whether assistance is required based on the preset criteria: If yes, drives the slave to reach the temporary assistance domain and then performs the second action; otherwise, performs the first action.

[0129] (2.2.1) Switching of working modes

[0130] In the scheme designed in this embodiment, the auxiliary machine has two working modes. In the two working modes, the role and execution actions of the auxiliary machine are different. Therefore, it is necessary to first determine the switching strategy of the auxiliary machine in different working modes. Specifically, in this embodiment, the auxiliary machine adopts the active mode to carry out tasks under most conditions, and only when the host triggers the need for the auxiliary machine to provide communication assistance services during work, the auxiliary machine will be required to switch to the auxiliary mode and assist the host. Correspondingly, when the host no longer meets the triggering conditions for the auxiliary machine to provide communication assistance services, the auxiliary machine exits the auxiliary mode and switches back to the active mode. That is: the switching of the auxiliary machine between the two working modes mainly depends on whether the task of the host triggers the conditions for the auxiliary machine to provide communication assistance services.

[0131] Before introducing the triggering conditions of different working modes of the slave device designed in this embodiment, the concept of temporary assistance domain is first introduced. In this embodiment, temporary assistance domain O is a domain centered on the host location and the host performs a single action for a specified number of consecutive times (such as ) time step moves a circular area with a radius of . That is, the size of the temporary assistance domain is Among them, the temporary assistance domain is an area whose size exceeds the service coverage of a single communication drone.

[0132] On this basis, this embodiment defines the criteria for whether the host needs to perform support (i.e., the triggering condition for the secondary machine mode switching) as:

[0133] ,

[0134] in, W It represents the communication reward value of the overlapping part of its own receptive field and the temporary assistance field when the auxiliary machine calculates the estimated arrival time according to the direction vector; Refers to the average expected value of the communication reward calculated by the host; T A constant representing the preset condition trigger threshold.

[0135] (2.2.2) Active mode

[0136] In the active mode of the slave, the second decision unit outputs the flight action that the slave should take, i.e., the second action, based on the input status information of the host, slave, and users in the service scene. Like the first decision unit, the second decision unit is also pre-trained by the DDQN algorithm. In the active mode part of the second decision unit, the collaborative reward function of the DDQN algorithm is set. R 1 You can also use the total reward objective function R total .

[0137] In addition, in a further optimized solution of this embodiment, in order to reduce the energy consumption caused by the blind exploration of the auxiliary machine in the early stage, a reward mechanism based on information entropy can be introduced in the part of the second decision network used to generate a type of action in the active mode. The reward mechanism based on information entropy calculates the degree of exploration of the auxiliary machine to the environment and gives positive rewards. Among them, due to the presence of a large number of unexplored grids in the early stage, the information entropy is basically a negative reward. When a large number of grids are explored, the system obtains more positive rewards. The overall trend is that as the degree of exploration increases, the negative rewards become smaller and smaller, and the rewards begin to tend to be positive. Finally, the optimized reward function is used to encourage the auxiliary machine to explore more comprehensively.

[0138] After introducing the reward mechanism based on information entropy, the collaborative reward function R1 satisfies the following formula:

[0139] R1= R total + R H ,

[0140] In the above formula, R H Represents the auxiliary reward provided by the reward mechanism based on information entropy. The following is a detailed introduction to the reward mechanism based on information entropy introduced in this embodiment:

[0141] S01: Initialize a trajectory matrix, which is a two-dimensional matrix of size W*H, used to record the exploration trajectory of the auxiliary aircraft. According to the perception field of the auxiliary aircraft, the equidistant grids within the perception range are counted into the exploration degree. T ij ,in, i and j Represents the row and column numbers of the grid respectively.

[0142] S02: Setting a threshold for exploration K , exceeding the threshold K , it is considered that the depth exploration is completed. After the scene information exploration is completed, the exploration degree of each grid in the previous step is normalized to obtain the probability of exploring the current grid. P ij , its mathematical expression is as follows:

[0143] ,

[0144] S03: Combine the probability obtained in the previous step P ij , the information entropy of the current grid can be calculated H ij , and then the overall information entropy matrix H is obtained. Among them, information entropy H ij The calculation formula is:

[0145] ,

[0146] After the above processing, the exploration degree of the secondary machine to the current environment is converted into a specific information entropy value, and an entropy matrix with the same shape as the initial trajectory matrix is ​​obtained. This entropy value indicates the value of the grid exploration. If the exploration degree is full, the entropy value of the area is 0; the larger the entropy value of the unexplored area, the easier it is to give the secondary machine rewards through entropy value to encourage exploration.

[0147] S03: Considering that the entropy value of a single grid has limited influence, it is necessary to calculate the global total entropy value H total The mathematical expression is as follows:

[0148] S04: In order to avoid overly average exploration and strengthen the focus on under-explored areas, it is necessary to balance the relationship between global exploration and local in-depth exploration, so the local entropy difference term is introduced D To optimize the reward distribution and encourage diversified exploration, the mathematical expression is as follows:

[0149] ,

[0150] In the above formula, N Represents the information entropy matrix H The number of elements in .

[0151] S05: Calculate the reduction in the overall entropy value to directly reflect the macroscopic progress of the exploration. It is necessary to calculate the difference between the target entropy value and the overall entropy, that is, the overall entropy difference term Δ H , its mathematical expression is as follows:

[0152] Δ H = H initial - H total ,

[0153] In the above formula, H initial represents the initial target entropy value. In this embodiment, H initial =1.0.

[0154] S06: Combine the local entropy difference term of the previous steps D and the overall entropy difference term Δ H , we can get the overall entropy reward function R H , its mathematical expression is as follows:

[0155] ,

[0156] In the above formula, and The overall entropy difference term Δ H and the local entropy difference D The trade-off factor of , both of which are between (0, 1), is used to weigh the overall entropy difference term Δ H and the local entropy difference D The impact of the final complete overall entropy reward function R H It can be expressed as:

[0157] .

[0158] final, R H Auxiliary reward provided as an introduction to the information entropy-based reward mechanism.

[0159] (2.2.3) Assisted mode

[0160] In this embodiment, compared with the active mode, the slave in the auxiliary mode stops the currently executing flight action, obtains the position of the host, and immediately flies to the temporary assistance domain of the host, and then flies according to the first action output by the second decision network.

[0161] Specifically, during the training phase of the second decision network, the auxiliary reward function used in the auxiliary mode-related part is R The design process of 2 is as follows:

[0162] S001: Constructing temporary assistance domain:

[0163] Using the host receptive field and current scene information obtained by the secondary machine, the temporary assistance domain of the host at the current moment is planned. The size of the temporary assistance domain is .

[0164] S002: Calculate the expected value of the host's average communication reward :

[0165] According to the above steps, the host's receptive field size is , calculate the estimated time for the host to traverse the temporary assistance area t 1 satisfies the following formula:

[0166] .

[0167] Assume that the number of terrestrial mobile users in the temporary assistance area is n , the total communication reward value is V total , then the average communication reward value obtained by the host per time step is for:

[0168] ,

[0169] According to the time constraint of the decay of the communication reward value of ground mobile users introduced above, the expected value of the average communication reward obtained by the host in a unit time step after independently exploring the temporary assistance domain is calculated to be The expression is as follows:

[0170] .

[0171] S003: Calculate the fastest time required for the secondary aircraft to reach the temporary assistance area t 2:

[0172] Assume that the auxiliary engine speed is v , then the fastest time required for the secondary aircraft to reach the temporary assistance area is t 2: t 2=L / v , L represents the Euclidean distance between the master and slave.

[0173] S004: Calculate the expected average communication reward value of the slave machine :

[0174] Time from the secondary aircraft to the temporary assistance area t 2Start until the host completes the task t 1, calculate the communication bonus value of the slave , its mathematical expression is as follows:

[0175] ,

[0176] In the above formula, Includes slave t The total communication reward value obtained by the auxiliary aircraft from the time the temporary assistance area mission is completed from 2:00 onwards.

[0177] Then the average communication reward value of the slave is expected to be for:

[0178] ,

[0179] S005: Constructing assistance reward function R 2:

[0180] ,

[0181] In the above formula, and is the collaborative weight factor, which takes a value between (0, 1) and is used to balance the communication reward value between the primary and secondary machines. The complete expression of the assistance reward function is finally obtained as follows:

[0182] .

[0183] In summary, in the scheme designed in this embodiment, by establishing trigger conditions and temporary assistance domains, the slave can determine whether the host currently meets the assistance conditions: if so, the slave will no longer fly according to the original action within the mission time, but will reach the temporary assistance domain by the shortest route to assist the host in performing the communication task. In the second decision network used by the slave, by setting a differentiated reward function, the actions generated by the second decision network can be made more realistic, encouraging the slave to actively assist the host to complete the communication task more efficiently.

[0184] Example 2

[0185] Based on the solution of Example 1, this embodiment further provides a multi-UAV cooperative communication system, which includes: multiple communication UAVs, a first decision module and a second decision module.

[0186] In a multi-UAV cooperative communication system, one of the multiple communication UAVs is designated as the host, and the others are slaves; each communication UAV cruises in the service area according to the received motion control instructions, and then provides communication services to users within its own communication coverage distance.

[0187] The first decision module runs a first decision network pre-trained in the optimization method for multi-UAV cooperative communication in Example 1. The first decision module is used to obtain status information of the host and users in the service scene, and then generate instructions for controlling the flight of the host.

[0188] The second decision module runs a second decision network pre-trained in the aforementioned optimization method for cooperative communication of multiple UAVs. The second decision module is used to obtain the status information of the auxiliary aircraft, the host and the user in the service scene, and then generate the first action and the second action for controlling the flight of the auxiliary aircraft. The second decision module also includes a regional update unit and a state judgment unit. Among them, the regional update unit is used to generate the current location information of the temporary assistance domain according to the real-time location of the host. The state judgment unit is used to determine whether support is required according to a preset criterion. The first decision module is used to execute the following decision: when support is required, a control instruction to reach the temporary assistance domain is first sent to the auxiliary aircraft, and then a control instruction to execute the second type of action is sent. When no support is required, a control instruction to execute the first type of action is sent.

[0189] It should be noted that in actual applications, the first decision module and the second decision module are deployed in a variety of ways. Figure 2 In the scheme 1 shown, the first decision module and the second decision module can be deployed in the host and the slave respectively, and the host and the slave can also communicate with each other to achieve mutual transmission of each other's status data. Figure 3In the scheme 2 shown, the first decision module and the second decision module are deployed in a cloud service and serve as a scheduling system running in the cloud server. The host and the slave communicate with the cloud server respectively, and the cloud server obtains the status information of the two and executes the corresponding scheduling strategy. Figure 4 In the scheme 2 shown, multiple edge servers are deployed in the service areas of the host and the slave. When the two arrive at different areas, they interact with the critical edge servers respectively. The first decision module and the second decision module are deployed in each edge server respectively. The edge server obtains the status information of the host and the slave and executes the corresponding scheduling strategy.

[0190] Example 3

[0191] This embodiment provides an optimization device for cooperative communication of multiple UAVs, which includes a memory, a processor, and a computer program stored in the memory and running in the processor. When the processor executes the computer program, the optimization method for cooperative communication of multiple UAVs as described above is implemented to optimize the flight trajectory of each communicating UAV in the service range.

[0192] The optimization device for cooperative communication of multiple UAVs provided in this embodiment is essentially a computer device for implementing the solution in Embodiment 1. The computer device may be an embedded module, or an intelligent terminal capable of executing a program, a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc.

[0193] The computer device indicated in this embodiment includes at least but is not limited to: a memory and a processor that can be connected to each other through a system bus. Among them, the memory (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.

[0194] The processor may be a central processing unit (CPU), a graphics processing unit (GPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor is generally used to control the overall operation of a computer device. In this embodiment, the processor is used to run program codes stored in a memory or process data.

[0195] Simulation test

[0196] In order to verify the effect of the optimization scheme for multi-UAV cooperative communication provided by the present invention, the technicians formulated an experimental plan and trained and tested the method based on a self-built two-dimensional simulation scene. The total reward obtained by the UAV communication system and the average number of flight steps and training rounds required by the secondary UAV to complete the same communication information collection task are shown in the following figure. Figure 5 and Figure 6 shown.

[0197] analyze Figure 5 It can be seen that as the number of training rounds increases, the rewards obtained by the secondary drone fluctuate over a large range in the early stages of training, then gradually increase and finally remain between 1290-1340. During the training process, its reward value gradually increases and finally converges. Figure 6It can be seen that as the number of training rounds increases, the number of steps required for the drone to complete the communication task gradually decreases, and finally remains between 140 and 180. The above experimental results show that the secondary drone can still maintain the completion of the ground communication task while performing the auxiliary task.

[0198] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for optimizing cooperative communication of multiple UAVs, characterized in that: It includes: Specify the master and slave among multiple communicating drones; The flight trajectory of the host in the service range is optimized through a first decision network pre-trained based on the DDQN algorithm; the first decision network takes the status information of itself and the users in the service scene as input, combined with the preset total reward objective function R total , generating actions for controlling the host's flight, which are then executed by the host; The flight trajectory of the secondary aircraft in the service range is optimized through a second decision network pre-trained based on the DDQN algorithm; the second decision network takes the status information of the host, secondary aircraft and users in the service scene as input, combined with the preset collaborative reward function R 1 Generate a type of action for controlling the flight of the auxiliary aircraft in active mode; and combine it with the preset auxiliary reward function R 2 Generate the second type of action for controlling the auxiliary aircraft in auxiliary mode; The slave receives the first and second actions and makes the following decisions: (1) Generates a temporary assistance domain related to the real-time position of the host; (2) Determines whether assistance is required based on the preset criteria: If yes, drives the slave to reach the temporary assistance domain and then performs the second action; otherwise, performs the first action.

2. The optimization method for multi-UAV cooperative communication according to claim 1, characterized in that: The scenario modeling of the first decision network and the second decision network in the training phase includes: The service range is gridded, and the location information of the service range is encoded into a matrix. The spatial locations of the host, slave, and user are represented by a two-dimensional position vector (W, H); The actions of the communication drone and the user are connected to the nine directions of motion vectors: east, south, west, north, southeast, southwest, northeast, northwest, and stationary. u 1~ u 9 for characterization; The circular area with the communication drone as the center and the communication coverage distance D as the radius is regarded as the receptive field of the communication drone, and the communication demand of the users contained in the receptive field is regarded as the communication reward value of the communication drone. V 0; And / or, define the communication reward value generated by any user to decay over time to meet the timeliness constraint; And / or, define the hovering time and motion power consumption of the communication drone E 1 and communication power consumption E 2, satisfying the energy constraint.

3. The optimization method for multi-UAV cooperative communication according to claim 2, characterized in that: The temporary assistance domain is a circular area with the host position as the center and the distance moved by the host performing a single action for a specified number of time steps as the radius; and / or The criteria for defining whether execution support is needed are: ， in, W It represents the communication reward value of the overlapping part of its own receptive field and the temporary assistance field when the auxiliary machine calculates the estimated arrival time according to the direction vector; Refers to the average expected value of the communication reward calculated by the host; T A constant representing the preset condition trigger threshold.

4. The optimization method for multi-UAV cooperative communication according to claim 2, characterized in that: The expression of the timeliness constraint is: ， In the above formula, V 0 represents the initial value of the communication reward corresponding to the user's communication demand. V t Indicates that the communication reward value corresponding to the user's communication demand is t The remaining value after the period; Represents the attenuation factor.

5. The optimization method for multi-UAV cooperative communication according to claim 4, characterized in that: The training process of the first decision network is: S1: Define the host's action space as A: A=( u 1, u 2,…, u 9) Establish the final total reward objective function within the task cycle R total : ， In the above formula, Indicates the end time of the task. Represents the discount factor used to balance the proportion of future rewards in the current accumulated rewards. R t represents the immediate reward function; S2: Generate the host's state based on the host's receptive field s t , input it into the decision network of the DDQN algorithm, the decision network generates the value matrix Q of all possible actions, and the host selects an action to execute based on the state transition probability P; S3: Decision network combines the actions performed a t Calculating Reward Value r t , determine the new state s t+1 , and add a new experience Δ to the experience replay pool: Δ=( s t , a t , r t , s t+1 ); S4: Randomly sample multiple experiences from the experience replay pool for offline learning, and use the gradient descent method to update the weights of the decision network , and after every preset number of actions, update the parameters of the target network in the DDQN algorithm to finally obtain the optimal strategy.

6. The optimization method for multi-UAV cooperative communication according to claim 5, characterized in that: In step S2, the calculation formula of the state transition probability is as follows: ， In the above formula, Represents the greedy coefficient in the action selection process; Q A value matrix representing all possible actions; and / or Loss function used in offline learning process for: ， The weight update formulas for the decision network and the target network are as follows: ， In the above formula, represents the function for obtaining the expectation; represents the discount factor of the reward; represents the learning rate; To accumulate rewards; Indicates execution of an action a t The next new state reached later; Indicates that in the next state Next, use the target network to select the network with the largest Q Actions of values; Represents the parameters of the target network; Represents the parameters of the decision network.

7. The method for optimizing multi-UAV cooperative communication according to claim 5, characterized in that: The collaborative reward function R 1 Using the total reward objective function R total ; or, In the second decision network, a reward mechanism based on information entropy is introduced for generating a class of actions in the active mode; The collaborative reward function R 1 satisfies the following formula: R 1= R total + R H ， In the above formula, R H represents the auxiliary reward provided by the reward mechanism based on information entropy, which satisfies: ， In the above formula, H initial represents the initial target entropy value; P ij Indicates the service scope i Line j exploration probability of the grid of columns; N Represents the number of elements in the information entropy matrix; and Respectively R H The trade-off factor between the overall entropy difference term and the local entropy difference term contained in .

8. The method for optimizing multi-UAV cooperative communication according to claim 5, characterized in that: The auxiliary reward function R The expression of 2 is: ， In the above formula, t 1 indicates the estimated time for the host to traverse the temporary assistance domain; t 2 represents the shortest time for the secondary aircraft to reach the temporary assistance area; Indicates the communication reward value of the host; Indicates the communication reward value of the slave machine; and are the collaborative weight factors of the host and slave respectively.

9. A multi-UAV cooperative communication system, characterized in that: It includes: Multiple communication drones, one of which is designated as the main drone and the others as slave drones; each communication drone cruises in the service area according to the received motion control instructions, and then provides communication services to users within its own communication coverage distance; A first decision module, in which a first decision network pre-trained in the optimization method for multi-UAV cooperative communication according to any one of claims 1 to 8 is run; the first decision module is used to obtain status information of the host and users in the service scene, and then generate instructions for controlling the flight of the host; A second decision module, in which a second decision network pre-trained in the optimization method for multi-UAV cooperative communication as described in any one of claims 1 to 8 runs; the second decision module is used to obtain the status information of itself and the host and users in the service scene, and then generate the first action and the second action for controlling the flight of the auxiliary aircraft; the second decision module also includes a region update unit and a state judgment unit; the region update unit is used to generate the location information of the current temporary assistance domain according to the real-time position of the host; the state judgment unit is used to judge whether support is needed according to a preset criterion; the first decision module is used to execute the following decisions: when support is needed, a control instruction to reach the temporary assistance domain is first sent to the auxiliary aircraft, and then a control instruction to execute the second type of action is sent; when no support is required, a control instruction to execute the first type of action is sent.

10. An optimization device for cooperative communication of multiple unmanned aerial vehicles, comprising a memory, a processor, and a computer program stored in the memory and running in the processor, characterized in that: When the processor executes the computer program, it implements the optimization method for multi-UAV cooperative communication as described in any one of claims 1 to 8, and optimizes the flight trajectory of each communicating UAV within the service range.

Images