Method, system and device for optimizing cooperative communication of multiple unmanned aerial vehicles

Abstract

The invention belongs to the field of unmanned aerial vehicle communication, and particularly relates to a multi-unmanned aerial vehicle cooperative communication optimization method, system and device. The method comprises the following steps: specifying a main engine and an auxiliary engine in a plurality of communication unmanned aerial vehicles; and optimizing the flight paths of the main engine and the auxiliary engine through a first decision network and a second decision network pre-trained based on a DDQN algorithm. Wherein the first decision network takes the state information of the host and the user as input; and the second decision network takes the state information of the host, the auxiliary machine and the user as input. The second decision-making module can also generate two different actions according to different working modes. The auxiliary machine is combined with the temporary assistance domain of the main machine to judge whether support needs to be executed; if yes, the auxiliary machine is driven to reach the temporary assistance domain, and then the action of the auxiliary mode is executed; otherwise, executing the action of the active mode. According to the invention, the problems of high unmanned aerial vehicle scheduling difficulty, high communication coverage rate improvement difficulty and poor user experience in the existing multi-unmanned aerial vehicle cooperative communication system are solved.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) communication, specifically relating to an optimization method for multi-UAV collaborative communication, a multi-UAV collaborative communication system, and an optimization device for multi-UAV collaborative communication. Background Technology

[0002] In recent years, with the rapid development of aviation and automation technologies, unmanned aerial vehicles (UAVs) have been widely used in various fields such as military, agriculture, transportation, and public administration due to their superior maneuverability, low cost, and ease of operation. By installing small communication devices on UAVs, they can act as relay nodes to provide communication services to ground users. Especially in the field of emergency communication, UAVs can be used as mobile base stations to provide temporary communication for ground users and to provide real-time on-site information for remote command. These applications make aerial UAV communication of significant research importance. However, facing the widely distributed needs of users, a single UAV often cannot effectively meet the communication requirements of ground users. Therefore, research on collaborative communication among multiple UAVs is particularly important.

[0003] Existing literature, "Path Planning for UAV-Mounted Mobile Edge Computing With Deep Reinforcement Learning," utilizes artificial neural networks to explore the relationship between scene information such as UAV location, energy state, and throughput. Meanwhile, "Deep Reinforcement Learning for UAV Navigation Through Massive MIMO Technique" employs convolutional neural networks to model scene information as a tensor composed of multiple channels, making the scene feature hierarchy clearer. R. Zhong et al., in "Multi-agent reinforcement learning in noma-aided UAV networks for cellular offloading," proposed a multi-agent MDQN algorithm to learn such strategies.

[0004] However, based on existing research, some challenges remain: First, there is a lot of redundant exploration in the early stages of UAV training, resulting in a waste of resources and time; second, the existing auxiliary behaviors of multiple UAVs are vaguely defined, requiring decision-making algorithms to explore on their own, which is quite limited and often difficult to implement in open scenarios. Summary of the Invention

[0005] To address the challenge of scheduling drones in existing multi-drone collaborative communication systems, this invention provides an optimization method, a multi-drone collaborative communication system, and an optimization device for multi-drone collaborative communication.

[0006] The technical solution provided by this invention is as follows: An optimization method for multi-UAV cooperative communication, comprising: A primary and secondary drone are designated among multiple communication drones. A first decision network, pre-trained using the DDQN algorithm, optimizes the primary drone's flight trajectory within its service range. The first decision network takes its own state information and that of users within the service scenario as input, combined with a pre-defined total reward objective function. R total The system generates actions to control the host aircraft's flight, which are then executed by the host. A second decision network, pre-trained using the DDQN algorithm, optimizes the flight trajectory of the slave aircraft within the service area. This second decision network takes the state information of the host, slave aircraft, and users within the service scenario as input, and combines this with a pre-defined collaborative reward function. R 1. Generate a type of action used to control the flight of the secondary aircraft in active mode; and combine it with a preset auxiliary reward function. R 2. Generate two types of actions in auxiliary mode to control the flight of the secondary aircraft.

[0007] The slave receives a type I action and a type II action, and performs the following decisions: (1) generates a temporary assistance domain related to the real-time location of the host; (2) determines whether support needs to be performed based on a preset criterion: if yes, the slave is driven to reach the temporary assistance domain and then performs a type II action; otherwise, a type I action is performed.

[0008] As a further improvement of the present invention, the scene modeling of the first decision network and the second decision network during the training phase includes: (1) The service range is gridded and the location information of the service range is encoded into a matrix. The spatial location of the host, slave and user is represented by a two-dimensional location vector (W,H).

[0009] (2) The actions of the communication drone and the user are associated with motion vectors in nine directions: east, south, west, north, southeast, southwest, northeast, northwest, and stationary. u 1~ u 9. Characterization.

[0010] (3) The circular area centered on the communication UAV and with a communication coverage distance D as the radius is regarded as the receptive field of the communication UAV, and the communication demand of users contained within the receptive field is used as the communication reward value of the communication UAV. V 0.

[0011] Furthermore, in this invention, scene modeling may also include the following: (4) Define the communication reward value generated by any user as decaying over time to satisfy the timeliness constraint.

[0012] (5) Define the loiter time and motion power consumption of the communication UAV. E 1 and communication power consumption E 2. Related to energy constraints.

[0013] As a further improvement of the present invention, the temporary assistance domain is a circular area centered on the host's location and with a radius equal to the distance moved by the host in a specified number of time steps while performing a single action.

[0014] As a further improvement of this invention, the criterion for whether support needs to be performed is defined as follows: , in, W This represents the communication reward value of the overlapping portion of the secondary aircraft's receptive field and the temporary assistance domain when it is expected to arrive, calculated based on the direction vector. This refers to the average expected value of communication reward calculated by the host. T It is a constant representing a preset condition trigger threshold.

[0015] As a further improvement of this invention, the expression for the timeliness constraint is: , In the above formula, V 0 represents the initial value of the communication reward corresponding to the user's communication needs. V t This indicates the communication reward value corresponding to the user's communication needs. t Remaining value after the time period; This represents the attenuation factor.

[0016] As a further improvement of the present invention, the training process of the first decision network is as follows: S1: Define the host's action space as A: A=( u 1, u 2,…, u 9) Establish the objective function for the final total reward within the task cycle. R total : , In the above formula, Indicates the end time of the mission. This represents a discount factor used to balance the proportion of future rewards in current accumulated rewards. R t This represents the instant reward function.

[0017] S2: Generate the host's state based on the host's receptive field. s t The value matrix Q of all possible actions is then input into the decision network of the DDQN algorithm. The decision network generates the value matrix Q of all possible actions, and the host selects an action to execute based on the state transition probability P.

[0018] S3: Decision network combined with execution actions a t Calculate reward value r t Determine the new state s t+1 And add a new experience Δ to the experience replay pool: Δ = ( s t , a t , r t , s t+1 ); S4: Randomly sample multiple experiences from the experience replay pool for offline learning, and update the weights of the decision network using gradient descent. And after a set number of actions, the parameters of the target network in the DDQN algorithm are updated to obtain the optimal strategy.

[0019] As a further improvement of the present invention, the formula for calculating the state transition probability in step S2 is as follows: , In the above formula, This represents the greed coefficient during the action selection process; Q A value matrix representing all possible actions; As a further improvement of this invention, the loss function used in the offline learning process is... for: , The weight update formulas for the decision network and the target network are as follows: , In the above formula, This represents a function that calculates the expected value. A discount factor representing the reward; Indicates the learning rate; For cumulative rewards; Indicates the execution of an action a t The next new state that is reached later; Indicates the next state Next, the target network is used to select the one with the largest... QThe value of the action; These represent the parameters of the target network. This represents the parameters of the decision network; the parameters of the target network are periodically copied from the parameters of the online network, and their function is to calculate the target value, thereby avoiding... Q The value was overestimated.

[0020] As a further improvement of the present invention, the cooperative reward function R 1. Using the total reward objective function R total .

[0021] Alternatively, an information entropy-based reward mechanism can be introduced into the part of the second decision network used to generate a class of actions in the proactive mode. In this case, the collaborative reward function... R 1 satisfies the following formula: R 1= R total + R H , In the above formula, R H The auxiliary reward provided by the information entropy-based reward mechanism satisfies: , In the above formula, H initial This represents the initial target entropy value; P ij Indicates the first in the service scope i Line 1 j The probability of exploring a grid of columns; N This represents the number of elements in the information entropy matrix; and They represent R H The overall entropy difference term and the local entropy difference term are included, and the trade-off factor is denoted by .

[0022] As a further improvement to this invention, an auxiliary reward function R The expression for 2 is: , In the above formula, t 1 indicates the estimated time for the host to traverse the temporary assistance domain; t 2 indicates the shortest time for the secondary aircraft to reach the temporary assistance zone; This represents the communication reward value of the host. This represents the communication reward value of the secondary unit; and These are the collaborative weighting factors for the main unit and the secondary unit, respectively.

[0023] The present invention also includes a multi-UAV collaborative communication system, which includes: multiple communication UAVs, a first decision module and a second decision module.

[0024] In this system, one communication drone is designated as the master drone, while the others are slave drones. Each communication drone patrols within its service area based on the received motion control commands, thereby providing communication services to users within its communication coverage range.

[0025] The first decision module runs a pre-trained first decision network, as described in the aforementioned optimization method for multi-UAV cooperative communication. This first decision module acquires the status information of the host and users within the service scenario, and then generates commands to control the host's flight.

[0026] The second decision module runs a pre-trained second decision network, as described in the aforementioned optimization method for multi-UAV cooperative communication. This module acquires the status information of the slave aircraft, the host aircraft, and users within the service scenario, and then generates first and second actions to control the slave aircraft's flight. The second decision module also includes a region update unit and a status judgment unit. The region update unit generates the current location information of the temporary assistance domain based on the host aircraft's real-time location. The status judgment unit determines whether support is needed based on preset criteria. The first decision module executes the following decisions: when support is needed, it first sends a control command to the slave aircraft to reach the temporary assistance domain, and then sends a control command to execute the second type of action. When support is not needed, it sends a control command to execute the first type of action.

[0027] The present invention also includes an optimization device for multi-UAV cooperative communication, comprising a memory, a processor, and a computer program stored in the memory and running in the processor. When the processor executes the computer program, it implements the aforementioned optimization method for multi-UAV cooperative communication, optimizing the flight trajectories of each communicating UAV within its service range.

[0028] The technical solution provided by this invention has the following beneficial effects: This invention provides a novel strategy for scheduling communication drones in multi-drone collaborative scenarios. This new strategy categorizes communication drones into two types: master drones and slave drones. The master drone uses the DDQN algorithm to execute learnable flight maneuvers based on actual user communication needs within its service range. The slave drones, depending on whether the master drone requires support, assume different roles and execute differentiated flight maneuvers using the DDQN algorithm. When support is needed, the slave drone acts as a "wingman" for the master drone, expanding its communication service range. When no support is needed, the slave drone works alongside the master drone, both providing a higher quality communication service.

[0029] This invention also addresses novel communication drone scheduling strategies by modeling tasks and designing and training two decision networks using the DDQN algorithm to execute the corresponding scheduling strategies. This enables automated collaborative control of various communication drones, thereby significantly improving the user's communication experience within the drone communication system. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the optimization method for multi-UAV cooperative communication provided in Embodiment 1 of the present invention.

[0031] Figure 2 This is a topology diagram of the multi-UAV collaborative communication system using an embedded module in Embodiment 2 of the present invention.

[0032] Figure 3 This is a topology diagram of the multi-UAV collaborative communication system using a cloud server in Embodiment 2 of the present invention.

[0033] Figure 4 This is a topology diagram of the multi-UAV collaborative communication system using an edge server in Embodiment 2 of the present invention.

[0034] Figure 5 This is a test experiment showing the curve of the total reward obtained by the UAV communication system as a function of the number of training rounds.

[0035] Figure 6 This is a test experiment to show the curve of the average number of flight steps required by the secondary UAV to complete the same amount of communication information collection task as a function of the number of training rounds. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0037] Example 1 This embodiment provides an optimized method for multi-UAV cooperative communication. In the solution of this embodiment, as follows: Figure 1 As shown, communication drones are divided into two categories: main communication drones (hereinafter referred to as host drones) and auxiliary communication drones (hereinafter referred to as slave drones). The host drone's role is to patrol within a designated service area according to a learnable strategy, meeting the dynamically changing needs of users below and providing corresponding communication services. The host drone's flight path is only affected by its own and the users' communication needs, in order to maximize the satisfaction of user demands.

[0038] The secondary drone has two operating modes: active mode and auxiliary mode. In active mode, the secondary drone functions identically to the primary drone, patrolling within a designated service area according to a learnable strategy to meet the dynamically changing needs of users below and provide corresponding communication services. However, unlike the primary drone, in the secondary drone's learnable strategy, it is affected not only by its own and users' communication needs but also by the primary drone operating concurrently. That is, the secondary drone's flight path and communication coverage should avoid overlapping with the primary drone as much as possible to achieve higher communication coverage. Therefore, considering the influence of the primary drone on the secondary drone, if the primary drone is defined as a fully autonomous (unaffected by other communication drones) communication drone, then in active mode, the secondary drone is a semi-autonomous (affected by the primary drone) communication drone.

[0039] When the communication coverage provided by the main unit is insufficient to meet user needs, the secondary unit in this embodiment enters auxiliary mode. In auxiliary mode, the main unit no longer operates as an independent communication drone, but rather as a "wingman." At this time, the secondary unit will fly alongside the main unit to extend the coverage of the communication service provided by the main unit, thereby achieving full coverage of the user's communication needs.

[0040] Specifically, the optimization method for multi-UAV cooperative communication provided in this embodiment is implemented in the following way: I. Role Assignment The solution in this embodiment first requires designating a master and slave drone among multiple communication drones. In practical applications, this solution can be applied to scenarios with only two communication drones, where one drone acts as the master and the other as the slave. When this solution is applied to a scenario with two communication drones, the allocation of the master and slave drones can be flexibly designed based on actual scenario factors such as service range and demand distribution. For example, one communication drone can be selected as the master, and the rest as slaves. When the master requires communication assistance, a nearby drone can be selected from the slaves to perform a chaperone mission to provide auxiliary communication. Alternatively, the entire communication service range can be divided into sections, and two communication drones can be configured for each section. The two communication drones only provide services within their respective sections. Within their respective sections, one drone acts as the master and the other as the slave, and they operate according to the dual-drone collaborative communication strategy described above, providing communication services to users within their corresponding sections.

[0041] Furthermore, it is important to emphasize that in the practical application of this embodiment, the function of each communication drone—whether it acts as a master or slave—can be flexibly switched according to mission requirements; the roles of master and slave are not inherent attributes of any communication drone. For example, within one mission cycle, drone number one can act as the master and drone number two as the slave. In the next mission cycle, the roles can be reversed, with drone number two acting as the slave and drone number one as the master.

[0042] II. Scheduling and Execution The above describes the role allocation of each drone in a multi-communication drone scenario and the task content designed for each role. To enable each drone to perform its assigned task according to its designated role, thereby providing better communication coverage and service, this embodiment also designs a corresponding scheduling system for the various communication drones within the multi-drone collaborative communication system. This scheduling system consists of two parts: one part controls the master communication drones, and the other part controls the slave communication drones.

[0043] 2.1 Host Scheduling Strategy The host's task is to patrol within a designated area based on real-time user communication demands to ensure that customer communication needs are met. Therefore, host scheduling essentially involves adjusting the host's flight path based on predicted future user communication demands. While user communication demands are unpredictable, they do exhibit certain statistical patterns over significant time and spatial dimensions. To address this, this embodiment utilizes a Dual Deep Reinforcement Learning (DDQN) model. Based on historical state data, including user communication demands within the service area and the host's operational process, a decision network is trained to output the host's ideal flight maneuvers based on user demands and the host's real-time status.

[0044] In the technical solution provided in this embodiment, a first decision network pre-trained based on the DDQN algorithm is used to optimize the flight trajectory of the host within the service range. The first decision network takes its own state information and the state information of users within the service scenario as input, and combines it with a preset total reward objective function. R total It generates actions to control the flight of the host aircraft, which are then executed by the host aircraft.

[0045] In order to make the DDQN algorithm applicable to the host scheduling problem, this embodiment performs scenario modeling on the scheduling problem addressed by the first decision network during the training phase, including: (1) The service range is gridded and the location information of the service range is encoded into a matrix. The spatial location of the host, slave and user is represented by a two-dimensional location vector (W,H).

[0046] In this embodiment, the drone operates at a fixed altitude within its service range. h The communication drone patrols in the sky, while users with communication needs move on the ground. Therefore, in practical applications, both the communication drone and the user move at different altitudes. To more concisely represent the spatial positions of the communication drone and the user, this embodiment meshes the service area (corresponding to the communication drone's operating scenario). Combining the numerous equidistant grids, the location information of the service area can be encoded into a matrix. In this state, the communication drone flies between W*H equidistant grids. Ground mobile users are randomly distributed at the center of each grid.

[0047] (2) The actions of the communication drone and the user are associated with motion vectors in nine directions: east, south, west, north, southeast, southwest, northeast, northwest, and stationary. u 1~ u 9. Characterization.

[0048] In this embodiment, the communication drone and the user have two different states: moving and stationary. In the scene modeling of this embodiment, both the communication drone and the user are defined to move at a constant speed according to their respective typical speeds. Within their respective planes, the movement directions of the user and the communication drone can be categorized into eight types: east, south, west, north, southeast, southwest, northeast, and northwest. Combining the different states and movement directions, this embodiment uses nine motion vectors to represent the movement trends of the user and the communication drone. These nine motion vectors—east, south, west, north, southeast, southwest, northeast, northwest, and stationary—are denoted as: u 1~ u 9. Based on this, optimizing the flight trajectory of a communication drone essentially involves generating corresponding motion vectors at different times to control the flight attitude of the communication drone.

[0049] (3) The communication demand of users within the sensing field is used as the communication reward value for the communication drone. V 0.

[0050] The DDQN algorithm evaluates the reward value of different actions to assess the benefit of the selected action, thereby driving the target to execute the action with the highest reward. In the UAV communication scenario of this embodiment, the essence of the optimization problem is to meet the user's communication needs. Therefore, in this embodiment, the communication needs of the user satisfied by the communication service provided by the UAV are used as the communication reward value in the DDQN algorithm. Based on this, considering that the communication coverage range that each communication UAV can provide is limited, this embodiment considers a circular area centered on the communication UAV with a communication coverage distance D as the receptive field of the communication UAV; the UAV can provide communication services to ground mobile users within the receptive field. When a user who makes a communication request is located within the current receptive field of any communication UAV, it means that the user's communication need at that moment has been met. Conversely, when a user who makes a communication request is located outside the current receptive field of all communication UAVs, it means that the user's communication need at that moment has not been met.

[0051] Furthermore, in order to achieve a more refined mathematical description of the optimization problem, the scene modeling in this embodiment may also include the following: (4) Define the communication reward value generated by any user. V 0 decays over time, satisfying timeliness constraints.

[0052] The timeliness constraint describes how a user's user experience changes over time when they submit a communication request. It is well known that any user's communication request should be met immediately, or at the very least, after a specified delay following the submission of the request. If the waiting time is too long, the user's communication request may disappear, resulting in the worst user experience. Therefore, in this embodiment, the timeliness constraint of the communication reward value corresponding to a user's communication request in each grid is characterized by the following expression: , In the above formula, V 0 represents the initial value of the communication reward corresponding to the user's communication needs. V t This indicates the communication reward value corresponding to the user's communication needs. t Remaining value after the time period; This represents the attenuation factor. Based on the above expression, it can be seen that, considering the timeliness of information, as the duration increases, the communication reward value obtained by the communication drone when its receptive field covers the user gradually decreases.

[0053] (5) Define the loiter time and motion power consumption of the communication UAV. E 1 and communication power consumption E 2. Related to energy constraints.

[0054] When a communication drone flies in the sky to provide communication services to users on the ground, its flight time is not unlimited. Generally speaking, the flight time of a communication drone is constrained by two factors: its own energy supply and overall energy consumption. Considering that the energy supply (battery or fuel tank, etc.) carried by each communication drone is fixed, the flight time of the communication drone is only constrained by overall energy consumption. In this embodiment, overall energy consumption is further divided into two aspects: one is motion power consumption. E 1. The other is communication power consumption. E 2. Among them, communication drones typically cruise at a constant speed at a designated altitude, therefore, motion power consumption... E 1 is usually constant and can be characterized as a fixed value. Communication power consumption, however, is closely related to the communication needs of ground users; the correlation between the two is as follows: , In the above formula, V r This indicates the rate at which a communication drone transmits signals during communication. e tr Indicates the transmission efficiency of the communication process; c This indicates the bandwidth limitations during communication between the drone and ground users. d The communication distance between the drone and the ground user is calculated using the following formula: , In the above formula, h Indicates the height of the drone above the horizontal ground. express t The horizontal coordinate projection of the drone in the scene. express t The horizontal coordinate of the ground user in the scene at any time. express t The vertical coordinate projection of the drone in the scene at any time. express t The vertical coordinate of the ground user in the scene at any given time.

[0055] In detail, the dual deep reinforcement learning model is a classic reinforcement learning algorithm that introduces two identical networks: an action selection network (decision network) and a target network. The drone uses the action selection network to predict the Q-value of each possible action in different time-varying scenarios, selecting the action with the highest Q-value. After executing the action, the drone observes new scene information and receives a reward. The target network then evaluates the Q-value of each action in this new scene state, selecting the largest Q-value as the best estimate for that action. This largest Q-value is then used to update the parameters of the action selection network. However, the parameters of the target network are not updated at every step; instead, they are copied to the target network after several actions at specified intervals, making the target network's estimates more stable.

[0056] Specifically, the process of training the required first decision network using the DDQN algorithm in this embodiment is roughly as follows: S1: Define the host's action space as A: A=( u 1, u 2,…, u 9); in, u 1~ u The number 9 corresponds to nine different actions of the drone: flying in the east, south, west, north, southeast, southwest, northeast, and northwest directions, as well as remaining stationary.

[0057] Establish the objective function for the final total reward within the task cycle. R total : , In the above formula, Indicates the end time of the mission. This represents a discount factor used to balance the proportion of future rewards in the current cumulative rewards. The discount factor is used to adjust the weight of future rewards in the current cumulative rewards. R t This indicates an immediate reward. In the algorithm of this embodiment, after the communication drone performs an action, it will obtain a new state and a corresponding reward value.

[0058] Among them, instant rewards R t express t Real-time drones and their sensing range d The total transmission rate between ground users and The difference is expressed as follows: , In the above formula, This indicates the communication transmission rate between the drone and the user. The expression for the drone's power consumption model during time period t is as follows: , In the above formula, p f Indicates the flight power of the drone; v uav Indicates the flight speed of the drone; express t Real-time projection coordinates of the drone on the ground; express t The projected coordinates of the UAV on the ground at time -1. in, The expression is as follows: , In the formula: W e This represents the bandwidth allocated to each ground communication user by the drone; SNR is the signal-to-noise ratio.

[0059] S2: Generate the host's state based on the host's receptive field. s t The value matrix Q of all possible actions is then input into the decision network of the DDQN algorithm. The decision network generates the value matrix Q of all possible actions, and the host selects an action to execute based on the state transition probability P.

[0060] Specifically, the state information of the aforementioned service scenarios involving users is collected, and combined with the host's receptive field, the host's state is obtained. s t , s t Indicates the current t Scene observation information at any given time. The scene information and the host's receptive field are used as input to the decision network. The decision network outputs a value matrix Q for all possible actions of the UAV. Actions are selected for execution based on the state transition probabilities P, and its mathematical expression is: , In the above formula, This represents the greed coefficient during the action selection process; s t+1 Indicates that the drone is performing actions a t Scene observation information at the next moment after; a t This indicates the action selected by the drone at time t.

[0061] The host is based on the state transition probability PAfter the selected action is executed, the scene information and reward value change, and the optimal strategy is updated accordingly. Its mathematical expression is as follows: , S3: Decision network combined with execution actions a t Calculate reward value r t Determine the new state s t+1 And add a new experience Δ to the experience replay pool: Δ = ( s t , a t , r t , s t+1 ).

[0062] In this embodiment, the DQNN algorithm is used to establish an experience replay pool to store the experience values ​​of each state transition of the communication UAV. Each experience value includes: the current state information of the host. s t Selected action a t The reward value of the currently executed action r t New state information obtained after performing the action s t+1 .

[0063] S4: Randomly sample multiple (corresponding to the batch size) experiences from the experience replay pool for offline learning, and update the weights of the decision network using gradient descent. And after a set number of actions, the parameters of the target network in the DDQN algorithm are updated to obtain the optimal strategy.

[0064] In this embodiment, the loss function used in the offline learning process is... for: , The weight update formulas for the decision network and the target network in the DQN algorithm are as follows: , In the above formula, This represents a function that calculates the expected value. A discount factor representing the reward; Indicates the learning rate; For cumulative rewards; Indicates the execution of an action a t The next new state that is reached later; Indicates the next state Next, the target network is used to select the one with the largest... Q The value of the action; These represent the parameters of the target network. This represents the parameters of the decision network; the parameters of the target network are periodically copied from the parameters of the online network, and their function is to calculate the target value, thereby avoiding... Q The value was overestimated.

[0065] The above content details the key information defined when the first decision network is trained using the DDQN algorithm in this embodiment, including actions, states, rewards, loss functions, and weight update formulas. Other non-core technical details of the DDQN algorithm during the training phase will not be elaborated in this embodiment.

[0066] 2.2 Scheduling Strategy for Secondary Units The previous section introduced the first decision network, pre-trained using a DDQN network, which outputs the subsequent flight actions the host should take based on the user and host states. In this implementation, based on the same approach, a second decision network pre-trained using the DDQN algorithm can be used to optimize the flight trajectory of the slave aircraft within the service area. Unlike the host, which can make flight action decisions completely autonomously, the flight action commands generated by the slave aircraft using the second decision network need to consider the states of existing hosts within the scene to avoid collisions or competition. Therefore, the second decision network in this embodiment takes the state information of the host, slave aircraft, and users within the service scene as input. Furthermore, considering that the slave aircraft in this embodiment needs to select different operating modes based on actual conditions during operation, the second decision unit in the slave aircraft should incorporate a preset cooperative reward function during the training and application phases. R 1. Generate a type of action used to control the flight of the secondary aircraft in active mode; and combine it with a preset auxiliary reward function. R 2. Generate two types of actions in auxiliary mode to control the flight of the secondary aircraft.

[0067] In summary, in this embodiment, the second decision network actually uses the same historical state data, but trains the DDQN algorithm using different methods to obtain two network units. One is designed and trained according to the working scenario and requirements of the active mode, and it can generate the flight maneuvers that the secondary aircraft should take in the active mode. The other is designed and trained according to the working scenario and requirements of the auxiliary mode, and it can generate the flight maneuvers that the secondary aircraft should take in the auxiliary mode. The secondary aircraft can then select and execute the corresponding flight maneuver according to its own working mode. That is: The slave receives a type I action and a type II action, and performs the following decisions: (1) generates a temporary assistance domain related to the real-time location of the host; (2) determines whether support needs to be performed based on a preset criterion: if yes, the slave is driven to reach the temporary assistance domain and then performs a type II action; otherwise, a type I action is performed.

[0068] (2.2.1) Switching between working modes In this embodiment, the secondary machine has two operating modes, each with different roles and actions. Therefore, it's necessary to first determine the switching strategy for the secondary machine in each mode. Specifically, in this embodiment, the secondary machine operates in active mode most of the time. Only when the primary machine triggers a communication assistance requirement from the secondary machine will it switch to auxiliary mode to support the primary machine. Conversely, when the primary machine no longer meets the triggering conditions requiring communication assistance from the secondary machine, the secondary machine exits auxiliary mode and switches back to active mode. In other words, the switching between the two operating modes depends primarily on whether the primary machine's task triggers a condition requiring communication assistance from the secondary machine.

[0069] Before introducing the triggering conditions for the different working modes of the secondary machine designed in this embodiment, the concept of a temporary assistance domain is first introduced. In this embodiment, a temporary assistance domain O is a domain centered on the host location, where the host performs a single action for a specified duration (e.g., ...). The time step moves a circular region with a radius equal to the distance of the circle. That is, the size of the temporary aid domain is... The temporary assistance domain is an area whose size exceeds the service coverage of a single communications drone.

[0070] Based on this, this embodiment defines the criterion for whether the host needs to perform support (i.e., the triggering condition for switching to slave mode) as follows: , in, W This represents the communication reward value of the overlapping portion of the secondary aircraft's receptive field and the temporary assistance domain when it is expected to arrive, calculated based on the direction vector. This refers to the average expected value of communication reward calculated by the host. T It is a constant representing a preset condition trigger threshold.

[0071] (2.2.2) Active mode In the active mode of the secondary aircraft, the second decision unit outputs the flight action that the secondary aircraft should take, i.e., the second action, based on the input status information of the primary aircraft, the secondary aircraft, and the users within the service scenario. Like the first decision unit, the second decision unit is also pre-trained using the DDQN algorithm. Specifically, in the active mode portion of the second decision unit, the cooperative reward function of the DDQN algorithm is set.R Alternatively, a total reward objective function can be used. R total .

[0072] Furthermore, in a further optimized scheme of this embodiment, to reduce energy consumption caused by the secondary machine's blind exploration in the early stages, an information entropy-based reward mechanism can be introduced into the part of the second decision network used to generate a type of action in active mode. The information entropy-based reward mechanism provides positive rewards by calculating the secondary machine's exploration degree of the environment. Initially, due to the large number of unexplored grids, the information entropy rewards are mostly negative. As a large number of grids are explored, the system receives more positive rewards. The overall trend is that as the exploration degree increases, the negative rewards decrease, and the rewards begin to tend towards positive. Ultimately, the optimized reward function encourages the secondary machine to explore more comprehensively.

[0073] After introducing a reward mechanism based on information entropy, the collaborative reward function R1 satisfies the following equation: R1= R total + R H , In the above formula, R H This refers to the auxiliary reward provided by the information entropy-based reward mechanism. The following provides a detailed description of the information entropy-based reward mechanism introduced in this embodiment: S01: Initialize a trajectory matrix, a two-dimensional matrix of size W*H, to record the exploration trajectory of the secondary aircraft. Based on the secondary aircraft's perception field, include the exploration degree in the equidistant grid within the perception range. T ij ,in, i and j These represent the row and column numbers of the grid, respectively.

[0074] S02: Set a threshold for exploration level K Exceeding the threshold K This is considered a completion of depth exploration. After the scene information exploration is complete, the exploration degree values ​​of each grid in the previous step are normalized to obtain the probability of exploring the current grid. P ij Its mathematical expression is as follows: , S03: Probability obtained from the previous steps P ij The information entropy of the current grid can be calculated. H ij Thus, the overall information entropy matrix H is obtained. Wherein, information entropy... H ij The calculation formula is: , After the above processing, the exploration progress of the secondary machine in the current environment is converted into a specific information entropy value, and an entropy matrix with the same shape as the initial trajectory matrix is ​​obtained. This entropy value indicates the value of the grid exploration. If the exploration progress is full, the entropy value of the area is 0; the larger the entropy value of an unexplored area, the easier it is to reward the secondary machine with entropy value to incentivize exploration later.

[0075] S03: Considering the limited impact of the entropy value of a single grid cell, it is necessary to calculate the global total entropy value. H total To increase the impact, the mathematical expression is as follows: S04: To avoid overly uniform exploration and to increase attention to underexplored areas, it is necessary to balance the relationship between global exploration and in-depth local exploration. Therefore, a local entropy difference term is introduced. D To optimize reward distribution and incentivize diverse exploration, its mathematical expression is as follows: , In the above formula, N Represents the information entropy matrix H The number of elements in the text.

[0076] S05: To calculate the reduction in overall entropy to directly reflect the macroscopic progress of the exploration, it is necessary to calculate the difference between the target entropy and the overall entropy, i.e., the overall entropy difference term Δ. H Its mathematical expression is as follows: Δ H = H initial - H total , In the above formula, H initial This represents the initial target entropy value. In this embodiment, H initial =1.0.

[0077] S06: Local entropy difference term combined with the preceding steps D and the overall entropy difference term Δ H The overall entropy reward function can be obtained. R H Its mathematical expression is as follows: , In the above formula, and The overall entropy difference term Δ H and local entropy difference term D The two factors, both taking values ​​between (0, 1), are used to weigh the overall entropy difference term Δ. Hand local entropy difference term D The impact of the final complete overall entropy reward function R H It can be represented as: .

[0078] final, R H This serves as an auxiliary reward provided by the introduced information entropy-based reward mechanism.

[0079] (2.2.3) Assist Mode In this embodiment, compared with the active mode, the secondary aircraft in the auxiliary mode will stop the currently executing flight action, obtain the location of the host, and immediately fly to the host's temporary assistance domain, and then fly according to the first action output by the second decision network.

[0080] Specifically, during the training phase of the second decision network, the auxiliary reward function used for the part related to the auxiliary mode is... R The design process for 2 is as follows: S001: Construct a temporary assistance domain: Using the host's receptive field and current scene information obtained from the secondary machine, the host's temporary assistance domain at the current moment is planned, and the size of the temporary assistance domain is [size missing]. .

[0081] S002: Calculate the expected average communication reward of the host. : Based on the previous step, the receptive field size of the host is... The estimated time for the host to traverse the temporary aid area is calculated. t 1 satisfies the following formula: .

[0082] Assuming the number of ground mobile users within the temporary assistance zone is n The total communication reward value is V total The average communication reward value obtained by the host per unit time step for: , Based on the time-sensitivity constraint of the decay of ground mobile user communication reward value over time introduced above, the average expected communication reward obtained by the host independently exploring the temporary assistance domain per unit time step is calculated to be [value missing]. The expression is as follows: .

[0083] S003: Calculate the fastest time required for the secondary aircraft to reach the temporary aid zone. t 2: Assume the speed of the secondary machine is v The fastest time required for the co-pilot to reach the temporary assistance area is... t 2: t 2=L / v L represents the Euclidean distance between the master and slave machines.

[0084] S004: Calculate the expected average communication reward value of the secondary unit. : Time from the co-pilot's arrival at the temporary aid zone t 2. Start time until the host completes the task. t Up to 1, calculate the communication reward value of the slave unit. Its mathematical expression is as follows: , In the above formula, Includes the secondary machine from t From moment 2 until the completion of the temporary assistance area mission, the total communication reward value obtained by the secondary aircraft.

[0085] The expected average communication reward value of the secondary machine for: , S005: Constructing the aid reward function R 2: , In the above formula, and The collaborative weighting factor, with a value between (0, 1), is used to balance the communication reward values ​​between the master and slave units. The final complete expression for the assistance reward function is: .

[0086] In summary, in the scheme designed in this embodiment, by establishing triggering conditions and a temporary assistance domain, the slave aircraft can determine whether the master aircraft currently meets the assistance conditions. If it does, during the mission time, the slave aircraft will no longer fly according to its original actions, but will instead reach the temporary assistance domain via the shortest route to assist the master aircraft in performing the communication task. In the second decision network adopted by the slave aircraft, the differentiated reward function can make the actions generated by the second decision network more realistic, encouraging the slave aircraft to actively assist the master aircraft in completing the communication task more efficiently.

[0087] Example 2 Based on the scheme in Embodiment 1, this embodiment further provides a multi-UAV collaborative communication system, which includes: multiple communication UAVs, a first decision module and a second decision module.

[0088] In a multi-UAV collaborative communication system, one of the multiple communication UAVs is designated as the master, and the rest are slaves. Each communication UAV patrols within its service area according to the received motion control commands, thereby providing communication services to users within its communication coverage range.

[0089] The first decision module runs a pre-trained first decision network, as described in the optimized method for multi-UAV cooperative communication in Example 1. The first decision module is used to acquire the status information of the host and users within the service scenario, and then generate commands to control the host's flight.

[0090] The second decision module runs a pre-trained second decision network, as described in the aforementioned optimization method for multi-UAV cooperative communication. This module acquires the status information of the slave aircraft, the host aircraft, and users within the service scenario, and then generates first and second actions to control the slave aircraft's flight. The second decision module also includes a region update unit and a status judgment unit. The region update unit generates the current location information of the temporary assistance domain based on the host aircraft's real-time location. The status judgment unit determines whether support is needed based on preset criteria. The first decision module executes the following decisions: when support is needed, it first sends a control command to the slave aircraft to reach the temporary assistance domain, and then sends a control command to execute the second type of action. When support is not needed, it sends a control command to execute the first type of action.

[0091] It should be noted that in practical applications, the first and second decision-making modules are deployed in various ways. For example, in... Figure 2 In Scheme 1 shown, a first decision module and a second decision module can be deployed in the host and slave machines respectively, and the host and slave machines can also communicate with each other to achieve mutual transmission of their status data. Alternatively, in Figure 3 In Scheme 2, the first and second decision modules are deployed in a cloud service, serving as a scheduling system running on the cloud server. The master and slave servers communicate with the cloud server, which obtains their status information and executes corresponding scheduling strategies. Alternatively, in... Figure 4 In Scheme 2 shown, multiple edge servers are deployed within the service areas of the host and the slave. When they arrive at different areas, they interact with the critical edge server respectively. The first decision module and the second decision module are deployed in each edge server. The edge server obtains the status information of the host and the slave and executes the corresponding scheduling strategy.

[0092] Example 3 This embodiment provides an optimization device for multi-UAV cooperative communication, which includes a memory, a processor, and a computer program stored in the memory and running in the processor. When the processor executes the computer program, it implements the aforementioned optimization method for multi-UAV cooperative communication, optimizing the flight trajectories of each communicating UAV within its service range.

[0093] The optimized device for multi-UAV collaborative communication provided in this embodiment is essentially a computer device used to implement the solution in Embodiment 1. This computer device can be an embedded module, or it can be a smart terminal capable of executing programs, such as a tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers).

[0094] The computer device described in this embodiment includes, but is not limited to, a memory and a processor that can be interconnected via a system bus. The memory (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as the hard disk or RAM of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory can also include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. Furthermore, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0095] In some embodiments, a processor may be a central processing unit (CPU), a graphics processing unit (GPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of a computer device. In this embodiment, the processor is used to run program code stored in memory or process data.

[0096] Simulation test To verify the effectiveness of the optimized multi-UAV cooperative communication scheme provided by this invention, technicians formulated an experimental plan and trained and tested the method based on a self-built two-dimensional simulation scenario. The results of the tests show the changes in the total reward obtained by the UAV communication system and the average number of flight steps and training rounds required for the secondary UAV to complete the same amount of communication information collection task. Figure 5 and Figure 6 As shown.

[0097] analyze Figure 5 It can be seen that as the number of training rounds increases, the reward obtained by the secondary drone fluctuates significantly in the early stages of training, then gradually increases and finally stabilizes between 1290 and 1340. During the training process, its reward value gradually increases and eventually converges. Analysis Figure 6 It can be seen that as the number of training rounds increases, the number of steps required for the UAV to complete the communication task gradually decreases, eventually settling between 140 and 180. The above experimental results indicate that the secondary UAV can maintain the completion rate of the ground communication task while performing auxiliary tasks.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for optimizing cooperative communication of multiple UAVs, characterized in that: It includes: Specify the master and slave among multiple communicating drones; The flight trajectory of the host in the service range is optimized through a first decision network pre-trained based on the DDQN algorithm; the first decision network takes the status information of itself and the users in the service scene as input, combined with the preset total reward objective function R total , generating actions for controlling the host's flight, which are then executed by the host; The flight trajectory of the secondary aircraft in the service range is optimized through a second decision network pre-trained based on the DDQN algorithm; the second decision network takes the status information of the host, secondary aircraft and users in the service scene as input, combined with the preset collaborative reward function R 1 Generate a type of action for controlling the flight of the auxiliary aircraft in active mode; and combine it with the preset auxiliary reward function R 2 Generate the second type of action for controlling the auxiliary aircraft in auxiliary mode; The slave receives the first and second actions and makes the following decisions: (1) Generates a temporary assistance domain related to the real-time position of the host; (2) Determines whether assistance is required based on the preset criteria: If yes, drives the slave to reach the temporary assistance domain and then performs the second action; otherwise, performs the first action.

2. The optimization method for multi-UAV cooperative communication according to claim 1, characterized in that: The scenario modeling of the first decision network and the second decision network in the training phase includes: The service range is gridded, and the location information of the service range is encoded into a matrix. The spatial locations of the host, slave, and user are represented by a two-dimensional position vector (W, H); The actions of the communication drone and the user are connected to the nine directions of motion vectors: east, south, west, north, southeast, southwest, northeast, northwest, and stationary. u 1~ u 9 for characterization; The circular area with the communication drone as the center and the communication coverage distance D as the radius is regarded as the receptive field of the communication drone, and the communication demand of the users contained in the receptive field is regarded as the communication reward value of the communication drone. V 0; And / or, define the communication reward value generated by any user to decay over time to meet the timeliness constraint; And / or, define the hovering time and motion power consumption of the communication drone E 1 and communication power consumption E 2, satisfying the energy constraint.

3. The optimization method for multi-UAV cooperative communication according to claim 2, characterized in that: The temporary assistance domain is a circular area with the host position as the center and the distance moved by the host performing a single action for a specified number of time steps as the radius; and / or The criteria for defining whether execution support is needed are: ， in, W It represents the communication reward value of the overlapping part of its own receptive field and the temporary assistance field when the auxiliary machine calculates the estimated arrival time according to the direction vector; Refers to the average expected value of the communication reward calculated by the host; T A constant representing the preset condition trigger threshold.

4. The optimization method for multi-UAV cooperative communication according to claim 2, characterized in that: The expression of the timeliness constraint is: ， In the above formula, V 0 represents the initial value of the communication reward corresponding to the user's communication demand. V t Indicates that the communication reward value corresponding to the user's communication demand is t The remaining value after the period; Represents the attenuation factor.

5. The optimization method for multi-UAV cooperative communication according to claim 4, characterized in that: The training process of the first decision network is: S1: Define the host's action space as A: A=( u 1, u 2,…, u 9) Establish the final total reward objective function within the task cycle R total : ， In the above formula, Indicates the end time of the task. Represents the discount factor used to balance the proportion of future rewards in the current accumulated rewards. R t represents the immediate reward function; S2: Generate the host's state based on the host's receptive field s t , input it into the decision network of the DDQN algorithm, the decision network generates the value matrix Q of all possible actions, and the host selects an action to execute based on the state transition probability P; S3: Decision network combines the actions performed a t Calculating Reward Value r t , determine the new state s t+1 , and add a new experience Δ to the experience replay pool: Δ=( s t , a t , r t , s t+1 ); S4: Randomly sample multiple experiences from the experience replay pool for offline learning, and use the gradient descent method to update the weights of the decision network , and after every preset number of actions, update the parameters of the target network in the DDQN algorithm to finally obtain the optimal strategy.

6. The optimization method for multi-UAV cooperative communication according to claim 5, characterized in that: In step S2, the calculation formula of the state transition probability is as follows: ， In the above formula, Represents the greedy coefficient in the action selection process; Q A value matrix representing all possible actions; and / or Loss function used in offline learning process for: ， The weight update formulas for the decision network and the target network are as follows: ， In the above formula, represents the function for obtaining the expectation; represents the discount factor of the reward; represents the learning rate; To accumulate rewards; Indicates execution of an action a t The next new state reached later; Indicates that in the next state Next, use the target network to select the network with the largest Q Actions of values; Represents the parameters of the target network; Represents the parameters of the decision network.

7. The method for optimizing multi-UAV cooperative communication according to claim 5, characterized in that: The collaborative reward function R 1 Using the total reward objective function R total ; or, In the second decision network, a reward mechanism based on information entropy is introduced for generating a class of actions in the active mode; The collaborative reward function R 1 satisfies the following formula: R 1= R total + R H ， In the above formula, R H represents the auxiliary reward provided by the reward mechanism based on information entropy, which satisfies: ， In the above formula, H initial represents the initial target entropy value; P ij Indicates the service scope i Line j exploration probability of the grid of columns; N Represents the number of elements in the information entropy matrix; and Respectively R H The trade-off factor between the overall entropy difference term and the local entropy difference term contained in .

8. The method for optimizing multi-UAV cooperative communication according to claim 5, characterized in that: The auxiliary reward function R The expression of 2 is: ， In the above formula, t 1 indicates the estimated time for the host to traverse the temporary assistance domain; t 2 represents the shortest time for the secondary aircraft to reach the temporary assistance area; Indicates the communication reward value of the host; Indicates the communication reward value of the slave machine; and are the collaborative weight factors of the host and slave respectively.

9. A multi-UAV cooperative communication system, characterized in that: It includes: Multiple communication drones, one of which is designated as the main drone and the others as slave drones; each communication drone cruises in the service area according to the received motion control instructions, and then provides communication services to users within its own communication coverage distance; A first decision module, in which a first decision network pre-trained in the optimization method for multi-UAV cooperative communication according to any one of claims 1 to 8 is run; the first decision module is used to obtain status information of the host and users in the service scene, and then generate instructions for controlling the flight of the host; A second decision module, in which a second decision network pre-trained in the optimization method for multi-UAV cooperative communication as described in any one of claims 1 to 8 runs; the second decision module is used to obtain the status information of itself and the host and users in the service scene, and then generate the first action and the second action for controlling the flight of the auxiliary aircraft; the second decision module also includes a region update unit and a state judgment unit; the region update unit is used to generate the location information of the current temporary assistance domain according to the real-time position of the host; the state judgment unit is used to judge whether support is needed according to a preset criterion; the first decision module is used to execute the following decisions: when support is needed, a control instruction to reach the temporary assistance domain is first sent to the auxiliary aircraft, and then a control instruction to execute the second type of action is sent; when no support is required, a control instruction to execute the first type of action is sent.

10. An optimization device for cooperative communication of multiple unmanned aerial vehicles, comprising a memory, a processor, and a computer program stored in the memory and running in the processor, characterized in that: When the processor executes the computer program, it implements the optimization method for multi-UAV cooperative communication as described in any one of claims 1 to 8, and optimizes the flight trajectory of each communicating UAV within the service range.

Images