A channel allocation method for multi-UAV base station assisted ground communication

Abstract

This paper discloses a channel allocation method for multi-UAV base station-assisted ground communications. This method utilizes a Markov game framework to model trajectory optimization and channel allocation decisions for a UAV-assisted communication system. A multi-intelligence reinforcement learning model based on an event-driven Transformer is proposed. Its event-driven monitoring strategy optimization mechanism aims to learn the optimal event-driven listening strategy based on the current channel state, effectively reducing unnecessary information listening and processing by multiple UAV base stations. The Transformer network extracts useful information and patterns from a large amount of observational information, enabling the UAV base station evaluation network to more accurately assess state information, thereby achieving dynamic and intelligent channel allocation for multiple UAV base stations. The proposed method demonstrates good strategy convergence performance.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a channel allocation method for multi-UAV base station-assisted ground communication. Background Art

[0002] In drone-assisted communication, reasonable channel allocation is a key factor in improving transmission rates and achieving efficient drone-assisted communication. Selecting highly available channels for information transmission is a method for ensuring smooth communication and improving spectrum utilization. When multiple drone base stations are used to provide communication coverage in an area, the constantly changing positions of drones and the varying electromagnetic environments in which each drone operates make selecting appropriate communication channels a key challenge in achieving high-quality drone-assisted communication. In practice, the joint decision-making process for channel allocation among multiple drone base stations constitutes a Markov game, and solving the Nash equilibrium of this communication-assisted game ensures that each multi-drone base station in the game achieves the optimal decision. However, existing Nash equilibrium solutions often use analytical solutions using multi-constraint polynomials with known prior knowledge or employ convex optimization theory to solve the game equilibrium. This paper addresses the channel allocation decision-making problem for multi-drone-assisted communication by employing a model-free reinforcement learning approach. This approach enables multiple drone base stations to optimize their movement trajectories and channel allocation strategies through interactive learning with the environment, providing new insights for the future development of wireless communication technology. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention proposes a channel allocation method for multi-UAV base station assisted ground communication, which more effectively solves the problem of channel allocation for multiple UAV assisted communications.

[0004] In order to achieve the above object, the technical solution of the present invention is as follows:

[0005] A channel allocation method for multi-UAV base station assisted ground communication includes the following steps:

[0006] Obtain information about M drone base stations and N ground users in the drone-assisted communication system, and initialize the joint strategy of several drone base stations;

[0007] A multi-agent reinforcement learning model based on an event-driven Transformer is constructed. The multi-agent reinforcement learning model includes states, drone base stations, actions, and rewards. The actions include movement vectors, vectors for selecting whether to pair with each ground user, channel selection vectors, and event-driven monitoring strategy outputs. The state of the drone base station in each time slot includes the location of the ground user currently paired with the drone base station, as well as pairing and channel selection information for other drone base stations.

[0008] Input the state observation data of the drone base station in the previous time slot into the multi-intelligence reinforcement learning model and calculate the state value function V(s,a);

[0009] Calculate and compare event-driven conditions. If the conditions are met, trigger the event-driven monitoring strategy to monitor and obtain the broadcasts of other drone base stations. The location of the ground user paired with the current drone base station and the pairing and channel selection information of other drone base stations obtained by monitoring are used as the input of the Transformer model. Output V(q,κ,μ) is used to calculate the advantage function based on V(q,κ,μ). If the condition is not met, the advantage function is calculated based on the position of the ground user paired with the current drone base station

[0010] Based on the advantage function, the action data corresponding to the state observation data of the previous time slot is determined. The drone base station interacts with the simulation environment according to the action data, calculates the output of the Critic network, that is, the Q value, and feeds the Q value back to the Actor network of the drone base station to assist the drone base station in updating its strategy.

[0011] Preferably, the joint strategy for initializing several drone base stations includes setting up a drone-assisted communication system in which each ground user can only be paired with one drone base station, and each drone base station will serve at least one ground user; in each time slot, multiple drone base stations independently decide to pair with ground users in turn, and select a channel from the shared channel, and broadcast the pairing information and channel selection to other drone base stations; several drone base stations will design trajectories in real time according to the location information of the paired ground users, and decide whether to listen to the broadcasts of other drone base stations based on an event-driven strategy.

[0012] Preferably, an over-monitoring penalty operator ε'(t) is added to the reward mechanism of the multi-intelligence reinforcement learning model, and the formula is as follows:

[0013] ε'(t)=c1*Onehot event-driven (t) (13)

[0014] Among them, Onehot event-driven It is a one-hot vector indicating whether the monitoring event occurs, and c1 is the penalty coefficient.

[0015] Preferably, the optimization objectives of the M drone base stations are defined as:

[0016]

[0017]

[0018] p m,n (t) = Ptotal (19)

[0019]

[0020] b m,n (t)≥b min ,p m,n (t)≥p min ,n∈N (21)

[0021]

[0022] Where p m,n (t) and b m,n (t) are the transmission power and bandwidth resources allocated to ground user n by UAV base station m; b min is the minimum divisible bandwidth; p min is the minimum divisible power.

[0023] Preferably, the strategy is updated using the following formula:

[0024]

[0025] Where C is the penalty coefficient, γ is the discount factor, is the approximate objective function, is the KL divergence between the current policy and the candidate policy.

[0026] Based on the above technical solution, the beneficial effects of the present invention are as follows: the present invention targets the channel allocation strategy optimization problem of multiple drone base stations in drone-assisted communication scenarios, and uses the Markov game framework to model the trajectory optimization and channel allocation decision of the multi-drone assisted communication system; the present invention proposes a multi-agent reinforcement learning strategy optimization method based on event-driven Transformer. The event-driven monitoring strategy optimization mechanism aims to learn the optimal event-driven listening strategy based on the current channel state, effectively reducing unnecessary information listening and processing of multiple drone base stations. The Transformer network extracts useful information and patterns from a large amount of state information, enabling the evaluation network of the drone base station to more accurately evaluate the state information, thereby realizing dynamic and intelligent channel allocation of multiple drone base stations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a schematic diagram of a drone-assisted communication system in one embodiment;

[0028] Figure 2 The invention relates to a timing decision process for allocating channels of several UAV base stations in a channel allocation method for multi-UAV base station assisted ground communication in an embodiment;

[0029] Figure 3 This is a block diagram of a multi-intelligence reinforcement learning model based on an event-driven Transformer in one embodiment;

[0030] Figure 4 The trajectory design and channel selection results of three UAV base stations in one embodiment are shown;

[0031] Figure 5 Schematic diagram of average reward curves for three drone base stations in an embodiment, where (a) is a schematic diagram of average reward curves for drone base station 1 using HAPPO, HATRPO, TransformerMATR, and EDGT-MARL, respectively; (b) is a schematic diagram of average reward curves for drone base station 2 using HAPPO, HATRPO, TransformerMATR, and EDGT-MARL, respectively; (c) is a schematic diagram of average reward curves for drone base station 3 using HAPPO, HATRPO, TransformerMATR, and EDGT-MARL, respectively;

[0032] Figure 6 1 is a box plot of the average rewards of three drone base stations in an embodiment, where (a) is the box plot of the average rewards of drone base station 1 using HAPPO, HATRPO, TransformerMATR and EDGT-MARL respectively; (b) is the box plot of the average rewards of drone base station 2 using HAPPO, HATRPO, Transformer MATR and EDGT-MARL respectively; (c) is the box plot of the average rewards of drone base station 3 using HAPPO, HATRPO, TransformerMATR and EDGT-MARL respectively. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0034] like Figures 1 to 3 As shown, this embodiment provides a channel allocation method for multi-UAV base station assisted ground communication, which is applied to a UAV assisted communication system. The schematic diagram of the UAV assisted communication system is shown in FIG. Figure 1 As shown in Figure 1, we first model the channel allocation problem for multiple UAV-assisted communications. In this system, M UAV base stations provide wireless communication services to N ground users, and multiple UAV base stations share L channels. The set of UAV base stations is defined as M = {1, 2, ..., M}, the set of ground users is defined as N = {1, 2, ..., N}, and the set of shared channels is defined as L = {1, 2, ..., L}.

[0035] In a dynamic assisted communication environment with multiple UAV base stations, each UAV base station makes sequential decisions by interacting with the unknown environment. This joint decision-making process is usually called a random game, or Markov game.

[0036] Therefore, the auxiliary communication of multiple UAV base stations is modeled as a Markov game and interacts with the environment according to the following process: at time t, based on the current state s t ∈S, drone base station i adopts strategy π i The joint action of multiple drone base stationsπ=(π 1 ,...,π n ) corresponds to the joint strategy Then, the drone gets the reward r t =r(s t ,a t ), and according to the probability P(s t+1 |s t ,a t )Transfer to the next state s t+1 .

[0037] In this UAV-assisted communication system, all ground users move randomly on the two-dimensional ground, and each UAV base station flies in three-dimensional space and uses frequency division multiplexing technology to provide wireless services to ground users.

[0038] Step 1: Initialize the joint strategy of multiple drone base stations Specifically, within each time slot, multiple drone base stations independently decide to pair with a ground user and select a channel from L shared channels. They then broadcast the pairing information and channel selection to other drone base stations. They then plan their trajectories in real time based on the location of the paired ground user and decide whether to listen to other drone base stations' broadcasts based on an event-driven strategy.

[0039] In order to improve the communication quality of the system and avoid resource waste, we assume that each ground user can only be paired with one UAV base station, and each UAV base station will serve at least one ground user.

[0040] In the assisted communication model, the coordinates of the UAV and the ground user are expressed as and in Among them, x m ,x n ∈(-100,100) and z m ∈(0,100).

[0041] First, the receiving power and channel loss of ground users are modeled as shown in formulas (1)-(3):

[0042] A statistical probability-based air-to-ground communication model is used. Furthermore, to simplify the calculation, it is assumed that there is no line of sight between the drone base station and the ground user, and a line-of-sight (LoS) link is used. Therefore, at time t, the total path loss in the LoS link from drone base station m to ground user n can be described as:

[0043]

[0044] in, is the instantaneous path loss from UAV base station m to ground user n in the LoS link, is the additional loss of free space transmission.

[0045]

[0046] Where, f c is the carrier frequency, is the straight-line distance from the drone base station m to the ground user n, and c is the speed of light. m,n (t) is the power allocated to ground user n by UAV base station m at time t.

[0047] Therefore, the received signal power available at terrestrial user n is:

[0048]

[0049] Then the motion model of the UAV base station is constructed, as shown in formulas (4)-(5):

[0050] Considering the flight safety of the drone base station, its minimum flight altitude is set to 5 meters.

[0051] In the Cartesian coordinate system, the velocity of the drone base station m is expressed as:

[0052]

[0053] Then the position update equation of the drone base station m is:

[0054]

[0055] Then, the pairing and channel selection model of the UAV base station is described as shown in formulas (6)-(7):

[0056] Ground users move randomly at a slower speed than the drone base station. At each time t, a ground user will be paired with a drone base station and provided with communication services. In addition, each drone base station must serve at least one ground user at time t.

[0057] Using binary variables σ m,n(t)∈{0,1} represents the connection relationship between the drone base station m and the ground user n. When the ground user n is paired with the drone base station m, σ m,n Set to 1 if yes, otherwise set to 0.

[0058]

[0059] Defining binary variables To control the channel selection of the drone base station. When the drone base station m selects channel l, is 1 if the value is set, otherwise it is 0.

[0060]

[0061] In the UAV-assisted communication system, the action of the UAV base station a m ∈A m Including the movement vector v m , vector vector1m for selecting whether to pair with each ground user, channel selection vector onehot2m, and event-driven monitoring strategy output event-driven policy , that is, a m =v m ,vector1m,onehot2m,event-driven policy The dimension of vector1m is the number of ground users N, and the dimension of onehot2m is the number of shared channels L. Event-driven strategy output event-driven policy Is a continuous value used as an event-driven gating threshold.

[0062] For the UAV base station m, the total power allocated is P total , according to the total number of paired ground users, the total bandwidth B total For uniform distribution, formula (8) and formula (9) give the mathematical description of power and bandwidth resource allocation:

[0063] P m,n (t) = P total (8)

[0064]

[0065] where s m (t) represents the number of ground users paired with the UAV base station m in time slot t.

[0066] Finally, a description of the communication rate and co-channel interference is given, as shown in formula (10) and formula (11).

[0067] Define n0 as the power spectral density, B m,n(t) is the bandwidth resource from UAV base station m to ground user n. Based on Shannon channel capacity theory, the communication rate of the link from UAV base station m to ground user n is:

[0068]

[0069] Where, is the co-channel interference generated by all other UAV base stations except UAV base station m on channel l to ground user n in time slot t, which is described as follows:

[0070]

[0071] Where h i,n (t) is the interference path loss from UAV base station i to ground user n on the LoS link, which is calculated in the same way as the path loss between UAV base station m and ground user n; P i is the interference power of UAV base station i.

[0072] In order to maximize the throughput of the multi-UAV assisted communication system while reducing co-channel interference, each UAV base station first needs to optimize the pairing strategy and channel strategy, and then optimize the flight trajectory and event-driven monitoring strategy. max As the time period of an episode, t is used as the time interval for determination:

[0073] t∈T={1,2,...,T max} (12)

[0074] Figure 2 The time-series decision-making process for a multi-UAV-assisted communication network is presented. At each time slot t, each UAV base station first needs to decide which ground user to pair with and allocate channels to the paired ground users. Specifically, UAV base stations select ground users for pairing based on their priority. For example, in an environment with three UAV base stations, UAV base station 1 has the highest priority and is the first to pair with a user; UAV base station 3 has the lowest priority and is the last to pair with a user.

[0075] In this paper, to reduce interference and improve communication quality, each ground user will be paired with only one drone base station. Each drone base station will serve at least one ground user to avoid wasting resources. Ground users already paired with a previous drone base station will not be paired with other drone base stations, thus avoiding duplicate pairings. Furthermore, each drone base station will select an unoccupied or lightly occupied channel based on current channel conditions to avoid co-channel interference. The drone base station will then broadcast pairing and channel selection information to all other drone base stations. Finally, each drone base station will adjust its trajectory in real time based on the current location of its paired ground user.

[0076] Step 2: Calculate the state value function V(s,a): For each UAV base station, the state information in each time slot includes the location of the paired ground user and the pairing and channel selection information of other UAV base stations.

[0077] In order to avoid excessive monitoring events, an excessive monitoring penalty operator ε'(t) is added to the reward mechanism to reduce the communication overhead of the drone base station.

[0078] ε'(t)=c1*Onehot event-driven (t) (13)

[0079] Among them, Onehot event-driven Is a one-hot vector indicating whether the monitoring event occurs. event-driven If Onehot is 1, a monitoring event occurs and the drone base station will monitor the broadcast information of all other drone base stations to assist its own decision-making; event-driven If it is 0, no monitoring event will occur and the drone base station will make decisions based only on its own local state information. c1 is the penalty coefficient.

[0080] The optimization objectives of several drone base stations are defined as:

[0081]

[0082] p m,n (t) = P total (19)

[0083]

[0084] b m,n (t)≥b min ,p m,n (t)≥p min ,n∈N (21)

[0085]

[0086] Where p m,n (t) and b m,n (t) are the transmission power and bandwidth resources allocated to ground user n by UAV base station m; b min is the minimum divisible bandwidth; p min is the minimum divisible power.

[0087] After modeling and optimizing the problem description, this paper proposes a novel event-driven transformer multi-agent reinforcement learning framework (EDT-MARL) to solve the trajectory design and channel allocation problems in multi-UAV assisted communication. This event-driven monitoring strategy learns the optimal monitoring time based on the current channel state, enabling the UAV base station to adaptively adjust the monitoring trigger threshold to decide whether to monitor global information. This reduces unnecessary information monitoring by the UAV base station and improves efficiency.

[0088] Furthermore, to avoid excessive event-driven monitoring, a new reward mechanism was designed that incorporates a weighted over-monitoring penalty into the drone base station's reward function, reducing its communication overhead. The drone base station uses the proposed method to optimize its trajectory design and channel allocation strategy within the communication network, thereby maximizing its cumulative reward. Furthermore, under the monotonic policy improvement guarantee of the proposed method, the drone base station gradually achieves joint policy convergence.

[0089] Most communication scenarios using multi-agent reinforcement learning rely on wireless communication to share observations. As the number of agents and the dimensionality of the data increase, communication performance will be affected by packet loss and communication latency. Furthermore, the bandwidth of real-world communication networks is limited. Therefore, minimizing the communication overhead between agents is crucial. To help achieve this, we have conducted extensive research on related work, among which event-driven communication is one of the most effective approaches. We propose a new event-driven monitoring policy optimization method to reduce unnecessary information monitoring by drone base stations.

[0090] Step 3: Calculate and compare event-driven trigger conditions Onehot trig Specifically, the action network of each drone base station will output an adaptive event-driven policy value event-driven policy , as the threshold for gating, such as Figure 3 Then, the event-driven strategy value is compared with the output value of the pre-evaluation network. For comparison, if event-driven policy Greater than value, then listen to the broadcasts of other drone base stations, otherwise.

[0091] Step 4: If the event-driven monitoring strategy is triggered, the monitored state information o(s,a) is encoded and the transformer value function V(q,κ,μ) is calculated, and then the advantage function is calculated

[0092] The specific calculation is as follows:

[0093]

[0094] Where q is the query; κ and μ form a key-value pair; d κ are the columns of the q and κ matrices, i.e. the vector dimensions.

[0095]

[0096] Step 5: Otherwise, the event-driven monitoring strategy is not triggered, which enables the UAV base station to learn the event-driven monitoring strategy from historical interaction data, thereby improving the communication efficiency of the multi-UAV assisted communication system.

[0097] Then, a lightweight Transformer network is introduced to extract useful information and patterns from a large amount of observation data, so that the evaluation network of the drone base station can more accurately evaluate the state information to improve its own decision-making. Compared with the classic Transformer network architecture, the lightweight Transformer network used in this invention eliminates the residual and normalization layers, thereby reducing the computational complexity. Figure 3 As shown in Figure 2, the input data of the lightweight Transformer network is the status information of the drone base station.

[0098] The Transformer, originally designed for machine translation tasks, is an encoder-decoder architecture in which the encoder maps an input sequence of tokens to a latent representation, and the decoder then generates the desired output sequence in an autoregressive manner. At each step, the Transformer takes all previously generated tokens as input. The attention mechanism with residual connections is one of the most important components of the Transformer, a technique used to build deep neural networks. Residual connections help alleviate the vanishing and exploding gradient problem in deep network training and simplify the network training process.

[0099] In the present invention, when the event-driven monitoring condition is activated, the drone base station needs to collect broadcast information from others to help itself make decisions. However, when the number of drone base stations is large, the dimension of information that needs to be processed is also exponential. Therefore, the present invention uses a lightweight Transformer model and embeds it into the Critic network of the drone base station, with the aim of extracting the most valuable information from the massive interactive information, while assisting the decision output of the drone base station Actor network. The input data of the lightweight Transformer model used in the present invention specifically includes

[0100] Figure 3 The framework of event-driven Transformer multi-agent reinforcement learning is given. Figure 3 As shown in the figure, based on the centralized training distributed execution (CTDE) structure, an event-driven monitoring optimization strategy is introduced as one of the output strategies of the Actor network. In this way, each drone base station can monitor the broadcast information of other drone base stations to learn the optimal event-driven monitoring strategy.

[0101] Each drone base station will pre-process the status information through the Transformer network before inputting it into the evaluation network.

[0102] Step 6: Calculate the output of the Critic network, that is, the Q value: The output Q value is fed back to the Actor network of the drone base station to assist in the strategy update of the drone base station.

[0103] Step 7: Update the policy Where C is the penalty coefficient, γ is the discount factor, is the approximate objective function, It is the KL divergence between the current strategy and the candidate strategy, which is used to measure the gap between the two strategies.

[0104] experiment

[0105] Specific experimental results: Figure 4 The movement trajectories and channel selection results of three drone base stations are given. Ground user 1 (3,2|3,1) means that ground user 1 is paired with drone base station 3 through channel 2 at the beginning, and paired with drone base station 3 through channel 1 at the end, and so on. Figure 5 The average reward curves of three drone base stations are given; Figure 6 The average reward box plots for the three drone base stations are shown; Table 1 shows the final rewards for the three drone base stations. It can be seen that the proposed method shows better policy convergence performance compared with benchmark algorithms such as Heterogeneous-Agent Proximal Policy Optimization (HAPPO), Heterogeneous-Agent Trust Region Policy Optimization (HATRPO), and Transformer Multi-agent Trust Region (Transformer MATR).

[0106] Table 1 Final rewards of three drone base stations

[0107]

[0108] It should be understood that, although the various steps in the above flow chart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0109] The above description is merely a preferred embodiment of the channel allocation method for multi-UAV base station-assisted ground communication disclosed in the present invention and is not intended to limit the scope of protection of the embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of this specification shall be included in the scope of protection of the embodiments of this specification.

Claims

1. A channel allocation method for multi-UAV base station assisted ground communication, characterized in that: The steps include: Obtain information about M drone base stations and N ground users in the drone-assisted communication system, and initialize the joint strategy of several drone base stations; Build a multi-agent reinforcement learning model based on an event-driven Transformer. The multi-agent reinforcement learning model includes states, drone base stations, actions, and rewards. The actions include movement vectors, vectors for selecting whether to pair with each ground user, channel selection vectors, and event-driven monitoring policy outputs. The state of the UAV base station in each time slot includes the location of the ground user currently paired with the UAV base station and the pairing and channel selection information of other UAV base stations; Input the state observation data of the drone base station in the previous time slot into the multi-intelligence reinforcement learning model and calculate the state value function V(s,a); Calculate and compare event-driven conditions. If the conditions are met, trigger the event-driven monitoring strategy to monitor and obtain the broadcasts of other drone base stations. The location of the ground user paired with the current drone base station and the pairing and channel selection information of other drone base stations obtained by monitoring are used as the input of the Transformer model. Output V(q,κ,μ) is used to calculate the advantage function based on V(q,κ,μ). If the condition is not met, the advantage function is calculated based on the position of the ground user paired with the current drone base station Based on the advantage function, the action data corresponding to the state observation data of the previous time slot is determined. The drone base station interacts with the simulation environment according to the action data, calculates the output of the Critic network, that is, the Q value, and feeds the Q value back to the Actor network of the drone base station to assist the drone base station in updating its strategy.

2. The channel allocation method for multi-UAV base station assisted ground communication according to claim 1 is characterized in that: The strategy for initializing a joint operation of multiple UAV base stations includes assuming that each ground user in the UAV-assisted communication system can only be paired with one UAV base station, and each UAV base station will serve at least one ground user; in each time slot, multiple UAV base stations independently and sequentially decide to pair with the ground user, select a channel from the shared channel, and broadcast the pairing information and channel selection to other UAV base stations; Several drone base stations will design trajectories in real time based on the location information of paired ground users, and decide whether to listen to the broadcasts of other drone base stations based on event-driven strategies.

3. The channel allocation method for multi-UAV base station assisted ground communication according to claim 1, characterized in that: The over-monitoring penalty operator ε'(t) is added to the reward mechanism of the multi-agent reinforcement learning model. The formula is as follows: ε’(t)=c1*Onehot event-driven (t) (13) Among them, Onehot event-driven It is a one-hot vector indicating whether the monitoring event occurs, and c1 is the penalty coefficient.

Images