A resource allocation method for UAV-assisted ultra-reliable low-latency communication

Abstract

This application provides a resource allocation method in UAV-assisted ultra-reliable low-latency communication, relating to the field of wireless communication technology. The method includes: acquiring a global state vector of a UAV-assisted ultra-reliable low-latency communication network; the global state vector representing the position information, remaining energy state, and channel gain information between each UAV and ground users in the UAV-assisted ultra-reliable low-latency communication network; extracting cross-UAV state coupling features from the global state vector through a multi-head self-attention layer in a trained policy network; determining a joint action strategy based on the state coupling features; the joint action strategy including UAV flight control parameters and ground user transmit power control parameters; and adjusting the flight state of each UAV and the transmit power of each ground user based on the joint action strategy to complete resource allocation. This method is used for precise joint optimization of UAV flight trajectories and ground user transmit power.

Description

Technical Field

[0001] This application relates to the field of wireless communication technology, and in particular to a resource allocation method in UAV-assisted ultra-reliable low-latency communication. Background Technology

[0002] As mobile communication technology evolves towards 5G-Advanced and 6G, building an integrated three-dimensional network encompassing air, space, and ground has become an inevitable trend. As an important component of non-terrestrial networks (NTNs), unmanned aerial vehicles (UAVs), with their advantages of high mobility, flexible deployment, and line-of-sight (LoS) transmission, can effectively extend the coverage of terrestrial networks, playing a crucial role, especially in emergency communication and temporary hotspot enhancement scenarios.

[0003] Ultra-Reliable and Low-Latency Communications (URLLC), as one of the three core service scenarios of 5G / 6G, places extremely high demands on end-to-end latency and transmission reliability (such as 99.999% reliability and millisecond-level latency). However, in actual UAV-assisted communication scenarios, the communication link exhibits highly dynamic time-varying characteristics due to the three-dimensional maneuvering of UAVs, and the flight and communication behavior of UAVs is subject to strict energy constraints due to the limited onboard battery capacity. Therefore, how to maximize the system's energy efficiency while ensuring the quality of URLLC service by jointly optimizing the UAV's flight trajectory, flight speed, and the user's transmission power is a challenging problem.

[0004] Currently, traditional methods for resource allocation and trajectory design in UAV communication networks mainly rely on convex optimization theory, game theory, or heuristic algorithms. While these methods can achieve good results in specific static scenarios, they typically require complex iterative calculations, are difficult to adapt to rapid changes in the wireless channel environment, and often only yield local optima, failing to meet the real-time requirements of URLLC services. Summary of the Invention

[0005] This application provides a resource allocation method for UAV-assisted ultra-reliable low-latency communication. It introduces a multi-head self-attention mechanism into a deep reinforcement learning policy network to extract cross-UAV state coupling features contained in the global state vector of the UAV-assisted ultra-reliable low-latency communication network, thereby achieving accurate joint optimization of UAV flight trajectory and ground user transmission power in a high-dimensional dynamic environment.

[0006] In a first aspect, this application provides a resource allocation method in UAV-assisted ultra-reliable low-latency communication (ULLP). The method includes: acquiring a global state vector of a UAV-assisted ULLP network; the global state vector representing the position information, remaining energy state, and channel gain information between each UAV and local ground users in the UAV-assisted ULLP network; extracting cross-UAV state coupling features from the global state vector through a multi-head self-attention layer in a trained policy network; determining a joint action strategy based on the state coupling features; the joint action strategy including UAV flight control parameters and ground user transmit power control parameters; and adjusting the flight state of each UAV and the transmit power of local ground users based on the joint action strategy to complete resource allocation.

[0007] Optionally, state coupling features across UAVs are extracted from the global state vector through a multi-head self-attention layer in the trained policy network. This includes: linearly embedding the state sub-blocks corresponding to each UAV in the global state vector to obtain initial embedding features; inputting the initial embedding features into an L-layer stacked multi-head self-attention layer, with each layer sequentially processed through linear mapping, multi-head self-attention calculation, residual connection and layer normalization, feedforward neural network, residual connection and layer normalization, to obtain the final hidden state of each UAV. The final hidden state is used to represent the state coupling features across UAVs; L is a positive integer greater than or equal to 2.

[0008] Optionally, the processing of each multi-head self-attention layer includes: receiving the output features of the previous multi-head self-attention layer as the input features of this layer; the input features of the first multi-head self-attention layer are the initial embedding features; linearly mapping the input features of this layer using the query weight matrix, key weight matrix, and value weight matrix respectively to obtain the query vector, key vector, and value vector of each UAV, and concatenating the query vector, key vector, and value vector of all UAVs to form the query matrix, key matrix, and value matrix respectively; dividing the query matrix, key matrix, and value matrix into multiple attention heads along the feature dimension, each attention head independently... The dot product of the query matrix and the transpose of the key matrix is ​​calculated, and the attention weights are obtained by normalizing the exponential function. The value matrix is ​​then weighted using these attention weights to obtain the output features of the attention head. The output features of all attention heads are concatenated and linearly projected to obtain the multi-head attention output. The multi-head attention output is then residually connected to the input features of this layer, and layer normalization is performed to obtain the intermediate features. The intermediate features are then input into a feedforward neural network consisting of two fully connected layers and Gaussian error linear unit activation functions. The output of the feedforward neural network is again residually connected to the intermediate features, and layer normalization is performed to obtain the output features of this layer.

[0009] Optionally, based on the state coupling characteristics, a joint action strategy is determined, including: performing average pooling on the state coupling characteristics to obtain a global feature vector; processing the global feature vector through a mean network and a log-standard deviation network respectively to generate the mean and log-standard deviation of the action distribution; and combining the noise sampled from the standard normal distribution with the mean and log-standard deviation to generate a joint action strategy.

[0010] Optionally, based on a joint action strategy, the flight status of each UAV and the transmission power of each ground user are adjusted to complete resource allocation, including: mapping the UAV flight control parameters and ground user transmission power control parameters to their respective preset constraint ranges through an activation function; determining the predicted position of the UAV at a future moment based on the mapped UAV flight control parameters and the current position of the UAV; adjusting the flight status of the UAV based on the predicted position; and controlling the ground user to send uplink data signals to the UAV at the mapped ground user transmission power.

[0011] Optionally, the policy network is trained as follows: A soft Actor-Critic framework is constructed; the soft Actor-Critic framework includes a policy network, two identical action value networks, a state value network, and a target state value network; in each time slot of the UAV-assisted ultra-reliable low-latency communication network, a joint action policy is generated and executed based on the current global state vector through the policy network, obtaining the immediate reward calculated based on the reward function and the global state vector of the next time slot, and storing the current global state vector, joint action policy, immediate reward, and global state vector of the next time slot as an experience tuple in the experience replay buffer; a batch of experience tuples is randomly sampled from the experience replay buffer, and for each experience tuple, based on the immediate reward and the target state... The state value network constructs a temporal difference target value based on the state value output by the global state vector in the next time slot. The parameters of the two action value networks are updated by minimizing the mean square error between the action values ​​output by each of the two action value networks and the temporal difference target value. The expected value of the squared error between the action values ​​output by each of the two action value networks and the temporal difference target value is calculated and used as the loss function for each of the two action value networks. The parameters of the two action value networks are updated by minimizing their respective loss functions. The parameters of the target state value network are softly updated based on the parameters of the state value network using an exponential moving average method. The parameters of the policy network are updated using a weighted sum of maximizing the minimum action value output by the two action value networks and the policy entropy as the loss function, resulting in the trained policy network.

[0012] Optionally, the reward function satisfies the following relationship: ; in, Indicates in time slot In global state Execute joint actions The instant rewards received; Indicates the drone index; Indicates the first A ground user index within the coverage area of ​​the drone; Indicates in time slot Ground users To drones The achievable uplink transmission rate; This represents the energy penalty weighting coefficient; Indicates drone In the time slot The energy consumption of flight propulsion; Indicates drone In the time slot Energy consumption of edge computing.

[0013] Optionally, the loss function satisfies the following relationship: ; in, The parameters representing the policy network; This represents the value of the loss function; Represents the mathematical expectation; This represents the experience replay buffer; This represents the global state vector sampled from the experience playback buffer; This represents a noise vector sampled from a standard normal distribution with a mean of 0 and a covariance of the identity matrix; This represents the entropy regularization coefficient; Indicates the state given The policy network then outputs a joint action policy. The logarithmic probability density; Indicates that the action value network is based on and The input is the value of the action output; This represents the parameters of the action value network.

[0014] Secondly, this application provides a resource allocation apparatus, which includes various functional modules for the method provided in the first aspect above.

[0015] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory storing instructions executable by the processor; the processor being configured to, when executing the instructions, cause the electronic device to perform the method provided in the first aspect above.

[0016] Fourthly, this application provides a computer-readable storage medium comprising: computer software instructions; when the computer software instructions are executed in an electronic device, the electronic device causes the electronic device to implement the method provided in the first aspect above.

[0017] Fifthly, this application provides a computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method provided in the first aspect above.

[0018] The resource allocation method for UAV-assisted ultra-reliable low-latency communication provided in this application can obtain the global state vector of the UAV-assisted communication network. The global state vector represents the position information, remaining energy state, and channel gain information between each UAV and ground users in the UAV-assisted ultra-reliable low-latency communication network. The position information, remaining energy state, and channel gain information between each UAV and ground users are fused into a complete system state representation. Then, the state coupling features across UAVs in the global state vector are extracted through the multi-head self-attention layer in the trained policy network. The multi-head self-attention mechanism is used to globally model the complex nonlinear dependencies between different UAVs in the high-dimensional heterogeneous state vector. Compared with traditional multilayer perceptron networks, it can more effectively capture the deep coupling features between UAVs and between UAVs and users caused by factors such as channel interference, position changes, and energy differences, thereby achieving more accurate state understanding in dynamic and time-varying communication environments. Subsequently, based on the state coupling features, a joint action strategy is determined. This joint action strategy includes both UAV flight control parameters and ground user transmit power control parameters, enabling flight trajectory planning and communication resource allocation to be optimized collaboratively rather than independently. Finally, based on the aforementioned joint action strategy, the flight status of each UAV and the transmission power of ground users are adjusted. Under the premise of meeting the extremely high requirements of ultra-reliable low-latency communication for end-to-end latency and transmission reliability, the system's energy efficiency is maximized by jointly optimizing flight energy consumption and communication energy consumption. This effectively solves the problem of real-time and accurate resource allocation in high-dimensional dynamic environments that is difficult to achieve in existing technologies. Attached Figure Description

[0019] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of the resource allocation method in UAV-assisted ultra-reliable low-latency communication provided in the embodiments of this application. Figure 2A flowchart illustrating the resource allocation method in UAV-assisted ultra-reliable low-latency communication provided in this application embodiment; Figure 3 Another flowchart illustrating the resource allocation method in UAV-assisted ultra-reliable low-latency communication provided in this application embodiment; Figure 4 This is a schematic diagram of the strategy network structure provided in an embodiment of this application; Figure 5 A schematic diagram of the training method for the policy network provided in this application embodiment; Figure 6 A schematic diagram of the composition of the resource allocation device provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the composition of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Unless the context otherwise requires, throughout the specification and claims, the term "comprise" and other forms such as the third-person singular "comprises" and the present participle "comprising" are interpreted as open-ended and encompassing, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiments," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this application. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples.

[0023] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0024] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0025] In addition, the use of “based on” implies openness and inclusivity, because processes, steps, calculations or other actions “based on” one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0026] As mobile communication technology evolves towards 5G-Advanced and 6G, building an integrated three-dimensional network encompassing air, space, and ground has become an inevitable trend. As an important component of non-terrestrial networks (NTNs), unmanned aerial vehicles (UAVs), with their advantages of high mobility, flexible deployment, and line-of-sight (LoS) transmission, can effectively extend the coverage of terrestrial networks, playing a crucial role, especially in emergency communication and temporary hotspot enhancement scenarios.

[0027] Ultra-Reliable and Low-Latency Communications (URLLC), as one of the three core service scenarios of 5G / 6G, places extremely high demands on end-to-end latency and transmission reliability (such as 99.999% reliability and millisecond-level latency). However, in actual UAV-assisted communication scenarios, the communication link exhibits highly dynamic time-varying characteristics due to the three-dimensional maneuvering of UAVs, and the flight and communication behavior of UAVs is subject to strict energy constraints due to the limited onboard battery capacity. Therefore, how to maximize the system's energy efficiency while ensuring the quality of URLLC service by jointly optimizing the UAV's flight trajectory, flight speed, and the user's transmission power is a challenging problem.

[0028] Currently, traditional methods for resource allocation and trajectory design in UAV communication networks mainly rely on convex optimization theory, game theory, or heuristic algorithms. While these methods can achieve good results in specific static scenarios, they typically require complex iterative calculations, are difficult to adapt to rapid changes in the wireless channel environment, and often only yield local optima, failing to meet the real-time requirements of URLLC services.

[0029] Based on this, this application provides a resource allocation method in UAV-assisted ultra-reliable low-latency communication, which introduces a multi-head self-attention mechanism into a deep reinforcement learning policy network to extract the cross-UAV state coupling features contained in the global state vector of the UAV-assisted ultra-reliable low-latency communication network, thereby achieving accurate joint optimization of UAV flight trajectory and ground user transmission power in a high-dimensional dynamic environment.

[0030] The following description is provided in conjunction with the accompanying drawings.

[0031] Figure 1 This is a schematic diagram illustrating an application scenario of the resource allocation method in UAV-assisted ultra-reliable low-latency communication provided in this application embodiment. For example... Figure 1 As shown, this application scenario can include three types of entities: drones, ground base stations, and ground users. Drones are equipped with Mobile Edge Computing (MEC) servers and batteries as their power supply unit. Ground base stations are equipped with base station servers. Ground base stations and drones are connected via drone-ground base station communication links, and ground users and drones are connected via user-drone communication links. Figure 1 The coverage area of ​​each drone is marked by an elliptical area, and ground users are distributed within the coverage area of ​​each drone.

[0032] The resource allocation method in UAV-assisted ultra-reliable low-latency communication provided in this disclosure is executed by a resource allocation device. This resource allocation device can be an electronic device with computing capabilities, such as a computer or server (e.g., the MEC server mounted on the aforementioned UAV). Alternatively, the resource allocation device can be a processor (e.g., a central processing unit, CPU) in the aforementioned electronic device; or, it can be an application (APP) with resource allocation capabilities installed in the aforementioned electronic device; or, it can be a software system or platform deployed in the aforementioned electronic device; or, it can be a functional module in the aforementioned electronic device used to execute the resource allocation method, etc. This application does not impose any limitations on these aspects.

[0033] For simplicity, the following description will use the resource allocation device as the execution subject of the resource allocation method in UAV-assisted ultra-reliable low-latency communication provided in the embodiments of this application.

[0034] Figure 2 This is a flowchart illustrating the resource allocation method in UAV-assisted ultra-reliable low-latency communication provided in an embodiment of this application. Figure 2 As shown, the method may include the following steps: S101. Obtain the global state vector of the UAV-assisted ultra-reliable low-latency communication network.

[0035] The global state vector is used to represent the location information, remaining energy status, and channel gain information between each UAV and local ground users in the UAV-assisted ultra-reliable low-latency communication network.

[0036] In some embodiments, prior to S101 above, a communication scenario for an unmanned aerial vehicle-assisted ultra-reliable low-latency communication network can also be constructed.

[0037] For example, it can be defined This represents a collection of drones, among which This represents the total number of drones, each equipped with an edge server; definition Indicates belonging to the first The system identifies a set of ground users within the coverage area of ​​the drone; it sets the ground base station as the central controller to collect global status information and make decisions; and it discretizes the system runtime into a set. ,in This represents the total number of time slots. The duration of each time slot.

[0038] Set all drones to a fixed altitude Flight within a two-dimensional horizontal plane; definition For drones In the time slot The horizontal position coordinates; the position update of the UAV follows the kinematic equations, i.e. as well as And flight speed The heading angle is limited by the maximum flight speed. and scope.

[0039] The communication link between the drone and the ground user is divided into LosS and non-line-of-sight (NLoS) links; based on the elevation angle between the drone and the user... Calculate the probability of line-of-sight transmission occurring. And calculate the non-line-of-sight transmission probability. The path loss under the two states is calculated based on the path loss model, and the instantaneous channel gain is calculated by weighting. .

[0040] Define user In the time slot The received signal-to-interference-plus-noise ratio (SINR) is: The interference term consists of co-channel interference from other users within the same service area and inter-channel interference from users in other service areas. The background noise is additive white Gaussian noise. To take advantage of the short packet transmission characteristics of URLLC, the information bits are encoded into a length of [length missing]. The base blocks are used to ensure the target block error rate. Based on the theory of Finite Block Length Coding (FBC), the reachable spectral efficiency of ground users is calculated. ,in The channel dispersion calculation formula is: (Inverse Gaussian Q-function) .

[0041] As an example, S101 above may specifically include: in each time slot, obtaining the position coordinates and remaining energy state of each UAV, as well as the channel state information between each UAV and local ground users, and calculating the reachable spectral efficiency of the users based on finite length coding theory; and constructing a global state vector for the current time slot based on the position coordinates, remaining energy state, channel state information, and reachable spectral efficiency.

[0042] For example, resource allocation devices can acquire drones. In the current time slot The flight speed is defined as the flight propulsion energy consumption of the UAV in the current time slot. The calculation formula is as follows: ,in and These are parameters related to the physical characteristics of the drone.

[0043] Define the edge computing energy consumption of a drone processing user-uploaded data within its coverage area in the current time slot as determined by... The calculation formula is as follows: ,in This represents the calculated energy efficiency coefficient related to the chip. Indicates system bandwidth. This indicates the computational frequency of the drone. This indicates the uplink achievable spectrum efficiency for the user.

[0044] Define drones The total energy consumption in the current time slot must not exceed its remaining energy, that is... ,in Indicates drone In the time slot The remaining energy state (e.g., remaining electrical charge).

[0045] The action vector of the current time slot t is defined by... This means that the motion vector includes the flight heading angle, flight speed of all UAVs, and the transmission power of the ground user, i.e. . Indicates the flight heading angle. Indicates flight speed. Indicates the transmission power of ground users. This represents the action space, which is the set of all possible action vectors.

[0046] Get the current time slot The state information of all UAVs is defined by the global state vector. It is indicated that its composition formula is as follows: This includes the drone's two-dimensional position, remaining energy state, and uplink channel gain between the user and the drone. Indicates drone In the time slot Horizontal two-dimensional position coordinates, Indicates in time slot drones Its ground users within its coverage area Uplink channel gain between Indicates the importance of drones All ground users within the coverage area will use the corresponding uplink channel gain. This means that for all UAVs, the above-mentioned horizontal two-dimensional position coordinates, remaining energy state, and uplink channel gain are taken. It represents the state space, which is the set of all possible global state vectors.

[0047] The global state vector can be divided into: Each drone corresponds to Each sub-block defines the characteristics for a single drone. The low-dimensional state sub-blocks are Its dimensions are The global state vector can be decomposed into: And the total dimension of the global state vector is .

[0048] S102. Extract cross-UAV state coupling features from the global state vector through the multi-head self-attention layer in the trained policy network.

[0049] As an example, Figure 3 This is another flowchart illustrating the resource allocation method in UAV-assisted ultra-reliable low-latency communication provided in this application embodiment. For example... Figure 3 As shown, the above S102 may specifically include the following steps: S1021. Perform linear embedding processing on the state sub-blocks corresponding to each UAV in the global state vector to obtain the initial embedding features.

[0050] For example, drones The initial hidden state (i.e., the initial embedded features) after linear projection is represented as: Its calculation formula is ,in Indicates drone The low-dimensional state sub-blocks, This represents the state projection matrix of the linear embedding layer. Represents the bias vector; S1022. Input the initial embedded features into a multi-head self-attention layer with L stacked layers. Each layer is processed sequentially through linear mapping, multi-head self-attention calculation, residual connection and layer normalization, feedforward neural network, residual connection and layer normalization to obtain the final hidden state of each UAV. The final hidden state is used to represent the state coupling features across UAVs.

[0051] Where L is a positive integer greater than or equal to 2. For example, L can be 2, 3, 4, or 5, etc. The embodiments of this application do not limit the specific value of L.

[0052] As an example, the processing of each multi-head self-attention layer may include the following steps: Step 1: Receive the output features of the previous multi-head self-attention layer as the input features of this layer; the input features of the first multi-head self-attention layer are the initial embedding features.

[0053] Step 2: For the input features of this layer, perform linear mapping using the query weight matrix, key weight matrix, and value weight matrix respectively to obtain the query vector, key vector, and value vector of each UAV. Then, concatenate the query vector, key vector, and value vector of all UAVs to form the query matrix, key matrix, and value matrix respectively.

[0054] For example, the first can be defined Drone in hidden state query vector Key vector Sum value vector The calculation formulas are as follows: ,in Indicates the first The input hidden state (i.e., input features) of the layer. They represent the first The layer's query, key, and value weight parameter matrix.

[0055] Step 3: Divide the query matrix, key matrix, and value matrix into multiple attention heads along the feature dimension. Each attention head independently calculates the dot product of the query matrix and the transpose of the key matrix. The attention weight is obtained by normalizing the exponential function. The value matrix is ​​then weighted with the attention weights to obtain the output feature of that attention head.

[0056] For example, the resource allocation device can concatenate the query vectors, key vectors, and value vectors of all UAVs to form a query matrix. Key matrix Sum matrix And it is segmented into feature dimensions. The first point of attention; targeting the first Each attention head defines its output features as... The calculation formula is as follows: ,in, This represents the embedding dimension of the multi-head self-attention layer, which is also the vector length of each hidden state. This represents the feature dimension of each attention head. This represents the matrix transpose operation. This represents the normalized exponential function. Indicates the first Layer The query matrix corresponding to each attention head. Indicates the first Layer The key matrix corresponding to each attention head Indicates the first Layer The value matrix corresponding to each attention head.

[0057] Step 4: Concatenate the output features of all attention heads and perform linear projection to obtain the multi-head attention output.

[0058] For example, you can follow the formula Calculate the multi-head attention output. Among them, Indicates the first The aggregated output of the multi-head self-attention layer Indicates the first The linear projection weight matrix of the output layer. This indicates a splicing operation. Indicates the first The layer outputs the bias vector of the layer.

[0059] Step 5: After performing residual connection between the multi-head attention output and the input features of this layer, perform layer normalization to obtain intermediate features.

[0060] For example, you can follow the formula Calculate intermediate features. Among them, Indicates the first The intermediate features (or intermediate hidden states) of the multi-head self-attention layer after the first residual connection and layer normalization. Presentation layer normalization operation, Indicates the first The input features (or input hidden states) of the multi-head self-attention layer.

[0061] Step 6: Input the intermediate features into the feedforward neural network consisting of two fully connected layers and Gaussian error linear unit activation functions. Then, perform residual connection and layer normalization on the output of the feedforward neural network and the intermediate features again to obtain the output features of this layer.

[0062] For example, you can follow the formula The output of the feedforward neural network is calculated, and residual connections and layer normalization are performed again to obtain the input features (or input hidden states) of the next layer. .in, Indicates the first Output features of a feedforward neural network with multiple self-attention layers. Represents the Gaussian error linear unit activation function. This represents the weight matrix of the first fully connected layer of the feedforward neural network. This represents the weight matrix of the second fully connected layer of the feedforward neural network. This represents the bias vector of the first fully connected layer of the feedforward neural network. This represents the bias vector of the second fully connected layer in the feedforward neural network. Indicates the first The input features (or input hidden states) of the multi-head self-attention layer.

[0063] As shown above, after L layers of multi-head self-attention layers, the final state coupling feature (or the final hidden state) can be represented as follows: .

[0064] S103. Determine the joint action strategy based on the state coupling characteristics.

[0065] The joint action strategy includes UAV flight control parameters and ground user transmit power control parameters.

[0066] As an example, S103 above may specifically include: performing average pooling on the state coupling features to obtain a global feature vector; processing the global feature vector through a mean network and a log-standard deviation network respectively to generate the mean and log-standard deviation of the action distribution; and combining the noise sampled from the standard normal distribution with the mean and log-standard deviation to generate a joint action strategy.

[0067] For example, you can follow the formula Calculate the global feature vector. Wherein, Represents the global feature vector. express The first layer after processing by the L-layer multi-head self-attention layer The final hidden state vector of the drone.

[0068] For example, the mean of the action distribution can be represented by a vector. This means that the logarithmic standard deviation can be represented by a vector. The calculation formulas are as follows: as well as .in, The parameters represent the policy network. This represents the weight matrix of a mean network. The weight matrix of the hourly log-standard deviation network. This represents the bias vector of the mean network. This represents the bias vector of the log-standard deviation network.

[0069] For example, you can follow the formula Generate a joint action strategy. Among them, Indicates in time slot The generated joint action policy vector (or action vector). This represents an exponential function with base e. This indicates an element-wise multiplication operation. This represents a noise vector sampled from a standard normal distribution with a mean of 0 and a covariance of the identity matrix.

[0070] S104. Based on the joint action strategy, adjust the flight status of each UAV and the transmission power of ground users in various locations to complete resource allocation.

[0071] As an example, S104 may specifically include: mapping the UAV flight control parameters and the ground user transmit power control parameters to their respective preset constraint ranges through an activation function; determining the predicted position of the UAV at a future time based on the mapped UAV flight control parameters and the current position of the UAV; adjusting the flight state of the UAV based on the predicted position; and controlling the ground user to send uplink data signals to the UAV at the mapped ground user transmit power.

[0072] For example, the unbounded parameters output by the policy network can be mapped to a preset range, i.e., the drone's flight speed, through an activation function. satisfy Flight heading angle satisfy Ground user transmission power satisfy .

[0073] For example, you can follow the formula Determine the predicted location of the drone at a future time. Among these, and Indicates drone In the time slot The predicted location coordinates, and Indicates drone In the time slot Location coordinates, It indicates the duration of a time slot.

[0074] In some embodiments, the resource allocation device may also perform energy constraint verification before each time slot to verify the propulsion energy consumption of each UAV in that time slot. Computational energy consumption The sum does not exceed the current remaining energy. That is, satisfying the constraints. If an action results in an excess of remaining energy, the action will be penalized or restricted.

[0075] Based on the understanding of the above embodiments, Figure 4 This is a schematic diagram of the policy network structure provided in an embodiment of this application. Figure 4 As shown, the policy network first receives the original state blocks as input, obtains the initial embedding features through a linear mapping layer, and then passes through L stacked multi-head self-attention layers, a feedforward neural network, and layer normalization to obtain the final hidden state features. The final hidden state features are then subjected to average pooling to obtain the global features h. global (That is, the global feature vector mentioned above), and output the action distribution by passing the mean and logarithmic standard deviation branches of the global feature.

[0076] The resource allocation method in UAV-assisted ultra-reliable low-latency communication provided in this application embodiment can obtain the global state vector of the UAV-assisted communication network. The global state vector is used to represent the position information, remaining energy state, and channel gain information between each UAV and ground users in the UAV-assisted ultra-reliable low-latency communication network. The position information, remaining energy state, and channel gain information between each UAV and ground users are fused into a complete system state representation. Then, the state coupling features across UAVs in the global state vector are extracted through the multi-head self-attention layer in the trained policy network. The multi-head self-attention mechanism is used to globally model the complex nonlinear dependencies between different UAVs in the high-dimensional heterogeneous state vector. Compared with traditional multilayer perceptron networks, it can more effectively capture the deep coupling features between UAVs and between UAVs and users caused by factors such as channel interference, position changes, and energy differences, thereby achieving more accurate state understanding in a dynamic and time-varying communication environment. Subsequently, based on the state coupling features, a joint action strategy is determined. This joint action strategy includes both UAV flight control parameters and ground user transmit power control parameters, enabling flight trajectory planning and communication resource allocation to be optimized collaboratively rather than independently. Finally, based on the aforementioned joint action strategy, the flight status of each UAV and the transmission power of ground users are adjusted. Under the premise of meeting the extremely high requirements of ultra-reliable low-latency communication for end-to-end latency and transmission reliability, the system's energy efficiency is maximized by jointly optimizing flight energy consumption and communication energy consumption. This effectively solves the problem of real-time and accurate resource allocation in high-dimensional dynamic environments that is difficult to achieve in existing technologies.

[0077] In some embodiments, prior to S101 described above, the resource allocation device may also acquire the trained policy network. Figure 5 This is a schematic diagram of the training method for the policy network provided in an embodiment of this application. Figure 5 As shown, the training process of the policy network can include the following steps: S201. Construct the soft Actor-Critic framework.

[0078] The soft Actor-Critic framework includes a policy network, two identical action value networks, a state value network, and a target state value network.

[0079] S202. In each time slot of the UAV-assisted ultra-reliable low-latency communication network, a joint action policy is generated and executed based on the current global state vector through the policy network. The immediate reward calculated based on the reward function and the global state vector of the next time slot are obtained. The current global state vector, the joint action policy, the immediate reward and the global state vector of the next time slot are stored as an experience tuple in the experience replay buffer.

[0080] As an example, the reward function satisfies the following relationship: ; in, Indicates in time slot In global state Execute joint actions The instant rewards received; Indicates the drone index; Indicates the first A ground user index within the coverage area of ​​the drone; Indicates in time slot Ground users To drones The achievable uplink transmission rate; This represents the energy penalty weighting coefficient; Indicates drone In the time slot The energy consumption of flight propulsion; Indicates drone In the time slot Energy consumption of edge computing.

[0081] S203. Randomly sample a batch of experience tuples from the experience replay buffer. For each experience tuple, construct the temporal difference target value based on the state value output by the immediate reward and the target state value network based on the global state vector of the next time slot.

[0082] For example, you can follow the formula Calculate the temporal difference objective value. Among them, Indicates the target value of time-series difference. Indicates the discount factor. Value network representing the target state Input the global state vector for the next time slot The state value is output later.

[0083] S204. Calculate the expected value of the squared error between the action value output by each of the two action value networks and the temporal difference target value, and use it as the loss function of each of the two action value networks. Then, update the parameters of the two action value networks by minimizing their respective loss functions.

[0084] For example, you can follow the formula Calculate the loss function of each of the two action value networks separately, and update the first action value network using gradient descent. Parameters and second action value network The parameters. Among them, The parameters representing the action value network, Taking 1 indicates The parameters, Taking 2 means The parameters, express The loss function value, Denotes the mathematical expectation, and represents the expectation from the experience replay buffer. Calculate the average value of a batch of empirical tuples sampled from the data. Represents the action value network (i.e. or ) in input and The value of the action output afterward.

[0085] For example, the parameters of the two action value networks can be updated using gradient descent based on their respective loss functions.

[0086] S205. The parameters of the target state value network are softly updated based on the parameters of the state value network using an exponential moving average method.

[0087] For example, you can follow the formula Update the parameters of the target state value network. Among them, The parameters representing the target value network, The parameters representing the state-value network, This represents the exponential moving average smoothing factor. This indicates an assignment operation, meaning that the result of the calculation on the right is assigned to the result on the left.

[0088] S206. Using the weighted sum of the minimum action value output by the two action value networks and the policy entropy as the loss function, update the parameters of the policy network to obtain the trained policy network.

[0089] As an example, the loss function of a policy network satisfies the following relationship: ; in, The parameters representing the policy network; This represents the value of the loss function; Represents the mathematical expectation; This represents the experience replay buffer; This represents the global state vector sampled from the experience playback buffer; This represents a noise vector sampled from a standard normal distribution with a mean of 0 and a covariance of the identity matrix; This represents the entropy regularization coefficient; Indicates the state given The policy network then outputs a joint action policy. The logarithmic probability density; Indicates that the action value network is based on and The input is the value of the action output; This represents the parameters of the action value network.

[0090] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, each device, such as a resource allocation device, includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the algorithm steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Experts may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0091] This application embodiment can divide the resource allocation device into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one functional module. The integrated module can be implemented in hardware or software. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the example of dividing each functional module according to each function.

[0092] In an exemplary embodiment, this application provides a resource allocation device. Figure 6 This is a schematic diagram illustrating the composition of a resource allocation device provided in an embodiment of this application. Figure 6 As shown, the resource allocation device may include an acquisition module 601 and a processing module 602.

[0093] The acquisition module 601 is used to acquire the global state vector of the UAV-assisted ultra-reliable low-latency communication network; the global state vector is used to represent the position information, remaining energy status and channel gain information between each UAV and local ground users in the UAV-assisted ultra-reliable low-latency communication network. The processing module 602 is used to extract state coupling features across UAVs from the global state vector through the multi-head self-attention layer in the trained policy network; determine a joint action strategy based on the state coupling features; the joint action strategy includes UAV flight control parameters and ground user transmit power control parameters; and adjust the flight state of each UAV and the transmit power of each ground user based on the joint action strategy to complete resource allocation.

[0094] In some embodiments, the processing module 602 is specifically used to perform linear embedding processing on the state sub-blocks corresponding to each UAV in the global state vector to obtain initial embedding features; input the initial embedding features into a multi-head self-attention layer with L stacked layers, and each layer sequentially undergoes linear mapping, multi-head self-attention calculation, residual connection and layer normalization, feedforward neural network, residual connection and layer normalization processing to obtain the final hidden state of each UAV, and the final hidden state is used to represent the state coupling features across UAVs; L is a positive integer greater than or equal to 2.

[0095] In some embodiments, the processing module 602 is specifically configured to receive the output features of the previous multi-head self-attention layer as the input features of this layer; the input features of the first multi-head self-attention layer are the initial embedding features; the input features of this layer are linearly mapped using the query weight matrix, key weight matrix, and value weight matrix respectively to obtain the query vector, key vector, and value vector of each UAV, and the query vector, key vector, and value vector of all UAVs are concatenated to form the query matrix, key matrix, and value matrix respectively; the query matrix, key matrix, and value matrix are divided into multiple attention heads along the feature dimension, and each attention head is independent. The dot product of the query matrix and the transpose of the key matrix is ​​calculated, and the attention weights are obtained by normalizing the exponential function. The value matrix is ​​then weighted using these attention weights to obtain the output features of the attention head. The output features of all attention heads are concatenated and linearly projected to obtain the multi-head attention output. The multi-head attention output is then residually connected to the input features of this layer, and layer normalization is performed to obtain the intermediate features. The intermediate features are then input into a feedforward neural network consisting of two fully connected layers and Gaussian error linear unit activation functions. The output of the feedforward neural network is again residually connected to the intermediate features, and layer normalization is performed to obtain the output features of this layer.

[0096] In some embodiments, the processing module 602 is specifically used to perform average pooling on the state coupling features to obtain a global feature vector; process the global feature vector through a mean network and a log-standard deviation network respectively to generate the mean and log-standard deviation of the action distribution; and combine the noise sampled from the standard normal distribution with the mean and log-standard deviation to generate a joint action strategy.

[0097] In some embodiments, the processing module 602 is specifically used to map the UAV flight control parameters and the ground user transmit power control parameters to their respective preset constraint ranges through an activation function; determine the predicted position of the UAV at a future time based on the mapped UAV flight control parameters and the current position of the UAV; adjust the flight state of the UAV based on the predicted position; and control the ground user to send uplink data signals to the UAV with the mapped ground user transmit power.

[0098] In some embodiments, the processing module 602 is further configured to construct a soft Actor-Critic framework; the soft Actor-Critic framework includes a policy network, two identical action value networks, a state value network, and a target state value network; in each time slot of the UAV-assisted ultra-reliable low-latency communication network, a joint action policy is generated and executed based on the current global state vector through the policy network, the immediate reward calculated based on the reward function and the global state vector of the next time slot are obtained, and the current global state vector, the joint action policy, the immediate reward, and the global state vector of the next time slot are stored as an experience tuple in the experience replay buffer; a batch of experience tuples are randomly sampled from the experience replay buffer, and for each experience tuple, the immediate reward and the target state are calculated and executed. The state value network constructs a temporal difference target value based on the state value output by the global state vector in the next time slot. The parameters of the two action value networks are updated by minimizing the mean square error between the action values ​​output by each of the two action value networks and the temporal difference target value. The expected value of the squared error between the action values ​​output by each of the two action value networks and the temporal difference target value is calculated and used as the loss function for each of the two action value networks. The parameters of the two action value networks are updated by minimizing their respective loss functions. The parameters of the target state value network are softly updated based on the parameters of the state value network using an exponential moving average method. The parameters of the policy network are updated using a weighted sum of maximizing the minimum action value output by the two action value networks and the policy entropy as the loss function, resulting in the trained policy network.

[0099] In some embodiments, the reward function satisfies the following relationship: ; in, Indicates in time slot In global state Execute joint actions The instant rewards received; Indicates the drone index; Indicates the first A ground user index within the coverage area of ​​the drone; Indicates in time slot Ground users To drones The achievable uplink transmission rate; This represents the energy penalty weighting coefficient; Indicates drone In the time slot The energy consumption of flight propulsion; Indicates drone In the time slot Energy consumption of edge computing.

[0100] In some embodiments, the loss function satisfies the following relationship: ; in, The parameters representing the policy network; This represents the value of the loss function; Represents the mathematical expectation; This represents the experience replay buffer; This represents the global state vector sampled from the experience playback buffer; This represents a noise vector sampled from a standard normal distribution with a mean of 0 and a covariance of the identity matrix; This represents the entropy regularization coefficient; Indicates the state given The policy network then outputs a joint action policy. The logarithmic probability density; Indicates that the action value network is based on and The input is the value of the action output; This represents the parameters of the action value network.

[0101] It should be noted that the above Figure 6 Modules in a module can also be called units; for example, a processing module can be called a processing unit. Additionally, in... Figure 6 In the embodiments shown, the names of the modules may not be the same as those shown in the figure. For example, the acquisition module may also be called the transceiver module or the communication module.

[0102] Figure 6 If the various modules in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of software products. These software products are stored in a storage medium and include several instructions to cause an electronic device or processor to execute all or part of the steps of the methods of the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] In an exemplary embodiment, this application also provides an electronic device. Figure 7 This is a schematic diagram illustrating the composition of an electronic device provided in an embodiment of this application. For example... Figure 7As shown, the electronic device includes a processor 702, a communication interface 703, and a bus 704. As an example, the electronic device may also include a memory 701.

[0104] Processor 702 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 702 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 702 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0105] The communication interface 703 is used to connect to other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0106] The memory 701 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0107] As one possible implementation, the memory 701 can exist independently of the processor 702. The memory 701 can be connected to the processor 702 via a bus 704 and is used to store instructions or program code. When the processor 702 calls and executes the instructions or program code stored in the memory 701, it can implement the resource allocation method in UAV-assisted ultra-reliable low-latency communication provided in the embodiments of this application.

[0108] In another possible implementation, the memory 701 can also be integrated with the processor 702.

[0109] The 704 bus can be an extended industry standard architecture (EISA) bus, etc. The 704 bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0110] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the resource allocation device can be divided into different functional modules to complete all or part of the functions described above.

[0111] In exemplary embodiments, this application also provides a computer-readable storage medium including computer software instructions that, when executed in an electronic device, cause the electronic device to implement the methods described in the above embodiments. The readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. Further, the readable storage medium can include both internal storage units and external storage devices of the electronic device. The readable storage medium is used to store the software instructions and other programs and data required by the electronic device. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0112] In an exemplary embodiment, this application also provides a computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the above method embodiments.

[0113] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer-executable instructions. When these computer-executable instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-executable instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, Bluetooth, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape) or an optical medium (e.g., DVD), etc.

[0114] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0115] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A resource allocation method in UAV-assisted ultra-reliable low-latency communication, characterized in that, The method includes: Obtain the global state vector of the UAV-assisted ultra-reliable low-latency communication network; The ultra-reliable low-latency communication network is achieved through a collection of unmanned aerial vehicles (UAVs). The drones in the middle provide assistance, Indicates the total number of drones; the first Ground user group within the coverage area of ​​the drone drones In the time slot The horizontal position coordinates are The drone's location update follows as well as ; The duration of each time slot, , This represents the total number of time slots. Indicates flight speed. Indicates the heading angle; the communication link between the UAV and the ground user includes line-of-sight (LAS) links and non-LAS links; The global state vector is composed of express, ; Indicates in time slot drones Its ground users within its coverage area Uplink channel gain between The path loss is calculated by weighting the path loss of line-of-sight links and non-line-of-sight links. Indicates drone In the time slot The remaining energy state, , Indicates drone The propulsion energy consumption for flight in the current time slot, , and For parameters related to the physical characteristics of the drone, This refers to the edge computing energy consumption of drones processing user-uploaded data within their coverage area. , This represents the calculated energy efficiency coefficient related to the chip. Indicates system bandwidth. Indicates the drone's computing frequency; Indicates in time slot Ground users To drones Uplink achievable speed , Indicates user In the time slot The received signal-to-interference-plus-noise ratio, It is an inverse Gaussian Q-function. Indicates the target block error rate. Indicates channel dispersion, ; Indicates the importance of drones The corresponding uplink channel gain is applied to all ground users within the coverage area; This means that for all UAVs, we take the horizontal two-dimensional position coordinates, remaining energy state, and uplink channel gain; Representing the state space, The set that includes all possible global state vectors; The cross-UAV state coupling features in the global state vector are extracted by the multi-head self-attention layer in the trained policy network; the establishment function in the policy network satisfies the following relationship: ； in, Indicates in time slot In the global state vector Execute joint actions The instant rewards obtained , Indicates the transmission power of ground users. Represents the action space. The set that includes all possible action vectors; Indicates the drone index; Indicates the first A ground user index within the coverage area of ​​the drone; This represents the energy penalty weighting coefficient; Based on the aforementioned state coupling characteristics, a joint action strategy is determined; the joint action strategy includes UAV flight control parameters and ground user transmit power control parameters. Based on the aforementioned joint action strategy, the flight status of each UAV and the transmission power of ground users in various locations are adjusted to complete resource allocation.

2. The method according to claim 1, characterized in that, The extraction of cross-UAV state coupling features from the global state vector through a multi-head self-attention layer in the trained policy network includes: The state sub-blocks corresponding to each UAV in the global state vector are linearly embedded to obtain the initial embedding features; The initial embedded features are input into a multi-head self-attention layer with L stacked layers. Each layer is processed sequentially through linear mapping, multi-head self-attention calculation, residual connection and layer normalization, feedforward neural network, residual connection and layer normalization to obtain the final hidden state of each UAV. The final hidden state is used to represent the state coupling features across UAVs; L is a positive integer greater than or equal to 2.

3. The method according to claim 2, characterized in that, The processing steps for each multi-head self-attention layer include: The output features of the previous multi-head self-attention layer are used as the input features of this layer; the input features of the first multi-head self-attention layer are the initial embedding features. For the input features of this layer, linear mapping is performed using the query weight matrix, key weight matrix, and value weight matrix respectively to obtain the query vector, key vector, and value vector of each UAV. The query vector, key vector, and value vector of all UAVs are then concatenated to form the query matrix, key matrix, and value matrix respectively. The query matrix, key matrix, and value matrix are divided into multiple attention heads along the feature dimension. Each attention head independently calculates the dot product of the query matrix and the transpose of the key matrix. The attention weight is obtained by normalizing the exponential function. The value matrix is ​​then weighted with the attention weight to obtain the output feature of that attention head. The output features of all attention heads are concatenated and linearly projected to obtain the multi-head attention output; After performing a residual connection between the multi-head attention output and the input features of this layer, layer normalization is performed to obtain intermediate features; The intermediate features are input into a feedforward neural network consisting of two fully connected layers and Gaussian error linear unit activation functions. The output of the feedforward neural network and the intermediate features are then subjected to residual connection and layer normalization to obtain the output features of this layer.

4. The method according to claim 1, characterized in that, The determination of the joint action strategy based on the state coupling characteristics includes: The state coupling features are subjected to average pooling to obtain a global feature vector; The global feature vectors are processed by a mean network and a log-standard deviation network respectively to generate the mean and log-standard deviation of the action distribution; The noise sampled from the standard normal distribution is combined with the mean and logarithmic standard deviation to generate the joint action strategy.

5. The method according to claim 1, characterized in that, The method of adjusting the flight status of each UAV and the transmission power of ground users based on the joint action strategy to complete resource allocation includes: The activation function maps the UAV flight control parameters and the ground user transmit power control parameters to their respective preset constraint ranges. Based on the mapped UAV flight control parameters and the UAV's current position, the predicted position of the UAV at future moments is determined. The flight status of the drone is adjusted based on the predicted location; The ground user is controlled to send uplink data signals to the UAV at the mapped ground user transmit power.

6. The method according to claim 1, characterized in that, The policy network is trained in the following way: Construct a soft Actor-Critic framework; the soft Actor-Critic framework includes the policy network, two action value networks with identical structures, a state value network, and a target state value network; In each time slot of the UAV-assisted ultra-reliable low-latency communication network, a joint action policy is generated and executed based on the current global state vector through the policy network. The immediate reward calculated based on the reward function and the global state vector of the next time slot are obtained. The current global state vector, the joint action policy, the immediate reward and the global state vector of the next time slot are stored as an experience tuple in the experience replay buffer. A batch of experience tuples are randomly sampled from the experience replay buffer. For each experience tuple, a temporal difference target value is constructed based on the immediate reward and the state value output by the target state value network based on the global state vector of the next time slot. The parameters of the two action value networks are updated by minimizing the mean square error between the action value output by each of the two action value networks and the temporal difference target value. The expected value of the squared error between the action value output by each of the two action value networks and the temporal difference target value is calculated and used as the loss function of each of the two action value networks. The parameters of the two action value networks are updated by minimizing their respective loss functions. The parameters of the target state value network are softly updated based on the parameters of the state value network using an exponential moving average method. The parameters of the policy network are updated using the weighted sum of the minimum action value output by the two action value networks and the policy entropy as the loss function, resulting in a trained policy network.

7. The method according to claim 6, characterized in that, The loss function satisfies the following relationship: ； in, The parameters representing the policy network; This represents the value of the loss function; Represents the mathematical expectation; This represents the experience replay buffer; This represents the global state vector sampled from the experience playback buffer; This represents a noise vector sampled from a standard normal distribution with a mean of 0 and a covariance of the identity matrix; This represents the entropy regularization coefficient; Indicates the state given The policy network then outputs a joint action policy. The logarithmic probability density; Indicates that the action value network is based on and The input is the value of the action output; This represents the parameters of the action value network.

8. An electronic device, characterized in that, The electronic device includes: a memory and a processor; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device performs the method as described in any one of claims 1-7.

9. A computer program product, characterized in that, include: Computer instructions; When the computer instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-7.

Images