Unmanned aerial vehicle communication resource optimization method and system based on preview state collaborative decision making
By constructing dynamic heterogeneous graphs and extracting topological embeddings using graph neural networks, and combining continuous and discrete actions with reinforcement learning models, the hybrid optimization problem of UAV trajectory, user association, and RIS phase configuration in unmanned aerial vehicle (UAV) communication systems was solved, thereby improving system performance and stability.
Patent Information
- Application Number
- CN202611588154.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-14
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies struggle to effectively address the continuous-discrete hybrid optimization problem in UAV communication systems, which involves UAV trajectory, user association, NOMA power allocation, and RIS phase configuration. This is especially true under conditions of high-speed vehicle movement and imperfect channel state information, where temporal state mismatch and multi-agent non-stationarity lead to insufficient system performance.
A preview-state-based collaborative decision-making method is adopted. By constructing a dynamic heterogeneous graph and extracting topological embeddings using a graph neural network, and combining the outputs of continuous and discrete actions from the first and second reinforcement learning models, collaborative optimization of UAV trajectory control, power allocation, and RIS phase configuration is achieved. A hybrid reward function is constructed for joint training.
It effectively solves the problem of continuous-discrete hybrid optimization, improves the energy efficiency and transmission rate of the air-ground integrated vehicle network system, improves the system's policy convergence and training stability, and reduces the learning difficulty of the hybrid action space.
Smart Images

Figure CN123075498A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the intersection of wireless communication and artificial intelligence, and specifically to a method and system for optimizing UAV communication resources based on preview state collaborative decision-making. Background Technology
[0002] As 6G mobile communication technology evolves towards higher density, higher mobility, and lower latency in vehicle-to-everything (V2X) networks, the coverage reliability and user access stability of V2X networks in urban environments face severe challenges. Reconfigurable Intelligent Surfaces (RIS) mounted on Unmanned Aerial Vehicles (UAVs) can effectively improve air-to-ground link quality and enhance network deployment flexibility. However, due to limitations such as airborne energy constraints, the high-speed movement characteristics of vehicles, and the residual serial interference cancellation (SIC) error in Non-Orthogonal Multiple Access (NOMA), user association, UAV trajectory design, NOMA power allocation, and RIS phase configuration in the system constitute a strongly coupled continuous-discrete hybrid optimization problem.
[0003] Existing methods mainly employ deep reinforcement learning for online optimization, but they suffer from the following shortcomings: value-based methods are suitable for discrete actions but struggle to handle continuous control; policy gradient methods are suitable for continuous actions but struggle to efficiently handle discrete RIS codebook selection; in hierarchical hybrid action methods, if the RIS still performs phase selection based on the state before the action execution after the UAV's continuous actions change the network topology, it will lead to significant temporal state mismatch; existing reinforcement learning methods based on graph neural networks (GNNs) still have shortcomings in dynamic correlation modeling in RIS-UAV-NOMA systems.
[0004] Therefore, existing technologies mainly suffer from the following problems: UAV trajectory, user association, NOMA power allocation, and RIS phase configuration are highly coupled, making it difficult to handle continuous-discrete hybrid decision-making problems; high-speed vehicle movement leads to rapid changes in channel state, making it difficult to achieve stable optimization under imperfect channel state information; existing hierarchical reinforcement learning methods cannot effectively coordinate the temporal dependencies between continuous UAV control and discrete RIS phase selection, easily leading to state mismatch; and there is significant multi-agent non-stationarity in multi-UAV collaborative scenarios, resulting in insufficient convergence and stability of traditional deep reinforcement learning methods. Summary of the Invention
[0005] The purpose of this invention is to provide a method and system for optimizing UAV communication resources based on preview state collaborative decision-making. It can output continuous actions through a first reinforcement learning model, construct a preview state based on the continuous actions, and use a second reinforcement learning model to select reconfigurable intelligent surface phase configurations from a finite codebook according to the preview state. This enables temporal collaborative decision-making between continuous and discrete actions, thereby effectively solving the problem of continuous-discrete hybrid optimization and the temporal state mismatch between UAV continuous control and reconfigurable intelligent surface discrete decision-making, and improving the energy efficiency and transmission rate of the air-ground integrated vehicle network system.
[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On the one hand, this invention provides a method for optimizing UAV communication resources based on collaborative decision-making in preview state, including the following steps: Construct an air-ground integrated vehicle-to-everything (V2X) system consisting of base stations, RIS-UAVs, and ground users; the RIS-UAVs consist of unmanned aerial vehicles (UAVs) and their reconfigurable smart surfaces; model the base stations, RIS-UAVs, and ground users as graph nodes, construct the edges between the graph nodes, and form a dynamic heterogeneous graph with the graph nodes and the edges; The dynamic heterogeneous graph is encoded using a graph neural network, and node-level topological embedding and global-level topological embedding are obtained through a graph attention mechanism. The association between the ground users and RIS-UAV is corrected based on the node-level topology embedding, and a service cluster is formed in combination with capacity constraints; The first reinforcement learning model is used to output continuous actions, which include UAV trajectory control parameters and the original power allocation parameters of ground users within the service cluster. A preview state is constructed based on continuous actions. The preview state includes the preview position, the preview equivalent channel, and the global-level topology embedding. A second reinforcement learning model is used to output discrete actions, the discrete actions including codebook indices selected from a finite codebook, the codebook indices corresponding to reconfigurable smart surface phase configurations, and the decision of the discrete actions is based on the preview state; A hybrid reward function consisting of global and local rewards is constructed. Based on the hybrid reward function, the first reinforcement learning model and the second reinforcement learning model are jointly trained and their parameters are updated to achieve collaborative optimization of the UAV trajectory control parameters, the original power allocation parameters, and the reconfigurable intelligent surface phase configuration.
[0007] Furthermore, a graph neural network is used to encode the dynamic heterogeneous graph, and node-level topological embeddings and global-level topological embeddings are obtained through a graph attention mechanism. The specific steps are as follows: The edge weights between nodes in the graph are obtained using the following formula: ; Where n is the time slot number, and u and v are the graph node numbers. Let be the edge weight between graph node u and graph node v in time slot n. This is the distance weighting coefficient. Let be the distance between graph node u and graph node v in time slot n. This is the distance normalization factor. This is the link quality weight coefficient. The normalized link quality for time slot n; The edge weights of each time slot within the preset historical window length are exponentially decayed and smoothed to obtain the historical fusion edge weights; The first node of the central diagram Layer hidden features and the first neighbor graph node Layer implicit features are concatenated and multiplied with the historical fusion edge weights of the central graph node and its neighboring graph nodes. After normalization, the result is the first layer. Attention coefficient of layer The initial value is 1; Based on the first The attention coefficient of the layer affects the node of the neighbor graph. The hidden features of the layers are weighted and aggregated, and after nonlinear activation, the first node of the central graph is obtained. Hidden features of layers; Through multi-layer propagation with a preset number of layers L, the latent features of each graph node in the Lth layer are used as the node-level topological embeddings of each graph node, and the node-level topological embeddings of all graph nodes are aggregated by global mean to obtain the global-level topological embedding.
[0008] Furthermore, the association between the ground user and the RIS-UAV is corrected based on the node-level topology embedding; the specific steps are as follows: Based on the node-level topology embedding, a correlation score between ground users and RIS-UAV is constructed, and the formula for the correlation score is as follows: ; Where i is the ground user ID, and k is the RIS-UAV ID. The association score between the i-th ground user and the k-th RIS-UAV in time slot n is given. , For the i-th ground user, the transpose of the node-level topology embedding in time slot n. For the node-level topology embedding of the k-th RIS-UAV in time slot n, For the i-th ground user in time slot n, the node-level topology embedding is performed. For the Euclidean norm, The value should be a very small positive number to prevent the denominator from being 0; The effective distance from the ground user to the service cluster center is calculated based on the correlation score, and the formula for the effective distance is as follows: ; in, Let be the effective distance from the i-th ground user to the service cluster center corresponding to the k-th RIS-UAV. Let i be the horizontal position of the i-th ground user in time slot n. For the service cluster center corresponding to the k-th RIS-UAV, The square of the Euclidean norm; The ground user association variables are updated based on the effective distance, and the update formula for the ground user association variables is as follows: ; in, This is a ground user-related variable. It takes the value of 1 when the i-th ground user is assigned to the k-th RIS-UAV in time slot n, and otherwise takes the value of 0. For RIS-UAV sets, For the i-th ground user to the i-th The effective distance to the service cluster center corresponding to each RIS-UAV; argmin is the RIS-UAV number found in the RIS-UAV set that minimizes the effective distance; Ground users are assigned to service cluster centers based on the effective distance, and the service cluster centers are updated with weighted scores using the correlation scores. The formula for the service cluster centers is as follows: ; Where M represents the total number of ground users; Repeat the calculation of effective distance, the update of ground user related variables and the weighted update of service cluster center until the change in service cluster center is less than the preset threshold or the preset maximum number of iterations is reached.
[0009] Furthermore, the service clusters are updated periodically, with the time slot interval between two adjacent updates being a preset periodic value. In each updated service cluster, when the number of ground users in the service cluster exceeds a preset capacity limit, the ground user with the lowest association score is transferred to another service cluster with available capacity. When the number of ground users in the service cluster is lower than a preset capacity limit, the ground user with the highest association score is added from an adjacent service cluster.
[0010] Furthermore, the first reinforcement learning model is a policy-evaluation architecture, including a policy network and a dual evaluation network. The dual evaluation network includes a first evaluation network and a second evaluation network. The first evaluation network and the second evaluation network output a first Q-value and a second Q-value, respectively, and the training target is constructed using the smaller of the first Q-value and the second Q-value. The first reinforcement learning model adopts a centralized training and distributed execution framework. During the training phase, the joint state composed of the local states of all UAVs is used as the input of the first evaluation network and the second evaluation network. During the execution phase, each UAV outputs the continuous action only based on its own local state through the policy network.
[0011] Furthermore, the policy network outputs the original power allocation parameters, and after taking a positive value for the original power allocation parameters, it divides them by the sum of the positive values of all ground users in the service cluster to obtain the non-orthogonal multiple access power allocation parameters.
[0012] Furthermore, the preview state is constructed based on continuous actions, and the specific steps are as follows: Based on the continuous action, the horizontal displacement control quantity in the continuous action is obtained, and the horizontal displacement control quantity is the trajectory control parameter of the UAV. The preview position is calculated based on the current position of the UAV and the horizontal displacement control amount. The formula for the preview position is as follows: ; in, For the k-th drone in time slot Preview location, For projection operations on the flyable area of the drone, For the k-th drone in time slot Current location Let be the horizontal displacement control value of the k-th UAV; Based on the preview location, the estimated channel from the base station to the reconfigurable smart surface mounted on the UAV, and the estimated channel from the reconfigurable smart surface mounted on the UAV to the ground user are recalculated. The formula for the preview equivalent channel is as follows: ; in, Let $\frac{ ... For base stations The estimated direct channel to the i-th ground user Let be the conjugate transpose of the estimated channel vector from the reconfigurable smart surface mounted on the k-th UAV to the i-th ground user at the preview location. This is the conjugate transpose operation. For time slots Reconfigurable smart surface phase matrix, The estimated channel from the base station to the reconfigurable smart surface mounted on the k-th UAV at the preview location.
[0013] The global-level topology embedding is obtained, and the preview state includes the preview location, the preview equivalent channel, and the global-level topology embedding.
[0014] Furthermore, the second reinforcement learning model is a dual Q-network, which separates the state value function from the action advantage function, and the formula for outputting the Q-value is as follows: ; in, In preview mode Select codebook index below Q value, For the network parameters of the dual Q-network, For the state value function, The network parameters are the state value function. For the action advantage function, Here, Q represents the network parameters for the action advantage function, and Q is the codebook capacity of the finite codebook. For summation index; The finite codebook is generated as follows: based on the reconfigurable smart surface array structure and the number of phase quantization bits, a two-dimensional discrete Fourier transform beam grid is constructed to generate basic codewords. The phase of each reflection unit is discretized in combination with the number of phase quantization bits. The final finite codebook is obtained by limiting the codebook capacity and removing redundant codewords.
[0015] Furthermore, the global reward is composed of a weighted sum of the normalized value of the average total rate, the normalized value of energy efficiency, the Jain fairness index, and the constraint violation penalty term. The average ground user rate is calculated based on the non-orthogonal multiple access power allocation parameters. The constraint violation penalty term includes service quality constraint violation penalty, energy budget violation penalty, UAV safe distance violation penalty, UAV speed constraint violation penalty, and flight area boundary violation penalty. The local reward is calculated based on the service cluster rate gain of the current time slot relative to the previous time slot, and the phase switching cost is subtracted after a logarithmic transformation of the non-negative part of the service cluster rate gain. The phase switching cost is used to constrain the change range of the reconfigurable smart surface phase configuration between adjacent time slots, and the formula for the phase switching cost is as follows: ; in, Let n be the phase switching cost of the reconfigurable smart surface mounted on the k-th UAV in time slot n. denoted by m, which represents the total number of reflective elements on the reconfigurable smart surface mounted on the UAV, where m is the reflective element number. For modulo operation, The phase angle, Let be the complex phase coefficient of the m-th reflective element of the reconfigurable smart surface mounted on the k-th UAV in time slot n. For conjugate operations; The hybrid reward function is a weighted fusion of the global reward and the local reward.
[0016] On the other hand, the present invention provides a UAV communication resource optimization system based on preview state collaborative decision-making, specifically including: The topology perception and user association module is used to construct a dynamic heterogeneous graph of the air-ground integrated vehicle network system composed of base stations, RIS-UAVs and ground users. The RIS-UAVs are composed of UAVs and their reconfigurable smart surfaces. The module uses graph neural networks to extract node-level topology embeddings and global-level topology embeddings, and corrects the association between ground users and RIS-UAVs based on the node-level topology embeddings. It also forms service clusters by combining capacity constraints. The continuous action decision module is communicatively connected to the topology perception and user association module. It is used to receive the service cluster and optimize the continuous actions using a first reinforcement learning model. The continuous actions include UAV trajectory control parameters and the original power allocation parameters of ground users within the service cluster. A preview state construction module is communicatively connected to the continuous action decision module and is used to construct a preview state based on the continuous action. The preview state includes a preview position, a preview equivalent channel, and the global-level topology embedding. The discrete action decision module is communicatively connected to the preview state construction module. It is used to receive the preview state and select a codebook index from a finite codebook using a second reinforcement learning model. The codebook index corresponds to a reconfigurable smart surface phase configuration. The hybrid reward calculation module is communicatively connected to the discrete action decision module, and is used to receive the codebook index output by the discrete action decision module and construct a hybrid reward function consisting of global reward and local reward. The joint training optimization module is communicatively connected to the continuous action decision module, the discrete action decision module, and the hybrid reward calculation module, respectively, and is used to perform joint training and parameter updates on the first reinforcement learning model and the second reinforcement learning model based on the hybrid reward function.
[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention constructs a dynamic heterogeneous graph and uses a graph neural network to extract node-level and global-level topology embeddings, which can effectively characterize the network structure and time-varying characteristics in the air-ground integrated vehicle network environment, improve the state expression capability of the air-ground integrated vehicle network system under imperfect channel state information conditions, and provide more accurate topology information for subsequent ground user association and communication resource optimization.
[0018] This invention enables the discrete phase selection of reconfigurable smart surfaces to make decisions based on the state updates of continuous UAV actions by constructing a preview state, effectively alleviating the temporal state mismatch problem between the continuous actions and the discrete actions in existing methods.
[0019] This invention employs a first reinforcement learning model to output continuous actions and a second reinforcement learning model to output discrete actions, combined with a preview state to achieve sequential collaborative decision-making, effectively reducing the learning difficulty of the mixed action space and improving the efficiency of policy search.
[0020] This invention improves policy convergence and training stability in multi-UAV collaborative scenarios by using a hierarchical fusion mechanism of global and local rewards. This mechanism enables each UAV to perceive the direct impact of its own local actions during training while also taking into account the global objectives of the air-ground integrated vehicle network system.
[0021] This invention models the reconfigurable smart surface phase configuration as a discrete action decision problem of selecting a codebook index from a finite codebook. It performs discretization processing by combining the number of phase quantization bits and introduces a phase switching cost constraint to constrain the change range of the reconfigurable smart surface phase configuration between adjacent time slots. This reduces the control overhead of the reconfigurable smart surface and avoids the instability of the air-ground fusion vehicle network system caused by drastic phase jumps. Attached Figure Description
[0022] Figure 1 This is a flowchart of the present invention.
[0023] Figure 2 This is a model diagram of the air-ground integrated vehicle networking system of the present invention.
[0024] Figure 3 This is a performance comparison chart of the embodiments of the present invention with other methods.
[0025] Figure 4 Performance comparison chart of the embodiments of the present invention and the scheme without preview state. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.
[0027] like Figure 1 As shown, the UAV communication resource optimization method based on preview state collaborative decision-making is characterized by the following steps: An air-ground integrated vehicle network system is constructed, consisting of a base station, a RIS-UAV, and ground users. The RIS-UAV is composed of a drone and its reconfigurable smart surface. The base station, RIS-UAV, and ground users are modeled as graph nodes, and edges are constructed between the graph nodes and the edges to form a dynamic heterogeneous graph.
[0028] The dynamic heterogeneous graph is encoded using a graph neural network, and node-level topological embeddings and global-level topological embeddings are obtained through a graph attention mechanism.
[0029] The association between the ground users and RIS-UAV is corrected based on the node-level topology embedding, and a service cluster is formed by combining capacity constraints.
[0030] The first reinforcement learning model is used to output continuous actions, which include UAV trajectory control parameters and the original power allocation parameters of ground users within the service cluster.
[0031] A preview state is constructed based on continuous actions. The preview state includes the preview position, the preview equivalent channel, and the global-level topology embedding.
[0032] A second reinforcement learning model is used to output discrete actions, which include codebook indices selected from a finite codebook, the codebook indices corresponding to reconfigurable smart surface phase configurations, and the decision of the discrete actions is based on the preview state.
[0033] A hybrid reward function consisting of global and local rewards is constructed. Based on the hybrid reward function, the first reinforcement learning model and the second reinforcement learning model are jointly trained and their parameters are updated to achieve collaborative optimization of the UAV trajectory control parameters, the original power allocation parameters, and the reconfigurable intelligent surface phase configuration.
[0034] like Figure 2As shown, the air-to-ground integrated vehicle-to-everything (V2X) system utilizes multiple UAVs equipped with reconfigurable smart surfaces to assist downlink non-orthogonal multiple access communication. The system comprises a base station, K RIS-UAVs (Reconfigurable Integrated Vehicle-to-Availables), and M ground users. Each RIS-UAV consists of a UAV and its mounted reconfigurable smart surface, with a one-to-one correspondence between each RIS-UAV and the UAV. The k-th RIS-UAV corresponds to the k-th UAV (i.e., a rotary-wing UAV equipped with a passive reconfigurable smart surface). The reconfigurable smart surface mounted on the UAV acts as an aerial reconfigurable reflection platform, dynamically adjusting the phase response of the reconfigurable smart surface to improve the equivalent propagation environment between the base station and ground users. Let the set of UAVs be denoted as . The ground user set is The total operating time of the air-ground integrated vehicle-to-everything (V2X) system is divided into N equal-length time slots, each time slot having a length of [missing information]. The time slot set is denoted as To reduce modeling complexity, all UAVs fly at a fixed altitude H. The current position of the k-th UAV in time slot n is... Then the trajectory of the k-th UAV during the entire mission cycle can be: Ground users move dynamically within the horizontal plane. The horizontal position of the i-th ground user in time slot n can be... In the graph modeling process, the base station, the RIS-UAV, and the ground user are each modeled as graph nodes, and edges are constructed between the graph nodes and the edges to form a dynamic heterogeneous graph.
[0035] For ground user association, this invention employs a dynamic ground user clustering strategy. To determine whether the i-th ground user is assigned to the k-th RIS-UAV service in time slot n, i.e. Each ground user is served by only one RIS-UAV in the same time slot. The k-th UAV and its k-th reconfigurable smart surface together constitute the k-th RIS-UAV and correspond to the k-th service cluster. K is the total number of RIS-UAVs, which is also the total number of UAVs and service clusters. Ground users within a service cluster share time-frequency resources and access the network through power domain multiplexing. Ground user association is dynamically adjusted based on location, estimated channel, and service load.
[0036] To improve the quality of air-to-ground propagation links, each drone is equipped with a [device / system]. A reconfigurable smart surface composed of reflective units. Since the reconfigurable smart surface used in this invention is passive, the reflection amplitude of each reflective unit is set to 1. The phase response matrix of the reconfigurable smart surface carried by the k-th UAV in time slot n. for: ;in, For diagonal matrix operations, It is a natural constant. Imaginary unit, Let m be the phase of the m-th reflective unit in the reconfigurable smart surface carried by the k-th UAV in time slot n. .
[0037] Given that practical reconfigurable smart surfaces are limited by finite phase resolution, hardware quantization accuracy, and online control complexity, this invention does not directly optimize the phase of the reconfigurable smart surface in continuous phase space. Instead, it models the phase configuration of the reconfigurable smart surface as a finite codebook selection problem. The finite codebook is generated as follows: based on the reconfigurable smart surface array structure and the number of phase quantization bits, a two-dimensional discrete Fourier transform beam grid is constructed to generate basic codewords. The phase of each reflecting unit is discretized based on the number of phase quantization bits. The final finite codebook is obtained through codebook capacity constraints and redundant codeword removal. The finite codebook (used to configure the phase of the reconfigurable smart surface) is... Where Q is the codebook capacity of the finite codebook. Let be the phase matrix of the t-th predefined reconfigurable smart surface, where t ranges from 1 to Q. Let be the codebook index selected from the finite codebook in time slot n for the k-th reconfigurable smart surface. According to the codebook index, the phase matrix of the k-th reconfigurable smart surface in time slot n. Obtained by looking up the table: , For the first A predefined reconfigurable smart surface phase matrix.
[0038] In each time slot n, the base station allocates downlink transmit power to different service clusters based on ground user association results and power control strategies, and constructs a non-orthogonal multiple access (NMO) superimposed signal within each service cluster. For the k-th service cluster, the NMO superimposed signal is: ;in, The signal is the superimposed signal of the k-th service cluster in time slot n. Let n be the set of ground users who receive reconfigurable smart surface services carried by the k-th UAV within time slot n. The total transmit power of the k-th service cluster, The proportion of non-orthogonal multiple access power allocated to the i-th ground user (i.e., non-orthogonal multiple access power allocation parameter). Normalized information symbols.
[0039] To address the line-of-sight (LoS) propagation and non-line-of-sight (NLoS) scattering characteristics in complex urban environments, this invention employs a Rician fading model to describe three types of links: a direct link from the base station to a ground user, a link from the base station to a reconfigurable smart surface mounted on a UAV, and a link from the reconfigurable smart surface to a ground user. The direct link from the base station to the i-th ground user can be: ;in, This is the direct connection channel from the base station to the i-th ground user. The path loss at a reference distance (e.g., 1m) Let be the distance from the base station to the i-th ground user. This is the path loss index for directly connected links. Rice factor for direct links, For the deterministic component of sight distance, This refers to the non-line-of-sight random scattering component.
[0040] For a reconfigurable smart surface-assisted link, the channel vector from the base station to the reconfigurable smart surface mounted on the k-th UAV and the channel vector from the reconfigurable smart surface to the i-th ground user are respectively: ; ;in, Let be the channel vector from the base station to the reconfigurable smart surface mounted on the k-th UAV. The path loss at the reference distance from the base station to the reconfigurable smart surface link. Let n be the distance from the base station to the reconfigurable smart surface mounted on the k-th UAV in time slot n. This represents the path loss exponent for the link from the base station to the reconfigurable smart surface. Rice factor for the base station to reconfigurable smart surface link. Let be the array response vector from the base station to the reconfigurable smart surface mounted on the k-th UAV. The random scattering component from the base station to the reconfigurable smart surface mounted on the k-th UAV; Let be the channel vector from the reconfigurable smart surface mounted on the k-th UAV to the i-th ground user. The path loss at the reference distance for the reconfigurable smart surface to ground user link. Let n be the distance from the reconfigurable smart surface mounted on the k-th UAV to the i-th ground user in time slot n. The path loss index for the reconfigurable smart surface to ground user link. Rice factor for reconfigurable smart surface-to-ground user links, Let be the array response vector from the reconfigurable smart surface mounted on the k-th UAV to the i-th ground user. Let be the random scattering component from the reconfigurable smart surface mounted on the k-th UAV to the i-th ground user.
[0041] Combining the above direct links and reconfigurable smart surface-assisted links, the true equivalent channel for the i-th ground user relative to the reconfigurable smart surface service cluster mounted on the k-th UAV can be: ;in, Let be the true equivalent channel for the i-th ground user relative to the k-th service cluster. This is the conjugate transpose. The first term... For direct connection links, the second item It is a cascaded link formed by reconfigurable smart surface reflection.
[0042] Due to limitations in pilot resources, estimation algorithm errors, feedback delays, and the high-speed movement of vehicles, the air-to-ground integrated vehicle-to-everything (V2X) system typically cannot obtain true channel state information and can only make subsequent decisions based on estimated channel state information. The formula for estimating the equivalent channel is as follows: ;in, Let be the estimated equivalent channel for the i-th ground user relative to the k-th service cluster. Let be the estimated direct connection channel from the base station to the i-th ground user. Let be the conjugate transpose of the estimated channel vector from the reconfigurable smart surface mounted on the k-th UAV to the i-th ground user. This is the estimated channel vector from the base station to the reconfigurable smart surface mounted on the k-th UAV.
[0043] This invention employs an additive error model to characterize the deviation between the true equivalent channel and the estimated equivalent channel: , ;in, To estimate the error, With a mean of 0 and a variance of The complex Gaussian distribution. The average variance of the estimation error per channel dimension. ;in, To estimate the error covariance matrix, For the trace operation of a matrix, For channel vectors, The dimension of the channel vector; for the channel vector The formula for its estimation error covariance is as follows: ;in, The channel covariance matrix, , For mathematical expectation operations; The effective pilot signal-to-noise ratio (SNR) is the ratio of pilot signal power to pilot observation noise power. For pilot observation noise power, It is the identity matrix. Perform the matrix inversion operation. The formula for estimating the equivalent channel gain is as follows: ;in, This is for the modulo squaring operation.
[0044] This invention determines the decoding order of non-orthogonal multiple access based on estimated equivalent channel gain. (Ordering function) To estimate the equivalent channel gain within the k-th service cluster and sort the r-th ground users, we have: ;in, Let K be the number of ground users in the k-th service cluster. Here, r is the sorting function, and r is the sorting position index, with a value ranging from 1 to... , The estimated equivalent channel gain for the r-th ground user in the k-th service cluster.
[0045] Based on the principle of serial interference cancellation in non-orthogonal multiple access (NOA), ground users with stronger channel conditions first decode and cancel the signals of ground users with weaker channel conditions, and then decode their own signals. For ground users at sequence position r... Its received signal is: ;in, Sort the received signals of the r-th ground user in time slot n within the k-th service cluster. The true equivalent channels of the r-th ground user are sorted within the k-th service cluster. For the first The true equivalent channel of the r-th ground user in each service cluster. For the first The superimposed signals of a service cluster in time slot n, It is additive white Gaussian noise. This represents noise power.
[0046] Under ideal serial interference cancellation conditions, ground users It can completely eliminate ground user signals with a sequence position lower than r. However, in practical systems, due to channel estimation errors, finite word length effects, and receiver hardware non-ideals, decoded user signals cannot be completely eliminated. This invention uses a linear residual interference model to characterize imperfect serial interference cancellation. For ground users... The residual serial interference from the decoded user signal is eliminated as follows: ;in, To eliminate residual serial interference for the r-th ground user in time slot n within the k-th service cluster. The imperfect serial interference cancellation coefficient. , Sort the k-th service cluster by... The power ratio of non-orthogonal multiple access for each ground user in time slot n.
[0047] Furthermore, since the air-to-ground integrated vehicle-to-everything (V2X) system performs detection based on estimated channels, imperfect channel state information can further transform the error between the actual and estimated channels into equivalent additional interference. ;in, The equivalent interference caused by the channel state information error of the r-th ground user in time slot n within the k-th service cluster. For the first The total transmit power of the service cluster in time slot n For ground users Compared to the first The variance of the estimation error of each service cluster in time slot n For the first A set of ground users in time slot n for a service cluster. For the first The non-orthogonal multiple access power ratio of the p-th ground user in time slot n within a service cluster.
[0048] Under the combined effect of imperfect channel state information and imperfect serial interference cancellation, ground users The downlink signal-to-interference-plus-noise ratio is: ;in, For the r-th ground user in the k-th service cluster, sort the downlink signal-to-interference-plus-noise ratio in time slot n. The estimated equivalent channel gain is calculated for the r-th ground user in the k-th service cluster. Based on the above formula, the ground user... The downlink achievable rate in time slot n is: ;in, Sort the r-th ground user in the k-th service cluster. The downlink achievable rate in time slot n, The system bandwidth (i.e., the downlink transmission frequency band width of the air-to-ground integrated vehicle-to-everything (V2X) system). The total rate of the air-to-ground integrated V2X system in time slot n. for: .
[0049] Let the flight speed of the k-th UAV in time slot n be... for: ,in, Let the k-th UAV be located at time slot n+1. The propulsion energy consumption of the k-th UAV at time slot n is: ;in, Let n be the propulsion energy consumption of the k-th UAV in time slot n. The power of the airfoil in hovering state. The blade angular velocity, The rotor radius is... Induced power during hovering, The induced velocity of the drone as it flies forward. For the fuselage drag ratio, Where S is the air density and S is the rotor solids density. This represents the rotor disk area.
[0050] The communication energy consumption of the k-th service cluster in time slot n is: ;in, Let n be the communication energy consumption of the k-th service cluster in time slot n. For fixed communication power.
[0051] The total energy consumption of the air-ground integrated vehicle-to-everything (V2X) system in time slot n is: ;in, Let n be the total energy consumption of the air-ground integrated vehicle-to-everything (V2X) system in time slot n.
[0052] To evaluate the communication performance, energy efficiency, and user service balance in an air-to-ground integrated vehicle-to-everything (V2X) system, this invention uses average total rate, energy efficiency, and Jain fairness index as core performance indicators. Average total rate... for: Energy efficiency The total amount of data transmitted is divided by the total power consumption: Jain's fairness index is: ;in, Let Jain's fairness index be the value in time slot n. Let i be the rate of the i-th ground user in time slot n. It should be a very small positive number to prevent the denominator from being 0.
[0053] This invention aims to jointly optimize UAV trajectories, user-related variables, non-orthogonal multiple access power ratios, and reconfigurable intelligent surface codebook indexes to maximize the energy efficiency of the air-to-ground integrated vehicle-to-everything (V2X) system throughout the entire mission cycle. Additionally, the total transmit power of the k-th service cluster in time slot n... The power budget is pre-allocated from the base station side; this invention uses an average allocation method. To simplify the formulation of the joint optimization problem, this invention defines a set of joint optimization variables. for: ,in, Let these be user-related variables. The joint optimization problem can be represented as: ;in, To calculate the expected value of the estimated channel error, the constraints of the joint optimization problem include: each ground user can only be assigned to one service cluster, the number of ground users in each service cluster has an upper and lower limit, the power ratio is non-negative and sums to 1, the total transmit power of the service cluster is finite, each ground user meets the minimum service quality requirements, the reconfigurable smart surface phase can only be selected from a finite codebook, the flight area and speed of UAVs are limited, the energy of a single UAV is finite, and multiple UAVs must maintain a safe distance from each other.
[0054] In a dynamic network of reconfigurable smart surfaces and non-orthogonal multiple access mounted on UAVs, the movement of ground users, changes in UAV positions, and adjustments to the cascaded reflection links of the reconfigurable smart surfaces lead to continuous evolution of the network topology and equivalent channels. Directly using a fixed-dimensional state vector as the environment makes it difficult to explicitly characterize the spatial proximity between nodes, link coupling relationships, and changes in service load. To address these issues, this invention constructs a dynamic heterogeneous graph and utilizes graph neural networks to extract node-level and global-level topology embeddings with physical semantics to characterize the network structure and time-varying characteristics.
[0055] In time slot n, a dynamic heterogeneous graph is constructed based on the real-time state. ,in For a set of graph nodes, Let be the set of edges. The feature matrix of the graph nodes. Let be the edge weight matrix.
[0056] To characterize the interaction strength between graph nodes, this invention constructs edge weights by combining spatial distance and link quality. For graph node u and graph node v, their edge weights are: Where n is the time slot number, and u and v are the graph node numbers. Let be the edge weight between graph node u and graph node v in time slot n. This is the distance weighting coefficient. Let be the distance between graph node u and graph node v in time slot n. This is the distance normalization factor. This is the link quality weight coefficient. This represents the normalized link quality for time slot n.
[0057] The normalized link quality is obtained based on the graph node type and in combination with large-scale link quality or geometric proximity.
[0058] The normalized link quality ;in, This is the original link quality metric for time slot n. and These are the minimum and maximum logarithmic link quality values for all edges in the time slot, respectively.
[0059] The original link quality index is determined according to the graph node type as follows: when graph node u is a base station and graph node v is a ground user: ;in, This is the distance attenuation factor for the link from the base station to the i-th ground user in time slot n; when graph node u is a base station and graph node v is a RIS-UAV: ;in, This is the distance attenuation factor from the base station to the k-th RIS-UAV in time slot n; when both graph node u and graph node v are ground users, or both graph node u and graph node v are RIS-UAVs, geometric proximity is used: , The square of the Euclidean distance between node u and node v in time slot n; when node u is a ground user and node v is a RIS-UAV: ;in, For time slots Large-scale gain of direct links from base stations to ground users. For time slots Large-scale gain of the direct link from the base station to the k-th RIS-UAV. For time slots Large-scale gain from the k-th RIS-UAV to ground users; .
[0060] Given that vehicle movement and channel state information disturbances can cause instantaneous edge weight fluctuations, this invention introduces a historical topology fusion mechanism to perform recent topology fusion. The node features and edge weights of each time slot are subjected to exponential decay smoothing: , ;in, For the historical fusion graph node features of time slot n, For time slots Graph node features, For the historical fusion edge weight of time slot n, For time slots The right to the border, The time decay factor, For the length of the history window, This represents the historical time slot offset.
[0061] Furthermore, this invention employs a graph attention network to encode node features. For the central graph node... and its neighboring graph nodes The attention coefficient of the l-th layer is: ;in, Let be the attention coefficient of the l-th layer, and softmax be the normalized exponential function. For a linear rectified function with leakage, For the central graph node The latent features in the l-th layer, The historical fusion edge weights of the central graph node and its neighboring graph nodes. and These are learnable parameters. Based on the above attention coefficients, the node features are updated as follows: ;in, For the central graph node The hidden features in the (l+1)th layer It is a non-linear activation function. This represents the set of neighbors of the central graph node. After L layers of propagation, the latent features of each graph node in the Lth layer constitute the node-level topological embedding of each graph node. This invention uses global mean aggregation to obtain the global-level topological embedding. .
[0062] To enable graph neural networks to learn physically meaningful topological relationships, this invention employs an edge weight reconstruction task to pre-train the graph neural network: , ;in, Let be the predicted edge weights of graph node u and graph node v in time slot n, and sigmoid be the sigmoid function. Let u be the transpose of the node-level topological embedding of graph node u in time slot n. For the node-level topology embedding of graph node v in time slot n For the pre-training loss of graph neural networks, Let represent the edge between graph node u and graph node v. The total number of edges, For learnable weight matrix, This is the balance coefficient.
[0063] Let the horizontal position of the i-th ground user in time slot n be... The cluster center (i.e., service cluster center) of the reconfigurable smart surface service cluster carried by the k-th UAV is: This invention utilizes node-level topological embeddings from graph neural network outputs to construct a correlation score between ground users and RIS-UAV.
[0064] The formula for the correlation score is: Where i is the ground user ID and k is the RIS-UAV ID. The association score between the i-th ground user and the k-th RIS-UAV in time slot n is given. , For the i-th ground user, the transpose of the node-level topology embedding in time slot n. For the node-level topology embedding of the k-th RIS-UAV in time slot n, For the i-th ground user, the node-level topology embedding in time slot n, It is the Euclidean norm.
[0065] To incorporate graph neural network-related scoring into clustering, the effective distance from ground user i to the center of the k-th service cluster is: ;in, Let be the effective distance from the i-th ground user to the service cluster center corresponding to the k-th RIS-UAV. Let i be the horizontal position of the i-th ground user in time slot n. For the service cluster center corresponding to the k-th RIS-UAV, It is the square of the Euclidean norm.
[0066] The ground user association variables are updated based on the effective distance, and the update formula for the ground user association variables is as follows: ;in, This is a ground user-related variable. It takes the value of 1 when the i-th ground user is assigned to the k-th RIS-UAV in time slot n, and otherwise takes the value of 0. For RIS-UAV sets, For the i-th ground user to the i-th The effective distance to the service cluster center corresponding to each RIS-UAV; argmin is the RIS-UAV number found in the RIS-UAV set that minimizes the effective distance.
[0067] Ground users are assigned to service cluster centers based on the effective distance, and the service cluster centers are updated with weighted scores using the correlation scores. The formula for the service cluster centers is as follows: The process involves repeatedly calculating the effective distance, updating the ground user association variables, and updating the service cluster centers with weighted averages until the change in the service cluster centers is less than a preset threshold or the preset maximum number of iterations is reached. To avoid overload or idleness in individual service clusters, the final association result must meet the service cluster capacity constraint. An upper and lower limit for the number of ground users in each service cluster is set. When the number of users in a service cluster exceeds the upper limit, the ground user with the lowest association score in that service cluster is transferred to another service cluster with available capacity; when the number of users in a service cluster is lower than the lower limit, the ground user with the highest association score is added from neighboring service clusters. To balance association stability and dynamic adaptability, this invention employs a periodic re-clustering mechanism, performing the re-clustering every [period]. The graph neural network-assisted user association is re-executed once per time slot.
[0068] In each time slot, a graph neural network topology embedding is first constructed and extracted, and then a graph neural network-assisted user association is performed to determine the service cluster corresponding to each RIS-UAV. Subsequently, the first reinforcement learning model outputs the UAV displacement and the proportion of non-orthogonal multiple access power within the service cluster, based on which a preview state is constructed, and the second reinforcement learning model selects the reconfigurable smart surface codebook index.
[0069] In a dynamic network of reconfigurable smart surfaces and non-orthogonal multiple access (NOA) on UAVs, significant coupling exists between multi-UAV trajectory control, NOA power allocation, and user association. UAV movement alters the distance between users and the UAV-mounted reconfigurable smart surfaces, the equivalent channel quality, and the NOA decoding order, thereby affecting user rates, inter-cluster interference, and system energy efficiency. If each UAV independently updates its strategy based solely on local observations, the impact of other UAV actions on service load and interference distribution can easily be overlooked.
[0070] To address the aforementioned problems, this invention employs a policy-evaluation architecture as the first reinforcement learning model. This first reinforcement learning model includes a policy network and a dual-evaluation network. The dual-evaluation network comprises a first evaluation network and a second evaluation network. The two evaluation networks independently score the actions generated by the policy network, mitigating the Q-value overestimation problem by taking the smaller of the two scores. The first reinforcement learning model adopts a centralized training and distributed execution framework. During the training phase, the policy networks and evaluation networks of each UAV share an experience pool, and the state information of all UAVs is centrally used for training the evaluation network. During the execution phase, each UAV makes independent decisions based solely on its local state through the policy network.
[0071] The local state of the k-th UAV in time slot n. ;in, Let's consider the local state of the k-th UAV. For the remaining energy, The service cluster link state is obtained based on the estimated equivalent channel gain. This represents the non-orthogonal multiple access power ratio of the previous time slot. It is a continuous action.
[0072] During the training phase, the local states of all UAVs constitute the joint state. , The local state of the Kth UAV is given; the joint state serves as the input to the first evaluation network and the second evaluation network. During the execution phase, each UAV outputs the continuous actions through the policy network based solely on its own local state.
[0073] Continuous action The horizontal displacement of the UAV (i.e., the horizontal displacement control quantity) The raw power allocation parameters output by the policy network composition.
[0074] To satisfy the nonnegativity and normalization constraints of the non-orthogonal multiple access power ratio, the original power ratio is mapped to the effective power ratio: , ;in, The proportion of non-orthogonal multiple access power allocated to the i-th ground user. The value of the i-th ground user in the original power allocation parameters output by the policy network. The first of the original power allocation parameters output by the policy network The values for each ground user.
[0075] During the centralized training phase, given experience samples The temporal difference target and comment network loss for the k-th drone is: ; ;in, Let the k-th UAV be the time-difference target in time slot n. The reward obtained by the k-th drone in time slot n is... As a discount factor, To evaluate the network for the target, Generate target actions for the target policy network and add pruned noise. To evaluate network loss, For the joint state of time slot n+1, For the first An evaluation network. The optimization objective of the policy network is to maximize the Q-value: ;in, The optimization objective of the policy network for the k-th drone is... This is the policy network for the k-th drone. The target network uses a soft update method. ;in, This is the soft update coefficient. and These are the parameters for the online evaluation network and the online policy network, respectively. and These are the parameters for the target evaluation network and the policy network, respectively.
[0076] After the first reinforcement learning model outputs the continuous displacement action of the UAV, the reconfigurable intelligent surface phase configuration should be adaptively adjusted according to the propagation environment after the UAV maneuver. If the phase configuration is still selected based on the old state before the UAV maneuver, the reconfigurable intelligent surface controller cannot perceive the new geometric relationship formed after the UAV maneuver, resulting in a temporal state mismatch between continuous trajectory control and discrete phase selection. To address the above problem, this invention designs a preview state mechanism, enabling the reconfigurable intelligent surface decision module to select discrete phase codewords based on the state after the UAV's continuous actions. After the k-th UAV outputs the continuous displacement action, the preview position of the UAV is first constructed: ;in, For the k-th drone in time slot Preview location, For projection operations on the flyable area of the drone.
[0077] Based on the preview location, the estimated channel from the base station to the reconfigurable smart surface mounted on the UAV, and the estimated channel from the reconfigurable smart surface mounted on the UAV to the ground user are recalculated; the formula for the preview equivalent channel is as follows: ;in, Let $\frac{ ... For base stations The estimated direct channel to the i-th ground user Let be the conjugate transpose of the estimated channel vector from the reconfigurable smart surface mounted on the k-th UAV to the i-th ground user at the preview location. This is the conjugate transpose operation. For time slots Reconfigurable smart surface phase matrix, The estimated channel from the base station to the reconfigurable smart surface mounted on the k-th UAV at the preview location.
[0078] The preview state is constituted by the preview position and the preview equivalent channel. The preview state is represented as follows: ;in, This represents the preview state of the k-th service cluster in time slot n. This is the index of the previous time slot codebook.
[0079] To improve the evaluation stability of discrete codebook actions, this invention employs a dual Q-network structure to estimate the action value of different codewords. The dual Q-network separates the state value function from the action advantage function, and the formula for outputting the Q-value is as follows: ;in, In preview mode Next, select the codebook index. Q value, For the network parameters of the dual Q-network, For the state value function, The network parameters are the state value function. For the action advantage function, Here, Q represents the network parameters for the action advantage function, and Q is the codebook capacity of the finite codebook. For summation index.
[0080] To mitigate the overestimation of the target Q-value, this invention further employs double-Q learning to construct the training target. The codebook index in the next state is selected by the online network parameters, and the target Q-value is evaluated by the target network parameters. ;in, The optimal codebook index for the next state (selected by the online network); ; For the target Q value, For mixed rewards, For target dual Q-networks. , This is the loss for the dual Q-network.
[0081] To encourage the dynamic user association in the graph neural network, the continuous control of the first reinforcement learning model, and the selection of the reconfigurable intelligent surface codebook to collaboratively optimize around a unified system objective, this invention designs a hierarchical reward mechanism consisting of global rewards, local rewards, and hybrid rewards. The global reward is used to evaluate the impact of continuous UAV actions on the overall system performance, comprehensively considering energy efficiency, speed, fairness, and constraint violation penalties. The formula for the global reward is: ;in, For global rewards, This is the normalized value of the average sum rate. J[n] is the normalized value of energy efficiency, and J[n] is the Jain fairness index. To constrain violations and penalties, , , , The weighting coefficients are as follows. The penalty items for constraint violations include penalties for service quality constraints, energy budget violations, human-machine safe distance violations, UAV speed constraints, and flight area boundary violations. ;in, Penalties for violating service quality constraints include penalties for ground users whose service quality does not meet the minimum speed requirements; Penalties for violating the energy budget are corresponding to penalties for a single drone's total energy consumption exceeding the energy budget; Penalties for violating the safe distance for drones, corresponding to penalties for drones being less than the safe distance between them; The penalty for violating drone speed constraints is the penalty for exceeding the maximum speed limit. Penalties for violating flight area boundary constraints, corresponding to penalties for drones exceeding preset flight areas.
[0082] Local rewards are used to evaluate the local effects of codebook selection on the reconfigurable smart surface, enabling the second reinforcement learning model to perceive the direct impact of codebook actions on its own service cluster. The local rate gain of the reconfigurable smart surface is the change in total rate of the current time slot relative to the previous time slot: ;in, For the local rate gain of the k-th service cluster in time slot n, The total rate of the k-th service cluster when the current codebook configuration is used in the current time slot. The total rate when the previous codebook configuration was used in the previous time slot.
[0083] Based on this local gain, the local reward is: ;in, For the local reward of the k-th service cluster in time slot n, For phase switching penalty weights, The phase switching cost of the reconfigurable smart surface mounted on the k-th UAV in time slot n.
[0084] Frequent switching of the reconfigurable smart surface codebook can cause drastic phase configuration changes between adjacent time slots, which is detrimental to stable system control. This invention introduces a phase switching cost. If the phase of a reconfigurable smart surface is represented by a complex unit vector, then the phase switching cost is the average of the phase changes of all reflecting units in adjacent time slots: ;in, Let m be the total number of reflective elements on the reconfigurable smart surface, and m be the reflective element number. For modulo operation, The phase angle, Let be the complex phase coefficient of the m-th reflective element of the reconfigurable smart surface mounted on the k-th UAV in time slot n. This is a conjugate operation.
[0085] Hybrid reward combines global and local rewards in a weighted manner to form the final training signal. ;in, For mixed rewards, and These are the combined weights for global rewards and local rewards, respectively.
[0086] This invention employs a strategy combining phased training and joint fine-tuning. In the first phase, the reconfigurable intelligent surface codebook strategy is fixed, and only the continuous control strategy for the UAV is trained. The reconfigurable intelligent surface phase is fixed as the default codebook, and the first reinforcement learning model learns basic trajectory control and non-orthogonal multiple access power allocation capabilities in this phase. In the second phase, the already trained first reinforcement learning model strategy is fixed, and only the reconfigurable intelligent surface codebook selection network is trained. All network parameters of the first reinforcement learning model are frozen, and the second reinforcement learning model is trained under relatively stable UAV trajectories, enabling it to learn how to select effective codewords based on different states. In the third phase, joint fine-tuning is performed, with the network parameters of the first and second reinforcement learning models updated simultaneously, enabling the continuous UAV, non-orthogonal multiple access control, and discrete reconfigurable intelligent surface phase selection to be collaboratively optimized under a unified reward mechanism. After training, the network parameters are no longer updated during the online execution phase; only forward inference and action output are performed.
[0087] The present invention sets up the following ablation versions and comparison schemes: the complete scheme of the present invention, ablation without preview state, ablation without action fusion, ablation with single agent continuous control, and greedy search baseline.
[0088] This invention (Proposed): The complete framework of this invention includes user association assisted by graph neural network, continuous action decision-making of the first reinforcement learning model, preview state construction, discrete action decision-making of the second reinforcement learning model, and a hybrid reward mechanism, with each step working in concert.
[0089] No Preview Scheme (w / o Preview): Based on the complete framework of this invention, the preview state construction step is removed, allowing the second reinforcement learning model to make discrete action decisions directly based on the state of the first reinforcement learning model before the action is executed.
[0090] Actionless Fusion (w / o Fusion): Based on the complete solution of this invention, the first reinforcement learning model and the second reinforcement learning model are trained independently and optimized alternately, and the second reinforcement learning model does not make decisions based on the preview state.
[0091] Prior Fusion, a deterministic policy-dual fusion scheme, employs a deterministic policy network to generate continuous power control actions and a dual network to evaluate discrete sub-channel selection actions. These two approaches achieve collaborative optimization through cascaded decision-making. While this scheme possesses continuous-discrete action fusion capabilities, it lacks a preview state mechanism, meaning the state changes following continuous action execution are not perceived by the discrete decision-making module. Therefore, this study directly verifies the incremental contribution of the preview state mechanism to action fusion and simultaneously validates the improvement effect of the first reinforcement learning model compared to the deterministic policy network.
[0092] Single-Agent Continuous Control Ablation Scheme: Based on the complete scheme of this invention, this scheme retains the preview state construction and discrete action decision-making of the second reinforcement learning model, but replaces the first reinforcement learning model of multiple UAVs with a single-agent continuous control model. The trajectory and power decisions of each UAV are centrally output by a single network, eliminating the centralized training-distributed execution multi-UAV collaborative architecture. This scheme is used to verify the necessity of multi-UAV collaborative training in multi-UAV scenarios.
[0093] Greedy search baseline scheme: A greedy search strategy is employed in each time slot to select the candidate action combination that optimizes the system performance in the current time slot, without involving historical experience learning or long-term policy optimization. This scheme serves as a lower bound reference for performance.
[0094] The simulation was conducted in a scenario involving a downlink UAV, a reconfigurable smart surface, and a non-orthogonal multiple access air-to-ground integrated vehicle network. The simulation parameter settings are shown in Table 1.
[0095] Table 1 Simulation parameter settings
[0096] like Figure 3 As shown, this invention exhibits faster convergence speed and more stable average reward during training, achieving an improvement of approximately 43% compared to the suboptimal no-action fusion ablation scheme. This is because it coordinates continuous UAV control and discrete codebook selection for reconfigurable intelligent surfaces through preview state coordination, and utilizes a first reinforcement learning model to improve policy stability during multi-agent continuous control. This invention maintains high energy efficiency across different training stages, with an average energy efficiency improvement of approximately 45% compared to the deterministic policy-dual fusion scheme. This is because the preview-based coordination mechanism reduces state mismatch between UAV trajectory adjustment and reconfigurable intelligent surface phase selection. Figure 4 As shown, the introduction of the preview state mechanism results in a higher and more stable average reward, which is about 6% higher than the scheme without preview state. The fundamental reason is that the preview state transmits the channel and topology changes caused by the UAV's actions to the reconfigurable smart surface decision-making end in advance, thereby alleviating the timing mismatch problem between continuous control and discrete codebook selection.
[0097] Another embodiment of the present invention provides a UAV communication resource optimization system based on preview state collaborative decision-making, specifically including: The topology perception and user association module is used to construct a dynamic heterogeneous graph of the air-ground integrated vehicle network system composed of base stations, RIS-UAVs and ground users. The RIS-UAVs are composed of UAVs and their reconfigurable smart surfaces. The module uses graph neural networks to extract node-level topology embeddings and global-level topology embeddings, and corrects the association between ground users and RIS-UAVs based on the node-level topology embeddings. It also forms service clusters by combining capacity constraints.
[0098] The continuous action decision module is communicatively connected to the topology perception and user association module. It is used to receive the service cluster and optimize the continuous actions using a first reinforcement learning model. The continuous actions include UAV trajectory control parameters and the original power allocation parameters of ground users within the service cluster.
[0099] The preview state construction module is communicatively connected to the continuous action decision module and is used to construct a preview state based on the continuous action. The preview state includes the preview position, the preview equivalent channel, and the global-level topology embedding.
[0100] The discrete action decision module is communicatively connected to the preview state construction module. It is used to receive the preview state and select a codebook index from a finite codebook using a second reinforcement learning model. The codebook index corresponds to a reconfigurable smart surface phase configuration.
[0101] The hybrid reward calculation module is communicatively connected to the discrete action decision module. It is used to receive the codebook index output by the discrete action decision module and construct a hybrid reward function consisting of global reward and local reward.
[0102] The joint training optimization module is communicatively connected to the continuous action decision module, the discrete action decision module, and the hybrid reward calculation module, respectively, and is used to perform joint training and parameter updates on the first reinforcement learning model and the second reinforcement learning model based on the hybrid reward function.
[0103] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for optimizing UAV communication resources based on collaborative decision-making in preview state, characterized in that, Includes the following steps: Construct an air-ground integrated vehicle-to-everything (V2X) system consisting of base stations, RIS-UAVs, and ground users; the RIS-UAVs consist of unmanned aerial vehicles (UAVs) and their reconfigurable smart surfaces; model the base stations, RIS-UAVs, and ground users as graph nodes, construct the edges between the graph nodes, and form a dynamic heterogeneous graph with the graph nodes and the edges; The dynamic heterogeneous graph is encoded using a graph neural network, and node-level topological embedding and global-level topological embedding are obtained through a graph attention mechanism. The association between the ground users and RIS-UAV is corrected based on the node-level topology embedding, and a service cluster is formed in combination with capacity constraints; The first reinforcement learning model is used to output continuous actions, which include UAV trajectory control parameters and the original power allocation parameters of ground users within the service cluster. A preview state is constructed based on continuous actions. The preview state includes the preview position, the preview equivalent channel, and the global-level topology embedding. A second reinforcement learning model is used to output discrete actions, the discrete actions including codebook indices selected from a finite codebook, the codebook indices corresponding to reconfigurable smart surface phase configurations, and the decision of the discrete actions is based on the preview state; A hybrid reward function consisting of global and local rewards is constructed. Based on the hybrid reward function, the first reinforcement learning model and the second reinforcement learning model are jointly trained and their parameters are updated to achieve collaborative optimization of the UAV trajectory control parameters, the original power allocation parameters, and the reconfigurable intelligent surface phase configuration.
2. The UAV communication resource optimization method based on preview state collaborative decision-making according to claim 1, characterized in that, The dynamic heterogeneous graph is encoded using a graph neural network, and node-level and global-level topological embeddings are obtained through a graph attention mechanism. The specific steps are as follows: The edge weights between nodes in the graph are obtained using the following formula: ; Where n is the time slot number, and u and v are the graph node numbers. Let be the edge weight between graph node u and graph node v in time slot n. This is the distance weighting coefficient. Let be the distance between graph node u and graph node v in time slot n. This is the distance normalization factor. This is the link quality weight coefficient. The normalized link quality for time slot n; The edge weights of each time slot within the preset historical window length are exponentially decayed and smoothed to obtain the historical fusion edge weights; The first node of the central diagram Layer hidden features and the first neighbor graph node Layer implicit features are concatenated and multiplied with the historical fusion edge weights of the central graph node and its neighboring graph nodes. After normalization, the result is the first layer. Attention coefficient of layer The initial value is 1; Based on the first The attention coefficient of the layer affects the node of the neighbor graph. The hidden features of the layers are weighted and aggregated, and after nonlinear activation, the first node of the central graph is obtained. Hidden features of layers; Through multi-layer propagation with a preset number of layers L, the latent features of each graph node in the Lth layer are used as the node-level topological embeddings of each graph node, and the node-level topological embeddings of all graph nodes are aggregated by global mean to obtain the global-level topological embedding.
3. The UAV communication resource optimization method based on preview state collaborative decision-making according to claim 2, characterized in that, The association between the ground user and the RIS-UAV is corrected based on the node-level topology embedding; the specific steps are as follows: Based on the node-level topology embedding, a correlation score between ground users and RIS-UAV is constructed, and the formula for the correlation score is as follows: ; Where i is the ground user ID, and k is the RIS-UAV ID. The association score between the i-th ground user and the k-th RIS-UAV in time slot n is given. , For the i-th ground user, the transpose of the node-level topology embedding in time slot n. For the node-level topology embedding of the k-th RIS-UAV in time slot n, For the i-th ground user in time slot n, the node-level topology embedding is performed. For the Euclidean norm, The value should be a very small positive number to prevent the denominator from being 0; The effective distance from the ground user to the service cluster center is calculated based on the correlation score, and the formula for the effective distance is as follows: ; in, Let be the effective distance from the i-th ground user to the service cluster center corresponding to the k-th RIS-UAV. Let i be the horizontal position of the i-th ground user in time slot n. For the service cluster center corresponding to the k-th RIS-UAV, The square of the Euclidean norm; The ground user association variables are updated based on the effective distance, and the update formula for the ground user association variables is as follows: ; in, This is a ground user-related variable. It takes the value of 1 when the i-th ground user is assigned to the k-th RIS-UAV in time slot n, and otherwise takes the value of 0. For RIS-UAV sets, For the i-th ground user to the i-th The effective distance to the service cluster center corresponding to each RIS-UAV; argmin is the RIS-UAV number found in the RIS-UAV set that minimizes the effective distance; Ground users are assigned to service cluster centers based on the effective distance, and the service cluster centers are updated with weighted scores using the correlation scores. The formula for the service cluster centers is as follows: ; Where M represents the total number of ground users; Repeat the calculation of effective distance, the update of ground user related variables and the weighted update of service cluster center until the change in service cluster center is less than the preset threshold or the preset maximum number of iterations is reached.
4. The UAV communication resource optimization method based on preview state collaborative decision-making according to claim 3, characterized in that, The service cluster is updated periodically, and the time slot interval between two adjacent updates is a preset period value. In each updated service cluster, when the number of ground users in the service cluster exceeds the preset capacity limit, the ground user with the lowest association score will be transferred to another service cluster with available capacity; when the number of ground users in the service cluster is lower than the preset capacity limit, the ground user with the highest association score will be added from the adjacent service cluster.
5. The UAV communication resource optimization method based on preview state collaborative decision-making according to claim 4, characterized in that, The first reinforcement learning model is a policy-evaluation architecture, including a policy network and a dual evaluation network. The dual evaluation network includes a first evaluation network and a second evaluation network. The first evaluation network and the second evaluation network output a first Q-value and a second Q-value, respectively, and the training target is constructed using the smaller of the first Q-value and the second Q-value. The first reinforcement learning model adopts a centralized training and distributed execution framework. During the training phase, the joint state composed of the local states of all UAVs is used as the input of the first evaluation network and the second evaluation network. During the execution phase, each UAV outputs the continuous action only based on its own local state through the policy network.
6. The UAV communication resource optimization method based on preview state collaborative decision-making according to claim 5, characterized in that, The strategy network outputs the original power allocation parameters. The original power allocation parameters are then taken as positive values and divided by the sum of the positive values of all ground users in the service cluster to obtain the non-orthogonal multiple access power allocation parameters.
7. The UAV communication resource optimization method based on preview state collaborative decision-making according to claim 6, characterized in that, The preview state is built based on continuous actions. The specific steps are as follows: Based on the continuous action, the horizontal displacement control quantity in the continuous action is obtained, and the horizontal displacement control quantity is the trajectory control parameter of the UAV. The preview position is calculated based on the current position of the UAV and the horizontal displacement control amount. The formula for the preview position is as follows: ; in, For the k-th drone in time slot Preview location, For projection operations on the flyable area of the drone, For the k-th drone in time slot Current location Let be the horizontal displacement control value of the k-th UAV; Based on the preview location, the estimated channel from the base station to the reconfigurable smart surface mounted on the UAV, and the estimated channel from the reconfigurable smart surface mounted on the UAV to the ground user are recalculated. The formula for the preview equivalent channel is as follows: ; in, Let $\frac{ ... For base stations The estimated direct channel to the i-th ground user Let be the conjugate transpose of the estimated channel vector from the reconfigurable smart surface mounted on the k-th UAV to the i-th ground user at the preview location. This is the conjugate transpose operation. For time slots Reconfigurable smart surface phase matrix, The estimated channel from the base station to the reconfigurable smart surface mounted on the k-th UAV at the preview location; The global-level topology embedding is obtained, and the preview state includes the preview location, the preview equivalent channel, and the global-level topology embedding.
8. The UAV communication resource optimization method based on preview state collaborative decision-making according to claim 7, characterized in that, The second reinforcement learning model is a dual Q-network, which separates the state value function from the action advantage function, and the formula for outputting the Q-value is as follows: ; in, In preview mode Select codebook index below Q value, For the network parameters of the dual Q-network, For the state value function, The network parameters are the state value function. For the action advantage function, Here, Q represents the network parameters for the action advantage function, and Q is the codebook capacity of the finite codebook. For summation index; The finite codebook is generated as follows: based on the reconfigurable smart surface array structure and the number of phase quantization bits, a two-dimensional discrete Fourier transform beam grid is constructed to generate basic codewords. The phase of each reflection unit is discretized in combination with the number of phase quantization bits. The final finite codebook is obtained by limiting the codebook capacity and removing redundant codewords.
9. The UAV communication resource optimization method based on preview state collaborative decision-making according to claim 8, characterized in that, The global reward is a weighted sum of the normalized average total rate, the normalized energy efficiency, the Jain fairness index, and the constraint violation penalty. The average ground user rate is calculated based on the non-orthogonal multiple access power allocation parameters. The constraint violation penalty includes service quality constraint violation penalties, energy budget violation penalties, UAV safety distance violation penalties, UAV speed constraint violation penalties, and flight area boundary violation penalties. The local reward is calculated based on the service cluster rate gain of the current time slot relative to the previous time slot, and the non-negative part of the service cluster rate gain is logarithmically transformed before subtracting the phase switching cost. The phase switching cost is used to constrain the change in reconfigurable smart surface phase configuration between adjacent time slots, and the formula for the phase switching cost is as follows: ; in, Let n be the phase switching cost of the reconfigurable smart surface mounted on the k-th UAV in time slot n. denoted by m, which represents the total number of reflective elements on the reconfigurable smart surface mounted on the UAV, where m is the reflective element number. For modulo operation, The phase angle, Let be the complex phase coefficient of the m-th reflective element of the reconfigurable smart surface mounted on the k-th UAV in time slot n. For conjugate operations; The hybrid reward function is a weighted fusion of the global reward and the local reward.
10. A UAV communication resource optimization system based on preview state collaborative decision-making, comprising the UAV communication resource optimization method based on preview state collaborative decision-making as described in any one of claims 1 to 9, characterized in that, The system includes: The topology perception and user association module is used to construct a dynamic heterogeneous graph of the air-ground integrated vehicle network system composed of base stations, RIS-UAVs and ground users. The RIS-UAVs are composed of UAVs and their reconfigurable smart surfaces. The module uses graph neural networks to extract node-level topology embeddings and global-level topology embeddings, and corrects the association between ground users and RIS-UAVs based on the node-level topology embeddings. It also forms service clusters by combining capacity constraints. The continuous action decision module is communicatively connected to the topology perception and user association module. It is used to receive the service cluster and optimize the continuous actions using a first reinforcement learning model. The continuous actions include UAV trajectory control parameters and the original power allocation parameters of ground users within the service cluster. A preview state construction module is communicatively connected to the continuous action decision module and is used to construct a preview state based on the continuous action. The preview state includes a preview position, a preview equivalent channel, and the global-level topology embedding. The discrete action decision module is communicatively connected to the preview state construction module. It is used to receive the preview state and select a codebook index from a finite codebook using a second reinforcement learning model. The codebook index corresponds to a reconfigurable smart surface phase configuration. The hybrid reward calculation module is communicatively connected to the discrete action decision module, and is used to receive the codebook index output by the discrete action decision module and construct a hybrid reward function consisting of global reward and local reward. The joint training optimization module is communicatively connected to the continuous action decision module, the discrete action decision module, and the hybrid reward calculation module, respectively, and is used to perform joint training and parameter updates on the first reinforcement learning model and the second reinforcement learning model based on the hybrid reward function.
