A decision-making method, apparatus, device, medium, and program product

Abstract

The application provides a decision method, device, equipment, medium and program product, relates to the technical field of communication, and is used for accelerating model training convergence speed and improving energy efficiency of a system. The specific technical scheme is as follows: at least two user state feature vectors are acquired, and each user state feature vector corresponds to a user; at least two user state feature vectors are processed through a multi-head self-attention module of a first model to obtain at least two context feature vectors; each context feature vector in the at least two context feature vectors comprises a user state feature vector corresponding to a user and a relationship feature vector between the user and other users; an aggregation operation is performed on the at least two context feature vectors to obtain an aggregated context feature vector; and an action or a Q value in an MDP is generated based on the aggregated context feature vector. The application is applied to a scene of model training in a UAV-RIS cooperative communication system.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a decision-making method, apparatus, device, medium, and program product. Background Technology

[0002] In a 6G UAV-RIS collaborative communication system, the key to maximizing network energy efficiency lies in the joint real-time optimization of the UAV's three-dimensional flight trajectory and the high-dimensional beamforming of the reflector. Because these two control variables are tightly coupled and the problem model is highly non-convex, traditional optimization algorithms face bottlenecks due to excessive computational complexity, making it difficult to meet the real-time decision-making requirements of dynamic environments.

[0003] Currently, deep reinforcement learning (DRL) methods are typically used to optimize computational complexity, such as deep deterministic policy gradient (DDPG) or twin-delayed deep deterministic policy gradient (TD3) algorithms. Multilayer perceptrons (MLPs) are used as the core network architecture for the agent to address coverage gaps and improve quality of service (QoS). In specific implementations, the system directly concatenates the state information (such as location and channel conditions) of multiple ground users into a fixed-length one-dimensional vector, which serves as the input to the MLP.

[0004] However, following the above method, when using MLP to handle multi-user states, MLP treats the input as an ordered vector and cannot handle the permutation invariance of user inputs. This results in slow model training convergence and low system energy efficiency. Summary of the Invention

[0005] This application provides a decision-making method, apparatus, device, medium, and program product for accelerating model training convergence speed and improving system energy efficiency.

[0006] In a first aspect, embodiments of this application provide a decision-making method, which includes: acquiring at least two user state feature vectors, each user state feature vector corresponding to a user; processing the at least two user state feature vectors through a multi-head self-attention module of a first model to obtain at least two context feature vectors, each of the at least two context feature vectors including: a user state feature vector corresponding to a user, and a relationship feature vector between the user and other users; performing an aggregation operation on the at least two context feature vectors to obtain an aggregated context feature vector; and generating an action or Q-value in a Markov Decision Process (MDP) based on the aggregated context feature vector.

[0007] The technical solution provided in this application offers at least the following advantages: it can acquire multiple user state feature vectors and relationship feature vectors between users, and perform aggregation operations on the acquired feature vectors to obtain aggregated context feature vectors. Furthermore, it generates actions or Q-values ​​in the MDP based on these aggregated context feature vectors. This allows user states to be treated as an unordered set, explicitly models the dependencies between users, and enables dynamic focus on key users, thus handling the permutation invariance of user input. This accelerates model training convergence and improves system energy efficiency.

[0008] One possible implementation, as described above, of obtaining at least two user state feature vectors, includes: obtaining the states of at least two users; and mapping the state of each user to a user state feature vector through the entity embedding layer of the first model, thereby obtaining at least two user state feature vectors.

[0009] Another possible implementation, which involves performing aggregation operations on at least two context feature vectors to obtain an aggregated context feature vector, includes: summing the at least two context feature vectors using a permutation-invariant aggregation function to obtain the aggregated context feature vector.

[0010] Another possible implementation, based on the aggregated context feature vector, generates actions or Q-values ​​in the MDP, including: concatenating the aggregated context feature vector and the UAV feature vector to obtain a concatenated feature vector; the UAV feature vector is obtained by mapping the UAV state through the entity embedding layer of the first model; and the concatenated feature vector is processed through the fully connected layer of the first model to obtain the actions or Q-values ​​in the MDP.

[0011] Another possible implementation involves, after obtaining the action in the MDP, executing the action; obtaining the reward function and state function after the action is executed, and storing the reward function and state function in the replay pool; randomly sampling from the replay pool to obtain sampled data; determining the first Q value based on the sampled data; and updating the first model using the first Q value to obtain the updated first model.

[0012] Another possible implementation, based on the sampled data, is to determine the first Q value, which includes: processing the sampled data through a dual critic network of the first model to obtain a second Q value; and performing temporal difference (TD) error minimization on the second Q value to obtain the first Q value.

[0013] Secondly, embodiments of this application provide a decision-making apparatus, including: an acquisition module, a processing module, an aggregation module, and an output module; the acquisition module is used to acquire at least two user state feature vectors, each user state feature vector corresponding to a user; the processing module is used to process the at least two user state feature vectors through a multi-head self-attention module of a first model to obtain at least two context feature vectors, each of the at least two context feature vectors including: a user state feature vector corresponding to a user, and a relationship feature vector between the user and other users; the aggregation module is used to perform an aggregation operation on the at least two context feature vectors to obtain an aggregated context feature vector; the output module is used to generate actions or Q-values ​​in an MDP based on the aggregated context feature vector.

[0014] One possible implementation is that the aforementioned acquisition module is specifically used to acquire the states of at least two users; and through the entity embedding layer of the first model, the state of each user is mapped to a user state feature vector to obtain at least two user state feature vectors.

[0015] Another possible implementation is that the above-mentioned aggregation module is specifically used to sum at least two context feature vectors by using an aggregation function that is permutation-invariant, to obtain an aggregated context feature vector.

[0016] Another possible implementation is that the above output module is specifically used to concatenate the aggregated context feature vector and the UAV feature vector to obtain a concatenated feature vector; the UAV feature vector is obtained by mapping the UAV state through the entity embedding layer of the first model; and the concatenated feature vector is processed through the fully connected layer of the first model to obtain the action or Q value in the MDP.

[0017] In another possible implementation, the above processing module is also used to execute the action after obtaining the action in the MDP; and to obtain the reward function and state function after the action is executed, and store the reward function and state function in the replay pool; and to randomly sample from the replay pool to obtain sampled data; to determine the first Q value based on the sampled data; and to update the first model using the first Q value to obtain the updated first model.

[0018] Another possible implementation is that the aforementioned processing module is specifically used to process the sampled data through the dual critic network of the first model to obtain the second Q value; and to minimize the TD error of the second Q value to obtain the first Q value.

[0019] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the method of the first aspect described above.

[0020] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a computer, implement the method of the first aspect described above.

[0021] Fifthly, this application provides a computer program product stored in a storage medium, which, when executed by a computer, implements the method described in the first aspect.

[0022] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0023] The beneficial effects of the second to sixth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description

[0024] Figure 1 A schematic diagram of a network architecture for an application of a decision-making method provided in an embodiment of this application;

[0025] Figure 2 A flowchart illustrating a decision-making method provided in an embodiment of this application;

[0026] Figure 3 A schematic diagram of a UAV-RIS-assisted downlink communication system provided in an embodiment of this application;

[0027] Figure 4 A flowchart illustrating yet another decision-making method provided in an embodiment of this application;

[0028] Figure 5 A flowchart illustrating another decision-making method provided in an embodiment of this application;

[0029] Figure 6 A flowchart illustrating yet another decision-making method provided in an embodiment of this application;

[0030] Figure 7 A flowchart illustrating yet another decision-making method provided in an embodiment of this application;

[0031] Figure 8 A flowchart illustrating another decision-making method provided in an embodiment of this application;

[0032] Figure 9 A schematic diagram of the network architecture of an attention-enhanced dual-delay deep deterministic policy gradient (A-TD3) algorithm provided for embodiments of this application;

[0033] Figure 10 A trajectory visualization diagram of a decision-making device provided in an embodiment of this application;

[0034] Figure 11 This is a schematic diagram comparing the performance metrics of the algorithm provided in the embodiments of this application with those of algorithms in related technologies;

[0035] Figure 12 This is a schematic diagram comparing the performance metrics of the algorithm provided in the embodiments of this application with those of algorithms in related technologies;

[0036] Figure 13 This is a schematic diagram comparing the performance metrics of the algorithm provided in the embodiments of this application with those of algorithms in related technologies;

[0037] Figure 14 This is a schematic diagram comparing the performance metrics of the algorithm provided in the embodiments of this application with those of algorithms in related technologies;

[0038] Figure 15 This is a schematic diagram of the structure of a decision-making device provided in an embodiment of this application;

[0039] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0041] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0042] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0043] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0044] The decision-making method, apparatus, device, medium, and program product provided in this application embodiment can be applied to model training scenarios under the UAV-RIS collaborative communication system.

[0045] With the development of 6G networks, the key to maximizing network energy efficiency in UAV-RIS collaborative communication systems lies in the joint real-time optimization of the UAV's three-dimensional flight trajectory and the high-dimensional beamforming of the reflector. Because these two control variables are tightly coupled and the problem model is highly non-convex, traditional optimization algorithms face bottlenecks such as excessive computational complexity, making it difficult to meet the real-time decision-making requirements of dynamic environments.

[0046] Currently, DRL methods are typically used to optimize computational complexity, such as the DDPG or TD3 algorithms, and MLP is used as the core network architecture for the intelligent agent to address coverage holes and improve QoS. In specific implementations, the system directly concatenates the state information (such as location and channel conditions) of multiple ground users into a fixed-length one-dimensional vector, which serves as the input to the MLP.

[0047] However, among the MLP and concatenated input methods mentioned above, MLP is sensitive to input order and suffers from an information bottleneck. MLP treats the input as an ordered vector and cannot handle the permutation invariance of user input. If the order of two users in the input vector is swapped, the network may output completely different actions, even though its physical state (system topology) remains unchanged. Because simple concatenation masks the relational structure between users, MLP struggles to extract spatial dependencies between users (e.g., user clustering or isolation) from the flattened vector. It cannot achieve optimal energy efficiency in dynamic multi-user scenarios and cannot flexibly prioritize users based on their real-time QoS requirements. This makes it difficult for the agent to dynamically allocate "attention" to the most critical users based on real-time channel conditions. Consequently, training convergence is slow, and the system's energy efficiency is low.

[0048] To address the aforementioned technical problems, embodiments of this application provide a decision-making method, apparatus, device, medium, and program product. This method can acquire multiple user state feature vectors and relationship feature vectors between users, and aggregate these feature vectors to obtain an aggregated context feature vector. Furthermore, it generates actions or Q-values ​​in an MDP based on this aggregated context feature vector. This allows user states to be treated as an unordered set, clearly modeling the dependencies between users, and enabling dynamic focus on key users, thus handling the permutation invariance of user input. This accelerates model training convergence and improves system energy efficiency.

[0049] The decision-making methods, apparatus, devices, media, and program products provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0050] Figure 1 The network architecture for a decision-making method application provided in an embodiment of this application is illustrated. For example... Figure 1 As shown, the network architecture includes a decision-making device 101 and a terminal device 102. The decision-making device 101 and the terminal device 102 are interconnected.

[0051] In some embodiments, the decision-making device 101 may be a server, a computer, or a processor or processing unit within a server or computer. The server may be a single server or a server cluster comprising multiple servers. It should be noted that the embodiments of this application do not limit the specific device form of the decision-making device 101. Figure 1 The decision-making device 101 is shown as a single server as an example.

[0052] In some embodiments, the terminal device may be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., and the embodiments of this application do not specifically limit it. Figure 1 The example shown is a mobile phone, with terminal device 102 as an example.

[0053] It should be noted that the network architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As network architectures evolve, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0054] See Figure 2 This is a flowchart illustrating a decision-making method provided in an embodiment of this application. Figure 2 As shown, the decision-making method provided in this application embodiment can be implemented by the above-mentioned decision-making device, specifically including the following steps 201 to 204.

[0055] Step 201: The decision-making device acquires at least two user state feature vectors.

[0056] In this context, each of the at least two user state feature vectors corresponds to one user.

[0057] In some embodiments, each user state feature vector is used to characterize the user's own information.

[0058] It should be noted that the decision-making method provided in this application embodiment can be applied to UAV-RIS-assisted downlink communication systems. For example... Figure 3 As shown, the UAV-RIS-assisted downlink communication system includes a ground base station (BS) 31, a rotary-wing UAV 32 equipped with the RIS, and One ground user.

[0059] In this scenario, due to the obstruction of tall buildings in the city, the direct link between the base station and the user is blocked, and communication relies entirely on the reflection link provided by the UAV-RIS. In this scenario, the UAV flies in three-dimensional space, and its position is at... Each moment is represented by three-dimensional coordinates:

[0060] ;

[0061] To accurately simulate real-world communication environments, this embodiment employs a 3D channel model combining large-scale fading and line-of-sight (LoS) components. For the... The user, in the 1st The achievable rate per time slot Calculated according to Shannon's formula:

[0062] ;

[0063] in Represents bandwidth. For base station transmission power, For noise power, and This is a diagonal matrix composed of the phase shifts of the various reflection units of the RIS. Furthermore, given the limited onboard energy of the UAV, this invention employs an energy consumption model based on rotor hydrodynamics to evaluate the system cost. propulsion power per time slot Specifically, it is expressed as follows:

[0064] ;

[0065] in, , and It is a fixed constant. Hovering speed, and These are the velocities in the vertical and horizontal directions, respectively. For the speed of the drone's blades; , , and These are the fuselage drag ratio, rotor solidity, air density, and rotor disk area.

[0066] In some embodiments, combined with Figure 2 ,like Figure 4 As shown, step 201 above can be implemented through steps 201a and 201b.

[0067] Step 201a: The decision-making device acquires the status of at least two users.

[0068] Step 201b: The decision-making device maps the state of each user to a user state feature vector through the entity embedding layer of the first model, thereby obtaining at least two user state feature vectors.

[0069] In some embodiments, the first model described above can be an attention-enhanced dual-delay deep deterministic policy gradient (A-TD3) algorithm model.

[0070] In some embodiments, this method reconstructs the traditional Actor and Critic networks, introducing a hierarchical structure of "embedding-attention-aggregation." The system first embeds the UAV state and each user through an entity embedding layer. The states are mapped to high-dimensional feature vectors respectively. and .

[0071] In some embodiments, by mapping the user state to a user state feature vector, the user state feature vector can be input into the multi-head self-attention module, laying the foundation for the following steps.

[0072] Step 202: The decision-making device processes at least two user state feature vectors through the multi-head self-attention module of the first model to obtain at least two context feature vectors.

[0073] In some embodiments, each of the at least two context feature vectors includes: a user state feature vector corresponding to a user, and a relationship feature vector between the user and other users.

[0074] In some embodiments, the multi-head self-attention module described above can process the state feature vectors of multiple users, thereby capturing the complex dependencies between feature vectors and enhancing the model's expressive power and learning efficiency.

[0075] In some embodiments, these feature vectors are fed into a multi-head self-attention module. Unlike traditional methods that directly concatenate vectors, this module utilizes the following formula to calculate the interactions between user features:

[0076] (4)

[0077] In some embodiments, the context feature vector generated by this method It not only includes the user's own information, but also incorporates the user's relative distance and interference relationship with other users, thereby dynamically generating attention weights that reflect the user's real-time priority.

[0078] Step 203: The decision-making device performs an aggregation operation on at least two context feature vectors to obtain an aggregated context feature vector.

[0079] It should be noted that aggregation operations apply specific functions (such as summation, average, count, etc.) to a data set (such as arrays, lists, database tables, etc.) to transform scattered data into meaningful summary results. Here, the summation function is used.

[0080] In some embodiments, combined with Figure 2 ,like Figure 5 As shown, step 203 above can be implemented through step 203a as follows.

[0081] Step 203a: The decision-making device sums at least two context feature vectors using a permutation-invariant aggregation function to obtain an aggregated context feature vector.

[0082] In some embodiments, to ensure that the network is insensitive to the order of user input, the system uses a summation operation to aggregate the context features of all users to obtain an aggregated context feature vector. .

[0083] In some embodiments, based on the above-described 3D channel model, this invention jointly models the trajectory planning of the UAV and the beamforming problem of the RIS as a Markov decision process (MDP). In this MDP architecture, the state space... It is designed as a composite set that not only includes the UAV's real-time three-dimensional position and remaining energy, but also aggregates the relative position coordinates of all users and the current channel quality information.

[0084] In some embodiments, this method does not use traditional vector concatenation. Instead, it first embeds the user state independently, then extracts the relationship features between users through a self-attention mechanism, and finally generates the state representation by using a permutation-invariant aggregation function (such as summation). This solves the input order sensitivity problem and breaks through the information bottleneck.

[0085] Step 204: The decision-making device generates actions or Q-values ​​in the MDP based on the aggregated context feature vector.

[0086] In some embodiments, the action in the MDP can be a driving state transition, from state s1 to state s2.

[0087] In some embodiments, the Q value in the MDP can be the expected cumulative discount reward that can be obtained by following a certain strategy (such as the optimal strategy) after taking action a in state s.

[0088] In some embodiments, combined with Figure 2 ,like Figure 6 As shown, step 204 above can be implemented through steps 204a to 204b.

[0089] Step 204a: The decision-making device performs a concatenation operation on the aggregated context feature vector and the UAV feature vector to obtain a concatenated feature vector.

[0090] In some embodiments, the UAV feature vector is obtained by mapping the UAV state through the entity embedding layer of the first model.

[0091] In some embodiments, the above-mentioned aggregated context feature vector is obtained by aggregating the context feature vector using an aggregation function.

[0092] Step 204b: The decision-making device processes the concatenated feature vector through the fully connected layer of the first model to obtain the action or Q value in the MDP.

[0093] In some embodiments, the aggregated context feature vector is concatenated with the UAV features and then input into a fully connected layer to generate the final action or Q-value.

[0094] The decision-making method provided in this application embodiment can obtain feature vectors of multiple users and the relationships between users, and aggregate the user feature vectors to generate actions or Q-values ​​in the MDP. This allows for handling the permutation invariance of user inputs, thereby accelerating training convergence and improving the system's energy efficiency.

[0095] In some embodiments, combined with Figure 2 ,like Figure 7 As shown, after step 204 above, the decision-making method provided in this application embodiment may further include steps 205a to 205e.

[0096] Step 205a: After receiving the action from the MDP, the decision-making device executes the action.

[0097] In some embodiments, the system initializes the Actor network and two Critic networks. At each time step... UAV obtains current status The input is then fed into the Actor network, which uses the aforementioned attention mechanism to identify key users at the current moment (e.g., users at the coverage edge or with deteriorating channel conditions), and outputs a joint action that includes the flight speed and RIS phase adjustment for the next moment. .

[0098] In some embodiments, joint action The action space consists of the flight control variables of the UAV and the electromagnetic control variables of the RIS, specifically including the three-dimensional velocity vector of the UAV. and the continuous change in RIS phase shift .

[0099] Step 205b: The decision-making device obtains the reward function and state function after the action is executed, and stores the reward function and state function in the playback pool.

[0100] In some embodiments, an environmental feedback reward is obtained after performing this action. and new status This information is stored in the playback pool as an experience sample.

[0101] In some embodiments, in order to guide the agent to learn a strategy that maximizes the system's energy efficiency (EE), the reward function... Defined as a weighted sum of instantaneous energy efficiency and constraint penalty terms, i.e.:

[0102] (5)

[0103] The first term represents the system and rate per unit energy consumption, and the second term... It is a penalty function; when the UAV flies out of the preset boundary or its speed exceeds the physical limit, the system will apply a penalty coefficient. This forces the agent to explore the optimal solution within the feasible domain.

[0104] Step 205c: The decision-making device randomly samples from the playback pool to obtain sampled data.

[0105] In some embodiments, during the model update phase, the decision-making device randomly samples from the playback pool to obtain sampled data.

[0106] In some embodiments, the above-mentioned sampling data may be environmental feedback rewards obtained through random sampling. and new status Empirical samples, etc.

[0107] Step 205d: The decision-making device determines the first Q value based on the sampled data.

[0108] In some embodiments, combined with Figure 7 ,like Figure 8 As shown, step 205d can be implemented through steps 205d1 and 205d2.

[0109] Step 205d1: The decision-making device processes the sampled data through the dual critic network of the first model to obtain the second Q value.

[0110] It should be noted that the dual critic network is responsible for estimating the Q-value. By introducing two independently trained critic networks, the minimum of their outputs is taken as the target value, thereby improving the accuracy of value estimation.

[0111] In some embodiments, the decision-making device uses a dual-critic network to calculate the target Q-value and obtain a second Q-value.

[0112] In some embodiments, the decision-making device obtains a second Q value by minimizing the Q value to suppress the overestimation problem:

[0113] (6)

[0114] Step 205d2: The decision-making device performs TD error minimization processing on the second Q value to obtain the first Q value.

[0115] In some embodiments, the smaller the absolute value of the TD error, the more accurate the value function's prediction of future returns.

[0116] In some embodiments, the Q-value is obtained through a dual-critic network, and the obtained Q-value is then processed to minimize the TD error, thereby improving the accuracy of value assessment.

[0117] Step 205e: The decision-making device updates the first model using the first Q value to obtain the updated first model.

[0118] In some embodiments, the decision-making device uses a first Q value to update the network parameters by minimizing the TD error, thereby obtaining an updated first model.

[0119] Thus, through the above steps, a closed-loop interactive learning process can be achieved by using an environmental reward feedback function and a new state. The target Q value is calculated by randomly sampling from the replay pool and taking the minimum value to suppress the overestimation problem of Q value, thereby accelerating the training convergence speed and improving the energy efficiency of the system.

[0120] The decision-making method of this application will be described below through specific embodiments.

[0121] The implementation process of the decision-making method provided in this application includes the following S1 to S4:

[0122] S1. The decision-making device acquires at least two user state feature vectors, each user state feature vector corresponding to a user.

[0123] This embodiment constructs a UAV-RIS-assisted downlink communication system, which includes a ground base station (BS), a rotary-wing UAV equipped with a RIS, and... There are several ground users. Due to the obstruction of tall buildings in the city, the direct link between the base station and the users is blocked, and communication relies entirely on the reflection link provided by the UAV-RIS. In this scenario, the UAV flies in three-dimensional space, and its position is at the [missing information]. Each moment can be represented as a three-dimensional coordinate, which can be specifically represented by the above formula (1).

[0124] This application proposes an attention-enhanced dual-delay deep deterministic policy gradient (A-TD3) algorithm, whose network architecture is as follows: Figure 9 As shown.

[0125] The core innovation of this architecture lies in the reconstruction of the traditional Actor and Critic networks, introducing a hierarchical structure of "embedding-attention-aggregation". Specifically, the system first maps the UAV state and the state of each user into high-dimensional feature vectors through an entity embedding layer. and .

[0126] S2. The decision-making device processes at least two user state feature vectors through the multi-head self-attention module of the first model to obtain at least two context feature vectors.

[0127] These feature vectors are input into the multi-head self-attention module. Unlike traditional methods that directly concatenate vectors, this module can calculate the interactions between user features using the above formula (4).

[0128] This mechanism enables the generation of context-aware features. It not only includes the user's own information, but also incorporates the user's relative distance and interference relationship with other users, thereby dynamically generating attention weights that reflect the user's real-time priority.

[0129] To accurately simulate real-world communication environments, this embodiment employs a 3D channel model combining large-scale fading and line-of-sight (LoS) components. For the... The user, in the 1st The achievable rate per time slot It can be expressed according to the above formula (2).

[0130] Furthermore, given the limited onboard energy of UAVs, this invention employs an energy consumption model based on rotor hydrodynamics to evaluate system costs. propulsion power per time slot Specifically, it is expressed as: Formula (3) above, which will not be repeated here.

[0131] S3. The decision-making device performs an aggregation operation on at least two context feature vectors to obtain an aggregated context feature vector.

[0132] To ensure the network is insensitive to the order of user input, the system uses a summation operation to aggregate the contextual features of all users. .

[0133] Based on the aforementioned 3D channel model, this invention jointly models the trajectory planning of UAVs and the beamforming problem of RIS as a Markov Decision Process (MDP). In this MDP architecture, the state space... It is designed as a composite set that not only includes the UAV's real-time three-dimensional position and remaining energy, but also aggregates the relative position coordinates of all users and the current channel quality information.

[0134] S4. The decision-making device generates actions or Q-values ​​in the MDP based on the aggregated context feature vector.

[0135] The decision-making device concatenates the aggregated context feature vector with the UAV features and inputs it into the fully connected layer to generate the final action or Q value.

[0136] The above actions The action space consists of the flight control variables of the UAV and the electromagnetic control variables of the RIS, specifically including the three-dimensional velocity vector of the UAV. and the continuous change in RIS phase shift To guide the agent in learning a strategy that maximizes the system's energy efficiency (EE), the reward function... Defined as the weighted sum of instantaneous energy efficiency and constraint penalty terms, it can be expressed by the above formula (5), which will not be elaborated here.

[0137] When a UAV flies out of the preset boundary or exceeds the physical limit, the system will apply a penalty coefficient. This forces the agent to explore the optimal solution within the feasible domain.

[0138] In the initial phase, the system initializes the Actor network and two Critic networks. At each time step... UAV obtains current status The input is then fed into the Actor network, which uses the aforementioned attention mechanism to identify key users at the current moment (e.g., users at the coverage edge or with deteriorating channel conditions), and outputs a joint action that includes the flight speed and RIS phase adjustment for the next moment. After performing this action, the environment provides a reward. and new status This information is stored in the replay pool as empirical samples. During the model update phase, the algorithm randomly samples from the replay pool, calculates the target Q value using a dual Critic network, and suppresses the overestimation problem of the Q value by taking the minimum value: this can be expressed by the above formula (6).

[0139] Then, the network parameters are updated by minimizing the TD error.

[0140] Thus, the technical solution provided in this application brings at least the following beneficial effects: it can obtain multiple user state feature vectors and relationship feature vectors between users, and perform aggregation operations on the obtained feature vectors to obtain aggregated context feature vectors, and then generate actions or Q-values ​​in the MDP based on the aggregated context feature vectors. This allows user states to be treated as an unordered set, explicitly models the relationship dependencies between users, and achieves dynamic attention to key users, i.e., it can handle the permutation invariance of user input. This can accelerate model training convergence speed and improve the system's energy efficiency.

[0141] Furthermore, the practical effect of this application is demonstrated through… Figure 10 The trajectory visualization shown provides intuitive verification.

[0142] Furthermore, Figure 10 The dynamic trajectory in a multi-user scenario is shown: In the initial stage (time t1), the UAV is located in the center and performs normal coverage; when a user (such as User C) moves and causes a sharp deterioration in channel quality and faces the risk of QoS interruption (time t2), the A-TD3 algorithm automatically assigns a very high weight to the user through a self-attention mechanism (indicated by the bold line in the figure), driving the UAV to actively deviate from the original trajectory and move toward the user; once the service quality is restored (time t3), the attention weight is immediately redistributed, and the UAV then adjusts its course to take into account global energy efficiency.

[0143] Furthermore, Figures 11 to 14 The performance of the algorithm in this application (A-TD3, green curve) was compared with existing mainstream algorithms (TD3, DDPG) and heuristic algorithms from different dimensions. The horizontal axis represents the number of users (K), and the vertical axis represents various performance metrics.

[0144] Among them, such as Figure 11 As shown, the average energy efficiency is illustrated. It can be seen that as the number of users increases, the energy efficiency growth curve of A-TD3 is the steepest, significantly outperforming the comparison algorithm, thus verifying its high efficiency in large-scale user scenarios.

[0145] like Figure 12 As shown, the system and speed are illustrated. The A-TD3 consistently maintains the highest communication speed.

[0146] like Figure 13 As shown, the QoS satisfaction rate is illustrated. When user density is high (K=20), the performance of the comparison algorithm degrades significantly, while A-TD3 still maintains a satisfaction rate of over 95%.

[0147] like Figure 14As shown, the fairness index is illustrated. The A-TD3 curve is the most stable, proving that the algorithm balances efficiency with fairness among users.

[0148] Furthermore, this application proposes a novel method for handling multi-user states in UAV-RIS deep reinforcement learning control. Instead of traditional vector concatenation, it first embeds user states independently, then extracts relational features between users through a self-attention mechanism, and finally generates state representations using a permutation-invariant aggregation function (such as summation). This method solves the input order sensitivity problem and overcomes the information bottleneck.

[0149] Furthermore, this application protects a specific neural network architecture, namely the attention-enhanced A-TD3 network architecture, which embeds an "embedding layer + self-attention layer + aggregation layer" structure at the front end of the Actor and Critic networks to process variable-length, unordered user set inputs and combine them with the UAV's own state to make decisions.

[0150] Furthermore, this application utilizes the aforementioned network architecture to achieve a control method that jointly outputs the UAV three-dimensional velocity vector and the RIS phase offset increment, with the goal of maximizing system energy efficiency.

[0151] Furthermore, this application can be applied to related wireless communication network equipment or future 6G network architecture products.

[0152] It should be noted that the descriptions of each step S1 to S4 in this embodiment can be found in the descriptions in the above embodiments, and will not be repeated here.

[0153] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there is no conflict, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0154] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] This application embodiment can divide the decision-making device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0156] In some embodiments, this application also provides a decision-making apparatus. The decision-making apparatus may include one or more functional modules for implementing the decision-making methods of the above method embodiments.

[0157] For example, Figure 15 This is a schematic diagram of a decision-making device provided in an embodiment of this application. Figure 15 As shown, the decision-making device 900 includes: an acquisition module 901, a processing module 902, an aggregation module 903, and an output module 904.

[0158] The acquisition module 901 is used to acquire at least two user state feature vectors, each corresponding to a user; the processing module 902 is used to process the at least two user state feature vectors through the multi-head self-attention module of the first model to obtain at least two context feature vectors, each of the at least two context feature vectors including: a user state feature vector corresponding to a user, and a relationship feature vector between the user and other users; the aggregation module 903 is used to perform an aggregation operation on the at least two context feature vectors to obtain an aggregated context feature vector; and the output module is used to generate actions or Q-values ​​in the MDP based on the aggregated context feature vectors.

[0159] The decision-making device provided in this application can acquire multiple user state feature vectors and relationship feature vectors between users, and perform aggregation operations on the acquired feature vectors to obtain aggregated context feature vectors. Then, actions or Q-values ​​in the MDP are generated based on the aggregated context feature vectors. This allows user states to be treated as an unordered set, explicitly modeling the dependencies between users, and achieving dynamic focus on key users, thus handling the permutation invariance of user inputs. This accelerates model training convergence and improves the system's energy efficiency.

[0160] In some embodiments, the acquisition module 901 is specifically used to acquire the states of at least two users; and to obtain at least two user state feature vectors by mapping the state of each user to a user state feature vector through the entity embedding layer of the first model.

[0161] In other embodiments, the aggregation module 903 is specifically used to sum at least two context feature vectors by using an aggregation function that does not change the value, so as to obtain an aggregated context feature vector.

[0162] In some other embodiments, the output module 904 is specifically used to concatenate the aggregated context feature vector and the UAV feature vector to obtain a concatenated feature vector; the UAV feature vector is obtained by mapping the UAV state through the entity embedding layer of the first model; and the concatenated feature vector is processed through the fully connected layer of the first model to obtain the action or Q value in the MDP.

[0163] In some other embodiments, the processing module 901 is further configured to execute the action after obtaining the action in the MDP; and obtain the reward function and state function after the action is executed, and store the reward function and state function in the replay pool; and randomly sample from the replay pool to obtain sampled data; determine a first Q value based on the sampled data; and use the first Q value to update the first model to obtain the updated first model.

[0164] In some other embodiments, the processing module 901 is specifically used to process the sampled data through the dual critic network of the first model to obtain a second Q value; and to perform TD error minimization processing on the second Q value to obtain a first Q value.

[0165] It should be noted that the decision-making device can implement all the processes implemented in the above method embodiments and achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0166] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 16 As shown, the electronic device 90 includes: a processor 92, a communication interface 93, and a bus 94. Optionally, the electronic device 90 may also include a memory 91.

[0167] Processor 92 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0168] Communication interface 93 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0169] The memory 91 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0170] As one possible implementation, the memory 91 can exist independently of the processor 92. The memory 91 can be connected to the processor 92 via a bus 94 and is used to store instructions or program code. When the processor 92 calls and executes the instructions or program code stored in the memory 91, it can implement the decision-making method provided in the embodiments of this application.

[0171] In another possible implementation, memory 91 can also be integrated with processor 92.

[0172] Bus 94 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 94 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 16 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0173] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0174] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above decision-making method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0175] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0176] This application also provides a readable storage medium storing a program or instructions that, when executed by a computer, implement the decision-making method provided in the above embodiments. It is understood that all or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware; the readable storage medium can be any of the foregoing embodiments or memory; the readable storage medium can also be an external storage device of the service invocation device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the service invocation device. Further, the readable storage medium can include both internal storage units of the service invocation device and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the service invocation device. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0177] This application also provides a computer program product, which is stored in a storage medium and implements the decision-making method provided in the above embodiments when executed by a computer.

[0178] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0180] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A decision-making method, characterized in that, include: Obtain at least two user state feature vectors, with each user state feature vector corresponding to one user; The multi-head self-attention module of the first model processes the at least two user state feature vectors to obtain at least two context feature vectors. Each of the at least two context feature vectors includes: a user state feature vector corresponding to a user, and a relationship feature vector between the user and other users. The at least two context feature vectors are aggregated to obtain an aggregated context feature vector. Based on the aggregated context feature vector, the actions or Q-values ​​in the Markov Decision Process (MDP) are generated.

2. The decision-making method according to claim 1, characterized in that, The acquisition of at least two user state feature vectors includes: Get the status of at least two users; Through the entity embedding layer of the first model, the state of each user is mapped to a user state feature vector, thus obtaining the at least two user state feature vectors.

3. The decision-making method according to claim 1, characterized in that, The aggregation operation on the at least two context feature vectors to obtain the aggregated context feature vector includes: The aggregated context feature vector is obtained by summing the at least two context feature vectors using a permutation-invariant aggregation function.

4. The decision-making method according to claim 1, characterized in that, The step of generating actions or Q-values ​​in the MDP based on the aggregated context feature vector includes: The aggregated context feature vector and the UAV feature vector are concatenated to obtain a concatenated feature vector; the UAV feature vector is obtained by mapping the UAV state through the entity embedding layer of the first model; The concatenated feature vectors are processed through the fully connected layer of the first model to obtain the actions or Q-values ​​in the MDP.

5. The decision-making method according to any one of claims 1 to 4, characterized in that, The method further includes: After obtaining the action from the MDP, execute the action; Obtain the reward function and state function after the action is executed, and store the reward function and the state function in the replay pool; Random sampling is performed from the playback pool to obtain sampled data; Based on the sampled data, a first Q value is determined; The first model is updated using the first Q value to obtain the updated first model.

6. The decision-making method according to claim 5, characterized in that, Determining the first Q value based on the sampled data includes: The sampled data is processed by the dual critic network of the first model to obtain the second Q value; The second Q value is processed by minimizing the TD error to obtain the first Q value.

7. A decision-making device, characterized in that, include: The module consists of an acquisition module, a processing module, an aggregation module, and an output module. The acquisition module is used to acquire at least two user state feature vectors, each user state feature vector corresponding to one user; The processing module is used to process the at least two user state feature vectors through the multi-head self-attention module of the first model to obtain at least two context feature vectors. Each of the at least two context feature vectors includes: a user state feature vector corresponding to a user, and a relationship feature vector between the user and other users. The aggregation module is used to perform an aggregation operation on the at least two context feature vectors to obtain an aggregated context feature vector. The output module is used to generate actions or Q-values ​​in the MDP based on the aggregated context feature vector.

8. The decision-making device according to claim 7, characterized in that, The acquisition module is specifically used to acquire the states of at least two users; and through the entity embedding layer of the first model, to map the state of each user into a user state feature vector, thereby obtaining the at least two user state feature vectors.

9. The decision-making device according to claim 7, characterized in that, The aggregation module is specifically used to sum the at least two context feature vectors using an aggregation function that is permutation-invariant, to obtain the aggregated context feature vector.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions, when executed by the processor, implementing the decision-making method as described in any one of claims 1-6.

11. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a computer, implement the decision-making method as described in any one of claims 1-6.

12. A computer program product, characterized in that, The computer program product is stored in a storage medium, and when executed by a computer, the computer program product implements the decision-making method as described in any one of claims 1-6.

Images