RIS-assisted multi-user communication beam forming optimization method based on KAN-SAC

Abstract

The invention discloses an RIS assisted multi-user communication beam forming optimization method based on KAN-SAC, and belongs to the field of reconfigurable intelligent surface assisted communication. According to the method, the scene of RIS-assisted multi-user multi-input single-output communication is considered, the KAN-SAC algorithm is utilized to optimize the active beam forming of the base station and the passive beam forming of the RIS, then the system performance is maximized under implicit channel state information (CSI), and complex and expensive explicit channel estimation is avoided. According to the method, complex nonlinear mapping is better expressed through two layers of KAN modules, a softmax approximate quantization mechanism is fused in action output, the blocking problem of discrete actions on gradient propagation is effectively relieved, the model can process continuous and discrete hybrid control at the same time, and the method is suitable for large-scale popularization and application. And the method is simultaneously adaptive to the continuous active beam control of the base station and the discrete phase selection of the RIS. According to the method, the joint optimization problem of active beam forming and passive beam forming can be effectively solved without depending on perfect or imperfect explicit channel estimation.

Description

Technical Field

[0001] The present invention relates to the field of reconfigurable intelligent surface (RIS)-assisted communications, and in particular to a RIS-assisted multi-user multiple-input single-output (MU-MISO) system joint beamforming optimization method and system based on Kolmogorov-Arnold network (KAN)-soft actor-critic (SAC), which realizes the joint optimization of base station active beamforming and RIS passive beamforming through non-explicit channel state information (CSI). Background Art

[0002] RIS consists of multiple low-cost passive reflectors that can intelligently adjust the incident signal through a controller, thereby reconfiguring the wireless propagation environment, which is conducive to enhancing communication performance. In particular, in obstructed scenarios, the transmission rate of the communication system can be improved by placing a reconfigurable smart metasurface.

[0003] In recent years, RIS-assisted MU-MISO systems have garnered widespread attention. Researchers have proposed maximizing system transmission rates by optimizing the base station's active beamforming and the RIS's passive beamforming. However, most existing studies assume perfect channel state information and rarely consider imperfect channel state information. Traditional methods rely on explicit channel estimation, resulting in high computational overhead and resource consumption. While existing deep reinforcement learning-based solutions avoid explicit CSI, they still suffer from issues such as insufficient policy expressiveness and poor adaptability to hardware constraints. For example, existing SAC algorithms use a multilayer perceptron (MLP) as a policy network, making it difficult to efficiently approximate complex beamforming strategies and failing to fully consider the hardware limitations of RIS phase discretization. Therefore, there is an urgent need for a joint beamforming optimization method that does not require explicit channel estimation, adapts to hardware constraints, and possesses high expressiveness. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a RIS-assisted multi-user communication beamforming method based on KAN-SAC. For the RIS-assisted MU-MISO system, considering that perfect channel state information is not required, a balance between complexity and performance is achieved by jointly optimizing the active beamforming of the base station and the passive beamforming of the RIS. Considering directly utilizing the transmitted uplink pilot to jointly optimize the downlink beamforming, the downlink transmission rate of the RIS-assisted MU-MISO system can be maximized.

[0005] Technical solution: To achieve the above-mentioned purpose, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a RIS-assisted multi-user communication beamforming method based on KAN-SAC, comprising the following steps:

[0007] A RIS-assisted MU-MISO system model is constructed, and an optimization problem is formulated with the base station's active beamforming and the RIS's passive beamforming as optimization variables, with the goal of maximizing the downlink system rate.

[0008] Construct an aperiodic Markov decision model. The state space includes the decoupled pilot signal, the base station's active beamforming matrix at the previous time step, and the RIS's passive beamforming vector. The action space includes the real and imaginary parts of the base station's active beamforming matrix and the RIS's passive beamforming vector. The reward function is defined as the system's total downlink rate at each time step minus the average reward of previous time steps.

[0009] A deep reinforcement learning network architecture based on KAN-SAC is designed and trained. The policy network consists of two layers of KAN modules, a mean output layer, and a logarithmic standard output layer. The input dimension of the first layer KAN module is the dimension of the state space, and the output is sent to the second layer KAN module, which in turn sends the output to the mean output layer and the logarithmic standard output layer. A nonlinear mapping combining K-nearest neighbor sample selection and learnable spline fitting is used between the input and output of the two layers of KAN modules. The mean output layer is based on a fully connected layer, and the output dimension is the action space dimension. The structure of the logarithmic standard output layer is the same as that of the mean output layer, with an additional truncation operation to limit the output of the logarithmic standard to a preset minimum and maximum value. The mean output layer and the logarithmic standard deviation output layer are used to generate the mean and logarithmic standard deviation of the Gaussian action distribution, respectively. A softmax-weighted quantization method is used to map the RIS passive phase shift output by the policy network to a discrete set, achieving approximate discretization of the phase shift while maintaining differentiability.

[0010] According to the optimized deep reinforcement learning model, the optimal active beamforming and RIS phase shift are obtained based on the transmitted uplink pilot.

[0011] Furthermore, at time step t, the state of the system s (t) Including decoupled pilot signal The base station active beamforming matrix W at the previous time step t-1 (t-1) , and the RIS passive beamforming vector θ at the previous time step (t-1) , expressed as:

[0012]

[0013] Where vec(·) represents the column vector of the stacked matrix to realize the vectorized operation of the matrix. represents the operation of taking the real part of the complex matrix, Indicates the operation of taking the imaginary part of the complex matrix;

[0014] Action a (t) Contains the base station active beamforming matrix W (t) The real and imaginary parts of the RIS passive beamforming vector θ (t) , expressed as:

[0015]

[0016] The instantaneous reward of the system at each time step is It is defined as the total achievable rate of the downlink:

[0017]

[0018] where R k represents the downlink transmission rate of the kth user, K is the number of users, and the average reward mechanism is introduced. The reward used in training is r (t) It is defined as:

[0019]

[0020] in is the average reward estimated based on all time steps before time t, which is used to measure the improvement of the current strategy.

[0021] Furthermore, in order to meet the total transmission power of the base station Constraints, the generated actions are normalized before application; where P d is the downlink transmit power constraint, w k is the kth column vector of the active beamforming matrix.

[0022] Furthermore, the decoupled pilot signal Where L is the pilot sequence length, Y is the pilot signal received by the base station, The pilot sequence is organized into a pilot matrix by row, is the effective uplink channel matrix, P u Indicates the transmit power of the uplink pilot. represents the equivalent noise.

[0023] Furthermore, in the policy network, the first layer KAN module input state Output vector Among them A in =4MK+N,A out =2(A in-1), M represents the number of base station antennas, N is the number of RIS reflection units, and K is the number of users; the nonlinear mapping relationship between input and output is:

[0024]

[0025] where β c (s i ) represents the cth learnable spline basis function corresponding to the i-th state variable, C is the number of learnable spline basis functions, is the trainable weight of the first layer of KAN;

[0026] The input of the second layer KAN module is z (1) , dimension is A out , the output is The dimension is h, and the mapping method is also:

[0027]

[0028] in is the trainable weight of the second layer KAN; After the mean output layer and the logarithmic standard deviation output layer generate the mean μ and logarithmic standard deviation logσ of the Gaussian action distribution, the logarithmic standard deviation is converted into the standard deviation through the exponential function, and a continuous Gaussian distribution is constructed. The distribution is sampled using the reparameterization method:

[0029]

[0030] Thus, a derivable action output is obtained, where ⊙ represents element-by-element multiplication, Indicates that from the standard normal distribution The sampled random noise, I is the identity matrix, and the action output of the final policy network includes the real and imaginary parts of the unnormalized base station active beamforming matrix and the unquantized passive beamforming vector of RIS.

[0031] Furthermore, the deep reinforcement learning network architecture based on KAN-SAC also includes two Q networks. and Indicates that the parameters are φ1 and φ2, and each Q value network is equipped with a target network. and Indicates that the parameters are φ′1 and φ′2 respectively. Each Q network is a three-layer fully connected structure. The input dimension is the vector dimension after the state and action are concatenated. The output dimension is 1, and the output is two scalar Q value estimates.

[0032] Furthermore, the quantization method is: for each unquantized phase shift value output by the policy network and discrete phase sets Calculate the absolute difference for each value in the discrete phase set There are 2 F discrete phase values, is the fth quantized phase, and then these differences are multiplied by a predetermined scaling factor η, and the negative value is taken as the softmax input to calculate the corresponding weight distribution. The quantized phase shift value is expressed as where w n,f is the unquantized phase shift value For discrete phase sets The weight of the f-th quantized phase in .

[0033] In a second aspect, the present invention provides a RIS-assisted multi-user communication beamforming optimization system based on KAN-SAC, comprising:

[0034] The problem modeling module is used to build a RIS-assisted MU-MISO system model and establish an optimization problem with the base station's active beamforming and the RIS's passive beamforming as optimization variables, with the goal of maximizing the downlink system rate. It also constructs an aperiodic Markov decision algorithm. The state space includes the decoupled pilot signal, the base station's active beamforming matrix at the previous time step, and the RIS's passive beamforming vector. The action space includes the real and imaginary parts of the base station's active beamforming matrix and the RIS's passive beamforming vector. The reward function is defined as the system's total downlink rate at each time step minus the average reward of previous time steps.

[0035] The network construction and training module is used to design and train a deep reinforcement learning network architecture based on KAN-SAC. The policy network includes two layers of KAN modules, a mean output layer, and a logarithmic standard output layer. The input dimension of the first layer of KAN module is the dimension of the state space, and the output is sent to the second layer of KAN module. The second layer of KAN module outputs to the mean output layer and the logarithmic standard output layer. The input and output of the two layers of KAN modules are both mapped using a nonlinear mapping that combines K-nearest neighbor sample selection and learnable spline fitting. The mean output layer is based on a fully connected layer, and the output dimension is the action space dimension. The structure of the logarithmic standard output layer is the same as that of the mean output layer, with an additional truncation operation to limit the output of the logarithmic standard to between the preset minimum and maximum values. The mean output layer and the logarithmic standard deviation output layer are used to generate the mean and logarithmic standard deviation of the Gaussian action distribution, respectively. A softmax-weighted quantization method is used to map the RIS passive phase shift output by the policy network to a discrete set, achieving approximate discretization of the phase shift while maintaining differentiability.

[0036] The beamforming optimization module is used to obtain the optimal active beamforming and RIS phase shift based on the transmitted uplink pilot according to the optimized deep reinforcement learning model.

[0037] In a third aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method are implemented.

[0038] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method.

[0039] Beneficial effects: Compared with the prior art, the present invention proposes a RIS-assisted multi-user communication beamforming optimization method based on KAN-SAC, and proposes a KAN-SAC model. Its policy network has better expression ability for complex nonlinear mappings, and incorporates a softmax approximate quantization mechanism into the action output, which effectively alleviates the blocking problem caused by discrete actions (such as RIS phase selection) on gradient propagation, enabling the model to simultaneously process continuous and discrete hybrid control, so that it can simultaneously adapt to the continuous active beam control of the base station and the discrete phase selection of RIS. The present invention bypasses explicit channel estimation and directly uses pilot signals as input, reducing pilot overhead and computational complexity, adaptively optimizing RIS-assisted multi-user communication beamforming, and maximizing the total downlink transmission rate. It provides a key technical path for the large-scale deployment of RIS in future 6G networks, has both theoretical innovation and engineering practicality, and is of great significance to the performance improvement and cost optimization of intelligent wireless communication systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of an optimization method according to an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of the RIS-assisted MU-MISO model provided in an embodiment of the present invention;

[0042] Figure 3 The graph below shows the variation of downlink total transmission rate with base station transmit power under different algorithms. DETAILED DESCRIPTION

[0043] The technical solutions of the present invention are described in detail below through the accompanying drawings and specific embodiments. The embodiments described with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0044] In the field of wireless communications, beamforming optimization of multi-user multiple-input single-output (MU-MISO) systems assisted by reconfigurable intelligent metasurfaces (RIS) is key to improving communication performance. This embodiment of the present invention proposes a RIS-assisted multi-user communication beamforming optimization method based on KAN-SAC, aiming to achieve efficient beamforming optimization and improve the total downlink transmission rate. Unlike traditional methods, this solution does not require precise channel estimation, and reduces complexity and enhances system adaptability through innovative algorithm design. The following describes its specific implementation steps in detail from four main parts: system model construction and optimization problem establishment, problem transformation and environment construction, algorithm design and training that integrates KAN-SAC and approximate quantization strategies, and the execution stage.

[0045] Step 1: Construct a MU-MISO system model that includes a base station, RIS, and multiple users. Define the formulas related to channels, reflection matrices, and transmission rates. Use the RIS to assist in the reciprocity of the uplink and downlink channels of the multi-user, multiple-input, single-output (MU-MISO) system. Establish an uplink-based pilot signal. Use the base station's active beamforming and the RIS's passive beamforming as optimization variables, and solve an optimization problem aimed at maximizing the downlink system rate. Specifically, the following is the solution:

[0046] (1a) Construct a system model such as Figure 2 As shown in the figure, consider that the base station is equipped with M antennas, and the RIS contains N controllable reflection units, serving K single-antenna users. M represents the number of base station antennas, N is the number of RIS reflection units, and K is the number of users. For the downlink, we define represents the direct link channel from the base station to user k, G r Indicates the channel from the base station to the RIS, Denotes the channel from RIS to user k, considering the reciprocity of uplink and downlink channels, that is, the uplink channel matrix is ​​the transpose of the downlink channel matrix.

[0047] RIS passive beamforming vector The RIS passive beamforming matrix is ​​represented by Indicates that the symbol diag means placing a set of complex elements in the main diagonal position of the square matrix in sequence and setting the remaining elements to zero, thereby constructing the corresponding diagonal matrix. represents the phase offset of the nth RIS reflection unit, which controls the phase direction of the reflected signal, and Quantize to a set:

[0048]

[0049] Where F is the number of quantization bits, which represents the number of phase quantization bits, that is, each phase unit can be expressed as 2 F discrete phase values, represents the set of all possible discrete phase offsets.

[0050] The effective downlink channel from the base station to user k is Indicates that it can be achieved through Calculate the downlink transmission rate of user k as follows:

[0051]

[0052] where w k is the base station’s active beamforming vector for user k, is the variance of zero-mean complex Gaussian noise.

[0053] (1b) Construct an optimization problem. A joint beamforming optimization problem is constructed to maximize the total downlink transmission rate. It is based on uplink pilot transmission, skips channel estimation, does not require perfect channel state information, and uses the base station's active beamforming matrix and the RIS's passive beamforming vector as joint optimization variables. Specifically, in the uplink phase, each user sends a pilot sequence of length L to the base station. To achieve effective user separation at the base station, the pilot sequences assigned to different users are orthogonal to each other. These pilot sequences are organized into a pilot matrix by row. Meet XX H =LP u I K , where P u Specifically, the pilot sequence of the kth user is recorded as Corresponding to the kth row of matrix X, the pilot signal reaches the base station through the direct link and the RIS auxiliary link. Therefore, the pilot signal received by the base station can be expressed as:

[0054]

[0055] in represents the equivalent uplink channel matrix, represents additive white Gaussian noise. Under the assumption that there is no accurate channel state information, the only known information available to the system is Y. Secondly, in the downlink phase, the system needs to jointly design: the active beamforming matrix of the base station and RIS passive beamforming vectors with and Based on the above variables, the goal is to maximize the total rate of the system in the downlink. The optimization problem can be formally expressed as:

[0056]

[0057] where w k is the kth column vector of the active beamforming matrix, Pd is the maximum total downlink transmit power constraint of the base station.

[0058] Step 2: Convert the joint beamforming optimization problem into an aperiodic Markov decision process. Considering the base station as an intelligent agent, the base station receives the state of the RIS-assisted MU-MISO system at each time step t. Based on this state, the base station and RIS jointly execute an action through the controller. After the action is executed, the system returns a reward corresponding to the optimization objective. Specifically, the state, action, and reward are set as follows:

[0059] (2a) State construction: The base station uses the orthogonality of the user pilot sequence to decouple the received pilot signal. The decoupled signal is:

[0060]

[0061] Where Y is the pilot signal received by the base station, is the effective uplink channel matrix, represents equivalent noise;

[0062] At time step t, the state of the system s (t) Including decoupled pilot signal The base station active beamforming matrix W at the previous time step t-1 (t-1) , and the RIS passive beamforming vector θ at the previous time step (t-1) ,Right now:

[0063]

[0064] Where vec(·) represents the column vector of the stacked matrix to realize the vectorized operation of the matrix. represents the operation of taking the real part of the complex matrix, Represents the operation of taking the imaginary part of a complex matrix.

[0065] (2b) Action construction: Action a (t) Contains the base station active beamforming matrix W (t) The real and imaginary parts of the RIS passive beamforming vector θ (t) , expressed as:

[0066]

[0067] In order to meet the total transmission power of the base station (P d is the downlink transmit power constraint), the generated actions need to be normalized before being applied.

[0068] (2c) Reward construction: The instantaneous reward of the system at each time step is expressed as It is defined as the total achievable rate of the downlink:

[0069]

[0070] where R k represents the downlink transmission rate of the kth user. In order for the agent to formulate long-term strategies for non-periodic tasks, the impact of historical behavior results must be considered. The instantaneous reward itself is not enough to guide training. Therefore, the average reward mechanism is introduced, and the reward used in training is r (t) It is defined as:

[0071]

[0072] in is the average reward estimated based on all time steps before time t, which is used to measure the improvement of the current strategy.

[0073] Step 3: To optimize the above non-periodic Markov decision, a deep reinforcement learning network architecture based on KAN-SAC is designed, which specifically includes a policy network and two Q networks. The policy network is represented by π θ (a|s) represents the parameter θ, which consists of two layers of KAN modules, a mean output layer, and a logarithmic standard output layer. The input dimension of the first layer KAN module is A in is the dimension of the state function, i.e. A in =4MK+N, output dimension A out It is twice the input dimension minus one, that is, A out =2(A in -1), a nonlinear mapping combining K-nearest neighbor sample selection and learnable spline fitting is used between input and output.

[0074] The first layer of the KAN module is specifically the input state Output vector The nonlinear mapping relationship between input and output is:

[0075]

[0076] where β c (s i ) represents the cth learnable spline basis function corresponding to the i-th state variable. In this embodiment, the learnable spline basis function adopts a uniformly distributed cubic B-spline, C is the number of learnable spline basis functions, is the trainable weight of the first layer KAN; similarly, the specific architecture of the second layer KAN module is that the input is z (1) , dimension is A out , the output is The dimension is h, here h = 256, and the mapping method is also:

[0077]

[0078] in is the trainable weight of the second-layer KAN; the mean output layer is based on the fully connected layer, the input is the output dimension of the second-layer KAN, that is, h, and the output dimension is the action space dimension, that is, N+2MK; the structure of the logarithmic standard output layer is the same as the mean output layer, and is expressed as:

[0079] u=θ u z (2) +b u

[0080] logσ=clip(θ σ z (2) +b σ ,logσ min ,logσ max )

[0081] Where clip represents the truncation operation that limits the output of the logarithmic standard to between the minimum and maximum values ​​set in the preview. They represent the linear coefficient trainable parameters of the mean output layer and the logarithmic standard deviation output layer, They represent the bias trainable parameters of the mean output layer and the logarithmic standard deviation output layer respectively. After converting the logarithmic standard deviation into the standard deviation through the exponential function, a continuous Gaussian distribution is constructed and the distribution is sampled using the reparameterization method:

[0082]

[0083] This allows for a differentiable action output, ensuring that the policy gradient can be effectively back-propagated. The final action output of the policy network includes the real and imaginary parts of the unnormalized base station active beamforming matrix and the unquantized RIS passive beamforming vector, which can be expressed as:

[0084]

[0085] in represents the unnormalized base station active beamforming matrix, represents the passive beamforming vector of the unquantized RIS.

[0086] In addition, two Q networks are used and Indicates that the parameters are φ1 and φ2, and each Q value network is equipped with a target network. and Indicates that the parameters are φ′1 and φ′2 respectively. Each Q network is a three-layer fully connected structure. The input dimension is the vector dimension after the state and action are concatenated, that is, 2N+6MK. The output dimension is 1. In this embodiment, the intermediate dimensions are 256, 128, and 64 respectively, and the output is two scalar Q value estimates.

[0087] Step 4: Train the above deep reinforcement learning network and update the model parameters.

[0088] (4a) First, the unnormalized base station active beamforming matrix Normalize and quantize the passive beamforming vector of the unquantized RIS. Specifically, the active beamforming matrix that satisfies the transmit power is expressed as:

[0089]

[0090] The softmax-weighted quantization method is used to map the unquantized RIS passive beamforming vector output by the policy network to a discrete set, so as to achieve the approximate discretization of the phase shift and maintain differentiability. Specifically, each unquantized phase shift value output by the policy network is first mapped to a discrete set. and discrete phase sets Calculate the absolute difference for each value in the discrete phase set There are 2 F discrete phase values, For the fth quantized phase, these differences are then multiplied by a predetermined scaling factor η, and the negative value is taken as the softmax input to calculate the corresponding weight distribution, which is expressed by the following formula:

[0091]

[0092] where w n,f is the unquantized phase shift value For discrete phase sets The weight of the fth quantized phase in , so the quantized phase shift value is expressed as:

[0093]

[0094] (4b) Deep reinforcement learning model training is based on the maximum entropy framework to find the optimal policy π * , expressed as:

[0095]

[0096] where π is the strategy, π(a t |s t ) represents the policy function, given the state s t Take action a tThe probability of r(s t ,a t ) is the immediate reward function, indicating that in state s t Next take action t The immediate benefits obtained, Indicates that the policy is in state s t The entropy under the policy reflects the randomness of the strategy, α is the entropy coefficient, which is used to adjust the trade-off between the reward term and the entropy term, ρ π Indicates that under the strategy π, the state-action pair (s t ,a t ),

[0097] The policy network is updated by maximizing the entropy regularized expected value function, which is expressed as:

[0098]

[0099] in, Represents the experience replay buffer, s represents the experience replay buffer The batch status taken out from Represents the policy network π θ Batch actions generated for each state in s.

[0100] (4c) The Q-value network is used to evaluate the t and action a t Under this condition, the expected cumulative reward that the agent can obtain in the future is expressed as

[0101]

[0102] in, represents the experience replay buffer, s, a, r, s' represent the replay buffer from the experience The batched states, actions, rewards, and the states of the next time slot are taken out. is the target Q value, defined as:

[0103]

[0104] Where γ is the discount factor used to measure the decay of future rewards, a′~π θ (·|s′) represents the action of sampling the next state from the current policy. The parameters of the Q-value network are iteratively optimized by backpropagating the loss function and using the gradient descent algorithm.

[0105] (4d) For stable training, the target network adopts soft update method, i.e.

[0106] φ′ i ←τφ i+(1-τ)φ′ i ,i=1,2

[0107] where τ∈(0,1) is the soft update rate.

[0108] Step 5: In the execution phase, the trained model is loaded and beamforming is adjusted based on the pilot signal to improve system performance. Specifically:

[0109] (5a) Loading model: Loading the trained deep reinforcement learning model, which has learned the ability to select the optimal action (beamforming strategy) based on the system state during training.

[0110] (5b) Output beamforming results: The model constructs the system state based on the acquired uplink pilot signal and outputs the unnormalized active beamforming matrix of the base station Passive beamforming vector with unquantized RIS Achieve optimized adjustment of beamforming, improve system downlink transmission rate, and enhance communication performance.

[0111] (5c) Normalized unnormalized base station active beamforming matrix and the passive beamforming vector of the unquantized RIS Each unquantized phase shift value of Direct quantization to discrete phase sets The discrete value closest to it.

[0112] exist Figure 1 In

[15] , a flowchart of a RIS-assisted multi-user communication beamforming method based on an enhanced SAC algorithm is described. First, a deep reinforcement learning model for joint beamforming is trained, and the base station optimizes the joint beamforming based on the deep reinforcement learning model trained by the enhanced SAC algorithm.

[0113] exist Figure 2 In

[15] , the system model of RIS-assisted MU-MISO is described. It can be seen that based on the reciprocity of uplink and downlink, uplink pilots are used to optimize downlink joint beamforming.

[0114] exist Figure 3In the paper, it is described that the total downlink rate of different algorithms changes with the base station transmission power. As the downlink transmission power increases, the performance gap between the algorithm proposed in the present invention and the theoretical upper limit increases. However, since it does not require explicit channel estimation, the system overhead is greatly reduced, and the sum rate of the algorithm proposed in the present invention is better than the total transmission rate of other algorithms. The present invention uses the KAN network to significantly improve the expression ability of the strategy network for complex nonlinear mapping, so it is better than the traditional SAC network. Compared with the improved DDPG, the SAC algorithm adopted in the present invention is more suitable for integration with the softmax approximate quantization mechanism, and is more adaptable to the continuous active beam control of the adaptive base station and the discrete phase selection of RIS. This is a mixed control scenario of continuous and discrete actions, and has more advantages in practical applications.

[0115] Based on the same inventive concept, an embodiment of the present invention provides a RIS-assisted multi-user communication beamforming optimization system based on KAN-SAC, including: a problem modeling module, used to construct a RIS-assisted MU-MISO system model, and establish an optimization problem with the active beamforming of the base station and the passive beamforming of the RIS as optimization variables, and the goal of maximizing the downlink system rate; and construct a non-periodic Markov decision, the state space includes the decoupled pilot signal, the active beamforming matrix of the base station in the previous time step, and the passive beamforming vector of the RIS; the action space includes the real part and imaginary part of the active beamforming matrix of the base station and the passive beamforming vector of the RIS, and the reward function is defined as the total downlink rate of the system at each time step minus the average reward of the past time steps; a network construction and training module, used to design a deep reinforcement learning network architecture based on KAN-SAC and perform training, wherein the policy network includes two layers of KAN modules, a mean output layer and a The logarithmic standard output layer, the input dimension of the first-layer KAN module is the dimension of the state space, and the output is sent to the second-layer KAN module. The second-layer KAN module outputs to the mean output layer and the logarithmic standard output layer. The input and output of both KAN modules use a nonlinear mapping that combines K-nearest neighbor sample selection and learnable spline fitting. The mean output layer is based on a fully connected layer, and the output dimension is the action space dimension. The structure of the logarithmic standard output layer is the same as the mean output layer, with an additional truncation operation to limit the output of the logarithmic standard to between a preset minimum and maximum value. The mean output layer and the logarithmic standard deviation output layer are used to generate the mean and logarithmic standard deviation of the Gaussian action distribution, respectively. A softmax-based weighted quantization method is used to map the RIS passive phase shift output by the policy network to a discrete set, achieving approximate discretization of the phase shift while maintaining differentiability. The beamforming optimization module is used to obtain the optimal active beamforming and RIS phase shift based on the transmitted uplink pilot according to the optimized deep reinforcement learning model. The detailed implementation details of each module are referred to in the above method embodiment and are not repeated here.

[0116] An embodiment of the present invention further provides a computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method are implemented.

[0117] An embodiment of the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method.

[0118] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program code is executed by the processor or controller, the steps of the method of the present invention are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] Any process or method description described in a flowchart or otherwise herein may be understood to represent a module, segment, or portion of code comprising one or N executable instructions for implementing a custom logic function or process step, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which the embodiments of the present invention pertain. Any content not described in detail in this application belongs to the prior art known to those skilled in the art.

Claims

1. A RIS-assisted multi-user communication beamforming optimization method based on KAN-SAC, characterized in that: include: A RIS-assisted MU-MISO system model is constructed, and an optimization problem is formulated with the base station's active beamforming and the RIS's passive beamforming as optimization variables, with the goal of maximizing the downlink system rate. Construct an aperiodic Markov decision model. The state space includes the decoupled pilot signal, the base station's active beamforming matrix at the previous time step, and the RIS's passive beamforming vector. The action space includes the real and imaginary parts of the base station's active beamforming matrix and the RIS's passive beamforming vector. The reward function is defined as the system's total downlink rate at each time step minus the average reward of previous time steps. A deep reinforcement learning network architecture based on KAN-SAC is designed and trained. The policy network consists of two layers of KAN modules, a mean output layer, and a logarithmic standard output layer. The input dimension of the first layer KAN module is the dimension of the state space, and the output is sent to the second layer KAN module, which in turn sends the output to the mean output layer and the logarithmic standard output layer. A nonlinear mapping combining K-nearest neighbor sample selection and learnable spline fitting is used between the input and output of the two layers of KAN modules. The mean output layer is based on a fully connected layer, and the output dimension is the action space dimension. The structure of the logarithmic standard output layer is the same as that of the mean output layer, with an additional truncation operation to limit the output of the logarithmic standard to a preset minimum and maximum value. The mean output layer and the logarithmic standard deviation output layer are used to generate the mean and logarithmic standard deviation of the Gaussian action distribution, respectively. A softmax-weighted quantization method is used to map the RIS passive phase shift output by the policy network to a discrete set, achieving approximate discretization of the phase shift while maintaining differentiability. According to the optimized deep reinforcement learning model, the optimal active beamforming and RIS phase shift are obtained based on the transmitted uplink pilot.

2. The KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method according to claim 1, characterized in that: At time step t, the state of the system s (t) Including decoupled pilot signal The base station active beamforming matrix W at the previous time step t-1 (t-1) , and the RIS passive beamforming vector θ at the previous time step (t-1) , expressed as: Where vec(·) represents the column vector of the stacked matrix to realize the vectorized operation of the matrix. represents the operation of taking the real part of the complex matrix, Indicates the operation of taking the imaginary part of the complex matrix; Action a (t) Contains the base station active beamforming matrix W (t) The real and imaginary parts of the RIS passive beamforming vector θ (t) , expressed as: The instantaneous reward of the system at each time step is It is defined as the total achievable rate of the downlink: where R k represents the downlink transmission rate of the kth user, K is the number of users, and the average reward mechanism is introduced. The reward used in training is r (t) It is defined as: in is the average reward estimated based on all time steps before time t, which is used to measure the improvement of the current strategy.

3. The KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method according to claim 2, characterized in that: In order to meet the total transmission power of the base station Constraints, the generated actions are normalized before application; where P d is the downlink transmit power constraint, w k is the kth column vector of the active beamforming matrix.

4. The KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method according to claim 2, characterized in that: Decoupled pilot signal Where L is the pilot sequence length, Y is the pilot signal received by the base station, The pilot sequence is organized into a pilot matrix by row, is the effective uplink channel matrix, P u Indicates the transmit power of the uplink pilot. represents the equivalent noise.

5. The KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method according to claim 1, characterized in that: In the policy network, the first layer KAN module input state , output vector Among them A in =4MK+N,A out =2(A in -1), M represents the number of base station antennas, N is the number of RIS reflection units, and K is the number of users; the nonlinear mapping relationship between input and output is: where β b (s i ) represents the bth learnable spline basis function corresponding to the i-th state variable, B is the number of learnable spline basis functions, is the trainable weight of the first layer of KAN; The input of the second layer KAN module is z (1) , dimension is A out , the output is The dimension is h, and the mapping method is also: in is the trainable weight of the second layer KAN; logσ=clip(θ σ z (2) +b σ ,logσ min ,logσ max ) After the mean output layer and the logarithmic standard deviation output layer generate the mean μ and logarithmic standard deviation logσ of the Gaussian action distribution, the logarithmic standard deviation is converted into the standard deviation through the exponential function, and a continuous Gaussian distribution is constructed. The distribution is sampled using the reparameterization method: Thus, a derivable action output is obtained, where ⊙ represents element-wise multiplication and ∈ represents the output from the standard normal distribution. The sampled random noise, I is the identity matrix; the action output of the final policy network includes the real and imaginary parts of the unnormalized base station active beamforming matrix and the unquantized passive beamforming vector of RIS.

6. The KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method according to claim 1, characterized in that: The deep reinforcement learning network architecture based on KAN-SAC also includes two Q networks. and Indicates that the parameters are φ1 and φ2, and each Q value network is equipped with a target network. and Indicates that the parameters are φ1' and φ2' respectively. Each Q network is a three-layer fully connected structure. The input dimension is the vector dimension after the state and action are concatenated. The output dimension is 1, and the output is two scalar Q value estimates.

7. The KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method according to claim 1, characterized in that: The quantization method is: for each unquantized phase shift value output by the policy network and discrete phase sets Calculate the absolute difference for each value in the discrete phase set There are 2 F discrete phase values, is the fth quantized phase, and then these differences are multiplied by a predetermined scaling factor η, and the negative value is taken as the softmax input to calculate the corresponding weight distribution. The quantized phase shift value is expressed as where w n,f is the unquantized phase shift value For discrete phase sets The weight of the f-th quantized phase in .

8. A RIS-assisted multi-user communication beamforming optimization system based on KAN-SAC, characterized in that: include: The problem modeling module is used to build a RIS-assisted MU-MISO system model and establish an optimization problem with the base station's active beamforming and the RIS's passive beamforming as optimization variables and the goal of maximizing the downlink system rate; And construct an aperiodic Markov decision. The state space includes the decoupled pilot signal, the active beamforming matrix of the base station in the previous time step, and the passive beamforming vector of the RIS. The action space includes the real and imaginary parts of the active beamforming matrix of the base station and the passive beamforming vector of the RIS. The reward function is defined as the total downlink rate of the system at each time step minus the average reward of the previous time step. The network construction and training module is used to design and train a deep reinforcement learning network architecture based on KAN-SAC. The policy network includes two layers of KAN modules, a mean output layer, and a logarithmic standard output layer. The input dimension of the first layer of KAN module is the dimension of the state space, and the output is sent to the second layer of KAN module. The second layer of KAN module outputs to the mean output layer and the logarithmic standard output layer. The input and output of the two layers of KAN modules are both mapped using a nonlinear mapping that combines K-nearest neighbor sample selection and learnable spline fitting. The mean output layer is based on a fully connected layer, and the output dimension is the action space dimension. The structure of the logarithmic standard output layer is the same as that of the mean output layer, with an additional truncation operation to limit the output of the logarithmic standard to between the preset minimum and maximum values. The mean output layer and the logarithmic standard deviation output layer are used to generate the mean and logarithmic standard deviation of the Gaussian action distribution, respectively. A softmax-weighted quantization method is used to map the RIS passive phase shift output by the policy network to a discrete set, achieving approximate discretization of the phase shift while maintaining differentiability. The beamforming optimization module is used to obtain the optimal active beamforming and RIS phase shift based on the transmitted uplink pilot according to the optimized deep reinforcement learning model.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the KAN-SAC-based RIS-assisted multi-user communication beamforming optimization method are implemented.

Images