A RIS-NOMA resource allocation method based on deep reinforcement learning

Abstract

The application relates to the field of wireless communication systems and provides an RIS-NOMA resource allocation method based on deep reinforcement learning, which comprises the following steps: acquiring base station position information, RIS position information, the position information of each user, the total power of the base station, the SIC residual interference coefficient, and the user QoS condition as a state; adopting a distributed optimization strategy, using an RIS phase network, a power distribution network at the base station, and a non-circular coefficient distribution network to output the RIS element phase under each state, the power of the base station to transmit signals to all users, and the non-circular coefficient; constructing feedback according to the minimum transmission rate in the user QoS condition and the sum of the secret rates of all legal users of the system; and jointly optimizing the RIS phase network, the power distribution network, and the non-circular coefficient distribution network, so that the secret rate of the system is effectively improved and the risk of user eavesdropping is reduced under the limitation conditions of different transmission powers and minimum transmission rates.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless communication, in particular to a RIS-NOMA resource allocation method based on deep reinforcement learning. BACKGROUND

[0002] In a wireless communication system, Non-Orthogonal Multiple Access (NOMA) improves spectrum efficiency by allowing multiple users to share the same resource block at different power levels, but due to its inherent broadcast nature, the signals of the sending / receiving legitimate users are more susceptible to passive eavesdropping. To address this security challenge, physical layer security technology takes advantage of the inherent randomness and fading characteristics of the wireless channel to achieve information confidentiality, providing a supplement to resist eavesdropping, and becoming a promising technology.

[0003] NOMA uses non-orthogonal transmission at the sending end, actively introduces interference information, and realizes correct demodulation through a serial interference cancellation (SIC) receiver at the receiving end. However, in actual wireless communication scenarios, due to power, delay and other limitations, the SIC technology used to eliminate interference in NOMA cannot completely eliminate inter-user interference. In this case, introducing physical layer security technology, such as artificial noise, to improve the security performance of the system will further enhance inter-user interference, causing the system performance to degrade.

[0004] Since the advantage of NOMA is more pronounced when there is a difference in channel gain between users, reconfigurable intelligent surfaces (RIS) can intelligently adjust the reflection coefficient by using a series of low-cost, independently controllable reflection elements to enhance the received signal at the receiving end while weakening the signal at the eavesdropping end, thereby making the difference in channel gain between users more pronounced without the need to increase additional transmission power and spectrum resources, achieving the purpose of enhancing system performance. Although RIS can improve the security performance of the NOMA system to some extent, it still cannot eliminate the inter-user interference caused by the suboptimal SIC technology.

[0005] Improper Gaussian Signaling (IGS) relaxes the constraint on pseudo-covariance, increases the degree of freedom of signal design, and is a more general signal model. IGS has a significant advantage in interference management due to its flexible signal design. Studies have shown that compared with Proper Gaussian Signaling (PGS), introducing IGS to solve the interference problem of the NOMA system can improve the throughput of the NOMA system under the condition of imperfect successive cancellation. Although jointly optimizing user power and IGS non-circular coefficient can improve system performance to some extent, traditional resource allocation methods are mostly based on idealized assumptions in specific scenarios, such as fixed number of users, static channel or simplified interference model, etc. When the system is expanded to more complex communication scenarios, such as multi-user and multi-antenna systems, the traditional resource allocation method often has difficulty in maintaining the convergence and computational efficiency of the algorithm, and even cannot be directly applied.

[0006] In order to cope with more complex communication scenarios, the existing optimization method couples the originally independent user power and IGS non-circular coefficient into a new decision variable through a wide linear transformation, thereby simplifying the optimization structure in mathematical modeling. However, this coupling obscures the regulatory effect of IGS non-circular coefficient on user interference, cannot explicitly represent the advantage of IGS in interference management, and may limit system performance. On the other hand, by scaling the objective function or the constraint condition, the originally difficult-to-solve non-convex optimization problem is transformed into an easily solvable convex problem. However, this scaling strategy changes the original structure of the problem, so that the obtained solution only corresponds to a subspace or a relaxed solution of the original problem, and cannot guarantee global optimality, so that the optimal resource allocation strategy cannot be found.

[0007] In summary, although the introduction of RIS and IGS in the NOMA system can improve the security and throughput of the system to some extent, it greatly increases the model complexity of the resource allocation strategy of the wireless communication system, leading to difficulty in solving the optimization strategy and large computational overhead. The existing optimization method uses a wide linear transformation or a model scaling strategy, which obscures the role of IGS in managing user interference, and the obtained resource allocation strategy is not a globally optimal solution, so that the system security performance cannot be effectively improved. SUMMARY

[0008] Therefore, the technical problem to be solved by the present application is to overcome the difficulty in directly jointly optimizing the RIS phase, user power and non-circular coefficient of the RIS-NOMA downlink system in the existing NOMA resource allocation method, so as to effectively improve the system security performance.

[0009] To solve the above technical problems, the application provides a RIS-NOMA resource allocation method based on deep reinforcement learning, comprising: acquiring the base station position information, RIS position information, position information of each user, total power of the base station, SIC residual interference coefficient of the NOMA system and QoS condition of the user at the current moment of the RIS-NOMA system as the current state; wherein each user in the system includes a legal user and an eavesdropper; inputting the current state into the RIS phase network to output the RIS element phase at the current moment; inputting the current state into the power allocation network at the base station to generate the power of the signal transmitted by the base station to each user at the current moment; inputting the current state into the non-circular coefficient allocation network at the base station to generate the non-circular coefficient of the signal transmitted by the base station to each user at the current moment; based on the RIS element phase at the current moment, the power of the signal transmitted by the base station to each user and the non-circular coefficient, the sum of the secrecy rates of all legal users at the current moment is calculated to construct the feedback at the current moment; and the RIS phase network, the power allocation network and the non-circular coefficient allocation network are trained by using the experience replay pool.

[0010] Preferably, the RIS phase network, the power allocation network and the non-circular coefficient allocation network are all constructed based on the SAC algorithm.

[0011] Preferably, based on the RIS element phase at the current moment, the power of the signal transmitted by the base station to each user and the non-circular coefficient, the sum of the secrecy rates of all legal users at the current moment is calculated to construct the feedback at the current moment, comprising:

[0012] According to the RIS element phase at the current moment, the channel state of each user at the current moment is calculated; according to the power of the signal transmitted by the base station to each user and the non-circular coefficient at the current moment, the covariance matrix of the signal transmitted by the base station to each user at the current moment is obtained; according to the channel state of each user at the current moment and the covariance matrix of the signal transmitted by the base station to each user at the current moment, the achievable rate of each legal user at the current moment and the data rate of the eavesdropper decoding the signal of each legal user at the current moment are obtained; based on the achievable rate of each legal user at the current moment and the data rate of the eavesdropper decoding the signal of each legal user at the current moment, the secrecy rate of each legal user at the current moment is calculated; if the achievable rate of all legal users at the current moment is not less than the minimum transmission rate in the QoS condition, and the secrecy rate of all legal users at the current moment is not less than 0, the sum of the secrecy rates of all legal users at the current moment is taken as the feedback at the current moment; if there is a legal user whose achievable rate is less than the minimum transmission rate or a legal user whose secrecy rate is less than zero at the current moment, the feedback at the current moment is 0.

[0013] Preferably, according to the RIS element phase at the current moment, the channel state of each user at the current moment is calculated, and the expression is:

[0014] ;

[0015] in, For the current moment user Channel state, for RIS and its users Channel vectors and matrices between For the current moment RIS element phase The diagonal matrix formed For the current moment Channel matrix between base station and RIS For the current moment Base station to user Channel vector of direct link, For user indexing, 1 represents a strong user among legitimate users, 2 represents a weak user among legitimate users, and 3 represents an eavesdropper;

[0016] Based on the power and non-circular coefficients of the signals transmitted from the base station to each user at the current moment, the covariance matrix of the signals transmitted from the base station to each user at the current moment is obtained, and its expression is:

[0017] ;

[0018] in, Indicates the current time Base station to user The covariance matrix of the transmitted signal, For the current moment user The real vector of the circularly polarized signal. and Each represents the current time. Base station to user The power and non-circularity coefficient of the transmitted signal for and The corresponding geometric transformation matrix, where T represents the transpose.

[0019] Preferably, based on the channel state of each user at the current moment and the covariance matrix of the signals transmitted from the base station to each user, the achievable rate of each legitimate user at the current moment and the data rate at which the eavesdropper decodes each legitimate user are obtained, including:

[0020] According to the current time, the The channel status of the first legitimate user, the base station to the first The covariance matrix of the transmitted signals of the first legal user and the first The interference noise components at each legitimate user location are used to obtain the current time. No. The reachable rate for a number of legitimate users is expressed as:

[0021] ;

[0022] in, For the current moment No. The achievable rate for a legitimate user It is the identity matrix. For the current moment user The real-valued channel matrix, Current moment Base station to user The covariance matrix of the transmitted signal, where T denotes transpose. For the current moment No. Interference noise components at the location of a legitimate user;

[0023] Current moment user Real-valued channel matrix for:

[0024] ,

[0025] in, Indicates the current time user Channel state The real part, Indicates the current time user Channel state The imaginary part;

[0026] Current moment No. Interference noise components at legitimate user locations for:

[0027] ;

[0028] in, For the current moment t SIC residual interference coefficients of user k and user j signals Indicates user Belongs to user set ,and ; For the current moment No. The variance of the noise observed at each legitimate user; and For user indexing, 1 represents a strong user among legitimate users, 2 represents a weak user among legitimate users, and 3 represents an eavesdropper; For the current moment Base station to user The covariance matrix of the transmitted signal;

[0029] According to the current time The eavesdropper's channel state, base station to the first The covariance matrix of the signals transmitted by each legitimate user and the interference noise component at the eavesdropper's location are used to obtain the current time. Eavesdropper Decoding The data rate of a valid user signal is expressed as:

[0030] ;

[0031] in, For the current moment Eavesdropper Decoding The data rate of a legitimate user signal For the current moment The eavesdropper's real channel matrix, For the current moment Eavesdropper Decoding Interference noise components of a legitimate user signal: , For the current moment No. The variance of observed noise at each legitimate user location. For the current moment Base station to user The covariance matrix of the transmitted signal.

[0032] Preferably, the expression for the feedback at the current moment is:

[0033] ;

[0034] in, For the current moment No. The confidentiality rate for each legitimate user: ; Indicates the current time No. The achievable rate for each legitimate user is not less than the minimum transmission rate in the QoS conditions; Indicates the current time No. The achievable rate for a legitimate user is no less than the data transmission rate at which an eavesdropper decodes signals from a legitimate user. ,Right now .

[0035] Preferably, the current state is input into the RIS phase network, and the current RIS element phase is output, including:

[0036] The current state is input into the RIS phase network to generate the mean and variance of the distribution of all RIS element phases. Then, resampling is performed, and the tanh function is used to activate the sampling results. The activated result is multiplied by the range of values ​​of the RIS element phase to obtain the RIS element phase at the current time.

[0037] Preferably, the RIS phase network, power allocation network, and non-circular coefficient allocation network are trained using samples from the empirical replay pool, including:

[0038] Each moment tuple As training samples, they are stored in the experience replay pool. middle;

[0039] In the experience replay pool Random collection Group samples Training the RIS phase network, power allocation network, and non-circular coefficient allocation network includes:

[0040] Experience replay pool of RIS phase network Random collection Group samples Update the parameters of the value network, policy network, first value network, second value network, and target value network of the RIS phase network;

[0041] Experience replay pool of power distribution networks Random collection Group samples Update the parameters of the value network, policy network, first value network, second value network, and target value network of the power allocation network;

[0042] Experience replay pool for non-circular coefficient allocation networks Random collection Group samples Update the parameters of the value network, policy network, first value network, second value network, and target value network of the non-circular coefficient allocation network;

[0043] in, For a moment The state; For a moment The output action of the policy network, For a moment The phase of the RIS element, and They are time points the power and non-circular coefficient of the signal transmitted by the base station to the strong user, and respectively the state at time the power and non-circular coefficient of the signal transmitted by the base station to the strong user, and respectively the state at time the power and non-circular coefficient of the signal transmitted by the base station to the weak user, and respectively the state at time the power and non-circular coefficient of the signal transmitted by the base station to confuse the eavesdropper; is the feedback at time is the state at time is the state at time 、 、 、 、 respectively the current state, action, feedback and next state of the i-th sample in the experience replay pool, 、 、 and respectively the RIS element phase, the power and non-circular coefficient of the signal transmitted by the base station to each user in the i-th sample.

[0044] The above technical solutions of the present application have the following beneficial effects compared with the prior art:

[0045] The RIS-NOMA resource allocation method based on deep reinforcement learning provided by the present application introduces non-circular Gaussian signals in the RIS-NOMA system to weaken the inter-user interference, adopts a distributed deep reinforcement learning strategy, and uses multiple agents to respectively give the allocation scheme of the RIS element phase, the base station power and the non-circular coefficient of the RIS-NOMA downlink system, thereby effectively reducing the output dimension of the network and improving the performance of the resource allocation network. By jointly optimizing the power and IGS non-circular coefficient of the signal transmitted by the base station to each user, rather than integrating the power and non-circular coefficient into one decision variable, the present application more directly reflects the important influence of the IGS non-circular coefficient on improving the system security performance.

[0046] ​The present application calculates the signal state of the legitimate user and the eavesdropper in the system according to the RIS element phase output by the RIS phase network; calculates the achievable rate of the legitimate user in the system and the data transmission rate of the eavesdropper decoding the signal of the legitimate user according to the power and non-circular coefficient of the signal transmitted by the base station to the legitimate user output by the power distribution network and the non-circular coefficient distribution network, and the power and non-circular coefficient of the signal emitted by the base station to confuse the eavesdropper; and jointly optimizes the RIS phase network, the power distribution network and the non-circular coefficient distribution network with the optimization target of maximizing the sum of the secrecy rates of the legitimate users in the whole system while ensuring the minimum rate requirement of the user. The method provided by the present application for jointly optimizing the RIS phase, user power and non-circular coefficient can effectively utilize the reconfigurable intelligent surface, non-circular Gaussian signal and physical layer interference technology to improve the secrecy performance of the system and reduce the risk of the legitimate user of the RIS-NOMA system being eavesdropped under the limitation of different transmission powers and minimum transmission rates. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to make the content of the present application more easily understood, the present application will be further described in detail below according to specific embodiments of the present application and in conjunction with the drawings, in which:

[0048] Figure 1 is a flowchart of a resource allocation method of a RIS-NOMA system based on deep reinforcement learning provided by an embodiment of the present application;

[0049] Figure 2 is a schematic diagram of a RIS-NOMA system provided by an embodiment of the present application;

[0050] Figure 3 is a joint optimization schematic diagram of a RIS-NOMA resource allocation method based on a distributed SAC algorithm provided by an embodiment of the present application;

[0051] Figure 4 is a network parameter update framework schematic diagram of a RIS phase network / power distribution network / non-circular coefficient distribution network provided by an embodiment of the present application;

[0052] Figure 5 is a sum secrecy rate comparison schematic diagram of a joint RIS phase, user power and IGS non-circular coefficient allocation method based on deep reinforcement learning and an alternating iterative optimization algorithm under different base station powers provided by an embodiment of the present application;

[0053] Figure 6 is a sum secrecy rate comparison schematic diagram of a joint RIS phase, user power and IGS non-circular coefficient allocation method based on deep reinforcement learning and an alternating iterative optimization algorithm under different minimum transmission rates provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0055] As an important branch of machine learning, deep reinforcement learning combines traditional reinforcement learning with neural networks in deep learning. It continuously interacts with the environment, collects feedback information from the system, and dynamically adjusts network parameters to generate better decision-making schemes, thereby maximizing system performance. Therefore, deep reinforcement learning no longer requires a mathematical model or prior knowledge of the system, making it more suitable for resource optimization in wireless communication.

[0056] While the SAC (soft Actor-Critic) algorithm, a deep reinforcement learning algorithm, performs well in continuous control tasks, its effectiveness is less than ideal when directly applied to RIS-NOMA systems that incorporate non-circular Gaussian signals. This is because the IGS signal has non-zero pseudo-covariance, and its communication environment is more complex than that of traditional systems. This leads to mismatches in SAC's state modeling, policy optimization, and reward design, resulting in unstable learning of non-circular coefficients, difficulties in policy convergence, or performance degradation.

[0057] This invention provides a resource allocation method for a reconfigurable, super-intelligent surface-assisted non-orthogonal multiple access (NOMA) downlink system based on deep reinforcement learning. It introduces a non-circular Gaussian signal to reduce inter-user interference. Compared to traditional optimization methods, the method provided in this invention is a joint optimization approach based on a distributed SAC strategy, model-free RIS phase, user power, and IGS non-circular coefficients. This approach can more efficiently obtain the optimal resource allocation scheme and improve the security performance of the RIS-NOMA downlink system. Furthermore, it achieves good performance under various transmission power and minimum transmission rate constraints. By jointly optimizing user power and IGS non-circular coefficients, rather than integrating them into a single decision variable, the significant impact of IGS non-circular coefficients on improving system performance is more clearly demonstrated.

[0058] like Figure 1 As shown in the figure, the resource allocation method for a RIS-NOMA system based on deep reinforcement learning provided by this embodiment of the invention specifically includes the following steps:

[0059] Step S1: Obtain the current time Base station location information RIS location information Location information of each user Total power of base stations NOMA system SiC residual interference coefficient and Quality of Service (QoS) conditions of users , as the current state .

[0060] The expression of the current state is as follows: .

[0061] As shown in Figure 2 , in the two-user RIS-NOMA downlink system, the two legitimate users are divided into strong users and weak users according to the distance from the base station to the user, wherein the user close to the base station is the strong user, and the base station allocates smaller power to the strong user; the user far from the base station is the weak user, and the base station allocates larger power to the weak user. Both of the two legitimate users are in the coverage of the RIS, and the strong and weak users are randomly and uniformly distributed on both sides of the RIS, while the eavesdropper is randomly and uniformly distributed in the coverage of the RIS.

[0062] The basic principle of serial interference cancellation (SIC) is to gradually eliminate the interference of the user with the largest signal power, that is, the strong user decodes the signal of the weak user first, then eliminates the interference of the weak user signal from the received signal, and finally decodes the signal itself. However, due to various non-ideal factors, the interference cannot be completely eliminated, and the SIC residual interference coefficient is a quantitative index of the residual interference strength, and its unified representation form is :

[0063]

[0064] wherein is the SIC residual interference coefficient of user k and user j signals at the current time t , is the user index, which is specifically defined as: the index of the strong user is 1, the index of the weak user is 2, and the index of the eavesdropper is 3; the strong, weak and eavesdropper users together form the system user set . When , that is, when the signal power of user is greater than the signal power of user , the residual interference brought by user is times of the received signal .

[0065] Step S2: input the current state into the RIS phase network at the base station, and output the RIS element phase at the current time .

[0066] Input the current state into the RIS phase network constructed based on the SAC algorithm, and the double Q network outputs the RIS element phase , the network parameter of the RIS phase network at the current moment The Q value of all phases of the RIS is generated, and in order to prevent overestimation of the Q value, the smaller Q value in the double Q network is selected as the estimated value. After the RIS phase network generates the mean and variance of the distribution to which all RIS element phases are subjected, resampling is performed, the sampling result is activated by using a tanh function, the activated result is multiplied by the value range of the RIS element phase, and the RIS phase information at the current moment is obtained .

[0067] wherein the expression of the tanh function is , is the pre-activation input of the previous layer; the value range of the RIS element phase is , , represents a set composed of possible actions of all RIS element phases at the current moment.

[0068] Step S3: inputting the current state into the power allocation network at the base station, generating the power of the signal transmitted by the base station to all users at the current moment .

[0069] All users include legitimate users and eavesdroppers, and the legitimate users include strong users and weak users. In the present application, a physical layer security technology is introduced to improve the security performance of the RIS-NOMA system, and therefore the power of the signal transmitted by the base station to all users at the current moment includes the power of the signal transmitted by the base station to the strong users at the current moment , the power of the signal transmitted by the base station to the weak users at the current moment , and the power of the signal used by the base station to confuse the eavesdroppers at the current moment .

[0070] inputting the current state into the power allocation network based on the SAC algorithm, and the double Q network of the power allocation network uses the network parameter of the power allocation network at the current moment to give the power allocation strategy of the two legitimate user signals and the signal used to confuse the eavesdroppers, and in order to prevent overestimation of the Q value, the smaller Q value in the double Q network is selected as the estimated value. After the power allocation network generates the mean and variance of the distribution to which all user powers are subjected, resampling is performed, the sampling result is activated by using a softmax function, and the power of the signal transmitted by the base station to all users is obtained.

[0071] The expression for the softmax function is: , The first element in the pre-activation input vector of the previous layer Each element. , This represents the set of all power allocation schemes that satisfy the total power of the base station at the current moment.

[0072] Since user power is constrained by the total power of the base station, and the softmax function can be used to obtain the proportion of all user power to the total power of the base station, and the sum of the power proportions of all users is 1, the user power obtained by multiplying by the power of the base station will automatically meet the total power constraint of the base station. Therefore, in this embodiment, it is preferable to activate the user power sampling result using the softmax function.

[0073] Step S4: Input the current state into the non-circular coefficient allocation network at the base station to generate the non-circular coefficients of the signals transmitted by the base station to all users in the system at the current moment. ;

[0074] Among them, the current moment Non-circular coefficient of the signal transmitted by the base station to all users Including the current moment Non-circular coefficient of base station signal transmission to strong users Current moment Non-circular coefficient of base station signal transmission to weak users and the current moment Non-coefficient signals used by base stations to confuse eavesdroppers .

[0075] The current state is input into the non-circular coefficient allocation network constructed based on the SAC algorithm. The double-Q network of the non-circular coefficient allocation network uses the network parameters of the non-circular coefficient allocation network at the current time according to the current state. A strategy for allocating the non-circular coefficients of the IGS (Integrated Geometric Spectroscopy) for all user signals (including those used to confuse eavesdroppers) is presented. To prevent overestimation of Q-values, the smaller Q-value in the dual-Q network is selected as the estimate. After the non-circular coefficient allocation network generates the mean and variance of the distribution of the non-circular coefficients of all user signals, resampling is performed, and the sigmoid function is used to activate the sampling results to obtain the non-circular coefficients of the signals transmitted by the base station to all users.

[0076] The sigmoid function is expressed as follows: , This is the pre-activation input for the previous layer; , This represents the set of all non-circular coefficient allocation schemes that satisfy the constraints at the current moment.

[0077] This embodiment uses the sigmoid function because the non-circular coefficient of the user signal has a range of values ​​that are... The sigmoid function can be used to map the non-circular coefficients of all user signals to... It automatically satisfies the constraints.

[0078] Step S5: Based on the current RIS element phase, the power of the signal transmitted by the base station to all users, and the non-circularity coefficient, the base station communicates with users within the system, and constructs the current feedback using the sum of the confidential rates of all legitimate users at the current moment. ;

[0079] The base station obtains the RIS element phase according to the RIS phase network. Power obtained by the power distribution network Non-circular coefficients obtained from the non-circular coefficient allocation network It communicates data with users and returns confidential rates to all legitimate users as feedback.

[0080] S51: Based on the current time RIS element phase The current time is calculated. user Channel state :

[0081] ;

[0082] in, for RIS and its users Channel vectors and matrices between For the current moment The diagonal matrix formed by the phases of the RIS elements. For the current moment Channel matrix between base station and RIS For the current moment Base station to user Channel vector of direct link, Indicates the user's index. Indicates a strong user. Indicates a weak user. It refers to an eavesdropper.

[0083] Large-scale path fading in RIS-NOMA downlink systems for:

[0084] ;

[0085] in, For reference distance path loss at m, transmit antenna gain, link distance, path loss exponent.

[0086] The small-scale fading between the base station and the users is Rayleigh fading, i.e., the channel vector Each element in should follow a complex Gaussian distribution

[0087] The small-scale fading between the base station and the RIS is Rician fading, i.e., the channel matrix should satisfy:

[0088] ;

[0089] where, is the Rician factor, is the line of sight (LoS) component, is the non line of sight (NLoS) component at current time and follows Rayleigh fading; the NLoS component is a random variable, the LoS component is a deterministic quantity:

[0090] ;

[0091] where, denotes the vector generated by denotes the vector generated by denotes the conjugate transpose of ; and denote the angle of arrival and the angle of departure of the link signal between the base station and the RIS at the RIS end at current time, respectively, both of which follow a uniform distribution, , denotes the uniform distribution. S52: According to the current time the power of the signal transmitted by the base station to all users

[0092] and the non-circularity coefficient , the covariance matrix of the signal transmitted by the base station to the users at the current time is obtained by using the geometric shaping method ;

[0093]

[0094] where, ​​denotes the current time The base station transmits signals to users The covariance matrix of the transmitted signals, For the current time Users The real vector of the circular polarization signals of the users, And For the current time The power and non-circular coefficient of the signals transmitted by the base station to users at the base station output by the power allocation network and the non-circular coefficient allocation network, For And The transformation matrix corresponding to the geometric shaping, T denotes the transpose. The transformation matrix of the geometric shaping, including rotation, scaling, and other matrix transformation operations, is used to generate non-circular signals based on power and non-circular coefficients.

[0095] S53: According to the channel state of the current time The covariance matrix of the signals transmitted by the base station to the The interference noise component at the The achievable rate of the current time The The ;

[0096] ;

[0097] where, is the identity matrix, The real-valued channel matrix of the current time Users The covariance matrix of the signals transmitted by the base station to users at the current time T denotes the transpose, The interference noise component at the current time The The denotes a strong user, denotes a weak user. The real-valued channel matrix of the current time

[0098] Users is:

[0099] ,

[0100] where, denotes the current time Users​​ Channel state The real part, Indicates the current time user Channel state The imaginary part.

[0101] Based on the channel state of the legitimate users at the current moment, the covariance matrix of the signal transmitted from the base station to each user, the SIC residual interference coefficient, and the observed noise at the legitimate users, the interference noise components at the legitimate users at the current moment are obtained. No. Interference noise components at legitimate user locations for:

[0102] ;

[0103] in, Indicates user Belongs to user set ,and ; For the current moment No. The variance of observed noise at each legitimate user location. For the current moment Base station to user The covariance matrix of the transmitted signal. This matrix represents the signal transmitted from the base station to other legitimate users, as well as the signal used to confuse eavesdroppers, compared to the first... Gaussian noise at the location of the first legitimate user is used as the first Interference noise components at legitimate users.

[0104] S54: Based on the current time The eavesdropper's channel state, the base station's direction to the second The covariance matrix of the signals transmitted by each legitimate user and the interference noise component at the eavesdropper's location are used to obtain the current time. Eavesdropper Decoding Data rate of a legitimate user signal ;

[0105] ;

[0106] in, For the current moment The eavesdropper's real channel matrix, For the current moment Eavesdropper Decoding Interference noise components of a legitimate user signal.

[0107] Based on the channel state used by the base station to confuse eavesdroppers, the covariance matrix of the signals transmitted from the base station to each user, and the noise observed at the eavesdropper's location, the interference noise components that the eavesdropper uses to decode legitimate user signals are obtained, and the current time... Eavesdropper Decoding Interference noise components of a legitimate user signal The expression is:

[0108] ;

[0109] in, For the current moment The variance of the noise observed at the eavesdropper's location. For the current moment Base station to user The covariance matrix of the transmitted signal, when the eavesdropper decodes the first... When a legitimate user sends a signal, the signal transmitted to other legitimate users, as well as the signal used to confuse the eavesdropper, will be combined with Gaussian noise as interference noise components.

[0110] S55: Based on the reachability of each legitimate user and the data rate at which the eavesdropper decodes the signals of each legitimate user, the confidentiality rate of each legitimate user is calculated, and the sum of the confidentiality rates of all legitimate users in the system is taken as the system's total confidentiality rate.

[0111] Current moment The confidentiality rate expression for a legitimate user for:

[0112] ;

[0113] To ensure the data security of legitimate users, the achievable data rate for legitimate users should be greater than or equal to the data transmission rate at which the eavesdropper decodes the legitimate user's signal. .

[0114] S56: If the reachable rate of all legitimate users is not less than the minimum transmission rate in the QoS conditions, and the confidential rate of all legitimate users is not less than 0, then the sum of the confidential rates of all legitimate users is used as the feedback at the current moment; if there is a legitimate user whose reachable rate is less than the minimum transmission rate or a legitimate user whose confidential rate is less than zero, then the feedback at the current moment is 0, that is:

[0115] .

[0116] S6: Train the RIS phase network, power allocation network, and non-circular coefficient allocation network using the empirical replay pool.

[0117] S61: Get the next time step Base station location information , RIS position information , position information of each user , total power of base station , NOMA system SIC residual interference coefficient , and QoS condition of user , as next state .

[0118] S62: store the tuple of each time point as a training sample in the experience replay pool .

[0119] As shown in Figure 3 , in the RIS-NOMA downlink system introducing IGS, the distributed SAC algorithm is adopted, the experience replay mechanism and the double Q network structure are used for joint optimization of the RIS element phase, the user power and the non-circular coefficient, so as to further improve the sum rate and the secrecy rate of the RIS-NOMA system under the requirement of different minimum transmission rates.

[0120] wherein, is the output of the policy network at time point , is the RIS element phase at time point , and are respectively the power and the non-circular coefficient of the signal transmitted by the base station to each user at time point , and are respectively the power and the non-circular coefficient of the signal transmitted by the base station to the strong user at time point , and are respectively the power and the non-circular coefficient of the signal transmitted by the base station to the weak user at time point , and are respectively the power and the non-circular coefficient of the signal used by the base station to confuse the eavesdropper at time point .

[0121] S63: randomly collect group of samples from the experience replay pool to train the RIS phase network, the power allocation network and the non-circular coefficient allocation network.

[0122] The network parameter update framework of the RIS phase network / power allocation network / non-circular coefficient allocation network provided by the embodiment of the application is shown in Figure 4 .

[0123] ​In the embodiments of the present application, the parameters of the target value network in the RIS phase network, the power distribution network and the non-circular coefficient distribution network are updated in a soft updating manner, that is:

[0124]

[0125] wherein, is the parameter of the target value network, is the value network parameter, is the soft updating parameter. By using the soft updating manner, the target value network parameter can slowly follow the change of the current value network parameter, so as to avoid the influence of target value mutation on the training stability.

[0126] In the RIS phase network experience replay pool , a group of samples are randomly collected , and the parameters of the value network, the policy network, the first value network, the second value network and the target value network of the RIS phase network are updated:

[0127] The loss function of the value network of the RIS phase network is:

[0128] ,

[0129] ;

[0130] The loss function of the policy network of the RIS phase network is:

[0131] ;

[0132] The loss function of the value network of the RIS phase network is:

[0133] ;

[0134] ;

[0135] wherein, , , , are respectively the parameters of the value network, the policy network, the value network and the target value network of the RIS phase network, represents the output of the value network in the state performing the action , , represents the probability of the policy network outputting the action in the state , This indicates that the smaller Q-value in the dual Q-network of the RIS phase network is selected as the estimate to prevent overestimation of the Q-value. Indicates the target value. Indicates the state Next action The reward As a discount factor, This represents the target value network of the RIS phase network for the next state. The estimate, , : At that time, it was the first value network. This is the second value network.

[0136] Experience replay pool of power distribution networks Random collection Group samples Update the parameters of the value network, policy network, first value network, second value network, and target value network of the power allocation network:

[0137] Value network of power distribution network The loss function is:

[0138] ,

[0139] ;

[0140] Policy network of power allocation network The loss function is:

[0141] ;

[0142] Value network of power distribution networks The loss function is:

[0143] ;

[0144] ;

[0145] in, , , , These are the parameters of the value network, policy network, value network, and target value network of the power allocation network, respectively. Representing the value network in state Next action The output, , The policy network of the power allocation network is in state. Down Output Action The probability, This indicates that the smaller Q-value in the dual-Q network of the power distribution network is selected as the estimate. Indicates the target value. Indicates the state Next action The reward As a discount factor, The target value of the power distribution network is represented by the network's next state. The estimate, , : At that time, it was the first value network. This is the second value network.

[0146] Experience replay pool for non-circular coefficient allocation networks Random collection Group samples Update the parameters of the value network, policy network, first value network, second value network, and target value network of the non-circular coefficient assignment network:

[0147] Value network of non-circular coefficient allocation network The loss function is:

[0148] ,

[0149] ;

[0150] Policy network for non-circular coefficient allocation network The loss function is:

[0151] ;

[0152] Value network of non-circular coefficient allocation network The loss function is:

[0153] ;

[0154] ;

[0155] in, , , , These are the parameters of the value network, policy network, value network, and target value network of the non-circular coefficient allocation network, respectively. Representing the value network in state Next action The output, , The policy network of the non-circular coefficient allocation network is in state. Down Output Action the probability of, representing selecting a smaller Q value in a double Q network of a non-circular coefficient allocation network as an estimated value, representing a target value, representing a reward of performing an action in a state , is a discount factor, representing an estimation of a target value network of a non-circular coefficient allocation network for a next state , , : is a first value network when, is a second value network.

[0156] This example considers a RIS-NOMA downlink communication system, jointly optimizes the RIS phase, user power and IGS non-circular coefficient, the main parameters of the simulation scene of this example are shown in Table 1, and the SAC training parameters are shown in Table 2.

[0157] Table 1 Main simulation parameters of RIS-MONA downlink communication system

[0158]

[0159] Table 2 Training parameters of distributed SAC algorithm

[0160]

[0161] Figure 5 The figure from top to bottom is the result of the traditional alternating iteration algorithm in the RIS-NOMA system transmitting IGS, the RIS-NOMA system transmitting PGS, the NOMA system transmitting IGS, the NOMA system transmitting PGS, and the joint optimization method provided by the embodiment of the present application in the RIS-NOMA system transmitting IGS. It can be seen from Figure 5 that the method described in the present application can more effectively utilize the IGS non-circular coefficient and improve the sum and secrecy rate of the system by jointly optimizing the RIS phase, user power and IGS non-circular coefficient on the basis of meeting the minimum rate. Through comparison, it is found that the introduction of IGS can further improve the secrecy performance of the RIS-NOMA system.

[0162] Figure 6is a schematic diagram of the average and secrecy rate comparison of the joint RIS phase, user power and IGS non-circular coefficient allocation method and the alternating iterative optimization method based on deep reinforcement learning provided by the embodiment of the application under different minimum transmission rate constraints. The figure shows the average and secrecy rate of the IGS-SAC method and the alternating iterative optimization algorithm in the PGS, IGS model under the constraint of different residual interference coefficients of the RIS-NOMA system SIC Figure 6 It can be seen that IGS can further eliminate inter-user interference in the RIS-NOMA system, and the method provided by the embodiment of the application jointly optimizes the RIS phase, user power and IGS non-circular coefficient through the distributed SAC algorithm, which can further improve the average and secrecy rate of the RIS-NOMA system under the requirement of different minimum transmission rates.

[0163] In summary, the embodiment of the application provides a RIS-NOMA resource allocation method based on deep reinforcement learning. In the RIS-NOMA system, IGS is introduced to solve the interference problem of the NOMA system. However, the existing resource allocation method uses wide linear variation or scaling strategy for the model, which masks the role of IGS in managing user interference, and the obtained resource allocation strategy is not a global optimal solution. Moreover, since the IGS signal has a non-zero pseudo-covariance, its communication environment is more complex than that of the traditional environment, which leads to the problem of mismatch in state modeling, policy optimization and reward design when using traditional deep reinforcement learning algorithms to allocate resources in the RIS-NOMA downlink system. This may lead to unstable learning of non-circular coefficients, difficulty in policy convergence or performance degradation.

[0164] To solve the above problems, the present application uses a distributed SAC algorithm to jointly optimize the RIS element phase, user power and IGS non-circular coefficient. Multiple agents are used to give the allocation scheme of the RIS element phase, user power and IGS non-circular coefficient, which effectively reduces the output dimension of the network and improves the performance of the allocation network.

[0165] ​And the present application calculates the channel state of all users in the system according to the RIS element phase output by the RIS phase network; calculates the covariance matrix of the signal transmitted by the base station to all users according to the power and non-circular coefficient of the signal (including the signal used to confuse the eavesdropper) output by the power allocation network and the non-circular coefficient allocation network, thereby calculating the achievable rate of each legitimate user in the system and the data rate of the eavesdropper decoding the signal of each legitimate user, and finally obtaining the secrecy rate of each legitimate user; according to the minimum transmission rate in the QoS condition of the user and the sum secrecy rate of the system, a feedback mechanism of joint optimization strategy is constructed, so that the RIS phase network, the power allocation network and the non-circular coefficient allocation network trained based on the experience replay mechanism and the double Q network can generate a more optimized resource allocation strategy, so as to further improve the sum secrecy rate of the RIS-NOMA system while meeting the requirements of different minimum transmission rates.

[0166] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0167] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The means for performing the functions specified in one or more flows and / or blocks.

[0168] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The means for performing the functions specified in one or more flows and / or blocks.

[0169] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide steps for implementing the function specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0170] Obviously, the above embodiments are only examples for clearly illustrating the present application, and are not intended to limit the embodiments. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the embodiments are not required to be exhausted, and the obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A deep reinforcement learning based RIS-NOMA resource allocation method, characterized in that, The method comprises the following steps: Obtain the base station position information, RIS position information, position information of each user, total power of the base station, SIC residual interference coefficient of the NOMA system, and QoS condition of the user at the current moment of the RIS-NOMA system as the current state; wherein, the users in the system include legal users and eavesdroppers; Input the current state into the RIS phase network to output the RIS element phase at the current moment; Input the current state into the power allocation network at the base station to generate the power of the signal transmitted by the base station to each user at the current moment; Input the current state into the non-circular coefficient allocation network at the base station to generate the non-circular coefficient of the signal transmitted by the base station to each user at the current moment; Based on the RIS element phase at the current moment, the power of the signal transmitted by the base station to each user, and the non-circular coefficient, calculate the sum of the secrecy rates of all legal users at the current moment to construct the feedback at the current moment, comprising: According to the RIS element phase at the current moment, calculate the channel state of each user at the current moment; According to the power of the signal transmitted by the base station to each user at the current moment and the non-circular coefficient, obtain the covariance matrix of the signal transmitted by the base station to each user at the current moment; According to the channel state of each user at the current moment and the covariance matrix of the signal transmitted by the base station to each user at the current moment, obtain the achievable rate of each legal user at the current moment and the data rate of the eavesdropper decoding the signal of each legal user at the current moment; Based on the achievable rate of each legal user at the current moment and the data rate of the eavesdropper decoding the signal of each legal user at the current moment, calculate the secrecy rate of each legal user at the current moment; If the achievable rate of all legal users at the current moment is not less than the minimum transmission rate in the QoS condition, and the secrecy rate of all legal users at the current moment is not less than 0, then the sum of the secrecy rates of all legal users at the current moment is taken as the feedback at the current moment; If there is a legal user whose achievable rate is less than the minimum transmission rate or a legal user whose secrecy rate is less than zero at the current moment, the feedback at the current moment is 0; The RIS phase network, the power allocation network, and the non-circular coefficient allocation network are trained by using the experience replay pool.

2. The RIS-NOMA resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, The RIS phase network, the power allocation network, and the non-circular coefficient allocation network are all constructed based on the SAC algorithm.

3. The deep reinforcement learning based RIS-NOMA resource allocation method according to claim 1, wherein, According to the RIS element phase at the current moment, calculate the channel state of each user at the current moment, the expression is: ； in, For the current moment user Channel state, for RIS and its users Channel vectors and matrices between For the current moment RIS element phase The diagonal matrix formed For the current moment Channel matrix between base station and RIS For the current moment Base station to user Channel vector of direct link, For user indexing, 1 represents a strong user among legitimate users, 2 represents a weak user among legitimate users, and 3 represents an eavesdropper; According to the power of the signal transmitted by the base station to each user at the current moment and the non-circular coefficient, obtain the covariance matrix of the signal transmitted by the base station to each user at the current moment, the expression is: ； wherein denotes the current time instant the base station transmits to the user the covariance matrix of the transmitted signals, is the current time instant the user the real vector of the circular polarized signals of the user, and are the power and the non-circularity coefficient of the transmitted signals by the base station to the user at the current time instant the base station transmits to the user the power and the non-circularity coefficient of the transmitted signals by the base station to the user at the current time instant is and the transformation matrix of the corresponding geometric shaping, T denotes the transpose.

4. The deep reinforcement learning based RIS-NOMA resource allocation method according to claim 1, characterized in that, According to the channel state of each user at the current moment and the covariance matrix of the signal transmitted by the base station to each user at the current moment, obtain the achievable rate of each legal user at the current moment and the data rate of the eavesdropper decoding the signal of each legal user at the current moment, comprising: According to the current time, the The channel status of the first legitimate user, the base station to the first The covariance matrix of the transmitted signals of the first legal user and the first The interference noise components at each legitimate user location are used to obtain the current time. No. The reachable rate for a number of legitimate users is expressed as: ； in, For the current moment No. The achievable rate for a legitimate user It is the identity matrix. For the current moment user The real-valued channel matrix, Current moment Base station to user The covariance matrix of the transmitted signal, where T denotes transpose. For the current moment No. Interference noise components at the location of a legitimate user; Current time User Real-valued channel matrix is: ， wherein denotes the current time instant user channel state real part, denotes the current time instant user channel state imaginary part; current time the first interference noise component at the site of the legal user is: ； wherein, is the current time user with the user SIC residual interference coefficient of the signal, represents the user belongs to the user set , and ; is the current time the variance of the observation noise at the th legitimate user; and is the user index, 1 represents a strong user in the legitimate user, 2 represents a weak user in the legitimate user, and 3 represents an eavesdropper; is the current time the covariance matrix of the transmission signal of the base station to the user ; According to the current time the channel state of the eavesdropper, the covariance matrix of the signal transmitted by the base station to the first legitimate user, and the interference noise component at the eavesdropper, the data rate of the signal decoded by the eavesdropper from the first legitimate user at the current time is obtained , and is expressed as ； in, For the current moment Eavesdropper Decoding The data rate of a legitimate user signal For the current moment The eavesdropper's real channel matrix, For the current moment Eavesdropper Decoding Interference noise components of a legitimate user signal: For the current moment No. The variance of observed noise at each legitimate user location. For the current moment Base station to user The covariance matrix of the transmitted signal.

5. The deep reinforcement learning based RIS-NOMA resource allocation method according to claim 1, characterized in that, The expression of the feedback at the current moment is: ； wherein is the current time The secret rate of the i-th legitimate user: ; is the current time The achievable rate of the i-th legitimate user is not less than the minimum transmission rate in the QoS condition; is the current time The achievable rate of the i-th legitimate user is not less than the data transmission rate of the eavesdropper decoding the signal of the legitimate user , i.e. .

6. The deep reinforcement learning based RIS-NOMA resource allocation method according to claim 1, characterized in that, Input the current state into the RIS phase network to output the RIS element phase at the current moment, comprising: The current state is input into the RIS phase network, after generating the mean and variance of the distribution to which all RIS element phases are subject, resampling is performed, the sampling result is activated by using a tanh function, the activated result is multiplied by the value range of the RIS element phase, and the RIS element phase at the current moment is obtained.

7. The deep reinforcement learning based RIS-NOMA resource allocation method according to claim 1, characterized in that, The current state is input into the power allocation network at the base station, and the power of the signal transmitted by the base station to each user at the current moment is generated, including: The current state is input into the power allocation network, after generating the mean and variance of the distribution to which all user powers are subject, resampling is performed, the sampling result is activated by using a softmax function, and the power of the signal transmitted by the base station to each user at the current moment is obtained.

8. The deep reinforcement learning based RIS-NOMA resource allocation method according to claim 1, characterized in that, The current state is input into the non-circular coefficient allocation network at the base station, and the non-circular coefficient of the signal transmitted by the base station to each user at the current moment is generated, including: The current state is input into the non-circular coefficient allocation network, after generating the mean and variance of the distribution to which all user signal non-circular coefficients are subject, resampling is performed, the sampling result is activated by using a sigmoid function, and the non-circular coefficient of the signal transmitted by the base station to each user at the current moment is obtained.

9. The deep reinforcement learning based RIS-NOMA resource allocation method according to claim 1, characterized in that, The RIS phase network, the power allocation network and the non-circular coefficient allocation network are trained by using samples of the experience replay pool, including: The tuple of each time instant is stored as a training sample in the experience replay pool ;​ In the experience replay pool randomly sampling group samples training the RIS phase network, the power allocation network, and the non-circular coefficient allocation network, comprising: In the RIS phase network experience replay pool Randomly collect Group samples , update the parameters of the value network, policy network, first value network, second value network and target value network of the RIS phase network; randomly sampling in the power allocation network experience replay pool group samples updating parameters of the value network, the policy network, the first value network, the second value network, and the target value network of the power allocation network​ Randomly sampling in experience replay pool of non-circular coefficient allocation network Group samples updating parameters of the value network, the policy network, the first value network, the second value network, and the target value network of the non-circular coefficient allocation network​ wherein is the state at time ; is the output action of the policy network at time , is the RIS element phase at time , and are the power and non-circularity coefficient of the signal transmitted by the base station to each user at time , and are the power and non-circularity coefficient of the signal transmitted by the base station to the strong user at time , and are the power and non-circularity coefficient of the signal transmitted by the base station to the weak user at time , and are the power and non-circularity coefficient of the signal used by the base station to confuse the eavesdropper at time ; is the feedback at time , is the state at time ; , , , are the current state, action, feedback and next state of the th sample in the experience replay pool, , and are the RIS element phase, the power and non-circularity coefficient of the signal transmitted by the base station to each user in the th sample.

Images