Rapid user pairing method based on deep reinforcement learning in large-scale uplink access network

Abstract

The invention provides a fast user pairing method based on deep reinforcement learning in a large-scale uplink access network. The fast user pairing method comprises the following steps: step 1, modeling for an NOMA and OMA hybrid system; step 2, performing user pairing; step 3, determining a user access mode; 4, determining the transmitting power of the users, and calculating the reachable rates of the two users; step 5, the step 2 to the step 4 are cycled until all users complete pairing, access mode selection and power distribution; 6, calculating the total reachable communication rate of the system, and updating the weight of the neural network; after all the users complete pairing and power distribution, adding the user rates to obtain the total reachable communication rate of all the users; and 7, circulating the step 2 to the step 6 until the maximum execution step number is reached or the neural network is converged. According to the method, the access mechanism combining NOMA and OMA is determined, and the user pairing mode, the access mode and the transmitting power are optimized, so that the total reachable rate of the system is maximum.

Description

Technical Field

[0001] The present application relates to the field of wireless communication technology, and in particular to a fast user pairing method based on deep reinforcement learning in a large-scale uplink access network. Background Art

[0002] The sixth generation mobile communication network (6G) is a key driving technology for realizing the intelligent information society. Its user experience communication rate and connection density will reach 10 times that of the fifth generation mobile communication network (5G). To achieve such a high communication rate and connection density, a multiple access method different from 5G is required. Non-orthogonal multiple access (NOMA) uses power domain multiplexing to allow users to send signals simultaneously on the same frequency. It is considered to be one of the key technologies to solve the high connection density in 6G networks. Although NOMA allows users to send signals simultaneously on the same frequency, the receiving end needs to use serial interference cancellation (SIC) technology to demodulate the signal. When the number of users sending signals simultaneously increases, the complexity and delay of SIC technology will increase sharply. Therefore, in a large-scale user access network, a solution that balances effectiveness and complexity is to pair multiple users in pairs, and each pair of users accesses the same channel resource based on NOMA technology. In this large-scale user NOMA network, user pairing strategy and user power allocation scheme are important research issues.

[0003] The pairing schemes currently under study include rule-based head-to-tail pairing, uniform pairing, and other schemes. These rule-based algorithms are fast, but the single rule cannot adapt to changes in other factors, and the algorithm performance is average; the annealing algorithm and particle swarm algorithm based on iterative search have good performance, but the iteration time is too long to adapt to the dynamic changes of user channels; there are still two problems with user pairing in current research. One is that the existing algorithm cannot meet the dynamic changes in the number of users and the dynamic changes in the user channel state (reference: N. Trabelsi, L. Chaari Fourati, and CSChen, "Interference management in 5G and beyond networks: A comprehensive survey," Computer Networks, vol. 239, p. 110159, Feb. 2024,). In addition, existing studies rarely consider the situation where SIC technology cannot perfectly eliminate direct interference from users, and in actual scenarios, SIC is usually imperfect due to the presence of interference or noise. In this case, there is little research on how to choose between NOMA access and traditional orthogonal multiple access (OMA). Therefore, it is necessary to study a selection strategy for how to select the best access method under imperfect SIC conditions; as well as a user pairing and power allocation method for NOMA access that can meet the changes in the number of users and user channel states. Summary of the invention

[0004] The present application provides a fast user pairing method based on deep reinforcement learning in a large-scale uplink access network, which can be used to solve the technical problems of user pairing and power allocation that the existing technology cannot meet with the changes in the number of users and user channel states.

[0005] The present application provides a fast user pairing method based on deep reinforcement learning in a large-scale uplink access network, characterized in that the method includes:

[0006] Step 1: Model the hybrid system of non-orthogonal multiple access (NOMA) and traditional orthogonal multiple access (OMA), and construct an optimization problem;

[0007] Step 2: Perform user pairing;

[0008] Step 3: Determine whether the user's access mode is NOMA or OMA;

[0009] Step 4: determine the user transmission power and calculate the achievable rates of two users in the same pair respectively;

[0010] Step 5: loop step 2 to step 4 until all users complete pairing, access mode selection and power allocation;

[0011] Step 6, calculate the total achievable communication rate of the system, and use the trajectory reference mechanism and proximal gradient optimization algorithm to update the neural network weights;

[0012] When all users have completed pairing and power allocation, the user rates saved in step 4 are added together to obtain the total achievable communication rate R of all users. sum ;

[0013] Step 7: loop steps 2 to 6 until the maximum number of execution steps is reached or the neural network converges.

[0014] Furthermore, step 1 is to model a hybrid system of non-orthogonal multiple access NOMA and traditional orthogonal multiple access OMA, and construct an optimization problem; including:

[0015] In the NOMA uplink access network, there are N users and a central base station. The user set is represented as u∈U={1,2,...,N}. Users use OMA or NOMA to access the base station for communication, and the user channel gain can be obtained at the base station. When the user uses OMA access, the user signal-to-noise ratio SNR is as follows:

[0016]

[0017] Where P is the maximum transmission power of the user, h u is the user channel gain satisfying Rayleigh fading, σ 2 is the Gaussian white noise power; the achievable communication rate of the OMA user is expressed as:

[0018] R u =0.5Blog2(1+γ u ) (18)

[0019] Where 0.5 is the OMA multiplexing loss, and B is the channel bandwidth;

[0020] In the case of two-user NOMA, N users are divided into N / 2 user pairs, where i∈{0,1,...,N / 2} represents the i-th user pair, and the two users of each user pair are s i ,w i ∈U, the channel gain of the user with larger channel gain is The channel gain of the user with smaller channel gain is Right now The NOMA access signal to interference plus noise ratio (SINR) of the strong user and the weak user in the i-th user pair are respectively expressed as:

[0021]

[0022] in is the SINR of strong users, is the SINR of the weak user, α∈[0,1] represents the user's transmit power allocation ratio, and β∈[0,1] represents the imperfect coefficient; according to formula (1) and formula (3), we get:

[0023]

[0024] in, Respectively represent The signal-to-noise ratio of the corresponding user's OMA access method; therefore, the achievable communication rate of two NOMA users is expressed as:

[0025]

[0026] For each user pair, the achievable communication rate of the NOMA method is expressed as The achievable communication rate of the OMA method is expressed as are the achievable rates of strong and weak users in the i-th user pair using OMA access respectively; define the vector y = [y1, y2..., y N / 2 ], when y i = 0, indicating that the user uses OMA transmission mode for i; when y i =1 indicates that user i uses NOMA transmission mode; at this time, the total achievable communication rate of N users is expressed as

[0027]

[0028] The optimization problem is constructed as follows:

[0029]

[0030] In P0, by optimizing the user's transmission power allocation factor α and user access mode y i and user pairing strategy π to maximize the total achievable communication rate of the system, where (a) represents the constraint of the NOMA user power allocation factor, (b) represents the user's NOMA and OMA access methods, and (c) indicates that the strong user channel gain must be greater than that of the weak user.

[0031] Furthermore, step 2, performing user pairing, includes:

[0032] Step 21, obtain the channel gain of all users through channel estimation, expressed as Γ={γ1,..,γ n ,.,γ N};

[0033] Step 22: sort the current channel gains in ascending order, determine the environment state, and define the mask t is a 1×N vector, mask t,n =0 means the nth user has been paired, mask t,n =1 means the nth user is not paired; the value range of t is t∈{1,...,N / 2}. t ={γ1, .., γ n ,...,γ N ,γ t ,mask t} represents the current state, including the channel gains of all users and the channel gains of the users that currently need to be paired; γ t The value is the unpaired user with the smallest current channel gain; therefore, we can know that the state space size is 2N+1, and a complete pairing process contains N / 2 states.

[0034] Step 23, select and γ through the improved pointer network t Paired users: Deep reinforcement learning is used to pair users, and an improved pointer network is used to adapt to the dynamic changes in the number of users;

[0035] Both the encoder and decoder use convolutional neural networks; the encoder input is the channel gain sequence between all users, and the decoder input is the channel gain and mask of the user currently needing to be paired t The output of the encoder and decoder is used for attention calculation to obtain the paired user corresponding to the current user. The attention calculation method is as follows:

[0036] u=V T tanh(W ref ·ref+W q ·q) (24)

[0037] Among them, u is a vector of size 1×N, W ref ,W q The matrix of size d×d represents the fully connected layer parameters, V represents the learnable vector parameter is a vector of size 1×N, d is the number of hidden units in the CNN network; ref is the output of the encoder; in order to prevent the user from being selected multiple times, the mask is used to correct the relevance u:

[0038]

[0039] The probability distribution of paired users is:

[0040] p(ref,q;W ref ,W q ,V)=Softmax(u) (26)

[0041] p is the probability distribution of N users, lnp θ (π|θ) is the logarithmic probability distribution. Each pairing is sampled with probability p to select users. After sampling, S is saved. t ,lnp θ (π|θ) and the selected user into the buffer.

[0042] Further, step 3, determine the user access mode y i :include:

[0043] NOMA pairing is performed only when the transmission rate of two users using NOMA is not less than the rate of the corresponding user using OMA. That is, the condition for NOMA access is The remaining cases make the achievable communication rate of some users lower than that of OMA access, so OMA access is still adopted in the remaining cases;

[0044] When users adopt NOMA, they need to meet According to formula (4) and formula (5), we have

[0045]

[0046] Then get

[0047]

[0048] According to formula (12), we can get:

[0049]

[0050] Formula (29) shows that when the imperfection coefficient is less than the imperfection factor β that the user can accept L When satisfied There must be α * ∈[α L ,α U ] makes the transmission rates of the two NOMA users no less than the rates of the corresponding users using the OMA method, so NOMA access is adopted; when β<β L NOMA access is used when i =1, otherwise OMA access y is adopted i =0.

[0051] Further, step 4, determining the user transmission power and calculating the achievable rate of the two users; including:

[0052] When OMA access is used, since the signals sent by users are orthogonal to each other, the maximum transmission power P is used to increase the achievable communication rate;

[0053] When NOMA access is used, Therefore, the upper bound of the power allocation factor α is adopted U As the actual power allocation factor, the transmission powers of the two users are (1-α U ),α U ;

[0054] At this time, since the paired users, access mode and transmission power have been determined, using formula (6) (1-y i )R i O +y i R i NO (α) Calculate the achievable rates of the two users and save the corresponding rates.

[0055] Further, step 6, calculating the total achievable communication rate of the system and updating the neural network weights; including:

[0056] When all users have completed pairing and power allocation, the user rates saved in step 4 are added together to obtain the total achievable communication rate R of all users. sum ;

[0057] The trajectory benchmark is combined with the proximal gradient optimization algorithm to train the neural network;

[0058] The value function estimation module Critic adopts the self-evaluation method of trajectory benchmark, uses the same module as the Actor, and uses the best parameters of the Actor's historical reward to interact with the environment to evaluate the current Actor's action; at this time, the advantage function is:

[0059]

[0060] in To calculate the total achievable communication rate using Actors performing steps 2 to 5, To calculate the total achievable communication rate of using Critic to execute steps 2 to 5, if adv < 0, it means that the current Actor's performance is not as good as Critic, and Actor should learn from Critic. On the contrary, if adv > 0, it means that the Actor's learning direction is correct, and learning in this direction should be strengthened.

[0061] The loss function of the Actor network is expressed as:

[0062]

[0063] in

[0064]

[0065] Indicates the similarity between two parameters, pθ is the probability distribution of the neural network parameter θ when saving data, p θnow is the current neural network parameter θ now The probability distribution obtained under the condition; entropy is the probability distribution p θnow The entropy of J is ξ, the entropy coefficient is 0.01; θ Then use Adam and other optimizers to update the Actor parameters; when the mean of adv is greater than zero, assign the Actor parameters to the Critic

[0066] (1) The present invention comprehensively considers the impact of imperfect serial interference elimination and proposes a hybrid access mechanism based on non-orthogonal multiple access and orthogonal multiple access to ensure the achievable communication rate for users;

[0067] (2) The present invention changes the architecture of the pointer network, uses the method of environmental state to memorize paired users, and replaces the original LTSM network with CNN, thereby improving the network operation speed;

[0068] (3) For the training algorithm of reinforcement learning, a method combining trajectory benchmark and proximal gradient optimization importance sampling is used to make the neural network converge faster;

[0069] (4) Compared with other user pairing algorithms, the method proposed in the present invention has a faster response speed and the ability to adapt to dynamic changes in the number of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A system model diagram provided for an embodiment of the present application;

[0071] Figure 2 A network diagram provided for an embodiment of the present application;

[0072] Figure 3 A network model diagram of a pairing algorithm provided in an embodiment of the present application;

[0073] Figure 4 A pairing flow chart provided for an embodiment of the present application;

[0074] Figure 5 A graph showing the relationship between the number of users and the total achievable rate provided in the embodiment of the present application; wherein (a) a comparison graph of the total achievable rates of different schemes; (b) a comparison graph of the total achievable rates under multiple users;

[0075] Figure 6 A time comparison chart provided for an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0077] Step 1: Model the hybrid system of non-orthogonal multiple access (NOMA) and traditional orthogonal multiple access (OMA), and construct an optimization problem;

[0078] In the NOMA uplink access network, there are N users and a central base station. The user set is represented by u∈U={1,2,...,N}. The user uses OMA or NOMA to access the base station for communication, and the user channel gain can be obtained at the base station. When the user uses OMA access, the user signal-to-noise ratio (SNR) is as follows

[0079]

[0080] Where P is the maximum transmission power of the user, h u is the user channel gain satisfying Rayleigh fading, σ 2 is the Gaussian white noise power; the achievable communication rate of the OMA user is expressed as:

[0081] R u =0.5Blog2(1+γ u ) (34)

[0082] Where 0.5 is the OMA multiplexing loss, and B is the channel bandwidth;

[0083] In the case of two-user NOMA, N users are divided into N / 2 user pairs, where i∈{0,1,...,N / 2} represents the i-th user pair, and the two users of each user pair are s i ,w i ∈U, the channel gain of the user with larger channel gain is The channel gain of the user with smaller channel gain is Right now The NOMA access signal to interference plus noise ratio (SINR) of the strong user and the weak user in the i-th user pair are respectively expressed as:

[0084]

[0085] in is the SINR of strong users, is the SINR of the weak user, α∈[0,1] represents the user's transmit power allocation ratio, and β∈[0,1] represents the imperfect coefficient; According to formula (33) and formula (35), we get:

[0086]

[0087] in, Respectively represent The signal-to-noise ratio of the corresponding user's OMA access method; therefore, the achievable communication rate of two NOMA users is expressed as:

[0088]

[0089] For each user pair, the achievable communication rate of the NOMA method is expressed as The achievable communication rate of the OMA method is expressed as are the achievable rates of strong and weak users in the i-th user pair using OMA access respectively; define the vector y = [y1, y2..., y N / 2 ], when y i = 0, indicating that the user uses OMA transmission mode for i; when y i =1 indicates that user i uses NOMA transmission mode; at this time, the total achievable communication rate of N users is expressed as

[0090]

[0091] According to formula (38), the total achievable communication rate of the system mainly depends on the choice of OMA or NOMA, that is, y i The value of and the corresponding R i O ,R i N The size of is closely related to the channel gain and power allocation of NOMA users, so the optimization problem is constructed as follows:

[0092]

[0093] In P0, by optimizing the user's transmission power allocation factor α and user access mode y i and user pairing strategy π to maximize the total achievable communication rate of the system, where (a) represents the constraint of the NOMA user power allocation factor, (b) represents the user's NOMA and OMA access methods, and (c) indicates that the strong user channel gain must be greater than that of the weak user.

[0094] Step 2: Perform user pairing:

[0095] The steps involved are as follows:

[0096] Step 21, obtain the channel gain of all users through channel estimation, expressed as Γ={γ1,..,γ n ,.,γ N};

[0097] Step 22: sort the current channel gains in ascending order, determine the environment state, and define the maskt is a 1×N vector, mask t,n =0 means the nth user has been paired, mask t,n =1 means the nth user is not paired; the value range of t is t∈{1,...,N / 2}. t ={γ1, .., γ n ,...,γ N ,γ t ,mask t} represents the current state, including the channel gains of all users and the channel gains of the users that currently need to be paired; γ t The value is the unpaired user with the smallest current channel gain; therefore, we can know that the state space size is 2N+1, and a complete pairing process contains N / 2 states.

[0098] Step 23, select and γ through the improved pointer network t Paired users: Deep reinforcement learning is used to pair users, and an improved pointer network is used to adapt to the dynamic changes in the number of users;

[0099] Traditional pointer networks use long short-term memory networks (LSTM) in both encoders and decoders. LSTM is used to memorize the relationship between users, but LSTM is slow and takes a certain amount of time to converge. Figure 2 It is shown that both the encoder and decoder of the present invention adopt convolutional neural network (CNN); the network input includes not only the channel gain between users, but also the relationship between the current paired user and the paired users (i.e., the attached Figure 2 The decoder input in ; By inputting the determined user relationship, the use of LSTM to learn the relationship between the selected users is avoided, so that the network convergence speed can be improved to a certain extent. Similar to the traditional pointer network, the attention mechanism is used to enable the pointer network to pair the corresponding users; the attention calculation method is as follows:

[0100] u=V T tanh(W ref ·ref+W q ·q) (40)

[0101] Among them, u is a vector of size 1×N, W ref ,W q is a d×d matrix, representing the fully connected layer parameters, V represents the learnable vector parameter is a 1×N vector, d is the number of hidden units in the CNN network; ref is the output of the encoder; in order to prevent the user from being selected multiple times, the mask is used to correct the relevance u:

[0102]

[0103] The probability distribution of paired users is:

[0104] p(ref,q;W ref ,W q ,V)=Softmax(u) (42)

[0105] p is the probability distribution of N users, lnp θ (π|θ) is the logarithmic probability distribution. Each pairing is sampled with probability p to select users. After sampling, S is saved. t ,lnp θ (π|θ) and the selected user into the buffer.

[0106] Step 3: Determine the user access method i :

[0107] NOMA pairing is performed only when the transmission rate of two users using NOMA is not less than the rate of the corresponding user using OMA. That is, the condition for NOMA access is The remaining cases make the achievable communication rate of some users lower than that of OMA access, so OMA access is still adopted in the remaining cases;

[0108] When users adopt NOMA, they need to meet According to formula (36) and formula (37), we have

[0109]

[0110] Then get

[0111]

[0112] According to formula (44), we can get:

[0113]

[0114] Formula (45) shows that when the imperfection coefficient is less than the imperfection factor β that the user can accept L When satisfied There must be α * ∈[α L ,α U ] makes the transmission rates of the two NOMA users no less than the rates of the corresponding users using the OMA method, so NOMA access is adopted; when β<β L NOMA access is used (y i =1), otherwise OMA access is adopted (y i =0).

[0115] Step 4: Determine the user transmission power and calculate the achievable rate of the two users;

[0116] When OMA access is used, since the signals sent by users are orthogonal to each other, the maximum transmission power P is used to increase the achievable communication rate;

[0117] When NOMA access is used, Therefore, the upper bound of the power allocation factor α is adopted U As the actual power allocation factor, the transmission powers of the two users are (1-α U ),α U ;

[0118] At this time, since the paired users, access mode and transmission power have been determined, using formula (38) (1-y i )R i O +y i R i NO (α) Calculate the achievable rates of the two users and save the corresponding rates.

[0119] Step 5: loop step 2 to step 4 until all users complete pairing, access mode selection and power allocation;

[0120] Step 6, calculate the total achievable communication rate of the system according to formula (38) and update the neural network weights;

[0121] When all users have completed pairing and power allocation, the user rates saved in step 4 are added together to obtain the total achievable communication rate R of all users. sum .

[0122] As attached Figure 3 , the present invention combines the trajectory baseline (RolloutBaseline) with the proximal gradient optimization (PPO) algorithm to train the neural network; the PPO algorithm includes the speaker Actor and the critic Critic module, and uses importance sampling to achieve the purpose of multiple training of the same batch of data. Different from the PPO algorithm, the value function estimation module Critic of the present invention adopts the self-evaluation method of RolloutBaseline, uses the same module as the Actor, and uses the best parameters of the Actor's historical reward to interact with the environment to evaluate the current Actor's action; at this time, the advantage function is:

[0123]

[0124] in To calculate the total achievable communication rate using Actors performing steps 2 to 5, To calculate the total achievable communication rate of using Critic to execute steps 2 to 5, if adv < 0, it means that the current Actor's performance is not as good as Critic, and Actor should learn from Critic. On the contrary, if adv > 0, it means that the Actor's learning direction is correct, and learning in this direction should be strengthened.

[0125] The loss function of the Actor network is expressed as:

[0126]

[0127] in

[0128]

[0129] Indicates the similarity between two parameters, p θ is the probability distribution of the neural network parameter θ when saving data, p θnow is the current neural network parameter θ now The probability distribution obtained under the condition; entropy is the probability distribution p θnow The entropy of J is ξ, which is usually taken as 0.01. θ Then, optimizers such as Adam are used to update the Actor parameters; when the mean of adv is greater than zero, the Actor parameters are assigned to Critic.

[0130] Step 7: loop steps 2 to 6 until the maximum number of execution steps is reached or the neural network converges.

[0131] The present application is further described below through specific embodiments.

[0132] Embodiment 1: Figure 4 The pairing process after obtaining the user channel gain in the case of 6 users is given. The specific pairing network construction is as follows Figure 3 Assume that the SNR of the users in descending order is [7.02, 9.03, 14.00, 15.19, 21.35, 21.80], select the first pair of paired users [7.02, 21.35] according to step 2 above, and calculate β according to step 3 L1 =0.0274, assuming β = 0.001, which satisfies β < β L Therefore, the two users adopt NOMA access mode y1 = 1, and then calculate the transmission power of the two users according to step 4 (1-α U ),α U According to step 6, the pairing mode, access mode and transmission power of the remaining users are obtained in a loop. In this example, according to the method proposed by the present invention, the third pair of users β L1 =0.003.

[0133] Example 2: Assume that in an actual communication scenario, N = 20 users are randomly distributed in a circular area with a radius of 80m, the maximum user transmission power P = 10dBm, and the channel fading satisfies the Rayleigh fading channel model. Noise power σ 2 =-100dBm, the channel path loss exponent is 4, the loss is fixed to -20dB when the user is too close to the base station, the user bandwidth is B = 1, and the imperfection coefficient β = -30dB. In the training phase, the data batch size is 256 and the initial learning rate is 1e -3 , the learning rate decay rate is 0.95. In the training phase, users are selected according to the probability distribution sampling, the number of test samples is 100, and the greedy method is used to select users in the test phase.

[0134] Execute the above steps to obtain the specific pairing strategy, access mode and transmission power.

[0135] Figure 5 (a) The effects of different user pairing schemes on the total achievable rate as the number of users changes are compared, indicating that the proposed method can adapt to the dynamic changes in the number of users and has good performance in terms of the total achievable rate. For all schemes, the total achievable rate increases linearly with the increase in the number of users. It can be seen that the total achievable rate of the scheme using all OMA is the lowest, which shows that even under imperfect SIC conditions, the NOMA access method can still obtain certain gains. As the number of users changes, the scheme proposed in this paper is significantly better than the head-to-tail pairing; the DQN algorithm can only pair within the group. When the number of users is large, the difference in user channels is small and the algorithm performance is low. It is worth noting that the fully connected network structure used by DQN cannot adapt to the situation of different users. In order to compare the performance, this paper trained multiple DQNs for comparison, and the proposed scheme test used the model obtained from the same training. Even if the number of users exceeds the number of users used in training, the proposed scheme can still obtain good performance, indicating that the proposed method can adapt to the changes in the number of different users and has good generalization ability. The annealing algorithm and the scheme proposed in this paper are very close to the exhaustive search algorithm, and the performance of the scheme proposed in this paper is slightly better than the annealing algorithm. Since the exhaustive search algorithm takes an unacceptable amount of time when the number of users is large, and the performance of the annealing algorithm is close to that of the exhaustive search, the annealing algorithm is used as the benchmark algorithm for comparison below. Figure 5 (b) Increase the number of users to compare the change trend of the total achievable rate of different algorithms. This is very important in practical use, because usually only a limited number of users can be used to train the model, but the number of users in practical use is often not fixed. In addition, from Figure 5 (b) It can be seen that as the number of users increases, the improvement of the proposed scheme compared with head-to-tail pairing and DQN also increases.

[0136] Figure 6A comparison of the time taken by the algorithm proposed in this paper and the annealing algorithm when the number of users increases is given. The optimal algorithm requires an exhaustive search of n! possible permutations, which takes too long, so this paper does not compare the running time of the optimal algorithm. The head-to-tail pairing algorithm takes the shortest time due to its simple pairing method, but its total achievable rate is the worst. Although the annealing algorithm can also obtain an approximate optimal solution, the annealing algorithm requires multiple iterations, which may cause the problem of excessive communication delay. In other words, the annealing algorithm cannot meet the dynamic changes of the channel state. The algorithm proposed in this invention is much shorter than the annealing algorithm in terms of time, and as the number of users increases, the algorithm time consumption increases much faster than the annealing algorithm. In addition, the pointer network is similar to the DQN in terms of time, but the DQN algorithm needs to train multiple models to target different numbers of users, which makes it difficult to implement in actual deployment. The proposed network takes less time than DQN and the pointer network, and its operating efficiency is improved by about 40% compared to the pointer network.

[0137] The present invention proposes a fast user pairing algorithm based on deep reinforcement learning in a large-scale uplink access network. Considering the impact of imperfect SIC in actual NOMA communication scenarios, an access mechanism combining NOMA and OMA is designed to optimize the user pairing method, access method and transmission power to maximize the total achievable rate of the system. The pairing method proposed in the present invention can take into account the dynamic changes in the number of users and has good real-time performance and can adapt to the dynamic changes in channel status. The method proposed in the present invention is superior to other NOMA pairing and power optimization schemes.

[0138] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A fast user pairing method based on deep reinforcement learning in a large-scale uplink access network, characterized in that: The method comprises: Step 1: Model the hybrid system of non-orthogonal multiple access (NOMA) and traditional orthogonal multiple access (OMA), and construct an optimization problem; Step 2: Perform user pairing; Step 3: Determine whether the user's access mode is NOMA or OMA; Step 4: determine the user transmission power and calculate the achievable rates of two users in the same pair respectively; Step 5: loop step 2 to step 4 until all users complete pairing, access mode selection and power allocation; Step 6, calculate the total achievable communication rate of the system, and use the trajectory reference mechanism and proximal gradient optimization algorithm to update the neural network weights; When all users have completed pairing and power allocation, the user rates saved in step 4 are added together to obtain the total achievable communication rate R of all users. sum ; Step 7: loop steps 2 to 6 until the maximum number of execution steps is reached or the neural network converges.

2. The method according to claim 1, characterized in that Step 1: Model the hybrid system of non-orthogonal multiple access (NOMA) and traditional orthogonal multiple access (OMA), and construct an optimization problem; including: In the NOMA uplink access network, there are N users and a central base station. The user set is represented as u∈U={1,2,...,N}. Users use OMA or NOMA to access the base station for communication, and the user channel gain can be obtained at the base station. When the user uses OMA access, the user signal-to-noise ratio SNR is as follows: Where P is the maximum transmission power of the user, h u is the user channel gain satisfying Rayleigh fading, σ 2 is the Gaussian white noise power; the achievable communication rate of the OMA user is expressed as: R u =0.5Blog2(1+γ u ) (2) Where 0.5 is the OMA multiplexing loss, and B is the channel bandwidth; In the case of two-user NOMA, N users are divided into N / 2 user pairs, where i∈{0,1,...,N / 2} represents the i-th user pair, and the two users of each user pair are s i ,w i ∈U, the channel gain of the user with larger channel gain is h si ; The channel gain of the user with smaller channel gain is h wi , that is |h si |>|h wi |; The NOMA access signal to interference plus noise ratio (SINR) of the strong user and the weak user in the i-th user pair are respectively expressed as: in is the SINR of strong users, is the SINR of the weak user, α∈[0,1] represents the user's transmit power allocation ratio, and β∈[0,1] represents the imperfect coefficient; according to formula (1) and formula (3), we get: Among them, γ si ,γ wi Respectively represent The signal-to-noise ratio of the corresponding user's OMA access method; therefore, the achievable communication rate of two NOMA users is expressed as: For each user pair, the achievable communication rate of the NOMA method is expressed as The achievable communication rate of the OMA method is expressed as R i O =R si +R wi , R si ,R wi are the achievable rates of strong and weak users in the i-th user pair using OMA access respectively; define the vector y = [y1, y2..., y N / 2 ], when y i = 0, indicating that the user uses OMA transmission mode for i; when y i =1 indicates that user i uses NOMA transmission mode; at this time, the total achievable communication rate of N users is expressed as The optimization problem is constructed as follows: In P0, by optimizing the user's transmission power allocation factor α and user access mode y i and user pairing strategy π to maximize the total achievable communication rate of the system, where (a) represents the constraint of the NOMA user power allocation factor, (b) represents the user's NOMA and OMA access methods, and (c) indicates that the strong user channel gain must be greater than that of the weak user.

3. The method according to claim 1, characterized in that: Step 2: Perform user pairing, including: Step 21, obtain the channel gain of all users through channel estimation, expressed as Γ={γ1,..,γ n ,.,γ N }; Step 22: sort the current channel gains in ascending order, determine the environment state, and define the mask t is a 1×N vector, mask t,n =0 means the nth user has been paired, mask t,n =1 means the nth user is not paired; the value range of t is t∈{1,...,N / 2}; S t ={γ1, .., γ n ,...,γ N ,γ t ,mask t } represents the current state, including the channel gains of all users and the channel gains of the users that currently need to be paired; γ t The value is the unpaired user with the smallest current channel gain; therefore, the state space size is determined to be 2N+1, and a complete pairing process contains N / 2 states; Step 23, select and γ through the improved pointer network t Paired users: Deep reinforcement learning is used to pair users, and an improved pointer network is used to adapt to the dynamic changes in the number of users; Both the encoder and decoder use convolutional neural networks; the encoder input is the channel gain sequence between all users, and the decoder input is the channel gain and mask of the user currently to be paired t ; Perform attention calculation on the output of the encoder and decoder to obtain the paired user corresponding to the current user; the attention calculation method is as follows: u=V T tanh(W ref ·ref+W q ·q) (8) Among them, u is a vector of size 1×N, W ref ,W q is a d×d matrix, representing the fully connected layer parameters, V represents the learnable vector parameter is a 1×N vector, d is the number of hidden units in the CNN network; ref is the output of the encoder; in order to prevent the user from being selected multiple times, the mask is used to correct the relevance u: The probability distribution of paired users is: p(ref,q;W ref ,W q ,V)=Softmax(u) (10) p is the probability distribution of N users, lnp θ (π|θ) is the logarithmic probability distribution. Each pairing is sampled with probability p to select users. After sampling, S is saved. t ,lnp θ (π|θ) and the selected user into the buffer.

4. The method according to claim 2, characterized in that: Step 3: Determine the user access method i :include: NOMA pairing is performed only when the transmission rate of two users using NOMA is not less than the rate of the corresponding user using OMA. That is, the condition for NOMA access is The remaining cases make the achievable communication rate of some users lower than that of OMA access, so OMA access is still adopted in the remaining cases; When users adopt NOMA, they need to meet According to formula (4) and formula (5), we have Then get According to formula (12), we can get: Formula (13) shows that when the imperfection coefficient is less than the imperfection factor β accepted by the user L When satisfied There must be α * ∈[α L ,α U ] makes the transmission rates of the two NOMA users no less than the rates of the corresponding users using the OMA method, so NOMA access is adopted; when β<β L NOMA access is used when i =1, otherwise OMA access y is adopted i =0.

5. The method according to claim 2, characterized in that: Step 4, determining the user transmission power and calculating the achievable rate of the two users; including: When OMA access is used, since the signals sent by users are orthogonal to each other, the maximum transmission power P is used to increase the achievable communication rate; When NOMA access is used, Therefore, the upper bound of the power allocation factor α is adopted U As the actual power allocation factor, the transmission powers of the two users are (1-α U ),α U ; At this time, since the paired users, access mode and transmission power have been determined, using formula (6) (1-y i )R i O +y i R i NO (α) Calculate the achievable rates of the two users and save the corresponding rates.

6. The method according to claim 2, characterized in that Step 6, calculating the total achievable communication rate of the system and updating the neural network weights; including: When all users have completed pairing and power allocation, the user rates saved in step 4 are added together to obtain the total achievable communication rate R of all users. sum ; The trajectory benchmark is combined with the proximal gradient optimization algorithm to train the neural network; The value function estimation module Critic adopts the self-evaluation method of trajectory benchmark, uses the same module as the Actor, and uses the best parameters of the Actor's historical reward to interact with the environment to evaluate the current Actor's action; at this time, the advantage function is: in To calculate the total achievable communication rate using Actors performing steps 2 to 5, To calculate the total achievable communication rate of using Critic to execute steps 2 to 5, if adv < 0, it means that the current Actor's performance is not as good as Critic, and Actor should learn from Critic. On the contrary, if adv > 0, it means that the Actor's learning direction is correct, and learning in this direction should be strengthened. The loss function of the Actor network is expressed as: in Indicates the similarity between two parameters, p θ is the probability distribution of the neural network parameter θ when saving data, p θnow is the current neural network parameter θ now The probability distribution obtained under the condition; entropy is the probability distribution p θnow The entropy of J is ξ, where ξ is the entropy coefficient and takes the value as 0.01; θ Then, optimizers such as Adam are used to update the Actor parameters; when the mean of adv is greater than zero, the Actor parameters are assigned to Critic.

Images