A beam training and power allocation method and system for near-field NOMA system
Through the resource allocation network of deep reinforcement learning, the beam training and power distribution strategies of the near-field NOMA system are dynamically adjusted, which solves the problems of beam training and power distribution in the system, improves transmission rate and communication quality, and reduces interference and overhead.
Patent Information
- Application Number
- CN202410384122.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-01
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-04-01
AI Technical Summary
The prior art is difficult to effectively solve the problems of beam training and power distribution in near-field NOMA systems at the same time, especially in the high-dimensional state, the convergence performance of the deep Q network is poor, and traditional methods are difficult to deal with the combination of user packets and beam training.
A resource allocation network based on deep reinforcement learning is adopted, and a beam training network and a power distribution network are built through user location information acquisition strategies. The beam training strategy and power distribution strategy are dynamically adjusted using DQN and DDPG algorithms to maximize system rewards to improve transmission rate.
The transmission rate of the near-field communication system is improved, the communication quality requirements of future mobile communication systems are met, the interference between users is reduced, and the beam allocation and resource allocation with low overhead are realized.
Smart Images

Figure CN118282458B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of wireless communication technologies, and in particular to a beam training and power allocation method and system for a near-field NOMA system. Background Art
[0002] In the process of wireless communication, beam training is very important because it can greatly improve the beamforming gain. Current beam training schemes are usually based on predefined codebooks. However, since the predefined codebooks are usually large in scale, traditional beam scanning schemes are no longer applicable, so it is necessary to study more effective beam training methods. At the same time, in previous communication systems, the access methods are usually orthogonal. However, orthogonal access schemes are often not the best solutions. Therefore, NOMA (Non-orthgonal Multiple Access) was proposed to achieve higher system rates and ensure multi-user access. Unlike traditional OMA (Orthgonal Multiple Access), NOMA actively introduces interference information and achieves correct demodulation at the receiving end through continuous interference elimination.
[0003] At present, there are related works on near-field communication systems and NOMA technology. The research content of near-field communication mainly focuses on: channel model design, beamforming design and channel estimation, etc. The resource allocation problem in NOMA system has always been a hot research topic. Since user grouping and beam training in NOMA system is a discrete problem, while power allocation is a continuous problem, and there are many combinations of user grouping and beam training, it is difficult for traditional methods to solve these two problems at the same time. As a model-free method, DRL (Deep Reinforcement Learning) solves the problem of difficulty in establishing mathematical models. This method adopts a trial-and-error mechanism to output and optimize strategies through continuous interaction with the system environment. Among them, DQN (Deep Q Network) and DDPG (Deterministic Policy Gradient) networks are the most commonly used algorithms in DRL. Some work has used DQN and DDPG networks to solve the power allocation problem in far-field NOMA systems. However, there is still little research on beam training and power allocation based on deep reinforcement learning in near-field NOMA systems.
[0004] After searching the literature on the prior art, it was found that Cui et al. published "Channel Estimation for Extremely Large-Scale MIMO: Far-Field or Near-Field?" in IEEE Transactions on Communications, vol. 70, no. 4, pp. 2663-2677, April 2022. The paper designed a near-field codebook. Considering that the near-field channel depends not only on the angle domain characteristics but also on the distance, the article proposed a near-field codebook based on the polarization domain. By sampling in the angle domain and the distance domain at the same time, it can effectively adapt to the characteristics of near-field communication. However, the document did not propose a corresponding beam training method. If the proposed polar domain codebook is used through beam scanning, it will bring unbearable training overhead to the system. Another search found that Zhang et al. published "Energy-efficient resource allocation in uplink NOMA systems with deep reinforcement learning" in "2019 11th International Conference on Wireless Communications and Signal Processing (WCSP), 2019, pp. 1-6". The paper proposed a deep reinforcement learning method to solve the power allocation problem in NOMA wireless communication systems. It uses a wireless resource mapping method and a deep neural network (i.e., a deep Q network) to take certain variables in the system as the input of the neural network, map the power allocation results to the output of the network, and give a certain reward to each result of the network. Through continuous interaction with the system, the network can output the optimal power allocation result. However, the deep Q network used in this paper will lead to poor network convergence performance under high-dimensional state input. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a beam training and power allocation method and system for a near-field NOMA system, construct a DRL-based resource allocation network, obtain corresponding strategies and rewards according to user location information, dynamically adjust the beam training strategy and power allocation strategy through rewards, and complete beam training and power allocation. The present invention improves the transmission rate of the entire communication system and meets the requirements of future mobile near-field communication systems for communication quality.
[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0007] A beam training and power allocation method for a near-field NOMA system comprises the following steps:
[0008] S1. Acquire user location information and design codebook for near-field NOMA system.
[0009] S2. Build a resource allocation network based on DRL (Deep reinforcement learning) to generate the beam training strategy and power allocation strategy for the tth time slot according to the input user location information, and obtain the corresponding reward.
[0010] S3. Use the experience tuple to train the DRL-based resource allocation network to obtain a trained DRL-based resource allocation network; the experience tuple includes the user location information at the current moment and the next moment, the beam training strategy and power allocation strategy generated in step S2, and the reward obtained by the system executing the corresponding strategy.
[0011] S4. Input the location information of all users in the current time slot into the trained DRL-based resource allocation network. By maximizing the rewards obtained by the system, dynamically adjust the beam training strategy and power allocation strategy, obtain the strategy corresponding to the maximum and rate of the system, and complete the beam training and power allocation.
[0012] Furthermore, in step S1, user location information acquisition and codebook design include the following contents:
[0013] Assume that all M users of the service are within the near field range. The user's location information, including elevation angle and distance, is obtained through positioning. The near field downlink channel is expressed as:
[0014]
[0015] The transmitting antenna array is {1, 2, …, N}, where N represents the total number of transmitting antennas; L represents the total number of paths between the transmitting end and the user; H(t) represents the near-field spherical wave channel vector; g l (t) represents the channel gain of the lth path in the tth time slot; j represents the imaginary unit; λ c Represents the wavelength of electromagnetic waves; r l (t) represents the distance of the lth path in the tth time slot; θ l (t) represents the angle of the lth path in the tth time slot; b(θ l (t),r l (t)) represents the steering vector of the lth path in the tth time slot.
[0016] The codebook design for the near-field NOMA system is as follows:
[0017]
[0018] in, represents the set of codewords, θ 1 Indicates the first angle sampling value, r 1 1 represents the first distance sampling value on the first distance sampling ring, θ N Represents the Nth angle sampling value, r 1 N Indicates the Nth distance sampling value on the first distance sampling ring, Represents the first distance sampling value on the Sth distance sampling ring, represents the Nth distance sampling value on the Sth distance sampling ring, b(θ 1 ,r 1 1 ) represents the steering vector corresponding to the position determined by the first angle sampling and the first distance sampling ring, b(θ N ,r 1 N ) represents the steering vector corresponding to the position determined by the Nth angle sampling and the first distance sampling ring, Indicates the steering vector corresponding to the position determined by the first angle sampling and the Sth distance sampling ring, Represents the steering vector corresponding to the position determined by the Nth angle sampling and the Sth distance sampling ring; the steering vector is the codeword.
[0019] There are N×S codewords in the codeword set, where the number of angle sampling is determined by the number of antennas and is equal to the number of antennas. Therefore, there are N angle sampling values and S distance sampling rings.
[0020] Furthermore, in step S2, generating a beam training strategy and a power allocation strategy includes the following contents:
[0021] S201, construct a DRL-based resource allocation network, including BTN (Beam training network) and PAN (Power allocation network); BTN includes Q network Q (s t ,a t ;κ) and the target Q network Q(s t ,a t κ - ), the weights in the Q network will be copied to the target Q network; PAN includes the actor network π(s t ;φ), critic network Q(st ,a t ;ψ), target actor networkπ(s t ; φ - ) and the target critic network Q(s t ,a t ψ - ), κ and κ - Represent the parameters of the Q network and the target Q network, s t represents the state of the tth time slot, a t represents the beam training strategy and power allocation strategy for the tth time slot, φ and φ - Represent the parameters of the actor network and the target actor network, ψ and ψ respectively. - Represent the parameters of the critic network and the target critic network respectively.
[0022] S202, user grouping and beam training are actually value-taking problems on discrete intervals, so the deep Q network is used to construct BTN; the power allocation problem is a value-taking problem in continuous space, so DDPG is used to construct PAN.
[0023] The user location information of the tth time slot is taken as the state s t Input into the DRL-based resource allocation network, and output user grouping and beam training strategy through BTN W (1) (t),…,W (k) (t),…,W (K) (t)}, where s t ={L 1 (t),L 2 (t),…,L m (t),…,L M (t)},L m (t) represents the location of the mth user in the tth time slot, L m (t) = {θ m (t),γ m (t)},θ m (t) is the angle of the mth user relative to the base station in the tth time slot, γ m (t) represents the distance of the mth user relative to the base station in the tth time slot, L M (t) represents the location of the Mth user in the tth time slot, 1≤m≤M, M represents the total number of users, represents the grouping coefficient for determining whether the mth user is in the kth group, W (k) (t) represents the codeword allocated to the kth group, k = 1, 2, ..., K, and K represents the total number of user groups.
[0024] like This indicates that the mth user is not in the kth group. This indicates that the mth user is in the kth group; This includes information about which group each user belongs to and the codewords used by each group during beam training.
[0025] S203, the actor network is a policy network, which takes the current state as input and outputs the probability distribution of the action or the continuous action value. The critic network then evaluates the quality of the action and adjusts the policy. The actor-critic algorithm separates the action value evaluation and policy update process. The actor can fully explore the current environment and slowly update the policy. The critic only needs to evaluate the quality of the policy, so this integrated algorithm has relatively good performance. The target actor network and target critic network in PAN do not learn and update, but only copy the parameters of the actor network and the critic network once every period of time to calculate the objective function value.
[0026] Through the actor network in PAN, the power allocation strategy is output using the policy gradient algorithm where p m (t) represents the power of the mth user in the tth time slot, p M (t) represents the power of the Mth user; Includes the power allocated by the base station to each user.
[0027] S204: Obtaining the beam training strategy and power allocation strategy for the tth time slot
[0028] S205: According to the obtained beam training strategy and power allocation strategy, users are grouped, beam trained, and power allocated, and the results are fed back to the DRL-based resource allocation network, and the corresponding rewards are calculated through the reward function. The reward is the judgment of the neural network on the current output strategy. The better the output strategy, the higher the reward.
[0029] Furthermore, in step S3, obtaining a trained DRL-based resource allocation network includes the following:
[0030] In each time slot, the DRL-based resource allocation network interacts with the system environment to update the current state and obtain the user's location at the next moment to obtain the data sample {s t ,a t ,r t ,s t+1}, the data sample is stored in the experience replay pool, which is the experience tuple, where s t+1 represents the state of the t+1th time slot, r t represents the reward of the tth time slot; the training of the DRL-based resource allocation network is divided into two parts: training BTN and training PAN.
[0031] (1) Training BTN:
[0032] Randomly select U experience tuples {s t ,a t ,r t ,s t+1} Input into the target Q network of BTN to obtain the corresponding target Q value By minimizing the loss function Update the parameter κ to complete the training of BTN; where Q() represents the operation performed by the target Q network, s u represents the state in the u-th experience tuple, represents the strategy of BTN output in the u-th experience tuple, u = 1, 2, …, U.
[0033] (2) Training PAN:
[0034] Randomly select U′ experience tuples {s t ,a t ,r t ,s t+1} is input into the actor network of PAN, through Update the parameter φ by minimizing the loss function Update the parameter ψ in the critic network; update the parameter φ in the target actor network in a soft update manner - and the parameter ψ in the target critic network - , the specific formula is:
[0035] φ - =τφ+(1-τ)φ -
[0036] ψ - =τψ+(1-τ)ψ -
[0037] Among them, τ represents the soft update weight.
[0038] Soft updates only change the target Q value slowly, which can greatly improve learning stability.
[0039] Furthermore, in step S4, beam training and power allocation include the following:
[0040] Since the optimization goal of the system is to maximize the sum of the rates of all users while satisfying the total power limit of the transmitter, the DRL resource allocation network will tend to choose actions that make the reward function larger. Then, as the DRL resource allocation network and the system environment continue to interact, the DRL resource allocation network will choose beam training strategies and power allocation strategies that meet the power constraints, thereby achieving the goal of maximizing the sum of the rates of all users.
[0041] In the NOMA downlink system, each user receives the superimposed signals of all users in the same group from the base station, so the user end performs SIC technology to decode the received signal to achieve interference elimination. The superimposed signals transmitted by the base station contain information of different users, and these signal information are received by the users separately after the same channel fading.
[0042] The channel gain of the mth user in the kth group in the tth time slot is expressed as:
[0043] G m (t)=|W (k)H (t)H m (t)| 2
[0044] Among them, G m (t) represents the channel gain of the mth user in the tth time slot, W (k)H (t) represents W (k) The conjugate transpose of (t), H m (t) represents the near-field spherical wave channel vector of the m-th user in the t-th time slot.
[0045] When the channel gains of all users meet the following conditions:
[0046] |G 1 (t)| 2 >|G 2 (t)| 2 >…>|G M (t)| 2 ;
[0047] The rate of the mth user is expressed as:
[0048]
[0049] Among them, G M (t) represents the channel gain of the Mth user in the tth time slot, R m (t) represents the rate of the mth user in the tth time slot, p m (t) represents the power of the mth user in the tth time slot, represents the grouping coefficient for determining whether the qth user in the tth time slot is in the kth group, p q (t) represents the power of the qth user in the tth time slot, δ 2 represents the additive white Gaussian noise power.
[0050] When the sum of all user powers is less than or equal to the maximum power that the base station can provide, the reward r in the tth time slot is t Set as and as the system's sum rate, otherwise r will be rewarded t Set to 0; this setting ensures that the power allocation strategy of the proposed architecture meets the power constraint while maximizing the system and rate.
[0051] Furthermore, the present invention also proposes a beam training and power allocation system for a near-field NOMA system, comprising:
[0052] The user location information acquisition and codebook design module is used to acquire user location information and design codebooks for the near-field NOMA system.
[0053] The beam training strategy and power allocation strategy acquisition module is used to build a DRL-based resource allocation network, generate the beam training strategy and power allocation strategy for the tth time slot according to the input user location information, and obtain the corresponding reward.
[0054] The model training module is used to train the DRL-based resource allocation network using the experience tuple to obtain a trained DRL-based resource allocation network.
[0055] The beam training and power allocation module is used to input the location information of all users in the current time slot into the trained DRL-based resource allocation network, dynamically adjust the beam training strategy and power allocation strategy by maximizing the rewards obtained by the system, obtain the strategy corresponding to the maximum and rate of the system, and complete the beam training and power allocation.
[0056] Furthermore, the present invention also proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the beam training and power allocation method for a near-field NOMA system described above are implemented.
[0057] Furthermore, the present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is run by a processor, it executes the beam training and power allocation method of a near-field NOMA system described above.
[0058] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:
[0059] 1. The present invention makes full use of the advantages of deep reinforcement learning, effectively reduces the interference between users in the same group, and effectively improves the transmission rate of users while ensuring that the power constraints of the transmitter are met. By continuously interacting with the system during network training, the ability of the network to fit the system environment and the ability of beam training and power allocation can be effectively improved, thereby improving the transmission rate of the entire communication system, thereby meeting the requirements of future mobile near-field communication systems for communication quality.
[0060] 2. This invention combines near-field communication and NOMA technology for the first time, and introduces the user grouping problem on the basic near-field communication system. At the same time, when performing beam training, the allocation of codewords is no longer based on a single user, but on user groups. The use of deep reinforcement learning can effectively handle the decision-making problems of complex systems, especially in resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1 It is an overall implementation flow chart of the present invention.
[0062] Figure 2 It is a network structure diagram of the overall implementation of the present invention.
[0063] Figure 3 is a graph of convergence performance at different learning rates in an embodiment of the present invention.
[0064] Figure 4 It is a total rate performance diagram obtained by resource allocation in an embodiment of the present invention.
[0065] Figure 5 It is a comparison chart of the operation time of various methods in the embodiments of the present invention. DETAILED DESCRIPTION
[0066] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0067] In order to achieve low-cost and efficient beam allocation and resource allocation, the present invention proposes a beam training and power allocation method for a near-field NOMA system, such as Figure 1 , Figure 2 As shown, the specific steps are as follows:
[0068] S1. Acquire user location information and design codebook for near-field NOMA system. The specific contents are as follows:
[0069] Assume that all M users of the service are within the near field range. The user's location information, including elevation angle and distance, is obtained through positioning. The near field downlink channel is expressed as:
[0070]
[0071] The transmitting antenna array is {1, 2, …, N}, where N represents the total number of transmitting antennas; L represents the total number of paths between the transmitting end and the user; H(t) represents the near-field spherical wave channel vector; g l (t) represents the channel gain of the lth path in the tth time slot; j represents the imaginary unit; λ c Represents the wavelength of electromagnetic waves; r l (t) represents the distance of the lth path in the tth time slot; θ l (t) represents the angle of the lth path in the tth time slot; b(θ l (t),r l (t)) represents the steering vector of the lth path in the tth time slot.
[0072] In this embodiment, the base station is equipped with a uniform linear array of N = 64 antennas, and the antenna spacing is set to half a wavelength to avoid the grid effect. The number of users is M = 4, divided into two groups, each NOMA group contains two users. The noise power spectral density in the communication scenario is set to -154dbm / Hz, and the number of paths from the user to the base station is set to L = 3.
[0073] Where, H(t)=[-0.02658013+0.99964669j,0.95673525+0.29095989j,.....] T , T represents transpose; λ c =0.01m; r 1 (t) = 32.8, r 2 (t) = 8.6, r 3 (t) = 24.5; g 1 (t) = 0.85, g 2 (t) = 0.07, g 3 (t) = 0.08;
[0074] The steering vector for the first path in the first time slot is
[0075]
[0076] The codebook design for the near-field NOMA system is as follows:
[0077]
[0078] in, represents the set of codewords, i.e., the codebook; θ 1 Represents the first angle sampling value, which is r 1 1 Indicates the first distance sampling value on the first distance sampling ring, which is 1.02m; θ N Represents the Nth angle sampling value, which is r 1 N Indicates the Nth distance sampling value on the first distance sampling ring; Represents the first distance sampling value on the Sth distance sampling ring; represents the Nth distance sampling value on the Sth distance sampling ring; b(θ 1 ,r 1 1 ) represents the steering vector corresponding to the position determined by the first angle sampling and the first distance sampling loop; b(θ N ,r 1 N ) represents the steering vector corresponding to the position determined by the Nth angle sampling and the first distance sampling ring; represents the steering vector corresponding to the position determined by the first angle sampling and the Sth distance sampling ring; Represents the steering vector corresponding to the position determined by the Nth angle sampling and the Sth distance sampling ring; the steering vector is the codeword.
[0079] There are N×S codewords in the codeword set, where the number of angle sampling is determined by the number of antennas and is equal to the number of antennas. Therefore, there are N angle sampling values and S distance sampling rings.
[0080] In this embodiment, the codebook is
[0081]
[0082] S2. Build a resource allocation network based on DRL (Deep reinforcement learning) to generate the beam training strategy and power allocation strategy for the tth time slot according to the input user location information, and obtain the corresponding reward. The specific content is as follows:
[0083] S201, construct a DRL-based resource allocation network, including BTN (Beam training network) and PAN (Power allocation network); BTN includes Q network Q (s t ,a t ;κ) and the target Q network Q(s t ,a t κ -), the weights in the Q network will be copied to the target Q network; PAN includes the actor network π(s t ;φ), critic network Q(s t ,a t ;ψ), target actor networkπ(s t ; φ - ) and the target critic network Q(s t ,a t ψ - ), κ and κ - Represent the parameters of the Q network and the target Q network, s t Indicates the state, a t represents the beam training strategy and power allocation strategy for the tth time slot, φ and φ - Represent the parameters of the actor network and the target actor network, ψ and ψ respectively. - Represent the parameters of the critic network and the target critic network respectively.
[0084] S202, user grouping and beam training are actually value-taking problems on discrete intervals, so the deep Q network is used to construct BTN; the power allocation problem is a value-taking problem in continuous space, so DDPG is used to construct PAN.
[0085] The user location information of the tth time slot is taken as the state s t Input into the DRL-based resource allocation network, and output user grouping and beam training strategy through BTN where s t ={L 1 (t),L 2 (t),…,L m (t),…,L M (t)},L m (t) represents the location of the mth user in the tth time slot, L m (t) = {θ m (t),γ m (t)},θ m (t) is the angle of the mth user relative to the base station in the tth time slot, γ m (t) represents the distance of the mth user relative to the base station in the tth time slot, L M (t) represents the location of the Mth user in the tth time slot, 1≤m≤M, M represents the total number of users, represents the grouping coefficient for determining whether the mth user is in the kth group, W (k) (t) represents the codeword assigned to the kth group, k = 1, 2, ..., K, K represents the total number of user groups;
[0086] like This indicates that the mth user is not in the kth group. This indicates that the mth user is in the kth group; This includes information about which group each user belongs to and the codewords used by each group during beam training.
[0087] In this embodiment, In the first time slot,
[0088] S203, the actor network is a policy network, which takes the current state as input and outputs the probability distribution of the action or the continuous action value. The critic network then evaluates the quality of the action and adjusts the policy. The actor-critic algorithm separates the action value evaluation and policy update process. The actor can fully explore the current environment and slowly update the policy. The critic only needs to evaluate the quality of the policy, so this integrated algorithm has relatively good performance. The target actor network and target critic network in PAN do not learn and update, but only copy the parameters of the actor network and the critic network once every period of time to calculate the objective function value.
[0089] Through the actor network in PAN, the power allocation strategy is output using the policy gradient algorithm where p m (t) represents the power of the mth user in the tth time slot, p M (t) represents the power of the Mth user; Includes the power allocated by the base station to each user.
[0090] S204: Obtaining the beam training strategy and power allocation strategy for the tth time slot
[0091] In this embodiment, in the first time slot, but
[0092] S205: According to the obtained beam training strategy and power allocation strategy, users are grouped, beam trained, and power allocated, and the results are fed back to the DRL-based resource allocation network, and the corresponding rewards are calculated through the reward function. The reward is the judgment of the neural network on the current output strategy. The better the output strategy, the higher the reward.
[0093] S3. Train the DRL-based resource allocation network using the experience tuple to obtain a trained DRL-based resource allocation network; the experience tuple includes the user location information at the current moment and the next moment, the beam training strategy and power allocation strategy generated in step S2, and the reward obtained by the system executing the corresponding strategy, and the specific content is:
[0094] In each time slot, the DRL-based resource allocation network interacts with the system environment to update the current state and obtain the user's location at the next moment to obtain the data sample {s t ,a t ,r t ,s t+1}, the data sample is stored in the experience replay pool, which is the experience tuple, where s t+1 represents the state of the t+1th time slot, r t represents the reward of the tth time slot; the training of the DRL-based resource allocation network is divided into two parts: training BTN and training PAN.
[0095] (1) Training BTN:
[0096] Randomly select U experience tuples {s t ,a t ,r t ,s t+1} Input into the target Q network of BTN to obtain the corresponding target Q value By minimizing the loss function Update the parameter κ to complete the training of BTN; where Q() represents the operation performed by the target Q network, s u represents the state in the u-th experience tuple, represents the strategy of BTN output in the u-th experience tuple, u = 1, 2, …, U.
[0097] (2) Training PAN:
[0098] Randomly select U′ experience tuples {s t ,a t ,r t ,s t+1} is input into the actor network of PAN, through Update the parameter φ by minimizing the loss function Update the parameter ψ in the critic network; update the parameter φ in the target actor network in a soft update manner - and the parameter ψ in the target critic network - , the specific formula is:
[0099] φ - =τφ+(1-τ)φ -
[0100] ψ - =τψ+(1-τ)ψ -
[0101] Among them, τ represents the soft update weight.
[0102] Soft updates only change the target Q value slowly, which can greatly improve learning stability.
[0103] S4: Input the location information of all users in the current time slot into the trained DRL-based resource allocation network. By maximizing the rewards obtained by the system, dynamically adjust the beam training strategy and power allocation strategy, obtain the strategy corresponding to the maximum sum rate of the system, and complete the beam training and power allocation. The specific contents are as follows:
[0104] Since the optimization goal of the system is to maximize the sum of the rates of all users while satisfying the total power limit of the transmitter, the DRL resource allocation network will tend to choose actions that make the reward function larger. Then, as the DRL resource allocation network and the system environment continue to interact, the DRL resource allocation network will choose beam training strategies and power allocation strategies that meet the power constraints, thereby achieving the goal of maximizing the sum of the rates of all users.
[0105] In the NOMA downlink system, each user receives the superimposed signals of all users in the same group from the base station, so the user end performs SIC technology to decode the received signal to achieve interference elimination. The superimposed signals transmitted by the base station contain information of different users, and these signal information are received by the users separately after the same channel fading.
[0106] The channel gain of the mth user in the kth group in the tth time slot is expressed as:
[0107] G m (t)=|W (k)H (t)H m (t)| 2
[0108] Among them, G m (t) represents the channel gain of the mth user in the tth time slot, W (k)H (t) represents W (k) The conjugate transpose of (t), H m (t) represents the near-field spherical wave channel vector of the m-th user in the t-th time slot.
[0109] When the channel gains of all users meet the following conditions:
[0110] |G 1 (t)| 2 >|G 2 (t)| 2 >…>|G M (t)| 2 ;
[0111] The rate of the mth user is expressed as:
[0112]
[0113] Among them, G M (t) represents the channel gain of the Mth user in the tth time slot, R m (t) represents the rate of the mth user in the tth time slot, p m (t) represents the power of the mth user in the tth time slot, represents the grouping coefficient for determining whether the qth user in the tth time slot is in the kth group, p q (t) represents the power of the qth user in the tth time slot, δ 2 represents the additive white Gaussian noise power.
[0114] In this embodiment, the channel gains are: G 1 (t) = 0.83, G 2 (t) = 0.79, G 3 (t) = 0.67 and G 4 (t) = 0.54, the corresponding rates are: R 1 (t) = 1.32, R 2 (t) = 0.81, R 3 (t) = 1.18 and R 4 (t) = 0.51.
[0115] The signal sent by the base station to the mth user is X m (t), which satisfies E[|X m (t)| 2 ]=1.
[0116] When the sum of all user powers is less than or equal to the maximum power that the base station can provide, the reward r in the tth time slot is t Set as and as the system's sum rate, otherwise r will be rewarded t Set to 0; this setting ensures that the power allocation strategy of the proposed architecture meets the power constraint while maximizing the system and rate.
[0117] The present invention selects the total rate of all users as the optimization target, so the joint problem of beam training and power allocation can be expressed as:
[0118]
[0119] Among them, P max Indicates the maximum power that the base station can provide, W (a) (t) represents the codeword assigned to group a,
[0120] W (b) (t) represents the codeword assigned to group b; C1, C2, C3, C4, C5 and C6 all represent constraint formulas, C1 ensures that the total power allocated to all users by the base station is less than the maximum power limit, C3 ensures that the codewords assigned to different groups are not repeated, and C4, C5, C6 all represent group indexes Can only be 0 or 1. A user can only be in one group, and every user is in one group.
[0121] The performance of the method proposed in the present invention in the actual system is as follows Figure 3-5 As shown in Figure 2, reinforcement learning is performed for 300 rounds, each round contains 2000 time slots, the soft update weight is 0.9, and the experience replay area size is 500. Figure 3 As shown in FIG. 1 , the performance of the method is evaluated by changing the learning rate of the subchannel allocation network. It can be seen from the figure that the method can converge at a faster speed under different learning rates. The performance of the reinforcement learning method used in the present invention is shown in FIG. Figure 4 As shown in the figure, as a comparison, the same problem is solved using a full DQN-based solution (QB-QP) and a random selection method. Although the QB-QP method can effectively solve the beam training problem, when faced with the power problem of the continuous action space, it can only solve the problem by quantizing the action space, which not only increases the amount of computation, but also introduces additional errors. At the same time, in the random selection method, the agent randomly selects an action at any time slot without considering the quality of the actions taken at other times, which also affects the final performance. This method can obtain global information, thereby better improving the final performance. The average calculation time of several schemes is compared, as shown in the figure. Figure 5 As shown in Figure 2, it can be seen that the average running time of this method is of the same order of magnitude as other schemes and is faster than the QB-QP scheme because quantizing the action space will increase the amount of computation. Therefore, this method has lower computational time overhead and computational complexity.
[0122] In summary, the present invention studies the problem of dynamic resource allocation with the goal of maximizing the transmission rate and in the near-field downlink NOMA system. The present invention proposes a flexible deep reinforcement learning method to solve problems that are difficult to model in complex communication scenarios.
[0123] The embodiment of the present invention also proposes a beam training and power allocation system for a near-field NOMA system, including a user location information acquisition and codebook design module, a beam training strategy and power allocation strategy acquisition module, a model training module, a beam training and power allocation module, and a computer program that can be run on a processor. It should be noted that each module in the above system corresponds to the specific steps of the method provided in the embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.
[0124] The embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that when the processor executes the computer program, the specific steps of the method provided in the embodiment of the present invention are corresponding to the specific steps of the method provided in the embodiment of the present invention, and the processor has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.
[0125] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. It should be noted that when the computer program is executed by the processor, it corresponds to the specific steps of the method provided in the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.
[0126] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A beam training and power allocation method for a near-field NOMA system, characterized in that: include: S1. Acquire user location information and design codebook for near-field NOMA system; S2. Build a DRL-based resource allocation network and generate the beam training strategy and power allocation strategy a for the tth time slot according to the user location information. t , and receive corresponding rewards; S3, using the experience tuple to train the DRL-based resource allocation network to obtain a trained DRL-based resource allocation network; The experience tuple includes the user location information at the current moment and the next moment, the beam training strategy and power allocation strategy generated in step S2, and the reward obtained by the system executing the corresponding strategy; S4. Input the location information of the user in the current time slot into the trained DRL-based resource allocation network. By maximizing the reward obtained by the near-field NOMA system, dynamically adjust the beam training strategy and power allocation strategy, obtain the strategy corresponding to the maximum sum rate of the system, and complete the beam training and power allocation.
2. The beam training and power allocation method of the near-field NOMA system according to claim 1, characterized in that: In step S1, user location information acquisition and codebook design include the following: Assume that all M users of the service are within the near field range. The user's location information, including elevation angle and distance, is obtained through positioning. The near field downlink channel is expressed as: The transmitting antenna array is {1, 2, …, N}, where N represents the total number of transmitting antennas; L represents the total number of paths between the transmitting end and the user; H(t) represents the near-field spherical wave channel vector; g l (t) represents the channel gain of the lth path in the tth time slot; j represents the imaginary unit; λ c Represents the wavelength of electromagnetic waves; r l (t) represents the distance of the lth path in the tth time slot; θ l (t) represents the angle of the lth path in the tth time slot; b(θ l (t),r l (t)) represents the steering vector of the lth path in the tth time slot; The codebook design for the near-field NOMA system is as follows: in, represents a set of codewords, θ1 represents the first angle sampling value, r1 1 represents the first distance sampling value on the first distance sampling ring, θ N Indicates the Nth angle sampling value, r1 N Indicates the Nth distance sampling value on the first distance sampling ring, Represents the first distance sampling value on the Sth distance sampling ring, represents the Nth distance sampling value on the Sth distance sampling ring, b(θ1,r1 1 ) represents the steering vector corresponding to the position determined by the first angle sampling and the first distance sampling ring, b(θ N ,r1 N ) represents the steering vector corresponding to the position determined by the Nth angle sampling and the first distance sampling ring, Indicates the steering vector corresponding to the position determined by the first angle sampling and the Sth distance sampling ring, Represents the steering vector corresponding to the position determined by the Nth angle sampling and the Sth distance sampling ring; the steering vector is the codeword.
3. The beam training and power allocation method for a near-field NOMA system according to claim 1, characterized in that: In step S2, generating a beam training strategy and a power allocation strategy includes the following: S201, construct a DRL-based resource allocation network, including BTN and PAN; wherein BTN includes Q network Q(s t ,a t ; k) and target Q network Q(s t ,a t ;k - ), copy the weights in the Q network to the target Q network; PAN includes the actor network π(s t ;φ), critic network Q(s t ,a t ;ψ), target actor networkπ(s t ; φ - ) and the target critic network Q(s t ,a t ψ - ), the critic network is used to evaluate the output of the actor network, the target actor network copies the parameters of the actor network, and the target critic network copies the parameters of the critic network; κ and κ - Represent the parameters of the Q network and the target Q network, s t represents the state of the tth time slot, a t represents the beam training strategy and power allocation strategy for the tth time slot, φ and φ - Represent the parameters of the actor network and the target actor network, ψ and ψ respectively. - Represent the parameters of the critic network and the target critic network respectively; S202: Use the user location information of the tth time slot as state s t Input into the DRL-based resource allocation network, and output user grouping and beam training strategy through BTN W (1) (t),…,W (k) (t),…,W (K) (t)}, where s t ={L1(t),L2(t),…,L m (t),…,L M (t)},L m (t) represents the location of the mth user in the tth time slot, L m (t) = {θ m (t),γ m (t)},θ m (t) represents the angle of the mth user relative to the base station in the tth time slot, γ m (t) represents the distance of the mth user relative to the base station in the tth time slot, L M (t) represents the location of the Mth user in the tth time slot, 1≤m≤M, M represents the total number of users, represents the grouping coefficient for determining whether the mth user is in the kth group, W (k) (t) represents the codeword assigned to the kth group in the tth time slot, k = 1, 2, ..., K, K represents the total number of user groups; when This indicates that the mth user is not in the kth group. This indicates that the mth user is in the kth group; This includes information about which group each user belongs to and the codewords used by each group during beam training; S203, Through the actor network in PAN, the power allocation strategy is output using the policy gradient algorithm where p m (t) represents the power of the mth user in the tth time slot, p M (t) represents the power of the Mth user in the tth time slot; Including the power allocated by the base station to each user; S204: Obtaining the beam training strategy and power allocation strategy for the tth time slot S205. According to the obtained beam training strategy and power allocation strategy, users are grouped, beam trained, and power allocated, and the results are fed back to the DRL-based resource allocation network, and the corresponding rewards are calculated through the reward function.
4. The beam training and power allocation method for a near-field NOMA system according to claim 1, characterized in that: In step S3, obtaining the trained DRL-based resource allocation network includes the following: In each time slot, the DRL-based resource allocation network interacts with the system environment to update the current state and obtain the user's location at the next moment to obtain the data sample {s t ,a t ,r t ,s t+1 }, the data sample is stored in the experience replay pool, which is the experience tuple, where s t+1 represents the state of the t+1th time slot, r t represents the reward of the tth time slot; The training of the DRL-based resource allocation network is divided into two parts: (1) Training BTN: Randomly select U experience tuples {s t ,a t ,r t ,s t+1 } is input into the target Q network of BTN to obtain the corresponding target Q value y t , The parameter κ is updated by minimizing the loss function L(κ) to complete the training of BTN. in represents the user grouping and beam training strategy for the tth time slot, Q() represents the operation performed by the target Q network, κ and κ - Represent the parameters of the Q network and the target Q network, y u represents the target Q value of the u-th experience tuple, s u represents the state of the u-th experience tuple, represents the user grouping and beam training strategy for the u-th experience tuple, u = 1, 2, ..., U; (2) Training PAN: Randomly select U′ experience tuples {s t ,a t ,r t ,s t+1 } Input to the actor network of PAN, update the parameter φ by deterministic gradient descent, and update the parameter ψ in the critic network by minimizing the loss function L(ψ); Update the parameter φ in the target actor network in a soft update manner - and the parameter ψ in the target critic network - , the specific formula is: f - =τφ+(1-τ)φ - ψ - =tψ+(1-t)ψ - Among them, τ represents the soft update weight, s t represents the state of the tth time slot, ψ represents the parameters of the critic network, and φ represents the parameters of the actor network.
5. The beam training and power allocation method for a near-field NOMA system according to claim 1, characterized in that: In step S4, beam training and power allocation include the following: The channel gain of the mth user in the kth group in the tth time slot is expressed as: G m (t)=|W (k)H (t)H m (t)| 2 Among them, G m (t) represents the channel gain of the mth user in the tth time slot, 1≤m≤M, M represents the total number of users, W (k) (t) represents the codeword assigned to the kth group in the tth time slot, W (k)H (t) represents W (k) The conjugate transpose of (t), H m (t) represents the near-field spherical wave channel vector of the m-th user in the t-th time slot; When the channel gains of all users meet the following conditions: |G1(t)| 2 >|G2(t)| 2 >…>|G M (t)| 2 ; The rate of the mth user is expressed as: Among them, G M (t) represents the channel gain of the Mth user in the tth time slot, R m (t) represents the rate of the mth user in the tth time slot, p m (t) represents the power of the mth user in the tth time slot, represents the grouping coefficient for determining whether the qth user in the tth time slot is in the kth group, p q (t) represents the power of the qth user in the tth time slot, δ 2 represents the additive white Gaussian noise power; When the sum of all user powers is less than or equal to the maximum power that the base station can provide, the reward r in the tth time slot is t Set as and as the system's sum rate, otherwise r will be rewarded t Set to 0.
6. A system for beam training and power allocation method of a near-field NOMA system according to claim 1, characterized in that: include: User location information acquisition and codebook design module, used to acquire user location information and design codebook for near-field NOMA system; The beam training strategy and power allocation strategy acquisition module is used to build a DRL-based resource allocation network, generate the beam training strategy and power allocation strategy for the tth time slot according to the input user location information, and obtain the corresponding reward; A model training module is used to train the DRL-based resource allocation network using the experience tuple to obtain a trained DRL-based resource allocation network; The beam training and power allocation module is used to input the location information of all users in the current time slot into the trained DRL-based resource allocation network, dynamically adjust the beam training strategy and power allocation strategy by maximizing the rewards obtained by the system, obtain the strategy corresponding to the maximum and rate of the system, and complete the beam training and power allocation.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the beam training and power allocation method according to any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the beam training and power allocation method according to any one of claims 1 to 5 is executed.
