A smart beamwidth optimization method based on reinforcement learning

Abstract

The present invention discloses an intelligent beam width optimization method based on reinforcement learning. The method suppresses the beam drift effect in millimeter wave communication by dynamically adjusting the beam width. The algorithm models the beam width dynamic optimization problem as a Markov decision process, and selects the optimal beam width for data transmission at each decision moment. Its state design can characterize the severity of the beam drift effect of the current system from multiple angles, and each action corresponds to a different beam width, thereby optimizing and adjusting the width of the data transmission beam by sensing the speed of environmental changes. In each state, the optimal beam width for data transmission is selected according to the AC algorithm, and the policy network and value network are continuously updated during training to improve the rationality and reliability of model selection, while introducing a dynamic network update mechanism to reduce the computational burden of network updates. The throughput of the millimeter wave communication system under the beam drift effect is improved, thereby continuously ensuring the quality of the communication link.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communications, and in particular relates to an intelligent beam width optimization method based on reinforcement learning actor-critic (AC) in millimeter wave communications. Background Art

[0002] Millimeter waves have attracted a lot of attention as a potential technology to solve the demand for high-speed wireless data transmission due to their huge accessible bandwidth. However, compared with traditional low-frequency bands, despite the high bandwidth, the free space path loss of the millimeter wave band is large and the ability to bypass objects is poor. Therefore, array antennas consisting of dozens or even hundreds of antennas are required to concentrate energy through beamforming technology to improve the signal-to-noise ratio. Usually, the spacing and wavelength of antennas in the antenna array are of the same order of magnitude, which brings a lot of room for reducing the physical volume of the millimeter wave antenna array, so that the millimeter wave array antenna can be made small enough to meet the application of more scenarios. Beamforming makes millimeter waves highly directional, so beam tracking technology is needed to ensure the stability and quality of the communication link. It is worth noting that due to the high cost of the RF link and the high power consumption of ADC / DAC, millimeter wave communication systems usually adopt a hybrid digital-analog precoding architecture, and the number of RF chains is far less than the number of antennas.

[0003] Beamforming is only the basis for improving the signal-to-noise ratio. How to choose a reasonable direction for the directional beam is another major challenge to obtain a higher signal-to-noise ratio. The direction of the main beam after beamforming needs to be aligned with the user in real time, that is, beam training is required to ensure that the transmitter main beam is aligned and track the moving receiver. Most research schemes first perform beam training in a time slot, obtain a high-yield beam direction, and then use the beam direction to transmit data in the time slot. However, this is based on the premise that the communication environment between the user and the base station remains unchanged in each time slot. When the receiving end user moves continuously, its deviation angle or arrival angle always changes continuously. Even if the main beam of the transmitter and the receiver is aligned through beam training at the beginning of each time slot, the relative position of the receiver with the transmitter will change due to the movement of the receiver in the time slot, resulting in a decrease in beamforming / array gain. This phenomenon is called beam drift effect. In severe cases, it will affect the quality of the communication link and even cause communication interruption.

[0004] The process of beam training is actually a process of weighing the beam alignment success rate and beam training efficiency. Generally, narrower beams can bring higher beamforming gains, but at the same time they will generate a greater search burden and reduce search accuracy. When the beam drift effect is considered, the relative invariance of the environment within the time slot is broken. The faster the environment changes within the time slot, the more serious the impact on the beamforming gain. In order to suppress the beam drift effect, it is necessary to balance the beamforming gain and the effective time of data transmission. Although the beam drift effect can be alleviated by reducing the length of each time slot, it also increases the overhead of beam training and reduces the effective reachability performance. Putting aside the idea of ​​blindly using narrow beams, when the beam becomes wider, the range that a single beam can cover becomes larger. Although the maximum beamforming gain is affected compared to the narrow beam, it can still achieve better data transmission efficiency compared to the failure of beam alignment within the time slot. Therefore, adjusting the beam width in real time according to changes in the communication environment is an effective solution to deal with the beam drift effect. Summary of the invention

[0005] Technical problem: In order to suppress the beam drift effect and ensure the stability of the communication link, the present invention proposes an intelligent beam width optimization method, which offsets the influence of the beam drift effect on the beamforming gain by adaptively adjusting the beam width. This method can improve the throughput of the millimeter wave communication system and ensure the stability of the communication link quality during the entire communication process.

[0006] Technical solution: The present invention adopts an intelligent beam width optimization method based on reinforcement learning, which includes the following steps:

[0007] Step 1. Model the beam width optimization problem as a Markov decision process (MDP). Use the Actor-Critic algorithm based on a combination of policy learning and value learning. Define codebooks of different resolutions, i.e., beam widths, as action spaces. The policy network and value network are composed of fully connected networks. The policy network is responsible for selecting the best action at each moment. The value network is responsible for evaluating the choices fed back to the policy network based on the objective environment. Through continuous interaction with the environment, it perceives the dynamic characteristics of the environment and selects the best codebook codeword for data transmission.

[0008] Step 2. Initialization: Establish and initialize the policy network and value network, multi-resolution DFT codebook, data transmission initial codebook, time unit counting, channel environment construction and network update threshold, and open up storage space;

[0009] Step 3. At the beginning of each decision time slot, the current state is constructed based on the acquired information and the state design criteria, and the optimal action, i.e., beam width, is determined through the strategy network;

[0010] Step 4. Combine the optimal beam width with the optimal beam direction obtained by beam alignment, select the best codeword in the selected codebook and perform data transmission;

[0011] Step 5. Calculate the reward information based on the information in the data transmission process, compensate the reward, and determine whether the network update threshold is met. If so, execute step 6. If not, this loop will directly jump back to step 3 to start the next loop;

[0012] Step 6. Update the policy network and value network parameters based on the reward information.

[0013] in,

[0014] Step 1: First, the beam width optimization problem is characterized as an MDP, and the MDP model parameters including state design, action space, and reward design are effectively and reasonably designed. The beam width for data transmission is adaptively adjusted according to the severity of the beam drift effect, fully balancing the relationship between the beamforming gain and the effective time of data transmission.

[0015] If the time unit of the beam training process to determine the optimal beam direction is a time slot, and the time scale of beam width selection is defined as a time unit, and each time unit contains M time slots, then the MDP parameters are defined as follows:

[0016] Step 1.1. State design: In M time slots, that is, in one time unit, obtain M sets of beam training experience (S, A, R, S t+1 ), so M beam training rewards can be obtained, namely

[0017] R={R t ,R t+1 ,...,R i ,...,R t+M-2 ,R t+M-1} (1)

[0018] Where R i represents the reward of the i-th time slot in this time unit, that is, the effective achievable rate obtained by beam training in this time slot, and then adopts the collection of indirect results obtained by some meaningful calculations of the R sequence as the state, that is,

[0019] State = {R av ,R cv ,R kur} (2)

[0020] The meaning of each parameter is as follows:

[0021] ·R av represents the mean of the R sequence, which represents the average size of the R sequence;

[0022] ·R cv The coefficient of variation of the R sequence. The mean and dispersion are the two main characteristics of the sequence, and the dispersion is often expressed by variance or standard deviation. However, when a series of objective conditions such as the beam alignment of different time slots are different, it is likely that the measurement scales of different groups of data will differ greatly. At this time, it is not appropriate to use the standard deviation directly for comparison, and the coefficient of variation can eliminate the difference in measurement scales. Its calculation is as follows: First, calculate the standard deviation σ of the R sequence

[0023]

[0024] The coefficient of variation is a normalized measure of the dispersion of a probability distribution, which is defined as the square root of the standard deviation σ and the mean R. av The ratio is

[0025]

[0026] ·R kur The kurtosis represents the kurtosis of the R sequence. The kurtosis characterizes the characteristic number of the peak height of the probability density distribution curve at the mean value. It is a statistic that describes the steepness of the distribution of all values ​​in the population. The purpose of introducing kurtosis is to add more additional information for the system to learn. The kurtosis R kur The calculation is as follows:

[0027]

[0028] Step 1.2. Action design: Define actions as codebooks with different beam widths, i.e., different resolutions. The multi-resolution codebook is defined as The corresponding action space is defined as

[0029] Step 1.3. Reward design: The reward is defined as the sum rate of M time slots in each time unit, that is,

[0030]

[0031] Among them, R u,i It represents the effective transmission rate corresponding to the i-th time slot in the u-th time unit.

[0032] The initialization method of step 2 is as follows:

[0033] Step 2.1. Construct a value network V and a policy network π consisting of fully connected layers, and initialize the value network parameters ω and the policy network parameters θ;

[0034] Step 2.2. Construct multi-resolution DFT codebook And randomly select the initial codebook for data transmission;

[0035] Step 2.3. Initialize the time unit count u = 1, and initialize the network update threshold l based on experience max ;

[0036] Step 2.4. Open up storage space. For time unit u, the storage space stores the best action selected for this time unit. And the reward information generated by the M time slot beam training within this time unit and the state information obtained thereby.

[0037] The step 3 performs the following steps in sequence in the time unit u:

[0038] Step 3.1. Obtain the reward sequence R obtained in M ​​time slots within this time unit according to formula 1;

[0039] Step 3.2. Calculate the status information s of this time unit according to formulas 4, 5, and 6 u ;

[0040] Step 3.3. Obtain a set of probability distributions for all actions through the policy network π, for the action space have That is, in state s, the probability of the policy network π(θ) selecting all actions is added to 1, and the best action is selected And select the corresponding data transmission codebook

[0041] Step 4: Execute the following steps in order in time unit u:

[0042] Step 4.1. Assume that the beam training process to find the best beam direction uses a codebook Obtain the best beam for this time unit based on the beam alignment result Corresponding beam center angle

[0043] Step 4.2. Select the best codeword for data transmission, assuming in represents the i-th codeword, Indicates the number of codewords in the codebook. Medium Code The corresponding beam center angle is The best codeword for data transmission

[0044]

[0045] in Represents natural numbers from 1 to In the specific implementation process, the index relationship between codebooks with different resolutions has specific objective rules, so a direct mapping between indexes of different codebooks can be constructed.

[0046]

[0047] In order to directly calculate the optimal transmission codeword through the mapping function

[0048] Step 5: Execute the following steps in order in time unit u:

[0049] Step 5.1) Accept the external environment reward and calculate the reward R of this time unit according to formula (1) u Due to the deviation between the codebook used for beam training and the codebook used for data transmission, the reward needs to be compensated as follows. The default codebook used for beam training is The corrected R u ' represents the following:

[0050]

[0051] Where S S express The number of code words, express The number of code words;

[0052] Step 5.2) Determine whether the network update threshold is met and design the normalized criticism loss To characterize whether the network needs to be continuously updated, The calculation is as follows:

[0053]

[0054] Among them, ω represents the value network parameter; U l is the observation window. The neural network is trained.

[0055] Step 6: Execute the following steps in order in time unit u:

[0056] Step 6.1. Perform the best action in time unit u Select the corresponding codeword for data transmission and obtain compensation reward R u ′, the state s of the next time unit u+1 can be obtained according to formula (2) u+1 , and obtain the best action for the next time unit based on the policy network This action is only used for subsequent calculations and is not actually executed;

[0057] Step 6.2. Set the current time unit state s uand action a u Input the value network to get q u =q(s u ,a u ;ω u ), the next time unit state s u+1 and predict actions Input value network and And calculate the TD-error value δ u =q u -(R u ′+γ·q u+1 );

[0058] Step 6.3. Deriving the value network yields Next, we use the time difference to update the value network as shown in formula ω u+1 =ω u -α·δ u ·d ω,u ;

[0059] Step 6.4. Deriving the policy network yields Next, we use gradient ascent to update the policy network parameters as θ u+1 =θ u +β·q u ·d θ,u , where q u You can use δ u replace;

[0060] Step 6.5. Update the counter u←u+1.

[0061] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows:

[0062] 1) On the basis of using only a single codebook in beam alignment training, a more realistic scenario, namely the relative change characteristics of the environment within the time slot, is considered. A multi-resolution codebook is introduced, and by sensing the rate of change of the channel azimuth within the time slot, the codebook with the best resolution is adaptively selected at each decision point. Combined with beam training, the most efficient codeword is comprehensively selected to improve system performance.

[0063] 2) This algorithm does not require channel state information. Considering the real-time changing characteristics of the real environment, reinforcement learning is introduced to interact with the environment in real time, continuously learn and try and error and perceive the laws of environmental changes, and make the most reasonable decision by making full use of the existing computing power through neural networks.

[0064] 3) Under the beam drift effect, different from the traditional idea, a narrow beam is used for beam training and a multi-resolution codebook is introduced for data transmission, while taking into account the beamforming gain and the effective time of data transmission. After simulation, the performance of the algorithm is better than that of using a single codebook for data transmission, and the more drastic the environmental changes, that is, the more serious the beam drift effect, the more obvious its performance advantage.

[0065] 4) The AC algorithm makes action selection based on policy learning, which is more direct than other algorithms. This algorithm greatly reduces the burden of neural network training through reasonable and concise state design and the introduction of dynamic training mechanism, and does not require a large storage space to store a series of context information. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Flowchart of the smart beamwidth optimization algorithm;

[0067] Figure 2 Information transfer diagram for beam training module and data transmission module;

[0068] Figure 3 Schematic diagram of average EAR performance of different codebooks under different signal-to-noise ratios;

[0069] Figure 4 Schematic diagram of the relationship between average EAR performance and environmental change rate under different codebooks. DETAILED DESCRIPTION

[0070] In order to make the technical solutions and advantages of the present application more clear, the present application is further described in detail below in combination with examples and software simulation.

[0071] Considering the beam drift effect, that is, the change in the alignment angle between the transmitter and the receiver in the beam training time slot may cause the beamforming gain to be damaged. This algorithm interacts with the dynamic environment by introducing reinforcement learning without the need for channel state information, constantly perceives the law of environmental changes in the time slot, and adaptively adjusts the data transmission beam width to ensure the stability of the communication link quality.

[0072] In the specific implementation scheme, the beam width optimization problem is first modeled as an MDP model. Through reasonable and efficient action, state and reward design, the optimal beam width is selected in each time unit, and the optimal codeword is selected for data transmission in combination with the beam training alignment results. In each time unit cycle, the neural network parameters are updated according to the situation to make its selection more reasonable.

[0073] 1. Model building

[0074] In the MDP model, we first need to define the state space, action space, and reward. Assume that the minimum time unit is a time slot, and each M time slot is defined as a time unit, that is, each time unit includes M time slots.

[0075] Definition state: In M time slots, that is, in one time unit, M sets of beam training experience (S, A, R, S t+1 ), so M rewards can be obtained, namely

[0076] R={R t ,R t+1 ,...,R i ,...,R t+M-2 ,R t+M-1} (1)

[0077] Where R i represents the reward of the i-th time slot in this time unit, that is, the effective achievable rate obtained by beam training in this time slot, and then adopts the collection of indirect results obtained by some meaningful calculations of sequence R as the state, that is,

[0078] State = {R av ,R cv ,R kur} (2)

[0079] The meaning of each parameter is as follows:

[0080] ·R av represents the mean of the R sequence, which represents the average size of the R sequence;

[0081] ·R cv Represents the coefficient of variation of the R sequence. The mean and dispersion are the two most important representations of sequence characteristics, and the dispersion is often represented by variance or standard deviation. However, when a series of objective conditions such as beam alignment in different time slots are different, it is likely that the measurement scales of different groups of data will differ greatly. At this time, it is not appropriate to use the standard deviation directly for comparison, and the coefficient of variation can eliminate the difference in measurement scales. Its calculation is as follows: First, calculate the standard deviation σ of the R sequence

[0082]

[0083] The coefficient of variation is a normalized measure of the dispersion of a probability distribution, which is defined as the square root of the standard deviation σ and the mean R. av The ratio is

[0084]

[0085] ·R kurRepresents the kurtosis of the R sequence. Kurtosis characterizes the characteristic number of the peak height of the probability density distribution curve at the mean value. It is a statistic that describes the steepness of the distribution of all values ​​in the population. The purpose of introducing kurtosis is to add more additional information for the system to learn. When the sequence has n samples, the kurtosis R kur The calculation is as follows:

[0086]

[0087] Define action space: define actions as codebooks with different beam widths, i.e., different resolutions. Multi-resolution codebooks are defined as The corresponding action space is defined as

[0088] Reward definition: The reward is defined as the sum rate of M time slots in each time unit, that is,

[0089]

[0090] Among them, R u,i It represents the effective transmission rate corresponding to the i-th time slot in the u-th time unit.

[0091] The policy network selects the best codebook based on the state After that, the best codeword needs to be selected based on the beam alignment result. A direct mapping function can be established between codeword indexes in different codebooks:

[0092]

[0093] Then the optimal transmission codeword is directly calculated through the mapping function

[0094] After data transmission, the external environment feedback is accepted as a reward and the reward is compensated. The default codebook used for beam training is The corrected R u ' represents the following:

[0095]

[0096] Where S S express The number of code words, express The number of code words.

[0097] Then determine whether the network update threshold is met and design the normalized criticism loss To characterize whether the network needs to be continuously updated, The calculation is as follows:

[0098]

[0099] Among them, ω represents the value network parameter; U l is the observation window. , the neural network is trained; otherwise, the learning process is skipped.

[0100] 2. Implementation of AC-based beamwidth optimization algorithm

[0101] Input: Multi-resolution codebook Action Space

[0102] Initialization: Establish the value network V and the policy network π, initialize the neural network parameters ω and θ, and the network update threshold l max , open up storage space, and the initial data transmission codebook is Let the time unit count u=1.

[0103] Loop body: For each time unit u, repeat the following steps:

[0104] (1) According to formula (1), we can get the reward sequence R obtained by M time slots in this time unit:

[0105] (2) Calculate the state information s of this time unit according to formulas (4, 5, 6): u

[0106] (3) Select the best action through the policy network π And select the corresponding transmission codebook

[0107] (4) Determine according to formula (7) The best codeword for data transmission in and perform data transmission

[0108] (5) Receive the return value of the external environment and obtain the modified compensation reward R according to formula (8) u '

[0109] (6) Calculate the normalized evaluation loss When satisfied If yes, go to step (7) normally, otherwise jump directly to step (10)

[0110] (7) According to formula (2), the state s of the next time unit u+1 is obtained u+1 , and obtain the best action for the next time unit based on the policy network

[0111] (8) Calculate q through the value network u =q(s u ,a u ;ωu )and And calculate the TD-error value δ u =q u -(r u +γ·q u+1 );

[0112] (9) Update the value network parameters and strategy network parameters ω respectively u+1 =ω u -a·δ u ·d ω,u ,θ u+1 =θ u +β·q u ·d θ,u

[0113] (10) Update counter u←u+1

[0114] 3. Simulation environment and results

[0115] The following environment is used during simulation:

[0116] 1) This simulation generates a time-varying millimeter wave channel environment in an office. The center frequency of the base station is 28 GHz, the transmit power is 10 dBm, the size of the office is 20×20 meters, and the number of uniform linear array (ULA) sources N=64 is used.

[0117] 2) Channel path clusters are randomly generated between 2 and 5. For ease of tracking, each cluster contains only one path. The power allocation between line-of-sight paths (LOS) and non-line-of-sight paths (NLOS) is determined by the Rice factor K. The attenuation of all NLOS paths approximately follows an exponential distribution. Since the exponential attenuation is very fast, in fact only the first NLOS path may be relatively obvious, and the power of other paths is approximately 0.

[0118] 3) The uniform linear array (ULA) can cover an area of ​​180°, and the azimuth angle φ satisfies 0<φ<π, while different resolution codebooks divide the sine values ​​of 0~π at different granularities. Includes 4 codebooks with different resolutions The number of code words is 8, 16, 32, and 64 respectively. (i=1,2,3,4) is abbreviated as MRC i .

[0119] 4) Use δ to quantify the rate of environmental change. The larger the δ value, the faster the environment changes.

[0120] Simulation results analysis:

[0121] Figure 3The relationship between the EAR performance and the signal-to-noise ratio using different subcodebooks is shown. The average effective achievable rate (EAR) can represent the effective transmission rate per unit spectrum and can characterize the throughput of the system. It can be seen that as the signal-to-noise ratio increases, the EAR performance of different subcodebooks also increases. However, the EAR performance achieved by the BWS algorithm is much better than the EAR performance achieved by any fixed subcodebook. The reason is that the designed BWS algorithm can sense the rate of change of the environment and adjust the beam width in real time, which achieves a good trade-off between obtaining high array gain and alleviating the beam drift effect.

[0122] Figure 4 The relationship between the average EAR performance and the environmental change rate under different codebooks is shown. It can be seen that as the environment changes faster and faster, the impact of the beam drift effect becomes more and more serious, and the EAR performance achieved by different sub-codebooks also decreases accordingly. However, the EAR performance of the BWS algorithm is better than any fixed sub-codebook under all environmental change rates. And as the environment changes faster and faster, the performance gap between them becomes larger and larger. These observations show that the designed BWS algorithm can effectively offset the impact of the beam drift effect in a dynamic environment and ensure good effective reachability rate EAR performance.

[0123] In addition, it can be observed that when μ is small enough, that is, the environment changes slowly (for example, when μ < 0.08), the EAR performance achieved by MRC4 is the highest among the four subcodebooks. The reason is that in this case, the beam drift effect is not significant, and since the array gain of MRC4 with the narrowest beam width codeword is the highest, the EAR performance of the BWS algorithm is almost the same as that of MRC4. At the same time, as the rate of environmental change increases, the beam drift effect becomes more and more obvious, and the EAR performance of MRC4 deteriorates rapidly and becomes worse than that of MRC3, indicating that narrow beams are more susceptible to beam drift effects. The EAR performance achieved by MRC1 is the worst, because although the wide beam is robust to the beam drift effect, its array gain is relatively low, which in turn reduces the EAR performance. In contrast, our algorithm can achieve a good balance between achieving high array gain and mitigating beam drift effects, and perceive and adapt to the changing environment in real time.

[0124] In summary, the intelligent beamwidth optimization (BWS) algorithm can achieve a compromise and balance between achieving high array gain and suppressing beam drift effects, and can perceive and adapt to the changing environment in real time to improve system throughput.

Claims

1. An intelligent beamwidth optimization method based on reinforcement learning, characterized in that: The following steps are included: Step 1. Model the beam width optimization problem as a Markov decision process (MDP). Use the Actor-Critic algorithm based on a combination of policy learning and value learning. Define codebooks of different resolutions, i.e., beam widths, as action spaces. The policy network and value network are composed of fully connected networks. The policy network is responsible for selecting the best action at each moment. The value network is responsible for evaluating the choices fed back to the policy network based on the objective environment. Through continuous interaction with the environment, it perceives the dynamic characteristics of the environment and selects the best codebook codeword for data transmission. Step 2. Initialization: Establish and initialize the policy network and value network, multi-resolution DFT codebook, data transmission initial codebook, time unit counting, channel environment construction and network update threshold, and open up storage space; Step 3. At the beginning of each decision time slot, the current state is constructed based on the acquired information and the state design criteria, and the best action, i.e., the optimal beam width, is determined through the strategy network; Step 4. Combining the best action, i.e., the best beam width, with the best beam direction obtained by beam alignment, selecting the best codeword in the selected codebook and performing data transmission; Step 5. Calculate the reward information based on the information in the data transmission process, compensate the reward, and determine whether the network update threshold is met. If so, execute step 6. If not, this loop will directly jump back to step 3 to start the next loop; Step 6. Update the policy network and value network parameters based on the reward information; The step 1 first characterizes the beam width optimization problem as an MDP, and effectively and reasonably designs the MDP model parameters including state design, action space, and reward design, and adaptively adjusts the beam width for data transmission according to the severity of the beam drift effect, so as to fully balance the relationship between the beamforming gain and the effective time of data transmission; If the time unit of the beam training process to determine the optimal beam direction is a time slot, and the time scale of beam width selection is defined as a time unit, and each time unit contains M time slots, then the MDP parameters are defined as follows: Step 1.

1. State design: In M time slots, that is, in one time unit, obtain M sets of beam training experience (S, A, R, S t+1 ), so M beam training rewards can be obtained, namely R={R t ,R t+1 ,...,R i ,...,R t+M-2 ,R t+M-1 } (1) Where R i represents the reward of the i-th time slot in this time unit, that is, the effective achievable rate obtained by beam training in this time slot, and then the indirect result set obtained by calculating the R sequence is used as the state, that is, State={R av ,R cv ,R kur } (2) The meaning of each parameter is as follows: ·R av represents the mean of the R sequence, which represents the average size of the R sequence; ·R cv Represents the coefficient of variation of the R series; it is calculated as follows: first calculate the standard deviation σ of the R series The coefficient of variation is a normalized measure of the dispersion of a probability distribution, which is defined as the square root of the standard deviation σ and the mean R. av The ratio is ·R kur The kurtosis represents the kurtosis of the R sequence. The kurtosis characterizes the characteristic number of the peak height of the probability density distribution curve at the mean value. It is a statistic that describes the steepness of the distribution of all values ​​in the population. The purpose of introducing kurtosis is to add more additional information for the system to learn. The kurtosis R kur The calculation is as follows: Step 1.

2. Action design: Define actions as codebooks with different beam widths, i.e., different resolutions. The multi-resolution codebook is defined as The corresponding action space is defined as Step 1.

3. Reward design: The reward is defined as the sum rate of M time slots in each time unit, that is, Among them, R u,i represents the effective transmission rate corresponding to the i-th time slot in the u-th time unit; The initialization method of step 2 is as follows: Step 2.

1. Construct a value network V and a policy network π consisting of fully connected layers, and initialize the value network parameters ω and the policy network parameters θ; Step 2.

2. Construct multi-resolution DFT codebook And randomly select the initial codebook for data transmission; Step 2.

3. Initialize the time unit count u = 1, and initialize the network update threshold l based on experience max ; Step 2.

4. Open up storage space. For time unit u, the storage space stores the best action selected for this time unit. and the reward information generated by the M time slot beam training in this time unit and the state information obtained therefrom; The step 3 performs the following steps in sequence in the time unit u: Step 3.

1. Obtain the reward sequence R obtained in M ​​time slots within this time unit according to formula (1); Step 3.

2. Calculate the state information s of this time unit according to formulas (4), (5), and (6): u ; Step 3.

3. Obtain a set of probability distributions for all actions through the policy network π, for the action space have That is, in state s, the probability of the policy network π(θ) selecting all actions is added to 1, and the best action is selected from it And select the corresponding data transmission codebook The step 4 performs the following steps in sequence in the time unit u: Step 4.

1. Assume that the beam training process to find the best beam direction uses a codebook Obtain the best beam for this time unit based on the beam alignment result Corresponding beam center angle Step 4.

2. Select the best codeword for data transmission, assuming in represents the i-th codeword, Indicates the number of codewords in the codebook. Medium Code The corresponding beam center angle is The best codeword for data transmission for: in Represents natural numbers from 1 to In the specific implementation process, the index relationship between codebooks with different resolutions has specific objective rules, so a direct mapping between indexes of different codebooks can be constructed. In order to directly calculate the optimal transmission codeword through the mapping function The step 5 performs the following steps in sequence in the time unit u: Step 5.1) Accept the external environment reward and calculate the reward R of this time unit according to formula (1) u Due to the deviation between the codebook used for beam training and the codebook used for data transmission, the reward needs to be compensated as follows. The default codebook used for beam training is The corrected R u ' represents the following: Where S S express The number of code words, express The number of code words; Step 5.2) Determine whether the network update threshold is met and design the normalized criticism loss To characterize whether the network needs to be continuously updated, The calculation is as follows: Among them, ω represents the value network parameter; U l is the observation window. When , the neural network is trained; The step 6 performs the following steps in sequence in the time unit u: Step 6.

1. Perform the best action in time unit u Select the corresponding codeword for data transmission and obtain compensation reward R u ′, the state s of the next time unit u+1 can be obtained according to formula (2) u+1 , and obtain the best action for the next time unit based on the policy network Step 6.

2. Set the current time unit state s u and action a u Input the value network to get q u =q(s u ,a u ;ω u ), the next time unit state s u+1 and predict actions Input value network and And calculate the TD-error value δ u =q u -(R u ′+γ·q u+1 ); Step 6.

3. Deriving the value network yields Next, we use the time difference to update the value network as shown in formula ω u+1 =ω u -α·δ u ·d ω,u ; Step 6.

4. Deriving the policy network yields Next, we use gradient ascent to update the policy network parameters as θ u+1 =θ u +β·q u ·d θ,u , where q u You can use δ u replace; Step 6.

5. Update the counter u←u+1.

Images