Joint beam forming and power distribution method based on deep reinforcement learning

Abstract

The invention discloses a joint beam forming and power distribution method based on deep reinforcement learning. The invention provides a hybrid near-end strategy optimization algorithm under a multi-task hybrid action space architecture in order to coordinate multi-dimensional transmission resources among virtual reality users in multiple time slots aiming at the problems of joint beam forming and power allocation in a scene of a multi-user immersive communication service. Specifically, a transmission rate demand of a current video frame to be transmitted is used as state input, and a beam selection network in a discrete action space and a power distribution network in a continuous action space are designed. The two networks make a collaborative decision and carry out joint training based on a public evaluation network. The joint beam forming and power distribution method based on deep reinforcement learning provided by the invention ensures that users in an immersive communication scene can experience real-time interaction services with extremely high reliability and extremely low time delay.

Description

Technical Field

[0001] The present invention relates to the field of wireless communications, and in particular to a joint beamforming and power allocation method based on deep reinforcement learning. Background Art

[0002] The problem of intelligent beam steering in downlink transmission of immersive communication services has received widespread attention in recent years. In order to ensure that users can experience extremely reliable and low-latency real-time interactive services, fast and accurate beam steering technology is particularly important in immersive communications. However, when perfect channel state information cannot be obtained, relying on beam scanning to search for service beams for all users before each transmission will bring extremely high complexity. In the case of time-varying user needs, real-time decision-making on the beamforming scheme in each time slot has also become difficult. In addition, since the beamforming vectors in the beam codebook are normalized, the selection of beam codewords only determines the direction of the beam. In order to achieve more sophisticated resource management, beam steering often needs to be combined with power allocation technology to meet the diverse and time-varying transmission needs of users. A method based on deep reinforcement learning can be used to transform the specific optimization problem into a Markov decision process, and its powerful exploration ability can be used to optimize and design beamforming and power allocation schemes. However, faced with the problem of huge codebook space and difficulty in converging the learning network under the multi-task action network architecture, it is necessary to design some strategies to improve the efficiency of action space exploration and reduce the search space so that the reinforcement learning network can converge. Therefore, how to design a joint beamforming and power allocation method based on deep reinforcement learning has become a key issue. Summary of the invention

[0003] The purpose of the present invention is to address the deficiencies of the prior art and propose a joint beamforming and power allocation method based on deep reinforcement learning.

[0004] The object of the present invention is to achieve the following technical solution: a joint beamforming and power allocation method based on deep reinforcement learning, the method comprising:

[0005] S1. Initialize users and channels, perform beam scanning, and select candidate beam sets;

[0006] S2, activate users according to the arrival time of video frames, and establish a hybrid proximal strategy optimization network of the base station-side agent, including a multi-task action network and an evaluation network.

[0007] The multi-task action network includes a beam selection network and a power allocation network, which generates beam selection strategies and power allocation strategies for different users based on the state of the user's video frame to be transmitted and the candidate beam set;

[0008] The evaluation network obtains a reward value based on the transmission rate and transmission utility fed back by the user after the beam selection strategy and the power allocation strategy are adopted;

[0009] S3. Train the hybrid proximal strategy optimization network: Use the generated beam selection and power allocation scheme to serve active users in each time slot, obtain and store the experience of each time slot based on the state and network output results, and update the multi-task action network and evaluation network when the stored experience is greater than the batch training size until the network converges.

[0010] S4. Use the trained network to generate strategies for the state of the data to be transmitted to achieve real-time transmission.

[0011] Furthermore, the S1 specifically includes: initializing users and user channels, performing beam scanning, and each user selects the indexes of B beams with the highest signal strengths as a candidate beam set according to the received signal strengths.

[0012] Furthermore, in the hybrid proximal strategy optimization network, the generation strategy specifically includes:

[0013] In the tth time slot, use the remaining data to be transmitted q of the kth active user k，t Divided by the residual transmission delay τ k，t , the expected average transmission rate R of the user in this time slot can be calculated k,t =q k,t / τ k，t ; Order s t =R t , the rate is used as the state input into the hybrid proximal strategy optimization network to obtain the beam selection strategy in the current time slot and power allocation strategy Among them, the size of the action space selected by the initial scanning and dimensionality reduction is B K , K-bit B-ary coding is used to encode the indexes of all user service beams. In the tth time slot, the output action of the beam selection network is

[0014] Furthermore, in the hybrid proximal strategy optimization network, the specific structure is:

[0015] The inputs of the beam selection network, power allocation network and evaluation network are all states s t ,The beam selection network contains five hidden layers. The network outputs all the probabilities that can realize the beam selection strategy, and samples them after the Softmax function to obtain the final beam selection scheme;

[0016] The power allocation network contains three hidden layers, and the network output is K continuous actions sampled from independent Gaussian distributions; the mean μ and variance σ of all distributions are learned through neural networks, and the output of the power allocation network is normalized by the Softmax function to achieve the normalization of the power of all users.

[0017] The evaluation network directly adopts a fully connected network structure with three hidden layers, and the input state s t , outputs the value prediction for the mixed action.

[0018] Furthermore, the transmission rate and transmission utility calculation method is:

[0019] Use beam selection and power allocation scheme to serve active users and obtain user transmission rate c k =W log(1+SINR k ), where W is the channel bandwidth, SINR k is the signal-to-interference-noise ratio, and the utility function of the video frame of user k in the tth time slot is

[0020]

[0021] Where T is the unit time slot length, Q k The data size of the video frame to be transmitted.

[0022] Furthermore, the experience of each time slot is obtained and stored according to the state and network output results, and the experience specifically includes the current state, action, next moment state and reward pair; wherein the reward value r t It is calculated through the transmission rate and transmission utility fed back by users.

[0023]

[0024] Where β is the weight factor, T step is the time slot when the task is completed, It is the time slot at the end of the round. One round is the transmission process of a certain video frame. When the video frame transmission is completed or the remaining transmission delay is 0, the round ends and a new video frame transmission process starts.

[0025] Furthermore, in the updating of the multi-task action network and the evaluation network,

[0026] The loss function of the value network is:

[0027]

[0028] Among them, γ is the discount factor, V φ To evaluate the network output, the advantage function At is calculated as

[0029]

[0030] Where λ is the parameter of the generalized advantage function;

[0031] Power Distribution Network The objective function is designed as

[0032]

[0033] in, Represents the ratio of the new and old strategies; Indicates that in state s t Next, by the parameter θ p The probability distribution of actions generated by a certain policy, is the power of the kth user output by the power allocation network in time slot t; clip is the clipping function, Restricted to [1-∈, 1+∈], ∈ is a hyperparameter; is the policy entropy, α is the policy entropy coefficient; discrete policy network for beam selection The objective function is designed as:

[0034]

[0035] The superscripts b and p are used to distinguish the beam selection network from the power allocation network.

[0036] On the other hand, the present specification also provides a joint beamforming and power allocation device based on deep reinforcement learning, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the joint beamforming and power allocation method based on deep reinforcement learning.

[0037] On the other hand, the present specification also provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the method of joint beamforming and power allocation based on deep reinforcement learning is implemented.

[0038] On the other hand, the present specification also provides a computer program product, including a computer program, which, when executed by a processor, implements the joint beamforming and power allocation method based on deep reinforcement learning.

[0039] Beneficial effects of the present invention:

[0040] The joint beamforming and power allocation method based on deep reinforcement learning proposed in this invention can make decisions on the transmission scheme in real time according to the user's current transmission status, thereby ensuring that users in immersive communication scenarios can experience extremely high reliability and extremely low latency real-time interactive services. In addition, in order to further improve the convergence speed of the multi-task network, this method reduces the dimension of the action space based on the scene prior, thereby improving the training efficiency of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a simulation scene graph provided by an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of a joint beamforming and power allocation architecture provided by an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of the architecture of the action network and the evaluation network provided by an embodiment of the present invention;

[0044] Figure 4 is the relationship between the system sum rate and the number of VR users when comparing the proposed joint beamforming and power allocation method with the baseline method;

[0045] Figure 5 is the relationship between the video frame reception frame rate and the number of VR users when comparing the proposed joint beamforming and power allocation method with the baseline method;

[0046] Figure 6 It is a schematic diagram of a joint beamforming and power allocation device based on deep reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0048] like Figure 1 As shown, the present invention provides a joint beamforming and power allocation method based on deep reinforcement learning.

[0049] In this embodiment, a DeepMIMO channel dataset constructed based on ray tracing is used to comprehensively evaluate the proposed joint beam selection and power allocation method. Figure 1 As shown in the figure, consider an outdoor scene with three streets and two intersections. The base station located at the upper right corner of the left intersection in the figure is responsible for signal transmission. The base station is equipped with multiple antennas to support multiple single-antenna VR users randomly distributed on the main streets horizontally. In addition, the base station is equipped with an edge server to provide computing resources for the training of the agent.

[0050] This embodiment provides a joint beamforming and power allocation method, which includes the following steps:

[0051] S1: Initialize K users and user channels and perform beam scanning. Each user selects the index of the B beams with the highest signal strength as its candidate beam set based on the received signal strength;

[0052] S2: In the training phase, users are activated according to the arrival time of video frames, a hybrid proximal strategy optimization network HPPO of the base station-side agent is established, and the beam selection action network, power allocation action network and evaluation network are initialized;

[0053] Step S2 specifically includes:

[0054] Generate video data streams and activate users based on the arrival time of video frames. The arrival time of the first packet of each user's video frame and the size of the video frame follow the Gaussian truncated distribution. Initialize the hybrid proximal strategy optimization network of the base station-side agent, such as Figure 3 As shown in Figure 1, the architecture consists of two parts: a multi-task action network and an evaluation network. The hidden layers of all networks are constructed based on linear layers and Tanh activation functions. The multi-task action network includes a beam selection action network and a power allocation network. The input of all networks is the state s t The beam selection action network contains five hidden layers. The network outputs the probabilities of all possible beam selection strategies and obtains the final beam selection scheme after sampling through the Softmax function. The power allocation network contains three hidden layers. The network outputs continuous actions sampled from K independent Gaussian distributions. Among them, the mean μ and variance σ of all distributions are learned through neural network. In order to meet the total power constraint, the output of the power allocation network will pass through the Softmax function to normalize the power of all users. The evaluation network directly adopts a fully connected network structure with three hidden layers, inputs the state, and outputs the value prediction of the mixed action.

[0055] S3: The base station obtains the status of all user video frames to be transmitted and inputs them into the two action networks in S2 to obtain the strategies of beam selection and power allocation for the current transmission time slot respectively;

[0056] The status and strategy in step S3 are specifically:

[0057] In the tth time slot, use the remaining data to be transmitted q of the kth active user k，t Divided by the residual transmission delay τ k，t , the expected average transmission rate R of the user in this time slot can be calculated k,t =q k，t / τ k，tThe expected average transmission rate can effectively reflect the current transmission status of the user. Therefore, let s t =R t , R t is a vector of the average transmission rates of all users in time slot t, i.e., the average transmission rates of all R k，t The rate is used as the state input into the hybrid proximal strategy optimization network to obtain the beam selection strategy in the current time slot. and power allocation strategy Among them, through the initial scan, the size of the beam selection action space after dimensionality reduction is B K In order to more efficiently represent the beam selection strategies of all users, K-bit B-ary coding is further used to encode the indexes of all user service beams. In the tth time slot, the output action of the beam selection network is

[0058] S4: Based on the beam selection and power allocation strategy obtained in step S3, the VR user receives the video frame to achieve real-time interaction. The user side calculates the current transmission rate and transmission utility of each user and feeds back to the base station side.

[0059] The transmission rate and transmission utility calculation method described in step S4 is:

[0060] Use beam selection and power allocation scheme to serve active users and obtain user transmission rate c k =Wlog(1+SINR k ), where W is the channel bandwidth, SINR k is the signal to interference noise ratio. The utility function of the video frame of user k in the tth time slot is defined as

[0061]

[0062] Where T is the unit time slot length, Q k The data size of the video frame to be transmitted.

[0063] S5: The base station agent reacquires the state of the user's video frame to be transmitted, and calculates the reward value through the transmission rate and transmission utility fed back by the user. The agent's multi-task action network and evaluation network are updated based on the collected experience (current state-action-next state-reward pair);

[0064] The specific steps of network updating in step S5 are:

[0065] According to the transmission rate and utility function fed back by the user, the reward value under the current transmission state is calculated

[0066]

[0067] Where β is the weight factor, T step is the time slot when the task is completed, is the time slot at the end of the round (a round is the transmission process of a certain video frame. When the video frame transmission is completed or the remaining transmission delay is 0, the round ends and a new video frame transmission process begins). The above formula shows that before the current round ends, the reward function is calculated as the growth rate of the sum of user transmission rates relative to the sum of the historical highest transmission rates. The size of the growth rate directly reflects the performance of the agent in improving transmission efficiency. Once the current round ends, the reward function will add additional items related to transmission utility and completion time to comprehensively evaluate the transmission of the frame. At this time, the time slot T when the transmission task is completed step The smaller it is, the higher the reward value.

[0068] Get r t After that, update R k，t+1 ,s t+1 =R t+1 , collect the experience of time slot t And store it in the experience replay pool. When the experience stored in the replay pool is greater than the batch training size, the gradient descent algorithm is used to update the multi-task action network and the evaluation network. In particular, the loss function of the value network is calculated as

[0069]

[0070] Among them, γ is the discount factor, V φ To evaluate the network output. The network also uses the advantage function At to further evaluate the quality of the action. The advantage function is calculated as

[0071]

[0072] Where λ is the parameter of the generalized advantage function.

[0073] In order to enhance the stability of training, the network update introduces the policy gradient clipping mechanism to control the distance between the new and old policy distributions, thereby reusing the sample data obtained by the sampling strategy and improving the sample efficiency. In addition, in order to enhance the exploration ability of the action space, this method also incorporates the idea of ​​maximizing the policy entropy. Therefore, the continuous action network for power allocation is realized. The objective function is designed as

[0074]

[0075] in, Represents the ratio of the new and old strategies. Indicates that in state s t Next, by the parameter θ p The probability distribution of actions generated by a certain policy, is the power of the kth user output by the power allocation network in time slot t. clip is the clipping function. Restricted to [1-∈, 1+∈], ∈ is a hyperparameter. is the policy entropy. The larger the value, the stronger the randomness of the policy selection action. α is the policy entropy coefficient. Similarly, the discrete policy network that realizes beam selection The objective function is designed as

[0076]

[0077] Based on the principle of maximizing the above objective function, the two action networks can be updated separately.

[0078] S6: Continuously execute the process from step S2 to step S5 until the base station network converges;

[0079] S7: The testing phase is the actual operation phase. The base station-side agent directly uses the trained network to predict the beam selection and power allocation scheme based on the status of all users' data to be transmitted, thereby achieving real-time transmission in immersive communication scenarios.

[0080] Through computer simulation, we can see that: Figure 4 As shown, the combined beamforming and power allocation method of the present invention has a significant improvement in system and rate compared with the baseline power on-demand allocation and weighted minimum mean square error WMMSE scheme under the same VR user number condition. Figure 5 It shows that the joint beamforming and power allocation method proposed in the present invention has a significant improvement in the video frame reception frame rate compared to the baseline power on-demand allocation and weighted minimum mean square error WMMSE scheme. These advantages are mainly because the present invention utilizes reinforcement learning methods to strengthen the decision-making and evaluation mechanisms by continuously accumulating historical experience data to obtain the optimal beam selection and power allocation scheme. Therefore, the joint beamforming and power allocation method based on deep reinforcement learning proposed in the present invention has well solved the problem of high-reliability, low-latency real-time interactive services for users in immersive communications.

[0081] Corresponding to the aforementioned embodiment of a joint beamforming and power allocation method based on deep reinforcement learning, the present invention also provides an embodiment of a joint beamforming and power allocation device based on deep reinforcement learning.

[0082] See also Figure 6 An embodiment of the present invention provides a joint beamforming and power allocation device based on deep reinforcement learning, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a joint beamforming and power allocation method based on deep reinforcement learning in the above embodiment.

[0083] An embodiment of a joint beamforming and power allocation device based on deep reinforcement learning provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From a hardware perspective, if Figure 6 As shown, it is a hardware structure diagram of a joint beamforming and power allocation device based on deep reinforcement learning provided by the present invention, in which any device with data processing capability is located, except Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0084] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0085] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0086] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a joint beamforming and power allocation method based on deep reinforcement learning in the above embodiment is implemented.

[0087] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0088] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the joint beamforming and power allocation method based on deep reinforcement learning.

[0089] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0090] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A joint beamforming and power allocation method based on deep reinforcement learning, characterized in that: The method includes: S1. Initialize users and channels, perform beam scanning, and select candidate beam sets; S2, activate users according to the arrival time of video frames, and establish a hybrid proximal strategy optimization network of the base station-side agent, including a multi-task action network and an evaluation network. The multi-task action network includes a beam selection network and a power allocation network, which generates beam selection strategies and power allocation strategies for different users based on the state of the user's video frame to be transmitted and the candidate beam set; The evaluation network obtains a reward value based on the transmission rate and transmission utility fed back by the user after the beam selection strategy and the power allocation strategy are adopted; S3. Train the hybrid proximal strategy optimization network: Use the generated beam selection and power allocation scheme to serve active users in each time slot, obtain and store the experience of each time slot based on the state and network output results, and update the multi-task action network and evaluation network when the stored experience is greater than the batch training size until the network converges. S4. Use the trained network to generate strategies for the state of the data to be transmitted to achieve real-time transmission.

2. According to claim 1, a joint beamforming and power allocation method based on deep reinforcement learning is characterized in that: The S1 specifically includes: initializing users and user channels, performing beam scanning, and each user selects the indexes of B beams with the highest signal strength as a candidate beam set according to the received signal strength.

3. The method for joint beamforming and power allocation based on deep reinforcement learning according to claim 1, characterized in that: In the hybrid proximal strategy optimization network, the generation strategy specifically includes: In the tth time slot, use the remaining data to be transmitted q of the kth active user k,t Divided by the residual transmission delay τ k,t , the expected average transmission rate R of the user in this time slot can be calculated k,t =q k,t / τ k,t ; Order s t =R t , the rate is used as the state input into the hybrid proximal strategy optimization network to obtain the beam selection strategy in the current time slot and power allocation strategy Among them, the size of the action space selected by the initial scanning and dimensionality reduction is B K , K-bit B-ary coding is used to encode the indexes of all user service beams. In the tth time slot, the output action of the beam selection network is 4. The method for joint beamforming and power allocation based on deep reinforcement learning according to claim 1, characterized in that: In the hybrid proximal strategy optimization network, the specific structure is: The inputs of the beam selection network, power allocation network and evaluation network are all states s t ,The beam selection network contains five hidden layers. The network outputs all the probabilities that can realize the beam selection strategy, and samples them after the Softmax function to obtain the final beam selection scheme; The power allocation network contains three hidden layers, and the network output is K continuous actions sampled from independent Gaussian distributions. The mean μ and variance σ of all distributions are learned through neural networks. The output of the power allocation network is normalized by the Softmax function for all users. The evaluation network directly adopts a fully connected network structure with three hidden layers, and the input state s t , outputs the value prediction for the mixed action.

5. The method of joint beamforming and power allocation based on deep reinforcement learning according to claim 1, characterized in that: The transmission rate and transmission utility calculation method are as follows: Use beam selection and power allocation scheme to serve active users and obtain user transmission rate c k =Wlog(1+SINR k ), where W is the channel bandwidth, SINR k is the signal-to-interference-noise ratio, and the utility function of the video frame of user k in the tth time slot is Where T is the unit time slot length, Q k The data size of the video frame to be transmitted.

6. The method of joint beamforming and power allocation based on deep reinforcement learning according to claim 5, characterized in that: The experience of each time slot is obtained and stored according to the state and network output results. The experience specifically includes the current state, action, next moment state and reward pair; the reward value r t It is calculated through the transmission rate and transmission utility fed back by users. Where β is the weight factor, T step is the time slot when the task is completed, It is the time slot at the end of the round. One round is the transmission process of a certain video frame. When the video frame transmission is completed or the remaining transmission delay is 0, the round ends and a new video frame transmission process starts.

7. The method of joint beamforming and power allocation based on deep reinforcement learning according to claim 1, characterized in that: In the updating of the multi-task action network and the evaluation network, The loss function of the value network is: Among them, γ is the discount factor, V φ To evaluate the network output, the advantage function A t Calculated as δ t =r t +γV φ (s t+1 )-V φ (s t ) Where λ is the parameter of the generalized advantage function; Power Distribution Network The objective function is designed as in, Represents the ratio of the new and old strategies; Indicates that in state s t Next, by the parameter θ p The probability distribution of actions generated by a certain policy, is the power of the kth user output by the power allocation network in time slot t; clip is the clipping function, Restricted to [1-∈, 1+∈], ∈ is a hyperparameter; is the policy entropy, α is the policy entropy coefficient; discrete policy network for beam selection The objective function is designed as: The superscripts b and p are used to distinguish the beam selection network from the power allocation network.

8. A joint beamforming and power allocation device based on deep reinforcement learning, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a joint beamforming and power allocation method based on deep reinforcement learning is implemented as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a joint beamforming and power allocation method based on deep reinforcement learning as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a joint beamforming and power allocation method based on deep reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images