A beamforming method and related device in multi-TTI transmission under delay constraint

Abstract

The present invention discloses a beamforming method and related devices in multi-TTI transmission under delay constraints. It uses a reinforcement learning method to solve the framework of the multi-TTI transmission problem with delay constraints in MU-MIMO scenarios. Under the premise of considering the delay constraint of each user request, the beamforming vector of each user is constructed by the output of the neural network at the beginning of each TTI, and finally the goal of maximizing the sum rate of user requests successfully transmitted after multiple TTIs is achieved. The framework adopts the Proximal Policy Optimization (PPO) algorithm. The PPO algorithm establishes a multivariate normal distribution by outputting the mean of the multivariate normal distribution of each dimension of the action, and then samples from the distribution to obtain the final action to be executed, completing the processing of the continuous action space. The resulting system can better solve the multi-TTI transmission problem with delay constraints in MU-MIMO scenarios, and has good generalization for different user transmission data volumes and delay constraints.

Description

Technical Field

[0001] The present invention belongs to the field of communications and relates to intelligent beamforming technology in wireless communications, in particular to a beamforming method and related devices in multi-TTI transmission under delay constraints. Background Art

[0002] As wireless communication technology develops, a myriad of application scenarios and communication demands continue to pose new challenges to communication systems. In MU-MIMO (Multi-User Multiple-Input Multiple-Output) scenarios, beamforming technology, as a key technology for mitigating interference between users, improving data transmission and speed, and increasing resource utilization, remains a key consideration in the design of various communication systems.

[0003] In practical applications, data transmission is often subject to latency constraints. A certain amount of data must be transmitted within this latency limit; otherwise, data integrity will be compromised when the latency limit is reached. Therefore, multi-TTI (Transmission Time Interval) transmission with latency constraints in MU-MIMO scenarios requires beamforming at the beginning of each TTI to construct a beamforming vector for each user, ultimately maximizing the sum of user requests successfully transmitted over multiple TTIs.

[0004] The forward and backward coupling of beamforming for multi-TTI transmission under delay and data integrity constraints requires consideration of the impact of multiple TTI transmissions to maximize the sum of successful transmission rates. Existing traditional algorithms, such as ZF (zero forcing) and MRT (maximum ratio transmission), as well as some iterative algorithms, primarily perform beamforming for the current TTI transmission. In the paper "MU-MIMO Precoding Design in the Presence of Delay-Constrained Users," the authors consider multi-user scenarios based on delay tolerance and delay constraints. They iteratively solve for precoding vectors to achieve the optimization goal of maximizing spectral efficiency. However, due to the complex computations and iterations involved in the solution, the latency performance is limited and cannot meet the extremely low latency requirements of users in real-world scenarios. The paper "Resource Allocation for Multi-User Downlink MISO OFDMA-URLLC Systems" optimizes beamforming vectors to maximize the sum rate of multi-user transmission under multi-user delay constraints, minimum transmission data size constraints, and total power constraints. However, this paper focuses on beamforming design for transmission within a single TTI and fails to address the problem from the perspective of multi-TTI transmission scenarios, making it unsuitable for multi-TTI transmission scenarios. In the paper "Optimal multiuser transmit beamforming: a difficult problem with a simple solution structure [lecture notes]," the authors transform the rate maximization problem for two receivers into a problem of selecting an MRT and a ZF, achieving some results. Subsequently, in the paper "Machine Learning-Based Beamforming in K-User MISO Interference Channels," the authors extend this MRT and ZF selection scheme to multi-receiver scenarios, providing a solution for beamforming design. However, the MRT and ZF algorithms require only simple calculations based on the channel or channel covariance matrix, resulting in correspondingly lower performance. The sum rate obtained using these two special beamforming vectors differs significantly from the maximization problem. The above research results cannot effectively solve the beamforming problem in multi-TTI transmission under delay constraints. Therefore, a new and effective algorithm is needed that can not only solve the problem from the perspective of multi-TTI transmission but also meet the timeliness requirements of the algorithm for low-latency transmission.

[0005] Machine learning, a rapidly developing discipline in recent years, has been applied to numerous areas of production and life. Researchers have also conducted relevant research on beamforming design. The papers "Optimal multiuser transmit beamforming: a difficult problem with a simple solution structure [lecturenotes]" and "Machine Learning-Based Beamforming in K-User MISO Interference Channels" apply deep neural networks to predict MRT and ZF selection schemes. The final prediction results are limited by the performance of the MRT and ZF algorithms. In the paper "Learning to optimize: Training deep neural networks for interference management," the authors analyze three classic beamforming optimization algorithms: neural network methods for signal-to-interference-noise ratio (SIR) balancing, power minimization, and rate maximization. They use convolutional neural networks to predict parameters, but the modeling does not consider multi-TTI transmission scenarios. The paper "A deep learning framework for optimization of MISO downlink beamforming" also studied single-cell scenarios and constructed a BPNet, but similarly did not consider multi-TTI transmission scenarios. The paper "Fast beamforming design via deep learning" converted latency requirements into average rate constraints. Based on a user scheduling algorithm using distributed multi-agent Deep Q-learning, it used a trained model to solve the rate-maximizing beamforming problem in multi-cell systems. However, this approach is not applicable when multiple users have different latency requirements.The papers "Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications" and "Beamforming Design for Multiuser uRLLC With Finite Blocklength Transmission" both study the beamforming design problem for downlink multi-user uRLLC systems with the goal of maximizing multi-user reachability and rate. They convert delay constraints into instantaneous rate constraints. The paper "Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications" uses the DRL algorithm to optimize the network energy efficiency and QoS requirements of the URLLC system, but to make the optimization problem easier to handle, it also converts delay constraints into data rate constraints. These approaches are based on the premise that multiple users have the same delay constraints. However, in actual application scenarios, each user may have different service types and different delay constraint requirements. Therefore, converting multi-user delay constraints into instantaneous rate constraints is not accurate enough and can affect system performance. Reinforcement learning, a type of machine learning algorithm, offers unique advantages in sequential decision-making. However, no research has yet applied reinforcement learning to address beamforming in multi-TTI transmissions under latency constraints. When using neural networks to approximate beamforming algorithms, the accuracy of the algorithm is closely related to the network structure. Directly using the beamforming vector as the neural network output can lead to significant errors due to the large output dimensionality, and can also affect the convergence of neural network training. Summary of the Invention

[0006] The purpose of the present invention is to solve the problem of multi-TTI transmission with delay constraint in MU-MIMO scenario in the prior art, and to provide a beamforming method and related apparatus in multi-TTI transmission with delay constraint.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a beamforming method in multi-TTI transmission under delay constraint, comprising the following steps:

[0009] Construct a maximum sum rate model for multi-TTI transmission under latency constraints in MU-MIMO scenarios;

[0010] Perform dimensionality reduction on the maximization and rate model of multi-TTI transmission;

[0011] Combining the multi-TTI transmission maximization and rate model with the reinforcement learning framework;

[0012] The combined reinforcement learning framework is trained using the PPO algorithm.

[0013] In a second aspect, the present invention provides a beamforming system for multi-TTI transmission under delay constraints, including:

[0014] A model building module is used to build a maximum sum rate model for multi-TTI transmission under latency constraints in MU-MIMO scenarios;

[0015] A dimensionality reduction processing module, used for performing dimensionality reduction processing on the maximization and rate model of multi-TTI transmission;

[0016] A combined computation module is used to combine the multi-TTI transmission maximization and rate model with the reinforcement learning framework;

[0017] The model training module is used to train the combined reinforcement learning framework using the PPO algorithm.

[0018] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] Under the premise of considering the delay constraints of each user's request, the present invention constructs the beamforming vector for each user from the output of the neural network at the beginning of each TTI, ultimately achieving the goal of maximizing the sum rate of user requests successfully transmitted after multiple TTIs. In the reinforcement learning model, the base station that completes beamforming is modeled as an intelligent agent, and the current channel, the amount of remaining data for each user, and the remaining TTIs within each user's arrival delay constraints are modeled as states. When making decisions, the intelligent agent considers not only the channel conditions of the current TTI but also the amount of remaining transmission data and delay constraints of each user. This ensures that beamforming decisions for multi-TTI transmissions are interconnected, thereby solving the problem of front-to-back coupling in multi-TTI transmissions.

[0022] The framework of the present invention further adopts the Proximal Policy Optimization (PPO) algorithm. The PPO algorithm establishes a multivariate normal distribution by outputting the mean of the multivariate normal distribution of each dimension of the action, and then samples from the distribution to obtain the final action to be executed, completing the processing of the continuous action space.

[0023] Furthermore, the network training process of the present invention takes a multi-TTI problem from the first TTI to the completion of transmission to all users or the reaching of the delay constraint by all users as a trajectory, and at the beginning of the trajectory, the transmission data volume Q of each user is generated according to the uniform distribution. k and delay constraint T k ,A complex Gaussian channel is randomly generated at the beginning of each TTI. After a large amount of training, the mapping from state to action is completed. The final system can ,better solve the multi-TTI transmission problem with delay constraints in MU-MIMO scenarios, and ,has good generalization for different user transmission data amounts and ,delay constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 Schematic diagram of the process of the present invention.

[0026] Figure 2 Schematic diagram of the system of the present invention.

[0027] Figure 3 This invention provides a reinforcement learning framework model for solving the beamforming problem in multi-TTI transmission under delay constraints in MU-MIMO scenarios.

[0028] Figure 4 This is the sum of the transmission rates of the present invention and the ZF algorithm in 50 transmission scenarios when the amount of data transmitted is large.

[0029] Figure 5 This is the amount of data successfully transmitted by the present invention and the ZF algorithm in 50 transmission scenarios when the amount of data transmitted is large.

[0030] Figure 6 This is the sum of the transmission rates of the present invention and the ZF algorithm in 50 transmission scenarios when the amount of data transmitted is relatively small.

[0031] Figure 7This is the amount of data successfully transmitted by the present invention and the ZF algorithm in 50 transmission scenarios when the amount of transmitted data is small. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0034] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not require further definition or explanation in subsequent drawings.

[0035] In the description of the embodiments of the present invention, it should be noted that if the terms "upper," "lower," "horizontal," "inner," etc. appear, the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the inventive product is typically placed when in use. These terms are merely for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first," "second," etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0036] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0037] In the description of the embodiments of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0038] The present invention is described in further detail below with reference to the accompanying drawings:

[0039] See also Figure 1 The embodiment of the present invention discloses a beamforming method in multi-TTI transmission under delay constraint, comprising the following steps:

[0040] S1 builds a maximization and rate model for multi-TTI transmission under delay constraints in MU-MIMO scenarios.

[0041] The maximum sum rate model for multi-TTI transmission is as follows:

[0042]

[0043]

[0044]

[0045] Among them, W k,t is the beamforming matrix of user k in the t-th TTI, K is the number of users, R k is the transmission rate of user k, Q k is the amount of data that user k needs to transmit, τ k is the delay constraint of user k, d k,t is the amount of data transmitted by user k at the tth TTI, W k,t The conjugate transpose of p t is power limitation;

[0046] The amount of data d transmitted by user k at the tth TTI k,t as follows:

[0047] d k,t =-log(det(MMSE k,t )) (2)

[0048]

[0049] Among them, MMSE k,t is the MMSE matrix corresponding to user k at the tth TTI, I is the identity matrix, W k,t is the beamforming vector corresponding to user k in the tth TTI, W k,t The conjugate transpose of H k,t is the channel matrix corresponding to user k in the tth TTI, H k,t The conjugate transpose of W i,t is the beamforming matrix of user i in the t-th TTI, W i,t The conjugate transpose of .

[0050] S2 performs dimensionality reduction on the maximization and rate model of multi-TTI transmission, as follows:

[0051] The virtual uplink channel method is used to transform the beamforming matrix W into the uplink beamforming matrix B and the power real matrices P and U, as follows:

[0052]

[0053] Among them, H i,t is the channel matrix corresponding to user k in the i-th TTI, H i,t The conjugate transpose of is the normalized uplink beamforming matrix of user i at the tth TTI, for The conjugate transpose of U i,t is the power real matrix of i in the tth TTI, is the normalized uplink beamforming matrix of user k in the tth TTI, is the normalized uplink beamforming matrix of user k in the tth TTI, P k,t is the power real matrix of user k in the tth TTI.

[0054] S3 combines the multi-TTI transmission maximization and rate model with the reinforcement learning framework as follows:

[0055] The elements of the reinforcement learning framework: agent, environment, state, action, and reward are combined with the maximization and rate model of multi-TTI transmission as follows:

[0056] The intelligent agent is the base station;

[0057] The environment refers to the channel status and user transmission conditions;

[0058] Status t is the input of the neural network, as follows:

[0059] s t ={H t ,Q_left t ,T t} (5)

[0060] Among them, H t is the channel at the tth timestep, and H t =[H 1,t ,H 2,t ,…,H K,t], Q_left t is the remaining transmission data volume of each user at the tth timestep, and Q_left t =[Q_left 1,t ,Q_left 2,t ,…,Q_left K,t ], T t is the remaining TTI constrained by the arrival delay of each user at the t-th timestep, and T t =[T 1,t ,T 2,t ,…,T K,t ];

[0061] The remaining data volume of each user Q_left t The details are as follows:

[0062]

[0063] Among them, Q k is the total amount of data that user k needs to transmit, d k,i is the amount of data transmitted by user k during the i-th TTI;

[0064] Action a t is the neural network acceptance state s t The output after , the action is modeled as the uplink beamforming matrix B and the power real matrices P and U, as follows:

[0065] a t =[P t ,U t ,B t ] (7)

[0066] Among them, P t 、U t is the power real matrix at the tth timestep, and P t =[P 1,t ,P 2,t ,…,P K,t ], U t =[U 1,t ,U 2,t ,…,U K,t ], B t is the normalized uplink beamforming matrix at the tth timestep, and B t =[B 1,t ,B 2,t ,…,B K,t ];

[0067] Reward t The agent observes the state s t, output action a t After that, the environment gives feedback to the agent; the reward of the current TTI is the sum of the rewards of all users, as follows:

[0068]

[0069]

[0070] Among them, rew k,t is the reward component of user k at the tth timestep, and a is the penalty coefficient.

[0071] S4 uses the PPO algorithm to train the combined reinforcement learning framework as follows:

[0072] Step 1: Initialize the communication system and the relevant parameters of the PPO algorithm, including the power limit p t , the number of transmitting antennas N, the number of receiving antennas M, the number of users K, the learning rate of the PPO algorithm l r , discount factor γ;

[0073] Step 2: Generate an initial state s0 = {H0, Q_left0, T0} and start the learning iteration process. Here, the complex Gaussian channel H is randomly generated, the amount of data Q required to be transmitted by each user and the delay constraint T are generated according to a uniform distribution, and Q_left0 = Q and T0 = τ.

[0074] Step 3: Train the deep neural network. Before each round of training, an initial state s0 = {H0, Q_left0, T0} is generated to start the iterative learning process. Here, a randomly generated 0-1 complex Gaussian channel t is used. The amount of data Q required to be transmitted by each user and the delay constraint T are generated according to a uniform distribution, with Q_left0 = Q and T0 = τ.

[0075] In state s t Next, the neural network outputs action a t The mean of the multivariate normal distribution of , and the multivariate normal distribution is established through the given covariance matrix, and the action a is sampled from the normal distribution t =[P t ,U t ,B t ];

[0076] Step 4: Calculate the beamforming matrix W, and then calculate the amount of data d transmitted by each user in the current TTI t , determine whether each user has completed the transmission and whether the delay constraint is met, and calculate the reward r t ;

[0077] Step 5: Calculate the remaining transmission data volume of each user at this time and the remaining TTI of the arrival delay constraint, together with the randomly generated complex Gaussian channel to form the state s of the next TTI t+1 ;

[0078] Step 6: Update network parameters using the PPO algorithm: The agent receives the state s t , output action a t 、Get rewards t And get the next state s t+1 The whole process is a timestep, and after T timesteps, the data obtained by interacting with the environment in these T timesteps (s t ,a t ,r t ) Calculate the advantage estimate A k :

[0079] A k =-V(s k )+r k +γr k+1 +…+γ T-k+1 r T-1 +γ T-k V(s T ) (10)

[0080] Among them, V(s k ) is the state s received by the Critic network in the Actor-Critic framework of the PPO algorithm k Output after input, k=1,2,…,T, T is the maximum length of the trajectory, r k is the reward at time k, γ is the discount factor, γ T-k+1 is the (T-k+1)th power of γ, T-k is the (Tk)th power of γ, r k+1 is the reward at time k+1, r T-1 is the reward at time T-1, V(s T ) is the state s received by the Critic network T Output after input, s T is the state at time T;

[0081] After obtaining the advantage estimate, the loss function of the Actor network and the Critic network is calculated:

[0082] L actor (θ)=E k [min(p k (θ)A k ,clip(p k (θ),1-∈,1+∈)A k)] (11)

[0083]

[0084] Among them, L actor (θ) is the loss function of the actor network, E[·] is the expectation of the variables in [], and p k (θ) is the correction coefficient in importance sampling, θ is the parameter used to update the actor network using the Adam optimizer, and p θ (a k |s k ) means in state s k Next select action a k The probability of ,∈ is the parameter that controls the upper and lower bounds in the clip function; is the loss function of the critic network, To update the parameters of the critic network using the Adam optimizer, γ i-k is the (ik)th power of γ, r i is the reward at time i;

[0085] The calculated loss function is used to update the network parameters of the Actor network and the Critic network using the Adam optimizer. After updating L times, θ old ←θ,

[0086] Step 7: Repeat steps 2 to 6 until the set number of iterations is reached or the loss function converges, at which point the training process of the reinforcement learning framework ends;

[0087] Step 8: Input the channel matrix H for multiple TTIs, the amount of data each user needs to transmit Q, and the delay limit τ into the reinforcement learning framework. The reinforcement learning framework outputs the beamforming solution for each TTI, ultimately achieving the goal of optimizing multiple TTIs and rates.

[0088] See also Figure 2 The embodiment of the present invention discloses a beamforming system for multi-TTI transmission under delay constraint, including:

[0089] A model building module is used to build a maximum sum rate model for multi-TTI transmission under latency constraints in MU-MIMO scenarios;

[0090] A dimensionality reduction processing module, used for performing dimensionality reduction processing on the maximization and rate model of multi-TTI transmission;

[0091] A combined computation module is used to combine the multi-TTI transmission maximization and rate model with the reinforcement learning framework;

[0092] The model training module is used to train the combined reinforcement learning framework using the PPO algorithm.

[0093] Principle of the present invention:

[0094] like Figure 3 As shown, Figure 3 The reinforcement learning solution framework model for the beamforming problem in multi-TTI transmission under delay constraints in the MU-MIMO scenario of the present invention clarifies the model and framework of the entire system. The specific steps are as follows:

[0095] Step 1: Modeling of multi-TTI transmission and rate maximization under latency constraints in MU-MIMO scenarios.

[0096] Given the amount of data each user needs to transmit and the latency constraints, the multi-TTI transmission and rate maximization problem is as follows:

[0097]

[0098]

[0099]

[0100] Among them, W k,t is the beamforming matrix of user k in the t-th TTI, R k is the transmission rate of user k, Q k is the amount of data that user k needs to transmit, τ k is the delay constraint of user k, d k,t is the amount of data transmitted by user k in the t-th TTI. Constraint (1b) is to ensure the integrity of the data. If user k successfully transmits the required data amount Q within the delay constraint k , then the rate is If there is no complete transmission within the delay constraint, the rate is 0. Constraint (1c) is the total power constraint. The sum of the power of all users at the tth TTI does not exceed p t .

[0101] d k,t The calculation is as follows:

[0102] d k,t =-log(det(MMSE k,t )) (2)

[0103]

[0104]

[0105] Among them, MMSE k,tis the MMSE matrix corresponding to user k in the tth TTI, W k,t is the beamforming vector corresponding to user k in the tth TTI, W k,t The conjugate transpose of H k,t is the channel matrix corresponding to user k in the tth TTI. The channel used in the present invention is a complex Gaussian channel, which is randomly generated at the beginning of each TTI.

[0106] Step 2: The beamforming matrix W is converted into the uplink beamforming matrix B and the power real matrices P and U.

[0107] When using a neural network to approximate a beamforming algorithm, the accuracy of the algorithm is closely related to the network structure. Directly using the beamforming vector as the neural network output can lead to significant errors due to the large output dimension and affect the convergence speed of neural network training. Therefore, the virtual uplink channel method (described in the paper "Fast beamforming design via deep learning" by H. Huang, Y. Peng, J. Yang, W. Xia, and G. Gui) is used to represent the beamforming matrix W as the uplink beamforming matrix B and the real power matrices P and U. The calculation results are shown in the following equation.

[0108]

[0109] in, is the normalized uplink beamforming matrix of user k in the tth TTI, U k,t and P k,t is the power real matrix of user k in the t-th TTI. Since the neural network cannot output complex numbers, in the actual simulation process, the real and imaginary parts of the complex numbers need to be output separately and then combined. For the problem where the number of transmitting antennas is N, the number of receiving antennas is M, and the number of users is K, it is necessary to output the real and imaginary parts of the beamforming matrix of K users, and the output dimension is (2*M*N*K). After the above conversion process, it is necessary to output the normalized uplink beamforming matrix of K users and two power real matrices. Among them, the power real matrix has only real diagonal elements, and the final output dimension is (2*K*M 2 +2*M*K). For example, in the simulation, a scenario with 8 users and 32 transmitters and 2 receivers is considered. This conversion method reduces the output dimension of the neural network from 1024 to 96, which is very helpful for the convergence and accuracy of the neural network.

[0110] Step 3: Combining the beamforming problem in multi-TTI transmission under delay constraints with the reinforcement learning framework

[0111] Reinforcement learning algorithms have unique advantages in solving such sequential decision-making problems. The key to leveraging these advantages is how to integrate the reinforcement learning framework with the original problem. The key elements of the reinforcement learning framework, namely the agent, environment, state, action, and reward, are defined as follows:

[0112] Agent: The agent is the center of the reinforcement learning framework, equivalent to the human "brain". The agent obtains status and makes decisions by observing the environment, and ultimately receives rewards. The present invention considers the beamforming problem of the base station during downlink data transmission, and therefore uses the base station as the agent.

[0113] Environment: In reinforcement learning, all content that interacts with the agent is called the environment. The agent obtains its current state from the environment and performs actions to change the environment. In this invention, the environment refers to the channel state, user transmission status, etc.

[0114] Status: Status t It is the input of the neural network and the basis for the agent to perform actions. In the present invention, in order to solve the beamforming problem in multi-TTI transmission, the agent must consider not only the channel conditions of the current TTI but also the remaining transmission data volume and delay constraints of each user when making decisions, so that the beamforming decisions of multi-TTI transmission are closely linked, thereby solving the problem of front-to-back coupling in multi-TTI transmission. Therefore, the current channel H t , the remaining data volume of each user Q_left t , each user's arrival delay constraint residual TTIT t Modeled as a state, as shown below:

[0115] s t ={H t ,Q_left t ,T t} (6)

[0116] Among them H t =[H 1,t ,H 2,t ,…,H K,t ], Q_left t =[Q_left 1,t ,Q_left 2,t ,…,Q_left K,t ], T t =[T 1,t ,T 2,t ,…,T K,t ], the remaining data volume of each user Q_left t The calculation of is shown in formula (7):

[0117]

[0118] Action: Action a t is the neural network acceptance state s t After the output, the agent changes the environment by performing actions. In the present invention, the agent needs to analyze the current state to give the beamforming matrix W. In the previous step, we have introduced how to convert the beamforming matrix W into the uplink beamforming matrix B and the power real matrices P and U to achieve dimensionality reduction. Therefore, the action is modeled as the uplink beamforming matrix B and the power real matrices P and U, as shown in the following formula:

[0119] a t =[P t ,U t ,B t ] (8)

[0120] Among them, P t =[P 1,t ,P 2,t ,…,P K,t ],U t =[U 1,t ,U 2,t ,…,U K,t ], B t =[B 1,t ,B 2,t ,…,B K,t ].

[0121] Reward: Reward t The agent observes the state s t , output action a t After that, the environment gives feedback to the agent. The agent takes action a t After that, we need to know whether this action meets or is close to the optimization goal, so as to decide whether to increase or decrease the action a when selecting the action later. t probability, so as to better meet the needs of system design. t As the environment for action a t The feedback is for action a t The optimization goal of the present invention is to maximize the sum rate in multi-TTI transmission, and the reward r t It should be related to the optimization objective function. In order to reflect the optimization goal of multi-TTI transmission in the reward of each TTI, the optimization goal of maximizing the amount of data successfully transmitted in the original problem of multi-TTI transmission is transformed into maximizing the amount of effective data transmitted in each TTI. If the user has completed the transmission in the current TTI, the effective data amount transmitted is 0. For users who have not completed the transmission, if the data amount d calculated by the beamforming matrix is k,t Greater than the remaining transmission data amount Q_left k,t, then the effective amount of transmitted data is Q_left k,t , if the amount of data d calculated by the beamforming matrix k,t Less than Q_left k,t , then the effective transmission data volume is d k,t For users who do not meet the delay constraint, the reward is defined as the amount of valid data transmitted in the current TTI, while for users who exceed the delay constraint, a certain negative reward is given. Finally, the reward for the current TTI is the sum of the rewards of all users. t The design is as follows:

[0122]

[0123]

[0124] Step 4: The neural network training process in the reinforcement learning solution framework for the beamforming problem in multi-TTI transmission under delay constraints in MU-MIMO scenarios.

[0125] Since the defined actions, i.e., the outputs of the neural network, are continuous values, some reinforcement learning algorithms, such as Deep QNetwork (DQN), can only process discrete action spaces. Therefore, the Proximal Policy Optimization (PPO) algorithm is used. The PPO algorithm establishes a multivariate normal distribution by outputting the mean of the multivariate normal distribution of each dimension of the action, and then samples from the distribution to obtain the final action to be executed, thus completing the processing of the continuous action space. The main implementation process of the algorithm is shown in Table 1:

[0126] Table 1 Beamforming code flow in multi-TTI transmission under delay constraints based on deep reinforcement learning

[0127]

[0128] The specific execution process of the algorithm is as follows:

[0129] Step 1: Initialize the relevant parameters in the communication system and PPO algorithm, including but not limited to the power limit p in Table 2 below: t , the number of transmitting antennas N, the number of receiving antennas M, the number of users K, the learning rate of the PPO algorithm l r , discount factor γ.

[0130] Table 2 Algorithm simulation related parameter settings

[0131]

[0132]

[0133] Step 2: Generate the initial state s0 = {H0, Q_left0, T0} and start the learning iteration process, where the complex Gaussian channel H is randomly generated, Q_left0 = Q, T0 = τ, and the amount of data Q that each user needs to transmit and the delay constraint T are generated according to a uniform distribution.

[0134] Step 3: Train the deep neural network. Before each round of training, first generate the initial state s0 = {H0, Q_left0, T0} and start the learning iterative process, where H is a randomly generated 0-1 complex Gaussian channel, Q_left0 = Q, T0 = τ, and the amount of data Q that each user needs to transmit and the delay constraint T are generated according to a uniform distribution.

[0135] In state s t Next, input action a through the neural network t The mean of the multivariate normal distribution is obtained by using the given covariance matrix to establish the multivariate normal distribution and sample the action a from the normal distribution. t =[P t ,U t ,B t ].

[0136] Step 4: Calculate the beamforming matrix W using equation (5), and then calculate the amount of data d transmitted by each user in the current TTI using equation (2-4) t , determine whether each user has completed the transmission and whether the delay constraint is met, and calculate the reward r t .

[0137] Step 5: Calculate the remaining transmission data volume of each user and the remaining TTI of the arrival delay constraint, and together with the randomly generated complex Gaussian channel, form the state s of the next TTI t+1 .

[0138] Step 6: Update network parameters using the PPO algorithm: receive the agent into state s t , output action a t , get reward r t , get the next state s t+1 The whole process is called a timestep. After T timesteps, the data (s t ,a t ,r t ) Compute the advantage estimate:

[0139] A k =-V(s k )+r k +γr k+1 +…+γ T-k+1 r T-1 +γT-k V(s T ) (11)

[0140] Where k = 1, 2, ..., T, V (s k ) is the state s received by the Critic network in the PPO algorithm Actor-Critic framework k The output after input, γ is the discount factor. After obtaining the advantage estimate, use equations (12)-(13) to calculate the loss function of the actor network and the critic network respectively:

[0141] L actor (θ)=E k [mim(p k (θ)A k ,clip(p k (θ),1-∈,1+∈)A k )] (12)

[0142] in, p θ (a k |s k ) means in state s k Next select action a k The probability of ,∈ is the parameter that controls the upper and lower bounds in the clip function. The clip function is used here to prevent the parameters of the Actor network from changing too drastically during the update process, which would lead to inaccurate importance sampling.

[0143]

[0144] The calculated loss function is used to update the network parameters of the Actor network and the Critic network using the Adam optimizer. After updating L times, θ old ←θ,

[0145] Step 7: Repeat steps 2 to 6 until the set number of iterations is reached or the loss function converges, at which point the training process of the reinforcement learning framework ends.

[0146] Step 8: The reinforcement learning framework for solving the beamforming problem in multi-TTI transmission under latency constraints in MU-MIMO scenarios works as follows: The channel matrix H for multiple TTIs, the amount of data required to be transmitted by each user, Q, and the latency limit τ are input into the solution framework. The framework then generates a beamforming solution for each TTI, ultimately achieving the goal of optimizing multiple TTIs and data rates. Note that in actual applications, the critic network no longer functions.

[0147] The following analysis is performed using an example: Assume that the MU-MIMO scenario has 8 users, the base station has 32 transmitting antennas, and each user has 2 receiving antennas. The method provided in the embodiment of the present invention is used to design beamforming in multi-TTI transmission under delay constraints. To verify the beneficial effects of this embodiment, it is compared with the ZF beamforming algorithm in the same scenario. First, the effects of different algorithms are compared in scenarios with a large amount of transmitted data. The amount of data transmitted by each user obeys a uniform distribution of 16 to 24, and the delay constraint obeys a uniform distribution of 12 to 20. Simulations are performed under 50 different user demand scenarios (transmission data volume requirements and delay constraints). The simulation results are as follows: Figure 4 、 Figure 5 , as shown in Table 3:

[0148] Table 3 Allocation results of different algorithms when the amount of transmitted data is large

[0149]

[0150] As can be seen from the chart, the amount of data successfully transmitted and the sum rate of completed transmission in different transmission scenarios of the DRL algorithm are greater than the results of the ZF algorithm. The Q_sum of the DRL algorithm is mean The value is 44.1% higher than that of the ZF algorithm, R_sum mean 52.0% higher. This is because the ZF algorithm, as a static beamforming algorithm, only considers the channel state of a single TTI and does not account for the impact of beamforming results between previous and subsequent TTIs. The DRL algorithm, on the other hand, incorporates the remaining delay constraints and remaining transmission data volume of each user in the current TTI into its state, taking into account the impact between previous and subsequent TTIs. This allows for more resources to be provided to users whose delay constraints are about to be reached, reduces the number of users with transmission failures, and improves the final transmission sum rate.

[0151] Furthermore, in order to reflect the applicability of this embodiment in different scenarios, the amount of user transmission data is set to obey a uniform distribution of 12 to 20, and the other simulation conditions are the same. The results are as follows: Figure 6 、 Figure 7 , as shown in Table 4:

[0152] Table 4 Allocation results of different algorithms when the amount of transmitted data is small

[0153]

[0154] As can be seen from the chart, compared with the simulation results when the amount of data transmitted is large, when the amount of data that each user needs to transmit is reduced, the Q_sum of the two algorithms is mean Value and R_sum meanThe DRL algorithm slightly outperforms the ZF algorithm in both metrics. However, the gap between the two algorithms decreases compared to the simulation results with larger data volumes. This is because when the required data volume decreases while other parameters remain unchanged, the number of users who can successfully transmit within the latency constraint increases, resulting in an increase in both the amount of successfully transmitted data and the data rate. Furthermore, since most users can complete transmission under these simulation conditions, the DRL algorithm's advantage over the ZF algorithm decreases.

[0155] Simulation results show that in the MU-MIMO scenario, the present invention considers the beamforming problem in multi-TTI transmission subject to delay constraints and data integrity constraints. The amount of data successfully transmitted and the sum rate when completing the transmission using the DRL algorithm are greater than those of the ZF algorithm. In addition, the DRL algorithm has a significant improvement over the ZF algorithm when the amount of transmitted data is large. This makes the present invention competitive in solving the beamforming problem of multi-TTI transmission under such delay constraints.

[0156] Principle of the present invention:

[0157] The present invention discloses a framework for solving the delay-constrained multi-TTI transmission problem in MU-MIMO scenarios using a reinforcement learning method. Taking into account the delay constraints of each user's request, the output of a neural network constructs the beamforming vector for each user at the beginning of each TTI, ultimately achieving the goal of maximizing the sum rate of user requests successfully transmitted after multiple TTIs.

[0158] The beamforming problem during multiple TTI transmissions in this problem is coupled. Traditional algorithms such as the ZF (zero forcing) algorithm, the MRT (maximum ratio transmission) algorithm, and some iterative algorithms mostly perform beamforming for the current TTI transmission. They cannot solve the problem from the perspective of the entire multi-TTI transmission scenario, and the final effect cannot meet the problem requirements. In actual use scenarios, users often require extremely low transmission latency. Due to the complex calculations and iterations in the solution process, the above optimization algorithms also lack latency indicators. Reinforcement learning algorithms have unique advantages. After training, the neural network directly provides relevant beamforming parameters based on different TTI scenarios. This provides a beamforming solution from the perspective of the entire multi-TTI transmission problem in a very short time, meeting the requirements of beamforming algorithms in actual use scenarios.

[0159] In the reinforcement learning model, the base station that completes the beamforming is modeled as an intelligent agent, and the current channel, the amount of remaining data for each user, and the remaining TTIs for each user's arrival delay constraints are modeled as states. When making decisions, the intelligent agent not only considers the channel conditions of the current TTI, but also considers the remaining transmission data volume and delay constraints of each user, so that the beamforming decisions for multi-TTI transmission are closely linked, thereby solving the problem of front-to-back coupling in multi-TTI transmission. In the modeling of actions, the method in

[10] is used to convert the beamforming matrix W into the uplink beamforming matrix B and the power real matrices P and U, reducing the output of the neural network from (2×M×N×K) dimensions to (2×K×M 2 +2×M×K) dimensions, where N is the number of transmit antennas, M is the number of receive antennas, and K is the number of users. In the setting of the reward obtained after taking an action in each TTI, the optimization goal of maximizing the amount of data successfully transmitted in multiple TTIs is transformed into maximizing the amount of effective data transmitted in each TTI. If the user has completed the transmission in the current TTI, the amount of effective data transmitted is 0. For users who have not completed the transmission, if the data amount R calculated by the beamforming matrix is k,t Greater than the remaining transmission data amount Q_left k,t , then the effective amount of transmitted data is Q_left k,t , if R k,t Less than Q_left k,t , then the effective transmission data volume is R k,t For users that do not meet the delay constraint, the reward is defined as the amount of valid data transmitted in the current TTI, while for users that exceed the delay constraint, a certain negative reward is given. Finally, the reward for the current TTI is the sum of the rewards of all users.

[0160] The framework uses the Proximal Policy Optimization (PPO) algorithm. The PPO algorithm establishes a multivariate normal distribution by outputting the mean of the multivariate normal distribution of each dimension of the action, and then samples from the distribution to obtain the final action to be executed, completing the processing of the continuous action space.

[0161] The network training process takes a multi-TTI transmission problem from the first TTI to the completion of transmission for all users or the reaching of the delay constraint for all users as a trajectory. At the beginning of the trajectory, the transmission data volume Q of each user is generated according to the uniform distribution. k and delay constraint T k At the beginning of each TTI, a complex Gaussian channel is randomly generated. After a large amount of training, the mapping from state to action is completed. The final system can better solve the multi-TTI transmission problem with delay constraints in MU-MIMO scenarios, and has good generalization for different user transmission data volumes and delay constraints.

[0162] One embodiment of the present invention provides a computer device. The computer device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of each of the aforementioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in each of the aforementioned apparatus embodiments are implemented.

[0163] The computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to accomplish the present invention.

[0164] The computer device may be a desktop computer, a notebook computer, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a processor and a memory.

[0165] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0166] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0167] If the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0168] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A beamforming method for multi-TTI transmission under delay constraints, characterized in that: The following steps are involved: Construct a maximum sum rate model for multi-TTI transmission under delay constraints in MU-MIMO scenarios; the maximum sum rate model for multi-TTI transmission is as follows: in, It is TTI users The beamforming matrix, is the number of users, For users The transmission rate, For users The amount of data that needs to be transferred, For users The delay constraint, For the TTI hour user The amount of data transferred, for The conjugate transpose of is power limitation; No. TTI hour user Amount of data transferred as follows: in, For users In the The MMSE matrix corresponding to TTI is: is the identity matrix, For users In the The beamforming vector corresponding to each TTI is: express The conjugate transpose of For users In the The channel matrix corresponding to TTI, for The conjugate transpose of For the TTI users The beamforming matrix, for The conjugate transpose of Perform dimensionality reduction on the maximization and rate model of multi-TTI transmission; Combining the multi-TTI transmission maximization and rate model with the reinforcement learning framework; The combined reinforcement learning framework is trained using the PPO algorithm.

2. The beamforming method in multi-TTI transmission under delay constraint according to claim 1, characterized in that The dimensionality reduction processing of the maximization and rate model of multi-TTI transmission includes: The beamforming matrix is ​​transformed into Converted into uplink beamforming matrix and power real matrix and , as follows: in, For users In the The channel matrix corresponding to each TTI is: for The conjugate transpose of user In the The normalized uplink beamforming matrix of TTIs is: for The conjugate transpose of For use In the The power real matrix of each TTI, For users In the The normalized uplink beamforming matrix of TTIs is: For users In the The normalized uplink beamforming matrix of TTIs is: For users In the The power real matrix of each TTI.

3. The beamforming method in multi-TTI transmission under delay constraint according to claim 1 or 2, characterized in that: The combination of the multi-TTI transmission maximization and rate model with the reinforcement learning framework includes: The elements of the reinforcement learning framework: agent, environment, state, action, and reward are combined with the maximization and rate model of multi-TTI transmission as follows: The intelligent agent is the base station; The environment refers to the channel status and user transmission conditions; state is the input of the neural network, as follows: in, ,and , For the The remaining transmission data volume of each user at timestep is , For the The arrival delay of each user constrains the remaining TTI at timestep, and ; Remaining data volume for each user The details are as follows: in, For users The total amount of data that needs to be transferred, For the TTI hour user The amount of data transferred; action is the neural network acceptance state The output after , the action is modeled as an uplink beamforming matrix and power real matrix and , as follows: in, 、 For the The power real matrix at timestep, and , , For the The normalized uplink beamforming matrix at timestep is ; Rewards The agent observes the state , output action After that, the environment gives feedback to the agent; the reward of the current TTI is the sum of the rewards of all users, as follows: in, For the timestep when the user The amount of reward, is the penalty coefficient.

4. The beamforming method in multi-TTI transmission under delay constraint according to claim 3, characterized in that The PPO algorithm is used to train the combined reinforcement learning framework, including: Step 1: Initialize the communication system and PPO algorithm parameters, including power limit , number of transmitting antennas N, number of receiving antennas M, number of users K, PPO algorithm learning rate , discount factor γ; Step 2: Generate initial state , starts the learning iterative process; among them, the complex Gaussian channel The amount of data Q and delay constraint T that each user needs to transmit are generated randomly, and , ; Step 3: Train the deep neural network and generate the initial state before each round of training , start the learning iterative process; among them, the randomly generated 0~1 complex Gaussian channel , the amount of data Q that each user needs to transmit and the delay constraint T are generated according to uniform distribution, and , ; In state Output action through neural network The mean of the multivariate normal distribution of , and the multivariate normal distribution is established through the given covariance matrix, and the action is sampled from the normal distribution ; Step 4: Calculate the beamforming matrix W, and then calculate the amount of data transmitted by each user in the current TTI , determine whether each user has completed the transmission and whether the delay constraint is met, and calculate the reward ; Step 5: Calculate the remaining transmission data volume of each user at this time and the remaining TTI of the arrival delay constraint, together with the randomly generated complex Gaussian channel to form the state of the next TTI ; Step 6: Update network parameters using the PPO algorithm: The agent receives the state , output action , get rewards And get the next state The whole process is a timestep. After the timestep, use this Data obtained by interacting with the environment in timestep Calculating advantage estimates : in, The state received by the Critic network in the PPO algorithm Actor-Critic framework Output after input, , is the maximum length of the trajectory, for The reward of the moment, is the discount factor, for of( ) power, for of( ) power, for The reward of the moment, T The reward of the moment, Received status for the Critic network Output after input, is the state at time T; After obtaining the advantage estimate, the loss function of the Actor network and the Critic network is calculated: in, is the loss function of the actor network, To find the expectation of the variables in [ ], is the correction coefficient in importance sampling, To update the parameters of the actor network using the Adam optimizer, and , Indicates that the status Next select action The probability of Parameters that control the upper and lower bounds in the clip function; is the loss function of the critic network, To update the parameters of the critic network using the Adam optimizer, for of( ) power, for Rewards at the moment; The calculated loss function is used to update the network parameters of the Actor network and the Critic network using the Adam optimizer. After updating L times, , ; Step 7: Repeat steps 2 to 6 until the set number of iterations is reached or the loss function converges, at which point the training process of the reinforcement learning framework ends; Step 8: Combine the channel matrix H of multiple TTIs and the amount of data that each user needs to transmit and latency constraints The reinforcement learning framework is input, and the reinforcement learning framework outputs the beamforming solution for each TTI, ultimately achieving the goal of optimizing multiple TTIs and rates.

5. A beamforming system for multi-TTI transmission under delay constraints for implementing the method of claim 1, characterized in that: include: A model building module is used to build a maximum sum rate model for multi-TTI transmission under latency constraints in MU-MIMO scenarios; A dimensionality reduction processing module, used for performing dimensionality reduction processing on the maximization and rate model of multi-TTI transmission; A combined computation module is used to combine the multi-TTI transmission maximization and rate model with the reinforcement learning framework; The model training module is used to train the combined reinforcement learning framework using the PPO algorithm.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images