Digital twinning-assisted flow-aware cellular-free MIMO (Multiple Input Multiple Output) beam selection method

Abstract

The invention provides a digital twin-assisted flow sensing cellular-free MIMO (Multiple Input Multiple Output) beam selection method, which comprises a digital twin module and a reinforcement learning (RL) beam selection module, and comprises the following implementation steps of: in the initial stage of algorithm deployment, carrying out beam selection by adopting an expert strategy, and putting environmental information observed each time into a data warehouse; after enough environment information samples are collected, a digital twin module is constructed, supervised and trained to serve as a virtual environment to perform off-line interaction with an RL beam selection module, and state and reward information required by RL training are provided; after a certain number of rounds are trained, the RL beam selection module is deployed in a real environment, so that an expert strategy is replaced, and compared with an existing RL beam selection scheme, the digital twinning assistance scheme avoids unsafety in the initial stage of exploration and has higher initial performance and higher convergence speed.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology, and in particular relates to a digital twin-assisted traffic-aware cellular-free MIMO beam selection method. Background Art

[0002] The allocation of wireless communication resources such as beam resources and time-frequency resources is crucial to wireless communication systems. Traditional beam management solutions rely on specific models to make decisions, cannot adapt to high-dimensional dynamically changing networks, and have problems such as high computational complexity.

[0003] The RL-based beam management scheme interacts with the environment and autonomously learns complex environments based on the feedback rewards after actions. It can dynamically adapt to changes in the system and overcome the dependence on prior knowledge. However, in the early stage of network training, RL needs to explore the environment to find the optimal strategy. At this time, the algorithm is similar to a random strategy. The agent obtains rewards for different actions by randomly selecting actions, thereby judging the pros and cons of different actions under the same state. The random exploration of RL brings uncertainty to the action, has a negative impact on the real environment, and cannot guarantee the user's QoS requirements. At the same time, each interaction of the RL agent needs to wait for the real environment to feedback rewards after the end of the current time slot, and the time of one interaction is limited by the duration of a time slot specified by the environment. Therefore, the entire training process has the problems of difficulty in obtaining data samples, long time consumption and poor user experience.

[0004] To solve the above problems, digital twins can be used to speed up RL convergence and ensure safety during training. In the early stages of training, RL agents do not interact with the real environment, but with the virtual environment, namely the digital twin. The digital twin provides system status information and feedback rewards, which can avoid the insecurity of exploration. At the same time, the data transmission, processing and feedback delay of the digital twin is much smaller than that of the real environment, which can significantly speed up the convergence of RL strategies. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of poor performance and unsafe exploration in the initial training of RL beam selection schemes, and to provide a digital twin-assisted traffic-aware cellular-free MIMO beam selection method to ensure user QoS in the RL training phase and accelerate RL convergence;

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a digital twin-assisted traffic-aware cellular-free MIMO beam selection method, characterized in that it includes:

[0007] Step 1: Use the LCB beam selection module to make beam decisions. At the same time, the LCB beam selection module puts the real environment information observed each time into the data warehouse;

[0008] Step 2: After collecting enough environmental information samples in the data warehouse, use these collected environmental information samples to train the LSTM network to simulate the real environment, and then combine with the random traffic generator to build a digital twin module;

[0009] Step 3: Construct a QMIX beam selection module, and interact the QMIX beam selection module with the digital twin module offline to learn the optimal strategy. At the beginning of each interaction round, a set of data is randomly selected from the data warehouse to set the initial state for all agents in the QMIX beam selection module. In each round of interaction, the QMIX beam selection module determines the action according to the current state, and then the digital twin module generates the next state and reward, and feeds it back to the QMIX beam selection module, where the agent represents the base station;

[0010] Step 4: When the number of rounds of offline interaction between the QMIX beam selection module and the digital twin module in step 3 is greater than the preset number of rounds of interaction, the QMIX beam selection module is used to replace the LCB beam selection module to make beam decisions, and the QMIX beam selection module puts the environmental information obtained in each round of interaction into the data warehouse to update its own strategy and digital twin module;

[0011] Step 5: When the environmental information changes significantly, repeat steps 1-4.

[0012] Furthermore, the digital twin module includes:

[0013] The LSTM channel predictor is used to more accurately determine the user location and mobility pattern using the joint information of multiple cooperative base stations, and predict the complete CSI of the user to all base stations in the future based on the past channel information from the user to all base stations, as shown below:

[0014] h u (t+1)=g(h u (t),h u (t-1),L,h u (1));

[0015] The random traffic generator is used to randomly generate new traffic for each user in each round of interaction according to the Poisson distribution of the traffic arrival rate λ to update the user waiting queue q u (t), get the user queue length q of the next time slot u (t+1).

[0016] Further, the QMIX beam selection module includes:

[0017] Local observation: It is defined as the combination of the waiting queue length of all users and the channel between base station b and all users, expressed as:

[0018]

[0019] Among them, u represents the uth, is the user index, and h b,u (t) represents the channel between base station b and the u-th user at time slot t;

[0020] Global state: It is the concatenation of all agents’ local observations, which is obtained by combining the system CSI and the current waiting queue lengths of all users, expressed as:

[0021]

[0022] Among them, h u (t) represents the channel between all base stations and the u-th user at time slot t;

[0023] Action: It is defined as the set of all beam indices in the serving beam set of base station b, expressed as:

[0024]

[0025] Among them, i b,u (t) represents the index of the u-th beam at time slot t;

[0026] Reward: It is defined as the global reward received by the agent from the environment and is expressed as:

[0027]

[0028] Beneficial effect: Compared with the QMIX algorithm that only interacts with the real environment, the digital twin-assisted QMIX beam selection algorithm (QMIX-DT) of the present invention has a smaller variance in the training process and requires fewer rounds for convergence. The duration of each round of interaction with the digital twin is affected by the CPU performance and is much faster than a round of interaction with the real environment. Therefore, the convergence speed can be significantly improved. In addition, the digital twin-assisted QMIX algorithm avoids the insecurity of exploration, improves the performance lower bound of the algorithm, and ensures the user's QoS. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the present invention;

[0030] Figure 2 It is a schematic diagram of the flow of traffic-aware cellular-free MIMO beam selection assisted by the digital twin of the present invention;

[0031] Figure 3 Convergence diagram of the QMIX-DT algorithm deployed in the real environment for 500 rounds of interaction with the digital twin throughout the process;

[0032] Figure 4The performance comparison chart of QMIX-DT deployed in a real environment and QMIX algorithm for 500 rounds of interaction with the digital twin;

[0033] Figure 5 Performance comparison chart of QMIX-DT deployed in a real environment and QMIX algorithm after 1500 rounds of interaction with the digital twin. DETAILED DESCRIPTION

[0034] The present invention will be further explained below in conjunction with the accompanying drawings.

[0035] First, the non-cellular system joint transmission uplink scenario is modeled, including the channel model, transmission model, queue dynamic changes and the optimization problem of minimizing the long-term average delay cost of the system. The specific steps include:

[0036] Step S1: Considering the scenario of non-cellular system joint transmission uplink in which B base stations simultaneously serve U users, define the cooperative base station set and the service user set is Each base station is equipped with M antennas, and each user is equipped with a single antenna. The base station is connected to the CPU via a forward link for unified information processing. The channel between base station b and user u is It can be expressed as:

[0037]

[0038] Where L represents the number of propagation paths, g b,u,l =α b,u,l β b,u,l represents the complex gain of the lth path, α b,u,l and β b,u,l are large-scale fading and small-scale fading, θ b,u,l The base station antenna array uses a uniform linear array with a half-wavelength spacing between adjacent antennas.

[0039] Step S2: For non-cellular uplink transmission, the channel between user u and all cooperative base stations is Define the complete CSI of the system as Considering that the system adopts a hybrid beamforming architecture, base station b uses the standard DFT codebook Select U beams from the , the index of the uth beam is i b,u Base station b uses simulated beamforming service users, then the system's analog receiver Beam-domain equivalent channel matrix constructed based on simulated reception Using ZF receiver in CPU By digitally combining, the hybrid receiver of the system can be expressed as represents the u-th row of the hybrid reception combining matrix. The received signal of user u after processing can be expressed as:

[0040]

[0041] Where P u represents the average normalized transmit power, s u Represents the signal sent by user u, satisfying represents the received noise, where represents the receiver white noise of base station b. The SINR of user u can be expressed as:

[0042]

[0043] The bandwidth allocated to each user is W u , without considering the system beam training time, the achievable rate of user u is:

[0044] R u =W u log2(1+ρ u ).

[0045] Step S3: Use FIFO queue to store the data that the user has not sent. The newly generated data of the user enters the waiting queue at the end of each time slot. The newly arrived traffic size A of user u in time slot t u (t) is subject to the parameter λ u Poisson distribution of :

[0046]

[0047] At the same time, the user traffic between different time slots arrives independently and identically distributed. The length of each time slot is τ, and the amount of data that user u can transmit in time slot t is τR u (t), then in the next time slot t+1, the user queue length is q u (t) is updated as follows:

[0048] q u (t+1)=max{q u (t)-τR u (t),0}+A u (t).

[0049] Step S4: According to Little's theorem, minimizing the average queue length is equivalent to minimizing the delay. The entire cycle of the upload traffic contains T time slots, and the long-term average delay cost of user u in this cycle is defined as:

[0050]

[0051] Each base station uses a service beam set to transmit user data. Different beam selection strategies correspond to different user rates, which brings different delay perceptions to users. The delay performance of the system is optimized through appropriate beam selection. The optimization problem of minimizing the long-term average delay cost of the system can be expressed as:

[0052]

[0053] For the optimization problem of minimizing the long-term average delay cost of the system, such as Figure 1-2 As shown, the present invention provides a digital twin-assisted traffic-aware cellular-free MIMO beam selection method, comprising:

[0054] Step 1: Use the LCB beam selection module to make beam decisions. At the same time, the LCB beam selection module puts the real environment information observed each time into the data warehouse.

[0055] Step 2: After collecting enough environmental information samples in the data warehouse, use these collected environmental information samples to train the LSTM network to simulate the real environment, and then combine them with the random traffic generator to build a digital twin module.

[0056] Step 3: Construct a QMIX beam selection module, and interact the QMIX beam selection module with the digital twin module offline to learn the optimal strategy. At the beginning of each interaction round, a set of data is randomly selected from the data warehouse to set the initial state for all agents in the QMIX beam selection module. In each round of interaction, the QMIX beam selection module determines the action based on the current state, and then the digital twin module generates the next state and reward, and feeds it back to the QMIX beam selection module, where the agent represents the base station.

[0057] Step 4: When the number of rounds of offline interaction between the QMIX beam selection module and the digital twin module in step 3 is greater than the pre-set number of rounds of interaction, the QMIX beam selection module is used to replace the LCB beam selection module to make beam decisions, and the QMIX beam selection module puts the environmental information obtained in each round of interaction into the data warehouse to update its own strategy and digital twin module.

[0058] Step 5: When the environmental information changes significantly, repeat steps 1-4.

[0059] In step 2, the digital twin module includes:

[0060] The LSTM channel predictor is used to more accurately determine the user location and mobility pattern using the joint information of multiple cooperative base stations, and predict the complete CSI of the user to all base stations in the future based on the past channel information from the user to all base stations, as shown below:

[0061] h u(t+1)=g(h u (t),h u (t-1),L,h u (1));

[0062] The random traffic generator is used to randomly generate new traffic for each user according to the Poisson distribution using the traffic arrival rate λ in each round of interaction to update the user waiting queue q u (t), get the user queue length q of the next time slot u (t+1).

[0063] In this embodiment, during the user movement, the channels have spatial correlation, and the correlation of the channels during the user movement is expressed as follows:

[0064]

[0065] LSTM is used to process the input channel sequence data and perform channel prediction for each user. For an LSTM with an input sequence length of N, the input is the channel from user u to all cooperative base stations in the previous N time slots, and the output is the channel from user u to all cooperative base stations in the next time slot, which can be expressed as:

[0066] h u (t+1)=g(h u (t),h u (t-1),L,h u (t-N+1)).

[0067] In step 3, the QMIX beam selection module includes:

[0068] Local observation: It is defined as the combination of the waiting queue length of all users and the channel between base station b and all users, expressed as:

[0069]

[0070] Among them, u represents the uth, is the user index, and h b,u (t) represents the channel between base station b and the u-th user at time slot t;

[0071] Global state: It is the concatenation of all agents’ local observations, which is obtained by combining the system CSI and the current waiting queue lengths of all users, expressed as:

[0072]

[0073] Among them, h u (t) represents the channel between all base stations and the u-th user at time slot t;

[0074] Action: It is defined as the set of all beam indices in the serving beam set of base station b, expressed as:

[0075]

[0076] Among them, i b,u (t) represents the index of the u-th beam at time slot t;

[0077] Reward: It is defined as the global reward received by the agent from the environment and is expressed as:

[0078]

[0079] In this embodiment, QMIX is a multi-agent RL algorithm with centralized training and distributed execution. In the training phase, global state information is used to achieve good training results. In the execution phase, each base station uses a local network to make distributed decisions. The QMIX beam selection module is obtained according to the above-mentioned Markov decision process modeling, and QMIX is used to solve it. QMIX selects the optimal action in each time slot.

[0080] like Figure 3 As shown in the figure, due to the deviation of the statistical characteristics between the digital twin module and the real environment, QMIX-DT will experience a certain performance degradation after switching from the digital twin to the real environment.

[0081] like Figure 4 As shown in the figure, the pre-trained QMIX algorithm provided by the digital twin can achieve better performance than without the assistance of the digital twin at the beginning of interaction with the real environment, and converges faster.

[0082] like Figure 5 As shown in the figure, the initial reward of QMIX-DT that interacts with the digital twin for 1500 rounds is higher than that of QMIX-DT that interacts with the digital twin for 500 rounds, demonstrating the accuracy of digital twins in simulating the real environment and the effectiveness of offline training.

[0083] As shown in Table 1, the user queue backlog of different algorithms deployed in the real environment, among which the QMIX algorithm only interacts with the real environment, while the QMIX-DT algorithm only interacts with the digital twin module offline, and the LCB algorithm does not need iteration. The results show that the LCB algorithm has the best performance, and almost all data packets can be sent out of the queue within one time slot. However, the LCB algorithm needs to calculate the real rate corresponding to 21952 global actions in the action space and the weighted sum of the rate and queue length, which brings extremely high computational complexity and processing delay, and has poor scalability and cannot cope with larger-scale non-cellular systems. Due to the slight deviation of LSTM channel prediction, the performance of the QMIX-DT algorithm that only interacts with the digital twin is slightly reduced on user 1, while the performance of user 2 is more significantly reduced. Despite this, the user QoS of the QMIX-DT algorithm that only interacts with the digital twin is still within an acceptable range.

[0084] Table 1

[0085]

[0086] The digital twin module of the present invention assists the QMIX beam selection module in the training process with smaller variance, and requires fewer rounds for convergence. The duration of each round of interaction with the digital twin module is affected by the CPU performance, and is much faster than a round of interaction with the real environment. Therefore, the convergence speed can be significantly improved. In addition, the digital twin module assists the QMIX beam selection module to avoid unsafe exploration, improves the performance lower bound of the algorithm, and ensures the user's QoS.

[0087] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A digital twin-assisted traffic-aware cellular-free MIMO beam selection method, characterized in that: include: Step 1: Use the LCB beam selection module to make beam decisions. At the same time, the LCB beam selection module puts the real environment information observed each time into the data warehouse; Step 2: After collecting enough environmental information samples in the data warehouse, use these collected environmental information samples to train the LSTM network to simulate the real environment, and then combine with the random traffic generator to build a digital twin module; Step 3: Construct a QMIX beam selection module, and interact the QMIX beam selection module with the digital twin module offline to learn the optimal strategy. At the beginning of each interaction round, a set of data is randomly selected from the data warehouse to set the initial state for all agents in the QMIX beam selection module. In each round of interaction, the QMIX beam selection module determines the action according to the current state, and then the digital twin module generates the next state and reward, and feeds it back to the QMIX beam selection module, where the agent represents the base station; Step 4: When the number of rounds of offline interaction between the QMIX beam selection module and the digital twin module in step 3 is greater than the preset number of rounds of interaction, the QMIX beam selection module is used to replace the LCB beam selection module to make beam decisions, and the QMIX beam selection module puts the environmental information obtained in each round of interaction into the data warehouse to update its own strategy and digital twin module; Step 5: When the environmental information changes significantly, repeat steps 1-4.

2. According to claim 1, the digital twin-assisted traffic-aware cellular-free MIMO beam selection method is characterized in that: The digital twin module includes: The LSTM channel predictor is used to more accurately determine the user location and mobility pattern using the joint information of multiple cooperative base stations, and predict the complete CSI of the user to all base stations in the future based on the past channel information from the user to all base stations, as shown below: h u (t+1)=g(h u (t),h u (t-1),L,h u (1)); The random traffic generator is used to randomly generate new traffic for each user in each round of interaction according to the Poisson distribution of the traffic arrival rate λ to update the user waiting queue q u (t), get the user queue length q of the next time slot u (t+1).

3. According to claim 2, the digital twin-assisted traffic-aware cellular-free MIMO beam selection method is characterized in that: The QMIX beam selection module includes: Local observation: It is defined as the combination of the waiting queue length of all users and the channel between base station b and all users, expressed as: Among them, u represents the uth, is the user index, and h b,u (t) represents the channel between base station b and the u-th user at time slot t; Global state: It is the concatenation of all agents’ local observations, which is obtained by combining the system CSI and the current waiting queue lengths of all users, expressed as: Among them, h u (t) represents the channel between all base stations and the u-th user at time slot t; Action: It is defined as the set of all beam indices in the serving beam set of base station b, expressed as: Among them, i b,u (t) represents the index of the u-th beam at time slot t; Reward: It is defined as the global reward received by the agent from the environment and is expressed as:

Images