A method and system for estimating the middle layer and precoding matrix in large-scale MIMO

Abstract

This invention relates to the field of communications, and particularly to a method and system for estimating the middle layer and precoding matrix in large-scale MIMO. It mainly includes: each base station acquiring its own and the UE's state data, using this state data as a training dataset to train a reinforcement learning model; deploying the trained reinforcement learning model on each base station, each base station acquiring current state data through interaction with the UE, and the reinforcement learning model on the base station calculating the optimal R / PM based on the UE channel matrix; the UE performing precoding based on the R / PM transmitted by each base station; each base station obtaining the UE channel matrix for the next time step based on the precoded network state, iteratively updating the training dataset, and retraining the model using the updated training dataset. This invention can dynamically calculate the number of layers and the precoding matrix, effectively reducing network latency and improving network throughput.

Description

Technical Field

[0001] This invention relates to the field of communications, and in particular to a method and system for estimating the middle layer and precoding matrix in large-scale MIMO. Background Technology

[0002] In network communication, the time and frequency domain resources of the spectrum are limited. Multiple-input multiple-output (MIMO) can reuse these resources in the air in a layered manner without increasing spectrum resources. Reasonable layering can improve spectrum utilization. In large-scale 5G applications, MIMO has a maximum of 16 layers, but in 6G ultra-large-scale MIMO, there may be more layers and precoding matrices. More layers require more fine-grained channel characteristics to distinguish them. Calculating the number of layers and precoding matrices using traditional algorithms cannot guarantee the best performance, such as a better user equipment (UE) experience or higher cell throughput.

[0003] Because different base stations have different channel environments, noise, and interference, using a fixed algorithm to calculate the layers and precoding matrix cannot maximize throughput. It is necessary to dynamically calculate the layers and precoding matrix based on the base station's channel environment. With the increasing number of network users, the network environment will become increasingly complex. How to dynamically generate estimated layer numbers (Rank, R) and precoding matrices (PM) based on the actual network conditions will become a challenge for future technologies.

[0004] Therefore, how to overcome the shortcomings of existing technologies and solve the complex phenomenon of generating R / PM based on real network states in large-scale MIMO scenarios is a problem to be solved in this technical field. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention solves the problem of the complexity of generating R / PM based on the real network state in large-scale MIMO scenarios.

[0006] The embodiments of the present invention adopt the following technical solutions:

[0007] In a first aspect, the present invention provides a method for estimating the mid-layer and precoding matrix in large-scale MIMO, specifically: each base station acquires its own and the UE's state data, and uses the state data as a training dataset to train a reinforcement learning model, wherein the state data includes the UE channel matrix, R / PM, signal-to-interference-plus-noise ratio (SINNR), and the UE channel matrix at the next time step, and the reward of the reinforcement learning model is the SINNR of the network; the trained reinforcement learning model is deployed on each base station, and each base station acquires the current state data through interaction with the UE, and the reinforcement learning model on the base station calculates the optimal R / PM based on the UE channel matrix; the UE performs precoding based on the R / PM sent by each base station, each base station acquires the UE channel matrix at the next time step based on the precoded network state, iteratively updates the training dataset, and retrains the model using the updated training dataset.

[0008] Preferably, the acquisition of status data for each base station includes: the base station obtaining the UE channel matrix through interactive messages, generating all available layers and precoding matrices as an action set based on the UE channel matrix; the base station sending P / PM to the UE through DCI, calculating the network signal-to-interference-plus-noise ratio based on the received UE feedback; the base station randomly selecting a set of layers and precoding matrices from the action set as the current action, and calculating the UE channel matrix for the next moment based on the current action.

[0009] Preferably, the state data is used as the training dataset to train the reinforcement learning model. Specifically, the base station uses each set of [UE channel matrix, action set, network signal-to-interference-plus-noise ratio, and next-time UE channel matrix] as the training data for the reinforcement learning model. The current UE channel matrix is ​​used as the state of reinforcement learning, the signal-to-interference-plus-noise ratio is used as the reward, and the number of layers and the precoding matrix are used as the action. The deep reinforcement learning model is used to perform layer selection and precoding matrix calculation.

[0010] Preferably, using state data as a training dataset to train the reinforcement learning model further includes: the reinforcement learning model selecting a layer number and a precoding matrix based on the current input UE channel matrix to maximize the signal-to-interference-plus-noise ratio of the network; during training, first estimating the parameters of the Q function, then fixing the parameters of the Q function, and using the gradient ascent algorithm to update the parameters of the Actor.

[0011] Preferably, using state data as a training dataset to train the reinforcement learning model further includes: the output of the Q function at the current time is the signal-to-interference-plus-noise ratio (SIR) of the network after the data packet is transmitted from the source UE to the target UE at the current time; the output of the Q function at the next time is the SIR of the network after the data packet is transmitted from the source UE to the target UE at the next time.

[0012] Preferably, each base station obtains current state data through interaction with the UE, and the reinforcement learning model on the base station calculates the optimal R / PM based on the UE channel matrix. Specifically, the base station and the UE obtain the UE channel matrix through message interaction and calculate the current network state data; the base station sends the UE channel matrix to the reinforcement learning model, the reinforcement learning model calculates the optimal R / PM, and sends it to the UE through DCI messages.

[0013] Preferably, each base station obtains current status data through interaction with the UE, specifically including: the base station sending a CSI-RS message to the UE, and the UE sending its UE channel matrix to the base station through a report message.

[0014] Preferably, the reinforcement learning model on the base station calculates the optimal R / PM based on the UE channel matrix. Specifically, the base station sends the UE's channel matrix to the real-time learning system, the real-time learning system calls the trained reinforcement learning model to output the R / PM, and sends the R / PM to the base station.

[0015] Preferably, using real data as the training dataset further includes: using data from the training dataset to train the GAN model, and adding the converged data of the GAN model to the training dataset, wherein the output data structure of the GAN model is the status data of the base station.

[0016] On the other hand, the present invention provides a system for estimating the mid-level and precoding matrix of a large-scale MIMO system, including a base station and at least one UE. Specifically, each UE obtains state data by interacting with the base station, inputs the UE channel matrix in the state data into a reinforcement learning model trained on the base station, and obtains the R / PM using the method for estimating the mid-level and precoding matrix of a large-scale MIMO system according to any one of claims 1-9; the base station sends the R / PM to the UE, and the UE performs precoding based on the R / PM; the base station obtains the UE channel matrix at the next time step after precoding by the UE, iteratively updates the training dataset, and retrains the model using the updated training dataset.

[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: the base station's real-time learning system can learn the optimal layer and precoding matrix for the UE based on changing environments. By relying on the large amounts of data generated by the network itself, and employing deep reinforcement learning algorithms to learn the channel characteristics contained in the network environment, the system can dynamically calculate the number of layers and the precoding matrix, effectively reducing network latency and improving network throughput. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 This refers to the PMI table configuration used in this embodiment of the invention;

[0020] Figure 2 This describes the antenna port configuration of the PMI used in this embodiment of the invention.

[0021] Figure 3 A flowchart of a method for estimating the middle layer and precoding matrix in large-scale MIMO provided in this embodiment of the invention;

[0022] Figure 4 This is a schematic diagram of the iterative learning of Rank in an embodiment of the present invention;

[0023] Figure 5 A flowchart illustrating another method for estimating the middle layer and precoding matrix in large-scale MIMO, provided in an embodiment of the present invention;

[0024] Figure 6 A flowchart illustrating another method for estimating the middle layer and precoding matrix in large-scale MIMO, provided in an embodiment of the present invention;

[0025] Figure 7 This is a schematic diagram of the iterative learning process in an embodiment of the present invention;

[0026] Figure 8 This is a schematic diagram of the neural network structure in an embodiment of the present invention;

[0027] Figure 9 This is a schematic diagram of a system architecture for estimating the middle layer and precoding matrix in a large-scale MIMO model, provided as an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0029] This invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly describe the functional logic relationship of each structural module, and do not limit the specific software and hardware implementation methods.

[0030] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] Example 1:

[0032] In the current network implementation, the transmitter traverses all possibilities through layer and antenna configurations to find the optimal precoding matrix that maximizes capacity. Based on the optimal precoding matrix (PM), the transmitter sequentially sends one item from the matrix to the receiver, i.e., it sends the corresponding i... 1,1, i 1,2 i 1,3 The data is sent to the receiver, which then uses the received i... 1,1, i 1,2 i 1,3 Then calculate the precoding matrix.

[0033] In a specific scenario, the Precoding Matrix Indicator (PMI) table and antenna port configuration are as follows: Figure 1 and Figure 2 As shown.

[0034] In this scenario, CSI-RS port = 4, N1 = 2, N2 = 1, O1 = 4, O2 = 1, i 1,1 =1,i 1,2 =0,i 1,3 =1,i2=1,k1=O1=4,k2=0,l=i 1,1 =1,l'=i 1,1 +k1=1+4=5,m=i 1,2 =0,m'=i 1,2 +k2=0+0=0, n=i2=1.Number of layers=2.

[0035] Therefore, u can be calculated. m vector

[0036] Because N2 = 1, therefore u m =1, thus deriving v l,m vector

[0037] Given l = 1 and m = 0, substituting these values ​​into the above equation, we get...

[0038] After transposing and simplifying, we get

[0039] Next, calculate Phi.

[0040] Next, derive v l,m Based on this vector

[0041] Substituting l' and m' yields the vector

[0042] After transposing and simplifying, we get

[0043] Substitute all the calculated parameters into the formula in the table to obtain the precoding matrix;

[0044]

[0045] Substitute parameters further

[0046] After substituting the parameters, the final PM is obtained.

[0047]

[0048] In traditional algorithms, PM is calculated using the fixed formula mentioned above. However, changes in the network environment represented by the CSI-RS port can affect the accuracy of channel estimation. Therefore, it is necessary to adjust the number of layers and the precoding matrix to adapt to the current environment. In large-scale MIMO scenarios, the computational cost of adjusting the number of layers and the precoding matrix is ​​very high, making traditional algorithms computationally inefficient.

[0049] like Figure 3 As shown in the figure, the specific steps of the method for estimating the middle layer and precoding matrix in large-scale MIMO provided by the embodiments of the present invention are as follows:

[0050] Step 101: Each base station acquires its own and the UE's state data, and uses the state data as a training dataset to train the reinforcement learning model.

[0051] In this embodiment, deep reinforcement learning is used to estimate the layers and precoding matrix. To ensure the reinforcement learning model produces correct outputs, it needs to be trained using real data obtained from the existing network. The training data includes the UE channel matrix, the number of layers (Rank, R), and the precoding matrix (PM) corresponding to the layer and precoding matrix estimation, as well as the signal-to-interference-plus-noise ratio (SINR) related to the precoding communication performance.

[0052] In this field, the signal-to-interference-plus-noise ratio (SINR) is a key technical indicator for measuring the reliability of communication system quality; a higher SINR generally indicates better communication quality. SINR can be expressed as: SINR = S / (I+N). Here, S is the power of the measured useful signal, primarily focusing on RS and PDSCH signals and channels; I is the power of the measured signal or channel interference signal, including interference from other cells in the system and interference from other systems; and N is the noise floor, which is related to the specific measurement bandwidth and receiver noise figure. In this embodiment, SINR is used as the reward for the reinforcement learning model, ensuring that the number of layers and precoding matrix output by the reinforcement learning model achieve the best communication performance.

[0053] To evaluate the long-term benefits of the changes, the UE channel matrix at the next time step needs to be used as training data. To further improve access efficiency in scenarios with large amounts of data, the state data used as the training dataset can be stored and managed using a database.

[0054] Specifically, the implementation uses the UE channel matrix as input for deep reinforcement learning. Each UE corresponds to a state vector, and the state vectors of multiple UEs are combined to form the channel matrix. The channel matrix information contains the features pre-coded by the UE. The deep learning algorithm extracts features from this data and maps the channel matrix to the optimal pre-coding matrix.

[0055] In practical implementation, the UE channel matrix data can refer to the following data structure.

[0056]

[0057] m: represents the number of antennas. For example: 10.

[0058] n: represents the number of antenna subarrays. For example: 1024.

[0059] h mn : Represents the weight of an antenna subarray.

[0060] In practice, for uplink MIMO, the channel matrix represents the uplink channels of the current UE. For downlink MIMO, the channel matrix represents the downlink channels of the current UE.

[0061] Furthermore, in practical implementation, the UE channel matrix in the state data can also use the channel impulse response. The channel matrix is ​​frequency domain data, while the channel impulse response is time domain data after the channel matrix has undergone IFFT transformation. Both refer to the same physical object in actual communication and can be used as input for deep reinforcement learning to estimate layers and precoding matrices.

[0062] In reinforcement learning models, choosing a suitable reward can help the model converge quickly. According to Shannon's theorem, C = B log2(1 + SINR), the maximum transmission rate is proportional to SINR. Therefore, in the method provided in this embodiment, the signal-to-interference-plus-noise ratio (SIR) in the network is used as the reward. Reinforcement learning aims to maximize long-term future gains by selecting an Action, so the reward is the sum of all SIRs in the future t+n+1.

[0063] For MIMO, at low frequencies, a local reward can be used, i.e., r t =SINR t =Sum(SINR1+SINR2+……+SINRn), where SINRn is the SINR of the nth layer. At high frequencies, a global reward, i.e., r, can be used. t =Sum(SINR) ue1 +SINR ue2 +SINR ue3 +……+SINR ueN SINR ueN This is the SINR of the Nth UE. If a global reward is used, the state of the deep reinforcement learning model also needs to use the channel matrix of all UEs, and the action also needs to be the action of all UEs.

[0064] In practice, throughput can also be used as the reward. At low frequencies, a local reward, r, can be used. t =Sum(Throughput1+Throughput2+……+Throughput n Throughput n This is the throughput of the nth layer. At high frequencies, the full reward can be used, i.e., r. t =Sum(Throughput1+Throughput2+……+Throughput n Throughput n It is the throughput of the nth UE.

[0065] Step 102: Deploy the trained reinforcement learning model on each base station. Each base station obtains the current state data through interaction with the UE. The reinforcement learning model on the base station calculates the optimal R / PM based on the UE channel matrix.

[0066] After training the deep reinforcement learning model using the method in step 101 and saving the trained model, the model can be used to estimate the layers and precoding matrix. Real data from the live network can be used as input to the trained deep reinforcement learning model to output the optimal Rank and Precoding Matrix.

[0067] In reinforcement learning, the intelligent learner selects an action based on the current environmental state; this action belongs to an action set. In the method of this embodiment, the action set includes R (Rank) and PM (Precoding Matrix). An action is a finite set, and R is a positive integer. In practice, R is determined based on the number of layers actually supported by the network hardware; currently, R ∈ [1, 2, 3, 4, ..., 100] is supported. PM is the precoding matrix, which can be represented in the following matrix form.

[0068]

[0069] In order to obtain more R and PM, the system needs to adjust the new action based on the initial action.

[0070] Assume the initial R = 2,

[0071] Randomly select an action, R∈[Initial RN, Initial R+N].

[0072]

[0073] N and θ are the step size, where N is an integer and θ∈[0,2π].

[0074] e.g. the selected

[0075]

[0076]

[0077] If the environment state is a pre-coded matrix, the AI ​​model will output a phase. PM∈[-θ,θ],

[0078]

[0079]

[0080]

[0081] The transmitter sends θ, RI, and PMI to the receiver. Upon receiving these signals, the receiver adds the phase θ to PM for MIMO calculations.

[0082] Following the above process, such as Figure 4 The diagram illustrates an example of iterative learning of Rank in a specific implementation scenario. Initially, R = 28, and the step size is N. After continuous iterations, the AI ​​model will find the optimal R = 40.

[0083] The iteration of PM is similar to that of R, using a step size of θ. In a specific instance, θ has 3 possible values, and the size of PM is 3*3=9. The iterative learning will start from 3... 9 The best combination is selected from among the possible combinations. In practice, each θ is typically... nm The learning directions are different, and it is quite difficult to find the best combination. The step size of PM iterative learning can be set to a larger value to complete the iteration quickly.

[0084] Step 103: The UE performs precoding based on the R / PM sent by each base station. Each base station obtains the UE channel matrix for the next time step based on the precoded network state, iteratively updates the training dataset, and retrains the model using the updated training dataset.

[0085] To achieve better long-term benefits and enable timely dynamic adjustments based on the current network conditions, the training dataset needs to be iteratively updated. The reward in reinforcement learning uses a long-term reward value, representing the reward obtained by the current actor at time t+n+1 after choosing an action based on the current environment. Therefore, when acquiring sample data, the signal-to-interference-plus-noise ratio (SIR / NDR) at both time t+1 and t+n+1 needs to be used as training data. The reinforcement learning model trained using this dataset can optimize not only the reward in the current period but also the reward at time t+n+1, resulting in better long-term benefits.

[0086] After steps 101-103 provided in this embodiment, the estimated number of layers and precoding matrix can be dynamically generated based on the actual network state, making full use of network resources and obtaining higher spectrum utilization and throughput.

[0087] like Figure 5 As shown, the following steps can be used to obtain the status data of each base station required in the above steps.

[0088] Step 201: The base station obtains the UE channel matrix through interactive messages, and generates all available layers and precoding matrices as an action set based on the UE channel matrix.

[0089] The required state data includes: UE channel matrix, layer number, precoding matrix, and UE channel matrix for the next time step.

[0090] The UE channel matrix can be obtained directly through the interaction messages between the base station and the UE. For ease of explanation, the UE channel matrix will be represented as s below. t .

[0091] The number of layers and the precoding matrix need to be generated using the algorithm provided in step 102, based on the obtained UE channel matrix. During training and estimation, each set of layers and precoding matrices is combined into an action, and all available actions are combined into an action set. The neural network model is reinforced by randomly selecting one action each time and evaluating its reward to ultimately obtain the optimal R / PM. For ease of illustration, the action is denoted as a below. t .

[0092] Step 202: The base station sends the P / PM to the UE via DCI and calculates the network's signal-to-interference-plus-noise ratio based on the received feedback from the UE.

[0093] After the base station sends the R / PM to the UE via Downlink Control Information (DCI), it receives feedback from the UE and calculates the network's signal-to-interference-plus-noise ratio (SIR). For ease of explanation, the SIR will be expressed as r below. t .

[0094] Step 203: The base station randomly selects a set of layers and precoding matrices from the action set as the current action, and calculates the UE channel matrix for the next moment based on the current action.

[0095] The base station randomly selects action 'a' from the action set. t The corresponding layer number and signal-to-interference-plus-noise ratio (SINNR) of the precoding matrix are obtained through the reinforcement learning model, and the UE channel matrix for the next time step is calculated accordingly. For ease of explanation, the UE channel matrix for the next time step is represented as s. t+1 .

[0096] After obtaining the status data, the base station can [s] for easier use. t ,a t ,r t ,s t+1 The data is stored in the database as training data for the AI ​​model.

[0097] After steps 201-203 provided in this embodiment, the state data used for training and estimation can be obtained.

[0098] In this embodiment, a real-time learning system (RTLS) can be used to encapsulate the reinforcement learning model and provide corresponding external interfaces. The RTLS includes a data storage unit and a deep reinforcement learning AI algorithm unit. The RTLS and the base station exchange messages to obtain the UE's channel matrix. The base station stores each set of [UE channel matrix, action set, network signal-to-interference-plus-noise ratio (SIR), next-moment UE channel matrix] as training data for the reinforcement learning model in the database of the data storage unit. During offline training, the AI ​​algorithm unit retrieves training data from the database and updates the model's parameters. During online application, the AI ​​algorithm unit calculates the layers and precoding matrices in real time based on the UE's channel matrix in the network and performs uplink and downlink spatial multiplexing. The AI ​​algorithm unit uses the current UE channel matrix as the reinforcement learning state, the SIR as the reward, and the layer number and precoding matrix as the action, and uses the deep reinforcement learning model to perform layer selection and precoding matrix calculation.

[0099] Specifically, this invention employs the Actor-Critic algorithm, where both the actor and the Q-function are neural network models. In this example, a convolutional neural network, including but not limited to CNN, is used. The reinforcement learning model selects a layer number and a precoding matrix based on the current input UE channel matrix to maximize the network's signal-to-interference-plus-noise ratio (SNR). During training, the parameters of the Q-function are first estimated, then the parameters of the Q-function are fixed, and the Actor's parameters are updated using a gradient ascent algorithm.

[0100] In specific training, at the current time, i.e., time t, the output r of the Q function... t This represents the signal-to-interference-plus-noise ratio (SIR) of the network after the data packet is transmitted from the source UE to the target UE. At the next time step, i.e., time t+n+1, the output r of the Q function... t+n+1 The signal-to-interference-plus-noise ratio (SIR / NNR) of the network after the data packet is transmitted from the source UE to the target UE.

[0101] In practice, the acquisition of state data and the use of the channel matrix need to be completed jointly by the base station and the UE. For example... Figure 6 As shown, in specific implementation, the following steps can be used to complete the process of each base station obtaining current status data through interaction with the UE, and the reinforcement learning model on the base station calculating the optimal R / PM based on the UE channel matrix.

[0102] Step 301: The base station and the UE obtain the UE channel matrix through message interaction and calculate the current network status data.

[0103] Specifically, the base station sends a CSI-RS message to the UE, and the UE sends its own UE channel matrix to the base station through a report message, and further obtains other status data required according to steps 201-203.

[0104] Step 302: The base station sends the UE channel matrix to the reinforcement learning model. The reinforcement learning model calculates the optimal R / PM and sends it to the UE via DCI message.

[0105] The base station sends the UE's channel matrix to the real-time learning system via a Get_R_PM request message. The real-time learning system calls the reinforcement learning model trained in the AI ​​algorithm unit to output R and PM, and sends the optimal Rank and PM to the base station via a Get_R_PM response message. The base station then sends the optimal Rank and PM back to the UE via a DCI message.

[0106] After steps 301-302, the data interaction process between the base station and the UE can be realized.

[0107] Furthermore, AI algorithms require a large amount of training data, but in reality, it is difficult to obtain a large number of samples. Therefore, in this embodiment, the data in the training dataset is used to train the GAN model, and the converged data of the GAN model is added to the training dataset and stored in the database to save the time and cost of data collection. GAN stands for Generative Adversarial Network, which consists of a generator and a discriminator. The generator generates target data based on the input data (vector), and the discriminator is used to distinguish whether the data generated by the generator is real or fake. The two are in a game process, and the model converges when the discriminator cannot distinguish the data generated by the generator. A convolutional neural network can be used as the discriminator of the GAN, and a deconvolutional neural network can be used as the generator of the GAN. In this embodiment, the GAN model inputs a random vector from the collected training dataset into the generative adversarial network. When the GAN converges, the generator outputs data in the same format as the collected state data, namely [UE channel matrix, layer number / precoding matrix, signal-to-interference-plus-noise ratio, UE channel matrix at the next time step].

[0108] The method for estimating the middle layer and precoding matrix in large-scale MIMO provided in this embodiment relies on the big data generated by the network itself and uses a deep reinforcement learning algorithm to learn the channel characteristics contained in the network environment, thereby dynamically calculating the number of layers and the precoding matrix. This can effectively reduce network latency, improve network throughput, and effectively improve spectrum utilization and throughput.

[0109] Example 2:

[0110] In the method for estimating the middle layer and precoding matrix of large-scale MIMO provided in Example 1, a reinforcement learning model is used for deep reinforcement learning. This example provides specific configuration methods and parameters for some available neural network models as reinforcement learning models used in Example 1. In actual implementation, the following parameters can be referenced to select and configure specific neural network models according to actual needs.

[0111] Example 1 uses a deep reinforcement learning model to learn layer selection and pre-encoding matrices. In reinforcement learning, after an agent interacts with the environment and takes an action, the current state becomes the state at the next time step, and the agent receives a reward. If the action is positive, a higher reward is obtained, and the agent will increase the probability of choosing this action when encountering a similar environment again. Conversely, if the action is negative, a lower reward or a negative score is obtained, and the agent will decrease the probability of choosing this action when encountering a similar environment again.

[0112] In preferred embodiments, the Actor-Critic algorithm can be used to learn the optimal layer selection and precoding matrix, but it is not limited to this method. Actor-Critic is a player critic model that integrates the advantages of "policy-based learning" and "value function-based learning." The Actor is responsible for learning the policy, and the Q-function is responsible for scoring the learned policy; a higher score indicates a better policy learned by the Actor. During training, the parameters of Q-function are updated based on gradient ascent, then the parameters learned by Q-function are used to update the parameters of the Actor, and then the parameters of Q-function are used to update the parameters of the Actor again, until the entire network converges.

[0113] In specific implementation, the neural networks (NN) used in this invention can be used as the network for Actor and Q, which together form a large network. The input to the entire model is the UE channel matrix, and the output of the Actor is one of the action sets, i.e., a weight vector. In Example 1, the weight vector is R / PM. The output of Q is a score, and a higher score is better. In Example 1, the signal-to-interference-plus-noise ratio (SIR) or average throughput is used as the score. When the model converges, the algorithm can select a weight vector based on the current input (UE channel matrix) to make Q output a larger value. That is, the algorithm finds a layer number and precoding matrix based on the current network environment to maximize the network's reward (SIR or average throughput). During training, the parameters of Q are estimated first, and then Q is fixed to update the parameters of the Actor. The gradient ascent algorithm is used to update the Actor, i.e.

[0114] When calculating the action, a noise parameter method is used. That is, the Q-network parameters are added with noise. This generates actions, allowing the model to systematically learn actions instead of learning them randomly. The learned actions can then... maximize.

[0115] Furthermore, in the method provided in Example 1, since the reward in reinforcement learning is a long-term gain, representing the reward obtained by the current actor at time t+n+1 after choosing an action based on the current environment, the average throughput at both time t+1 and t+n+1 needs to be stored in the database when acquiring sample data. The advantage of doing this is that it makes it easier to retrieve data from time t to t+n+1 in batches from the database when training the model.

[0116] r at the current moment t Let r be the average network throughput after a data packet is transmitted from the source UE to the target UE, and r be the throughput at time t+n+1. t+n+1 Let Q(s) be the average network throughput after data packets are transmitted from the source UE to the destination UE. The objective function then is to make Q(s) equal to the average network throughput after data packets are transmitted from the source UE to the destination UE. t ,a t r close to the current time t r at time t+n+1 t+n+1 Adding time t+n+1 This is the target network, initialized to Q. During training, it needs to be fixed. The parameters are used to update the parameters of Q.

[0117] The reward in reinforcement learning is a crucial metric, and choosing an appropriate reward can help the model converge faster. In Example 1, the signal-to-interference-plus-noise ratio (SINR) of the network is preferably used as the reward. This is because reinforcement learning aims to maximize long-term gains after selecting an action, so the reward is the sum of all average throughputs over the next t+n+1 steps.

[0118] Furthermore, to more accurately predict the impact of an action on future returns, a discount factor γ can be added. γ is a discount factor that represents the degree of impact of the current action on future returns. The current action has a smaller impact on nodes further away from the current time point. The value can be selected according to the actual situation, and is generally (0,1).

[0119]

[0120] r t: Represents the system's average throughput at the current time.

[0121] R t : Represents the system's long-term expected return

[0122] γ: Discount factor

[0123] The above formula can be used to calculate the reward and obtain the reward value under the current layer and precoding matrix for evaluation.

[0124] Specifically, such as Figure 7 As shown, the agent π learns knowledge by interacting with the environment. The Q-function is responsible for scoring the results learned by π, thereby learning a better π′, which is then assigned to π. This iterative learning process continues until convergence. During training, the parameters of Q are estimated first, and then Q is fixed while updating the parameters of π. The gradient ascent algorithm is used to update π, i.e. like Figure 8 As shown, both the actor and the Q-function are neural networks; in this embodiment, a convolutional neural network is used. The actor and Q-function form a single network. The actor's output is the value from the action set, i.e., the Rank and the Precoding Matrix. The Q-function's output is a scalar representing a score. When calculating actions, a noise parameter method is used. That is, the Q-network parameters are combined with noise. To generate actions. The actions learned at this point are what... The combination of the maximum number of layers and the precoding matrix.

[0125] The following provides a set of available model training methods and parameters. In actual use, the steps and parameters can be adjusted as needed.

[0126] The model's hyperparameters are configured as follows: learning rate 0.001; batch size 32; optimizer adam; kernel size 3*3.

[0127] The training method is as follows:

[0128] (1) Initialize the Q-function Q and the target Q-function make

[0129] Initialize actorπ and target actor make

[0130] (2) In each episode, for each time step t:

[0131] Based on actorπ, according to the current state s t Make an action a t , that is, a t =π(s) t ).

[0132] Earn Reward t And enter a new state s t+1 .

[0133] Storage (s) t ,a t ,r t ,s t+1 ) into memory.

[0134] Retrieve in batches from memory (s t ,a t ,r t ,…,s t+N ,a t+N ,r t+N ,s t+N+1 Multiple sample data.

[0135] Let target

[0136] Update the parameters of Q so that Q(s) t ,π(s t )) is close to y.

[0137] Update the parameters of π to maximize Q(s) t ,π(s t )),

[0138] After every C time steps, reset

[0139] The above method can be used to train a deep reinforcement learning model, enabling it to estimate the number of layers and the precoding matrix based on the input.

[0140] By selecting and setting the parameters of the neural network model described above, the reinforcement learning model required in Example 1 can be obtained, and a deep reinforcement learning-based layer and precoding matrix estimation model can be established and trained.

[0141] Example 3:

[0142] Based on the methods for estimating the mid-layer and precoding matrix of large-scale MIMO provided in Embodiments 1 and 2 above, the present invention also provides a system for estimating the mid-layer and precoding matrix of large-scale MIMO that can be used to implement the above methods, such as... Figure 9 The diagram shown is a schematic representation of the system architecture of an embodiment of the present invention.

[0143] The system for estimating the middle layer and precoding matrix in large-scale MIMO in this embodiment includes a base station and at least one UE. One agent of a reinforcement learning model corresponds to one base station, and the reinforcement learning model is deployed on the base station.

[0144] Each UE obtains state data by interacting with the base station, inputs the matrix into the reinforcement learning model trained on the base station, and obtains R / PM according to the method for estimating the mid-layer and precoding matrix of large-scale MIMO provided in Example 1 or Example 2. Specifically, the base station and UE obtain the UE channel matrix through message interaction, and then calculate the current network state data, i.e., (s t The base station sends the channel matrix to the AI ​​model, which then calculates the optimal Rank / Precoding Matrix (a). t ), and send it to the UE via DCI message. The UE, according to R / PM(a t Precoding is performed, and the base station obtains the UE's precoded state data for the next time step. Specifically, the UE channel matrix for the next time step, i.e., s, is calculated. t+1 And calculate the network signal-to-interference-plus-noise ratio (r) at the next time step. t The base station will [s] t ,a t ,r t ,s t+1 The data is stored in the database, the training dataset is iteratively updated, and the model is retrained using the updated training dataset. At this point, the entire system completes the closed loop of data acquisition and iteration.

[0145] In the system provided in this embodiment, combined with the methods provided in Embodiments 1 and 2, the complete time series of data acquisition and estimation includes two major iterations: the first is the data acquisition iteration, and the second is the AI ​​algorithm iteration.

[0146] (I) Data Acquisition Iteration

[0147] 1> The base station sends a CSI-RS signal to the UE.

[0148] 2> The UE sends its H, PMI, and RI to the base station via a report message. Here, h is the weight of an antenna subarray, PMI is the precoding matrix indicator, and RI (Rank Indicator) is the layer number indicator.

[0149] 3> The base station sends pre-coded data to the UE.

[0150] 4> The UE decodes the data based on the precoding matrix.

[0151] 5> Base station calculates downlink throughput.

[0152] 6> The base station stores [UE channel matrix, R, PM, signal-to-interference-plus-noise ratio, UE channel matrix at the next moment] in the database.

[0153] By iterating through the above steps, the base station continuously stores the collected data into the database.

[0154] (II) AI Algorithm Iteration. AI algorithm iteration includes GAN training iteration, deep reinforcement learning iteration, and online application iteration. The following section will use the downlink MIMO FDD mode as an example.

[0155] 1. Training iteration of GAN

[0156] 1> Use the data in the database to train the GAN model.

[0157] 2> After the GAN converges, the Generator's output is stored in the database. The format of the Generator's output data is [channel matrix, R, PM, signal-to-interference-plus-noise ratio, channel matrix at the next time step].

[0158] By repeating this process, GAN will generate more training data.

[0159] 2. Iterative Deep Reinforcement Learning

[0160] 1> Use big data from the database [UE channel matrix, R, PM, signal-to-interference-plus-noise ratio, UE channel matrix at the next moment] to train a deep reinforcement learning model in a real-time learning system.

[0161] 2> Iteratively update the model parameters.

[0162] This process is repeated iteratively to complete the training iteration of the reinforcement learning model.

[0163] 3. Online application iteration

[0164] 1> In practical applications, the base station sends CSI-RS messages to the UE.

[0165] 2> The UE sends its channel matrix to the base station via the report message.

[0166] 3> The base station sends the UE's channel matrix to the real-time learning system via the Get_R_PM request message.

[0167] 4> The real-time learning system calls the previously trained AI algorithm to output R and PM, and sends the optimal Rank and PM to the base station through the Get_R_PM response message.

[0168] 5> The base station sends the optimal Rank and PM to the UE via DCI messages.

[0169] 6> The base station sends pre-coded data to the UE.

[0170] 7> The UE decodes the data based on the precoding matrix.

[0171] 8> Base station calculates downlink throughput.

[0172] 9> The base station stores the [channel matrix, R, PM, signal-to-interference-plus-noise ratio, channel matrix at the next moment] into the database to increase the training dataset in the database.

[0173] This process is repeated iteratively, using the trained reinforcement learning model to obtain the optimal number of layers and precoding matrix.

[0174] Training data is generated iteratively in this way, the model is trained iteratively, the online application is iteratively updated, and the results are then fed back to the database. This process continues until the entire system is stable.

[0175] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for estimating the mid-layer and precoding matrix in large-scale MIMO, characterized in that, Specifically, it includes: Each base station acquires its own and the UE's state data, and uses the state data as a training dataset to train the reinforcement learning model. The state data includes the UE channel matrix, R / PM, signal-to-interference-plus-noise ratio (SIN / NOT), and the UE channel matrix at the next time step. The reward of the reinforcement learning model is the network's SIN / NOT. The trained reinforcement learning model is deployed on each base station. Each base station obtains the current state data through interaction with the UE. The reinforcement learning model on the base station calculates the optimal R / PM based on the UE channel matrix. The UE performs precoding based on the R / PM transmitted by each base station. Each base station obtains the UE channel matrix for the next time step based on the precoded network state, iteratively updates the training dataset, and retrains the model using the updated training dataset.

2. The method for estimating the mid-layer and precoding matrix in large-scale MIMO according to claim 1, characterized in that, Each base station acquires its own and the UE's status data, specifically including: The base station obtains the UE channel matrix through interactive messages, and generates all available layers and precoding matrices as an action set based on the UE channel matrix. The base station sends the R / PM to the UE via DCI and calculates the network's signal-to-interference-plus-noise ratio based on the received feedback from the UE. The base station randomly selects a set of layers and precoding matrices from the action set as the current action, and calculates the UE channel matrix for the next moment based on the current action.

3. The method for estimating the middle layer and precoding matrix in large-scale MIMO according to claim 1, characterized in that, The step of using state data as a training dataset to train the reinforcement learning model specifically includes: The base station uses each set of [UE channel matrix, action set, network signal-to-interference-plus-noise ratio, and UE channel matrix at the next moment] as training data for the reinforcement learning model. The current UE channel matrix is ​​used as the state for reinforcement learning, the signal-to-interference-plus-noise ratio (SINR) is used as the reward, and the number of layers and the precoding matrix are used as the action. A deep reinforcement learning model is used to select layers and calculate the precoding matrix.

4. The method for estimating the middle layer and precoding matrix in large-scale MIMO according to claim 1, characterized in that, The step of using state data as a training dataset to train the reinforcement learning model also includes: The reinforcement learning model selects a layer number and a precoding matrix based on the current input UE channel matrix to maximize the network's signal-to-interference-plus-noise ratio (SINR). During training, the parameters of the Q function are estimated first, then the parameters of the Q function are fixed, and the parameters of the Actor are updated using the gradient ascent algorithm.

5. The method for estimating the middle layer and precoding matrix in large-scale MIMO according to claim 1, characterized in that, The step of using state data as a training dataset to train the reinforcement learning model also includes: The output of the Q function at the current moment is the signal-to-interference-plus-noise ratio (SIR) of the network after the data packet is transmitted from the source UE to the target UE at the current moment; The output of the Q function at the next moment is the signal-to-interference-plus-noise ratio of the network after the data packet is transmitted from the source UE to the target UE at the next moment.

6. The method for estimating the middle layer and precoding matrix in large-scale MIMO according to claim 1, characterized in that, Each base station obtains current state data through interaction with the UE. The reinforcement learning model on the base station calculates the optimal R / PM based on the UE channel matrix, specifically including: The base station and the UE obtain the UE channel matrix through message exchange and calculate the current network status data; The base station sends the UE channel matrix to the reinforcement learning model, which calculates the optimal R / PM and sends it to the UE via DCI message.

7. The method for estimating the middle layer and precoding matrix in large-scale MIMO according to claim 1, characterized in that, Each base station obtains its current status data through interaction with the UE, specifically including: The base station sends a CSI-RS message to the UE, and the UE sends its own UE channel matrix to the base station via a report message.

8. The method for estimating the middle layer and precoding matrix in large-scale MIMO according to claim 1, characterized in that, The reinforcement learning model on the base station calculates the optimal R / PM based on the UE channel matrix, specifically including: The base station sends the UE's channel matrix to the real-time learning system. The real-time learning system calls the trained reinforcement learning model to output R / PM and sends R / PM back to the base station.

9. The method for estimating the middle layer and precoding matrix in large-scale MIMO according to claim 1, characterized in that, Also includes: The data in the training dataset is used to train the GAN model. The data after the GAN model converges is added to the training dataset. The output data structure of the GAN model is the status data of the base station.

10. A system for estimating the middle layer and precoding matrix in large-scale MIMO, characterized in that, Includes a base station and at least one UE, specifically: Each UE obtains state data by interacting with the base station, and inputs the UE channel matrix in the state data into the reinforcement learning model trained on the base station, and obtains R / PM by the method of estimating the middle layer and precoding matrix of large-scale MIMO according to any one of claims 1-9; The base station sends the R / PM to the UE, and the UE performs precoding based on the R / PM; The base station obtains the UE channel matrix for the next time step after UE precoding, iteratively updates the training dataset, and retrains the model using the updated training dataset.

Images