Multi-cell massive MIMO joint user grouping and time-frequency resource allocation method and system
By adopting the joint user grouping and time-frequency resource allocation methods of deep reinforcement learning in large-scale MIMO systems, the problems of strong interference and high complexity of multiple cells are solved, performance improvement and computational complexity are achieved, and greedy methods are broken through Limited, achieving efficient resource allocation effect.
Patent Information
- Application Number
- CN202311155261.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-08
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2043-09-08
AI Technical Summary
In large-scale MIMO multi-cell transmission scenarios, there are problems of strong interference from multiple cells and high complexity in resource allocation. The system performance improvement brought by existing algorithms is limited and the computational complexity is high.
The multi-cell large-scale MIMO joint user grouping and time-frequency resource allocation method based on deep reinforcement learning is adopted. By building a user grouping network and a time-frequency resource allocation network, the optimization goal of multiple base stations is achieved and the computing complexity is reduced. Specific methods include using a fully connected network of deep Q learning to perform time-frequency resource allocation and user grouping decisions, considering the mutual influence of the status and actions of each base station, and optimizing resource allocation.
It realizes the reduction of computational complexity while ensuring system performance, and improves algorithm performance and computing real-time performance. Experiments show that the time-frequency resource allocation method reaches 93% of the exhaustive performance under fixed user grouping, and combines user grouping and time-frequency resources. The performance of the allocation method is 28% higher than that of the greedy method.
Smart Images

Figure CN117202357B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a multi-cell large-scale MIMO system resource scheduling technology based on reinforcement learning, and in particular to user grouping and time-frequency resource allocation, and belongs to the technical field of wireless communication. Background Art
[0002] Massive Multiple-Input Multiple-Output (MIMO) is a key technology of the Fifth Generation Mobile Communication Systems (5G). Massive MIMO systems can further exploit spatial resources by equipping base stations (BS) with a large number of antenna arrays, serving more users simultaneously on the same time-frequency resources, greatly increasing the number of supported users and the spectrum efficiency of the system. However, with a limited number of antennas, the spatial resolution of the base station is also limited. When the user's spatial position is close or there are many scatterers, resulting in indistinguishability, there is still serious interference between users, especially in multi-cell scenarios. The use of the same spectrum resources by users in each cell will further aggravate the interference between users and seriously affect the system performance. The base station allocates users with severe interference to different time-frequency resources through user grouping and time-frequency resource allocation, effectively reducing interference between users and improving system performance. Commonly used user scheduling algorithms include proportional fairness (PF), greedy and round robin (RR), but these algorithms only bring limited improvement in system performance; semi-orthogonal selection (SUS), signal leakage to noise ratio (SLNR) and maximum fairness (MF) methods can bring higher system performance, but the computational complexity increases significantly with the increase in the number of antennas.
[0003] Reinforcement Learning (RL) is a learning mechanism that learns how to map states to actions in order to obtain the maximum reward. The agent is not told which actions to take, but must try to discover which actions will produce the greatest reward. Deep Reinforcement Learning (DRL) is a combination of deep learning and reinforcement learning. It uses the perception ability of deep learning to solve the modeling problems of strategies and value functions; at the same time, it uses the decision-making ability of reinforcement learning to define problems and optimize goals. According to the characteristics of wireless networks, deep reinforcement learning is introduced into wireless networks, and the multi-cell user grouping and time-frequency resource allocation problems are modeled as intelligent decision-making problems. Reducing computational complexity through deep reinforcement learning methods and improving the real-time nature of decision-making are the main research topics of this invention. Summary of the invention
[0004] Purpose of the invention: In the multi-cell transmission scenario of massive MIMO, in order to address the problems of strong interference among multiple cells and high complexity of resource allocation, the purpose of the present invention is to provide a multi-cell massive MIMO joint user grouping and time-frequency resource allocation method based on reinforcement learning. Compared with existing similar technical means, this method can reduce computational complexity while ensuring performance.
[0005] Technical solution: In order to achieve the above-mentioned purpose, the present invention adopts the following technical solution: a multi-cell large-scale MIMO joint user grouping and time-frequency resource allocation method based on reinforcement learning, the method performs joint user grouping and time-frequency resource allocation based on a constructed user grouping network and a time-frequency resource allocation network to achieve a common optimization goal for multiple base stations; the user grouping network calculates the optimal user grouping matrix according to the system state; the time-frequency resource allocation network calculates the corresponding optimal time-frequency resource allocation matrix according to the system state after the optimal user grouping matrix is updated; the trained time-frequency resource allocation network is nested in the user grouping network to evaluate the optimization goals under different user groups; the system state includes all base station states, and the single base station state includes channel correlation information, user grouping information and time-frequency resource allocation information;
[0006] The time-frequency resource allocation network is a combination of multiple fully connected networks based on deep Q learning, wherein the parameters of each fully connected network are the same, the number of fully connected networks included in the time-frequency resource allocation network corresponding to each base station is the total number of time-frequency resource allocation actions of the base station, each fully connected network input layer includes the system state and the time-frequency resource allocation actions that may be taken by a single base station, and the output layer is the score of the corresponding time-frequency resource allocation action; the time-frequency resource allocation network is used to make parallel decisions on the best time-frequency resource allocation scheme for each base station;
[0007] The user grouping network is a fully connected network based on deep Q learning. The size of the user grouping network corresponding to each base station is determined by the number of users served by each base station. The input layer contains the system state, and the output layer is the score of all user grouping actions of this base station. The user grouping network is used to serially decide the optimal user grouping scheme for each base station. The system state when the next base station makes a decision is determined by the decision result of the previous base station. Each base station selects the action with the highest score to update the user grouping matrix. The user grouping matrix after the last base station makes a decision is the result of one round of decision.
[0008] Preferably, a user grouping network and a time-frequency resource allocation network are constructed in the central scheduler to perform joint user grouping and time-frequency resource allocation; each cell user sends an uplink detection signal, the base station estimates the user channel information, and calculates the channel correlation coefficient between each user; each base station transmits the channel correlation coefficient matrix to the central scheduler; the central scheduler uses the channel correlation coefficient to jointly perform user grouping and time-frequency resource allocation, and transmits the user grouping matrix and the time-frequency resource allocation matrix to each base station.
[0009] Preferably, users in each cell send uplink detection signals, and the detection signals of users in adjacent cells are orthogonal to each other; each base station estimates the channel information from users in its own cell and adjacent cells to the base station based on the received detection signals, calculates the channel correlation coefficients between users in its own cell and between users in its own cell and adjacent cells, and transmits them to the central scheduler; the central scheduler obtains the initial system state based on the channel correlation coefficients between users in each cell and the initial user grouping matrix.
[0010] Preferably, the time-frequency resource allocation network is composed of multiple fully connected networks, and the input layer of each fully connected network contains the system state and the time-frequency resource allocation action; the system state input to each fully connected network is the same, and the time-frequency resource allocation action is different, wherein the time-frequency resource allocation action is a time-frequency resource allocation scheme that a single base station may take; the output of each fully connected network approximates the score of the corresponding time-frequency resource allocation action taken by a single base station in the current state, wherein the score comprehensively considers the impact of future time-frequency resource allocation actions of other base stations.
[0011] Preferably, the score of a single base station taking a certain time-frequency resource allocation action in the current state is equal to the system optimization target value after the base station takes the action plus the product of the system optimization target value in the next state and the discount factor, wherein the discount factor is a real number greater than or equal to 0 and less than or equal to 1; the system optimization target value in the next state is the fixed user group and the time-frequency resource allocation action selected by the current base station, simulating j rounds of time-frequency resource allocation by other base stations, and calculating the system optimization target value under the new time-frequency resource allocation matrix, wherein j is a natural number greater than or equal to 1.
[0012] As a preference, a time-frequency resource allocation network is used to make parallel decisions on the optimal time-frequency resource allocation scheme for each base station. The initial system state of each base station input into the network is the same, and the combination of the highest-scoring actions of each base station is the optimal time-frequency resource allocation matrix.
[0013] Preferably, when the number of users served by each base station is the same, each base station reuses the same fully connected network when making decisions. The input layer of the fully connected network includes the system state, and the multiple outputs are scores of different user grouping actions taken by a single base station in the current state, wherein the user grouping action is a user grouping scheme that may be taken by a single base station; the scores of different user grouping actions are evaluated by calling the time-frequency resource allocation network.
[0014] Preferably, the score of a single base station for taking a user grouping action in the current state is equal to the reward after the base station takes the action plus the product of the next state score of the base station and the discount factor, wherein the reward is the system optimization target value calculated by the time-frequency resource allocation network after the base station takes the action in the current state; after serially simulating the decisions of all base stations, a new system state is obtained, the new system state is input into the user grouping network, and the maximum value of the network output is the next state score of the base station.
[0015] As a preference, the user grouping network is called to sequentially decide the best user grouping for each base station, and multiple rounds of decisions may be performed, with the decision result of the previous round being the initial user grouping matrix for the next round of decisions, until the best user grouping matrix is obtained.
[0016] Based on the same inventive concept, the present invention also provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, the reinforcement learning-based multi-cell large-scale MIMO joint user grouping and time-frequency resource allocation method is implemented.
[0017] Beneficial effects: The multi-cell large-scale MIMO joint user grouping and time-frequency resource allocation method based on reinforcement learning provided by the present invention, each small network in the time-frequency resource allocation network is used to calculate the corresponding score of each action, the network scale is small and the physical meaning is clear, and the convergence speed during training is fast. When training the user grouping network, the time-frequency resource allocation network calculation system is called to optimize the target value, and the score of each action is more accurate, thereby improving the performance of the algorithm. Further, the action scoring and selection of each base station in the time-frequency resource allocation network take into account the influence of the joint actions of other base stations, which can break through the limitations of the greedy method. The running time of the method of the present invention is much lower than that of the exhaustive method and the greedy method, which reduces the computational complexity and enhances the real-time performance of the computation. Experiments show that the time-frequency resource allocation method based on reinforcement learning can achieve 93% of the exhaustive performance under fixed user grouping conditions, and the joint user grouping and time-frequency resource allocation method based on reinforcement learning improves the performance by 28% relative to the greedy method. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 Block diagram for joint user grouping and time-frequency resource allocation based on reinforcement learning.
[0019] Figure 2 Block diagram of the time-frequency resource allocation algorithm based on reinforcement learning.
[0020] Figure 3 Block diagram of the user grouping algorithm based on reinforcement learning.
[0021] Figure 4 Performance comparison chart of time-frequency resource allocation algorithms.
[0022] Figure 5 Performance comparison chart of joint user grouping and time-frequency resource allocation algorithms. DETAILED DESCRIPTION
[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0024] like Figure 1 As shown, an embodiment of the present invention discloses a method for joint user grouping and time-frequency resource allocation for multi-cell large-scale MIMO based on reinforcement learning, which performs joint user grouping and time-frequency resource allocation based on a constructed user grouping network and a time-frequency resource allocation network, achieves a common optimization goal for multiple base stations, and effectively reduces the computational complexity compared to traditional algorithms. The user grouping network calculates the optimal user grouping matrix according to the system state; the time-frequency resource allocation network calculates the corresponding optimal time-frequency resource allocation matrix according to the system state updated by the optimal user grouping matrix; the trained time-frequency resource allocation network is nested in the user grouping network to evaluate the optimization goals under different user groups; the system state includes the states of all base stations, and the single base station state includes channel correlation information, user grouping information, and time-frequency resource allocation information.
[0025] The time-frequency resource allocation network is a combination of multiple fully connected networks based on deep Q learning, in which the parameters of each fully connected network are the same. The number of fully connected networks contained in the time-frequency resource allocation network corresponding to each base station is the total number of time-frequency resource allocation actions of this base station. The input layer of each fully connected network contains the system status and the time-frequency resource allocation actions that a single base station may take, and the output layer is the score of the corresponding time-frequency resource allocation action. The time-frequency resource allocation network is used to make parallel decisions on the optimal time-frequency resource allocation plan for each base station.
[0026] The user grouping network is a fully connected network based on deep Q learning. The size of the user grouping network corresponding to each base station is determined by the number of users served by each base station. The input layer contains the system state, and the output layer is the score of all user grouping actions of this base station. The user grouping network is used to serially decide the optimal user grouping scheme for each base station. The system state when the next base station makes a decision is determined by the decision result of the previous base station. Each base station selects the action with the highest score to update the user grouping matrix. The user grouping matrix after the last base station makes a decision is the result of one round of decision.
[0027] In specific scenarios, a user grouping network and a time-frequency resource allocation network can be constructed in the central scheduler to perform joint user grouping and time-frequency resource allocation; each cell user sends an uplink detection signal, the base station estimates the user channel information, and calculates the channel correlation coefficient between each user; each base station transmits the channel correlation coefficient matrix to the central scheduler; the central scheduler uses the channel correlation coefficient to jointly perform user grouping and time-frequency resource allocation, and transmits the user grouping matrix and the time-frequency resource allocation matrix to each base station.
[0028] Specifically, the time-frequency resource allocation network is composed of multiple fully connected networks, and the input layer of each fully connected network contains the system state and the time-frequency resource allocation action; the system state input to each fully connected network is the same, and the time-frequency resource allocation action is different, where the time-frequency resource allocation action is the time-frequency resource allocation scheme that a single base station may take; the output of each fully connected network approximates the score of the corresponding time-frequency resource allocation action taken by a single base station in the current state, where the score integrates the influence of the future time-frequency resource allocation actions of other base stations. The score of a single base station taking a certain time-frequency resource allocation action in the current state is equal to the system optimization target value after the base station takes the action plus the product of the next state system optimization target value and the discount factor; the next state system optimization target value is the fixed user group and the time-frequency resource allocation action selected by the current base station, simulating several rounds of time-frequency resource allocation by other base stations, and calculating the system optimization target value under the new time-frequency resource allocation matrix. The time-frequency resource allocation network is used to parallelly decide the optimal time-frequency resource allocation scheme for each base station. The initial system state of each base station input network is the same, and the combination of the actions with the highest score of each base station is the optimal time-frequency resource allocation matrix.
[0029] Specifically, when the number of users served by each base station is the same, each base station reuses the same fully connected network when making decisions. The input layer of the fully connected network contains the system state, and the multiple outputs are the scores of different user grouping actions taken by a single base station in the current state, where the user grouping action is the user grouping scheme that a single base station may take; the scores of different user grouping actions are evaluated by calling the time-frequency resource allocation network. The score of a single base station taking a certain user grouping action in the current state is equal to the reward after the base station takes the action plus the product of the next state score of the base station and the discount factor, where the reward is the system optimization target value calculated by the time-frequency resource allocation network after the base station takes the action in the current state; after serially simulating the decisions of all base stations, a new system state is obtained, and the new system state is input into the user grouping network, and the maximum value of the network output is the next state score of the base station. The central scheduler calls the user grouping network to sequentially decide the best user grouping of each base station, and multiple rounds of decisions can be made. The decision result of the previous round is the initial user grouping matrix of the next round of decisions until the best user grouping matrix is obtained.
[0030] The method of the embodiment of the present invention is described in detail below in conjunction with a specific scenario example. It should be noted that the multi-cell massive MIMO joint user grouping and time-frequency resource allocation method based on reinforcement learning of the present invention is not limited to this specific scenario example.
[0031] 1. System Configuration
[0032] Consider a multi-cell transmission scenario of massive MIMO-OFDM, where N base stations use L time-frequency resources to serve K user terminals. The base station set is denoted as The user set is denoted as The time-frequency resource set is denoted as Each base station is equipped with M transmitting antennas, and each user terminal is equipped with a single receiving antenna. Before optimizing resource allocation, users in each cell send uplink detection signals, and the detection signals of users in adjacent cells are orthogonal to each other; each base station estimates the channel information from users in the cell and adjacent cells to the base station based on the received detection signals, calculates the channel correlation coefficients between users in the cell and between users in the cell and adjacent cells, and transmits them to the central scheduler; the central scheduler obtains the initial system state based on the channel correlation coefficients between users in each cell and the initial user grouping matrix.
[0033] 2. Multi-frequency multi-cell channel model and user grouping and time-frequency resource allocation problem statement
[0034] The signal sent by the nth base station to user k on the lth time-frequency resource is The corresponding channel vector is The signal received by the kth user on the lth time-frequency resource is expressed as
[0035]
[0036] Among them, α n→k,l ∈{0,1} indicates whether the nth base station sends a signal to the kth user on the lth time-frequency resource. If it does, the value is 1, otherwise it is 0, and Ensure that every user can establish communication with the base station. k,l is additive Gaussian white noise with a mean of 0 and a variance of σ 2 The complex Gaussian distribution of . Send signal x n→k,l It can be expressed as
[0037] x n→k,l =p n→k,l w n→k,l s k (2) Among them, is the precoding vector, and its modulus is 1. k is a data signal, which satisfies a complex Gaussian distribution with a mean of 0 and a variance of 1. n→k,lis the transmit power, which must meet the base station's maximum transmit power constraint
[0038]
[0039] Therefore, formula (1) can be further written as
[0040]
[0041] SINR of user k on the lth time-frequency resource k,l It can be expressed as
[0042]
[0043] Then the rate of user k on the lth time-frequency resource is
[0044]
[0045] The system and rate are
[0046]
[0047] Consider the multi-frequency and multi-cell three-dimensional coordinated scheduling strategy to maximize the system and rate. Suppose the strategy matrix α n→k,l ∈{0,1}. For the convenience of calculation, the average power allocation p is used. n→k,l =p and maximum ratio transmit (MRT) precoding Where * represents vector conjugate transpose, and ||.|| represents vector modulus. The channel correlation coefficient between users is defined as
[0048]
[0049] Where T represents the vector transpose. The correlation coefficient can be used to simplify the rate of user k on the lth time-frequency resource to
[0050]
[0051] The optimization problem can be reformulated as follows
[0052]
[0053] The above problem is a nonlinear integer programming problem, which is usually difficult to solve. The traditional method to solve this problem is the heuristic search algorithm. However, most of these algorithms are inefficient, take a long time to run, and cannot be adjusted online in real time. A deep reinforcement learning algorithm will be used below to solve the above optimization problem. It should be noted that the optimization objectives of joint user grouping and time-frequency resource allocation can be quality of service (QoS), system energy efficiency, system spectrum efficiency, system sum rate, etc. This case takes the optimization objective of system sum rate as an example.
[0054] 3. Joint user grouping and time-frequency resource allocation method based on reinforcement learning
[0055] Base stations and communication networks are regarded as intelligent agents and environments, forming a multi-intelligence system. Since multiple base stations need to achieve the common goal of maximizing the system and rate through user grouping and time-frequency resource allocation, the base stations are in a "cooperative" relationship, that is, the achievement of the common goal is related to the joint behavior obtained by combining the behaviors of each individual. The strategy learning of the intelligent agent needs to consider the effect of the joint action, as well as the influence of other intelligent agents with decision-making capabilities. Therefore, a deep Q-learning network based on cooperation is used to solve the above problems.
[0056] In the multi-frequency multi-cell user grouping and time-frequency resource allocation problem, it is assumed that each base station serves s users and each base station estimates the user channel at time t. Then calculate the channel correlation coefficient according to formula (8) to obtain the correlation coefficient matrix Defining the Strategy Matrix And A t The user group matrix G t and the time-frequency resource allocation matrix V t Determine, among which represents the user scheduled by the nth base station, Corresponds to The time-frequency resources allocated to each user.
[0057] The period of interaction between the agent and the environment is T, and it is assumed that the channel remains unchanged within T. In a multi-frequency and multi-cell wireless communication network environment, reinforcement learning is used to solve the user grouping and time-frequency resource allocation problem. The proposed joint user grouping and time-frequency resource allocation algorithm based on reinforcement learning is shown in Figure 1 shown.
[0058] The agent, state, action, and reward functions are defined as follows:
[0059] Agent: n∈[1:N], which refers to each base station in the environment.
[0060] State: Single base station state represents the observation information of base station n at time t, It means that base station n interacts with the environment to obtain the observation information of other base stations at time t. The sum of all agent states is called the system state
[0061] Action: Single base station action represents the user scheduled by base station n at the next moment, express The time-frequency resource allocation of the corresponding user. The joint action of all agents
[0062] Reward Function: r(S t ,a t )=R t+1 , the reward is defined as the system in state S t Perform joint action a t After entering state s t+1 The sum rate of the system after
[0063] (1) Time-frequency resource allocation module
[0064] The time-frequency resource allocation module is applied to the user grouping algorithm as part of the system sum rate calculation. Its goal is to obtain the corresponding user time-frequency resource allocation matrix V that maximizes the system sum rate after given the user grouping information G. opt Since multiple base stations perform time-frequency resource allocation to achieve the common goal of maximizing the system and rate, the achievement of the common goal is related to the joint action obtained by combining the actions of each base station. A deep Q learning network can be used to solve the above problem.
[0065] The above single base station status Contains channel correlation information, user grouping information and time-frequency resource allocation information. When solving the time-frequency resource allocation, the user grouping information is fixed and the result is At this time, the action of each base station The total number of actions is L s , where L is the number of time-frequency resources and s is the number of users served by a single base station. t ,a t ), is the system performs joint action a t Then enter state S t+1 The system and rate R t+1 The proposed time-frequency resource allocation method is as follows: Figure 2 shown.
[0066] The time-frequency resource allocation module uses a combination of homogeneous fully connected networks to simulate Q value calculation. Each fully connected network outputs a score for an action. The action with the highest score is the time-frequency resource allocation solution v for a single base station. n,opt The channel correlation is calculated based on the channel information, and the current state of base station n is obtained by integrating the user grouping information and the initial time-frequency resource allocation information. If base station n takes the i-th action, the time-frequency resource allocation matrix is
[0067]
[0068] Where 1≤i≤L s , Indicates that the nth base station adopts the i-th time-frequency resource allocation scheme, Indicates the time-frequency resource allocation of other base stations. s indivual Corresponding to L s Time-frequency resource allocation matrix Get L s Strategy Matrix Defining Network Inputs
[0069]
[0070] in, u∈g n .Will The Q-value score of the action is calculated by inputting each fully connected network, where the number of neurons in the output layer of each fully connected network is 1 and has the same network parameter θ. The output of the fully connected network is defined as According to the meaning of Q value,
[0071]
[0072] Among them, γ∈[0,1], The user group is G, and the time-frequency resource allocation is The following systems and rates. Is fixed Simulate the time-frequency resource matrix after other base stations perform joint actions in time-frequency resource allocation in round j, then Where A′ n,i The user grouping matrix G and the new time-frequency resource allocation matrix Determine. DQN aims to search for the optimal parameter θ to minimize the loss, that is
[0073]
[0074] The gradient of θ is expressed as
[0075]
[0076] (2) User Grouping Module
[0077] In the user grouping algorithm, the trained time-frequency resource allocation network is part of the calculation system and rate module. The purpose is to obtain the maximum system and rate under different groups and better evaluate the performance of different user groups. Multiple base stations perform user grouping to achieve the common goal of maximizing the system and rate. The achievement of the common goal is related to the joint action obtained by combining the actions of each base station. The deep Q learning network can be used to solve the above problem.
[0078] Single base station status Contains channel correlation information, user grouping information and time-frequency resource allocation information. Optimal time-frequency resource allocation V t By G t and channel information. Therefore, the single base station status can be simplified Actions of each base station The total number of actions is Where K is the total number of users and s is the number of users served by a single base station. The proposed user grouping method is as follows Figure 3 shown.
[0079] In actual environments, the user location and the distance from the base station have a huge impact on the user grouping. The number of base stations that jointly schedule users in the same area is small, so the serial scheduling method can be used. The channel correlation is calculated based on the channel information, and the current state of base station n is obtained by integrating the user grouping information. Defining Network Inputs
[0080]
[0081] Where c is the set of integers greater than or equal to 0 and less than n, and b is the set of integers greater than n and less than N. Input DQN network to calculate the score q of each action i . Define the network output as where φ n represents the user group network parameters corresponding to the nth base station. The action with the highest score selected by base station n Thus, we get the new user grouping matrix Status of base station n+1 By the channel correlation ρ t and Determine, and repeat the above process until N base stations complete the action selection. Network input of the next state of base station n award is to perform a joint action t After getting Then the maximum system sum rate under this user group is calculated by the time-frequency resource allocation module In order to ensure that every user can establish communication with the base station,
[0082]
[0083] In DQN network training, experience replay is used to avoid instability. The base station state, action, reward and next moment state are recorded in the form of Stored in the experience replay pool The quasi-static target network method is also applied in network training. Two independent networks are defined for training, with parameters φn The training network and parameters are φ n,target Target network. In each iterative update, the target network is used to calculate the target label
[0084]
[0085] Minimizing the mean square Bellman error based on experience replay pool
[0086]
[0087] Using the gradient descent method, Random sampling from And use
[0088]
[0089] Update φ n .
[0090] 4. Implementation Effect
[0091] In order to enable those skilled in the art to better understand the scheme of the present invention, the following is a performance demonstration and computational complexity comparison of the user grouping and time-frequency resource allocation method based on reinforcement learning and the existing scheduling method in this embodiment under a specific system configuration. The simulation uses the QuaDRiGa platform to generate channels. The system is configured with the number of base stations N = 3, the number of antennas on the base station side M = 64, and the number of time-frequency resources L = 2. The total number of users K = 12, each base station serves s = 3 users, and each user device is equipped with a single antenna.
[0092] Table 1 shows the running time results of the joint user grouping and time-frequency resource allocation method based on DQN, the greedy method, and the exhaustive method in this embodiment.
[0093] Table 1. Comparison of running time
[0094]
[0095] It can be seen that the running time of the joint user grouping and time-frequency resource allocation method based on DQN is much lower than that of the other two methods, which reduces the computational complexity and enhances the real-time performance of the computation.
[0096] Figure 4 The performance results of the time-frequency resource allocation method based on reinforcement learning, the greedy time-frequency resource allocation method, the exhaustive time-frequency resource allocation method, and the random time-frequency resource allocation method are compared in the case of fixed user grouping in this embodiment. The proposed method is better than the greedy method and can achieve 93% of the exhaustive performance. This is because the action scoring and selection of each base station in the proposed time-frequency resource allocation method considers the influence of the joint actions of other base stations, breaking through the limitations of the greedy method.
[0097] Figure 5 The performance comparison of the joint user grouping and time-frequency resource allocation method based on reinforcement learning, the greedy user grouping and the time-frequency resource allocation method based on DQN, and the greedy user grouping and the greedy time-frequency resource allocation method in this embodiment is given. The proposed joint user grouping and time-frequency resource allocation method based on reinforcement learning is significantly better than the greedy method, with a performance improvement of 28% relative to the greedy method.
[0098] Based on the same inventive concept, an embodiment of the present invention also discloses a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the reinforcement learning-based multi-cell large-scale MIMO joint user grouping and time-frequency resource allocation method is implemented.
[0099] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.
Claims
1. A multi-cell massive MIMO joint user grouping and time-frequency resource allocation method based on reinforcement learning, characterized in that: The method performs joint user grouping and time-frequency resource allocation based on the constructed user grouping network and time-frequency resource allocation network to achieve a common optimization goal for multiple base stations; The user grouping network calculates the best user grouping matrix according to the system state; the time-frequency resource allocation network calculates the corresponding best time-frequency resource allocation matrix according to the system state after the best user grouping matrix is updated; The trained time-frequency resource allocation network is nested in the user grouping network to evaluate the optimization objectives under different user groups; the system state includes all single base station states, and the single base station state includes channel correlation information, user grouping information and time-frequency resource allocation information; The time-frequency resource allocation network is a combination of multiple fully connected networks based on deep Q learning, wherein the parameters of each fully connected network are the same, the number of fully connected networks included in the time-frequency resource allocation network corresponding to each base station is the total number of time-frequency resource allocation actions of the base station, each fully connected network input layer includes the system state and the time-frequency resource allocation actions that may be taken by a single base station, and the output layer is the score of the corresponding time-frequency resource allocation action; the time-frequency resource allocation network is used to make parallel decisions on the best time-frequency resource allocation scheme for each base station; The user grouping network is a fully connected network based on deep Q learning. The size of the user grouping network corresponding to each base station is determined by the number of users served by each base station. The input layer contains the system state, and the output layer is the score of all user grouping actions of this base station. Use the user grouping network to serially decide the optimal user grouping scheme for each base station; The system state when the next base station makes a decision is determined by the decision result of the previous base station. Each base station selects the action with the highest score to update the user grouping matrix. The user grouping matrix after the last base station makes a decision is the result of one round of decision. A user grouping network and a time-frequency resource allocation network are constructed in the central scheduler to perform joint user grouping and time-frequency resource allocation; each cell user sends an uplink detection signal, the base station estimates the user channel information, and calculates the channel correlation coefficient between each user; each base station transmits the channel correlation coefficient matrix to the central scheduler; the central scheduler uses the channel correlation coefficient to jointly perform user grouping and time-frequency resource allocation, and transmits the user grouping matrix and the time-frequency resource allocation matrix to each base station.
2. According to the reinforcement learning-based multi-cell massive MIMO joint user grouping and time-frequency resource allocation method of claim 1, it is characterized in that: Each cell user sends an uplink detection signal, and the detection signals of users in adjacent cells are orthogonal to each other. Each base station estimates the channel information from users in its own cell and adjacent cells to the base station based on the received detection signals, calculates the channel correlation coefficients between users in its own cell and between users in its own cell and adjacent cells, and transmits them to the central scheduler. The central scheduler obtains the initial system state based on the channel correlation coefficients between users in each cell and the initial user grouping matrix.
3. The method for multi-cell massive MIMO joint user grouping and time-frequency resource allocation based on reinforcement learning according to claim 1, characterized in that: The time-frequency resource allocation network is composed of multiple fully connected networks, and the input layer of each fully connected network includes a system state and a time-frequency resource allocation action; the system state input to each fully connected network is the same, and the time-frequency resource allocation action is different, wherein the time-frequency resource allocation action is a time-frequency resource allocation scheme that a single base station may take; the output of each fully connected network approximates the score of the corresponding time-frequency resource allocation action taken by a single base station in the current state, wherein the score comprehensively considers the impact of future time-frequency resource allocation actions of other base stations.
4. The method for multi-cell massive MIMO joint user grouping and time-frequency resource allocation based on reinforcement learning according to claim 2, characterized in that: The score of a single base station taking a certain time-frequency resource allocation action in the current state is equal to the system optimization target value after the base station takes the action plus the product of the system optimization target value of the next state and the discount factor, where the discount factor is a real number greater than or equal to 0 and less than or equal to 1; the system optimization target value of the next state is the fixed user group and the time-frequency resource allocation action selected by the current base station, simulating j rounds of time-frequency resource allocation by other base stations, and calculating the system optimization target value under the new time-frequency resource allocation matrix, where j is a natural number greater than or equal to 1.
5. The method for multi-cell massive MIMO joint user grouping and time-frequency resource allocation based on reinforcement learning according to claim 1, characterized in that: The time-frequency resource allocation network is used to make parallel decisions on the optimal time-frequency resource allocation scheme for each base station. The initial system state of each base station input into the network is the same, and the combination of the highest-scoring actions of each base station is the optimal time-frequency resource allocation matrix.
6. The method for multi-cell massive MIMO joint user grouping and time-frequency resource allocation based on reinforcement learning according to claim 1, characterized in that: When the number of users served by each base station is the same, each base station reuses the same fully connected network when making decisions. The input layer of the fully connected network contains the system state, and the multiple outputs are the scores of different user grouping actions taken by a single base station in the current state, where the user grouping action is a user grouping scheme that a single base station may take; the scores of different user grouping actions are evaluated by calling the time-frequency resource allocation network.
7. The method for multi-cell massive MIMO joint user grouping and time-frequency resource allocation based on reinforcement learning according to claim 6, characterized in that: The score of a single base station taking a user grouping action in the current state is equal to the reward after the base station takes the action plus the product of the next state score of the base station and the discount factor, where the reward is the system optimization target value calculated by the time-frequency resource allocation network after the base station takes the action in the current state; after serial simulation of all base station decisions, the new system state is obtained, the new system state is input into the user grouping network, and the maximum value of the network output is the next state score of the base station.
8. The method for multi-cell massive MIMO joint user grouping and time-frequency resource allocation based on reinforcement learning according to claim 1, characterized in that: The user grouping network is called to sequentially decide the best user grouping for each base station. Multiple rounds of decisions can be made, and the decision result of the previous round is the initial user grouping matrix of the next round of decisions, until the best user grouping matrix is obtained.
9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is loaded into a processor, the method for multi-cell massive MIMO joint user grouping and time-frequency resource allocation based on reinforcement learning is implemented.
Citation Information
Patent Citations
Downlink multi-cell OFDMA resource allocation method based on multi-agent deep reinforcement learning
CN112601284A
Multi-user MIMO system user scheduling method based on deep reinforcement learning
CN115103372A
