A deep reinforcement learning based pilot allocation method for cell-free massive multiple-input multiple-output system
Patent Information
- Application Number
- CN202610790474.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-03
- Publication Date
- 2026-09-15
Smart Images
Figure CN122764451A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of pilot allocation technology, and specifically to a pilot allocation method for cellular-free large-scale multiple-input multiple-output systems based on deep reinforcement learning. Background Technology
[0002] Cellular-free massive MIMO systems have become a transformative solution for next-generation wireless networks. Unlike traditional cellular systems constrained by cell boundaries, this new architecture achieves significant macro-diversity gain and improved service quality by having multiple access points collaboratively serve each user equipment. Coherent operation relies on obtaining accurate channel state information, a process primarily dependent on uplink channel estimation. However, practical applications of channel estimation face a thorny problem: the finite length of pilot sequences and the large number of user equipment lead to the reuse of pilot signals, inevitably introducing pilot pollution into the channel estimation process. When multiple user equipment use the same pilot sequence, mutual interference occurs, resulting in a degraded overall network performance.
[0003] Pilot allocation is a key technology for suppressing severe physical layer interference. By assigning orthogonal pilot signals to user equipment with similar spatial characteristics, pilot allocation can effectively avoid the inherent strong interference problems in pilot reuse scenarios. Currently, pilot allocation methods are mainly divided into two categories: model-based methods and learning-based methods.
[0004] Model-based methods utilize mathematical principles for optimization; for example, interference sensing methods suppress interference by differentiating between different user devices. While statistical analysis of the number of shared access points validates the superior performance of this method, the intelligent algorithm-based approach suffers from excessively high computational complexity.
[0005] Reference [1] proposes a matching-based solution: first, the user equipment is divided into several groups through a matching mechanism, and then orthogonal pilot signals are allocated according to the proportion of shared access points in each group. This matching scheme also faces significant computational complexity challenges.
[0006] Recent advances in deep learning have driven the development of learning-based methods in wireless network resource management. Early research on physical access has been reported in non-cellular massive MIMO systems. Reference [2] models pilot allocation as a graph cut problem and proposes a spectrum clustering method to separate user equipment that may suffer severe physical interference. Reference [3] proposes a multi-agent deep reinforcement learning method for pilot allocation. Despite significant achievements, these learning-based methods still face two major challenges: first, the problem of generalization ability, i.e., the performance of the method when dealing with diverse data samples. Such learning methods usually perform well in specific scenarios, but their performance degrades in other scenarios; second, the problem of scalability, which limits their application in large-scale network configurations. Summary of the Invention
[0007] To address the aforementioned problems, this invention aims to provide a pilot allocation method for cellular-free large-scale multiple-input multiple-output systems based on deep reinforcement learning, which offers good generalization ability and scalability.
[0008] The main idea of the technical solution adopted in this invention is as follows: A novel deep reinforcement learning algorithm framework is proposed, redefining pilot allocation as a sequential decision-making process. This framework models pilot allocation as a reverse clustering problem and optimizes it using a deep Q-network. A reverse clustering mechanism based on Euclidean distance is introduced, driven by a single deep Q-network. This deep Q-network is applied iteratively to each user device, allocating pilots using only the geometric information of the user device and the dynamic cluster centroid. A customized reward function is designed to maximize intra-cluster separation while balancing cluster size, thereby directly alleviating the physical access problem.
[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0010] A pilot assignment method for cellular-free large-scale multiple-input multiple-output systems based on deep reinforcement learning includes the following steps:
[0011] Obtain the two-dimensional location coordinates of each user device;
[0012] Based on two-dimensional location coordinates, user equipment is subjected to balanced inverse clustering to obtain several clusters;
[0013] A pilot allocation model is constructed based on a deep Q-network, and the decision mechanism of the pilot allocation model is defined: the pilot allocation model takes the two-dimensional coordinates of the user equipment to be allocated, the centroid position of each cluster, and the current size of each cluster as the state of the model input, and the action of allocating the user equipment to be allocated to a specific cluster as the action of the model output.
[0014] Define a reward function that calculates a reward value based on the minimum distance between user devices within a specific cluster and the balance of all cluster sizes after assigning the current user device to that cluster.
[0015] The pilot allocation model is trained using state, action, and reward function to obtain the target pilot allocation model. In each training round, the model outputs an action based on the current state, performs allocation, receives a reward calculated based on the reward function, and then updates the model parameters.
[0016] Using the target pilot allocation model, clusters are assigned to all user equipment in sequence, and the same pilot sequence is assigned to user equipment belonging to the same cluster, while orthogonal pilot sequences are assigned to user equipment belonging to different clusters.
[0017] Furthermore, based on the above technical solution, balanced back clustering of user equipment is performed using two-dimensional location coordinates to obtain several clusters. The expression for balanced back clustering is as follows:
[0018]
[0019] in, Indicates the first Groups, Indicates user equipment and The Euclidean distance between them To balance the penalty factor, Indicates the first The size of each group, The expected size for each group, The number of modulation symbols, This refers to the number of user devices.
[0020] The above technical solution is further characterized by:
[0021] The pilot assignment model defines the tuple as follows: ,in These represent the state, action, reward, and next state, respectively.
[0022] Pilot assignment models obtain state by observing the environment. Then perform the corresponding action. The environment is a pilot assignment model that returns a reward. And move to the next state. .
[0023] Based on the above technical solution, the expression for the state is further as follows:
[0024]
[0025] in, For decision-making steps state, For the first The normalized position of the centroid of each cluster, For the first The normalized size of each cluster, For the first Normalized location of each user device.
[0026] Through the above technical solutions, further: Action Used to determine the first The cluster to which each user device belongs:
[0027]
[0028] in, It is the set of behaviors of an intelligent agent.
[0029] Based on the above technical solution, the reward function expression is further as follows:
[0030]
[0031] in For instant rewards, For the next state The corresponding clustering results They are respectively Two user devices in the system.
[0032] Further, using the above technical solution, the pilot allocation model is trained using state, action, and reward function to obtain the target pilot allocation model; wherein, in each round of training, the model outputs an action based on the current state, performs allocation, receives a reward calculated based on the reward function, and then updates the model parameters, wherein:
[0033] The pilot assignment model outputs a Q-value, and the basic training strategy is to select actions that maximize the Q-value.
[0034] Here, the Q value represents the expected cumulative reward that can be obtained by assigning the current user device to a certain cluster.
[0035] Based on the above technical solutions, the expression for the Q value is further as follows:
[0036]
[0037] The optimal Q value represents the state. Execute action The maximum expected cumulative reward that can be obtained afterward; As a discount factor, For the next action.
[0038] A pilot allocation system for a non-cellular large-scale multiple-input multiple-output system based on deep reinforcement learning, used to implement the method described above, includes: an information acquisition module for acquiring the two-dimensional position coordinates of each user device;
[0039] The balanced inverse clustering module performs balanced inverse clustering of user equipment based on two-dimensional location coordinates to obtain several clusters;
[0040] The model building module constructs a pilot allocation model based on a deep Q-network and defines the decision mechanism of the pilot allocation model: the pilot allocation model takes the two-dimensional coordinates of the user equipment to be allocated, the centroid position of each cluster, and the current size of each cluster as the state of the model input, and assigns the user equipment to be allocated to a specific cluster as the action of the model output.
[0041] The reward design module is used to define the reward function. The reward function calculates the reward value based on the minimum distance between user devices in a specific cluster and the balance of all cluster sizes after the current user device is assigned to a specific cluster, so as to guide the training of the pilot allocation model.
[0042] The training module is used to train the pilot allocation model using states, actions, and a reward function to obtain the target pilot allocation model. In each round of training, the model outputs an action based on the current state, performs allocation, receives a reward calculated based on the reward function, and then updates the model parameters. The pilot allocation execution module is used to allocate clusters to all user equipment in sequence according to the target pilot allocation model, allocate the same pilot sequence to user equipment belonging to the same cluster, and allocate orthogonal pilot sequences to user equipment belonging to different clusters.
[0043] The beneficial effects of this invention are as follows: Through detailed numerical simulation and comparative analysis, this invention achieves significant advantages over existing technologies (such as interference sensing, matching, and spectral clustering), mainly in the following aspects:
[0044] Significantly improved channel estimation accuracy: Under different access point selection thresholds (0.3, 0.5, 0.7) and different numbers of user equipment (K=40, 60, 80), the method of this invention achieved the lowest normalized mean square error (NMSE) in most cases. This means more accurate channel estimation, laying a solid foundation for subsequent data transmission. Higher channel estimation accuracy directly translates to lower pilot pollution.
[0045] Effectively Improves System Spectrum Efficiency: Through cumulative distribution function analysis, the method of this invention demonstrates highly competitive performance in terms of user equipment spectrum efficiency. Statistical results confirm that this method can consistently improve the average spectrum efficiency of user equipment. Spectrum efficiency is a core indicator for measuring wireless network throughput, and its improvement directly translates into increased network capacity and improved user experience.
[0046] Excellent generalization ability and scalability: Simulation results verify that the proposed method maintains stable and superior performance under different network sizes (K=40, 60, 80) and different system parameters (different access point selection thresholds), solving the problem of performance degradation of existing learning-based methods when faced with diverse network configurations. This makes the invention highly practical and adaptable to the dynamic changes in the number of users and the complex and diverse environments in real-world networks.
[0047] Low complexity and real-time performance: On a regular laptop, this method can process the pilot allocation problem for 40 user devices in about 0.01 seconds, which is much smaller than the time scale of user device location updates (2 seconds). This proves that the method not only has superior performance but also low computational complexity, which fully meets the requirements of online real-time operation of practical systems and has great potential for engineering applications. Attached Figure Description
[0048] Figure 1 A schematic diagram of the training process with different learning rates when K=40;
[0049] Figure 2 A comparative diagram showing the average NMSE when the access point threshold is set to 0.3;
[0050] Figure 3 A comparative diagram showing the average NMSE when the access point threshold is set to 0.5;
[0051] Figure 4 A comparative diagram showing the average NMSE when the access point threshold is set to 0.7;
[0052] Figure 5 A schematic diagram of the cumulative distribution function of the spectral efficiency of each user equipment when the access point threshold is selected as 0.3;
[0053] Figure 6 A schematic diagram of the cumulative distribution function of the spectral efficiency of each user equipment when the access point threshold is set to 0.5;
[0054] Figure 7 A schematic diagram of the cumulative distribution function of the spectral efficiency of each user equipment when the access point threshold is selected as 0.7. Detailed Implementation
[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0056] Example 1: A Pilot Allocation Method for Cellular-Free Large-Scale Multiple-Input Multiple-Output Systems Based on Deep Reinforcement Learning
[0057] See Figures 1-7 This application discloses a pilot allocation method for cellular-free large-scale multiple-input multiple-output systems based on deep reinforcement learning.
[0058] I. System Model
[0059] Consider a non-cellular massive MIMO system comprising L single-antenna access points and K single-antenna user equipment. All access points are connected to a central processing unit (CPU) for joint transmission. To simplify the model, assume an ideal fronthaul network between the CPU and the access points. The system operates in standard massive MIMO time-division duplex mode. Time-frequency resources are allocated based on coherent resource blocks, each containing... There are modulation symbols. Assume that among them... One symbol is used for channel estimation, and the remaining... These symbols are used for downlink payload data transmission. This invention focuses only on downlink performance analysis.
[0060] L represents the number of access points, and K represents the number of user devices. The number of modulation symbols within each coherent resource block. The number of modulation symbols used for channel estimation in each coherent resource block.
[0061] Access point With user equipment The channel between them is modeled as Rayleigh fading:
[0062]
[0063] in For the first The user equipment and the first Large-scale fading coefficients between access points describe path loss and shadow fading; Gaussian distributions are used to model small-scale fading (SSF) that describes signal multipath propagation.
[0064] 1.1 Channel Estimation
[0065] Assume there is modulation symbols ,satisfy and , for A vector with one row and one column; For the first One modulation symbol, express The amplitude is It can be used for pilot signal allocation. This represents the assignable pilot signal. During the channel estimation phase, all... Multiple user equipments simultaneously use the allocated pilot signals for transmission, access point The received superimposed signal is:
[0066]
[0067] in The pilot transmit power of the user equipment (assuming all user equipment transmits pilots at full power). To be allocated to user equipment pilot sequence, It is additive noise and follows a distribution. , For noise power, Access point Received superimposed signal, For user equipment and access point The channel between them, where T is the transpose, This indicates that it follows a complex Gaussian random distribution.
[0068] Access point Based on the received signal Channel estimation By right multiplication It can eliminate interference from other pilot sequences:
[0069]
[0070] in Indicates the use of pilot signals Pilot pollution originates from co-pilot interference between user equipment within a set of user equipment. For the first One modulation symbol, Indicates the received signal The part that has undergone conjugate decoding; Indicates the first One user device.
[0071] The minimum mean square error (MMSE) estimator is denoted as Its distribution is as follows:
[0072]
[0073] in, For the first The user equipment and the first The variance of channel estimation between access points.
[0074]
[0075] in, For the first The user equipment and the first Large-scale fading coefficients between access points.
[0076] Channel estimation error is defined as Its distribution is as follows:
[0077]
[0078] 1.2 Downlink Data Transmission
[0079] Access point Based on the estimated channel, payload data is sent to the user equipment it serves. Access point selection matrix, used to indicate access points Is it a user device? To provide services, that is, when its elements Time indicates access point For user equipment Service, otherwise User equipment The received signal can be represented as:
[0080]
[0081] in Access point Assigned to user equipment power, For beamforming vectors, To send to user equipment The expected signal satisfies , For noise, For user equipment The received signal.
[0082] Conjugate beamforming is used:
[0083]
[0084] in, For the first The user equipment and the first The conjugate transpose of the channel estimates between access points For the first The user equipment and the first The expectation of the square of the channel estimate between access points.
[0085] User equipment The spectral efficiency can be expressed as:
[0086]
[0087] This formula employs a "use and discard" technique, where the first... The signal-to-interference-plus-noise ratio of each user equipment is:
[0088]
[0089] The numerator of equation (10) represents the desired signal power, the first term of the denominator is the sum of interference from other user equipment, and the second term is the channel fluctuation caused by pilot pollution. Equation (10) shows that pilot allocation has a significant impact on the spectral efficiency of user equipment, specifically in two aspects: firstly, the numerator of equation (10), reasonable pilot allocation can reduce channel estimation errors. Increase the desired signal strength; secondly, the second term in the denominator of equation (10) can be reduced by reasonable pilot allocation.
[0090] The objective of this invention is to maximize the average spectral efficiency of user equipment by optimizing pilot allocation, as expressed mathematically:
[0091]
[0092] 2 Pilot Allocation Based on Deep Reinforcement Learning
[0093] This paper details a novel deep reinforcement learning pilot allocation framework suitable for non-cellular massive MIMO systems. The optimization problem of Equation (11) is reconstructed into a reverse clustering task, which sequentially divides user equipment into different clusters using a unified deep Q-network. The deep Q-network spatially separates user equipment that is prone to severe pilot pollution based on the two-dimensional coordinates of the user equipment, assuming that this coordinate information is available when needed.
[0094] 2.1 Problem Modeling
[0095] A common strategy to mitigate pilot pollution is to divide user equipment (UEs) into different groups. The basic principle is to assign orthogonal pilot sequences to UEs experiencing strong mutual interference. However, the effectiveness of such methods largely depends on the access point selection, which itself may not be optimal or have poor generalization performance under different network configurations.
[0096] To address this issue, this invention proposes separation using only the geometric information (i.e., two-dimensional coordinates) of the user equipment. Severe pilot pollution can easily occur between adjacent user equipment. From another perspective, sharing the same pilot sequence between user equipment that are physically far apart is expected to achieve better performance gains. Based on this, this invention models pilot allocation as a reverse clustering task, with the mathematical expression:
[0097]
[0098] in Indicates the first Groups, Indicates user equipment and The Euclidean distance between them. Since the set is empty, directly optimizing equation (12) may yield a trivial solution, meaning all user equipment is grouped into the same cluster. To avoid this problem, a cluster balancing penalty term is introduced into equation (12), resulting in:
[0099]
[0100] in To balance the penalty factor, Indicates the first The size of each group, The expected size for each group. Equation (13) models pilot assignment as a balanced inverse clustering task, which can simultaneously avoid trivial solutions and severe inter-group pilot contamination. However, since the objective function varies with the location of the user equipment, the optimization problem of Equation (13) has non-convexity and nonlinear solvability (NPhard) characteristics. A deep Q-network-based optimization method will be proposed later.
[0101] 2.2 Optimization Method Based on Deep Q-Network
[0102] Deep Q-networks are suitable for scenarios where agents make decisions based on Markov decision processes. For the reverse clustering problem in equation (13), this invention defines tuples... Modeling the agent-environment interaction process, where These represent the state, action, reward, and next state, respectively. The agent acquires its state by observing its environment. Then perform the corresponding action. The environment is where the agent returns a reward. And move to the next state. A pilot allocation model is constructed based on a deep Q-network. The pilot allocation model takes the two-dimensional coordinates of the user equipment to be allocated, the centroid position of each cluster, and the current size of each cluster as state inputs, and assigns the current user equipment to a specific cluster as the action output.
[0103] Intelligent agents are based on The update strategy is represented by the action to be executed. This strategy is obtained through experience training to ensure that the correct action is selected in each decision. To address the problem in equation (13), a sequential process is defined for reverse clustering: starting from any user device, iterative execution continues until all user devices are assigned to their respective clusters. In the step of processing the t-th user device... The intelligent agent collects current user device data (i.e., the first...) The pilot assignment model takes geometric and balance information as input and outputs a Q-value vector, where... Each output neuron (equivalent to a pilot signal) corresponds to a potential cluster. The Q-value estimated by the first neuron represents the allocation of the current user device to the first neuron. The expected cumulative reward that a cluster can obtain.
[0104] 2.2.1 Status
[0105] Decision-making steps status Includes the following information:
[0106]
[0107] in For the first The normalized position of the centroid of each cluster, For the first The normalized size of each cluster, For the first Normalized location of each user device. Location coordinates. The normalization method is ,in and Let be the length and width of the coverage area, respectively, to ensure that all spatial inputs of the pilot assignment model fall within the unit interval [0,1]. Equation (14) shows that the input dimension of the pilot assignment model is... .
[0108] State from arrive The transition is the current state, the action being performed, and the first... The deterministic function for the location of a user equipment is defined as:
[0109]
[0110] This is the state transition function.
[0111] in:
[0112]
[0113] 2.2.2 Actions
[0114] action Used to determine the first The cluster to which each user device belongs:
[0115]
[0116] Equation (17) shows that the output layer of the pilot assignment model contains One neuron. For the set of agent behaviors, the state definition in equation (14) and the action definition in equation (17) model the back-clustering as a Markov decision process, where the state... It provides comprehensive statistical information, capturing the complete dynamic state, including geometric information and load balancing across all clusters; actions A discrete control space is defined, directly controlling the state trajectory. These models transform inherently complex problems into easily tractable temporal decision-making tasks, and the pilot assignment model can efficiently learn the optimal policy through the reward function defined below.
[0117] 2.2.3 Rewards
[0118] The reward function, by integrating the minimum distance within the selected cluster and the balance penalty term, directly corresponds to the optimization objective of equation (12):
[0119]
[0120] in , For immediate reward. It should be noted that the calculation of equation (18) is based on the next state. Corresponding clustering results .
[0121] The pilot assignment model is learned through an experience replay mechanism. The basic strategy of the agent is to select the action that maximizes the Q value, and the mathematical expression is:
[0122]
[0123] This indicates the optimal action.
[0124] The Q-value integrates immediate reward and expected future reward through the Bellman equation:
[0125]
[0126] in As a discount factor, The equation recursively defines the optimal Q value. , indicating from state Execute action The maximum expected cumulative reward that can be obtained afterward.
[0127] The learning process of the pilot assignment model is to... The approximation is given by denoting the pilot assignment model as follows: The intelligent agent iteratively collects interaction experience. And stored in the experience replay buffer. In this process, an independent target network is used. Estimating equation (20) item:
[0128]
[0129] Among them, the target network Every Each training cycle from Copy the parameters once. Therefore, The parameter update objective is to minimize the loss function, ultimately obtaining the target pilot assignment model.
[0130] loss function Based on the experience replay buffer The invention employs small-batch transfer sample calculations based on random sampling to break temporal correlation. Finally, to balance the trade-off between exploration and utilization, the invention adopts... A greedy strategy makes the agent perform actions:
[0131]
[0132] 2.2.4 Deep Q-Network Training Algorithm
[0133] The training process of a deep Q-network is as follows:
[0134] Input: User equipment location Small batch sample size .
[0135] Output: After training .
[0136] 1: Initialization , and experience replay buffer .
[0137] 2: Regarding training rounds .
[0138] 3: Randomly generated Cluster center.
[0139] 4: Regarding :
[0140] 5: Observation Status: .
[0141] 6: Execution Action: Adopt Greedy strategy.
[0142] 7: Rewards received:
[0143] 8: State transition: .
[0144] 9: Record experiences: Store to D.
[0145] 10: End the t loop.
[0146] 11: If :
[0147] 12: Learning: Update based on random mini-batch samples .
[0148] 13: Termination condition judgment.
[0149] 14: Synchronization: Every training rounds .
[0150] 15: Attenuation: .
[0151] 16: End the loop.
[0152] Example 2: Numerical Simulation
[0153] 3. In this embodiment, a CF M-MIMO system is configured within a 500m × 500m square area. 100 single-antenna access points and 40 single-antenna user equipments are independently and evenly distributed within this area, with a height difference of 1.5 meters between the access points and user equipments. The carrier frequency is set to 2GHz, the available bandwidth is 20MHz, the transmit power of both access points and user equipment is limited to 100mW, and the noise power is -94dBm. The moving speed of the user equipment within the area is 1.5m / s, and the duration of the coherent resource block is 50ms. The calculation yielded, where The wavelength is given. The large-scale fading coefficient remains constant within 2 seconds (40 times the coherence time), meaning the user equipment's location is updated every 2 seconds. Assume each coherent resource block contains... 1 symbol, of which One symbol is used for pilot assignment.
[0154] Large-scale fading coefficient The path loss and shadow fading are modeled using the model in reference [4]:
[0155]
[0156] in Access point With user equipment The distance between them For shadow fading, the variance is 4dB.
[0157] 3.1 Target Pilot Assignment Model Hyperparameter Settings
[0158] A deep Q-network is constructed using a multilayer perceptron, consisting of four fully connected layers. The dimensions of the input and output layers are as follows: and Both hidden layers contain 512 neurons. To mitigate overfitting, Dropout regularization with a dropout rate of 0.2 is used. Discount factor Set it to 0.5. Set the number to 50, and the training batch size to 64. Use dynamic... Greedy strategy It decays linearly from an initial value of 0.4 to a final value of 0.1.
[0159] 3.2 Training Process Analysis
[0160] First, we analyze the impact of the learning rate on the training process of the target pilot assignment model, such as... Figure 1 As shown, the cumulative reward curve with training rounds indicates that the learning rate affects both the convergence speed and the final performance. When the learning rate is 0.1, the agent exhibits instability and fails to converge. When the learning rate is 0.01, the performance improves but is still not optimal. When the learning rate is 0.001, the best result is achieved, with a stable performance improvement. However, when the learning rate is 0.0001, the convergence speed is slow due to insufficient policy updates.
[0161] In subsequent simulations, a learning rate of 0.01 was chosen because it showed a good balance between convergence speed and training stability.
[0162] 3.3 Performance Comparison Analysis
[0163] To evaluate the performance of the proposed pilot allocation method based on the target pilot allocation model, the normalized mean square error (NMSE) of channel estimation was used as the evaluation index. Three benchmark methods were selected for comparison: the interference-aware method in reference [5], the matching method in reference [1], and the spectral clustering method in reference [2]. To ensure the fairness of the comparison, the access point selection strategy in reference [1] was adopted.
[0164] Figure 2-4 The performance comparison results of average NMSE under different access point selection thresholds are presented. The performance comparison results of average normalized mean square error (NMSE) under different access point selection thresholds (0.3, 0.5, 0.7) show that the proposed method exhibits superior performance in most scenarios; at a threshold of 0.3, the proposed method achieves the lowest NMSE across all user equipment configurations (K=40, 60, 80); a similar performance advantage is observed at a threshold of 0.7; while at a threshold of 0.5, the matching-based method shows competitive results. and The proposed method achieves the minimum normalized mean square error at that time. Maintain an advantage.
[0165] Figure 5-7 It provides further insights into performance distribution, demonstrating its correlation with... Figure 2-4 The cumulative distribution function of the spectral efficiency per user equipment is presented. Analysis of the cumulative distribution function of the spectral efficiency of user equipment shows that, although the differences between the curves are not significant, statistical analysis confirms that the proposed deep Q-network method achieves competitive performance in terms of spectral efficiency of user equipment. The numerical results above verify the good generalization and scalability of the proposed method.
[0166] 3.4 Real-time Analysis
[0167] The proposed method is used on Huawei MateBooks equipped with Intel i7-1260P processors. The execution time for the pilot allocation problem is approximately 0.01s. Considering that the timescale for user equipment location updates is 2s, this result fully meets the requirements for real-time operation.
[0168] 4. Conclusion
[0169] This invention proposes a novel deep Q-network pilot allocation method suitable for CF M-MIMO systems. It cleverly reconstructs the complex pilot allocation problem into a reverse clustering task, using a unified deep Q-network to sequentially separate user equipment. To achieve efficient learning, a customized reward function is designed to maximize intra-cluster Euclidean distance while maintaining a balanced cluster size.
[0170] Extensive simulation results demonstrate that the proposed method achieves competitive performance under different network configurations, effectively addressing the issues of poor generalization and scalability in existing pilot allocation methods. It outperforms mainstream benchmark methods in both channel estimation accuracy (normalized mean square error) and spectral efficiency, and has a short execution time, meeting real-time operation requirements. This provides an efficient and scalable solution for pilot allocation in CFM-MIMO systems.
[0171] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
[0172] References:
[0173] [1]Y.Gao,H.Hu,J.Chen,X.Wang,X.ChuandJ.Zhang, "Amatching-basedpilotassignment algorithm for cell-free massive MIMO networks," IEEE
[0174] Trans. Veh. Tech., vol. 73, no. 1, pp. 1453-1457, Jan. 2024.
[0175] [2]L. Zhang, S. Yang and Z. Han, “Pilot assignment for cell-freemassive MIMO: A spectral clustering approach,” IEEE Wireless Commun. Lett., vol. 13, no. 1, pp. 243-247, Jan. 2024.
[0176] [3] MS Oh, AB Das, S. Hosseinalipour, T. Kim, DJ Love and CG
[0177] Brinton, “A Decentralized Pilot Assignment Algorithm for Scalable O
[0178] RAN Cell-Free Massive MIMO,” in IEEE J. Sel. Areas. Commun., vol.
[0179] 42, no. 2, pp. 373-388, Feb. 2024.
[0180] [4] Ozlem Tugfe Demir, Emil Bj “ ornson and Luca Sanguinetti,“Foundations of “
[0181] user-centric cell-free massive MIMO,” Foundations and Trends inSignal
[0182] Processing, vol. 14: no. 3-4, pp 162-472, 2021.
[0183] [5] S. K. Singh, Y. Chakravarthy, and A. K. Sah, “An interferenceaware
[0184] user-group pilot assignment for cell-free massive MIMO systems,” IEEE
[0185] Wireless Commun. Lett., vol. 12, no. 7, pp. 1169–1173,Jul. 2023.
Claims
1. A pilot allocation method for a cellular-free large-scale multiple-input multiple-output system based on deep reinforcement learning, characterized in that, Includes the following steps: Obtain the two-dimensional location coordinates of each user device; Based on two-dimensional location coordinates, user equipment is subjected to balanced inverse clustering to obtain several clusters; A pilot allocation model is constructed based on a deep Q-network, and the decision mechanism of the pilot allocation model is defined: the pilot allocation model takes the two-dimensional coordinates of the user equipment to be allocated, the centroid position of each cluster, and the current size of each cluster as the state of the model input, and the action of allocating the user equipment to be allocated to a specific cluster as the action of the model output. Define a reward function that calculates a reward value based on the minimum distance between user devices within a specific cluster and the balance of all cluster sizes after assigning the current user device to that cluster. The pilot allocation model is trained using state, action, and reward function to obtain the target pilot allocation model. In each training round, the model outputs an action based on the current state, performs allocation, receives a reward calculated based on the reward function, and then updates the model parameters. Using the target pilot allocation model, clusters are assigned to all user equipment in sequence, and the same pilot sequence is assigned to user equipment belonging to the same cluster, while orthogonal pilot sequences are assigned to user equipment belonging to different clusters.
2. The method according to claim 1, characterized in that: Based on two-dimensional location coordinates, balanced back clustering of user equipment is performed to obtain several clusters. The expression for balanced back clustering is as follows: ； in, Indicates the first Groups, Indicates user equipment and The Euclidean distance between them To balance the penalty factor, Indicates the first The size of each group, The expected size for each group, For the number of modulation symbols, For the number of user devices, It is an empty set.
3. The method according to claim 2, characterized in that: The pilot assignment model defines the tuple as follows: ,in These represent the state, action, reward, and next state, respectively. Pilot assignment models obtain state by observing the environment. Then perform the corresponding action. The environment is a pilot assignment model that returns a reward. And move to the next state. .
4. The method according to claim 3, characterized in that: The expression for the state is: ； in, For decision-making steps state, For the first The normalized position of the centroid of each cluster. For the first The normalized size of each cluster, For the first Normalized location of each user device.
5. The method according to claim 4, characterized in that: action Used to determine the first The cluster to which each user device belongs: ； in, It is the set of behaviors of an intelligent agent.
6. The method according to claim 5, characterized in that: The reward function expression is: ； in For instant rewards, For the next state The corresponding clustering results They are respectively Two user devices in the system.
7. The method according to claim 6, characterized in that: The pilot assignment model is trained using state, action, and reward function to obtain the target pilot assignment model. In each training round, the model outputs an action based on the current state, performs the assignment, receives a reward calculated based on the reward function, and then updates the model parameters. The pilot assignment model outputs a Q-value, and the training strategy is to select actions that maximize the Q-value. Here, the Q value represents the expected cumulative reward that can be obtained by assigning the current user device to a certain cluster.
8. The method according to claim 7, characterized in that: The expression for the Q value is: ； The optimal Q value represents the state. Execute action The maximum expected cumulative reward that can be obtained afterward; As a discount factor, For the next action.
9. A pilot allocation system for a cellular-free large-scale multiple-input multiple-output system based on deep reinforcement learning, characterized in that, The method for implementing any one of claims 1-8 includes: an information acquisition module, used to acquire the two-dimensional position coordinates of each user device; The balanced inverse clustering module performs balanced inverse clustering of user equipment based on two-dimensional location coordinates to obtain several clusters; The model building module constructs a pilot allocation model based on a deep Q-network and defines the decision mechanism of the pilot allocation model: the pilot allocation model takes the two-dimensional coordinates of the user equipment to be allocated, the centroid position of each cluster, and the current size of each cluster as the state of the model input, and assigns the user equipment to be allocated to a specific cluster as the action of the model output. The reward design module is used to define the reward function. The reward function calculates the reward value based on the minimum distance between user devices in a specific cluster and the balance of all cluster sizes after the current user device is assigned to a specific cluster, so as to guide the training of the pilot allocation model. The training module is used to train the pilot allocation model using state, action, and reward function to obtain the target pilot allocation model. In each round of training, the model outputs an action based on the current state, performs allocation, receives a reward calculated based on the reward function, and then updates the model parameters. The pilot allocation execution module is used to allocate clusters to all user equipment in sequence according to the target pilot allocation model, and to allocate the same pilot sequence to user equipment belonging to the same cluster, and to allocate orthogonal pilot sequences to user equipment belonging to different clusters.