Cellular-free large-scale MIMO high-energy-efficiency hybrid DRL power distribution and sleep control method and system

Abstract

The invention provides a cellular-free large-scale MIMO high-energy-efficiency hybrid DRL power distribution and sleep control method, which comprises the following steps that: each access point carries out channel estimation according to a received pilot signal, and returns the channel estimation to a central processing unit; the central processing unit gathers channel state information returned by all the access points, obtains an access point dormancy control decision and a downlink transmission power distribution coefficient through a deployed deep reinforcement learning agent based on a parameterized Markov decision process, and obtains the access point dormancy control decision and the downlink transmission power distribution coefficient through a discrete embedded table and a conditional variation auto-encoder; mapping the hybrid action to a low-dimensional potential representation space, decoding to obtain a reconstructed transmitting power distribution coefficient, and controlling the access point to perform downlink data transmission according to the hybrid action; the user calculates spectrum efficiency and uploads the spectrum efficiency to the intelligent agent; and the intelligent agent performs intelligent agent strategy updating by taking the total energy efficiency of the system as a reward, updates parameters of the conditional variation auto-encoder and the decoder, and returns to loop execution. The energy efficiency of the system is higher.

Description

TECHNICAL FIELD

[0001] The present application relates to wireless communication technology, in particular to a high-energy-efficiency hybrid DRL power allocation and sleep control method and system for cell-free massive MIMO. BACKGROUND

[0002] With the continuous evolution of the sixth generation mobile communication technology (6G), communication systems are facing multiple challenges of higher data rates, lower latency and higher energy efficiency. In order to achieve the goals of ultra-dense connection, global coverage and green communication, the large-scale MIMO system of cell-free architecture is widely considered as one of the key supporting technologies in 6G network. This architecture connects distributed access points (APs) with central processing units, and uses cooperative transmission technology to make all APs provide services for user equipment, thereby significantly improving the consistency of spectrum utilization and network coverage.

[0003] However, the cell-free massive MIMO system brings significant energy consumption problems while deploying a large number of APs to improve service capacity. On the one hand, some APs in the system continue to run in a low load or idle state, causing energy waste; on the other hand, if the power allocation and sleep control of the APs cannot be optimized cooperatively, it will lead to the intensification of the contradiction between spectrum efficiency and energy efficiency, and reduce the overall performance of the system. Some existing researches try to reduce energy consumption by turning off the APs that are not fully utilized, but most of the methods only optimize the on-off state (active state) or power allocation of the APs independently, ignoring the coupling relationship between the two, and it is difficult to achieve a globally optimal energy efficiency control strategy in a complex dynamic environment.

[0004] Therefore, how to cooperatively optimize the power allocation and sleep control of access points on the basis of ensuring the quality of service of users to improve the energy efficiency of the system has become an important research problem in the design of the current cell-free massive MIMO system. SUMMARY

[0005] In view of the problems existing in the prior art, the purpose of the present application is to provide a high-energy-efficiency hybrid DRL power allocation and sleep control method and system for cell-free massive MIMO with higher system energy efficiency.

[0006] In order to achieve the above-mentioned application purpose, the present application provides the following technical scheme:

[0007] A high-energy-efficiency hybrid DRL power allocation and sleep control method for cell-free massive MIMO, comprising the following steps:

[0008] (1) Each user sends a row pilot signal to each access point, and each access point receives the signal and performs channel estimation, and then returns the estimated channel state information to the central processing unit;

[0009] (2) The central processing unit summarizes the channel state information returned by all access points and obtains access point sleep control decisions and downlink transmit power allocation coefficients through a deployed deep reinforcement learning agent based on parameterized Markov decision process. The agent uses the current activation state and channel state information of all access points as its state and uses a mixture of discrete access point sleep control decisions and continuous downlink transmit power allocation coefficients as its actions.

[0010] (3) The agent of the central processing unit maps the hybrid actions to a low-dimensional latent representation space through a discrete embedding table and a conditional variational autoencoder, and uses a decoder to decode and obtain the reconstructed downlink transmit power allocation coefficients.

[0011] (4) The central processing unit controls the access point to perform downlink data transmission according to the hibernation control decision and the reconstructed downlink transmit power allocation coefficient in the hybrid operation;

[0012] (5) The user calculates the spectrum efficiency based on the downlink data transmission and uploads the spectrum efficiency to the central processing unit through the access point;

[0013] (6) The agent at the central processing unit calculates the total energy efficiency of the system based on the current spectral efficiency and the activation state of the access point, and updates the agent's policy with the total energy efficiency of the system as a reward. It also updates the parameters of the conditional variational autoencoder and decoder, and returns to the execution step (1).

[0014] Furthermore, the intelligent agent is configured as follows:

[0015] State: s = {s m =(b m ∪SINR k |m=1,...,M,k=1,...,K};

[0016] Mixed actions: H = {(a m ,η mk |m=1,...,M,k=1,...,K};

[0017] In the formula, s represents the state at a certain moment. m Indicates the state of access point m, b m SINR indicates the activation state of access point m, with a value of 1 indicating an active state and a value of 0 indicating a dormant state. k The signal-to-interference-plus-noise ratio (SIR) at user k is used as channel state information. H represents the mixing action, and a m This represents the hibernation control decision; a value of 1 indicates that access point m is active, and a value of 0 indicates that access point m is hibernating. η mkThis represents the downlink transmit power allocation coefficient for the downlink channel between access point m and user k, where M is the number of access points and K is the number of users.

[0018] Furthermore, step (3) includes:

[0019] (3.1) The agent uses a discrete action embedding table T ξ The sleep control decision for each access point in the hybrid action is a m The mapping is a low-dimensional continuous vector c m This enables a low-dimensional representation of discrete actions, wherein the discrete action embedding table... For the low-dimensional representation space mapping table, each row of the discrete action embedding table is a d1-dimensional continuous vector representation of the sleep control decision of an access point, where M is the number of access points and d1 is the dimension of the low-dimensional representation space.

[0020] (3.2) Convert the low-dimensional continuous vector c m Downlink transmit power allocation coefficient η mk The encoder in the conditional variational autoencoder, corresponding to state s, yields the latent variable z in the low-dimensional latent representation space, where the encoder performs the computation q. φ (z|η mk ,s,c m ), q φ () represents the encoder function;

[0021] (3.3) The latent variable z, the corresponding state s, and the low-dimensional continuous vector c are... m The decoder corresponding to the input conditional variational autoencoder is used to obtain the reconstructed downlink transmit power allocation coefficients. The decoder performs the following calculations: p ψ () is the decoder function.

[0022] Furthermore, the method for updating the parameters of the conditional variational autoencoder specifically includes:

[0023] The encoder parameters are updated according to the loss function set in the following formula:

[0024]

[0025] In the formula, L VAE Let E[] denote the loss function, and D[] denote the average value. KL () denotes the KL divergence, N(0,I) denotes the standard normal distribution with a mean of 0 and a covariance of the identity matrix, and I denotes the identity matrix.

[0026] Furthermore, the agent is also used to update the decoder parameters according to the loss function set by the following formula:

[0027]

[0028] in, Let κ be the Huber loss function. t =s t ′-s t s represents the state residual (the deviation between the actual state and the model-estimated state). t ′、s t These represent the actual observed state and the model estimated state, respectively. This represents the state residual predicted by the conditional VAE decoder, where δ is the threshold.

[0029] A non-cellular massive MIMO high-efficiency hybrid DRL power distribution and sleep control system includes a user end, an access point end, and a central processing unit end;

[0030] The user terminal includes:

[0031] The pilot signal transmission module is used to send line pilot signals to each access point;

[0032] The spectrum efficiency calculation module is used to calculate the spectrum efficiency based on the downlink data transmission situation, and then upload the spectrum efficiency to the central processing unit through the access point.

[0033] The access point includes:

[0034] The channel estimation module is used to perform channel estimation after receiving the user's signal and to send the estimated channel state information back to the central processing unit.

[0035] The central processing unit includes:

[0036] The channel state information aggregation module is used to aggregate the channel state information transmitted back from all access points.

[0037] An agent, based on a parameterized Markov decision process, uses the current activation state of all access points and channel state information as its state, and outputs a hybrid action consisting of discrete access point sleep control decisions and continuous downlink transmit power allocation coefficients. The agent includes a spatial mapping module, which maps the hybrid action to a low-dimensional latent representation space using a discrete embedding table and a conditional variational autoencoder, and then decodes it using a decoder to obtain the reconstructed downlink transmit power allocation coefficients. The agent also includes a policy update module, which calculates the total system energy efficiency based on the current spectral efficiency and the access point activation state, updates the agent's policy using the total system energy efficiency as a reward, and updates the parameters of the conditional variational autoencoder and decoder.

[0038] The access point control module is used to control the access point to perform downlink data transmission according to the hibernation control decision and the reconstructed downlink transmit power allocation coefficient in the hybrid operation.

[0039] Furthermore, the intelligent agent is configured as follows:

[0040] State: s = {s m =(b m ∪SINR k |m=1,...,M,k=1,...,K};

[0041] Mixed actions: H = {(a m ,η mk |m=1,...,M,k=1,...,K};

[0042] In the formula, s represents the state at a certain moment. m Indicates the state of access point m, b m SINR indicates the activation state of access point m, with a value of 1 indicating an active state and a value of 0 indicating a dormant state. k The signal-to-interference-plus-noise ratio (SIR) at user k is used as channel state information. H represents the mixing action, and a m This represents the hibernation control decision; a value of 1 indicates that access point m is active, and a value of 0 indicates that access point m is hibernating. η mk This represents the downlink transmit power allocation coefficient for the downlink channel between access point m and user k, where M is the number of access points and K is the number of users.

[0043] Furthermore, the spatial mapping module includes:

[0044] Low-dimensional mapping units are used to employ discrete action embedding tables T ξ The sleep control decision for each access point in the hybrid action is a m The mapping is a low-dimensional continuous vector c m This enables a low-dimensional representation of discrete actions, wherein the discrete action embedding table... For the low-dimensional representation space mapping table, each row of the discrete action embedding table is a d1-dimensional continuous vector representation of the sleep control decision of an access point, where M is the number of access points and d1 is the dimension of the low-dimensional representation space.

[0045] Latent representation space mapping unit, used to map low-dimensional continuous vectors c m Downlink transmit power allocation coefficient η mk The encoder in the conditional variational autoencoder, corresponding to state s, yields the latent variable z in the low-dimensional latent representation space, where the encoder performs the computation q. φ (z|η mk ,s,c m ), qφ () represents the encoder function;

[0046] The decoding unit is used to decode the latent variable z, the corresponding state s, and the low-dimensional continuous vector c. m The decoder corresponding to the input conditional variational autoencoder is used to obtain the reconstructed downlink transmit power allocation coefficients. The decoder performs the following calculations: p ψ () is the decoder function.

[0047] Furthermore, in the policy update module, the method for updating the conditional variational autoencoder parameters specifically includes:

[0048] The encoder parameters are updated according to the loss function set in the following formula:

[0049]

[0050] In the formula, L VAE Let E[] denote the loss function, and D[] denote the average value. KL () denotes the KL divergence, N(0,I) denotes the standard normal distribution (mean is 0, covariance is the identity matrix), and I denotes the identity matrix.

[0051] Furthermore, in the policy update module, the method for updating the decoder parameters is as follows:

[0052] The decoder parameters are updated according to the loss function set in the following formula:

[0053]

[0054] in, Let κ be the Huber loss function. t =s t ′-s t s represents the state residual (the deviation between the actual state and the model-estimated state). t ′、s t These represent the actual observed state and the model estimated state, respectively. This represents the state residual predicted by the conditional VAE decoder, where δ is the threshold.

[0055] Compared with existing technologies, the advantages of this invention are as follows: The deep reinforcement learning method based on hybrid action representation proposed in this invention effectively solves the problem of joint optimization of discrete sleep control and continuous power allocation in non-cellular massive MIMO systems by constructing a decodable potential hybrid action space. Specifically, this invention uses embedding tables and conditional variational autoencoders to represent hybrid actions in low dimension, and combines a dual-delay deep deterministic policy gradient algorithm to optimize the latent policy, significantly reducing the policy search complexity while avoiding the performance loss caused by discretization or continuous approximation in traditional methods. In addition, the state residual prediction mechanism further enhances the robustness of the algorithm in complex channel environments, providing an efficient and reliable solution for energy-saving optimization of non-cellular massive MIMO systems. Attached Figure Description

[0056] Figure 1 This is a flowchart of the non-cellular large-scale MIMO high-efficiency hybrid DRL power allocation and sleep control method provided by the present invention;

[0057] Figure 2 This is a diagram of the non-cellular massive MIMO system architecture of the present invention;

[0058] Figure 3 This is a comparison chart of the simulation results of the method of the present invention with other methods under 16AP and 10UE conditions;

[0059] Figure 4 This is a schematic diagram illustrating how the simulation effect of the method of the present invention changes with the number of APs;

[0060] Figure 5 This is a schematic diagram illustrating how the simulation effect of the method of the present invention changes with the number of UEs. Detailed Implementation

[0061] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0062] Example 1

[0063] This invention provides a method for high-efficiency hybrid DRL power allocation and sleep control in cellular-free massive MIMO, such as... Figure 1 As shown, it includes the following steps:

[0064] (1) Each user sends a line pilot signal to each access point. After receiving the signal, each access point performs channel estimation and sends the estimated channel state information back to the central processing unit.

[0065] Specifically, the steps include the following:

[0066] (1.1) The system model of the present invention is as follows: Figure 2As shown, during the uplink training phase, each user synchronously sends orthogonal pilot signals to all access points in the system. The access points use the minimum mean square error estimation method to estimate the uplink channel and obtain the channel estimation vector between the user and the access point. The channel estimation variance is affected by the pilot length, pilot signal energy, and large-scale fading coefficient. The channel estimation vector can be expressed as:

[0067]

[0068] in, g mk Let τ represent the estimated and actual channel vectors, respectively. p ρ represents the uplink training time interval. p β represents the normalized energy of each pilot symbol. mk This represents the channel coefficient between the m-th AP and the k-th user, where K is the number of users, and φ is the channel coefficient between them. k Let ||φ| represent the pilot signal sent by user k, and ||φ|| k || 2 =1,k=1,...,K, Each element is independent and identically distributed. This represents the signal received by access point m. Indicates channel noise;

[0069] (1.2) Each access point uploads its locally estimated channel state information to the CPU through the backhaul link between the access point and the central processing unit (CPU). The CPU then obtains the global channel state information of the current system for subsequent joint optimization calculations.

[0070] (2) The central processing unit summarizes the channel state information returned by all access points and obtains the access point sleep control decision and downlink transmit power allocation coefficient through the deployed deep reinforcement learning agent based on parameterized Markov decision process.

[0071] Among them, the parameterized Markov decision process is composed of It means that, among them, Representing the state space, s m =b m ∪SINR k m = 1, ..., M, k = 1, ..., K represent the state of the AP and the current channel state; Represents the action space, a m ∈{0,1},m=1,...,M represents the closing or opening operation performed on the AP, i.e., activation or hibernation; This is the state transition function; Here, γ is the reward function; γ is the discount factor. This represents the hybrid action space, which consists of discrete AP switching states and continuous power allocation coefficients.

[0072] Based on the above description, the agent settings are as follows:

[0073] State: s = {s m =(b m ∪SINR k |m=1,...,M,k=1,...,K};

[0074] Mixed actions: H = {(a m ,η mk |m=1,...,M,k=1,...,K};

[0075] In the formula, s represents the state at a certain moment. m Indicates the state of access point m, b m SINR indicates the activation state of access point m, with a value of 1 indicating an active state and a value of 0 indicating a dormant state. k The signal-to-interference-plus-noise ratio (SIR) at user k is used as channel state information. H represents the mixing action, and a m This represents the hibernation control decision; a value of 1 indicates that access point m is active, and a value of 0 indicates that access point m is hibernating. η mk This represents the downlink transmit power allocation coefficient for the downlink channel between access point m and user k, where M is the number of access points and K is the number of users.

[0076] SINR k It is obtained by summarizing the channel status information returned by all access points, and the specific formula is as follows:

[0077]

[0078] Where, σ 2 This indicates the noise received by the user. express conjugate, p mi w represents the power allocated by the m-th AP to the i-th user. mi This represents the precoding vector between the m-th AP and the i-th user.

[0079] (3) The agent of the central processing unit maps the hybrid actions to a low-dimensional latent representation space through a discrete embedding table and a conditional variational autoencoder (CVAE), and uses a decoder to decode and obtain the reconstructed downlink transmit power allocation coefficients.

[0080] Step (3) includes:

[0081] (3.1) The agent uses a discrete action embedding table T ξThe sleep control decision for each access point in the hybrid action is a m The mapping is a low-dimensional continuous vector c m This enables a low-dimensional representation of discrete actions, wherein the discrete action embedding table... For the low-dimensional representation space mapping table, each row of the discrete action embedding table is a d1-dimensional continuous vector representation of the sleep control decision of an access point, where M is the number of access points and d1 is the dimension of the low-dimensional representation space.

[0082] (3.2) Convert the low-dimensional continuous vector c m Downlink transmit power allocation coefficient η mk The encoder in the corresponding state s input conditional variational autoencoder yields the latent variables located in the low-dimensional latent representation space. The encoder performs the calculation q. φ (z|η mk ,s,c m ), q φ () represents the encoder function;

[0083] (3.3) The latent variable z, the corresponding state s, and the low-dimensional continuous vector c are... m The decoder corresponding to the input conditional variational autoencoder is used to obtain the reconstructed downlink transmit power allocation coefficients. The decoder performs the following calculations: p ψ () is the decoder function.

[0084] This step can eliminate the error in the conversion between discrete and continuous actions.

[0085] (4) The central processing unit controls the access point to transmit downlink data according to the hibernation control decision and the reconstructed downlink transmit power allocation coefficient in the hybrid operation.

[0086] Specifically, the central processing unit obtains the access points that need to be activated from the hibernation control decision; controls the access points that need to be activated to transmit downlink data to the user according to the reconstructed downlink transmit power allocation coefficient, and uses maximum ratio combining technology for precoding.

[0087] (5) The user calculates the spectrum efficiency based on the downlink data transmission and uploads the spectrum efficiency to the central processing unit through the access point.

[0088] The user first calculates the signal-to-interference-plus-noise ratio (SINR) at the UE during downlink transmission. k Based on the calculated SINR k Further calculation of the corresponding spectral efficiency can be expressed as:

[0089]

[0090] Among them, SE k Let τ represent the spectral efficiency of the k-th user. c τ represents the length of a coherent gap. p This indicates the uplink training time.

[0091] (6) The agent at the central processing unit calculates the total energy efficiency of the system based on the current spectral efficiency and the activation state of the access point, and updates the agent's policy with the total energy efficiency of the system as a reward. It also updates the parameters of the conditional variational autoencoder and decoder, and returns to the execution step (1).

[0092] The energy efficiency (EE) of an intelligent agent system is calculated as the ratio of total system throughput to total system energy consumption.

[0093]

[0094] Where EE represents the total energy efficiency of the system, B represents the system bandwidth, and P represents the total energy efficiency of the system. total This indicates the total energy consumption of the system.

[0095] Total system energy consumption P total The energy consumption model includes the circuit power consumption, transmit power consumption, and backhaul link energy consumption of each access point. It comprehensively considers the number of access points, the circuit consumption of each antenna, the fixed backhaul link energy consumption, and the load-related dynamic energy consumption.

[0096]

[0097] Among them, P m This indicates the power consumption of each AP, including amplifier power consumption and circuit power consumption. This represents the transmit power expenditure at the m-th access point, including all user-specific beamforming operations. This includes both the effective transmit power serving K users and the associated fire chain power proportional to the number of effective spatial streams.

[0098] The CPU transforms the problem of maximizing energy efficiency into the problem of maximizing rewards in reinforcement learning, specifically as follows:

[0099]

[0100] st

[0101]

[0102] Among them, P T S0 represents the maximum transmission power of each AP, and S0 represents the user's minimum spectral efficiency.

[0103] The reward is represented as follows:

[0104] R = EE,

[0105] Where R represents the reward received by the agent.

[0106] The agent continuously interacts with the environment, perceiving the system state, outputting hybrid action decisions, and updating its policy based on the rewards received. During training, a delayed double-Q network and a policy network structure are employed. The network parameters are iteratively optimized using the policy gradient method, ultimately converging within a finite number of training rounds to a highly energy-efficient joint control policy of access point sleep and power allocation. Specifically, the agent's double-Q network deployed on the CPU is updated based on the following estimates:

[0107]

[0108] Where, θ i Let γ represent the parameters of the i-th network, γ represent the discount factor, and π represent the parameters of the i-th network. θ (s') represents the target policy learned by the agent;

[0109] The gradient of the policy network is represented as:

[0110]

[0111] Where ω represents the parameters of the policy network, and the soft updates of the target Q network and the target policy network are expressed as:

[0112] θ′ i ←τθ i +(1-τ)θ′ i ,

[0113] ω'←τω(1-τ)ω',

[0114] Where τ represents the soft update coefficient.

[0115] Dual-Q networks are designed to independently estimate state-action values, a structure that enhances the stability and performance of policy learning. Through continuous interaction and learning with the environment via the policy network, system efficiency can ultimately be effectively improved.

[0116] Encoder parameter updates specifically include:

[0117] The agent updates the encoder parameters according to the loss function set in the following formula:

[0118]

[0119] In the formula, L VAE Let E[] denote the loss function, and D[] denote the average value. KL () denotes the KL divergence, N(0,I) denotes the standard normal distribution (mean is 0, covariance is the identity matrix), and I denotes the identity matrix.

[0120] The agent is also used to update the decoder parameters by minimizing the loss function set according to the following formula:

[0121]

[0122] in, Let κ be the Huber loss function. t =s′ t -s t s represents the state residual (the deviation between the actual state and the model-estimated state). t ′、s t These represent the actual observed state and the model estimated state, respectively. This represents the state residual predicted by the conditional VAE decoder, where δ is the threshold.

[0123] To verify the effectiveness and efficacy of the method of the present invention, simulation analysis was conducted, as detailed below:

[0124] Obtain Figure 3 , Figure 4 , Figure 5 The simulation results show that DDQN is an optimization scheme based on a two-layer Q-network proposed by Mendoza et al. in their 2021 paper "Deepreinforcement learning for dynamic access point activation in cell-free MIMO networks," PPO is the algorithm used by Wen Li in their 2024 paper "Energy-Efficient Access Point Sleep Control in User-Centric Cell-Free Massive MIMO Systems," Always On is a heuristic algorithm that does not employ AP sleep control, and HyTD3 is the method of this invention. It is evident that the method of this invention significantly outperforms the two compared algorithms.

[0125] Example 2

[0126] This invention provides a non-cellular massive MIMO high-efficiency hybrid DRL power distribution and sleep control system, including a user terminal, an access point terminal, and a central processing unit terminal;

[0127] The user terminal includes:

[0128] The pilot signal transmission module is used to send line pilot signals to each access point;

[0129] The spectrum efficiency calculation module is used to calculate the spectrum efficiency based on the downlink data transmission situation, and then upload the spectrum efficiency to the central processing unit through the access point.

[0130] The access point includes:

[0131] The channel estimation module is used to perform channel estimation after receiving the user's signal and to send the estimated channel state information back to the central processing unit.

[0132] The central processing unit includes:

[0133] The channel state information aggregation module is used to aggregate the channel state information transmitted back from all access points.

[0134] An agent, based on a parameterized Markov decision process, uses the current activation state of all access points and channel state information as its state, and outputs a hybrid action consisting of discrete access point sleep control decisions and continuous downlink transmit power allocation coefficients. The agent includes a spatial mapping module, which maps the hybrid action to a low-dimensional latent representation space using a discrete embedding table and a conditional variational autoencoder, and then decodes it using a decoder to obtain the reconstructed downlink transmit power allocation coefficients. The agent also includes a policy update module, which calculates the total system energy efficiency based on the current spectral efficiency and the access point activation state, updates the agent's policy using the total system energy efficiency as a reward, and updates the parameters of the conditional variational autoencoder and decoder.

[0135] The access point control module is used to control the access point to perform downlink data transmission according to the hibernation control decision and the reconstructed downlink transmit power allocation coefficient in the hybrid operation.

[0136] The intelligent agent is configured as follows:

[0137] State: s = {s m =(b m ∪SINR k |m=1,...,M,k=1,...,K};

[0138] Mixed actions: H = {(a m ,η mk |m=1,...,M,k=1,...,K};

[0139] In the formula, s represents the state at a certain moment. m Indicates the state of access point m, b m SINR indicates the activation state of access point m, with a value of 1 indicating an active state and a value of 0 indicating a dormant state. k The signal-to-interference-plus-noise ratio (SIR) at user k is used as channel state information. H represents the mixing action, and a mThis represents the hibernation control decision; a value of 1 indicates that access point m is active, and a value of 0 indicates that access point m is hibernating. η mk This represents the downlink transmit power allocation coefficient for the downlink channel between access point m and user k, where M is the number of access points and K is the number of users.

[0140] The spatial mapping module includes:

[0141] Low-dimensional mapping units are used to employ discrete action embedding tables T ξ The sleep control decision for each access point in the hybrid action is a m The mapping is a low-dimensional continuous vector c m This enables a low-dimensional representation of discrete actions, wherein the discrete action embedding table... For the low-dimensional representation space mapping table, each row of the discrete action embedding table is a d1-dimensional continuous vector representation of the sleep control decision of an access point, where M is the number of access points and d1 is the dimension of the low-dimensional representation space.

[0142] Latent representation space mapping unit, used to map low-dimensional continuous vectors c m Downlink transmit power allocation coefficient η mk The encoder in the conditional variational autoencoder, corresponding to state s, yields the latent variable z in the low-dimensional latent representation space, where the encoder performs the computation q. φ (z|η mk ,s,c m ), q φ () represents the encoder function;

[0143] The decoding unit is used to decode the latent variable z, the corresponding state s, and the low-dimensional continuous vector c. m The decoder corresponding to the input conditional variational autoencoder is used to obtain the reconstructed downlink transmit power allocation coefficients. The decoder performs the following calculations: p ψ () is the decoder function.

[0144] The function for updating the discrete embedding table and the decoder specifically includes:

[0145] The discrete embedding table and encoder function are updated according to the loss function set in the following formula:

[0146]

[0147] In the formula, L VAE Let E[] denote the loss function, and D[] denote the average value. KL () denotes the KL divergence, N(0,I) denotes the standard normal distribution (mean is 0, covariance is the identity matrix), and I denotes the identity matrix.

[0148] The agent also includes a state residual prediction module, used to update the decoder parameters by minimizing the loss function set according to the following formula:

[0149]

[0150] in, Let κ be the Huber loss function. t =s′ t -s t s′ represents the state residual (the deviation between the actual state and the model-estimated state). t s t These represent the actual observed state and the model estimated state, respectively. This represents the state residual predicted by the conditional VAE decoder, where δ is the threshold.

[0151] The system provided in this embodiment of the invention can be used to execute the method provided in Embodiment 1 of the invention, and has the corresponding functions and beneficial effects of executing the method.

[0152] It is worth noting that in the embodiments of the above system, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of the present invention.

[0153] It should be understood that the embodiments and descriptions above are only the principles, main features and advantages of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope of the invention, and all such changes and modifications fall within the protection scope of the present invention.

Claims

1. A method for high-efficiency hybrid DRL power allocation and sleep control in a cellular-free large-scale MIMO system, characterized in that, Includes the following steps: (1) Each user sends a line pilot signal to each access point. After receiving the signal, each access point performs channel estimation and sends the estimated channel state information back to the central processing unit. (2) The central processing unit summarizes the channel state information returned by all access points and obtains access point sleep control decisions and downlink transmit power allocation coefficients through a deep reinforcement learning agent based on parameterized Markov decision process. The agent uses the current activation state and channel state information of all access points as its state and uses a mixture of discrete access point sleep control decisions and continuous downlink transmit power allocation coefficients as its actions. (3) The agent of the central processing unit maps the hybrid actions to a low-dimensional latent representation space through a discrete embedding table and a conditional variational autoencoder, and uses a decoder to decode and obtain the reconstructed downlink transmit power allocation coefficients. (4) The central processing unit controls the access point to perform downlink data transmission according to the hibernation control decision and the reconstructed downlink transmit power allocation coefficient in the hybrid operation; (5) The user calculates the spectrum efficiency based on the downlink data transmission and uploads the spectrum efficiency to the central processing unit through the access point; (6) The agent at the central processing unit calculates the total energy efficiency of the system based on the current spectral efficiency and the activation state of the access point, and updates the agent's policy with the total energy efficiency of the system as a reward. It also updates the parameters of the conditional variational autoencoder and decoder, and returns to the execution step (1).

2. The method for high-efficiency hybrid DRL power allocation and sleep control without cellular massive MIMO as described in claim 1, characterized in that, The intelligent agent is configured as follows: State: s = {s m =(b m ∪SINR k |m=1,...,M,k=1,...,K}; Mixed actions: H = {(a m ,η mk |m=1,...,M,k=1,...,K}; In the formula, s represents the state at a certain moment. m Indicates the state of access point m, b m SINR indicates the activation state of access point m, with a value of 1 indicating an active state and a value of 0 indicating a dormant state. k The signal-to-interference-plus-noise ratio (SIR) at user k is used as channel state information. H represents the mixing action, and a m This represents the hibernation control decision; a value of 1 indicates that access point m is active, and a value of 0 indicates that access point m is hibernating. η mk This represents the downlink transmit power allocation coefficient for the downlink channel between access point m and user k, where M is the number of access points and K is the number of users.

3. The method for high-efficiency hybrid DRL power allocation and sleep control without cellular massive MIMO as described in claim 1, characterized in that, Step (3) includes: (3.1) The agent uses a discrete action embedding table T ξ The sleep control decision for each access point in the hybrid action is a m The mapping is a low-dimensional continuous vector c m This enables a low-dimensional representation of discrete actions, wherein the discrete action embedding table... For the low-dimensional representation space mapping table, each row of the discrete action embedding table is a d1-dimensional continuous vector representation of the sleep control decision of an access point, where M is the number of access points and d1 is the dimension of the low-dimensional representation space. (3.2) Convert the low-dimensional continuous vector c m Downlink transmit power allocation coefficient η mk The encoder in the conditional variational autoencoder, corresponding to state s, yields the latent variable z in the low-dimensional latent representation space, where the encoder performs the computation q. φ (z|η mk ,s,c m ), q φ () represents the encoder function, and φ represents the encoder parameter; (3.3) The latent variable z, the corresponding state s, and the low-dimensional continuous vector c are... m The decoder corresponding to the input conditional variational autoencoder is used to obtain the reconstructed downlink transmit power allocation coefficients. The decoder performs the following calculations: p ψ () represents the decoder function, and ψ represents the decoder parameters.

4. The method for high-efficiency hybrid DRL power allocation and sleep control without cellular massive MIMO as described in claim 3, characterized in that, The method for updating the parameters of the conditional variational autoencoder specifically includes: The encoder parameters are updated according to the loss function set in the following formula: In the formula, L VAE Let E[] denote the loss function, and D[] denote the average value. KL () denotes the KL divergence, N(0,I) denotes the standard normal distribution with a mean of 0 and a covariance of the identity matrix, and I denotes the identity matrix.

5. The method for high-efficiency hybrid DRL power allocation and sleep control without cellular massive MIMO according to claim 4, characterized in that: The agent is also used to update the decoder parameters according to the loss function set by the following formula: in, Let k be the Huber loss function. t =s t ′-s t Represents the state residual, s t ′、s t These represent the actual state and the estimated state, respectively. This represents the state residual predicted by the conditional decoder, where δ is the threshold.

6. A cellless, large-scale MIMO, high-efficiency hybrid DRL power distribution and sleep control system, characterized in that, This includes the user terminal, the access point terminal, and the central processing unit terminal; The user terminal includes: The pilot signal transmission module is used to send line pilot signals to each access point; The spectrum efficiency calculation module is used to calculate the spectrum efficiency based on the downlink data transmission situation, and then upload the spectrum efficiency to the central processing unit through the access point. The access point includes: The channel estimation module is used to perform channel estimation after receiving the user's signal and to send the estimated channel state information back to the central processing unit. The central processing unit includes: The channel state information aggregation module is used to aggregate the channel state information transmitted back from all access points. An agent, based on a parameterized Markov decision process, uses the current activation state of all access points and channel state information as its state, and outputs a hybrid action consisting of discrete access point sleep control decisions and continuous downlink transmit power allocation coefficients. The agent includes a spatial mapping module, which maps the hybrid action to a low-dimensional latent representation space using a discrete embedding table and a conditional variational autoencoder, and then decodes it using a decoder to obtain the reconstructed downlink transmit power allocation coefficients. The agent also includes a policy update module, which calculates the total system energy efficiency based on the current spectral efficiency and the access point activation state, updates the agent's policy using the total system energy efficiency as a reward, and updates the parameters of the conditional variational autoencoder and decoder. The access point control module is used to control the access point to perform downlink data transmission according to the hibernation control decision and the reconstructed downlink transmit power allocation coefficient in the hybrid operation.

7. The non-cellular massive MIMO high-efficiency hybrid DRL power distribution and sleep control system according to claim 6, characterized in that, The intelligent agent is configured as follows: State: s = {s m =(b m ∪SINR k |m=1,...,M,k=1,...,K}; Mixed actions: H = {(a m ,η mk |m=1,...,M,k=1,...,K}; In the formula, s represents the state at a certain moment. m Indicates the state of access point m, b m SINR indicates the activation state of access point m, with a value of 1 indicating an active state and a value of 0 indicating a dormant state. k The signal-to-interference-plus-noise ratio (SIR) at user k is used as channel state information. H represents the mixing action, and a m This represents the hibernation control decision; a value of 1 indicates that access point m is active, and a value of 0 indicates that access point m is hibernating. η mk This represents the downlink transmit power allocation coefficient for the downlink channel between access point m and user k, where M is the number of access points and K is the number of users.

8. The non-cellular massive MIMO high-efficiency hybrid DRL power distribution and sleep control system according to claim 6, characterized in that, The spatial mapping module includes: Low-dimensional mapping units are used to employ discrete action embedding tables T ξ The sleep control decision for each access point in the hybrid action is a m The mapping is a low-dimensional continuous vector c m This enables a low-dimensional representation of discrete actions, wherein the discrete action embedding table... For the low-dimensional representation space mapping table, each row of the discrete action embedding table is a d1-dimensional continuous vector representation of the sleep control decision of an access point, where M is the number of access points and d1 is the dimension of the low-dimensional representation space. Latent representation space mapping unit, used to map low-dimensional continuous vectors c m Downlink transmit power allocation coefficient η mk The encoder in the conditional variational autoencoder, corresponding to state s, yields the latent variable z in the low-dimensional latent representation space, where the encoder performs the computation q. φ (z|η mk ,s,c m ), q φ () represents the encoder function; The decoding unit is used to decode the latent variable z, the corresponding state s, and the low-dimensional continuous vector c. m The decoder corresponding to the input conditional variational autoencoder is used to obtain the reconstructed downlink transmit power allocation coefficients. The decoder performs the following calculations: p ψ () is the decoder function.

9. The non-cellular massive MIMO high-efficiency hybrid DRL power distribution and sleep control system according to claim 8, characterized in that, The method for updating the parameters of the conditional variational autoencoder in the policy update module specifically includes: The encoder parameters are updated according to the loss function set in the following formula: In the formula, L VAE Let E[] denote the loss function, and D[] denote the average value. KL () denotes the KL divergence, N(0,I) denotes the standard normal distribution with a mean of 0 and a covariance of the identity matrix, and I denotes the identity matrix.

10. The non-cellular massive MIMO high-efficiency hybrid DRL power distribution and sleep control system according to claim 6, characterized in that, In the policy update module, the method for updating the decoder parameters is as follows: The decoder parameters are updated according to the loss function set in the following formula: in, Let κ be the Huber loss function. t =s t ′-s t Represents the state residual, s t ′、s t These represent the actual state and the estimated state, respectively. This represents the state residual predicted by the conditional decoder, where δ is the threshold.

Images