High-energy-efficiency energy management method based on nested DRL in cellular-free large-scale MIMO system

Abstract

The invention discloses a nested DRL-based high-energy-efficiency energy management method in a cellular-free large-scale MIMO system. The method comprises the following steps: constructing an optimization problem taking maximization of the total energy efficiency of the system as an optimization target based on a downlink communication transmission model; defining representations of states, actions, and rewards in energy management; it is assumed that power is evenly distributed and energy transfer does not exist, the optimization problem of maximizing the total energy efficiency of the system is split to obtain a first sub-problem, AP dynamic dormancy operation output by the first sub-problem is used as known information to be transmitted to a second sub-problem, power distribution and energy transfer are further optimized on the basis of AP dynamic dormancy, and the maximum system total energy efficiency is obtained. Processing the mixed action space; and the AP adaptively adjusts power distribution, energy transfer and dynamic dormancy in dynamic and uncertain environments by adopting a nested deep reinforcement learning algorithm consisting of a PPO network and an SAC network according to the current environment state information. According to the method, the total energy efficiency of the system is improved while the requirements of user communication reliability and real-time performance are met.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of wireless communication, and relates to system energy efficiency optimization, in particular to a high-energy-efficiency energy management method based on nested deep reinforcement learning (DRL) in a cell-free massive multi-input multi-output (MIMO) system. BACKGROUND

[0002] Massive MIMO is widely used in the fifth generation mobile communication system and future wireless communication systems. It is based on hundreds of antenna multiplexing and serves tens of users at the same time, and can achieve high spectral efficiency through linear transmission and reception technology. Compared with traditional centralized massive MIMO systems, the cell-free massive MIMO (CF-mMIMO) system uses a large number of small antennas distributed in the environment for data transmission and reception, achieving better spatial diversity and spectral efficiency. This architecture can reduce communication latency, improve signal quality and reliability, and has the potential to significantly reduce energy consumption.

[0003] In order to meet the communication needs of more mobile users, the number of antenna deployments has increased significantly, and the demand for energy supply has also increased dramatically. High-energy-efficiency communication can reduce energy consumption and reduce operating costs. However, there is a trade-off between latency and energy efficiency. Reducing latency usually requires more energy consumption, while improving energy efficiency may increase communication latency. By introducing renewable energy, co-supplying the system with the power grid has become a new direction to solve the problem. How to handle resource allocation in a hybrid energy CF-mMIMO system, determine the power allocation, energy transfer, and access point (AP) dynamic sleep strategy in a dynamic and uncertain environment is a problem that needs to be solved. SUMMARY

[0004] The purpose of the application is to overcome the deficiencies in the prior art, and to provide a high-energy-efficiency energy management method based on nested deep reinforcement learning in a CF-mMIMO system on the basis of hybrid energy supply, which improves the energy efficiency of the system while meeting the real-time and reliability requirements of user communication.

[0005] Technical solution: In order to achieve the above-mentioned application purpose, the application adopts the following technical solutions:

[0006] The high-energy-efficiency energy management method based on nested DRL in the cell-free massive MIMO system comprises the following steps:

[0007] S1: According to the statistical channel state information, an optimization problem is constructed based on a downlink communication transmission model to maximize the system total energy efficiency;

[0008] S2: The representation of state, action and reward in energy management is defined, wherein the state includes the achievable rate of each user, the battery power and sleep / activation state of each AP, the action includes power allocation, sleep / activation decision and energy transfer, and the reward considers the system long-term total energy efficiency and the punishment for the failure to meet the Quality of Service (QoS) constraint;

[0009] S3: A heuristic method is adopted to split the optimization problem of maximizing the system total energy efficiency, assuming uniform power allocation and no energy transfer, to obtain a first sub-problem, and the first sub-problem outputs the AP dynamic sleep operation as known information to a second sub-problem, which further optimizes power allocation and energy transfer based on the AP dynamic sleep to handle the mixed action space;

[0010] S4: The AP obtains the current environmental state information, and adopts a nested deep reinforcement learning algorithm composed of a Proximal Policy Optimization (PPO) network and a Soft Actor-Critic (SAC) network to adaptively adjust power allocation, energy transfer and dynamic sleep in a dynamic and uncertain environment, so as to realize energy efficiency optimization while meeting the QoS requirement.

[0011] Further, in step S1, the long-term total energy efficiency of the system is constructed according to the system total power consumption and the sum of the achievable rates of the users wherein R k (i) is the achievable rate of the kth user at the ith moment, P tot (i) is the total power consumption of the system, P is the total number of system running moments, and K is the number of users; the problem of maximizing the system total energy efficiency is expressed as:

[0012]

[0013] wherein represents the minimum required rate of the kth user at the ith moment, p lk (i) represents the transmission power of the lth AP to the kth user at the ith moment, b l (i) represents the activation state of the lth AP, represents the transmission power of the lth AP to the kth user provided by the power grid and the energy maintained for running the lth AP, represents the CPU communication power provided by the power grid, Pik represents the transmit power of the lth AP to the kth user provided by the EH device, Pik represents the transmit power of the lth AP to the kth user provided by the EH device, Pik represents the transmit power of the lth AP to the kth user provided by the EH device.

[0014] Further, in step S2, the representation of state in energy management is defined, specifically including:

[0015]

[0016] wherein R = [R1, R2,..., R K ] T , B = [B1, B2,..., B L ] T , S = [S1, S2,..., S L ] T wherein R k represents the achievable rate of the kth user, B l represents the battery level of the lth AP, S l ∈ {0, 1} represents the sleep / active state of the lth AP, and L is the number of APs.

[0017] Further, in step S2, the representation of action in energy management is defined, specifically including:

[0018]

[0019] wherein p = [p 11 , p 12 ,..., p LK ] T , b = [b1, b2,..., b L ] T , wherein p lk represents the transmit power of the lth AP to the kth user, b l ∈ {0, 1} represents the sleep / active decision of the lth AP, and the action b at time i is the state S at time i+1, Pik represents the transmit power of the lth AP to the kth user provided by the EH device.

[0020] Further, in step S2, the representation of reward in energy management is defined, specifically including:

[0021]

[0022] wherein EE represents the long-term total energy efficiency of the system, is a negative constant representing a penalty factor, R k and Let represent the reachable rate and threshold for the k-th user, respectively.

[0023] Furthermore, the first sub-problem after splitting in step S3 is as follows:

[0024]

[0025] The second subproblem after splitting is as follows:

[0026]

[0027] Furthermore, after the optimization problem is decomposed, the action space is correspondingly divided into a discrete action space and a continuous action space as follows:

[0028]

[0029] a1 = b,

[0030] a2=[p T E t,T ] T .

[0031] Furthermore, step S4 specifically includes:

[0032] S41: Considering the discreteness of AP dynamic sleep in the action space, the PPO algorithm is selected to solve the optimal strategy for AP dynamic sleep; the PPO agent interacts with the environment continuously to learn the optimal action selection strategy and output the AP dynamic sleep decision to the environment and the SAC agent.

[0033] S42: Considering the continuity of power allocation and energy transfer in the action space, the SAC algorithm is selected to solve the optimal strategy for power allocation and energy transfer; the SAC agent continuously interacts with the environment, aiming to learn the optimal action selection strategy and output power allocation and energy transfer decisions to the environment;

[0034] S43: Continuously update the policy. When the long-term reward converges, optimize the overall energy efficiency of the system. In the subsequent execution phase, the agent determines the action directly based on the observations using the trained Actor network.

[0035] The present invention also provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the high-efficiency energy management method based on nested DRL in the non-cellular massive MIMO system.

[0036] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the high energy efficiency energy management method based on nested DRL in the cell-free massive MIMO system.

[0037] Beneficial effects: Compared with the prior art, the application proposes an energy management method based on nested deep reinforcement learning, which simultaneously processes power allocation, energy transfer and AP dynamic hibernation in a hybrid energy supply CF-mMIMO system, jointly optimizes system energy efficiency from different aspects, and further improves system energy efficiency. Moreover, the application effectively handles the difficulty brought by the hybrid action space by splitting the optimization problem of maximizing the total system energy efficiency, realizes the maximization of system energy efficiency while meeting the QoS requirement, reduces the computational complexity compared with the method of solving optimization based on traditional mathematical conversion, has higher flexibility and optimization ability, and improves the total system energy efficiency compared with the AP hibernation continuous algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a method flowchart of an embodiment of the application;

[0039] Figure 2 is a nested deep reinforcement learning algorithm block diagram in an embodiment of the application;

[0040] Figure 3 is a total system energy efficiency simulation result graph in the training process of an embodiment of the application. DETAILED DESCRIPTION

[0041] The application will be further illustrated below in combination with the drawings and specific embodiments, and it should be understood that these embodiments are only used to illustrate the application and not used to limit the scope of the application, and after reading the application, various equivalent modifications of the application by those skilled in the art all fall within the scope defined by the appended claims.

[0042] The application provides a high energy efficiency energy management method based on nested deep reinforcement learning in a CF-mMIMO system, assuming that there are L single-antenna APs and K single-antenna users in a cell-free massive (CF-m) MIMO system, each AP is equipped with an energy collection device connected to a central energy storage unit, is powered by a power grid and a battery storing new energy, and each AP is connected to a CPU. L>>K is satisfied. The APs and users are randomly distributed in the coverage area and are assumed not to be at the same location.

[0043] As shown in Figure 1 , the method comprises the following steps:

[0044] S1: According to the statistical channel state information, an optimization problem is constructed based on a downlink communication transmission model to maximize the total energy efficiency of the system as an optimization objective;

[0045] Step S1 specifically comprises:

[0046] S11: The AP uses the minimum mean square error for channel estimation, adopts the maximum ratio transmission as the beamforming technology for downlink transmission, and calculates the effective spectral efficiency and achievable rate R k (i) of the kth user at the ith moment;

[0047] S12: The total power consumption P tot (i) of the system is composed of the energy required to maintain the operation of the AP and the energy required for signal transmission, and is jointly powered by the power grid and the EH device. The energy captured by the EH device is stored in the battery, and the battery of the AP is powered at the next moment. The battery power of the lth AP at the ith+1 moment can be given by:

[0048]

[0049] where B l (i) is the battery power of the lth AP at the ith moment, b l (i) is the activation state of the lth AP, A l (i) is the renewable energy harvesting amount, is the renewable energy consumption amount, which can be expressed as:

[0050]

[0051] wherein is the energy consumed by the lth AP provided by the EH device, T is the corresponding time of each moment, is the communication power of the lth AP with the CPU provided by the EH device, is the transmission power of the lth AP to the kth user provided by the EH device. and is the energy flowing into and out of the lth AP at the ith moment, which can be given by:

[0052]

[0053] wherein 0 l′l <1 is the energy transmission loss coefficient between the l'th AP and the lth AP, is the energy transmitted from the l'th AP to the lth AP.

[0054] When the AP enters the sleep state, no energy is consumed; B max is the maximum capacity of the battery.

[0055] S13: constructing long-term total energy efficiency of the system according to the total system power consumption and the sum of the reachable rates of users P is the total number of running time points, and then the problem of maximizing the total energy efficiency of the system can be expressed as:

[0056]

[0057] wherein, represents the minimum demand rate of the kth user at the ith time point, p lk (i) represents the transmission power of the lth AP to the kth user at the ith time point, b l (i) represents the activation state of the lth AP, represents the transmission power of the lth AP to the kth user provided by the power grid and the running energy maintained by the lth AP, represents the CPU communication power provided by the power grid, represents the transmission power of the lth AP to the kth user provided by the EH device and the running energy maintained by the lth AP, represents the CPU communication power provided by the EH device, represents the energy transferred from the lth AP to the l'th AP. The constraint C1.1 guarantees the reliability of communication; the constraint C1.2 guarantees that the transmission power of the lth AP does not exceed the maximum power p max ; the constraints C1.3-C1.6 guarantee the non-negativity of energy and allocated transmission power; and the constraint C1.7 represents the sleep state constraint of each AP.

[0058] S2: defining the representation of states, actions and rewards in energy management based on a downlink communication transmission model;

[0059] The problem established in the above step S1 is difficult to solve due to its nonlinearity and non-convexity, and a method based on traditional optimization has extremely high complexity and needs to be solved separately at each time point, which brings great computational overhead. In view of this, the original problem is converted into a decision theory problem based on Markov decision process, and the basic elements include state space, action space and reward function.

[0060] Step S2 specifically includes:

[0061] State: defining the representation of states in energy management, specifically including:

[0062]

[0063] wherein R = [R1, R2,..., R K ] T , B = [B1, B2,..., B L ] T, S = [S1, S2,..., S L ] T , where B l represents the battery energy S l ∈{0,1} represents the sleep / active state of each AP.

[0064] Action: define the representation of action in energy management, specifically including:

[0065]

[0066] where p = [p 11 ,p 12 ,...,p LK ] T , b = [b1, b2,..., b L ] T , where b l ∈{0,1} represents the sleep / active decision of each AP, the action b at time i is the state S at time i+1, represents the energy transferred from the lth AP to the l'th AP. Since the power allocation and energy transfer are continuous actions, the dynamic sleep of the AP is a discrete action, and the action space is a mixed action space.

[0067] Reward: define the representation of reward in energy management, specifically including:

[0068]

[0069] where is a large negative constant representing a penalty factor. The penalty term means that if the QoS constraint of at least one UE cannot be met, the agent will be severely punished.

[0070] S3: adopt a heuristic method to split the optimization problem of maximizing the total energy efficiency of the system to process the mixed action space;

[0071] The power allocation and energy transfer in the action space constructed in the above step S2 are continuous actions, and the dynamic sleep of the AP is a discrete action, which makes the action space a mixed action space, and existing deep reinforcement learning algorithms cannot be directly used. In view of this, the present application adopts a heuristic method to split the optimization problem, and the corresponding action space is also split into a discrete action space and a continuous action space.

[0072] Step S3 specifically includes:

[0073] S31: By heuristic method, assuming uniform power allocation and no energy transfer, the first sub-problem is as follows: maximize system total energy efficiency under QoS constraints by optimizing AP dynamic sleep.

[0074]

[0075] Constraint C2.1 guarantees the reliability of communication; constraint C2.2 represents the sleep state constraint of each AP.

[0076] S32: The first sub-problem outputs the AP dynamic sleep operation as known information to the second sub-problem, which further optimizes power allocation and energy transfer based on the AP dynamic sleep to achieve the maximum system total energy efficiency under QoS constraints. The second sub-problem is as follows:

[0077]

[0078] Constraint C2.1 guarantees the reliability of communication; constraint C2.2 guarantees that the transmission power of the lth AP does not exceed the maximum power; and constraints C2.3-C2.6 guarantee the non-negativity of energy and allocated transmission power.

[0079] S33: After the optimization problem is split, the action space is also split into discrete action space and continuous action space as follows:

[0080]

[0081] a1=b,

[0082] a2=[p T ,E t,T ] T .

[0083] S4: The AP obtains the current environmental state information, and uses a nested deep reinforcement learning algorithm composed of PPO network and SAC network to adaptively adjust power allocation, energy transfer and dynamic sleep in dynamic and uncertain environment according to the current environmental state information, so as to realize energy efficiency optimization while meeting QoS requirements. The structure of the nested deep reinforcement learning algorithm is as shown in Figure 2 .

[0084] Step S4 specifically includes:

[0085] S41: Considering the discreteness of AP dynamic sleep in the action space, and the fact that sleep strategy directly affects the network topology, a strategy with strong determinacy is needed to avoid network performance shock caused by sudden changes in strategy. Therefore, the PPO algorithm with stable performance is selected to solve the optimal strategy of AP dynamic sleep. The policy gradient is used to update the policy network π θThe parameter θ is updated, and the policy gradient can be expressed as:

[0086]

[0087] where (s i ,a 1,i ) is a time step sampled from the trajectory generated by the agent interacting with the environment using the policy network π θ θ old is the old parameter of the policy network that has not been updated, r i (θ) is the policy ratio, defined as represents the selection tendency of the current policy compared to the old policy for action a 1,i , is the advantage function, and the advantage function at each time step is calculated using the generalized advantage estimation, which can be expressed as:

[0088]

[0089] where δ i = r i + γV ω (s i+1 ) - V ω (s i ), r i is the corresponding reward of time step (s i ,a 1,i ), γ is the discount factor, λ ∈ [0, 1] is an adjustable parameter, P is the total number of time steps, and V ω (·) is the state value function estimated by the evaluation network with parameter ω, representing the expected return under state s t , and its output value is a scalar. In addition, the clipped surrogate objective is a technique in the PPO algorithm that simplifies the complexity of the algorithm and improves the stability of training by limiting the magnitude of policy change at each step, expressed as:

[0090]

[0091] where ε is the clipping coefficient, and then the target parameter θ new is obtained by maximizing J PPO (θ) :

[0092]

[0093] The parameter θ is updated by gradient ascent method:

[0094]

[0095] where α θ is the learning rate. The evaluation network updates the parameter ω by the time difference method:

[0096]

[0097] where alpha ω is the learning rate, v i = V ω (s i )-[r i + gamma V ω (s i+1 )].

[0098] The PPO agent continuously interacts with the environment, aiming to learn the optimal action selection strategy, and outputs the AP dynamic dormancy decision to the environment and the SAC agent.

[0099] S42: Considering the continuity of power allocation and energy transfer in the action space, and the power allocation and energy transfer are relatively complex continuous actions, strong exploration ability is needed to make better decisions. The SAC algorithm with strong exploration ability is selected to solve the optimal strategy of power allocation and energy transfer. The policy network has the following objective function:

[0100]

[0101] where rho π is the distribution learning rate of (s i , a 2,i ), H is the entropy in the evaluation network pi, and alpha is the temperature parameter, which can be automatically adjusted by minimizing the following formula:

[0102]

[0103] where H' is the target entropy. The input of the action network is the state of the agent, and the output is the mean and standard deviation of the agent action distribution. mu is the action network parameter, theta is the evaluation network parameter, and the corresponding loss function is:

[0104]

[0105] In the formula, D is the experience pool. The SAC algorithm uses two evaluation networks and two target evaluation networks. In order to calculate the target Q value, the smaller output of the two evaluation networks is selected to suppress overestimation. The soft Bellman equation can be minimized by training the soft Q function parameters:

[0106]

[0107] In the formula, gamma is the discount factor, theta' represents the parameters of the target network Q θ ', and the evaluation network is updated as follows:

[0108] theta' = tau theta + (1-tau) theta'.

[0109] where τ is a soft update factor. The SAC agent interacts with the environment continuously, aiming to learn the optimal action selection policy, and outputs the power allocation and energy transfer decision to the environment.

[0110] S43: continuously update the policy, when the long-term reward converges, complete the optimization of the total energy efficiency of the system, and in the subsequent execution stage, the agent directly determines the action based on the trained actor network according to the observation.

[0111] In summary, as shown in Figure 2 the processing flow of the two-layer network combined with PPO and SAC is as follows: in the first layer network, the PPO agent observes the state s, the action network calculates the advantage function through the GAE method, the action a1 is sampled from the action network and is transmitted to the evaluation network, the environment and the second layer network, the evaluation network obtains the state value, and the evaluation loss and the action loss are calculated to update the network parameters. In the second layer network, the SAC agent observes the state s and stores the state and the action of the first layer network in the buffer pool, calculates the action a2 through the action network and transmits it to the evaluation network and the environment, and updates the network parameters through the soft target update and back propagation of the evaluation network composed of the two-layer double Q network.

[0112] In order to verify the effectiveness and effect of the method of the application, simulation experiments and analysis are carried out in this embodiment, as follows:

[0113] Figure 3 In the simulation results of the SAC (no AP hibernation), the SAC algorithm is used to optimize only the continuous action power allocation and energy transfer; the SAC (AP hibernation continuous) uses the SAC algorithm, and the value of the AP hibernation is set to a continuous value in [0, 1] for truncation judgment; the proposed method is the method proposed in the application. As can be seen from Figure 3 , the method of the application simultaneously processes continuous and discrete actions, and compared with the SAC (AP hibernation continuous), the AP hibernation strategy is more determined, and the total energy efficiency of the system is improved.

[0114] The embodiment of the application also provides a computer system, which comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program realizes the steps of the high-energy-efficiency energy management method based on nested DRL in the cell-free large-scale MIMO system when executed by the processor.

[0115] The embodiment of the application also provides a computer program product, which comprises a computer program, and the computer program realizes the steps of the high-energy-efficiency energy management method based on nested DRL in the cell-free large-scale MIMO system when executed by the processor.

[0116] Program code for carrying out the methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, causes the steps of the methods of the present application to be implemented. The program code can execute entirely on a machine, partly on a machine, as a stand-alone software package, partly on a machine and partly on a remote machine or entirely on a remote machine or server. The present application is not limited by the programming languages used to write the program code.

Claims

1. A high-efficiency energy management method based on nested DRL in a cellular-free massive MIMO system, characterized in that, Includes the following steps: S1: Based on statistical channel state information, construct an optimization problem with the goal of maximizing the total energy efficiency of the system based on the downlink communication transmission model; S2: Define the representation of states, actions, and rewards in energy management, where states include the reachability rate of each user, the battery level of each AP, and the sleep / activation state; actions include power allocation, sleep / activation decisions, and energy transfer; and rewards consider the long-term total energy efficiency of the system and penalties for failure to meet QoS constraints. S3: Using a heuristic approach, assuming uniform power distribution and no energy transfer, the optimization problem of maximizing the total energy efficiency of the system is decomposed into the first subproblem. The AP dynamic sleep operation output by the first subproblem is passed as known information to the second subproblem. Based on the AP dynamic sleep, power distribution and energy transfer are further optimized to handle the mixed action space. S4: The AP obtains the current environmental status information and, based on this information, uses a nested deep reinforcement learning algorithm composed of a PPO network and a SAC network to adaptively adjust power allocation, energy transfer, and dynamic sleep in dynamic and uncertain environments, thereby achieving energy efficiency optimization while meeting QoS requirements.

2. The high-efficiency energy management method based on nested DRL in a non-cellular massive MIMO system according to claim 1, characterized in that, In step S1, the long-term total energy efficiency of the system is constructed based on the sum of the total system power consumption and the achievable speed of the user. Where R k (i) represents the reachable rate of the k-th user at time i, P tot (i) represents the total power consumption of the system, P represents the total number of system operating hours, and K represents the number of users; the problem of maximizing the total energy efficiency of the system is expressed as: in p represents the minimum demand rate of the k-th user at time i. lk (i) represents the transmit power of the l-th AP to the k-th user at the i-th time, b l (i) represents the activation state of the l-th AP. This represents the transmission power of the l-th AP to the k-th user and the energy required to maintain operation of the l-th A, provided by the power grid. This indicates the power supplied by the power grid for communication with the CPU. This indicates that the transmit power of the l-th AP to the k-th user provided by the EH equipment is equivalent to the operating power of the l-th AP. This indicates the power supplied by the EH device for communication with the CPU. This represents the energy transferred from the l-th AP to the l′-th AP.

3. The high-efficiency energy management method based on nested DRL in a cellular-free massive MIMO system according to claim 1, characterized in that, In step S2, the representation of the state in energy management is defined, specifically including: Where R = [R1, R2, ..., R] K ] T B = [B1, B2, ..., B L ] T S = [S1, S2, ..., S L ] T , where R k B represents the reachable rate of the k-th user. l S represents the battery level of the l-th AP. l ∈{0,1} represents the dormant / active state of the l-th AP, K is the number of users, and L is the number of APs.

4. The high-efficiency energy management method based on nested DRL in a cellular-free massive MIMO system according to claim 1, characterized in that, In step S2, the representation of the action in energy management is defined, specifically including: Where p = [p 11 ,p 12 ,...,p LK ] T b = [b1, b2, ..., b L ] T , Where p lk b represents the transmit power of the l-th AP to the k-th user. l ∈{0,1} represents the sleep / activation decision of the l-th AP, and the action b at time i is the state S at time i+1. This represents the energy transferred from the l-th AP to the l′-th AP.

5. The high-efficiency energy management method based on nested DRL in a non-cellular massive MIMO system according to claim 1, characterized in that, In step S2, the representation of rewards in energy management is defined, specifically including: Where EE represents the system's long-term total energy efficiency. R is a negative constant representing the penalty factor. k and Let represent the reachable rate and threshold for the k-th user, respectively.

6. The high-efficiency energy management method based on nested DRL in a cellular-free massive MIMO system according to claim 2, characterized in that, The first subproblem after splitting in step S3 is as follows: The second subproblem after splitting is as follows:

7. The high-efficiency energy management method based on nested DRL in a cellular-free massive MIMO system according to claim 4, characterized in that, After the optimization problem is decomposed, the action space is correspondingly divided into a discrete action space and a continuous action space as follows: a1 = b, a2=[p T ,HAVE BEEN t,T ] T .

8. The high-efficiency energy management method based on nested DRL in a cellular-free massive MIMO system according to claim 6, characterized in that, Step S4 specifically includes: S41: Considering the discreteness of AP dynamic sleep in the action space, the PPO algorithm is selected to solve the optimal strategy for AP dynamic sleep; the PPO agent interacts with the environment continuously to learn the optimal action selection strategy and output the AP dynamic sleep decision to the environment and the SAC agent. S42: Considering the continuity of power allocation and energy transfer in the action space, the SAC algorithm is selected to solve the optimal strategy for power allocation and energy transfer; the SAC agent continuously interacts with the environment, aiming to learn the optimal action selection strategy and output power allocation and energy transfer decisions to the environment; S43: Continuously update the policy. When the long-term reward converges, optimize the overall energy efficiency of the system. In the subsequent execution phase, the agent determines the action directly based on the observations using the trained Actor network.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the high-efficiency energy management method based on nested DRL in a non-cellular massive MIMO system according to any one of claims 1-8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the high-efficiency energy management method based on nested DRL in a non-cellular massive MIMO system according to any one of claims 1-8.

Images