A method for optimizing energy efficiency in hybrid power supply heterogeneous networks based on A2C algorithm

Abstract

The present invention discloses a method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on an A2C algorithm. The method determines the user location of a small base station according to the number distribution of macro base stations and small base stations. A single small base station is regarded as an intelligent agent, and the state space, action space and reward function of a Markov decision process are established. The small base station user and the environment interact randomly to obtain a state. The action information (s t ,a t ,r t ,s t+1 ) is passed to the critic network; the optimal action obtained by learning each small base station is passed to the macro base station as the state, and the small base station is repeatedly deployed within the coverage of the macro base station to obtain the optimal small base station deployment strategy, that is, the optimal resource allocation; users are connected to the corresponding small base station to obtain a better channel, maximizing the energy efficiency of the heterogeneous network system. The present invention improves the energy efficiency in the A2C algorithm by using the A2C algorithm in reinforcement learning, and uses the Gaussian distribution method to approximate the state action value function, saving the resources of the traditional power grid and the cost of power grid communication energy consumption.

Description

Technical Field

[0001] The present invention relates to the technical field of physical layer of communication system, and in particular to a method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on an A2C algorithm. Background Art

[0002] With the continuous increase in the number of terminals and the rapid growth of data service demand, the traditional single-layer network can no longer meet the needs of the rapid development of current technology, and wireless communication networks are also facing huge challenges. In order to alleviate the huge pressure on the communication network, researchers proposed the A2C (advantage actor-critic network) algorithm. The current wireless access network has developed into an A2C algorithm consisting of macro base stations that meet wide-area access needs and small base stations that meet small-area high-density access needs. In order to support high-speed mobile data services and provide better coverage, the next generation of cellular networks is expected to widely deploy micro-cell base stations or small cell base stations, which can offload some users and traffic from traditional macro base stations. Although the system capacity can be improved and the network coverage can be enhanced, the deployment of a large number of small base stations will also bring new challenges, such as cell interference, resource waste and huge energy consumption. Therefore, it is becoming increasingly important to optimize power grid resources from a global perspective. With the increase in the number of base stations and energy prices, energy efficiency has become an important issue in power grid management. Summary of the invention

[0003] The present invention provides a method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on an A2C algorithm, and optimizes energy efficiency based on the A2C algorithm in reinforcement learning.

[0004] To achieve the above effects, the technical solution of the present invention is as follows:

[0005] The method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on the A2C algorithm includes the following steps:

[0006] Step 1: Build a heterogeneous network system based on the optimization goal. The system consists of macro base stations, small base stations, and users. Consider a single small base station as an intelligent agent, establish the state space, action space, and reward function of the Markov decision process, and calculate the signal-to-interference-noise ratio from the macro base station to the user.

[0007] Step 2: Calculate the total conventional power of the macro base station, build the actor network and critic network according to the defined Markov decision process, and train the heterogeneous network system using the A2C algorithm;

[0008] Step 3: The small base station user and the environment interact to randomly obtain the state s t , the state s t The information is passed to the actor network, which responds to the current state of the environment st and the agent's own state to select the appropriate action a t , get instant rewards t and the state s at time t+1 t+1 , get the action information (s t , a t , r t ,s t+1 );

[0009] Step 4: Send the action information (s t , a t , r t ,s t+1 ) is passed to the critic network, the critic network parameters are updated and the reward is maximized to obtain the optimal small base station deployment strategy, that is, the optimal resource allocation; users are connected to the corresponding small base stations to obtain better channels and maximize the energy efficiency of the heterogeneous network system.

[0010] Among them, the heterogeneous network system includes a macro base station and multiple small base stations, and users are evenly distributed within the coverage area of ​​the base stations.

[0011] Furthermore, in step 1, K small base stations and M users are deployed in the macro base station, the set of small base stations is κ = {0, 1, 2, ..., K}, and the set of users is M = {0, 1, 2, ... M}. Assuming that the traditional macro base station and the attached small base station use the same radio spectrum, the signal to interference and noise ratio γm of user m during time slot t is:

[0012]

[0013] In the formula, g k,m (t) is the average channel gain of the macro base station for user m at time slot t, p k,m (t) is the transmission power of the small base station; g i,m (t) is the average channel gain of user m from other interfering base stations i at time slot t; p i (t) is for user u i (t) the total wireless transmission power of the serving base station i, is the variance of the additive white Gaussian noise at user m.

[0014] Furthermore, the total bandwidth W and bandwidth B of the agent in step 1 are divided into sub-channels of W / B, and during time slot t, user m is assigned to b m (t)∈{0,1,…,W / B}, and The total information rate r achieved by the agent during time slot t sum (t) is:

[0015]

[0016] Small base stations obtain energy from the power grid and renewable energy collection equipment.

[0017] Furthermore, the wireless transmission power obtained from the conventional power grid is set to a positive value, and the wireless transmission power obtained from the renewable energy device is set to a negative value; at time slot t, the total power of the macro base station for:

[0018]

[0019] In the formula, represents the static power, including the board processor, cooling components, etc.; η is a coefficient factor related to the efficiency of the wireless power amplifier; p k (t) is the total wireless transmission power of all users u(t) associated with the macro base station in time slot t;

[0020] The static power To obtain the static power of each base station from the conventional power grid, the total conventional power of the macro base station at time slot t is for:

[0021]

[0022] In the formula, p k,u (t) represents the power used by the user in the macro base station, u k (t) represents user.

[0023] Furthermore, the energy efficiency of the agent in step 1 during time slot t is defined as the ratio of the information rate to the conventional power consumption ρ t ,Right now:

[0024]

[0025] Furthermore, in step 1, the state is the signal-to-interference-noise ratio γ(t) of each user and the battery power e(t) of each small base station, that is:

[0026] s t =(γ1(t),γ2(t),...,γ M (t),e1(t),e2(t),...,e K (t));

[0027] Action a t Represented as the number of users u in the macro base station k (t), the number of subchannels b for user m m (t), the transmission power p of user m k,m (t), that is, action a t =(uk (t),b m (t),p k,m (t));

[0028] Reward t is equivalent to the energy efficiency of heterogeneous network systems, that is, r t =ρ t .

[0029] Furthermore, step 2 uses the A2C algorithm to train the heterogeneous network system. Specifically,

[0030] Step 2.1: Establish a heterogeneous network system according to the optimization goal; set the reinforcement learning framework, initialize the state space, select the optimal action based on the state obtained by the interaction between the intelligent agent and the A2C algorithm environment, and pass the optimal action as state information to the macro base station;

[0031] Step 2.2: Set the state s t Input into the heterogeneous network system to update the state action value function and state value function of the heterogeneous network system;

[0032] Step 2.3: The heterogeneous network undergoes multiple rounds of iterative training until the reward function converges to obtain a trained heterogeneous network.

[0033] Furthermore, in step 2.2, the state action value function and state value function of the A2C algorithm are updated as follows: the state action value function Q of the A2C algorithm is updated. w (s,a) and the state value function V v (s) is updated and the Gaussian distribution method is used to approximate the state action value function Q w (s,a); Use tile encoding to approximate the state value function V v (s), 32 tiles are used for coverage, each covering 4*4=16 tiles; the eligibility trace is introduced to update all action values ​​to different degrees from the beginning of each scene, and the update degree decays according to the time distance, and the advantage function is set in the actor network.

[0034] Furthermore, the state action value function Q w The expression of (s,a) is:

[0035]

[0036] In the formula, w T is a parameter vector, expressed as: w=(w1,w2,...,w n ) T ; Using parameters θ=(θ1,θ2,...,θ n ) T Building Strategy π θ=(a|s), expressed as π θ (s,a)=Pr(a|s,θ);π θ (s,a)~N(μ(s),σ 2 ), μ(s) is the mean value, σ is the standard deviation; ψ(s)=(γ1,γ2,...,γ M ,e1,e2,...,e K ) T , γ represents the signal-to-interference-noise ratio of each user, and e represents the battery power of each small base station.

[0037] Furthermore, the state value function V v (s) is expressed as:

[0038] V v (s) = w It(1) *1+w It(2) *1+...+w It(32) *1,

[0039] In the formula, w It(1) 、w It(2) 、....、w It(32) is the weight value of the activated tile, It(1), It(2), ...., It(32) are the index values ​​of the activated tiles, and other tiles are equal to the inactive feature value * 0.

[0040] Furthermore, step 4 is specifically to convert the action information (s t , a t , r t ,s t+1 ) is passed to the critic network, which returns a value q for the agent's action t Make an evaluation and use the TD error (the deviation between the estimated value and the existing value in the time difference learning method) to update the critic network parameters; t , a t ,q t ) is passed to the actor network, and the action selection probability is updated according to the policy gradient to maximize the reward r t ; The optimal actions learned by each small base station are passed to the macro base station as the state, and the small base stations are repeatedly deployed within the coverage of the macro base station to obtain the optimal small base station deployment strategy, that is, the optimal resource allocation; users are connected to the corresponding small base stations to obtain better channels and maximize the energy efficiency of the heterogeneous network system.

[0041] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0042] The present invention improves the energy efficiency of heterogeneous network systems by training heterogeneous network models using the A2C algorithm in reinforcement learning, uses a Gaussian distribution method to approximate the state-action value function, uses tile encoding to approximate the state-value function, helps the actor network understand the policy gradient, and uses the advantage function to further reduce the variance of the policy gradient, so that the A2C algorithm converges faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings are only used for illustrative purposes and are not to be construed as limitations of the present invention. To better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the drawings.

[0044] Figure 1 A schematic diagram of a flow chart of an energy efficiency optimization method provided by an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of a heterogeneous network system provided by an embodiment of the present invention;

[0046] Figure 3 It is a flow chart of heterogeneous network system training provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Example 1

[0049] For easier understanding, see Figure 1-Figure 2 An embodiment of a method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on an A2C algorithm provided by the present invention comprises the following steps:

[0050] Step 1: Determine the user location of the small base station based on the number distribution of macro base stations and small base stations; regard a single small base station as an intelligent agent, establish the state space, action space and reward function of the Markov decision process; set the reward as the energy efficiency of the intelligent agent;

[0051] Step 2: Construct the actor network and critic network according to the defined Markov decision process; train the heterogeneous network system using the A2C algorithm;

[0052] Step 3: The small base station user and the environment interact to randomly obtain the state s t , the state s t Passed to the actor network, the actor network is based on the state of the current environment s t and the agent's own state to select the appropriate action a t , get instant rewards t and the state s at time t+1 t+1 , get the action information (s t , a t , r t ,s t+1 );

[0053] Step 4: Send the action information (s t , a t , r t ,s t+1 ) is passed to the critic network, which returns q to the action taken by the agent. t The value is evaluated and the TD error is used to update the critic network parameters; (s t , a t ,q t ) is passed to the actor network, and the action selection probability is updated according to the policy gradient to maximize the reward r t ; The optimal actions learned by each small base station are passed to the macro base station as the state, and the small base stations are repeatedly deployed within the coverage of the macro base station to obtain the optimal small base station deployment strategy, that is, the optimal resource allocation; so that users can connect to the corresponding small base station to obtain better channels and maximize the energy efficiency of the heterogeneous network system. When the state of the critic network changes, you only need to re-enter the new state into the actor network to obtain a new small base station deployment strategy.

[0054] The present invention optimizes energy efficiency by using the A2C algorithm in reinforcement learning, saves resources of traditional power grids, and saves the cost of communication network energy consumption.

[0055] Example 2

[0056] Specifically, based on Example 1, the scheme is described in combination with specific implementation examples to further reflect the technical effects of the scheme. Specifically:

[0057] Specifically, in step 1, K small base stations and M users are deployed in the macro base station, the set of small base stations is κ = {0, 1, 2, ..., K}, and the set of users is M = {0, 1, 2, ... M}. Assuming that the traditional macro base station and the attached small base station use the same radio spectrum, the signal to interference noise ratio γ of user m during time slot t is m for:

[0058]

[0059] In the formula, k represents the macro base station, g k,m (t) is the average channel gain of the macro base station for user m at time slot t, p k,m (t) is the transmission power of the small base station; g i,m (t) is the average channel gain of user m from other interfering base stations i at time slot t; p i (t) is for user u i (t) the total wireless transmission power of the serving base station i, is the variance of the additive white Gaussian noise at user m.

[0060] Specifically, the total bandwidth W and bandwidth B of the agent in step 1 are divided into sub-channels of W / B. During time slot t, user m is assigned to b m (t)∈{0,1,…,W / B}, and The total information rate r achieved by the agent during time slot t sum (t) is:

[0061]

[0062] Specifically, the wireless transmission power obtained from the conventional power grid is set to a positive value, and the wireless transmission power obtained from the renewable energy device is set to a negative value; at time slot t, the total power of the macro base station is for:

[0063]

[0064] In the formula, represents the static power; η is a coefficient factor related to the efficiency of the wireless power amplifier; p k (t) is the total wireless transmission power of all users u(t) associated with the macro base station in time slot t;

[0065] The static power To obtain the static power of each base station from the conventional power grid, the total conventional power of the macro base station at time slot t is for:

[0066]

[0067] In the formula, p k,u (t) represents the power used by the user in the macro base station, u k (t) represents user.

[0068] Specifically, the energy efficiency of the agent in step 1 during time slot t is defined as the ratio of the information rate to the conventional power consumption ρt ,Right now:

[0069]

[0070] Specifically, in step 1, the state is the signal-to-interference-noise ratio γ(t) of each user and the battery power e(t) of each small base station, that is:

[0071] s t =(γ1(t),γ2(t),...,γ M (t),e1(t),e2(t),...,e K (t));

[0072] Action a t Represented as the number of users u in the macro base station k (t), the number of subchannels b for user m m (t), the transmission power p of user m k,m (t), that is, action a t =(u k (t),b m (t),p k,m (t)); reward r t is equivalent to the energy efficiency of heterogeneous network systems, that is, r t =ρ t .

[0073] Specifically, Figure 3 As shown, step 2 uses the A2C algorithm to train the heterogeneous network system. Specifically, step 2.1: set the reinforcement learning framework, initialize the state space, select the optimal action according to the state obtained by the interaction between the intelligent agent and the A2C algorithm environment, and pass the optimal action as the state information to the macro base station;

[0074] Step 2.2: Set the state s t Input into the heterogeneous network system to update the state action value function and state value function of the heterogeneous network system;

[0075] Step 2.3: The heterogeneous network undergoes multiple rounds of iterative training until the reward function converges to obtain a trained heterogeneous network.

[0076] In the specific implementation process, it is assumed that there are 20 users in a macro base station, the user positions are randomly distributed, and the initial state is randomly generated; the bandwidth of each subchannel is set to B = 1Hz, and the information rate of each user is r m = [0,1]; Assume that the maximum transmission distance of a single user is P max =1W, so the transmission power of user m is p k,m ∈[-1,1]; static field power η=1, discount factor β=0.99, attenuation factor λ=0.5; most parameters are normalized, and the renewable energy collected in the small base station battery is normalized to e k = [0,1], the signal-to-interference-to-noise ratio between the small base station and the user m it serves is normalized to γ m =[0,1]; the learning rate of the critic network is 0.02, and the learning rate of the actor network is gradually increased from 0.01 to 0.03. The training is done for 10,000 rounds, with 50 cycles per round.

[0077] Specifically, in step 2.2, the state action value function and state value function of the A2C algorithm are updated as follows: the state action value function Q of the A2C algorithm is updated. w (s,a) and the state value function V v (s) is updated and the Gaussian distribution method is used to approximate the state action value function Q w (s,a); Use tile encoding to approximate the state value function V v (s), 32 tiles are used to cover, each covering 4*4=16 tiles, to solve the problem of huge state actions; the critic network introduces the qualification trace to update all action values ​​to different degrees from the beginning of each scene, and the update degree decays according to the time distance, and sets the advantage function in the actor network.

[0078] The advantage function is expressed as: A(s t ,a t )=Q w (s t ,a t )-[τV v (s t )+(1-τ)V v (s t-1 )]; The Q table in the state-action-value function is obtained through reinforcement learning iteration by performing different actions under multiple state information.

[0079] State-action-value function Q w The expression of (s,a) is:

[0080]

[0081] In the formula, w T is a parameter vector, expressed as: w=(w1,w2,...,w n ) T ; Using parameters θ=(θ1,θ2,...,θ n ) T Building Strategy π θ =(a|s), expressed as π θ (s,a)=Pr(a|s,θ);πθ (s,a)~N(μ(s),σ 2 ), μ(s) is the mean value, σ is the standard deviation; ψ(s)=(γ1,γ2,...,γ M ,e1,e2,...,e K ) T , γ represents the signal-to-interference-noise ratio of each user, and e represents the battery power of each small base station.

[0082] Since the Bellman equation cannot be used for huge state actions, function approximation is used to approximate the function; the difference between the A2C algorithm and the AC algorithm is that the A2C algorithm introduces a baseline to construct the advantage function, further reducing the variance and making the A2C algorithm converge faster, so two sets of parameters need to be updated in the algorithm, namely the state action value function Q w (s,a) and the state value function V v (s).

[0083] State value function V v (s) is expressed as:

[0084] V v (s) = w It(1) *1+w It(2) *1+...+w It(32) *1,

[0085] In the formula, w It(1) 、w It(2) 、....、w It(32) is the weight value of the activated tile, It(1), It(2), ...., It(32) are the index values ​​of the activated tiles, and other tiles and inactive feature values ​​* 0 will not appear in the state value function.

[0086] In order to solve the energy waste problem in power grid communication and reduce the energy cost and energy burden of the power grid, the present invention proposes a new type of small base station, which integrates energy collection equipment in the small base station and uses renewable energy (such as wind energy and solar energy) to replace energy supply. The present invention uses a new type of small base station to power it with renewable energy, and uses traditional power grid to power it when the renewable energy in the device is exhausted. In the environment of the A2C algorithm, the new type of small base station is deployed in the coverage area of ​​the traditional macro base station. The small base station unloads traffic from the macro base station, and the macro base station makes joint user scheduling and resource allocation decisions. The A2C algorithm is applied to macro base stations and small base stations, and the small base stations are randomly deployed within the coverage area of ​​the macro base station; according to the type of users connected to the base station, they can be divided into macro base station users and small base station users. Generally, the number of users connected to the macro base station is far greater than that of users connected to the small base station; the A2C algorithm is used to improve the energy efficiency of hybrid power supply in heterogeneous network systems.

[0087] The present invention proposes to optimize the energy efficiency of hybrid power supply based on the A2C algorithm in reinforcement learning, using Gaussian distribution as a parameterized strategy in the actor network to generate continuous random behavior, and using the gradient ascent method to update the strategy parameters. Tile encoding is used in the critic network to estimate the performance of the strategy, and to help the actor network understand the policy gradient, and to use the advantage function to further reduce the variance of the policy gradient.

[0088] As shown below, the pseudo code of the A2C algorithm in an embodiment of the present invention is as follows:

[0089] Table 1 A2C algorithm pseudo code

[0090]

[0091] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for optimizing energy efficiency in hybrid power supply heterogeneous networks based on A2C algorithm. It is characterized in that The following steps are involved: Step 1: Build a heterogeneous network system based on the optimization goal. The system consists of macro base stations, small base stations, and users. Consider a single small base station as an intelligent agent, establish the state space, action space, and reward function of the Markov decision process, and calculate the signal-to-interference-noise ratio from the macro base station to the user. Step 2: Calculate the total conventional power of the macro base station, build the actor network and critic network according to the defined Markov decision process, and train the heterogeneous network system using the A2C algorithm; Step 3: The small base station user and the environment interact to randomly obtain the state s t , the state s t The information is passed to the actor network, which responds to the current state of the environment s t and the agent's own state to select the appropriate action a t , get instant rewards t and the state s at time t+1 t+1 , get the action information (s t , a t , r t ,s t+1 ); K small base stations and M users are deployed in the macro base station, the set of small base stations is κ={0, 1, 2, ..., K}, the set of users is M={0, 1, 2, ..., M}, The state is the signal-to-interference-to-noise ratio γ(t) of each user and the battery power e(t) of each small base station, that is: s t =(γ 1 (t),γ 2 (t),...,γ M (t),e 1 (t),e 2 (t),...,e K (t)); The action a t Denoted as the number of users u in base station k k (t), the number of subchannels b for user m m (t), the transmission power p of user m k,m (t), that is, action a t =(u k (t),b m (t),p k,m (t)) Reward t is equivalent to the energy efficiency of heterogeneous network systems, that is, r t =ρ t ; Step 4: Send the action information (s t , a t , r t ,s t+1 ) is passed to the critic network, the critic network parameters are updated and the reward is maximized to obtain the optimal small base station deployment strategy, that is, the optimal resource allocation; users are connected to the corresponding small base stations to obtain better channels and maximize the energy efficiency of the heterogeneous network system.

2. According to the method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on the A2C algorithm according to claim 1, It is characterized in that The signal to noise ratio (SINR) from the macro base station to the user is calculated in step 1 as follows: K small base stations and M users are deployed in the macro base station, and the user position of the small base station is determined according to the number distribution of the macro base station and the small base station; the set of small base stations is κ = {0, 1, 2, ..., K}, and the set of users is M = {0, 1, 2, ..., M}. Assuming that the traditional macro base station and the attached small base station use the same radio spectrum, the SINR γ of user m during time slot t is m for: In the formula, g k,m (t) is the average channel gain of the macro base station for user m at time slot t, p k,m (t) is the transmission power of the small base station; g i,m (t) is the average channel gain of user m from other interfering base stations i at time slot t; p i (t) is for user u i (t) the total wireless transmission power of the serving base station i, is the variance of the additive white Gaussian noise at user m.

3. According to the method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on the A2C algorithm according to claim 2, It is characterized in that The total conventional power of the macro base station is calculated in step 2 as follows: the total bandwidth W of the heterogeneous network system and the bandwidth B are divided into sub-channels of W / B, and during time slot t, user m is allocated b m (t)∈{0,1,…,W / B}subchannel, The total information rate r achieved by the heterogeneous network system during time slot t sum (t) is: The wireless transmission power obtained from the conventional power grid is set to a positive value, and the wireless transmission power obtained from the renewable energy device is set to a negative value; at time slot t, the total power of the macro base station for: In the formula, represents static power; η is a coefficient factor related to the efficiency of the wireless power amplifier; p k (t) is the total wireless transmission power of all users u(t) associated with the macro base station in time slot t; The static power To obtain the static power of each base station from the conventional power grid, the total conventional power of the macro base station at time slot t is for: In the formula, p k,u (t) represents the power used by the user in the macro base station, and uk(t) represents the user.

4. According to the method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on the A2C algorithm according to claim 3, It is characterized in that The energy efficiency of the heterogeneous network system in step 1 during time slot t is defined as the ratio of information rate to conventional power consumption ρ t ,Right now:

5. According to the method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on the A2C algorithm according to claim 1, It is characterized in that Step 2 uses the A2C algorithm to train the heterogeneous network system. Specifically, Step 2.1: Set up the reinforcement learning framework, initialize the state space, select the optimal action based on the state obtained by the interaction between the agent and the A2C algorithm environment, and pass the optimal action as state information to the macro base station; Step 2.2: Set the state s t Input into the heterogeneous network system to update the state action value function and state value function of the heterogeneous network system; Step 2.3: The heterogeneous network undergoes multiple rounds of iterative training until the reward function converges to obtain a trained heterogeneous network.

6. According to the method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on the A2C algorithm according to claim 5, It is characterized in that In step 2.2, the state action value function and state value function of the A2C algorithm are updated as follows: w (s,a) and the state value function V v (s) is updated and the Gaussian distribution method is used to approximate the state action value function Q w (s,a); Use tile encoding to approximate the state value function V v (s), 32 tiles are used for coverage, each covering 4*4=16 tiles; the eligibility trace is introduced to update all action values ​​to different degrees from the beginning of each scene, and the update degree decays according to the time distance, and the advantage function is set in the actor network.

7. According to the method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on the A2C algorithm according to claim 6, It is characterized in that The state action value function Q w The expression of (s,a) is: In the formula, w T is a parameter vector, expressed as: w = (w 1 ,w 2 ,...,w n ) T ; Use parameter θ = (θ 1 ,θ 2 ,...,θ n ) T ψ(s)=(γ 1 ,γ 2 ,...,γ M ,e 1 ,e 2 ,...,e K ) T , γ represents the signal-to-interference-noise ratio of each user, and e represents the battery power of each small base station.

8. According to the method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on the A2C algorithm according to claim 7, It is characterized in that The state value function V v (s) is expressed as: V v (s)=w It(1) *1+w It(2) *1+...+w It(32) *1, In the formula, w It(1) 、w It(2) 、....、w It(32) is the weight value of the activated tile, It(1), It(2), ...., It(32) are the index values ​​of the activated tiles, and other tiles are equal to the inactive feature value * 0.

9. The method for optimizing energy efficiency in a hybrid power supply heterogeneous network based on the A2C algorithm according to claim 8, It is characterized in that Step 4 is to convert the action information (s t , a t , r t ,s t+1 ) is passed to the critic network, which returns a value q for the agent's action t Make an evaluation and use the TD error to update the critic network parameters; (s t , a t ,q t ) is passed to the actor network, and the action selection probability is updated according to the policy gradient to maximize the reward r t ; The optimal actions learned by each small base station are passed to the macro base station as the state, and the small base stations are repeatedly deployed within the coverage of the macro base station to obtain the optimal small base station deployment strategy, that is, the optimal resource allocation; users are connected to the corresponding small base stations to obtain better channels and maximize the energy efficiency of the heterogeneous network system.

Images