A resource allocation method for optimizing energy and spectrum efficiency in dense heterogeneous networks

Abstract

The present invention provides a resource allocation method for optimizing energy and spectrum efficiency in dense heterogeneous networks, models the resource allocation problem of downlinks in dense heterogeneous networks as a multi-objective optimization problem with the energy efficiency and spectrum efficiency of the system as optimization targets, and the optimization model not only jointly optimizes the performance of the two networks but also considers the network performance of small cells and macro cells at the same time, so as to reduce the cross-layer interference caused by the dense deployment of small base stations to macro cells, solves the problem that resource allocation only jointly optimizes the energy efficiency and spectrum efficiency of small cell networks without considering the performance of macro cell networks, and solves the established multi-objective optimization problem through a resource allocation method for jointly optimizing energy efficiency and spectrum efficiency. The proposed method not only reduces the complexity of the problem but also has high efficiency, and solves the problem that traditional optimization methods have high complexity and low efficiency when solving multi-objective optimization problems in dense heterogeneous networks with complex structures and large scales.

Description

Technical Field

[0001] The present invention relates to the field of resource allocation in dense heterogeneous networks, and in particular to a method for optimizing resource allocation in dense heterogeneous networks with energy and spectrum efficiency. Background Art

[0002] In dense heterogeneous networks, by densely deploying small base stations, the distance between mobile users and service base stations can be reduced, thereby reducing signal power transmission loss. The dense deployment of base stations improves the reuse rate of the spectrum. Although the technology of dense heterogeneous networks solves the problems of hotspot coverage and edge coverage, increases network throughput, and improves the communication quality of users at the edge of the network, greatly promoting the structural progress of dense heterogeneous networks, due to the increase in the density of small base stations, the total power consumption of the system increases accordingly, and the interference between users becomes more and more serious.

[0003] There are some shortcomings in the existing resource allocation methods for optimizing energy efficiency or spectrum efficiency that need to be improved. First, dense heterogeneous networks have high network requirements for energy efficiency and spectrum efficiency. If only one network performance is optimized, other network performances are often sacrificed and resources cannot be used efficiently. Secondly, there are some other methods that study resource allocation problems based on the joint optimization of energy efficiency and spectrum efficiency by establishing multi-objective optimization models with energy efficiency and spectrum efficiency as the goals. However, the proposed resource allocation methods do not consider the network performance of macro cells, and for solving multi-objective optimization problems, the main method is to convert the multi-objective optimization problem into a single-objective optimization problem or to use traditional multi-objective optimization methods to solve it. The dense network structure is complex and large in scale. The traditional resource allocation method has high complexity and low efficiency when solving multi-objective optimization problems in dense heterogeneous networks. Therefore, we make improvements on this and propose a resource allocation method for optimizing energy and spectrum efficiency in dense heterogeneous networks. Summary of the invention

[0004] The purpose of the present invention is to: In view of the current problems raised by the background technology, in order to achieve the above-mentioned invention purpose, the present invention provides the following technical solutions: a method for optimizing resource allocation for energy and spectrum efficiency in a dense heterogeneous network to improve the above-mentioned problems. The present application specifically includes a dense heterogeneous network, wherein the dense heterogeneous network includes a macro base station and N small base stations. The macro base station is associated with M macro base station users, and the small base station B n Related U nThe invention relates to a small base station user, characterized in that the following steps are included: S1: establishing a signal-to-interference-and-noise ratio SINR model, S2: establishing a user rate model, S3: establishing a system capacity model, S4: establishing a system power and system bandwidth model, S5: establishing a system energy efficiency EE and spectrum efficiency SE model, S6: establishing a resource allocation optimization model based on the joint optimization of energy efficiency and spectrum efficiency in a dense heterogeneous network, and S7: an energy and spectrum efficiency optimization resource allocation method.

[0005] As a preferred technical solution of the present application, the set of N+1 base stations in the dense heterogeneous network is represented as B={B0, B1...B n ...B N}, where B0 represents the macro base station, B1-B N represents N small base stations, and the macro base station is associated with M macro base station users, and the set is expressed as MUE = {MUE1, MUE2, ... MUE m ,...MUE M}; Small base station B n Related U n Small base station users, the set is represented by B n UE={1,2,...u n ,...U n The network is based on OFDMA technology. The spectrum is divided into K resource blocks. Each resource block has the same bandwidth B. The set of resource blocks is represented by RB = {RB1, RB2...RB k ...RB K}.

[0006] As the preferred technical solution of the present application, S1 establishes a signal to interference and noise ratio SINR model. If the MUE m RB is assigned k , then MUE m In RB k The SINR on

[0007]

[0008] in, Is the macro base station in RB k Go up to MUE m The channel gain, It's MUE m In RB k N0 is the sum of interference from other small base stations, and N0 is the noise power;

[0009] If B n Associated u n RB is assigned k , then u n In RBk The SINR on

[0010]

[0011] in It is B n To u n The transmission power, It is B n In RB k Up to u n The channel gain, is u n In RB k The interference from macro base stations is is u n In RB k The sum of the interference received from other small base stations.

[0012] As a preferred technical solution of the present application, the S2 establishes a user rate model;

[0013] According to Shannon's formula, we can get MUE m and u n Rate

[0014]

[0015]

[0016] The sum of the rates of all MUEs is

[0017]

[0018] B n The total rate of the associated small base station users is

[0019]

[0020] As a preferred technical solution of the present application, the S3: establishing a system capacity model;

[0021] The system capacity is expressed as

[0022]

[0023] As the preferred technical solution of this application, the S4: establishes a system power and system bandwidth model; the power of each base station includes two parts: transmission power and circuit power, and the total power in the system is

[0024]

[0025] Among them, P MC is the circuit power of the macro base station, PSC is the circuit power of the small base station.

[0026] The total bandwidth consumed in the system is the sum of the bandwidth occupied by all users and is expressed as:

[0027]

[0028] Among them, if Represents RB k is occupied, otherwise it is not occupied.

[0029] As a preferred technical solution of the present application, the S5: establishing a system energy efficiency EE and spectrum efficiency SE model;

[0030] The system energy efficiency and spectrum efficiency are

[0031]

[0032]

[0033] As a preferred technical solution of the present application, the S6: establishing a resource allocation optimization model based on joint optimization of energy efficiency and spectrum efficiency in dense heterogeneous networks;

[0034] Under the constraints of resource blocks, transmit power, and macro base station user SINR threshold, resource blocks and transmit power are allocated simultaneously to jointly optimize the energy efficiency and spectrum efficiency of the system. The multi-objective optimization problem is expressed as:

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] Among them, C1-C3 means that each small base station user can only be allocated one resource block and different resource blocks are allocated between users associated with the same small base station; C4-C5 represents the transmission power of the small base station to its associated users. is a positive value and the total transmit power cannot exceed its maximum total transmit power P ST ; C6 indicates that the signal to noise ratio of each macro base station user on the resource block it occupies cannot be lower than the threshold γ min.

[0043] As the preferred technical solution of this application, S7: Energy and spectrum efficiency optimization resource allocation method:

[0044] S71: Select an agent in the method and define the agent's state, action and reward;

[0045] For the dense heterogeneous network scenario under study, macro base stations in the network are selected as agents, and the actions, states and rewards of the agents are defined according to the decision variables, constraints and optimization objectives of the optimization problem.

[0046] The action of the action (Action, A) agent determines the resource allocation strategy. The agent allocates a resource block and the transmission power of the small base station on the resource block to each small base station user, discretizes the transmission power of the small base station into D levels, and the action set of the agent includes a spectrum allocation action set and a set of transmission power allocation action sets, which are expressed as

[0047]

[0048] In the state (State, S), in order to ensure the minimum SINR requirement of the macro base station user, a state set S={0,1} is defined. If the executed action meets the minimum SINR requirement of each macro base station user on the resource block it occupies, then the state s=1, otherwise s=0;

[0049] The reward (Reward, R) represents the optimization target, which is expressed as R = [R EE ,R SE ], where R EE is a reward aimed at energy efficiency, R SE is the reward targeting spectral efficiency, and F is the set of storage vectors (EE, SE);

[0050] S72: Formulate an action selection strategy for the agent;

[0051] The agent uses a random number x, x∈(0,1), a greedy rate ε, ε∈(0,1), and if x<ε, it selects the best action, otherwise, it selects an action randomly. The value of ε increases with the number of iterations, expressed as:

[0052]

[0053] Among them, ε0 is the initial greed rate set, episode is the number of iterations completed by the agent, and max-episode is the maximum number of iterations set.

[0054] S73: Setting method parameters; Setting energy efficiency weight W EE , spectral efficiency weight W SE, learning rate α, discount rate γ, initial greedy rate ε0, maximum number of iterations max-episode,

[0055] Where W EE ∈[0,1], W SE ∈[0,1] and W EE +W SE =1;

[0056] S74: method initialization;

[0057] Initialize TQ, Q EE , Q SE , a set F of storage vectors (EE, SE);

[0058] S75: initialization state s;

[0059] S76: Select an action according to the greedy strategy;

[0060] S77: Calculation EE, SE;

[0061] S78: Obtain a new state s' according to the state definition;

[0062] S79: Calculate reward;

[0063] S710: Update Q value;

[0064] Under the action selection strategy π, action a is executed in state s, and the Q value update formulas with energy efficiency and spectrum efficiency as the goals are:

[0065]

[0066]

[0067] S711: normalized Q value;

[0068] In order to reflect the fairness of different target Q values, it is necessary to EE and q SE Normalize

[0069]

[0070]

[0071] where the minimum value is mapped to 0 and the maximum value is mapped to 1;

[0072] S712: Update TQ value

[0073] The key to the method is to establish a synthetic target Q value TQ that can reasonably express preferences. π (s,a)

[0074] TQ π (s,a)=W EE scale(q EE )+W SE scale(q SE )

[0075] S713: Update status: s=s';

[0076] S714: When s=1, repeat steps S75-S713 until the set maximum number of iterations is completed, and output the optimal solution in F, that is, the optimal energy efficiency and spectrum efficiency, otherwise, repeat steps S76-S712.

[0077] Compared with the prior art, the present invention has the following beneficial effects:

[0078] In the scheme of this application:

[0079] 1. By modeling the downlink resource allocation problem in dense heterogeneous networks as a multi-objective optimization problem with the system's energy efficiency and spectrum efficiency as optimization targets, the optimization model not only jointly optimizes the performance of the two networks but also considers the network performance of small cells and macro cells at the same time. This solves the problem that existing resource allocation methods only jointly optimize the energy efficiency and spectrum efficiency of small cell networks without considering the performance of macro cells. At the same time, the network performance of small cells and macro cells is optimized to reduce the cross-layer interference caused by the dense deployment of small base stations to macro cells.

[0080] 2. The established multi-objective optimization problem is solved by a resource allocation method that jointly optimizes energy efficiency and spectrum efficiency. The proposed method not only reduces the complexity of the problem but also has high efficiency. It solves the problem of high complexity and low efficiency of traditional optimization methods when solving multi-objective optimization problems in dense heterogeneous networks with complex structures and large scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A flowchart of establishing a multi-objective optimization model for the method for optimizing resource allocation for energy and spectrum efficiency in dense heterogeneous networks provided in this application;

[0082] Figure 2 A computational reward flow chart for the method for optimizing resource allocation for energy and spectrum efficiency in dense heterogeneous networks provided by this application;

[0083] Figure 3 The influence of the greediness rate of the method for optimizing resource allocation for energy and spectrum efficiency in dense heterogeneous networks provided by the present application on the method of the present invention;

[0084] Figure 4The energy efficiency and spectrum efficiency of the system under different small base station densities for the method for optimizing resource allocation for energy and spectrum efficiency in dense heterogeneous networks provided in this application;

[0085] Figure 5 The network capacity diagram of macro cells under different small base station densities for the method for optimizing resource allocation for energy and spectrum efficiency in dense heterogeneous networks provided by this application;

[0086] Figure 6 A schematic diagram of a dense heterogeneous network scenario for the method for optimizing resource allocation for energy and spectrum efficiency in dense heterogeneous networks provided in this application. DETAILED DESCRIPTION

[0087] To make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is a specific implementation of the present invention and is not limited to all embodiments.

[0088] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0089] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0090] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0091] like Figure 1-Figure 6 A method for optimizing resource allocation for energy and spectrum efficiency in a dense heterogeneous network, comprising a dense heterogeneous network, wherein the dense heterogeneous network comprises a macro base station and N small base stations, wherein the macro base station is associated with M macro base station users, and the small base station B n Related U n The invention discloses a small base station user, comprising the following steps: S1: establishing a signal to interference and noise ratio SINR model, S2: establishing a user rate model, S3: establishing a system capacity model, S4: establishing a system power and system bandwidth model, S5: establishing a system energy efficiency EE and spectrum efficiency SE model, S6: establishing a resource allocation optimization model based on the joint optimization of energy efficiency and spectrum efficiency in dense heterogeneous networks, and S7: a resource allocation method for optimizing energy and spectrum efficiency.

[0092] As a preferred implementation, based on the above method, further, the set of N+1 base stations in the dense heterogeneous network is expressed as B={B0, B1...B n ...B N}, where B0 represents the macro base station, B1-B N represents N small base stations, and the macro base station is associated with M macro base station users, and the set is expressed as MUE = {MUE1, MUE2, ... MUE m ,...MUE M}; Small base station B n Related U n Small base station users, the set is represented by B n UE={1,2,...u n ,...U n The network is based on OFDMA technology. The spectrum is divided into K resource blocks. Each resource block has the same bandwidth B. The set of resource blocks is represented by RB = {RB1, RB2...RB k ...RB K}.

[0093] As a preferred implementation, based on the above method, further, the S1 establishes a signal to interference and noise ratio SINR model. If the MUE m Assigned RB k , then MUE m In RB k The SINR on

[0094]

[0095] in, Is the macro base station in RB k Go up to MUE m The channel gain, It's MUE m In RB k N0 is the sum of interference from other small base stations, and N0 is the noise power;

[0096] If B n Associated u n RB is assigned k , then u n In RB k The SINR on

[0097]

[0098] in It is B n To u n The transmission power, It is B nIn RB k Up to u n The channel gain, is u n In RB k The interference from macro base stations is is u n In RB k The sum of the interference received from other small base stations.

[0099] As a preferred implementation, based on the above method, further, the S2: establishing a user rate model;

[0100] According to Shannon's formula, we can get MUE m and u n Rate

[0101]

[0102]

[0103] The sum of the rates of all MUEs is

[0104]

[0105] B n The total rate of the associated small base station users is

[0106]

[0107] As a preferred implementation, based on the above method, further, the S3: establishing a system capacity model;

[0108] The system capacity is expressed as

[0109]

[0110] As a preferred implementation, based on the above method, further, the S4: establishing a system power and system bandwidth model;

[0111] The power of each base station includes two parts: transmission power and circuit power. The total power in the system is

[0112]

[0113] Among them, P MC is the circuit power of the macro base station, P SC is the circuit power of the small base station.

[0114] The total bandwidth consumed in the system is the sum of the bandwidth occupied by all users and is expressed as:

[0115]

[0116] Among them, if Represents RB k is occupied, otherwise it is not occupied.

[0117] As a preferred implementation, based on the above method, further, S5: establish a system energy efficiency EE and spectrum efficiency SE model, the system energy efficiency and spectrum efficiency are respectively

[0118]

[0119]

[0120] As a preferred implementation, based on the above method, further, S6: establishing a resource allocation optimization model based on joint optimization of energy efficiency and spectrum efficiency in dense heterogeneous networks;

[0121] Under the constraints of resource blocks, transmit power, and macro base station user SINR threshold, resource blocks and transmit power are allocated simultaneously to jointly optimize the energy efficiency and spectrum efficiency of the system. The multi-objective optimization problem is expressed as:

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128]

[0129] Among them, C1-C3 means that each small base station user can only be allocated one resource block and different resource blocks are allocated between users associated with the same small base station; C4-C5 means the transmission power P of the small base station to its associated users. un is a positive value and the total transmit power cannot exceed its maximum total transmit power P ST ; C6 indicates that the signal to noise ratio of each macro base station user on the resource block it occupies cannot be lower than the threshold γ min .

[0130] As a preferred implementation, based on the above-mentioned method, further, S7: energy and spectrum efficiency optimization resource allocation method:

[0131] S71: Select an agent in the method and define the agent's state, action and reward;

[0132] For the dense heterogeneous network scenario under study, macro base stations in the network are selected as agents, and the actions, states and rewards of the agents are defined according to the decision variables, constraints and optimization objectives of the optimization problem.

[0133] The action of the action (Action, A) agent determines the resource allocation strategy. The agent allocates a resource block and the transmission power of the small base station on the resource block to each small base station user, discretizes the transmission power of the small base station into D levels, and the action set of the agent includes a spectrum allocation action set and a set of transmission power allocation action sets, which are expressed as

[0134]

[0135] In the state (State, S), in order to ensure the minimum SINR requirement of the macro base station user, a state set S={0,1} is defined. If the executed action meets the minimum SINR requirement of each macro base station user on the resource block it occupies, then the state s=1, otherwise s=0;

[0136] The reward (Reward, R) represents the optimization target, which is expressed as R = [R EE ,R SE ], where R EE is a reward aimed at energy efficiency, R SE is the reward targeting spectral efficiency, and F is the set of storage vectors (EE, SE);

[0137] S72: Formulate an action selection strategy for the agent;

[0138] The agent uses a random number x, x∈(0,1), a greedy rate ε, ε∈(0,1), and if x<ε, it selects the best action, otherwise, it selects an action randomly. The value of ε increases with the number of iterations, expressed as:

[0139]

[0140] Among them, ε0 is the initial greed rate set, episode is the number of iterations completed by the agent, and max-episode is the maximum number of iterations set.

[0141] S73: Setting method parameters; Setting energy efficiency weight W EE , spectral efficiency weight W SE , learning rate α, discount rate γ, initial greedy rate ε0, maximum number of iterations max-episode,

[0142] Where W EE∈[0,1], W SE ∈[0,1] and W EE +W SE =1;

[0143] S74: method initialization;

[0144] Initialize TQ, Q EE , Q SE , a set F of storage vectors (EE, SE);

[0145] S75: initialization state s;

[0146] S76: Select an action according to the greedy strategy;

[0147] S77: Calculation EE, SE;

[0148] S78: Obtain a new state s' according to the state definition;

[0149] S79: Calculate reward;

[0150] S710: Update Q value;

[0151] Under the action selection strategy π, action a is executed in state s, and the Q value update formulas with energy efficiency and spectrum efficiency as the goals are:

[0152]

[0153]

[0154] S711: normalized Q value;

[0155] In order to reflect the fairness of different target Q values, it is necessary to EE and q SE Normalize

[0156]

[0157]

[0158] where the minimum value is mapped to 0 and the maximum value is mapped to 1;

[0159] S712: Update TQ value

[0160] The key to the method is to establish a synthetic target Q value TQ that can reasonably express preferences. π (s,a)

[0161] TQ π (s,a)=W EE scale(q EE )+WSE scale(q SE )

[0162] S713: Update status: s=s';

[0163] S714: When s=1, repeat steps S75-S713 until the set maximum number of iterations is completed, and the optimal solution in F, that is, the optimal energy efficiency and spectrum efficiency, is output; otherwise, repeat steps S76-S712.

[0164] Working principle: During the use of the present invention, the dense heterogeneous network includes a macro base station and N small base stations. The set of N+1 base stations in the dense heterogeneous network is represented by B={B0,B1...B n ...B N}, where B0 represents the macro base station, B1-B N represents N small base stations, and the macro base station is associated with M macro base station users, and the set is expressed as MUE = {MUE1, MUE2, ... MUE m ,...MUE M}; Small base station B n Related U n Small base station users, the set is represented by B n UE={1,2,...u n ,...U n The network is based on OFDMA technology. The spectrum is divided into K resource blocks. Each resource block has the same bandwidth B. The set of resource blocks is represented by RB = {RB1, RB2...RB k ...RB K Step S1: Establishing a signal to interference plus noise ratio (SINR) model. Step S1: Establishing a signal to interference plus noise ratio (SINR) model. If MUE m RB is assigned k , then MUE m In RB k The SINR on

[0165]

[0166] in, Is the macro base station in RB k Go up to MUE m The channel gain, It's MUE m In RB k N0 is the sum of interference from other small base stations, and N0 is the noise power;

[0167] If B n Associated u n RB is assigned k, then u n In RB k The SINR on

[0168]

[0169] in It is B n To u n The transmission power, It is B n In RB k Up to u n The channel gain, is u n In RB k The interference from macro base stations is is u n In RB k The sum of interferences received from other small base stations;

[0170] S2: establishing a user rate model, wherein S2: establishing a user rate model;

[0171] According to Shannon's formula, we can get MUE m and u n Rate

[0172]

[0173]

[0174] The sum of the rates of all MUEs is

[0175]

[0176] B n The total rate of the associated small base station users is

[0177]

[0178] S3: establishing a system capacity model, wherein S3: establishing a system capacity model;

[0179] The system capacity is expressed as

[0180]

[0181] S4: Establishing a system power and system bandwidth model, wherein S4: Establishing a system power and system bandwidth model;

[0182] The power of each base station includes two parts: transmission power and circuit power. The total power in the system is

[0183]

[0184] Among them, P MC is the circuit power of the macro base station, P SC is the circuit power of the small base station;

[0185] The total bandwidth consumed in the system is the sum of the bandwidth occupied by all users and is expressed as:

[0186]

[0187] Among them, if Represents RB k occupied, otherwise unoccupied;

[0188] S5: Establishing a system energy efficiency EE and spectrum efficiency SE model, wherein S5: Establishing a system energy efficiency EE and spectrum efficiency SE model;

[0189] The system energy efficiency and spectrum efficiency are

[0190]

[0191]

[0192] S6 establishes a resource allocation optimization model based on the joint optimization of energy efficiency and spectrum efficiency in dense heterogeneous networks. S6: establishes a resource allocation optimization model based on the joint optimization of energy efficiency and spectrum efficiency in dense heterogeneous networks. Under the constraints of resource blocks, transmit power and macro base station user SINR threshold, resource blocks and transmit power are allocated at the same time to jointly optimize the energy efficiency and spectrum efficiency of the system. The multi-objective optimization problem is expressed as:

[0193]

[0194]

[0195]

[0196]

[0197]

[0198]

[0199]

[0200] Among them, C1-C3 means that each small base station user can only be allocated one resource block and different resource blocks are allocated between users associated with the same small base station; C4-C5 means the transmission power P of the small base station to its associated users. unis a positive value and the total transmit power cannot exceed its maximum total transmit power P ST ; C6 indicates that the signal to noise ratio of each macro base station user on the resource block it occupies cannot be lower than the threshold γ min S7: Energy and spectral efficiency optimization resource allocation, S7: Energy and spectral efficiency optimization resource allocation method:

[0201] S71: Select an agent in the method and define the agent's state, action and reward;

[0202] For the dense heterogeneous network scenario under study, macro base stations in the network are selected as agents, and the actions, states and rewards of the agents are defined according to the decision variables, constraints and optimization objectives of the optimization problem.

[0203] The action of the action (Action, A) agent determines the resource allocation strategy. The agent allocates a resource block and the transmission power of the small base station on the resource block to each small base station user, discretizes the transmission power of the small base station into D levels, and the action set of the agent includes a spectrum allocation action set and a set of transmission power allocation action sets, which are expressed as

[0204]

[0205] In the state (State, S), in order to ensure the minimum SINR requirement of the macro base station user, a state set S={0,1} is defined. If the executed action meets the minimum SINR requirement of each macro base station user on the resource block it occupies, then the state s=1, otherwise s=0;

[0206] The reward (Reward, R) represents the optimization target, which is expressed as R = [R EE ,R SE ], where R EE is a reward aimed at energy efficiency, R SE is the reward targeting spectral efficiency, and F is the set of storage vectors (EE, SE);

[0207] S72: Formulate an action selection strategy for the agent;

[0208] The agent uses a random number x, x∈(0,1), a greedy rate ε, ε∈(0,1), and if x<ε, it selects the best action, otherwise, it selects an action randomly. The value of ε increases with the number of iterations, expressed as:

[0209]

[0210] Among them, ε0 is the initial greed rate set, episode is the number of iterations completed by the agent, and max-episode is the maximum number of iterations set.

[0211] S73: Setting method parameters; Setting energy efficiency weight W EE , spectral efficiency weight W SE , learning rate α, discount rate γ, initial greedy rate ε0, maximum number of iterations max-episode,

[0212] Where W EE ∈[0,1], W SE ∈[0,1] and W EE +W SE =1;

[0213] S74: method initialization;

[0214] Initialize TQ, Q EE , Q SE , a set F of storage vectors (EE, SE);

[0215] S75: initialization state s;

[0216] S76: Select an action according to the greedy strategy;

[0217] S77: Calculation EE, SE;

[0218] S78: Obtain a new state s' according to the state definition;

[0219] S79: Calculate reward;

[0220] S710: Update Q value;

[0221] Under the action selection strategy π, action a is executed in state s, and the Q value update formulas with energy efficiency and spectrum efficiency as the goals are:

[0222]

[0223]

[0224] S711: normalized Q value;

[0225] In order to reflect the fairness of different target Q values, it is necessary to EE and q SE Normalize

[0226]

[0227]

[0228] where the minimum value is mapped to 0 and the maximum value is mapped to 1;

[0229] S712: Update TQ value

[0230] The key to the method is to establish a synthetic target Q value TQ that can reasonably express preferences. π (s,a)

[0231] TQ π (s,a)=W EE scale(q EE )+W SE scale(q SE )

[0232] S713: Update status: s=s';

[0233] S714: When s=1, repeat steps S75-S713 until the set maximum number of iterations is completed, and the optimal solution in F, that is, the optimal energy efficiency and spectrum efficiency, is output; otherwise, repeat steps S76-S712.

[0234] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.

Claims

1. A method for optimizing resource allocation for energy and spectrum efficiency in a dense heterogeneous network, comprising a dense heterogeneous network, wherein the dense heterogeneous network comprises a macro base station and N small base stations, wherein the macro base station is associated with M macro base station users, and the small base station B n Related U n A small base station user, characterized in that: The method comprises the following steps: S1: establishing a signal to interference and noise ratio SINR model, S2: establishing a user rate model, S3: establishing a system capacity model, S4: establishing a system power and system bandwidth model, S5: establishing a system energy efficiency EE and spectrum efficiency SE model, S6: establishing a resource allocation optimization model based on joint optimization of energy efficiency and spectrum efficiency in dense heterogeneous networks, and S7: energy and spectrum efficiency optimization resource allocation method; The set of N+1 base stations in the dense heterogeneous network is represented by B={B0,B1...B n ...B N }, where B0 represents the macro base station, B1-B N represents N small base stations, and the macro base station is associated with M macro base station users, and the set is expressed as MUE = {MUE1, MUE2, ... MUE m ,...MUE M }; Small base station B n Related U n Small base station users, the set is represented by B n UE={1,2,...u n ,...U n The network is based on OFDMA technology. The spectrum is divided into K resource blocks. Each resource block has the same bandwidth B. The set of resource blocks is represented by RB = {RB1, RB2...RB k ...RB K }; S1 establishes a signal-to-interference-plus-noise ratio (SINR) model. If MUE m Assigned RB k , then MUE m In RB k The SINR on in, Is the macro base station in RB k Go up to MUE m The channel gain, It's MUE m In RB k N0 is the sum of interference from other small base stations, and N0 is the noise power; If B n Associated u n Assigned RB k , then u n In RB k The SINR on in It is B n To u n The transmission power, It is B n In RB k Up to u n The channel gain, is u n In RB k The interference from macro base stations is is u n In RB k The sum of interferences received from other small base stations; S2: establishing a user rate model; According to Shannon's formula, we can get MUE m and u n Rate The sum of the rates of all MUEs is B n The total rate of the associated small base station users is S3: establishing a system capacity model; The system capacity is expressed as S4: establishing a system power and system bandwidth model; The power of each base station includes two parts: transmission power and circuit power. The total power in the system is Among them, P MC is the circuit power of the macro base station, P SC is the circuit power of the small base station; The total bandwidth consumed in the system is the sum of the bandwidth occupied by all users and is expressed as: Among them, if Represents RB k occupied, otherwise unoccupied; S5: establishing a system energy efficiency EE and spectrum efficiency SE model; The system energy efficiency and spectrum efficiency are S6: establishing a resource allocation optimization model based on joint optimization of energy efficiency and spectrum efficiency in dense heterogeneous networks; Under the constraints of resource blocks, transmit power, and macro base station user SINR threshold, resource blocks and transmit power are allocated simultaneously to jointly optimize the energy efficiency and spectrum efficiency of the system. The multi-objective optimization problem is expressed as: s.t.C1: C2: C3: C4: C5: C6: Among them, C1-C3 means that each small base station user can only be allocated one resource block and different resource blocks are allocated between users associated with the same small base station; C4-C5 represents the transmission power of the small base station to its associated users. is a positive value and the total transmit power cannot exceed its maximum total transmit power P ST ; C6 indicates that the signal to noise ratio of each macro base station user on the resource block it occupies cannot be lower than the threshold γ min ; S7: Energy and spectrum efficiency optimization resource allocation method: S71: Select an agent in the method and define the agent's state, action and reward; For the dense heterogeneous network scenario under study, macro base stations in the network are selected as agents, and the actions, states and rewards of the agents are defined according to the decision variables, constraints and optimization objectives of the optimization problem. The action of the agent in the above action determines the resource allocation strategy. The agent allocates a resource block and the transmit power of the small base station on the resource block to each small base station user, discretizes the transmit power of the small base station into D levels, and the action set of the agent includes a spectrum allocation action set and a set of transmit power allocation action sets, which are expressed as In the formula, A represents action; In order to ensure the minimum SINR requirement of the macro base station user in the state, a state set S={0,1} is defined. If the executed action satisfies the minimum SINR requirement of each macro base station user on the resource block it occupies, then the state s=1, otherwise s=0; The reward represents the optimization target, expressed as R = [R EE ,R SE ], where R EE is a reward aimed at energy efficiency, R SE is the reward targeting spectral efficiency, and F is the set of storage vectors (EE, SE); S72: Formulate an action selection strategy for the agent; The agent uses a random number x, x∈(0,1), a greedy rate ε, ε∈(0,1), and if x<ε, it selects the best action, otherwise, it selects an action randomly. The value of ε increases with the number of iterations, expressed as: Among them, ε0 is the initial greed rate set, episode is the number of iterations completed by the agent, and max-episode is the maximum number of iterations set. S73: Setting method parameters; Setting energy efficiency weight W EE , spectral efficiency weight W SE , learning rate α, discount rate γ, initial greedy rate ε0, maximum number of iterations max-episode, Where W EE ∈[0,1], W SE ∈[0,1] and W EE +W SE =1; S74: method initialization; Initialize TQ, Q EE , Q SE , a set F of storage vectors (EE, SE); S75: initialization state s; S76: Select an action according to the greedy strategy; S77: Calculation EE, SE; S77: Obtain a new state s' according to the state definition; S79: Calculate reward; S710: Update Q value; Under the action selection strategy π, action a is executed in state s, and the Q value update formulas with energy efficiency and spectrum efficiency as the goals are: S711: normalized Q value; In order to reflect the fairness of different target Q values, it is necessary to EE and q SE Normalize where the minimum value is mapped to 0 and the maximum value is mapped to 1; S712: Update TQ value The key to the method is to establish a synthetic target Q value TQ that can reasonably express preferences. π (s,a) TQ π (s,a)=W EE scale(q EE )+W SE scale(q SE ) S713: Update status: s=s'; S714: When s=1, repeat steps S75-S713 until the set maximum number of iterations is completed, and the optimal solution in F, that is, the optimal energy efficiency and spectrum efficiency, is output; otherwise, repeat steps S76-S712.

Images