A method for user association and resource allocation in cellular networks based on deep reinforcement learning

Abstract

The present invention discloses a cellular network user association and resource allocation method based on deep reinforcement learning, which belongs to the intersection of mobile communication technology and artificial intelligence, and includes the following steps: Step 1, in a dynamic heterogeneous cellular network, construct an uplink network system including macro base stations, micro base stations and femto base stations, a central controller and multiple mobile user devices; Step 2, construct a user association and resource allocation model under a dynamic heterogeneous cellular network, and describe the user association and resource allocation model as a mixed integer nonlinear programming optimization problem with constraints; Step 3, construct an independent reward mixed action space multi-agent proximal strategy optimization algorithm based on deep reinforcement learning; Step 4, adopt a centralized training and distributed execution architecture, and optimize the scheduling of user association and resource allocation decisions based on the algorithm. The present invention can efficiently perform user association, resource management and optimization on the basis of ensuring the quality of user service.

Description

Technical Field

[0001] The present invention belongs to the intersection of mobile communication technology and artificial intelligence, and specifically relates to a cellular network user association and resource allocation method based on deep reinforcement learning. Background Art

[0002] With the rapid development of mobile communication technology, the number of wireless users and IoT devices and their service demands are increasing, resulting in the dual challenges of scarce resources and rising energy consumption. User association revolves around effectively connecting users to the most appropriate cell type, taking into account factors such as signal strength, user mobility, and load balancing. Resource allocation involves the allocation of limited resources such as subchannels and power, and is key to optimizing network efficiency and user satisfaction. Heterogeneous cellular networks have become a key solution to achieve efficient wireless communications. However, heterogeneous network environments are highly dynamic, time-varying, and complex.

[0003] Traditional methods usually rely on preset rules or mathematical optimization models for user association and resource allocation, which cannot fully adapt to the dynamic changes in the network environment and the diversity of user needs. For example, the user's location, network congestion, and service requirements change at any time, and traditional methods are not responsive enough to deal with these dynamic and complex factors. Therefore, effective user association and resource allocation solutions are essential to improve spectrum efficiency and energy efficiency, unleash the true potential of heterogeneous cellular networks, and provide sustainable high-performance wireless communications.

[0004] Deep reinforcement learning (DRL) has developed into a mainstream method for solving the problem of target optimization in cellular heterogeneous networks. Reinforcement learning is a machine learning technology that can learn the optimal strategy through interaction with the environment. It can automatically learn how to optimize user association and resource allocation under different network conditions. At the same time, with the development of multi-agent reinforcement learning, by deploying multiple agents in the same environment, it can better handle the dynamics and complexity of the environment, promote cooperation and coordination between agents, and thus improve the efficiency of the system. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a cellular network user association and resource allocation method based on deep reinforcement learning. From the perspectives of user association, sub-channel and power resource allocation, a hybrid action space multi-agent proximal strategy optimization algorithm based on independent rewards is constructed, and a framework with centralized training and distributed execution is introduced to make joint decisions on user association, channel allocation and power allocation, aiming to maximize the long-term utility of the system while ensuring user service quality requirements.

[0006] The technical solution of the present invention is as follows:

[0007] A cellular network user association and resource allocation method based on deep reinforcement learning comprises the following steps:

[0008] Step 1: In a dynamic heterogeneous cellular network, an uplink network system including a macro base station, a pico base station, a femto base station, a central controller, and multiple mobile user devices is constructed;

[0009] Step 2: construct a user association and resource allocation model in a dynamic heterogeneous cellular network, and describe the user association and resource allocation model as a mixed integer nonlinear programming optimization problem with constraints;

[0010] Step 3: Construct a multi-agent proximal strategy optimization algorithm in an independent reward mixed action space based on deep reinforcement learning;

[0011] Step 4: Adopt a centralized training and distributed execution architecture to optimize scheduling user association and resource allocation decisions based on algorithms.

[0012] Furthermore, in step 1, the uplink network system is composed of Macro base stations, Pico base stations, A femtocell and a group of randomly distributed mobile user devices composition, For the Mobile user equipment, is the total number of mobile user devices; the pico base station, femto base station and mobile user devices are all located within the coverage of the macro base station, and all base stations in the system are ,in, is the base station serial number, is the total number of base stations, , each mobile user equipment is equipped with an antenna, and all mobile user equipment within the coverage of the base station share the uplink subchannel; the set of all subchannels in the system is , is the subchannel number, is the total number of subchannels;

[0013] In the uplink network system, each mobile user equipment moves randomly within the coverage area of ​​the base station, and the location of the mobile user equipment is updated in each time slot; No. The moving speed of a mobile user device is recorded as , is the minimum moving speed, is the maximum value of the moving speed, time slot No. The direction of a mobile user equipment is ; Each mobile user device selects a matching base station for communication according to environmental conditions, Mobile user equipment in time slot With Base station association express, ,in , when When A mobile user equipment selects a base station ,otherwise ; for Mobile user equipment in time slot With The association of base stations;

[0014] For subchannel allocation, Mobile user equipment in time slot Selected Subchannel express, ,in , if , then the subchannel By Mobile user equipment occupies, otherwise ; For the Mobile user equipment in time slot Selected sub-channels.

[0015] Furthermore, in the uplink network system, the mobile user equipment sends a signal to the base station in each time slot, and the upper limit of the transmission power is , the lower limit is , the transmission power is adjusted in real time according to the environmental status of the sub-channel, which includes status information and user needs; No. The mobile user equipment is The transmit power on the subchannel is recorded as ; In the time slot When occupying The subchannel A mobile user device sends The signal-to-noise ratio of the signal transmitted by a base station is:

[0016] ;

[0017] in, For time slot Occupancy The subchannel A mobile user device sends The signal-to-noise ratio of the signal transmitted by each base station; It is a time slot No. The subchannels are from Mobile user equipment to The channel gain of each base station; Different from No. Mobile user equipment; For the Mobile user equipment in time slot With The association of base stations; For the Mobile user equipment in time slot Selected sub-channels; For the time slot No. The mobile user equipment is The transmit power on each subchannel; For time slot No. The subchannels are from Mobile user equipment to The channel gain of each base station; represents additive white Gaussian noise;

[0018] The channel gain is determined by small-scale fading, path loss, and shadowing. At that time, On the subchannel The connection between the base stations Mobile User Equipment Maximum uplink transmission rate It is expressed as:

[0019] ;

[0020] in, is the bandwidth of each subchannel in the system;

[0021] Since it is assumed that each mobile user equipment can only connect to one base station and occupy one subchannel in the same time slot, Mobile User Equipment In time slot Rate for:

[0022] ;

[0023] Time slot Include Total transmission rate of mobile user equipment for:

[0024] ;

[0025] Time slot Spectral efficiency for:

[0026] ;

[0027] Time slot Energy efficiency for:

[0028] .

[0029] Furthermore, the specific process of step 2 is as follows:

[0030] The construction system optimization objective function is:

[0031] ;

[0032] in, For time slot The system optimization objective function; and are weight coefficients, which are used to indicate the relative importance of spectrum efficiency and energy efficiency. ;

[0033] Under the constraints of the number of optional base stations, spectrum resources, mobile user equipment transmit power threshold and service quality requirement threshold, a user association and resource allocation model is constructed, and user association, channel allocation and power control decisions are used as model optimization variables to maximize the long-term average utility return of the model; the user association and resource allocation model is expressed as the following mixed integer nonlinear programming optimization problem with constraints:

[0034] ;

[0035] in, To maximize the long-term average utility return of the model; is the total number of time slots; is the set of total time slots; Requirement threshold for quality of service; constraint Indicates Mobile User Equipment How to select the binary variables of base stations and subchannels, the value range is 0 or 1; constraints Indicates that a mobile user device can only be associated with one base station in each time slot; constraint Indicates that a mobile user device can only be associated with one subchannel in each time slot; constraint Indicates that the total resources selected for all mobile user equipment cannot exceed the available base stations; constraint Indicates that the total resources selected for all mobile user devices cannot exceed the upper limit of available channels; constraint Represents the transmit power range of each mobile user equipment; constraint Indicates the service quality requirement threshold for the uplink rate of mobile user equipment.

[0036] Furthermore, in step 3, according to the mobility of mobile user devices and the dynamic changes of heterogeneous cellular networks, the optimal user association and sub-channel and power resource allocation strategy is solved by an independent reward hybrid action space multi-agent proximal strategy optimization algorithm based on deep reinforcement learning;

[0037] Establish the global state space, local observation space, action space and reward space corresponding to the system. Each mobile user device in the heterogeneous cellular network acts as an independent intelligent agent. The number of intelligent agents is equal to the number of mobile user devices. Each mobile user device has independent observations and actions. Each mobile user device reports its local state and observation to the central controller in real time. The central controller collects and maintains the global state information as the global state space of the system.

[0038] In time slot The global state space is represented as ,in, Indicates that each mobile user equipment is in the time slot Whether the service quality requirements are met, , when the user meets the minimum rate requirement ,otherwise ; represents the number of interfered mobile user equipment in the occupied channel, Indicates that each mobile user equipment is in the time slot The size of the interference, Indicates that the mobile user equipment and the base station are in the time slot Communication distance;

[0039] The local observation space of each mobile user device is expressed as ,in, Indicates Mobile user equipment in time slot Whether the service quality requirements are met, when the mobile user equipment meets the minimum rate requirements ,otherwise , Indicates Mobile user equipment in time slot The channel gain of the occupied subchannel, Indicates Mobile user equipment in time slot The disturbance is small or large, Indicates Mobile user equipment in time slot The distance to the communicating base station;

[0040] The action space represents the set of actions that each mobile user device can take in the current given state. The action space is defined as , and Respectively represent Mobile user equipment in time slot The selected base station index and subchannel index; For the Mobile user equipment in time slot The transmission power, The value is a continuous value within the range of

[0041] The following reward function is designed in the reward space:

[0042] ;

[0043] in, For the Mobile user equipment in time slot The reward function value of is the weight coefficient used to control the importance of subtasks; For time slot No. The transmit power of each mobile user equipment; Indicates The user rate threshold for a mobile user device under the service quality requirement is defined as:

[0044] ;

[0045] Average reward of the system for:

[0046] ;

[0047] Furthermore, the algorithm is used to train the user association and resource allocation model, and each agent takes maximizing the weighted utility of system spectrum efficiency and energy efficiency as the optimization goal to achieve the learning of user association and resource allocation strategies;

[0048] During the training process, each agent has its own actor network, and all agents in the system share the algorithm architecture of a critic network. An independent reward mechanism is designed for each agent. Combined with the actions taken based on the agent's observation space, the trajectory is independently sampled and input into its corresponding actor network, and the network parameters are updated according to the trajectory. A branch network layer matching the number of action types is added to each actor network to handle mixed action spaces. The branch network layer shares several previous neural networks. The discrete action layer outputs the probabilities of different actions, the continuous action layer outputs the mean and standard deviation of the action, and the discrete action layer outputs the joint classification distribution, specifically:

[0049] ;

[0050] in, is the strategy for the discrete action part, which means that in a given local observation space Under the condition of The distribution of discrete actions in represents the discrete action part, For the The set of parameters of the actor network, The actor network corresponds to Agent, also corresponding to Mobile user equipment; is the sequence number of all possible actions in the class, Represents the number of all possible actions in the class; For Action probability; Is an indicator function, indicating whether a specific action is selected ; The output of the continuous action layer that conforms to the Gaussian distribution is expressed as:

[0051] ;

[0052] in, is the strategy for the continuous action part, which means that in a given local observation space Under the condition of The distribution of continuous actions in represents normal distribution; is the mean value of the action; is the standard deviation of the action;

[0053] The loss clipping strategy in the proximal policy optimization method is used to calculate the loss function to promote the training of the actor network and encourage exploration by maximizing entropy. An action masking method is designed to mask unavailable actions based on the observation information of the agent and set the selection probability of unavailable actions to zero. Each agent shares a global critic network to evaluate the action decision of each agent. The global critic network receives the global state and the actions of all agents in the current time slot as input, evaluates the value of the joint actions of all agents in the system, and guides the training process.

[0054] Furthermore, the specific process of step 4 is as follows:

[0055] Step 4.1, initialize the network model parameters and the interaction trajectory between the mobile user device and the environment in the independent reward mixed action space multi-agent proximal strategy optimization algorithm based on deep reinforcement learning;

[0056] Step 4.2, the mobile user equipment collects its own observation information, and the central controller collects the service requirements, location, and channel status of the mobile user equipment as the global environment status;

[0057] Step 4.3: input the global state space and the local observation space into the algorithm, and the algorithm outputs the configuration strategy of user association and resource allocation according to the state of the mobile user device;

[0058] Step 4.4: Calculate the system optimization target and reward, update the model state and mobile user device observation information, and obtain the next state;

[0059] Step 4.5: Save the action space, global state space, reward space and next state space information in the environment, and update the network model parameters of the algorithm;

[0060] Step 4.6: Conduct centralized iterative training to optimize the algorithm model;

[0061] Step 4.7: Each mobile user device executes its own tasks in a distributed manner according to its own model parameters, and finally realizes the best scheduling strategy for user association and resource allocation.

[0062] The beneficial technical effects brought about by the present invention are as follows: the present invention considers an uplink heterogeneous cellular network scenario including a macro base station, a micro-micro base station and a femto-micro base station, a central controller and multiple movable user devices, and solves the user association and resource allocation problems in heterogeneous cellular networks by introducing an independent reward hybrid action space multi-agent proximal strategy optimization algorithm based on deep reinforcement learning, so that users in the system can perform real-time optimized resource scheduling, have intelligent scheduling capabilities, improve the rationality of resource allocation and user association tasks, and while ensuring that the user service quality QoS meets the requirements, improve the system utility of SE and EE, and achieve efficient and energy-saving green communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flowchart of the steps of the method for cellular network user association and resource allocation based on deep reinforcement learning of the present invention.

[0064] Figure 2 It is a structural diagram of the heterogeneous cellular network system of the present invention.

[0065] Figure 3 This is a diagram of the spectrum efficiency training results obtained by optimizing the multi-agent proximal strategy optimization algorithm in an independent reward mixed action space based on deep reinforcement learning in an embodiment of the present invention.

[0066] Figure 4 This is a diagram of the energy efficiency training results obtained by optimizing the multi-agent proximal strategy optimization algorithm in an independent reward mixed action space based on deep reinforcement learning in an embodiment of the present invention.

[0067] Figure 5 This is a diagram of the system optimization target training results obtained by optimizing the multi-agent proximal strategy optimization algorithm in an independent reward mixed action space based on deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0069] In heterogeneous cellular networks, traditional user association and resource allocation methods often rely on pre-set rules or simple optimization algorithms, which are difficult to adapt to the dynamically changing network environment and user needs, resulting in low resource utilization efficiency, high energy consumption, and failure to meet user QoS requirements. This invention proposes a cellular network user association and resource allocation method based on deep reinforcement learning. By introducing an independent reward hybrid action space multi-agent proximal policy optimization algorithm (MAPPO) based on deep reinforcement learning, and adopting a centralized training and distributed execution architecture, it aims to solve the user association and resource allocation problems in heterogeneous cellular networks, improve the system's resource utilization efficiency and user service quality, and promote the further development and application of heterogeneous cellular networks. Figure 1 As shown, the method of the embodiment of the present invention mainly includes the following steps:

[0070] Step 1: In a dynamic heterogeneous cellular network, an uplink network system including a macro base station, a pico base station, a femto base station, a central controller, and multiple mobile user devices is constructed, wherein the uplink network system supports user association, channel allocation, and power allocation in a network uplink;

[0071] The system structure of the embodiment of the present invention is as follows Figure 2As shown in FIG. 1 , a typical heterogeneous cellular network uplink system including macro base stations, pico base stations and femto base stations, a central controller and multiple mobile user devices (hereinafter referred to as users) is considered. The uplink network system consists of Macro base stations MBS, pico base station PBS, A femtocell base station FBS and a group of randomly distributed mobile user devices composition, For the Mobile user equipment, is the total number of mobile user devices. The elliptical dotted areas in the figure correspond to the coverage areas of macro base stations, pico base stations and femto base stations respectively. The curved dotted lines represent the movement trajectories of mobile user devices. The solid arrows represent the uplink transmission links, and the dotted arrows represent the interference links. The pico base stations, femto base stations and mobile user devices are all located within the coverage area of ​​the macro base stations. Let all base stations in the system model be ,in, is the base station serial number, is the total number of base stations, , each mobile user equipment is equipped with an antenna, and all mobile user equipment within the coverage of the base station share the uplink subchannel, where the set of all subchannels in the system model is , is the subchannel number, is the total number of subchannels, which are modulated using orthogonal frequency division multiplexing (OFDM). The bandwidth of each subchannel is ;

[0072] In the uplink network system, each mobile user equipment moves randomly within the coverage area of ​​the base station, and the location of the mobile user equipment is updated in each time slot. No. The moving speed of a mobile user device is recorded as , is the minimum moving speed, is the maximum value of the moving speed, time slot No. The direction of a mobile user equipment is ; Each mobile user device selects a matching base station for communication according to environmental conditions, Mobile user equipment in time slot With The association of base stations can be express, ,in , when When A mobile user equipment selects a base station ,otherwise ; for Mobile user equipment in time slot With The association of base stations;

[0073] For subchannel allocation, Mobile user equipment in time slot Selected Subchannels can be used express, ,in , if , then the subchannel By Mobile user equipment occupies, otherwise ; For the Mobile user equipment in time slot Selected sub-channels;

[0074] The mobile user equipment sends a signal to the base station in each time slot, and the upper limit of the transmission power is , the lower limit is , the transmission power can be adjusted in real time according to the sub-channel status information and user needs and other environmental conditions. No. The mobile user equipment is The transmit power on the subchannel is recorded as ; In the time slot When occupying The subchannel A mobile user device sends The signal-to-noise ratio (SINR) of the signal transmitted by a base station is:

[0075] ;

[0076] in, For time slot Occupancy The subchannel A mobile user device sends The signal-to-noise ratio of the signal transmitted by each base station; It is a time slot No. The subchannels are from Mobile user equipment to The channel gain of each base station; Different from No. Mobile user equipment; For the Mobile user equipment in time slot With The association of base stations; For the Mobile user equipment in time slot Selected sub-channels; For the time slot No. The mobile user equipment is The transmit power on each subchannel; For time slot No. The subchannels are from Mobile user equipment to The channel gain of each base station; represents additive white Gaussian noise;

[0077] The channel gain is determined by small-scale fading, path loss, and shadowing. At that time, On the subchannel The connection between the base stations Mobile User Equipment Maximum uplink transmission rate It is expressed as:

[0078] ;

[0079] in, is the bandwidth of each subchannel in the system model;

[0080] Since it is assumed that each mobile user equipment can only connect to one base station and occupy one subchannel in the same time slot, Mobile User Equipment In time slot Rate for:

[0081] ;

[0082] Time slot Include Total transmission rate of mobile user equipment for:

[0083] ;

[0084] Time slot Spectral efficiency for:

[0085] ;

[0086] Time slot Energy efficiency for:

[0087] .

[0088] Step 2: Construct a user association and resource allocation model in a dynamic heterogeneous cellular network, and describe the user association and resource allocation model as a mixed integer nonlinear programming optimization problem with constraints. The specific contents are as follows:

[0089] In each time slot, each mobile user device in the system can perform three tasks, including mobile user device association, subchannel allocation and uplink transmission power control, which respectively realize the selection of mobile user device associated base station, the selection of mobile user device occupied subchannel and the adjustment of uplink transmission power of each mobile user device. These tasks need to maximize the weighted sum of spectrum efficiency SE and energy efficiency EE of the system model by dynamically adjusting user association and resource allocation on the basis of meeting the QoS requirements of mobile user devices and system resource constraints, so as to improve the utility of heterogeneous cellular networks. Mobile User Equipment , its SINR should meet the service quality requirement threshold , , the corresponding system model optimization objective function is expressed as:

[0090] ;

[0091] in, For time slot The objective function of the system model is optimized; the objective function is a multi-objective optimization problem. Considering the different importance of each objective, and is set as the weight coefficient to indicate the relative importance of SE and EE, where ;

[0092] Under the constraints of the number of optional base stations, spectrum resources, mobile user equipment transmit power threshold and service quality requirement threshold, a user association and resource allocation model is constructed, and user association, channel allocation and power control decisions are used as model optimization variables to maximize the long-term average utility return of the model; the user association and resource allocation model is expressed as the following mixed integer nonlinear programming optimization problem with constraints:

[0093] ;

[0094] in, To maximize the long-term average utility return of the model; , , are decision variables in the objective function, where , is a discrete variable, is a continuous variable; is the total number of time slots; is the set of total time slots; constraints Indicates Mobile User Equipment How to select the binary variables of base stations and subchannels, the value range is 0 or 1; constraints Indicates that a mobile user device can only be associated with one base station in each time slot; constraint Indicates that a mobile user device can only be associated with one subchannel in each time slot; constraint Indicates that the total resources selected for all mobile user equipment cannot exceed the available base stations; constraint Indicates that the total resources selected for all mobile user devices cannot exceed the upper limit of available channels; constraint Represents the transmit power range of each mobile user equipment; constraint Indicates the service quality requirement threshold for the uplink rate of the mobile user equipment;

[0095] Through the construction and dynamic optimization of the above model, efficient user association and resource allocation can be achieved in heterogeneous cellular networks, maximizing system utility and improving network performance.

[0096] Step 3: Construct a multi-agent proximal strategy optimization algorithm in an independent reward mixed action space based on deep reinforcement learning; the specific contents are as follows:

[0097] In an embodiment, an independent reward mixed action space multi-agent proximal policy optimization algorithm MAPPO based on deep reinforcement learning can be used to solve the optimal user association and sub-channel and power resource allocation strategy according to the mobility of mobile user devices and the dynamic changes of heterogeneous cellular networks.

[0098] First, according to the above system optimization objectives, the target problem is modeled as a Markov decision process, and the corresponding global state space, local observation space, action space and reward space of the system are established. Each mobile user device in the heterogeneous cellular network is regarded as an independent intelligent agent. The number of intelligent agents is equal to the number of mobile user devices. Each mobile user device has independent observations and actions. Each mobile user device reports its local state and observation to the central controller in real time. The central controller collects and maintains the global state information as the global state space of the system.

[0099] In time slot The global state space can be expressed as ,in, Indicates that each mobile user equipment is in the time slot Whether the service quality requirements are met, , when the user meets the minimum rate requirement ,otherwise ; represents the number of interfered mobile user equipment in the occupied channel, Indicates that each mobile user equipment is in the time slot The size of the interference, Indicates that the mobile user equipment and the base station are in the time slot Communication distance;

[0100] The local observation space of each mobile user device is expressed as ,in, Indicates Mobile user equipment in time slot Whether the service quality requirements are met, when the mobile user equipment meets the minimum rate requirements ,otherwise , Indicates Mobile user equipment in time slot The channel gain of the occupied subchannel, Indicates Mobile user equipment in time slot The disturbance is small or large, Indicates Mobile user equipment in time slot The distance to the communicating base station;

[0101] The action space represents the set of actions that each mobile user device can take in the current given state. The action space is defined as ,Due to the limitations of resources and power, the action space is a bounded space, where, and Respectively represent Mobile user equipment in time slot The selected base station index and subchannel index have discrete values. For the Mobile user equipment in time slot The transmission power, The value is a continuous value within the range of

[0102] A reward function is designed in the reward space to encourage users to effectively increase the rate and reduce energy consumption while meeting QoS requirements, thereby improving the system SE and EE goals. The reward function is designed as:

[0103] ;

[0104] in, For the Mobile user equipment in time slot The reward function value of is the weight coefficient used to control the importance of subtasks; For time slot No. The transmit power of each mobile user equipment; Indicates The user rate threshold for a mobile user device under the service quality requirement is defined as:

[0105] ;

[0106] Average reward of the system for:

[0107] ;

[0108] The MAPPO algorithm is used to train the user association and resource allocation model. Each agent takes maximizing the weighted utility of system SE and EE as the optimization goal to achieve the learning of user association and resource allocation strategies.

[0109] During the training process, each agent has its own actor network, and all agents in the system share a MAPPO algorithm architecture of the critic network. An independent reward mechanism is designed for each agent. Combined with the actions taken based on the agent's observation space, the trajectory is independently sampled and input into its corresponding actor network, and the network parameters are updated according to the trajectory. A branch network layer matching the number of action types is added to each actor network to handle mixed action spaces. The branch network layer shares several previous neural networks. The discrete action layer outputs the probabilities of different actions, the continuous action layer outputs the mean and standard deviation of the action, and the discrete action layer outputs the joint classification distribution, specifically:

[0110] ;

[0111] in, is the strategy for the discrete action part, which means that in a given local observation space Under the condition of The distribution of discrete actions in represents the discrete action part, For the The set of parameters of the actor network, The actor network corresponds to Agent, also corresponding to Mobile user equipment; is the sequence number of all possible actions in the class, Represents the number of all possible actions in the class; For Action probability; Is an indicator function, indicating whether a specific action is selected ; The output of the continuous action layer that conforms to the Gaussian distribution is expressed as:

[0112] ;

[0113] in, is the strategy for the continuous action part, which means that in a given local observation space Under the condition of The distribution of continuous actions in represents normal distribution; is the mean value of the action; is the standard deviation of the action;

[0114] The loss clipping strategy in the proximal policy optimization method is used to calculate the loss function to promote the training of the actor network, and exploration is encouraged by maximizing entropy. An action masking method is designed to mask out unavailable operations based on the observation information of the agent, and their probability is always set to zero. Each agent shares a global critic network to evaluate the action decision of each agent. The global critic network receives the global state and the actions of all agents in the current time slot as input, evaluates the value of the joint actions of all agents in the system, and guides the training process.

[0115] Step 4: Use the centralized training and distributed execution architecture to optimize the scheduling of mobile user device association and resource allocation decisions. The specific process is as follows: In the training phase, the global state information of all agents is collected centrally to the central controller. Each agent uses its own observed environmental information and corresponding independent rewards to make action decisions through its own actor network algorithm. The value of the joint action of the agents in the system is evaluated through the global critic network to achieve centralized training network parameter updates; In the distributed execution phase, each agent makes independent decisions based on the centrally trained actor network, without the need for a global critic network. The actor network only relies on the observation of each agent and maps the observation results to the probability distribution of the action, or to the mean and standard deviation, to achieve user association and resource allocation.

[0116] The heterogeneous cellular network user association and resource allocation method proposed in the embodiment of the present invention is implemented based on the hybrid action space MAPPO algorithm with independent rewards. The method adopts a centralized training and distributed execution architecture. The specific process is as follows:

[0117] Step 4.1, initialize the network model parameters in the MAPPO algorithm, the interaction trajectory of the mobile user device and the environment, etc.;

[0118] Step 4.2, the mobile user equipment collects its own observation information, and the central controller collects the mobile user equipment service requirements, location, channel status and other information as the global environment status;

[0119] Step 4.3: input the global state space and the local observation space into the MAPPO algorithm, and the MAPPO algorithm outputs a configuration strategy for user association and resource allocation according to the state of the mobile user device;

[0120] Step 4.4: Calculate the system optimization target and reward, update the model state and mobile user device observation information, and obtain the next state;

[0121] Step 4.5, save the action space, global state space, reward space and next state space information in the environment, and update the network model parameters of the MAPPO algorithm;

[0122] Step 4.6: Conduct centralized iterative training to optimize the MAPPO algorithm model;

[0123] Step 4.7: Each mobile user device executes its own tasks in a distributed manner according to its own model parameters, and finally realizes the best scheduling strategy for user association and resource allocation.

[0124] According to the steps of the embodiment, the experiment was carried out, and the parameters were adjusted and set according to the needs. Figure 3-Figure 5 The experimental results. Figure 3-Figure 5 They are respectively the spectrum efficiency, energy efficiency, and system model optimization target training result diagrams obtained by optimizing the hybrid action space MAPPO method based on independent rewards in the embodiment of the present invention, where the gray line part is the unprocessed original data and the black dotted line is the smoothed data. Figure 3-Figure 5 It can be seen that after training with the implementation method of the present invention, as the number of rounds increases, the spectrum efficiency, energy efficiency and overall utility of the system model optimization target are greatly improved and stabilized at the convergence value. Therefore, the embodiment of the present invention can effectively improve the system SE and EE as well as the system optimization target while ensuring that the user QoS meets the requirements, and realizes efficient and energy-saving green communication.

[0125] The specific process pseudo code of the MAPPO algorithm of the present invention is shown in Table 1:

[0126] Table 1 MAPPO algorithm pseudo code table

[0127] .

[0128] in, and are the learning rates of the actor network and the critic network, respectively. is the discount factor, is the generalized advantage estimation process hyperparameter, is the cutoff coefficient, is the entropy coefficient hyperparameter; For the Agents in The action space at the moment, For the Agents in Rewards at all times, For the An agent at time The next moment of observation, For at the moment The state of the next moment; is the state value function, which is used to evaluate the long-term effect of executing the current strategy under a specific state. is the loss function for critic network training, is the loss function for actor network training, are the parameters of the critic network, For the The parameters of an actor network; each agent corresponds to an actor network; episode is the number of independent rounds to be executed by the algorithm, and each round represents a complete process; step is the number of time steps, and there are multiple time steps in each round; For the The initial observation value of an intelligent agent; UE is a mobile user equipment; is the total number of mobile user devices.

[0129] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for user association and resource allocation in a cellular network based on deep reinforcement learning, characterized in that: The steps include: Step 1: In a dynamic heterogeneous cellular network, an uplink network system including a macro base station, a pico base station, a femto base station, a central controller, and multiple mobile user devices is constructed; Step 2: construct a user association and resource allocation model in a dynamic heterogeneous cellular network, and describe the user association and resource allocation model as a mixed integer nonlinear programming optimization problem with constraints; Step 3: Construct a multi-agent proximal strategy optimization algorithm in an independent reward mixed action space based on deep reinforcement learning; Step 4: Adopt a centralized training and distributed execution architecture to optimize scheduling user association and resource allocation decisions based on algorithms; In step 1, the uplink network system consists of M mbs Macro base stations, M pbs pico base stations, M fbs A femtocell and a group of randomly distributed mobile user devices Composition, u n is the nth mobile user equipment, N is the total number of mobile user equipment; the pico base station, femto base station and mobile user equipment are all located within the coverage of the macro base station, let all base stations in the system be Where m is the base station number, M is the total number of all base stations, M = M mbs +M pbs +M fbs , each mobile user equipment is equipped with an antenna, and all mobile user equipment within the coverage of the base station share the uplink subchannel; the set of all subchannels in the system is k is the subchannel number, K is the total number of subchannels; Each mobile user equipment in the uplink network system moves randomly within the coverage area of ​​the base station, and the position of the mobile user equipment is updated in each time slot; the moving speed of the nth mobile user equipment in time slot t is recorded as v n (t)∈[v min , v max ],v min is the minimum value of the moving speed, v max is the maximum value of the moving speed, and the direction of the nth mobile user equipment in time slot t is θ n (t)∈[0, 2π]; each mobile user device selects a matching base station for communication according to the environmental conditions. The association between the nth mobile user device and the mth base station in time slot t is express, in 1≤m≤M, when When , it means that the nth mobile user equipment has selected base station m, otherwise is the association of n mobile user equipments with the Mth base station in time slot t; For subchannel allocation, the kth subchannel selected by the nth mobile user equipment in time slot t is used express, in 1≤k≤K, if Then subchannel k is occupied by the nth mobile user equipment, otherwise The Kth subchannel selected for the nth mobile user equipment in time slot t; In the uplink network system, the mobile user equipment sends a signal to the base station in each time slot, and the upper limit of the transmission power is p max , the lower limit is p min , the transmission power is adjusted in real time according to the environmental status of the subchannel, which includes status information and user needs; the transmission power of the nth mobile user equipment on the kth subchannel in time slot t is recorded as At time slot t, the signal-to-noise ratio of the signal transmitted by the nth mobile user equipment occupying the kth subchannel to the mth base station is: in, is the signal-to-noise ratio of the signal transmitted by the nth mobile user equipment occupying the kth subchannel in time slot t to the mth base station; is the channel gain of the kth subchannel from the nth mobile user equipment to the mth base station in time slot t; u i Different from u n The i-th mobile user equipment; is the association between the i-th mobile user equipment and the m-th base station in time slot t; The kth subchannel selected for the i-th mobile user equipment in time slot t; is the transmission power of the i-th mobile user equipment on the k-th subchannel in time slot t; is the channel gain of the k-th subchannel from the i-th mobile user equipment to the m-th base station in time slot t; represents additive white Gaussian noise; The channel gain is determined by small-scale fading, path loss and shadowing. At time slot t, the channel gain between the mth base station and the nth mobile user equipment u on the kth subchannel is n Maximum uplink transmission rate It is expressed as: Where W is the bandwidth of each subchannel in the system; Assuming that each mobile user equipment can only connect to one base station and occupy one subchannel in the same time slot, the nth mobile user equipment u n The rate r in time slot t n (t) is: The total transmission rate r of N mobile user equipment in time slot t total (t) is: Spectral efficiency η of time slot t SE (t) is: Energy efficiency η of time slot t EE (t) is: The specific process of step 2 is: The construction system optimization objective function is: F(t)=a·h SE (t)+β·η EE (t); Wherein, F(t) is the system optimization objective function of time slot t; α and β are weight coefficients, which are used to represent the relative importance of spectrum efficiency and energy efficiency, respectively, α+β=1; Under the constraints of the number of optional base stations, spectrum resources, mobile user equipment transmit power threshold and service quality requirement threshold, a user association and resource allocation model is constructed, and user association, channel allocation and power control decisions are used as model optimization variables to maximize the long-term average utility return of the model; the user association and resource allocation model is expressed as the following mixed integer nonlinear programming optimization problem with constraints: Among them, P1 is the long-term average utility return problem of the maximization model; T is the total number of time slots; is the set of total time slots; ρ n is the service quality requirement threshold; constraint C1 represents the nth mobile user equipment u n How to select a binary variable for a base station and a subchannel, with a value range of 0 or 1; Constraint C2 indicates that a mobile user device can only be associated with one base station in each time slot; Constraint C3 indicates that a mobile user device can only be associated with one subchannel in each time slot; Constraint C4 indicates that the total resources selected for all mobile user devices cannot exceed the available base stations; Constraint C5 indicates that the total resources selected for all mobile user devices cannot exceed the upper limit of the available channels; Constraint C6 indicates the transmission power range of each mobile user device; Constraint C7 indicates the service quality requirement threshold for the uplink rate of the mobile user device; In step 3, according to the mobility of mobile user devices and the dynamic changes of heterogeneous cellular networks, the optimal user association and sub-channel and power resource allocation strategy is solved by an independent reward hybrid action space multi-agent proximal strategy optimization algorithm based on deep reinforcement learning; Establish the global state space, local observation space, action space and reward space corresponding to the system. Each mobile user device in the heterogeneous cellular network acts as an independent intelligent agent. The number of intelligent agents is equal to the number of mobile user devices. Each mobile user device has independent observations and actions. Each mobile user device reports its local state and observation to the central controller in real time. The central controller collects and maintains the global state information as the global state space of the system. The global state space at time slot t is represented as s t = {q t , c t , z t , d t }, where q t Indicates whether each mobile user equipment meets the service quality requirements in time slot t, q t ∈{0, 1}, when the user meets the minimum rate requirement q t =1, otherwise q t =O; C t represents the number of interfered mobile user equipment in the occupied channel, z t represents the interference level of each mobile user equipment in time slot t, d t represents the communication distance between the mobile user equipment and the base station in time slot t; The local observation space of each mobile user device is expressed as in, Indicates whether the nth mobile user equipment meets the service quality requirements in time slot t. When the mobile user equipment meets the minimum rate requirement otherwise represents the channel gain of the subchannel occupied by the nth mobile user equipment in time slot t, Indicates the interference experienced by the nth mobile user equipment in time slot t. represents the distance from the nth mobile user equipment to the communicating base station at time slot t; The action space represents the set of actions that each mobile user device can take in the current given state. The action space is defined as and. represent the base station index and subchannel index selected by the nth mobile user equipment in time slot t respectively; is the transmission power of the nth mobile user equipment in time slot t, and takes a continuous value in the range of [pmin, pmax]; The following reward function is designed in the reward space: in, is the reward function value of the nth mobile user device in time slot t; δ is the weight coefficient used to control the importance of the subtask; pn(t) is the transmission power of the nth mobile user device in time slot t; It represents the rate threshold of the nth mobile user equipment under the service quality requirement constraint, which is defined as: Average reward of the system for: The algorithm is used to train the user association and resource allocation model. Each agent takes maximizing the weighted utility of system spectrum efficiency and energy efficiency as the optimization goal to achieve the learning of user association and resource allocation strategies. During the training process, each agent has its own actor network, and all agents in the system share the algorithm architecture of a critic network. An independent reward mechanism is designed for each agent. Combined with the actions taken based on the agent's observation space, the trajectory is independently sampled and input into its corresponding actor network, and the network parameters are updated according to the trajectory. A branch network layer matching the number of action types is added to each actor network to handle mixed action spaces. The branch network layer shares several previous neural networks. The discrete action layer outputs the probabilities of different actions, the continuous action layer outputs the mean and standard deviation of the action, and the discrete action layer outputs the joint classification distribution, specifically: in, is the strategy for the discrete action part, which means that in a given local observation space Under the condition of The distribution of discrete actions in ; d represents the discrete action part, θ n is the parameter set of the nth actor network, which corresponds to the nth agent and the nth mobile user device; j is the sequence number of all possible actions in the class, and J represents the number of all possible actions in the class; P j (·) is the probability of action j; Is an indicator function, indicating whether a specific action is selected The output of the continuous action layer that conforms to the Gaussian distribution is expressed as: in, is the strategy for the continuous action part, which means that in a given local observation space Under the condition of The distribution of continuous actions in represents normal distribution; μ(·) is the mean of the action; σ 2 (·) is the standard deviation of the action; The loss clipping strategy in the proximal policy optimization method is used to calculate the loss function to promote the training of the actor network and encourage exploration by maximizing entropy. An action masking method is designed to mask unavailable actions based on the observation information of the agent and set the selection probability of unavailable actions to zero. Each agent shares a global critic network to evaluate the action decision of each agent. The global critic network receives the global state and the actions of all agents in the current time slot as input, evaluates the value of the joint actions of all agents in the system, and guides the training process.

2. The method for cellular network user association and resource allocation based on deep reinforcement learning according to claim 1, characterized in that: The specific process of step 4 is as follows: Step 4.1, initialize the network model parameters and the interaction trajectory between the mobile user device and the environment in the independent reward mixed action space multi-agent proximal strategy optimization algorithm based on deep reinforcement learning; Step 4.2, the mobile user equipment collects its own observation information, and the central controller collects the service requirements, location, and channel status of the mobile user equipment as the global environment status; Step 4.3: input the global state space and the local observation space into the algorithm, and the algorithm outputs the configuration strategy of user association and resource allocation according to the state of the mobile user device; Step 4.4: Calculate the system optimization target and reward, update the model state and mobile user device observation information, and obtain the next state; Step 4.5: Save the action space, global state space, reward space and next state space information in the environment, and update the network model parameters of the algorithm; Step 4.6: Conduct centralized iterative training to optimize the algorithm model; Step 4.7: Each mobile user device executes its own tasks in a distributed manner according to its own model parameters, and finally realizes the best scheduling strategy for user association and resource allocation.

Images