Base station energy-saving regulation and control method based on diffusion model and reinforcement learning

Abstract

The invention discloses a base station energy-saving regulation and control method based on a diffusion model and reinforcement learning, and belongs to the field of communication, and the method specifically comprises the following steps: S1, constructing a simulated base station environment with dynamic flow change; s2, designing a strategy network of reinforcement learning in combination with the characteristics of the diffusion model; s3, designing a reward function of reinforcement learning; and S4, training a diffusion model in the simulated base station environment constructed in the step S1, so that the diffusion model makes a dynamic decision according to the observation condition. According to the method, a simulation scene capable of reflecting the dynamic change of the user traffic is constructed, the model is subjected to iterative training in the scene, and in the scene, the model is subjected to a series of iterative training so as to adapt to a complex and continuously changing traffic environment and make a dynamic energy-saving control decision. The method and the system have excellent performance in the aspect of energy conservation, and maintain a certain level of user service quality.

Description

Technical Field

[0001] The present invention belongs to the field of communications, and in particular relates to a base station energy-saving control method based on a diffusion model and reinforcement learning. Background Art

[0002] By the end of May 2023, China has built a total of 2.844 million 5G base stations and more than 2.05 billion mobile IoT terminal users. By the first quarter of 2024, my country has built a total of 3.647 million 5G base stations, and the penetration rate of 5G users has exceeded 60%. In 2019, the power consumption of 5G base stations in my country accounted for about 0.05% of the total social power consumption. However, by 2023, the power consumption of 5G base stations is expected to account for 1.3% of the total social power consumption, and by 2026, the power consumption of 5G base stations will rise to 2.1% of the total social power consumption.

[0003] In response to the need for energy saving in base stations, the industry has proposed a variety of solutions to meet the challenge. Key energy-saving technologies include simplifying the design of public signals, adopting a very simplified cell architecture, implementing a discontinuous transmission / reception strategy for cells, introducing a wake-up mechanism for base stations, and optimizing the management of sleep modes. However, in real operating environments, it is not easy to adaptively implement these energy-saving measures for base stations. Reinforcement learning is considered to be a key technology to solve this problem.

[0004] However, there are still problems: the changing multimodal environmental characteristics make it difficult for the reinforcement learning policy network to accurately capture the exact characteristics of the environment, which in turn affects decision-making.

[0005] In recent years, diffusion models have been very successful in natural language processing and computer vision, and can capture multimodal distributions well. Combining diffusion models with reinforcement learning has also attracted much attention in the current research field. However, value estimation in online reinforcement learning has high noise, which also fluctuates with the adjustment of strategies, which adds additional difficulty to training diffusion models.

[0006] In addition, since base station control usually involves discrete actions, current reinforcement learning methods based on diffusion models are not suitable. Summary of the invention

[0007] In order to solve the above-mentioned problems, the present invention provides a base station energy-saving control method based on diffusion model and reinforcement learning, which solves the problem that the reinforcement learning strategy network mentioned in the background technology is difficult to capture the characteristics of the multimodal environment, online reinforcement learning faces high noise values, and the noise will increase with the change of strategy, making the training of the diffusion model more complicated. Base station control mostly involves discrete actions, which makes the current diffusion model reinforcement learning method unapplicable.

[0008] To achieve the above object, the present invention provides a base station energy-saving control method based on diffusion model and reinforcement learning, which specifically includes the following steps:

[0009] Step S1: construct a simulated base station environment with dynamically changing traffic;

[0010] Step S2: Design a reinforcement learning strategy network based on the characteristics of the diffusion model;

[0011] Step S3: Design a reward function for reinforcement learning;

[0012] Step S4: training the diffusion model in the simulated base station environment constructed in step S1, so that the diffusion model can make dynamic decisions based on the observed conditions.

[0013] Preferably, in step S1, the dynamic change of traffic refers to modeling the traffic as a Poisson process. During the modeling process, the user equipment UE requests a single network flow, and the traffic is modeled as having an average arrival rate of λ t Poisson process with an average arrival rate λ t is the probability of generating a new UE in the area at time step dt, λ t Following the equation:

[0014]

[0015] Where S t is the regional traffic time density, dt is the fixed time slot, x max is the size of user demand, P(X=k) is the Poisson distribution probability, k is the number of users arriving, e is a natural constant, k! represents factorial operation, X represents probability variable, λ t k Represents the exponentiation operation.

[0016] Preferably, in step S2, the specific steps include:

[0017] Step S21: The reverse denoising process of the diffusion model is used as the process of the policy network outputting actions in reinforcement learning, and the degree of depression is applied to each state. The policy network restores the output action distribution of the policy by passing the Gaussian distribution through the denoising process of the diffusion model;

[0018]

[0019] For state s, the policy network restores the output action distribution of the policy by applying a Gaussian distribution to the denoising process of the diffusion model, where a i represents the denoising action of the i-th step in the denoising process, Represents a normal distribution.

[0020] Step S22: Model the forward noise addition process of the diffusion model as the policy network training process in reinforcement learning, add noise to the original action into a Gaussian distribution, then sample the action distribution through the above forward process, and perform gradient descent training according to the loss function;

[0021]

[0022] The original action is noised into a Gaussian distribution, where q represents the data distribution, x0 is the original data that follows the q distribution, and x t represents the data after t-step noise addition, β t Represents the predefined noise variance estimate of the t-th step; then the action distribution is sampled through the above denoising process, and finally gradient descent training is performed according to the loss function.

[0023] Step S23: Based on the discrete characteristics of the base station control action, the loss function of the strategy network is redesigned, and the loss of the diffusion model part is recorded as L d (θ), defined as:

[0024]

[0025] where θ represents the parameters optimized by the neural network, ∈ represents the noise estimate in the diffusion model, represents a uniform distribution on a discrete set, i represents an element in this set, represents the standard normal distribution, s represents the state, and a represents the action; Represents the expectation of the elements in the set that meet the conditions, ∈ θ represents the noise prediction value, Represents the weight coefficient of the prediction noise in the i-th step.

[0026] Design the strategy loss regularization term to prevent the model from falling into a unimodal strategy, denoted as L H (θ):

[0027]

[0028] Among them, H represents the entropy operation, π θ Q stands for strategy θ represents the Q value estimation, α represents the temperature coefficient, represents the expectation of the elements in the set that meet the conditions, π t represents the policy network output at time t, s t Represents the state at time t.

[0029] The training objective of the policy network for reinforcement learning based on the diffusion model is:

[0030]

[0031] Where L d (θ) is the noise estimation loss function for training the diffusion model, L H (θ) is the policy loss regularization term defined above;

[0032] Preferably, in step S3, a reward function for reinforcement learning is designed, which specifically includes the following steps:

[0033] S31: define a user equipment UE u k QOS Rewards qos，k :

[0034]

[0035] In the formula, ρ k Represents the user equipment UE u k The ratio of the actual rate to the required rate;

[0036] S32: Define the reward function r for power consumption pc :

[0037] r pc =-pc;

[0038] Where, pc is the total energy consumption, kw;

[0039] S33: During the action interval, the reward R is defined as:

[0040]

[0041] Where α and β are the weight coefficients of QoS reward and power consumption reward.

[0042] Preferably, in step S4, the model is trained in a simulated base station environment, specifically comprising the following steps:

[0043] S41: Initialize policy network π θ , critic value function network and the target network π θ' ,

[0044] S42: Start interacting with the environment, randomly selecting actions based on an adaptive ε-greedy strategy or outputting actions through a policy network;

[0045] S43: Collect data on interaction with the environment and store it in the replay buffer;

[0046] S44: If the total number of steps is greater than the number of steps to start training, start the training process, collect data from the replay buffer, and update it using the following formula and

[0047]

[0048] In the formula, r represents the reward function, γ represents the discount factor, represents the value function network, a t Represents action output, represents the value function target network,

[0049] Update the policy network π by the following formula θ :

[0050]

[0051] Update the temperature coefficient α by minimizing the formula:

[0052]

[0053] The target network is updated by the following formula:

[0054] θ'=ρθ'+(1-ρ)θ,φ' i' =ρφ' i' +(1-ρ)φ i ,i={1,2};

[0055] In the formula, θ' represents the parameters of the policy target network, φ' i' represents the parameters of the target network of the value function,

[0056] If the total number of steps is not greater than the number of steps at the start of training, the training process ends;

[0057] S47: End the training process and return to step S42 to continue interacting with the environment until the iteration round condition is met.

[0058] Therefore, the present invention adopts the above-mentioned base station energy-saving control method based on diffusion model and reinforcement learning, which has the following beneficial effects:

[0059] (1) The present invention can accurately extract environmental features in complex multimodal environments and thus make better decisions.

[0060] (2) The present invention can significantly reduce the energy consumption of the base station while ensuring the quality of communication services by precisely controlling the energy consumption of the base station.

[0061] (3) The present invention improves the efficiency of resource utilization and reduces energy loss caused by resource waste by dynamically adjusting the power allocation of the base station.

[0062] (4) The present invention utilizes a reinforcement learning algorithm to adaptively adjust the operating status of the base station and realize intelligent response to different environments and user needs.

[0063] (5) The present invention is applicable to different types of base stations and communication networks, has good flexibility and scalability, and is easy to use in future technology upgrades and network expansions.

[0064] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart of the energy-saving control method of a base station based on a reinforcement learning diffusion model of the present invention;

[0066] Figure 2 This is a schematic diagram of the structure of the model system in an embodiment of the present invention;

[0067] Figure 3 It is a performance comparison curve diagram in the comparative example of the present invention. DETAILED DESCRIPTION

[0068] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0069] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0070] The words "include" or "comprises" and the like used in the present invention mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of also including other elements. The orientation or position relationship indicated by the terms "inside", "outside", "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. When the absolute position of the described object changes, the relative position relationship may also change accordingly. In the present invention, unless otherwise clearly specified and limited, the terms "attachment" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral body; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0071] like Figure 1-Figure 2 As shown, a base station energy-saving control method based on diffusion model and reinforcement learning specifically includes the following steps:

[0072] Step S1: construct a simulated base station environment with dynamically changing traffic;

[0073] In step S4, the model is trained in a simulated base station environment, specifically including the following steps:

[0074] S41: Initialize policy network π θ , critic value function network and the target network π θ' ,

[0075] S42: Start interacting with the environment, randomly selecting actions based on an adaptive ε-greedy strategy or outputting actions through a policy network;

[0076] S43: Collect data on interaction with the environment and store it in the replay buffer;

[0077] S44: If the total number of steps is greater than the number of steps to start training, start the training process, collect data from the replay buffer, and update it using the following formula and

[0078]

[0079] In the formula, r represents the reward function, γ represents the discount factor, represents the value function network, a t Represents action output, represents the value function target network,

[0080] Update the policy network π by the following formula θ :

[0081]

[0082] Update the temperature coefficient α by minimizing the formula:

[0083]

[0084] The target network is updated by the following formula:

[0085] θ'=ρθ'+(1-ρ)θ,φ' i' =ρφ' i' +(1-ρ)φ i ,i={1,2};

[0086] In the formula, θ' represents the parameters of the policy target network, φ' i' Represents the parameters of the target network of the value function;

[0087] If the total number of steps is not greater than the number of steps at the start of training, the training process ends;

[0088] S47: End the training process and return to step S42 to continue interacting with the environment until the iteration round condition is met. In step S1, the dynamic change of traffic refers to modeling the traffic as a Poisson process. During the modeling process, the user equipment UE requests a single network flow and models the traffic as having an average arrival rate of λ t Poisson process with an average arrival rate λ t is the probability of generating a new UE in the area at time step dt, λ t Following the equation:

[0089]

[0090] Where S t is the regional traffic time density, dt is the fixed time slot, x max is the size of user demand, P(X=k) is the Poisson distribution probability, k is the number of users arriving, e is a natural constant, k! represents factorial operation; X represents probability variable; λ t k represents the power operation;

[0091] Step S2: Design a reinforcement learning strategy network based on the characteristics of the diffusion model; in step S2, the specific steps include:

[0092] Step S21: The reverse denoising process of the diffusion model is used as the process of the policy network outputting actions in reinforcement learning, and the degree of depression is applied to each state. The policy network restores the output action distribution of the policy by passing the Gaussian distribution through the denoising process of the diffusion model;

[0093]

[0094] For state s, the policy network restores the output action distribution of the policy by applying a Gaussian distribution to the denoising process of the diffusion model, where a i represents the denoising action of the i-th step in the denoising process, Represents a normal distribution.

[0095] Step S22: Model the forward noise addition process of the diffusion model as the policy network training process in reinforcement learning, add noise to the original action into a Gaussian distribution, then sample the action distribution through the above forward process, and perform gradient descent training according to the loss function;

[0096]

[0097] The original action is noised into a Gaussian distribution, where q represents the data distribution, x0 is the original data that follows the q distribution, and x t represents the data after t-step noise addition, βt Represents the predefined noise variance estimate of the t-th step; then the action distribution is sampled through the above denoising process, and finally gradient descent training is performed according to the loss function.

[0098] Step S23: Based on the discrete characteristics of the base station control action, the loss function of the strategy network is redesigned, and the loss of the diffusion model part is recorded as L d (θ), defined as:

[0099]

[0100] where θ represents the parameters optimized by the neural network, ∈ represents the noise estimate in the diffusion model, represents a uniform distribution on a discrete set, i represents an element in this set, represents the standard normal distribution, s represents the state, and a represents the action; Represents the expectation of the elements in the set that meet the conditions, ∈ θ represents the noise prediction value, Represents the weight coefficient of prediction noise.

[0101] Design the strategy loss regularization term to prevent the model from falling into a unimodal strategy, denoted as L H (θ):

[0102]

[0103] Among them, H represents the entropy operation, π θ Represents strategy, Q θ represents the Q value estimation, α represents the temperature coefficient, represents the expectation of the elements in the set that meet the conditions, π t represents the output of the policy network at time t, s t Represents the state at time t.

[0104] The training objective of the policy network for reinforcement learning based on the diffusion model is:

[0105]

[0106] Where L d (θ) is the noise estimation loss function for training the diffusion model, L H (θ) is the policy loss regularization term defined above;

[0107] In addition, the Q value function of the critic network is obtained by the traditional method. This method requires minimizing the Bellman equation and combining the double Q learning technique to enhance the learning process by initializing two Q value networks. And the corresponding target network During the training process, by calculating and The minimum value in is taken as the final target value. This method uses two different Q value networks and their target networks to reduce the estimation bias and improve the stability and performance of the learning algorithm.

[0108] Step S3: Design a reward function for reinforcement learning;

[0109] In step S3, a reward function for reinforcement learning is designed, which specifically includes the following steps:

[0110] S31: define a user equipment UE u k QoS reward r qos,k :

[0111]

[0112] In the formula, ρ k Represents the user equipment UE u k The ratio of the actual rate to the required rate;

[0113] S32: Define the reward function r for power consumption pc :

[0114] r pc =-pc;

[0115] Where, pc is the total energy consumption, kw;

[0116] S33: During the action interval, the reward R is defined as:

[0117]

[0118] Where α and β are the weight coefficients of QoS reward and power consumption reward.

[0119] Step S4: training the diffusion model in the simulated base station environment constructed in step S1, so that the diffusion model can make dynamic decisions based on the observed conditions.

[0120] In step S4, the model is trained in a simulated base station environment, specifically including the following steps:

[0121] S41: Initialize the policy network π θ , critic value function network and the target network π θ' ,

[0122] S42: Start interacting with the environment, randomly selecting actions based on an adaptive ε-greedy strategy or outputting actions through a policy network;

[0123] S43: Collect data on interaction with the environment and store it in the replay buffer;

[0124] S44: If the total number of steps is greater than the number of steps at the start of training, start the training process, collect data from the replay buffer, and update it using the following formula and

[0125]

[0126] In the formula, r represents the reward function, γ represents the discount factor, represents the value function network, a t Represents action output, represents the value function target network,

[0127] Update the policy network π by the following formula θ :

[0128]

[0129] Update the temperature coefficient α by minimizing the formula:

[0130]

[0131] The target network is updated by the following formula:

[0132] θ′=ρθ′+(1-ρ)θ,φ′ i′ =ρφ′ i′ +(1-ρ)φ i , i = {1, 2};

[0133] In the formula, θ′ represents the parameters of the policy target network, φ′ i′ represents the parameters of the target network of the value function,

[0134] If the total number of steps is not greater than the number of steps at the start of training, the training process ends;

[0135] S47: End the training process and return to step S42 to continue interacting with the environment until the iteration round condition is met. In step S1, the dynamic change of traffic refers to modeling the traffic as a Poisson process. During the modeling process, the user equipment UE requests a single network flow and models the traffic as having an average arrival rate of λ t Poisson process with an average arrival rate λ t is the probability of generating a new UE in the area at time step dt, λ t Following the equation:

[0136]

[0137] Where St is the regional traffic time density, dt is the fixed time slot, x max is the size of user demand, P(X=k) is the Poisson distribution probability, k is the number of users arriving, e is a natural constant, k! represents factorial operation, X represents probability variable, λ t k Represents the exponentiation operation.

[0138] Example

[0139] This embodiment provides a base station energy-saving control method based on a diffusion model and reinforcement learning. With reinforcement learning as the core, the optimal strategy is learned through the interaction between the intelligent agent and the environment, thereby achieving long-term goals. The specific process is as follows:

[0140] (1) Construct a simulated base station scenario with dynamic traffic changes with Markov properties.

[0141] (2) The reverse denoising process of the diffusion model is used as the process of outputting actions of the policy network in reinforcement learning;

[0142] (3) Model the forward noise addition process of the diffusion model as the policy network training process in reinforcement learning;

[0143] (4) Combine energy consumption and QoS indicator data to jointly construct a reinforcement learning reward function;

[0144] (5) Train the algorithm model in a simulated base station and compare the energy consumption with models trained by other methods.

[0145] The method is applicable to a base station environment with Markov properties. Through a base station energy-saving control method based on a diffusion model and reinforcement learning, energy consumption can be reduced while maintaining a certain user service quality.

[0146] Comparative Example

[0147] like Figure 3 As shown in the figure, compared with the baseline algorithm and other reinforcement learning algorithms, this scheme adopts a reinforcement learning algorithm based on the diffusion model. Under the scenario of complex dynamic traffic changes and a certain QoS quality, it solves the problem of inaccurate value function estimation in the fusion of the diffusion model and online reinforcement learning, and the limitation of the diffusion model for discrete actions in reinforcement learning.

[0148] Therefore, the present invention adopts the above-mentioned base station energy-saving control method based on diffusion model and reinforcement learning to promote the transformation of base stations to green energy, realize intelligent management of base station energy consumption, optimize energy efficiency, and reduce operating costs.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A base station energy-saving control method based on diffusion model and reinforcement learning, characterized in that: The specific steps include: Step S1: construct a simulated base station environment with dynamically changing traffic; Step S2: Design a reinforcement learning strategy network based on the characteristics of the diffusion model; Step S3: Design a reward function for reinforcement learning; Step S4: training the diffusion model in the simulated base station environment constructed in step S1, so that the diffusion model can make dynamic decisions based on the observed conditions.

2. A base station energy-saving control method based on diffusion model and reinforcement learning according to claim 1, characterized in that: In step S1, the dynamic change of traffic refers to modeling the traffic as a Poisson process. During the modeling process, the user equipment UE requests a single network flow and the traffic is modeled as having an average arrival rate of λ t Poisson process with an average arrival rate λ t is the probability of generating a new UE in the area at time step dt, λ t Following the equation: Where S t is the regional traffic time density, dt is the fixed time slot, x max is the size of user demand, P(X=k) is the Poisson distribution probability, k is the number of users arriving, e is a natural constant, k! is a factorial operation, X represents the probability variable, λ t k Tabular exponentiation.

3. According to the base station energy-saving control method based on diffusion model and reinforcement learning as described in claim 1, it is characterized by: In step S2, the specific steps include: Step S21: The reverse denoising process of the diffusion model is used as the process of the policy network outputting actions in reinforcement learning, and the degree of depression is applied to each state. The policy network restores the output action distribution of the policy by passing the Gaussian distribution through the denoising process of the diffusion model; Step S22: Model the forward noise addition process of the diffusion model as the policy network training process in reinforcement learning, add noise to the original action into a Gaussian distribution, then sample the action distribution through the above forward process, and perform gradient descent training according to the loss function; Step S23: Based on the discrete characteristics of the base station control action, the loss function of the strategy network is redesigned, and the loss of the diffusion model part is recorded as L d (θ), defined as: where θ represents the parameters optimized by the neural network, ∈ represents the noise estimate in the diffusion model, represents a uniform distribution on a discrete set, i represents an element in this set, represents the standard normal distribution, s represents the state, and a represents the action; Represents the expectation of the elements in the set that meet the conditions, ∈ θ represents the noise prediction value, Represents the weight coefficient of the prediction noise in the i-th step. Design the strategy loss regularization term to prevent the model from falling into a unimodal strategy, denoted as L H (θ): Among them, H represents the entropy operation, π θ Q stands for strategy θ represents the Q value estimation, α represents the temperature coefficient, represents the expectation of the elements in the set that meet the conditions, π t represents the policy network output at time t, s t Represents the state at time t. The training objective of the policy network for reinforcement learning based on the diffusion model is: Where L d (θ) is the noise estimation loss function for training the diffusion model, L H (θ) is the policy loss regularization term defined above.

4. A base station energy-saving control method based on diffusion model and reinforcement learning according to claim 1, characterized in that: In step S3, a reward function for reinforcement learning is designed, which specifically includes the following steps: S31: define a user equipment UE u k QoS reward r qos,k : In the formula, ρ k Represents the user equipment UE u k The ratio of the actual rate to the required rate; S32: Define the reward function r for power consumption pc : r pc =-pc; Where, pc is the total energy consumption, kw; S33: During the action interval, the reward R is defined as: Where α and β are the weight coefficients of QoS reward and power consumption reward.

5. According to the base station energy-saving control method based on diffusion model and reinforcement learning as described in claim 1, it is characterized by: In step S4, the model is trained in a simulated base station environment, specifically including the following steps: S41: Initialize policy network π θ , critic value function network and the target network S42: Start interacting with the environment, randomly selecting actions based on an adaptive ε-greedy strategy or outputting actions through a policy network; S43: Collect data on interaction with the environment and store it in the replay buffer; S44: If the total number of steps is greater than the number of steps to start training, start the training process, collect data from the replay buffer, and update it using the following formula and In the formula, r represents the reward function, γ represents the discount factor, represents the value function network, a t Represents action output, represents the value function target network, Update the policy network π by the following formula θ : Update the temperature coefficient α by minimizing the formula: The target network is updated by the following formula: θ'=ρθ'+(1-ρ)θ,φ' i' =rφ' i' +(1-r)φ i ,i={1,2}; In the formula, θ' represents the parameters of the policy target network, φ' i' Represents the parameters of the target network of the value function; If the total number of steps is not greater than the number of steps at the start of training, the training process ends; S47: End the training process and return to step S42 to continue interacting with the environment until the iteration round condition is met.

Images