A base station energy consumption optimization method based on improved Decision Transformer model

Abstract

The present invention discloses a base station energy consumption optimization method based on an improved Decision Transformer model, which belongs to the field of base station energy consumption management, including modeling the environment, generating a decision data set by reinforcement learning, calculating the trajectory return in the decision data, and calculating the mean μ of each time step of the calculated trajectory return by the Bootstrap method. t and standard deviation σ t , the distribution of modeling data samples; standardize the trajectory returns, calculate their distance distribution in the data samples, and normalize them as the prompt P of the trajectory sequence t ; Reconstruct the input trajectory sequence τ, use the DecisionTransformer model to train the reconstructed trajectory sequence τ until convergence, and obtain the optimal base station energy consumption management strategy. The present invention adopts the above-mentioned base station energy consumption optimization method based on the improved Decision Transformer model, improves the problem of the Decision Transformer model's excessive dependence on the quality of the data set, overcomes the defect that the model cannot learn the optimal strategy, and provides a new solution and plan for reducing base station energy consumption.

Description

Technical Field

[0001] The present invention relates to the technical field of base station energy consumption management, and in particular to a base station energy consumption optimization method based on an improved Decision Transformer model. Background Art

[0002] With the rapid development and widespread application of 5G technology, the number and energy consumption of 5G base stations have increased rapidly, becoming a key challenge for the sustainable development of the communications industry. Effective management of 5G base station energy consumption is of great significance for reducing operating costs, improving energy efficiency, and reducing environmental impact. Traditional base station energy-saving methods usually rely on fixed energy-saving strategies or manual experience, which are difficult to adapt to dynamically changing environments and business needs.

[0003] As a method of artificial intelligence, reinforcement learning has made significant progress in the field of intelligent decision-making. However, traditional reinforcement learning faces challenges in migration and generalization in base station energy management. This may be due to the overfitting of the algorithm to the training environment and the lack of the ability to effectively explore and adapt in new environments, resulting in the inability of the learned strategy to generalize well to new scenarios with different dynamic characteristics.

[0004] Recently, the Decision Transformer model has attracted attention as a new type of offline reinforcement learning algorithm. The model uses a self-attention mechanism and can effectively handle long sequence decision problems. By learning long-term dependencies in historical data, it can autonomously learn and adjust decision strategies to adapt to different dynamic environments and provide more efficient energy management quality. In addition, the Decision Transformer model excels in transfer generalization performance. It can use historical data from multiple tasks or environments to learn general strategies, so it can maintain good performance even in new and unseen environments. This feature makes it possible to overcome the challenges faced by traditional reinforcement learning and provide more intelligent and adaptive energy-saving strategies for base station energy management.

[0005] However, the Decision Transformer model still has some limitations when applied to base station energy management scenarios. These limitations include the model's over-reliance on the quality of the training dataset, the inability to learn the best strategy on non-optimal datasets, and the actual return generated during reasoning often failing to reach the expected return, where the expected return is the target specified during model reasoning and the actual return is the actual return generated by the interaction between the model and the environment. Therefore, it is necessary to propose new methods to improve the effect of the Decision Transformer model in base station energy management applications in order to further improve the efficiency of energy management. Summary of the invention

[0006] The purpose of the present invention is to provide a base station energy consumption optimization method based on an improved Decision Transformer model to solve the problems mentioned in the above background technology.

[0007] To achieve the above object, the present invention provides a base station energy consumption optimization method based on an improved Decision Transformer model, comprising the following steps:

[0008] S1. Model the environment, define the elements of the Markov decision process in the base station energy consumption management scenario, including state, action and reward mechanism, and use reinforcement learning to train and generate a decision data set for base station energy consumption management, calculate the trajectory return in the decision data set, and calculate the mean μ of each time step for the calculated trajectory return using the Bootstrap method. t and standard deviation σ t , the distribution of modeling data samples;

[0009] S2. Standardize the trajectory return, calculate its distance distribution in the data sample, and normalize it as a hint P of the trajectory sequence t ;

[0010] S3, reconstruct the input trajectory sequence τ, use the Decision Transformer model to train the reconstructed trajectory sequence until convergence, and obtain the optimal base station energy consumption management strategy.

[0011] Preferably, the states in S1 include base station power, total number of users, total transmission rate, base station sleep mode and number of antennas in use; actions include adjusting the sleep mode of each base station, the number of antennas in use and user association mode; rewards are aimed at reducing base station energy consumption, and user service quality is used as a penalty item.

[0012] Preferably, in S1, the cumulative reward after each decision point in the decision data set is represented as a trajectory return.

[0013] Preferably, S2 indicates P t The specific expression is as follows:

[0014]

[0015] Among them, R t is the trajectory return at time step t. The distribution of the trajectory return at time step t in the data sample is calculated through standardization, and then the result of the standardization is scaled to between 0 and 1 through the sigmoid function as the trajectory prompt P t .

[0016] Preferably, the specific expression of the reconstructed trajectory sequence τ is as follows:

[0017]

[0018] Among them, (R0,…,R T ) represents the trajectory return from time step 0 to T; (P0,…,P T ) represents the prompt from time step 0 to T; represents the state of the 0th to mth base stations from time step 0 to T; represents the actions of the 0th to the mth base station from time step 0 to T;

[0019] When training the model, a corresponding loss function is designed for discrete actions. The specific expression is as follows:

[0020]

[0021] in, is the true action probability at time step t; is the action probability predicted by the model at time step t; L is the length of the one-hot encoding of a single action; K is the size of the action space.

[0022] Therefore, the present invention adopts the above-mentioned base station energy consumption optimization method based on the improved Decision Transformer model, which has the following beneficial effects:

[0023] (1) An improved Decision Transformer model is proposed to align the expected return and actual return of the model during reasoning;

[0024] (2) By adding prompts, the improved Decision Transformer model can better learn the characteristics of sample data, solving the problem of the model's over-dependence on the quality of the dataset, and enabling the model to learn the best strategy even on non-optimal datasets;

[0025] (3) The Decision Transformer model improved by the present invention can show excellent performance when facing new tasks, which is no less than the current advanced reinforcement learning algorithm.

[0026] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0028] Figure 2 The diagram is an architecture diagram of the improved DecisionTransformer model according to an embodiment of the present invention;

[0029] Figure 3This is an analysis diagram of the expected return and actual return error during reasoning of the improved Decision Transformer model of an embodiment of the present invention;

[0030] Figure 4 This is a test histogram of the improved Decision Transformer model under a new task according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Example

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Reference Figure 1-3 The present invention discloses a base station energy consumption optimization method based on an improved Decision Transformer model, comprising the following steps:

[0034] S1. Model the environment, define the elements of the Markov decision process in the base station energy consumption management scenario, including state, action and reward mechanism, and use reinforcement learning to generate decision data. The cumulative reward after each decision point is expressed as a trajectory return. The calculated trajectory return is calculated by the Bootstrap method for the mean and standard deviation of each time step to model the distribution of data samples. In this embodiment, a regional-level large-scale MIMO cellular network scenario is simulated. The base stations in the network are distributed in a standard cellular structure. Each base station has 3 sleep modes. The power consumption discount factors corresponding to each sleep mode are 0.69, 0.5, and 0.29, respectively. Each base station can open up to 64 antennas at the same time. The total power consumption of the base station includes antenna transmission power consumption and fixed power consumption that is independent of the load; user services include 3 categories, and the corresponding budget delays are 50ms, 150ms, and 300ms, respectively. Set the simulation time interval to every millisecond, and one week is a complete cycle. The states include base station power, total number of users, total transmission rate, base station sleep mode, number of antennas in use, etc.; actions include adjusting the sleep mode of each base station, the number of antennas in use and the user association mode; rewards are aimed at reducing base station energy consumption, and user service quality is used as a penalty item; reinforcement learning methods are used to train and generate decision data sets. The states in the generated data set are standardized, and discrete actions are one-hot encoded. The trajectory returns in the decision data set are calculated, where the trajectory return is the cumulative reward obtained from the current time step to the end of the entire trajectory. In all samples, the mean μ of the trajectory return at each time step is estimated by the Bootstrap resampling method t and standard deviation σ t ,Modeling data sample distribution, set the sampling rounds to 1000 times, and each time draw 70% of the total samples.

[0035] S2, standardize the trajectory return, calculate its distance distribution in the data sample, and normalize it as a prompt for the trajectory sequence; trajectory prompt P t The expression is as follows:

[0036]

[0037] Where R t It is the trajectory return at time step t. The distribution of the trajectory return at time step t in the data sample is calculated through standardization, and then the result of the standardization is scaled to between 0 and 1 through the sigmoid function as a trajectory prompt.

[0038] S3, reconstruct the input trajectory sequence, and use the Decision Transformer model to train the reconstructed trajectory sequence until convergence to obtain the optimal base station energy consumption management strategy. Add trajectory prompts to the trajectory, and the specific expression of the reconstructed trajectory sequence τ is as follows:

[0039]

[0040] Among them, (R0,…,R T ) represents the trajectory return from time step 0 to T, (P0,…,P T ) represents the prompt from time step 0 to T, represents the state of the 0th to mth base stations from time step 0 to T, represents the actions of the 0th to the mth base station from time step 0 to T;

[0041] When training the model, a corresponding loss function is designed for discrete actions. The specific expression is as follows:

[0042]

[0043] in, is the true action probability at time step t; is the action probability predicted by the model at time step t; L is the length of the one-hot encoding of a single action; K is the size of the action space; when training the Decision Transformer model, the model observes the state information of all base stations at the current time step t and the previous trajectory sequence information, using the state Predict the action of the corresponding base station at time step t The model parameters are updated by gradient back propagation through the loss(p,q) loss function. Specifically, when training the Decision Transformer model, the output dimension of the embedding layer of the Decision Transformer model is set to 384, the number of decoder layers is 6, the number of attention heads in the multi-head attention is 1, and the learning rate is 0.001.

[0044] After training, evaluate the performance of the model and test its generalization in new tasks. Figure 3 The error between the expected return and the actual return during model reasoning is shown in , where DT is the unimproved Decision Transformer model and PDT is the improved Decision Transformer model. By comparing the absolute error between the expected return and the actual return during their reasoning, the improved Decision Transformer model can better learn the characteristics of the dataset, reduce the impact of the good and bad distribution of the dataset, and has an advantage in aligning the expected return and the actual return. The new task is defined as changing the size of the region and the number of base stations it contains to form a new whole for joint decision-making. By comparing the normalized total cumulative returns, in Figure 4The performance of the improved Decision Transformer model, the unimproved Decision Transformer model, and the PPO (Proximal Policy Optimization) reinforcement learning algorithm on new tasks is shown in the figure. Among them, PDT is the improved Decision Transformer model, DT is the Decision Transformer model, and PPO is the Proximal Policy Optimization reinforcement learning algorithm. The three were tested on the same new task. The results showed that the improved Decision Transformer model overcame the defects of the original model, and benefited from the model's own trajectory splicing ability. Its performance in the new task is no less than that of the current advanced PPO reinforcement learning algorithm. Therefore, it shows that the improved Decision Transformer model has good learning ability and strong generalization ability.

[0045] Therefore, the present invention adopts the above-mentioned base station energy consumption optimization method based on the improved Decision Transformer model. In the base station energy consumption management environment, by introducing and improving the Decision Transformer model, the distribution information of the modeled sample data is used as a trajectory prompt, thereby reducing the model's excessive dependence on the quality of the data set, enabling the model to better learn the sample data characteristics and learn the optimal strategy, and has strong generalization ability in new tasks, providing new solutions and solutions for saving base station energy consumption.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A base station energy consumption optimization method based on an improved DecisionTransformer model, characterized in that: The following steps are involved: S1. Model the environment, define the elements of the Markov decision process in the base station energy consumption management scenario, including state, action and reward mechanism, and use reinforcement learning to train and generate a decision data set for base station energy consumption management, calculate the trajectory return in the decision data set, and calculate the mean μ of each time step for the calculated trajectory return using the Bootstrap method. t and standard deviation σ t , the distribution of modeling data samples; The status includes base station power, total number of users, total transmission rate, base station sleep mode and number of antennas in use; the action includes adjusting the sleep mode of each base station, the number of antennas in use and the user association mode; the reward is aimed at reducing base station energy consumption, and the user service quality is the penalty item; The cumulative reward after each decision point in the decision dataset is expressed as the trajectory return; S2. Standardize the trajectory return, calculate its distance distribution in the data sample, and perform normalization as the prompt P of the trajectory sequence t ; S3, reconstruct the input trajectory sequence τ, use the Decision Transformer model to train the reconstructed trajectory sequence τ until convergence, and obtain the optimal base station energy consumption management strategy; The specific expression of the reconstructed trajectory sequence τ is as follows: Among them, (R0,…,R T ) represents the trajectory return from time step 0 to T; (P0,…,P T ) represents the prompt from time step 0 to T; represents the state of the 0th to mth base stations from time step 0 to T; represents the actions of the 0th to the mth base station from time step 0 to T; When training the model, a corresponding loss function is designed for discrete actions. The specific expression is as follows: in, is the true action probability at time step t; is the action probability predicted by the model at time step t; L is the length of the one-hot encoding of a single action; K is the size of the action space.

2. According to claim 1, a base station energy consumption optimization method based on an improved DecisionTransformer model is characterized in that: S2 prompts P t The specific expression is as follows: Among them, R t is the trajectory return at time step t. The distribution of the trajectory return at time step t in the data sample is calculated through standardization, and then the result of the standardization is scaled to between 0 and 1 through the sigmoid function as the trajectory prompt P t .

Images