Reinforcement learning-based universal base station temperature control method and system

Abstract

The application discloses a general base station temperature control method and system based on reinforcement learning, and the system is composed of an information sensing system, a central processor, an intelligent controller, a base station air conditioning system and a machine room dynamic environment monitoring system. The method uses data provided by the machine room dynamic environment monitoring system as the input of the method, and can be deployed in the BBU, without requiring excessive additional hardware investment to obtain input information and deploy algorithms. The method performs local model reasoning on base stations with good data quality and rich data sources, and constructs a model library. For base stations with less data and poor data preparation, a migration model and an initial control strategy are obtained through integrated learning and migration learning technical means, so that the method can effectively learn from existing data in the case of no data accumulation and a small amount of input parameters, and the migration and generalization ability of the method for different base stations is improved. Subsequently, through the technical means of reinforcement learning, local model reasoning is performed on the collected data to optimize the control strategy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of base station energy saving of communication system, and particularly relates to a general base station temperature control method and system based on reinforcement learning. BACKGROUND

[0002] With the rapid growth of data traffic and information services, mobile communication is entering the fifth generation (5G). Compared with 4G network, 5G network has faster rate, lower delay, larger capacity, higher reliability and other technical characteristics, and is the main development direction of future information technology.

[0003] Compared with 4G, 5G network needs to consume more energy due to its larger bandwidth, more channels, complex device structure and higher base station site and antenna density. In addition, with the expansion of 5G network application scenarios and supporting devices, energy consumption is expected to further increase. The communication industry is becoming a high-energy-consuming industry, and it is estimated that it will consume 20% of the global electricity by 2025, and the electricity cost will become the highest operating cost of mobile network operators, accounting for more than 15% of the total operating cost. In addition, the global electronic sustainable development proposes that the greenhouse gas emissions of the information and communication industry will reach 13 billion tons by 2020. The energy consumption of 5G base station accounts for more than 80% of the total system energy consumption, so the energy saving and consumption reduction of 5G base station is of great significance to the sustainable development of communication technology.

[0004] Because the devices in the base station dissipate heat to the base station room, the refrigeration system of the base station needs to consume a large amount of energy for thermal management to control the working environment within an acceptable temperature range, so the temperature control energy consumption of the base station refrigeration system brings considerable energy consumption and electricity cost, accounting for more than 40% of the total energy consumption of the base station. Therefore, the base station temperature control has great energy saving potential and space. However, the current industry base station energy saving means mainly focuses on the hardware of base station telecommunication equipment and the shutdown of main equipment, such as optimizing hardware architecture design, symbol shutdown, carrier shutdown, time slot shutdown and other technologies, and less research work focuses on base station temperature control energy saving. The existing methods related to the base station refrigeration system mainly assist or replace the air conditioner for heat dissipation, such as exhaust fan, phase change material, heat pump and the like. However, the energy saving effect of these methods is limited, and the installation and operation cost is high, which is not conducive to engineering application and promotion. SUMMARY

[0005] In order to solve the above problems, the application provides a general base station temperature control method and system based on reinforcement learning, which can meet the device safety, improve the energy efficiency of the base station and achieve the minimum running cost.

[0006] A general base station temperature control method based on reinforcement learning, comprising the following steps:

[0007] S1, local model inference is performed using the qualified base station data to obtain a local model, and the parameters of the local model are used for global model aggregation to obtain a model library;

[0008] S2, determine whether the base station is a qualified base station: if the base station is qualified, perform S8 and S10; otherwise, jump to S3;

[0009] S3, obtain base station basic information, and perform clustering-based matching of the base station basic information and the model library to obtain a migration model;

[0010] S4, obtain weather information and data fed back by a dynamic environment monitoring system, and construct a data set;

[0011] S5, construct a training set according to the data set for training to obtain a base station site thermal environment model;

[0012] S6, local model inference is performed: a Q-table is trained using the data set and the base station site thermal environment model to obtain a local model;

[0013] S7, determine whether the local model is better than the migration model: if yes, perform S8; otherwise, perform S9;

[0014] S8, select an action according to the Q-table obtained by the local model inference;

[0015] S9, select an action according to the current state of the site and the migration model;

[0016] S10, obtain the action output by S8 or S9, and output a temperature control strategy according to the relationship between the set air conditioner temperature set point and the action.

[0017] Further, in step S1, the model library is constructed by the following steps:

[0018] S1.1, for each qualified base station: obtain existing data of the site, the data including site basic information, external weather history data and dynamic environment monitoring system history data, and construct a site basic information vector;

[0019] S1.2, data preprocessing is performed on the obtained data, and a data set is constructed according to the preprocessed data wherein, is the surface temperature of all batteries at time t, is the temperature of the i-th group of batteries at time t, i = 1, … n b , n b is the number of batteries; is the inlet air temperature of all BBUs at time t, is the inlet air temperature of the j-th BBU at time t, j = 1, … n BBUn BBU is the number of BBUs; is the indoor temperature monitored by the motion monitoring system; is the outdoor temperature feature; is the air conditioning power feature;

[0020] S1.3, establishing an STGCN model, processing the data set obtained in S1.2 to obtain a training set and training the STGCN to obtain a base station site thermal environment model;

[0021] S1.4, updating the Q-Table using the data set and the base station site thermal environment model to obtain a local model, the local model including the Q-Table and a site basic information vector; the Q-Table storing the action value function Q(s, a) of the state-action pair (s, a) in the form of a table;

[0022] S1.5, uploading the local model to a cloud platform to obtain a model library by aggregating global models.

[0023] Further, step S1.3 includes the following steps:

[0024] S1.3.1, selecting five types of nodes, namely, batteries, BBUs, air conditioners, indoors, and outdoors, to construct a graph, that is, V t is a finite node set; the air conditioner node AC is represented by a load rate; the battery node b i is represented by its surface temperature , i = 1, … n b ; the BBU node BBU j is represented by its inlet air temperature , j = 1, … n BBU ; the indoor node indoor is defined as the center point of the site; and the outdoor node outdoor is defined as the center of the sunny wall;

[0025] S1.3.2, constructing an adjacency matrix W, satisfying where w ij is an element constituting W, representing the weight of the edge between any two nodes i and j in the graph, d i,j represents the distance between nodes i and j, σ 2 and l are threshold values for controlling the distribution and sparsity of the adjacency matrix W;

[0026] S1.3.3, constructing an STGCN model M(G t ; θ), where θ represents the model parameters to be trained;

[0027] S1.3.4, constructing a training set where are The tensor obtained by integrating in the time period t = n - r,..., n - 1, t represents the t-th moment, r represents the length of the time sequence, and n represents the current moment, respectively The first six items represent input features, and the last three items represent label features at t = n. The M(G t ; θ) is trained by using the training set to obtain the base station site thermal environment model.

[0028] Further, step S1.4 includes the following steps:

[0029] S1.4.1, set the cycle number Epoch, let epoch = 1; S1.4.2, the decision time granularity of the method is Δt, for the data set , every time two data with a time interval of Δt are taken out and They are the state transition of the environment caused by the action a t of the intelligent agent After data processing, the state of time t , the state of time t+Δt and the action a t at time t are obtained The value function is updated by using the update formula

[0030] S 1.4.3, take and a t as the input of the base station site thermal environment model obtained in S1.3 to obtain the output Calculate the difference between and s t+Δt If , execute S1.4.4, otherwise, jump to S1.4.5; is the threshold of the minimum accuracy requirement of the set base station site thermal environment model;

[0031] S1.4.4, for all actions a∈A and a≠a t , take and as the input of the base station site thermal environment model obtained in S1.3 to obtain the output The value function is updated by using the update formula

[0032] S1.4.5, let epoch = epoch + 1, epoch < Epoch, jump to S1.4.1, otherwise, exit the loop.

[0033] Further, step S3 includes the following steps:​​​

[0034] S3.1 Retrieve the basic base station information {x1, x2, ..., x} from the model library. m-1}, using base station basic information x m and {x1,x2,…,x} m-1 The sample set X = {x1, x2, ..., x} is formed. m};

[0035] S3.2, Define the dimension k1 after dimensionality reduction, the dimension k2 after clustering, and the dimension x after cluster partitioning. m Clustering is performed using a threshold k for the size of the cluster, resulting in cluster partitioning.

[0036] S3.3, Let x m The samples in the cluster are x1,…,x n If n≥k, retrieve the corresponding Q-table from the model library: Q-Table = {Q-Table1, ..., Q-Table...} n Otherwise, retrieve all Q-Tables from the model library: Q-Table = {Q-Table1, ..., Q-Table...} m-1}; The transfer model is obtained, and the transfer model includes and each x obtained during the clustering process i With x m similarity between x i For use with Q-Table i ∈Q-Table, i=1,...n tb The corresponding base station basic information vector, n tb This indicates the number of Q-Tables contained within a Q-Table.

[0037] Furthermore, step S4 includes the following steps:

[0038] S4.1 Obtain weather information and data from the environmental monitoring system to construct a base station status representation at the current time t. Let be the surface temperature of all batteries at time t. Let be the temperature of the i-th group of batteries at time t, i = 1, ..., n b n b The number of batteries; Let be the inlet air temperature of all BBUs at time t. Let be the inlet air temperature of the j-th BBU at time t, j = 1, ..., n BBU n BBU The number of BBUs; The indoor temperature monitored by the environmental monitoring system; Characteristics of outdoor temperature;

[0039] S4.2, Constructing the dataset This refers to the power characteristics of the air conditioner.

[0040] Furthermore, step S7 includes the following steps:

[0041] S7.1 Record the actions selected at each historical time t based on the migration model and the local model respectively. and

[0042] S7.2, Set the evaluation threshold η, And window size K, for a time period t i ∈[t start ,t start +Δt,...,t start +KΔt], calculate each t i Cumulative rewards earned in the past day based on both the migration model and the local model and in, For state s t The action selected based on the transfer model For state s t The action is selected based on the local model; when it exceeds η*K, the action is selected based on the local model. i satisfy When the local model is better than the transfer model, the local model is better than the transfer model; otherwise, the transfer model is better than the local model.

[0043] Furthermore, step S8 includes the following steps:

[0044] S8.1 Obtain the base station status representation at the current time t. Let be the surface temperature of all batteries at time t. Let be the temperature of the i-th group of batteries at time t, i = 1, ..., n b n b The number of batteries; Let be the inlet air temperature of all BBUs at time t. Let be the inlet air temperature of the j-th BBU at time t, j = 1, ..., n BBU n BBU The number of BBUs; The indoor temperature monitored by the environmental monitoring system; Characteristics of outdoor temperature;

[0045] S8.2, s t As the state input to the Q-Table obtained from local model inference, a greedy threshold ε is set; the action with the largest Q value is selected with probability ε. Randomly select an action a with a probability of 1-εt Output action a t .

[0046] Furthermore, step S9 includes the following steps:

[0047] S9.1, Obtaining the Q-Table and x i With x m similarity between and the base station state at the current time t. x i For use with Q-Table i ∈Q-Table, i=1,...n tb The corresponding base station basic information vector, i = 1, ... n b n b The number of batteries;

[0048] S9.2, For each Q-Table i ∈Q-Table, s t Select the action with the largest Q value as the state input, and obtain

[0049] Right now Obtain vector middle The corresponding component is 1, and the rest of the components are 0;

[0050] S9.3, Calculation but Output action a t .

[0051] A general-purpose base station temperature control system based on reinforcement learning includes:

[0052] The information sensing system is used to collect basic information about base stations, outdoor weather information, time-of-use electricity price information, and data fed back by the environmental monitoring system.

[0053] The environmental monitoring system is used to monitor the temperature and power data of equipment including BBU, transmission equipment, and batteries, and transmits the data collected by the information sensing system to the central processing unit.

[0054] The central processing unit, integrated in the baseband processing unit, is used to analyze and process the received data, obtain the base station status representation and update the dataset; the dataset is used as input to the optimization algorithm to update the model, generate a real-time temperature control strategy, and send the real-time temperature control strategy to the intelligent controller.

[0055] The intelligent controller is used to generate temperature control commands based on real-time temperature control strategies and send them to the air conditioner in the form of infrared signals;

[0056] Air conditioning is used to generate corresponding temperature control operations according to temperature control commands, which changes the environment of the base station equipment room. The environmental monitoring system monitors and collects data in real time and inputs it into the central processing unit to update and correct the optimization algorithm and related models online.

[0057] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0058] Due to significant differences in site enclosure structures, geographical locations, and equipment structures, there is no universally applicable base station temperature control strategy. Existing base station temperature control solutions require a large number of input parameters and high-quality data, which is difficult to meet for base stations with limited or poorly prepared data. The challenge of setting up air conditioning temperature control strategies for base station sites with limited or no data is known as the cold start problem.

[0059] The method described in this invention utilizes ensemble learning and transfer learning techniques to effectively learn from existing base station data without data accumulation and with only a small number of input parameters, thereby obtaining an initial control strategy and improving the transferability and generalization ability of the method across different base stations. Furthermore, by employing reinforcement learning techniques, the method leverages subsequently collected data for local model inference, further optimizing the control strategy and achieving algorithm intelligence.

[0060] This method performs local model inference at base stations with good data quality and abundant data sources to obtain a local model. The parameters of the local model are then uploaded to the cloud platform for global model aggregation to build a model library. For base stations with limited data or poor data readiness, the basic information of the base station is matched with the model library using clustering to obtain a transfer model. This transfer model is used to obtain an initial temperature control strategy and to build a dataset. The constructed dataset is used to build a spatiotemporal graph convolutional network site thermal environment model. Simultaneously, local model inference is performed: the Q-Table is updated using the dataset and the site thermal environment model. By evaluating the performance of the locally inferred model and the transfer model, when the locally inferred model is superior to the transfer model, the temperature control strategy is obtained using the local model, achieving adaptive switching of methods. After the local model inference is completed, the local model parameters are uploaded to the cloud platform for global model aggregation again.

[0061] Furthermore, this method focuses on optimizing the air conditioning system within the base station, which has significant energy-saving potential. It can reduce the operating costs of the base station and ensure that the equipment operates within a safe temperature range.

[0062] The system described in this invention uses data provided by the data center environment monitoring system as input and can be deployed in the BBU without requiring excessive additional hardware investment to obtain input information and deploy algorithms. Attached Figure Description

[0063] Figure 1 A schematic diagram of a general-purpose base station temperature control system based on reinforcement learning;

[0064] Figure 2 A flowchart of a general base station temperature control method based on reinforcement learning;

[0065] Figure 3 A comparison of air conditioning operating costs before and after optimization based on the above scheme for four base stations in Dongguan, Guangdong.

[0066] Figure 4 The average total site operating cost and average air conditioning operating cost of a base station in Dongguan, Guangdong, during the continuous testing phase. Detailed Implementation

[0067] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0068] This invention achieves the transferability and universality of energy-saving solutions based on parameter consensus and transfer learning, while ensuring the performance of the solutions.

[0069] Reference Figure 1 The embodiments of the present invention provide a 5G base station temperature control system, including a central processing unit, an intelligent controller (hereinafter referred to as the controller), a base station air conditioning system (hereinafter referred to as the air conditioner), an information sensing system, and a communication power supply and equipment room environment monitoring system (hereinafter referred to as the environmental monitoring system). The system can sense the status of the base station and set the optimal air conditioning temperature according to the current base station status, so as to minimize the power cost.

[0070] In this system: the software and programs based on the Q-Learning optimization algorithm and the Spatiotemporal Graph Convolutional Network (STGCN) model are deployed on the central processing unit (CPU), which is integrated into the baseband processing unit (BBU). The information sensing system collects basic base station information, outdoor weather information, time-of-use electricity price information, and data fed back from the environmental monitoring system. Basic base station information is obtained from the base station operator or through on-site surveys; outdoor weather information is obtained from the network; and time-of-use electricity price information is obtained from the local power department's website. Basic base station information includes: network architecture (C-RAN / D-RAN), indoor / outdoor site, equipment room size (small / medium / large), building envelope materials, battery type (lead-acid / lithium-ion), number of batteries, number of BBUs, and the average annual temperature of the city where the base station is located. Weather information includes outdoor temperature and weather conditions. The environmental monitoring system is a set of hardware and software components within the base station equipment room that monitors the environment and power environment of the equipment room, including the temperature of equipment such as BBUs, transmission equipment, and batteries, as well as the power data of the air conditioning system. The data collected by the information sensing system is sent to the central processing unit (CPU) as necessary input parameters for the optimization algorithm, and the CPU analyzes and processes the received data to obtain the base station status representation and update the dataset. The dataset is then used as input to the optimization algorithm for model updates: training the site thermal environment model, obtaining a Q-Table based on Q-Learning, and generating a real-time temperature control strategy. The CPU sends the generated real-time temperature control strategy to the intelligent controller via serial port commands. The intelligent controller learns the infrared code values ​​of the base station's air conditioner and sends infrared code commands for the temperature control strategy. Upon receiving the infrared code commands from the controller, the air conditioner automatically adjusts its temperature setpoint and performs corresponding operations. Changes in air conditioner operation affect the overall base station environment. The environmental monitoring system dynamically senses these changes and collects real-time equipment temperature and power data, inputting it to the CPU as feedback information to update and correct the temperature control strategy online.

[0071] The system receives a model library from the cloud platform, performs cluster-based matching between the base station's basic information and the model library to obtain a transfer model, and uses this transfer model to obtain an initial temperature control strategy. During the execution of the initial temperature control strategy, it accumulates data (including outdoor weather information and feedback data from the environmental monitoring system) and constructs a dataset, while simultaneously performing local model inference. It adaptively evaluates the performance of the local model and the transfer model, and when the locally inferred model is superior to the transfer model, it switches to using the local model to obtain the temperature control strategy. The system uses outdoor weather information and feedback data from the environmental monitoring system to construct a state representation of the base station, uses it as input to the model, and outputs the optimal temperature control strategy online. After the local model inference is completed, the local model parameters are uploaded to the cloud platform for global model aggregation again.

[0072] The real-time temperature control strategy generated by the central processing unit is sent to the intelligent controller via serial port.

[0073] The intelligent controller is used to generate temperature control commands based on real-time temperature control strategies and send them to the air conditioner in the form of infrared signals.

[0074] Air conditioning is used to generate corresponding temperature control operations according to temperature control commands, which changes the environment of the base station equipment room. The environmental monitoring system monitors and collects data in real time and inputs it into the central processing unit to update and correct the optimization algorithm and related models online.

[0075] For base stations with high-quality data and abundant data sources, the system directly performs local model inference to obtain a local model. For base stations with limited data and poor data readiness, the system performs cluster-based matching between the base station's basic information and the cloud platform's model library to obtain a transfer model, and then performs local model inference. By evaluating the merits of the local model and the transfer model, the system achieves adaptive switching between methods. The system can also sense the operating environment status of the base station and optimize the control of air conditioning while ensuring the safe and stable operation of the equipment, thereby reducing the operating cost of the data center's temperature control and achieving the goals of energy conservation and emission reduction.

[0076] Figure 2 This is a flowchart illustrating a base station energy-saving method according to an embodiment of the present invention. Specifically, it includes the following steps:

[0077] S1. Utilize base station data with good data quality and abundant data sources, perform local model inference through Q-Learning to obtain a local model, and upload the local model parameters to the cloud platform for global model aggregation to obtain a model library;

[0078] S2. Determine if the base station has good data quality and abundant data sources: If the base station has good data quality and abundant data sources, execute S8 and S10; otherwise, jump to S3.

[0079] S3. Obtain basic base station information, and perform cluster-based matching between the basic base station information and the model library to obtain the transfer model;

[0080] S4. Acquire weather information and data from the environmental monitoring system, and construct a dataset;

[0081] S5. Construct the STGCN training set based on the dataset and train it to obtain the base station site thermal environment model. The steps are the same as in S1.3.

[0082] S6. Perform local model inference: Train the Q-table using the dataset and the base station site thermal environment model to obtain the local model. The steps are the same as in S1.4.

[0083] S7. Determine if the local model is superior to the migration model; if yes, proceed to S8; otherwise, proceed to S9.

[0084] S8. Select the Q-Table action based on the local model inference;

[0085] S9. Select an action based on the current site status and migration model;

[0086] S10. Obtain the action output by S8 or S9, and output the temperature control strategy according to the relationship between the set air conditioning temperature setpoint and action a, setPoint = f(a,t), where setPoint is the air conditioning temperature setpoint and t represents the current time.

[0087] Furthermore, in S1, the model library is constructed through the following steps:

[0088] S1.1. Assess data quality in terms of data integrity, uniqueness, validity, and accuracy. For each base station with high data quality and abundant data sources: acquire existing data of the site, including basic site information, historical weather data, and historical data from the environmental monitoring system. Basic site information includes: network architecture (C-RAN / D-RAN), indoor / outdoor site, land area (small / medium / large), enclosure structure, battery type (lead-acid / lithium battery), number of batteries, number of BBUs, and the average annual temperature of the city where the site is located, and construct a basic site information vector.

[0089] S1.2. The acquired data is preprocessed, mainly including missing data imputation and outlier handling; the battery temperature characteristics are obtained. in, Let be the surface temperature of all batteries at time t. Let be the temperature of the i-th group of batteries at time t, i = 1, ..., n b n b Number of batteries; BBU temperature characteristics Let be the inlet air temperature of all BBUs at time t. Let be the inlet air temperature of the j-th BBU at time t, j = 1, ..., n BBU n BBU Number of BBUs; Indoor temperature characteristics Indoor temperature and outdoor temperature characteristics monitored by the environmental monitoring system. in α is the outdoor air temperature, β is the absorptivity of the outer surface of the building envelope to solar radiation, and α is the absorptivity of the outer surface of the building envelope to solar radiation. out To maintain the convective heat transfer coefficient of the outer surface of the structure, I represents the solar radiation intensity; air conditioning power characteristics. Where t represents time t. Construct the dataset.

[0090] S1.3 Establish the STGCN model, process the dataset obtained in S1.2 to obtain the training set and train STGCN to obtain the base station site thermal environment model;

[0091] S1.4 Perform local model inference: Update the Q-Table using the dataset and the base station site thermal environment model to obtain the local model. The local model includes the Q-Table and the site basic information vector. The Q-Table stores the action value function Q(s,a) of the state-action pair (s,a) in tabular form.

[0092] S1.5 Upload the local model to the cloud platform for global model aggregation to obtain the model library.

[0093] Furthermore, step S1.3 includes the following steps:

[0094] Temperature prediction is a typical time series forecasting problem, which involves predicting the temperature at the next time step based on previous measurements. The thermal process of the base station's indoor environment is defined graphically. The results between observation points are not independent but are connected by weighted paired nodes in the graph, and are considered a graphical signal. The graph at time t can be represented as G. t =(V t (E,W), V t E is a finite set of nodes, E is a finite set of edges, and W is the adjacency matrix.

[0095] S1.3.1, Select five types of nodes—battery, BBU, air conditioner, indoor, and outdoor—to construct the diagram, i.e. Each node is represented by its own characteristics: the air conditioning node (AC) is represented by its load rate. It means that, among them and These represent the power consumption during time phase t and the rated power in air conditioning cooling mode, respectively, with τ representing the duration of each time phase; battery node b i Due to its surface temperature This means that i = 1, ..., n b ;BBU node BBU j Due to its air inlet temperature This means that j = 1, ..., n BBU The indoor node is defined as the center point of the site, and is determined by... The outdoor node is defined as the center of the sun-facing wall, and is formed by... express;

[0096] S1.3.2 Construct the adjacency matrix W, satisfying Where w ijThe elements that make up W represent the weights of the edges between any two nodes i and j in the graph, d. i,j σ represents the distance between nodes i and j. 2 l and l are thresholds used to control the distribution and sparsity of the adjacency matrix W;

[0097] S1.3.3 Constructing the STGCN model M(G t ;θ), where θ represents the model parameters to be trained;

[0098] S1.3.4 Constructing the training set in They are respectively The tensor obtained by integrating time series t = n-1-r,...,n-1, where t represents time t, r represents the length of the time series, and n represents the current time. They are respectively At t=n, the first six terms represent the input features, and the last three terms represent the label features. M(G) is trained using the training set. t ;θ), to obtain the thermal environment model of the base station site.

[0099] Furthermore, step S1.4 includes the following steps:

[0100] Reinforcement learning consists of two main parts: the agent and the environment. The RL problem is defined on a Markov Decision Process (MDP). An MDP typically consists of a state space, an action space, a state transition probability function, a reward function, and a discount factor. At each time step, the environment has a state s, and the state space S is the set of all possible states; action a is the decision made by the agent based on the current state, and the action space A is the set of all possible actions; reward r is a value returned to the agent by the environment after the agent performs an action; state transition is the process by which the agent moves from the current state s at time t to the next state s′, and the state transition probability function is used to describe the state transition, denoted as p(s′|s,a); the discount factor represents the probability of state s at time t. t In terms of the current reward r t and future rewards r t+i The importance of this is that the goal of reinforcement learning is to obtain a policy π, which makes decisions at each time step based on the observed state, and these decisions are based on the action-value function. Given the current state s t The agent should execute the action with the highest value in action A, i.e. In reinforcement learning, the learning process is typically achieved through direct interaction with the environment, and the action-value function needs to be updated after each interaction according to the following formula:

[0101]

[0102] Where Q(s,a) represents the value of the agent taking action a in state s; s′ is the next state; r ss′ α represents the reward when the agent takes action 'a' to transition from state 's' to state 's'; α is the learning parameter; γ represents the discount factor.

[0103] The state at time t is represented as Action is represented as Directly calculating the state transition function is a very challenging task, so a neural network is used to train and estimate the state transition. The thermal process at a base station site is a complex physical process, influenced by time-varying uncertainties such as solar radiation, weather conditions, and network traffic load, making it time-varying. Furthermore, the thermal process at a base station site primarily involves convective heat exchange, with energy and heat transfer occurring between energy nodes, leading to its spatial dependence. Therefore, STGCN is used to capture this spatiotemporal dependence, establishing a thermal environment model of the base station site for estimating the state transition. The reward function is defined as...

[0104] R(s t ,a t )=-E t +f(a t ),

[0105] Where E t and f(a) t ) represent the operating cost and penalty function of the air conditioner during time period t, respectively. t Represented as: Where c t This represents the time-of-use electricity price for period t. f(a) t ) is a piecewise function:

[0106]

[0107] in and ... t ) = 0; otherwise, f(a) = 0. t It is set to be based on action a t The function is a monotonically increasing function. The reason is that when at least one device does not meet the redline temperature constraint, the agent (i.e., the air conditioning system) should prioritize actions with high load rates to provide more cooling. As long as monotonicity is satisfied, f(a) is a monotonically increasing function. t () can be any form other than a univariate linear function.

[0108] S1.4.1. Set the number of loop iterations Epoch, and let epoch = 1;

[0109] S1.4.2. The decision time granularity of the method is Δt. For the data set in it, take out two pieces of data with a time interval of Δt each time and They can be regarded as an action a of the agent t causing a state transition of the environment After data processing, obtain the state at time t the state at t + Δt and the action a at time t t , and use the update formula to update the value function;

[0110] S1.4.3. Take and a t as the input of the base station site thermal environment model obtained in S1.3, and obtain the output Calculate and the difference between s t+Δt If then execute S1.4.4, otherwise, jump to S1.4.5. Where is the threshold of the minimum accuracy requirement of the set base station site thermal environment model;

[0111] S1.4.4. For all actions where a ∈ A and a ≠ a t Take and as the input of the base station site thermal environment model obtained in S1.3, and obtain the output Use the update formula to update the value function;

[0112] S1.4.5. Let epoch = epoch + 1. When epoch < Epoch, jump to S1.4.1, otherwise, exit the loop.

[0113] Furthermore, in steps S1 and S2:

[0114] The judgment criteria for base stations with good data quality and rich data sources are:

[0115] There are many sensor deployment quantities, and the occurrences of data missing and outliers are few. Specifically, the sensor deployment points should include: each wall, each group of storage batteries, each BBU, indoor and outdoor temperatures, and the probabilities of data missing and outliers should be lower than 1%.

[0116] Furthermore, step S3 includes the following steps: ​

[0117] Spectral clustering is used for clustering. The main idea of ​​spectral clustering is to treat all samples as points in space, which can be connected by edges. The edge weight between two points that are far apart has a lower weight, while the edge weight between two points that are close together has a higher weight. By slicing the graph formed by all data points, the goal is to minimize the sum of edge weights between different subgraphs and maximize the sum of edge weights within a subgraph, thereby achieving the purpose of clustering.

[0118] S3.1 Retrieve the basic base station information {x1, x2, ..., x} from the model library. m-1}, using base station basic information x m and {x1,x2,…,x} m-1 The sample set X = {x1, x2, ..., x} is formed. m};

[0119] S3.2, Define the dimension k1 after dimensionality reduction, the dimension k2 after clustering, and the dimension x after cluster partitioning. m Clustering is performed based on a threshold k of the cluster size, resulting in cluster partitioning. The clustering process is as follows:

[0120] 1) Construct the similarity matrix Φ of the samples using the fully connected method, and use the Gaussian kernel function to define the weight of the edge between two points, i.e. Where φ ij This represents the similarity between node i and node j. For x i x j L2 norm distance between them, σ 2 Indicates variance;

[0121] 2) Construct an adjacency matrix W based on the similarity matrix Φ, satisfying w ij =φ ij , where w ij Represent the weight of the edge between node i and node j; construct the degree matrix D, where the elements d in the degree matrix D are... i satisfy d i This represents the degree of node i, which is the number of edges connected to that node;

[0122] 3) Calculate the Laplace matrix L = DW;

[0123] 4) Construct the standardized Laplace matrix D -1 / 2 LD -1 / 2 ;

[0124] 5) Calculate D -1 / 2 LD -1 / 2 The eigenvector z corresponding to the k1 smallest eigenvalues;

[0125] 6) Standardize the matrix formed by the corresponding eigenvectors z by row, and finally form an m×k1 dimensional eigenma matrix Z;

[0126] 7) For each row in Z, take it as a k1-dimensional sample, and there are m samples in total. Use K-Means to cluster the samples, and the cluster dimension is k2.

[0127] 8) Obtain cluster partitioning

[0128] S3.3 After cluster partitioning, let x m The samples in the cluster are x1,…,x n If n≥k, retrieve the corresponding Q-table from the model library: Q-Table = {Q-Table1, ..., Q-Table...} n Otherwise, retrieve all Q-Tables from the model library: Q-Table = {Q-Table1, ..., Q-Table...} m-1}; The transfer model is obtained, and the transfer model includes and each x obtained during the clustering process i With x m similarity between x i For use with Q-Table i ∈Q-Table, i=1,...n tb The corresponding base station basic information vector, n tb This indicates the number of Q-Tables contained within a Q-Table.

[0129] Furthermore, step S4 includes the following steps:

[0130] S4.1 Obtain weather information and data from the environmental monitoring system to construct a base station status representation at the current time t.

[0131] S4.2, Constructing the dataset

[0132] Furthermore, step S7 includes the following steps:

[0133] The algorithm evaluates the performance of the migration model and the local model by comparing the operating costs generated by specifying air conditioning operation strategies using the migration model and the local model the previous day.

[0134] S7.1 Record the actions selected at each historical time t based on the migration model and the local model respectively. and

[0135] S7.2, Set the evaluation threshold η, And window size K, for a time period ti ∈[t start ,t start +Δt,...,t start +KΔt], calculate each t i Cumulative rewards earned in the past day based on both the migration model and the local model and in, For state s t The action selected based on the transfer model For state s t The action is selected based on the local model. When t exceeds η*K... i satisfy If the local model is superior to the transfer model, then the local model is considered superior; otherwise, the transfer model is considered superior to the local model.

[0136] Furthermore, step S8 includes the following steps:

[0137] S8.1 Obtain the base station state representation at the current time t obtained from step S4.

[0138] S8.2, s t As the state input to the Q-Table obtained from local model inference, a greedy threshold ε is set; the action with the largest Q value is selected with probability ε. Randomly select an action a with a probability of 1-ε t Output action a t ;

[0139] Furthermore, step S9 includes the following steps:

[0140] S9.1 Obtain the Q-Table and the result obtained in step S3. and the base station state representation at the current time t obtained from step S4.

[0141] S9.2, For each Q-Table i ∈Q-Table, s t Select the action with the largest Q value as the state input, and obtain Right now Obtain vector middle The corresponding component is 1, and the rest of the components are 0;

[0142] S9.3, Calculation but Output action a t .

[0143] Figure 3The average operating cost of four test base stations in Dongguan, Guangdong, based on the above scheme. Figure 4 It can be seen that after the above scheme is applied to optimize the base stations, the operating costs of the base stations are reduced significantly, saving more than 20%. That is, this method has a significant power-saving effect on all types of base stations.

[0144] To investigate the effectiveness of the method's evolution, a three-week test was conducted, divided into three one-week phases. Figure 4 The figures show the average total operating cost and average air conditioning operating cost measured at each stage for a base station in Dongguan, Guangdong. It can be seen that in stage one, using the migration model can achieve approximately 4% savings in operating costs. With the accumulation of data, the cost-saving effect will become increasingly better.

[0145] This invention provides an operation optimization device for general base station temperature control based on reinforcement learning, used to execute the aforementioned general base station temperature control optimization method based on reinforcement learning. The optimization calculation module can be functionally divided according to the above method examples. For example, each function can be divided into its own module, or two functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0146] Since the operation optimization device for general base station temperature control based on reinforcement learning provided in this embodiment of the invention can be used to execute the above-mentioned general base station temperature control optimization method based on reinforcement learning, the technical effects it can achieve can be referred to the above-mentioned method embodiments, and this embodiment of the invention will not be repeated here.

[0147] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0148] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0149] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely exemplary descriptions of the invention as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A general base station temperature control method based on reinforcement learning, characterized by, The method comprises the following steps: S1, using the required base station data to perform local model inference to obtain a local model, and using the parameters of the local model to perform global model aggregation to obtain a model library; S2, judging whether the base station is a required base station: if the base station is required, performing S8 and S10; otherwise, jumping to S3; S3, obtaining base station basic information, and performing clustering-based matching between the base station basic information and the model library to obtain a migration model; S4, obtaining weather information and data fed back by a dynamic environment monitoring system, and constructing a data set; S5, constructing a training set according to the data set to perform training to obtain a base station site thermal environment model; S6, performing local model inference: using the data set and the base station site thermal environment model to train a Q-table to obtain a local model; S7, judging whether the local model is better than the migration model: if yes, performing S8; otherwise, performing S9; S8, selecting an action according to the Q-table obtained by the local model inference; S9, selecting an action according to the current state of the site and the migration model; S10, obtaining the action output by S8 or S9, and outputting a temperature control strategy according to the relationship between the set air conditioner temperature set point and the action; In the step S1, the model library is constructed through the following steps: S1.1, for each required base station: obtaining existing data of the site, wherein the data comprises site basic information, external weather historical data and dynamic environment monitoring system historical data, and constructing a site basic information vector; S1.

2. Perform data preprocessing on the acquired data, and construct a dataset based on the preprocessed data. ,in, For all batteries t Surface temperature at any given time For the first i Battery packs t Temperature at any moment i =1,…, , The number of batteries; Let be the inlet air temperature of all BBUs at time t. For the first j A BBU in t The temperature of the incoming air at any given time j =1,…, , The number of BBUs; The indoor temperature monitored by the environmental monitoring system; Characteristics of outdoor temperature; This refers to the power characteristics of the air conditioner. S1.3, establishing an STGCN model, processing the data set obtained in S1.2 to obtain a training set, training the STGCN, and obtaining a base station site thermal environment model; S1.4, further updating the Q-Table using the dataset and the base station site thermal environment model to obtain a local model, the local model comprising the Q-Table and a site base information vector; the Q-Table storing action value functions for state-action pairs (s, a) in tabular form ; S1.5, uploading the local model to a cloud platform to perform global model aggregation to obtain a model library; The step S3 comprises the following steps: S3.1, retrieve base station base information in the model library , with base station base information and constitute a sample set ; S3.2, set the dimension after dimension reduction , the dimension after clustering , the cluster division after clustering , the threshold of the cluster size , cluster to obtain the cluster division after clustering; S3.3、Set the sample of the cluster where the UE is located ; if , take the corresponding Q-table from the model library ; otherwise, take all Q-tables from the model library: ; get the migration model, which includes and the similarity between each and obtained in the clustering process , is the base station basic information vector corresponding to , and represents the number of Q-tables contained in .

2. The reinforcement learning based universal base station temperature control method according to claim 1, characterized in that, The step S1.3 comprises the following steps: S1.3.1, select five types of nodes, battery, BBU, air conditioner, indoor, outdoor, to construct the graph, that is , is a finite node set; the air conditioner node AC is represented by the load rate; the battery node is represented by its surface temperature , i =1,…, ; the BBU node is represented by its inlet air temperature , j =1,…, ; the indoor node indoor is defined as the center point of the site; the outdoor node outdoor is defined as the center of the sunny wall; S1.3.2, Constructing the Adjacency Matrix , satisfying ; wherein, is an element of the set denotes the weight of the edge between any two nodes and in the graph, denotes the distance between nodes and , and are thresholds for controlling the distribution and sparsity of the adjacency matrix ; S1.3.3 Building the STGCN model wherein θ represent the model parameters to be trained; S1.3.4, constructing training set wherein , , , , are respectively , , , , in time period integrated tensor, t representing the t moment, r representing the length of time series, n representing the current moment, are respectively , , in t=n value of the first six items represent input features, and the last three items represent label features, training , to obtain the base station site thermal environment model.

3. The reinforcement learning based universal base station temperature control method of claim 1, wherein, The step S1.4 comprises the following steps: S1.4.1, set the cycle number Epoch, let epoch = 1; S1.4.2, the decision time granularity of the method is , for the data set , every time two data with a time interval of are taken out, which are a state transition of the environment caused by the action of the agent , after data processing, the state at time , the state at time and the action at time are obtained, and the value function is updated by using the update formula ; S1.4.3, calculate and as input to the base station site thermal environmental model obtained in S1.3, obtain output , calculate and the difference , if then execute S1.4.4, otherwise, jump to S1.4.5; is a threshold value of the minimum accuracy requirement of the base station site thermal environmental model set. S1.4.

4. For all actions of the base station site , the and are obtained as inputs to the base station site thermal environment model obtained in S1.3, obtaining outputs , the value function is updated using the update formula ; for learning parameters, for the agent to take actions from a state to a state at time t, for a discount factor, for the actions possible at the next state s', parameters meanings for the next state, is a set of all possible actions; S1.4.5, let epoch = epoch + 1, and jump to S1.4.1 when epoch < Epoch, otherwise, exit the loop.

4. The reinforcement learning based universal base station temperature control method of claim 1, wherein, The step S4 comprises the following steps: S4.1, acquire weather information, data fed back by the dynamic environment monitoring system, and construct the base station state representation at the current time ; ; , Tsurf(t) is the surface temperature of all the batteries at the time t, t , , T(t) is the temperature of the i-th battery group at the time t, i , t =1,…, i , , is the number of batteries; , Tair(t) is the air inlet temperature of all the BBUs at the time t, , j , t , j =1,…, , is the number of BBUs; , Troom is the indoor temperature monitored by the dynamic environment monitoring system; , Toutside is the outdoor temperature characteristic; S4.2, Constructing dataset ; For air conditioner power characteristics.

5. The reinforcement learning based universal base station temperature control method of claim 1, wherein, The step S7 comprises the following steps: S7.1, record at each historical time step actions selected according to the migration model and the local model, respectively and ; S7.2, set evaluation threshold , and window size K , for a period of time , calculate each cumulative reward obtained according to the migration model and the local model in the past day and , wherein is the action selected according to the migration model in state , is the action selected according to the local model in state ; when of satisfies , the local model is superior to the migration model; otherwise, the migration model is superior to the local model.

6. The reinforcement learning based universal base station temperature control method of claim 1, wherein, The step S8 comprises the following steps: S8.1, Get the current time Base station status representation , For all batteries t Surface temperature at any given time For the first i Battery packs t Temperature at any moment i =1,…, , The number of batteries; Let be the inlet air temperature of all BBUs at time t. For the first j A BBU in t The temperature of the incoming air at any given time j =1,…, , The number of BBUs; The indoor temperature monitored by the environmental monitoring system; Characteristics of outdoor temperature; S8.2, will As the state input to the Q-Table obtained from local model inference, a greedy threshold is set. ;by The probability of choosing the action with the highest Q value ,by The probability of randomly selecting an action Output action , It refers to the set of all possible actions.

7. The reinforcement learning based universal base station temperature control method of claim 1, wherein, The step S9 comprises the following steps: S9.1, obtaining and a similarity between and a base station state representation of a current time point ; ; a base station base information vector corresponding to , i =1,…, , a number of accumulators; S9.2, for each , the action with the maximum Q-value is selected as the state input, resulting in , i.e. , resulting in the vector , in the corresponding component is 1 and the remaining components are 0; is the set of all possible actions; S9.3, compute then , output action .

8. A reinforcement learning based universal base station temperature control system for implementing the method of claim 1, characterized by It comprises: An information perception system for collecting base station basic information, outdoor weather information, time-of-use electricity price information and data fed back by a dynamic environment monitoring system; A dynamic environment monitoring system for monitoring the temperature of devices including BBU, transmission equipment and battery and the power data of air conditioners, and transmitting the data collected by the information perception system to a central processor; A central processor integrated in a baseband processing unit, for analyzing and processing the received data by using the method in any one of claims 1-7, obtaining a base station state representation and updating a data set; using the data set as an input of an optimization algorithm to update a model, generating a real-time temperature control strategy, and sending the real-time temperature control strategy to an intelligent controller; An intelligent controller for generating a temperature control instruction according to the real-time temperature control strategy, and sending the temperature control instruction to the air conditioner in the form of infrared signal. The air conditioner is used to generate corresponding temperature control operation according to temperature control instructions, to bring changes to the base station machine room environment, and the dynamic ring monitoring system monitors and collects data in real time and inputs to the central processor to update and correct the optimization algorithm and related models online.

Images