Reinforcement learning program, reinforcement learning method, and information processing device
The reinforcement learning method addresses constraint violations by predicting demand reliability and adjusting model parameters to manage base station states, ensuring efficient power usage and congestion prevention.
Patent Information
- Application Number
- JP2023081533
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-17
- Publication Date
- 2024-11-28
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In reinforcement learning, it is challenging to control the behavior of the model such that constraints are not violated, leading to issues like communication congestion in radio access networks due to exceeded load thresholds.
A reinforcement learning method that calculates future demand reliability and adjusts model parameters to ensure actions comply with environmental constraints, using prediction variance to manage base station states effectively.
Prevents constraint violations by optimizing base station operations, reducing power consumption while minimizing communication congestion.
Smart Images

Figure 2024165383000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a reinforcement learning program, a reinforcement learning method, and an information processing device. [Background technology]
[0002] Reinforcement learning is one of the machine learning techniques. In reinforcement learning, an agent in an environment observes the current state and decides which action to take. The agent obtains rewards from the environment by selecting an action. In reinforcement learning, a policy that will obtain the most rewards through a series of actions is learned. In some reinforcement learning, for example, constraints that must be satisfied are set. When constraints are set, a model is generated that determines actions that satisfy the constraints.
[0003] Constrained reinforcement learning can be used, for example, in base station control to optimize power consumption in a wireless access network. In this case, a model for controlling switching between active and sleep states of multiple base stations is generated by constrained reinforcement learning. A constraint in the model for base station control is, for example, that the load of each base station does not exceed an upper limit.
[0004] As a technology for controlling the power consumption of base stations and the like, for example, a technology for generating control information for performing control with reduced power consumption while reducing the risk of control failure has been proposed. In addition, a technology for reducing switching overhead due to frequent resource switching in a base station device that aggregates baseband processing for multiple base stations has also been proposed. A technology has also been proposed for analyzing past traffic data using an artificial intelligence model, predicting the amount of traffic that will exceed the capacity of the base station, and dispatching a drone to the base station. A technology has also been proposed for performing power saving operations in real time, preventing inappropriate power saving processing and serious user access congestion, and reducing the number of error bits. A technology has also been proposed for reducing power consumption based on the analysis of traffic patterns at base stations in a cellular communication network.
[0005] Furthermore, an energy-aware mobile traffic offloading scheme in heterogeneous networks is also proposed, which jointly applies Deep Q Network (DQN) decision-making and advanced traffic demand prediction.
[0006] For example, a neural network can be used as a model for reinforcement learning. As a technology related to neural networks, for example, a new class of network model obtained by combining a conventional neural network with a mixture density model has been proposed. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] JP 2016-189529 A [Patent Document 2] International Publication No. 2015 / 045444 [Patent Document 3] US Patent Application Publication No. 2022 / 0394512 [Patent Document 4] Special Publication No. 2015-515196 [Patent Document 5] US Patent Application Publication No. 2020 / 0045627 [Non-patent literature]
[0008] [Non-Patent Document 1] Chih-Wei Huang, Po-Chen Chen, "Mobile Traffic Offloading with Forecasting using Deep Reinforcement Learning", arXiv:1911.07452, 18 Nov 2019 [Non-Patent Document 2] Christopher M. Bishop, "Mixture Density Networks", Neural Computing Research Group Report (NCRG / 94 / 004), Aston University, February, 1994 Summary of the Invention [Problem to be solved by the invention]
[0009] In reinforcement learning, it is difficult to control the behavior of a model as intended. Due to such characteristics of reinforcement learning, in constrained reinforcement learning, the indicators related to the constraints often exceed the threshold as a result of the model's behavior after learning. If the constraints are not observed, problems may occur in control using the model. For example, in the control of base stations in a wireless access network, if the constraints on the load of the base station are violated, it may cause communication congestion.
[0010] In one aspect, the present case aims to deter restrictions from being violated. [Means for solving the problem]
[0011] One proposal provides the following reinforcement learning program: The computer calculates a second demand after a certain time and a reliability of the second demand based on a current first demand for a service provided in a given environment. The computer determines an action to be taken on the environment according to a machine learning model based on input data including the second demand, the reliability, and a current first state of the environment. The computer executes the determined action on the environment. Then, based on the second state of the environment after the action is taken and the reward, the computer updates parameters of the model by constrained reinforcement learning that increases the reward to the extent that the constraints on the state of the environment are satisfied. Effect of the Invention
[0012] According to one embodiment, constraints can be prevented from being violated. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of a reinforcement learning method according to a first embodiment. [Diagram 2] FIG. 2 illustrates an example of computer hardware. [Diagram 3] FIG. 1 is a block diagram showing an example of the functions of a computer for realizing constrained reinforcement learning. [Figure 4] FIG. 13 is a diagram illustrating an example of a constrained reinforcement learning operation. [Diagram 5] FIG. 13 is a diagram illustrating an example of prediction distribution. [Figure 6] FIG. 13 is a diagram illustrating an example of behavior based on constrained reinforcement learning in the second embodiment. [Figure 7] 11 is a flowchart illustrating an example of a processing procedure of constrained reinforcement learning. [Figure 8] FIG. 13 is a diagram showing an example of a prediction result by a model obtained by constrained reinforcement learning using prediction variance. [Figure 9] FIG. 13 is a diagram illustrating an example of behavior based on constrained reinforcement learning in the third embodiment. [Figure 10] 13 is a flowchart illustrating an example of a prediction process procedure using a model trained by constrained reinforcement learning. [Figure 11] FIG. 13 is a diagram showing an example of a verification result when a predicted traffic volume to which a margin according to a prediction variance is added is input. [Figure 12] FIG. 1 illustrates an example of a base station management system in a wireless access network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Hereinafter, the present embodiment will be described with reference to the drawings. Note that each embodiment can be implemented in combination with a plurality of other embodiments as long as no contradiction occurs. First Embodiment The first embodiment is a reinforcement learning method that effectively utilizes the reliability of the predicted demand amount of a service provided in an environment, thereby preventing the violation of constraints in constrained reinforcement learning.
[0015] Fig. 1 is a diagram showing an example of a reinforcement learning method according to a first embodiment. Fig. 1 shows an information processing device 10 that implements the reinforcement learning method. The information processing device 10 can implement the reinforcement learning method by, for example, executing a reinforcement learning program.
[0016] The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 is, for example, a memory or a storage device included in the information processing device 10. The processing unit 12 is, for example, a processor or an arithmetic circuit included in the information processing device 10.
[0017] The storage unit 11 stores, for example, the initial state of the model 2. The storage unit 11 also stores the model 2 after it has been trained by the processing unit 12. The processing unit 12 performs constrained reinforcement learning of the model 2. For example, the processing unit 12 calculates a second demand amount (predicted demand amount) after a certain time and the reliability of the second demand amount based on a current first demand amount (current demand amount) for a service provided in a predetermined environment 1. The reliability is a numerical value indicating the degree of reliability of the second demand amount calculated as a predicted value after a certain time. As the reliability, for example, the variance of the random variable of the second demand amount (the square of the difference between the candidate value of the second demand amount (a value that the random variable can take) and the expected value (average value)) can be used.
[0018] The processing unit 12 determines an action to be taken for the environment 1 according to the machine learning model 2 based on input data including the second demand amount, the reliability, and the current first state of the environment 1. For example, the processing unit 12 performs a calculation according to the model 2 using the second demand amount, the reliability, and the current first state of the environment 1 as input to the model 2, and obtains an output of the model 2. The output of the model 2 indicates an action to be taken for the environment 1. The processing unit 12 executes the determined action for the environment 1.
[0019] When an action is taken, the state of the environment 1 changes from a first state to a second state. The processing unit 12 updates the parameters of the model 2 by constrained reinforcement learning that increases the reward within a range that satisfies the constraints on the state of the environment 1, based on the second state of the environment 1 after the action is taken and the reward.
[0020] The processing unit 12 repeats updating the parameters of the model 2 by such processing. As a result, in the process of determining an action using the model 2, the constraints are prevented from being violated. That is, the constrained reinforcement learning updates the parameters of the model 2 so as to increase the reward within a range that satisfies the constraints, but it is not possible to completely prevent the environment 1 from transitioning to a state that does not satisfy the constraints as a result of the action determined according to the model 2. By including the reliability of the predicted second demand amount in the input data of the model 2, the model 2 is trained to take a large margin for satisfying the constraints when the reliability is low, and to take a small margin for satisfying the constraints when the reliability is high. As a result, the constraints are prevented from being violated.
[0021] A wireless access network communication environment, for example, can be considered as the environment 1. In this case, the service provided in the environment 1 is a wireless information communication service in response to a request from a user terminal.
[0022] When the amount of communication traffic is low, it is desirable for the base station of the wireless access network to be in a sleep state (a state in which it does not communicate with terminals) to reduce power consumption. On the other hand, if there are too many base stations in a sleep state, they will not be able to cope with a short-term increase in the amount of communication traffic, and congestion may occur. In a wireless access network, it is important to avoid congestion as much as possible. Therefore, reinforcement learning is performed under the constraint that the load on the base station does not exceed a load threshold, for example.
[0023] When generating a model 2 for controlling a base station of a wireless access network, a current first communication traffic volume in a communication environment of the wireless access network becomes a first demand volume. Then, the processing unit 12 calculates a second communication traffic volume after a certain time of the wireless access network as a second demand volume based on the first communication traffic volume. Also, the first state of the wireless access network is, for example, a load (first load) of a base station.
[0024] In the process of determining an action, the processing unit 12 determines whether to activate or sleep the base station based on the second traffic volume, the reliability, and the first load of the base station in the radio access network according to Model 2. Then, the processing unit 12 instructs a device for controlling the base station in the radio access network to control the state of the base station as determined.
[0025] After instructing the control of the base station, the processing unit 12 issues a penalty when the load (second load) of the base station after controlling the base station exceeds a threshold value related to the load of the base station. The processing unit 12 also sets a larger reward value as the power consumption of the base station after controlling the base station is smaller. The processing unit 12 then updates the parameters of the model 2 so as to increase the reward without issuing a penalty. For example, the penalty is given as a negative reward, and the processing unit 12 updates the parameters of the model 2 so as to increase the sum of the negative reward based on the penalty and the reward (positive reward) according to the load.
[0026] As a result, when it is difficult to accurately predict the amount of communication traffic after a certain time, Model 2 is generated so that the communication capacity of active base stations has some margin. Also, when it is possible to predict the amount of communication traffic after a certain time with sufficient accuracy, Model 2 is generated so that more base stations are put into sleep mode. If the device that controls the base stations is made to control the base stations using Model 2, it is possible to reduce overall power consumption while preventing congestion caused by the load of the base stations exceeding a threshold.
[0027] It is also possible to prevent the constraints from being violated by using the reliability of the predicted demand when determining an action using the generated model 2, rather than when learning the model 2. For example, the processing unit 12 performs the following process using the model 2 generated by performing constrained reinforcement learning that increases the reward within a range that satisfies the constraints on the state of the environment 1, without using the reliability.
[0028] The processing unit 12 calculates a second demand amount after a certain time and the reliability of the second demand amount based on a current first demand amount for a service provided in a predetermined environment 1. The processing unit 12 calculates a third demand amount by adding a value according to the reliability to the second demand amount. Then, based on input data including the third demand amount and the current first state of the environment 1, the processing unit 12 determines an action to be taken with respect to the environment 1 according to a model 2 generated by constrained reinforcement learning that increases a reward within a range that satisfies a constraint on the state of the environment 1. The processing unit 12 executes the determined action with respect to the environment 1.
[0029] This allows the predicted demand to be overestimated when the confidence level is low, providing a larger margin for satisfying the constraints, thereby preventing the constraints from being violated.
[0030] Second Embodiment The second embodiment is a computer capable of generating a model for controlling a base station in a wireless access network by constrained reinforcement learning, in which behavior that violates constraints is suppressed.
[0031] 2 is a diagram showing an example of computer hardware. The entire computer 100 is controlled by a processor 101. A memory 102 and a plurality of peripheral devices are connected to the processor 101 via a bus 109. The processor 101 may be a multiprocessor. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP). At least a part of the functions realized by the processor 101 executing a program may be realized by an electronic circuit such as an application specific integrated circuit (ASIC) or a programmable logic device (PLD).
[0032] The memory 102 is used as a main storage device of the computer 100. The memory 102 temporarily stores at least a part of the OS (Operating System) program and application programs to be executed by the processor 101. The memory 102 also stores various data used in the processing by the processor 101. As the memory 102, for example, a volatile semiconductor storage device such as a RAM (Random Access Memory) is used.
[0033] The peripheral devices connected to the bus 109 include a storage device 103, a GPU (Graphics Processing Unit) 104, an input interface 105, an optical drive device 106, a device connection interface 107, and a network interface 108.
[0034] The storage device 103 electrically or magnetically writes and reads data to and from a built-in recording medium. The storage device 103 is used as an auxiliary storage device for the computer 100. The storage device 103 stores an OS program, application programs, and various data. Note that, for example, a hard disk drive (HDD) or a solid state drive (SSD) can be used as the storage device 103.
[0035] The GPU 104 is an arithmetic device that performs image processing. The GPU 104 is an example of a graphic controller. The monitor 21 is connected to the GPU 104. The GPU 104 displays an image on the screen of the monitor 21 in accordance with an instruction from the processor 101. The monitor 21 may be a display device using an organic EL (Electro Luminescence) display device or a liquid crystal display device.
[0036] The input interface 105 is connected to the keyboard 22 and the mouse 23. The input interface 105 transmits signals sent from the keyboard 22 and the mouse 23 to the processor 101. Note that the mouse 23 is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.
[0037] The optical drive device 106 uses a laser beam or the like to read data recorded on the optical disc 24 or write data to the optical disc 24. The optical disc 24 is a portable recording medium on which data is recorded so that it can be read by reflection of light. The optical disc 24 includes a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (ReWritable), etc.
[0038] The device connection interface 107 is a communication interface for connecting peripheral devices to the computer 100. For example, a memory device 25 or a memory reader / writer 26 can be connected to the device connection interface 107. The memory device 25 is a recording medium equipped with a function for communicating with the device connection interface 107. The memory reader / writer 26 is a device for writing data to the memory card 27 or reading data from the memory card 27. The memory card 27 is a card-type recording medium.
[0039] The network interface 108 is connected to the network 20. The network interface 108 transmits and receives data to and from other computers or communication devices via the network 20. The network interface 108 is a wired communication interface that is connected by a cable to a wired communication device such as a switch or a router. The network interface 108 may also be a wireless communication interface that is communicatively connected by radio waves to a wireless communication device such as a base station or an access point.
[0040] The computer 100 can realize the processing function of the second embodiment by the above-mentioned hardware. The computer 100 is an example of the information processing device 10 shown in the first embodiment. The information processing device 10 shown in the first embodiment can also be realized by the same hardware as the computer 100 shown in FIG.
[0041] The computer 100 executes a program recorded in, for example, a computer-readable recording medium to realize the processing function of the second embodiment. The program describing the processing contents to be executed by the computer 100 can be recorded in various recording media. For example, the program to be executed by the computer 100 can be stored in the storage device 103. The processor 101 loads at least a part of the program in the storage device 103 into the memory 102 and executes the program. The program to be executed by the computer 100 can also be recorded in a portable recording medium such as the optical disk 24, the memory device 25, or the memory card 27. The program stored in the portable recording medium becomes executable after being installed in the storage device 103 under the control of, for example, the processor 101. The processor 101 can also directly read out and execute the program from the portable recording medium.
[0042] The computer 100 can reproduce the wireless access network environment by software simulation. By simulating the wireless access network environment by the computer 100, the power consumption and load of the base station can be obtained without connecting to the actual wireless access network environment. This makes it possible to efficiently perform constrained reinforcement learning.
[0043] 3 is a block diagram showing an example of the functions of a computer for realizing constrained reinforcement learning. The computer 100 includes a storage unit 110, a base station control unit 120, an environment simulation unit 130, and a communication traffic prediction unit 140.
[0044] The storage unit 110 stores environment definition information 111. The environment definition information 111 is information related to the environment of a wireless access network. For example, the environment definition information 111 includes information such as the performance of each base station and the upper limit of the load on the base station.
[0045] The base station control unit 120 uses a machine learning model to control the state (active / sleep) of the base station based on the predicted value of the communication traffic volume obtained from the communication traffic prediction unit 140 and the reliability of the prediction, and dynamically corrects the parameters of the model. The reliability of the prediction is expressed, for example, by prediction variance. The prediction variance is an index showing how much the predicted output varies from the true value. The true value is, for example, the average value (expected value) of the probability distribution when the probability that each of the candidate values (possible values of the random variable) of the communication traffic volume is a correct value follows a predetermined probability distribution. The probability distribution is, for example, a Gaussian distribution (normal distribution). For example, the true value is the same value as the calculated predicted traffic volume.
[0046] When the base station control unit 120 determines the state of each base station as an action using a machine learning model, it instructs the environment simulation unit 130 of the state of each base station. As a result of applying the action, the base station control unit 120 acquires state information of the radio access network from the environment simulation unit 130. The acquired state information is, for example, the power consumption and load of each base station. The base station control unit 120 modifies the model based on the acquired state information. The model is, for example, a neural network. In this case, the base station control unit 120 modifies the weight parameters between neurons.
[0047] The environment simulation unit 130 simulates changes in the state of the environment defined by the reinforcement learning problem, based on the environment definition information 111. For example, the environment simulation unit 130 simulates the load and power consumption of each base station, based on the sleep control of the base stations in the wireless access network (which base station is to be put into a sleep state).
[0048] The environment simulation unit 130 also determines the current communication traffic volume according to, for example, a preset schedule. The environment simulation unit 130 may also determine the communication traffic volume at each time in the simulation so as to reproduce the time transition of the communication traffic volume in the actual wireless access network.
[0049] The environment simulation unit 130 transmits the load and power consumption of each base station obtained as a result of the simulation to the base station control unit 120. The environment simulation unit 130 also transmits the current communication traffic volume to the communication traffic prediction unit 140.
[0050] The communication traffic prediction unit 140 predicts future communication traffic volume based on past and current communication traffic volume. For example, the communication traffic prediction unit 140 predicts communication traffic volume using a trained model for communication traffic prediction. The model for predicting communication traffic volume is, for example, a neural network. When predicting the communication traffic volume after a predetermined time, the communication traffic prediction unit 140 simultaneously calculates a prediction variance indicating the reliability of the prediction result. The communication traffic prediction unit 140 transmits the predicted communication traffic volume and prediction variance to the base station control unit 120.
[0051] The function of each element shown in FIG. 3 can be realized, for example, by causing the processor 101 to execute a program module corresponding to that element. 4 is a diagram showing an example of constrained reinforcement learning operation. For example, the environment simulation unit 130 simulates, by software, the operations of a macro base station (MBS) 131, multiple small base stations (SBSs) 132, and multiple user equipments (UEs) 133. Hereinafter, when a base station is referred to, it includes the macro base station 131 and multiple small base stations.
[0052] For example, the environment simulation unit 130 sets the state of each of the multiple small base stations 132 in response to an instruction from the base station control unit 120 regarding the state (active or sleep) of each of the multiple small base stations 132 .
[0053] Furthermore, the environment simulation unit 130 reproduces the occurrence state of communication requests from each of the multiple user equipments 133 in accordance with a predetermined algorithm. The environment simulation unit 130 then reproduces the operation of selecting a base station to connect to by each of the multiple user equipments 133, and selects a connection destination. At this time, the connection destination can be the macro base station 131 or a small base station in an active state.
[0054] The environment simulation unit 130 calculates the load of each of the macro base station 131 and the multiple small base stations 132 based on the communication connection status of each of the multiple user equipment 133. The environment simulation unit 130 also calculates the power consumption of each of the macro base station 131 and the multiple small base stations 132 based on the communication connection status of each of the multiple user equipment 133 and the status of each base station.
[0055] The environment simulation unit 130 transmits the load and power consumption of each base station to the base station control unit 120. The environment simulation unit 130 also calculates the amount of communication traffic between each base station and each of the multiple user equipments 133. The environment simulation unit 130 transmits the calculated amount of communication traffic to the communication traffic prediction unit 140 as the current traffic amount.
[0056] The communication traffic prediction unit 140 predicts the future communication traffic volume between each base station and each of the multiple user equipments 133. The communication traffic prediction unit 140 also calculates the prediction variance of the traffic volume. Then, the communication traffic prediction unit 140 transmits the predicted traffic volume and the prediction variance to the base station control unit 120.
[0057] The base station control unit 120 acquires the current load and power consumption of each base station from the environment simulation unit 130. The base station control unit 120 also acquires the communication traffic volume and prediction dispersion from the communication traffic prediction unit 140. Then, the base station control unit 120 uses the model 121 based on the acquired information to determine the state of each of the multiple small base stations 132.
[0058] The base station control unit 120 also modifies the parameters of the model 121 based on the load and power consumption of each base station. At this time, the base station control unit 120 increases the reward as the power consumption of each base station decreases. Furthermore, when the load of each base station exceeds a threshold value indicated by the constraint on the load, the base station control unit 120 increases the penalty. Then, the base station control unit 120 modifies the parameters of the model 121 so that the reward becomes larger and the penalty becomes smaller. In this way, the constrained reinforcement learning proceeds.
[0059] 4, the communication traffic prediction unit 140 transmits the prediction variance in addition to the predicted traffic volume to the base station control unit 120, and the base station control unit 120 uses the prediction variance to find the states of the multiple small base stations 132. The reason why the prediction variance is used to find the states of the multiple small base stations 132 is to prevent the constraints from being violated.
[0060] In other words, when constrained reinforcement learning is performed without using prediction variance, the constraints on the load are often violated. Note that by increasing the penalty for the constraint being violated, the constraint becomes less likely to be violated, but if the penalty is too large, the reinforcement learning does not progress, making it difficult to generate a highly accurate model 121.
[0061] In constrained reinforcement learning, by using information on the reliability of prediction such as prediction variance, when the reliability is high, the base station control unit 120 can determine an action that significantly reduces power consumption while observing the constraint. When the reliability is low, the base station control unit 120 can determine an action that allows a large margin up to the load threshold so as not to violate the constraint. As a result, it is possible to prevent the constraint from being violated.
[0062] Next, prediction variance will be described. 5 is a diagram showing an example of prediction variance. The communication traffic prediction unit 140 assumes that the data sequence to be learned is based on a certain distribution, and predicts the mean and variance of the distribution as the output of the model 141. As the model 141, for example, a neural network in which a mixture density network (MDN) is provided on the output side of a Long Short-Term Memory (LSTM) network is used. In this case, the variance of the predicted traffic volume is calculated by the MDN.
[0063] For example, the communication traffic prediction unit 140 assumes a Gaussian mixture distribution using MDN, and predicts the Gaussian mixture coefficient π, mean μ, and variance δ of each distribution. In MDN, learning is performed using the minimum value of the negative log-likelihood shown in the following equation (1) as the loss function.
[0064]
number
[0065] The model 141 for predicting communication traffic has been trained in advance. For example, the communication traffic prediction unit 140 divides the area of the wireless access network into a plurality of partial areas. Then, the communication traffic prediction unit 140 calculates the predicted traffic volume and prediction variance for each partial area.
[0066] The predicted traffic volume and prediction variance for each partial region are used for learning and prediction of model 121 in base station control unit 120. 6 is a diagram showing an example of behavior by constrained reinforcement learning in the second embodiment. The prediction variance predicted by the communication traffic prediction unit 140 becomes an input to the constrained reinforcement learning method in the base station control unit 120. The base station control unit 120 performs learning of the model 121 using, for example, the state of the wireless access network (load of the base station), the predicted traffic volume, and a value reflecting the reliability (prediction variance) as input to the model 121.
[0067] For example, the base station control unit 120 determines the action regarding the state of the small base station (whether to make it active or sleep) based on the output from the model 121. Then, the base station control unit 120 instructs the environment simulation unit 130 of the determined action. Then, based on the determined action, the environment simulation unit 130 simulates the environment of the wireless access network and transmits the state (time label, load of each base station) and reward (power consumption, load of each base station) to the base station control unit 120. If the load of any base station exceeds a threshold, the base station control unit 120 imposes a penalty (negative reward). Furthermore, the base station control unit 120 increases the reward as the power consumption decreases. Furthermore, the base station control unit 120 may increase the reward as the load of the base station decreases. Then, the base station control unit 120 updates the parameters of the model 121 so that the reward becomes larger (the penalty becomes lower).
[0068] The base station control unit 120 improves the accuracy of the model 121 by repeatedly executing learning based on the predicted traffic volume and prediction variance. In this way, the base station control unit 120 learns the behavior of the model at the next time (for example, 10 minutes later), such as "high / low reliability (variance value) → risk not taken / taken." For example, when taking a risk, learning is performed so that the difference between the predicted load and the load threshold value is smaller than when not taking a risk.
[0069] As a result, after sufficient learning has been performed, base station control unit 120 keeps the number of small base stations put to sleep to a small number when the variance is high. On the other hand, base station control unit 120 puts many small base stations to sleep when the variance is low. This prevents the occurrence of a situation in which the load on a base station exceeds a threshold (constraints are violated).
[0070] In this way, by using the prediction variance, which indicates the reliability of the predicted traffic volume, in constrained reinforcement learning, even if the predicted traffic volume is the same, if the prediction variance is different, the determined action will be different. Then, by proceeding with learning so as to increase the reward, a model 121 that suppresses the occurrence of situations in which the constraint is violated is generated.
[0071] Next, a processing procedure of constrained reinforcement learning that prevents the violation of constraints will be described. 7 is a flowchart showing an example of a processing procedure for constrained reinforcement learning. The processing shown in FIG. 7 will be described below in order of step numbers.
[0072] [Step S101] The base station controller 120 initializes learning parameters. [Step S102] The environment simulation unit 130 initializes the environment of the radio access network.
[0073] [Step S103] The base station control unit 120 acquires the state of the environment. For example, the base station control unit 120 acquires the time label, the load of the base station, and the current traffic volume. The current traffic volume is, for example, the traffic volume for each of a plurality of partial areas obtained by dividing the area of the radio access network.
[0074] [Step S104] Based on the current traffic volume, the communication traffic prediction unit 140 calculates, for each partial region, the communication traffic volume and prediction variance for the next time (for example, 10 minutes from now) in the environmental simulation.
[0075] [Step S105] The base station control unit 120 predicts the behavior at the next time in the environmental simulation. For example, the base station control unit 120 performs calculations according to the model 121 using the current load of each base station, the predicted traffic volume for each partial region, and the prediction variance for each partial region as input data to the model 121. The calculation result indicates the predicted behavior.
[0076] The base station control unit 120 may include in the input data not only the current load of each base station but also the load during a predetermined period in the past. The base station control unit 120 may also include in the input data the current communication traffic volume (current demand volume).
[0077] [Step S106] The environment simulation unit 130 updates the environment based on the predicted behavior. For example, the environment simulation unit 130 transitions small base stations designated as sleeping to a sleep state with low power consumption. The environment simulation unit 130 also transitions small base stations designated as active to a state in which they communicate in response to a request from user equipment. The environment simulation unit 130 also calculates a reward from the state update result, and feeds back the reward to the base station control unit 120.
[0078] [Step S107] The base station control unit 120 adds an episode to the data set. The episode includes information such as the predicted traffic volume and prediction variance for each base station at a certain time in the environmental simulation, the predicted state of each base station, and the load and power consumption of each base station in the environment in that state.
[0079] [Step S108] The base station control unit 120 determines whether the number of episodes has reached a specified number. If the number of episodes has reached the specified number, the base station control unit 120 proceeds to step S109. If the number of episodes has not reached the specified number, the base station control unit 120 proceeds to step S103.
[0080] [Step S109] The base station control unit 120 inputs each episode included in the data set into the model 121, performs calculations according to the model 121, and predicts behavior. The base station control unit 120 also calculates the error between a value indicating the magnitude of reward assumed at the time of prediction for an active base station and a value indicating the magnitude of reward obtained from the fed-back load and power consumption. The value indicating the magnitude of reward is, for example, a value called a Q value.
[0081] [Step S110] The base station controller 120 feeds back the error to the model 121. For example, the base station controller 120 modifies the parameters of the model 121 to reduce the error.
[0082] [Step S111] The base station control unit 120 determines whether learning has ended. For example, the base station control unit 120 determines that learning has ended when the error becomes equal to or less than a certain value. If learning has ended, the base station control unit 120 proceeds to step S112. If learning is to continue, the base station control unit 120 proceeds to step S103.
[0083] [Step S 112 ] The base station control unit 120 stores the parameters of the trained model 121 . By controlling the state of the base station using the model 121 thus learned, the violation of the constraints is prevented. For example, a device for controlling the base station of an actual wireless access network is used instead of the environment simulation unit 130. This makes it possible to appropriately perform power saving control of the base station in the actual wireless access network within a range in which the load does not exceed a threshold value.
[0084] 8 is a diagram showing an example of a prediction result by a model obtained by constrained reinforcement learning using prediction variance. A prediction result comparison table 31 shows the difference in prediction results according to the difference in information input to the model 121. The types of information input to the model 121 are the following four patterns. 1. Current demand λ t 2. Forecast demand λ t+1 (λ has a ^) 3. Current demand λ t , predicted demand λ t+1 (λ has a ^) 4.Current demand λ t , predicted demand λ t+1 (λ is followed by a ^), prediction variance σ 2 In the wireless access network, the current demand is the current traffic volume. The predicted demand is the predicted traffic volume. The input to the model 121 during learning is the same as the input during verification. The load threshold set as a constraint is "0.1".
[0085] The reward is a value calculated based on the load and power consumption. A higher reward value indicates better performance of the model 121. A lower load value indicates better performance of the model 121. It is desirable to avoid the maximum load exceeding a threshold as much as possible. A lower power consumption value indicates better performance of the model 121.
[0086] As shown in the prediction result comparison table 31, the maximum load is reduced by including the prediction variance in the input. Moreover, when the prediction variance is included in the input, the maximum load does not exceed the load threshold. In other words, the constraint is met.
[0087] For example, if only the current demand is used to train the model 121, the maximum load is "0.10341±0.00526". In this case, there is a high possibility that the load will exceed the threshold value of "0.1". If only the predicted demand is used to train the model 121, the maximum load is "0.09601±0.00354". In this case, even if the load becomes the upper limit of the error of the maximum load, "0.09601+0.00354", it is still below the load threshold value, but it is quite close to the threshold value. Therefore, if there is an unexpected change in the environmental state, there is a risk that the load will exceed the threshold value. This is also the case when the current demand and the predicted demand are used to train the model 121.
[0088] In contrast, when the current demand, forecast demand, and forecast variance are used to train model 121, the maximum load is 0.08897 ± 0.00530. In this case, even if the load becomes the upper limit of the error of the maximum load, 0.08897 + 0.00530, there is still a margin of error before the load threshold (0.1). Therefore, even if an unexpected change occurs in the environment, the load is prevented from exceeding the threshold.
[0089] Third embodiment In the third embodiment, when base station control unit 120 predicts appropriate behavior using model 121, the predicted traffic volume to which a margin based on reliability is added is input to model 121. This allows each base station to be controlled on the safe side. For example, at times when there is a large variation in predictions, the predicted traffic volume is overestimated, so the behavior is determined so as not to put small base stations to sleep.
[0090] During learning of the model 121, it is not necessary to add a margin based on the reliability to the predicted traffic volume. That is, in the third embodiment, during learning, the communication traffic prediction unit 140 does not calculate the prediction variance. During learning, the base station control unit 120 does not include the prediction variance in the input to the model 121. Other processes during learning are the same as those in the second embodiment shown in FIG.
[0091] 9 is a diagram showing an example of behavior by constrained reinforcement learning in the third embodiment. In the third embodiment, the processing in the base station control unit 120 is different from that in the second embodiment. The base station control unit 120 does not use prediction variance during learning. Furthermore, the base station control unit 120 adds a margin according to the prediction variance to the predicted traffic volume during prediction.
[0092] For example, during learning, it is determined based on the predicted traffic volume of a certain partial area that one of the three surrounding small base stations is in active state and two are in sleep state. During prediction, when the same predicted traffic volume for that partial area as during learning is input to model 121, the number of active small base stations increases. In the example of Fig. 9, it is determined that all three small base stations surrounding the relevant partial area are to be active.
[0093] The amount of margin to be added to the predicted traffic volume is larger as the prediction variance is larger. For example, the "predicted traffic volume + prediction variance" is the predicted traffic volume input to the model 121 at the time of prediction. It is also possible for the user to arbitrarily set the size of the margin according to the prediction variance. In that case, the size of the margin is "a × prediction variance" (coefficient a is a positive real number). The user can arbitrarily set the size of the margin by specifying the value of coefficient a.
[0094] Next, a processing procedure at the time of prediction when adding a margin to the predicted traffic volume according to the magnitude of the prediction variance will be described. 10 is a flowchart showing an example of a prediction process using a model trained by constrained reinforcement learning. The process shown in FIG. 10 will be described below in order of step numbers.
[0095] [Step S201] The base station controller 120 initializes learning parameters. [Step S202] The environment simulation unit 130 initializes the environment of the radio access network.
[0096] [Step S203] The base station control unit 120 acquires the state of the environment. For example, the base station control unit 120 acquires the time label, the load of the base station, and the current traffic volume. The current traffic volume is, for example, the traffic volume for each of a plurality of partial areas obtained by dividing the area of the radio access network.
[0097] [Step S204] Based on the current traffic volume, the communication traffic prediction unit 140 calculates, for each partial region, the communication traffic volume and prediction variance for the next time (for example, 10 minutes from now) in the environmental simulation.
[0098] [Step S205] The base station controller 120 adds a margin to the predicted traffic volume. [Step S206] The base station control unit 120 predicts the behavior at the next time in the environmental simulation using the predicted traffic volume to which the margin has been added. For example, the base station control unit 120 adds a margin to the predicted traffic volume for each partial region according to the prediction variance of that partial region. The base station control unit 120 then includes the value after the margin has been added as the predicted traffic volume in the input data to the model 121, and performs calculations according to the model 121. The calculation result indicates the predicted behavior.
[0099] [Step S207] The environment simulation unit 130 updates the environment based on the predicted behavior. For example, the environment simulation unit 130 transitions small base stations designated as sleeping to a sleep state with low power consumption. The environment simulation unit 130 also transitions small base stations designated as active to a state in which they communicate in response to requests from user equipment.
[0100] [Step S208] The base station control unit 120 determines whether prediction has been applied up to a specified time step in the environmental simulation. If prediction has been applied up to the specified time step, the base station control unit 120 proceeds to step S209. If prediction has not been applied up to the specified time step, the base station control unit 120 proceeds to step S203.
[0101] [Step S209] The base station control unit 120 determines whether or not the prediction process is finished. For example, the base station control unit 120 determines that the prediction is finished when a command to finish the prediction is input from the user. If the prediction is finished, the base station control unit 120 proceeds to step S210. If the prediction is to be continued, the base station control unit 120 proceeds to step S203.
[0102] [Step S210] The base station control unit 120 stores the prediction result in the memory 102 or the storage device 103. The prediction result includes information such as the environmental state (including the load on the base station and the amount of communication traffic), the predicted traffic amount, the prediction variance, and the action at the next time.
[0103] By connecting to a device that actually controls the base station instead of the environment simulation unit 130 and controlling the base station based on the behavior predicted by the process shown in FIG. 10, it becomes possible to perform power-saving base station control that prevents the violation of constraints in the wireless access network.
[0104] Fig. 11 is a diagram showing an example of a verification result when a predicted traffic volume to which a margin according to the prediction variance is added is input. A prediction result comparison table 32 shows prediction results when a margin according to the prediction variance is included and not included at the time of prediction. The following two patterns of information were used as input to the model 121 at the time of prediction. 1. Current demand λ t 2. Forecast demand λ t+1 (λ has a ^) In the forecast result comparison table 32, when verifying the forecast accuracy for each input pattern, the forecast variance σ 2 The prediction results are shown for when the value of (σ is marked with a ^) is added and when it is not added. That is, when current demand is used as the input, the input to model 121 during learning remains the current demand value, and the input during verification is current demand + forecast variance. When forecast demand is used as the input, the input to model 121 during learning remains the forecast demand value, and the input during verification is forecast demand + forecast variance. The load threshold defined as a constraint is "0.1".
[0105] As shown in the forecast result comparison table 32, when a margin according to the forecast variance is added to the forecast demand, the maximum load value decreases. As a result, the maximum load value is prevented from exceeding the threshold value (violating the constraint).
[0106] For example, when using current demand, if a margin according to the forecast variance is not added, the maximum load will be "0.10341 ± 0.00526". In this case, there is a high possibility that the load will exceed the load threshold. In contrast, by adding a margin according to the forecast variance to the forecast demand, the maximum load becomes "0.09356 ± 0.00326". In this case, even if the load becomes the upper limit of the error, "0.09356 + 0.00326", it is still below the load threshold.
[0107] When the forecast demand is used, more accurate prediction is possible, and the maximum load value is lower than when the current demand is used. However, if a margin according to the forecast variance is not added, the maximum load will be "0.09601 ± 0.00354". In this case, the upper limit of the error "0.09601 + 0.00354" is below the load threshold value, but is quite close to the threshold value. Therefore, even if there is a slight unexpected change in the environment, there is a risk that the load will exceed the maximum value. In contrast, by adding a margin according to the forecast variance to the forecast demand, the maximum load is "0.09494 ± 0.00268". In this case, even if the load becomes the upper limit of the error "0.09494 + 0.00268", there is still a margin up to the load threshold. Therefore, even if there is an unexpected change in the environment, the load is prevented from exceeding the threshold value.
[0108] Other embodiments The models trained by the reinforcement learning shown in the second and third embodiments can be effectively used for managing wireless base stations in organizations that operate wireless access networks, such as mobile phone companies.
[0109] 12 is a diagram showing an example of a base station management system in a wireless access network. For example, a base station management server 200 manages the operation states (active or sleep) of wireless base stations 71-74. The base station management server 200 can also acquire information such as the traffic volume at each time in each of the base stations 71-74, the usage rate of the base station, and power consumption. The base station management server 200 transmits the acquired information to the computer 100 as status data 81.
[0110] The computer 100 executes reinforcement learning by imitating the operation of the wireless access network by the environment simulation unit 130, and generates a model in which a policy that maximizes the reward is set. The computer 100 then inputs the state indicated by the state data 81 acquired from the base station management server 200 into the model, and determines an action (whether to activate or sleep each base station). The computer 100 transmits action data 82 indicating the determined action to the base station management server 200. The base station management server 200 controls the operation states of the base stations 71 to 74 based on the action data 82.
[0111] When the computer 100 obtains the status data 81 indicating the status of the base stations 71-74 after transmitting the action data 82 from the base station management server 200, the computer 100 calculates the reward for the instructed action. Then, the computer 100 updates the policy of the model so as to increase the reward. This generates an accurate model and promotes power saving in the wireless access network.
[0112] In the example of FIG. 12, reinforcement learning is performed in a computer 100 separate from the base station management server 200, but reinforcement learning may be performed within the base station management server 200. Although the embodiments have been described above, the configurations of the parts shown in the embodiments can be replaced with other configurations having similar functions. Any other components or steps may be added. Furthermore, any two or more configurations (features) of the above-described embodiments may be combined. [Explanation of symbols]
[0113] 1 Environment 2 Models 10. Information processing device 11 Storage section 12 Processing section
Claims
1. Calculating a second demand amount after a certain time period and a reliability of the second demand amount based on a current first demand amount for a service provided in a predetermined environment; determining an action to be taken on the environment according to a machine learning model based on input data including the second demand, the confidence level, and a current first state of the environment; performing the determined action on the environment; updating parameters of the model by constrained reinforcement learning that increases the reward to a degree that satisfies a constraint on the state of the environment, based on a second state of the environment after the action is taken and a reward; Reinforcement learning program.
2. In the process of calculating the second demand and the reliability, a communication environment of a radio access network is set as the environment, a current first communication traffic volume of the radio access network is set as the first demand, and a second communication traffic volume of the radio access network after a certain time period is calculated as the second demand based on the first communication traffic volume; In the process of determining the action, a load of a base station in the radio access network is used as the first state to determine whether the base station is to be active or to be in sleep mode as the action; In the process of updating parameters of the model, a penalty is imposed when a second load of the base station after controlling the base station in accordance with the determined action exceeds a threshold value related to a load of the base station, a larger value is set as the reward as the power consumption of the base station after controlling the base station decreases, and the parameters of the model are updated so as to increase the reward without imposing the penalty. The reinforcement learning program according to claim 1.
3. In the process of calculating the second demand amount and the reliability, a variance of the second demand amount is calculated as the reliability. The reinforcement learning program according to claim 1.
4. Calculating a second demand amount after a certain time period and a reliability of the second demand amount based on a current first demand amount for a service provided in a predetermined environment; determining an action to be taken with respect to the environment in accordance with a model generated by constrained reinforcement learning that increases a reward within a range that satisfies a constraint on the state of the environment, based on input data including a third demand obtained by adding a value corresponding to the reliability to the second demand and a current first state of the environment; executing the determined action on the environment; Reinforcement learning program.
5. Calculating a second demand amount after a certain time period and a reliability of the second demand amount based on a current first demand amount for a service provided in a predetermined environment; determining an action to be taken on the environment according to a machine learning model based on input data including the second demand, the confidence level, and a current first state of the environment; performing the determined action on the environment; updating parameters of the model by constrained reinforcement learning that increases the reward to a degree that satisfies a constraint on the state of the environment, based on a second state of the environment after the action is taken and a reward; Reinforcement learning methods.
6. a processing unit that calculates, based on a current first demand for a service provided in a predetermined environment, a second demand after a certain time period and a reliability of the second demand, based on input data including the second demand, the reliability, and a current first state of the environment, determines an action to be taken on the environment in accordance with a machine learning model, executes the determined action on the environment, and updates parameters of the model by constrained reinforcement learning that increases the reward within a range that satisfies a constraint on the state of the environment, based on a second state of the environment after the action is performed and a reward; An information processing device having the above configuration.
Citation Information
Patent Citations
Power saving method and device for base stations
JP2015515196A
Wireless base station control information generating device and wireless communication system
JP2016189529A
Methods of reducing power consumption in a cellular network based on traffic analytics
US20200045627A1
Dynamic capacity management of a wireless network
US20220394512A1
Radio base station apparatus and resource allocation method
WO2015045444A1
Cited By
Learning device, control device, and learning method
WO2026154558A1
