Control device, control action calculation method, and program

Abstract

This control device is for performing control on an environment provided with a plurality of control targets that each provide a network service, and comprises a constraint optimization control unit that calculates, on the basis of an observation value from the environment and a control action plan that maximizes the power consumption reduction amount and the average communication quality, a control action such that the degree of similarity with the control action plan increases, and a constraint violation of the constraint in which the communication quality achieves a target value decreases.

Description

Control device, control action calculation method, and program

[0001] The present invention relates to techniques for reducing power consumption in networks.

[0002] In recent years, the rapid increase in traffic demand in cellular networks has led to constant demand for improved performance, while at the same time, the impact of environmental loads has led to the need to reduce the power consumption of communication equipment. More than half of the power consumed by communication equipment is consumed by base stations, and in order to reduce power consumption, it has been considered to temporarily put less used base stations into sleep mode.

[0003] Generally, base stations are designed based on the traffic usage volume during peak hours, and the traffic usage rate is low during non-peak hours. Non-Patent Document 1 discloses a technology that utilizes this property of base stations to reduce power consumption by putting the base station to sleep during non-peak hours.

[0004] J. Wu, Y. Zhang, M. Zukerman, and EK -N. Yung, "Energy-Efficient Base-Stations Sleep-Mode Techniques in Green Cellular Networks: A Survey," in IEEE Communications Surveys & Tutorials, vol. 17, no. 2, pp. 803-826, Secondquarter 2015, doi: 10.1109 / COMST.2015.2403395.

[0005] However, conventional technology that puts base stations to sleep can cause them to sleep excessively, which can degrade the communication quality of users who use communication services in the base station's area.

[0006] The present invention has been made in consideration of the above points, and aims to provide a technology that enables control over an environment in which network services are provided, while achieving power consumption reduction and satisfying communication quality constraints.

[0007] According to the disclosed technology, there is provided a control device for controlling an environment having a plurality of control objects that provide network services, the control device including a constraint optimization control unit that calculates a control action based on a control action plan that maximizes the amount of power consumption reduction and average communication quality and an observed value from the environment, so as to increase the similarity between the control action plan and the control action plan, and to reduce violations of the constraint that communication quality achieves a target value.

[0008] The disclosed technology provides a technology that enables control of an environment in which network services are provided, while achieving reduction in power consumption and satisfying communication quality constraints.

[0009] 1 is a configuration diagram of a control device 100; FIG. 2 is a flowchart for explaining the operation of the control device 100; FIG. 3 is a diagram for explaining an example of a numerical experiment; FIG. 4 is a diagram showing evaluation results; and FIG. 5 is a diagram showing an example of a hardware configuration of the control device 100.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] Hereinafter, an embodiment relating to sleep control of communication devices such as base stations in a wireless network for reducing power consumption while maintaining communication quality will be described. However, the embodiment described below is merely an example, and the network is not limited to a wireless network, and the controlled object is not limited to a base station or an antenna. For example, the controlled object may be an individual server in a data center that has many servers.

[0012] In the following, first, conventional technologies related to the technology according to the present embodiment and their problems will be specifically described, and then the technology according to the present embodiment will be described in detail. Note that the technologies disclosed in the documents mentioned below are publicly known, but the explanations of the problems are not publicly known.

[0013] (Regarding the prior art and its problems) As mentioned above, the technology of Non-Patent Document 1 may cause the base station to sleep excessively, which may degrade the communication quality of users who use communication services in that area.

[0014] In response to this, Li, Rongpeng, et al. "TACT: A transfer actor-critic learning framework for energy saving in cellular radio access networks," IEEE transactions on wireless communications 13.4 (2014): 2000-2011, discloses technology that aims not only to reduce power consumption but also to minimize degradation in communication quality. Furthermore, the paper uses a reinforcement learning approach that can learn from temporal fluctuations in traffic, including these fluctuations, and resolves the long learning time required in deep reinforcement learning by using a transfer learning approach.

[0015] Also, "Ye, Junhong, and Ying-Jun Angela Zhang. "DRAG: Deep reinforcement learning based base station activation in heterogeneous networks." IEEE Transactions on Mobile Computing 19.9 (2019): 2076-2087." Wu, Qiong, et al. "Deep reinforcement learning with spatio-temporal traffic forecasting for data-driven base station sleep control." IEEE / ACM Transactions on Networking 29.2 (2021): 935-948. "Li, Rongpeng, et al. "TACT: A transfer actor-critic learning framework for energy saving in cellular radio access networks." IEEE transactions on wireless communications 13.4 (2014): As with "2000-2011," a deep reinforcement learning approach is used to reduce power consumption and minimize degradation of communication quality. By adding a prediction mechanism for traffic fluctuations, it is possible to improve accuracy in response to fluctuations, demonstrating that dynamic control with high accuracy is possible even compared to conventional rule-based methods.

[0016] Generally, network operators set a target value for communication quality when providing communication services. Therefore, it is desirable to reduce power consumption while achieving this target value. In the deep reinforcement learning approach used in the above-mentioned conventional technology, although the objective function includes a term that minimizes degradation of communication quality, it cannot satisfy the constraint that the quality does not fall below the target value, and therefore cannot be used as is.

[0017] While it is desirable to reduce power consumption while satisfying the constraint of achieving a preset communication quality target, the deep reinforcement learning approach used in conventional technology does not provide a means for inputting a constraint equation. As a result, while the desired power reduction and average quality are maximized, an optimal search cannot be performed to satisfy the constraint, which may result in a degradation of the user's communication quality.

[0018] In this embodiment, the control device 100 applies deep reinforcement learning to minimize the cost and constraint violations regarding the similarity with the proposed control action based on the results of the action (proposed control action) that maximizes the amount of power reduction and average communication quality output by the previous control. This process realizes sleep control of the control target that reduces power consumption while satisfying the target value of communication quality.

[0019] (Device Configuration Example) Fig. 1 shows a configuration example of a control device 100 according to this embodiment. As shown in Fig. 1, the control device 100 includes an objective optimization control unit 110 and a constraint optimization control unit 120. Fig. 1 also shows an environment 200 that is the target of control and observation by the control device 100.

[0020] In this embodiment, it is assumed that the environment 200 is a cellular network equipped with one or more base stations. Note that the objective optimization control unit 110 may be provided outside the control device 100.

[0021] The control device 100 in this embodiment performs sleep control for switching one or more control targets (e.g., base stations or antennas provided in the base stations) between a state in which power consumption can be reduced and an active state. The state in which power consumption can be reduced is, for example, a sleep mode or a state in which the power is turned off.

[0022] The operation of the control device 100 will be outlined with reference to the flowchart shown in FIG.

[0023] In S1 (step 1), the objective optimization control unit 110 calculates and outputs a control action plan that maximizes the amount of power consumption reduction and the average communication quality based on the observed values ​​obtained from the environment 200.

[0024] In S2, the constraint optimization control unit 120 calculates and outputs a control action that minimizes the cost related to the similarity with the control action plan and the violation of the constraint that the communication quality achieves the target value, based on the control action plan output from the objective optimization control unit 110 and the observation value obtained from the environment 200 (e.g., communication quality at the UE).

[0025] The method for realizing the objective optimization control unit 110 and the constraint optimization control unit 120 is not limited to a specific method, but in this embodiment, it is assumed that the objective optimization control unit 110 and the constraint optimization control unit 120 are each a neural network model.

[0026] It is also assumed that the parameters of the above model are updated (optimized) by reinforcement learning. For example, each time an action is taken, the objective optimization control unit 110 and the constraint optimization control unit 120 each retain the state after the action (e.g., activation / deactivation of each antenna), and learn the policy so as to optimize the policy (measure) based on the state. The model parameters correspond to the policy. More specifically, the policy is learned so as to reduce power consumption and ensure that the communication quality at the UE obtained as an observation value satisfies the constraints.

[0027] The control device 100 may learn the policy while operating the control over the environment 200, or may operate the control over the environment 200 using the learned policy.

[0028] The operation of the control device 100 will be described in more detail below. In the following description, "A / B" means "A or B." In addition, in the text of the specification, for convenience of description, normal font is used for characters that represent a set, such as "A," but it is clear from the context that this means a set. In addition, in the text of the specification, a hat (^) intended to be written above a character is written before the character. "^ρ ｉ (t)" is an example.

[0029] (Formulation) Here, we consider a problem of selecting whether to activate or deactivate multiple antennas that provide different frequency bands and are installed at a single base station in a cellular network corresponding to environment 200. Under the control of the base station, there are one or more UEs (User Equipments) that communicate with the base station. The UEs may also be called terminals.

[0030] The antenna set is A ａｌｌ ={1,2,...,A ａｌｌ}, and from the viewpoint of maintaining connectivity, the set of antennas that cannot be stopped is designated as A ｃｏｖｅｒａｇｅ ={1,2,...,A ｃｏｖｅｒａｇｅ}, and the set of antennas that can be stopped is A ｃａｐａｃｉｔｙ ={1,2,...,A ｃａｐａｃｉｔｙ}. This is the antenna set A ｃａｐａｃｉｔｙ The coverage area that can provide service is ｃｏｖｅｒａｇｅ This means that the area is covered by the coverage area of ​​at least one antenna.

[0031] The set of antennas that are not shut down is called A. ｏｎ (∈A ａｌｌ ) and in sleep control, the antenna set A ｏｎ Determine the following. ｃａｐａｃｉｔｙ When the antenna of is stopped, the UE connected to the stopped antenna will ｏｎ The handover is performed to one of the antennas that has a coverage area overlapping with that of the stopped antenna.

[0032] The number of resource blocks (RB) each antenna has is B ｔｏｔａｌ Let RB usage rate of antenna i at time t be ρ ｉ Let (t)∈[0,1].

[0033] Power consumption P of all antennas at time t ｃ is expressed by the following formula:

[0034] Here, P ｉ represents the maximum power consumption of antenna i, and q ｉ∈(0,1) represents the proportion of the maximum power consumption that is consumed in a fixed manner. However, a different power consumption model may be used.

[0035] Antenna set A ａｌｌ Let N = {1, 2, ..., N} be the set of UEs served by the UE. If all antennas are not stopped (if no antennas are stopped) at the time of traffic demand at time t, the power consumption P ａ (t) is expressed by the following formula:

[0036] where ^ρ ｉ (t) is the RB usage rate of antenna i at time t when all antennas are not stopped. The power reduction amount P(t) at time t is P(t) = 1 - P ｃ (t) / P ａ It is expressed as (t).

[0037] Let Q(t) be the communication quality index at time t. Q(t) is a function consisting of one or more quality indexes that should be taken into consideration by a network operator. However, Q(t) may be set so that the larger Q(t) is, the better the communication quality will be, and may be a function expressed as, for example, a weighted linear sum of multiple indexes.

[0038] Here, the quality index is set as the average throughput, and the throughput of the UE at time t is T ｎ (t), the average throughput Q(t) at time t can be expressed by the following equation:

[0039] The set quality target is that the throughput of each UE is T ｔａｒｇｅｔ However, other indicators such as delay may be added as quality indicators or may be substituted, and statistics other than the average may be used.

[0040] (Operations Related to Reinforcement Learning) The objective optimization control unit 110 and the constraint optimization control unit 120 in the control device 100 output a control action plan and a control action, respectively, using a policy learned by reinforcement learning. Here, the design of reinforcement learning in this embodiment will be described.

[0041] Action a at time tｔ The antenna set A that can be stopped is ｃａｐａｃｉｔｙ Let us consider a vector that indicates the antenna state, either active or inactive, at time t. For example, if active is 1 and inactive is 0, then the action a ｔ Is A ｃａｐａｃｉｔｙ It should be noted that if a certain antenna is in an activated state at time t and is also in an activated state at time t+1, and transitions to the same state, no operation is performed.

[0042] State s at time t ｔ Let be a vector that represents the utilization rate of all antennas and the combination of activation / deactivation of all antennas. Also, let be the reward r ｔ is the power reduction amount P(t) and the average throughput Q(t) with the weight parameter w １ In addition, for all UEn, T ｎ ≧T ｔａｒｇｅｔ Based on these, the control device 100 determines the action a at time t+1. ｔ＋１ Here, the problem to be solved is expressed as the following equation.

[0043] where γ∈(0,1] denotes the discount rate for future rewards, and E ｓ，π [*] represents the expected value for the state and policy. s.t. is the constraint mentioned above, and the reward r ｔ As mentioned above, the power reduction amount P(t) and the average throughput Q(t) are calculated by the weight parameter w １ This is the sum of the two.

[0044] For the above problem, the control device 100 applies two-stage reinforcement learning to search for a solution that satisfies the constraints. First, the objective optimization control unit 110 generates a policy π that maximizes only the following objective: ｏ Learn.

[0045] Next, the constraint optimization control unit 120 calculates the policy π ｏ Action a output by ｔ Let ^a be the action that satisfies the constraints. ｔA policy π that converts ｃ The policy π ｏ Action a output by ｔ corresponds to the "control action plan" shown in Figure 1, and the policy π ｃ The action ^a output by ｔ corresponds to the "control action" shown in Figure 1.

[0046] In this embodiment, policy π ｃ We use two cost functions to learn the first cost function, which is the cost of the action a ｔ and action ^a ｔ Cost function C at time t for the similarity between ｓ (t). For example, a distance such as the l-1 norm, the l-2 norm, or the KL divergence may be used as the cost for the similarity, or a nonlinear function for the distance may be used. As the nonlinear function, a function that assigns a larger cost when two actions are farther apart may be used.

[0047] In this embodiment, the preceding policy π ｏ The action output by C is intended to maximize the expected value of the reward, which is the goal. However, if there is a constraint violation, the optimal point that eliminates the constraint violation is found by adding the minimum number of activated antennas. ｓ (t) is the previous policy π ｏ Action a output by ｔ Number of activated antennas in |A ｏｎ For |, the cost is designed as a convex function so that if the change in the number of activated antennas is small, the cost is small, and if the change is large, the cost is large. For example, C ｓ (t) is defined as follows:

[0048] Here, ||*|| １ represents the l-1 norm, and |A ｃａｐａｃｉｔｙ | represents the number of elements in the set. The -1 in the numerator and the -1 in the denominator are ｓ It is used to normalize (t) to 0-1.

[0049] Next, the second cost function will be described. The second cost function is a function that represents the cost related to the constraint violation at time t, and is C ｖ (t). For example, C ｖ As (t), the number of UEs violating the constraint or the amount of violated throughput may be used, or a nonlinear function that increases according to the number of violations may be used. Here, since a strict constraint is imposed on the number of UEs violating the constraint when the number of UEs violating the constraint is 1 or more, a function with a steep slope when the number of UEs violating the constraint changes from 0 to 1 is used. For example, as shown below, using the tanh function, C ｖ Define (t).

[0050] Here, I(*) is an indicator function that takes the value 1 if the condition in the parentheses is met and takes the value 0 if it is not met, and α represents a parameter of the tanh function.

[0051] The two cost functions described above are weighted by the weight parameter w ２ The cost c at time t is calculated by adding ｔ is defined as follows:

[0052] The constraint optimization control unit 120 ｔ Using the following problem, we can solve the policy π ｃ In other words, we learn a policy π ｃ Learn.

[0053] (Numerical Experiment Example) Next, a numerical experiment example will be described. ｃｏｖｅｒａｇｅ |=1, |A ｃａｐａｃｉｔｙ |=2, and as shown in FIG. ｃｏｖｅｒａｇｅ The antennas in set A have a sector-shaped coverage area with a radius of 250 m and a central angle of 120 degrees. ｃａｐａｃｉｔｙ The antenna has a sector-shaped coverage area with a radius of 200 m and a central angle of 120 degrees. Other parameters are set as follows:

[0054] Parameter setting: P ｉ =860[Watt](∀i∈A ａｌｌ ), q ｉ = 0.6 (∀i∈Aａｌｌ ), B ｔｏｔａｌ = 100, P ｉ =860[Watt](∀i∈A ａｌｌ ), T ｔａｒｇｅｔ = 1 [Mbps], α = 0.05, w １ = 1 / 6, w ２ = 0.3, γ = γ ｃ = 0.9 In each scenario, the number of UEs, N, was randomly determined from a uniform distribution between 5 and 250, and the number of UEs within a scenario remained constant. In each scenario, UEs were randomly placed within the coverage area, and each UE downloaded an average of 90 kbits of data following an exponential distribution. The throughput was calculated using the Shannon capacity based on the UE's position using a free-space radio propagation model, with the transmission power of each antenna set to 20 Watt.

[0055] The neural network parameters for reinforcement learning are: the number of layers in the network is 2, the size of the hidden layer is 256, and the learning rate is 5 x 10 －４ The batch size is set to 256, the ε value in the ε-greedy strategy is changed from 0.9 to 0.1, and the number of training episodes is set to 2 × 10 ４ The learning was performed with 20 learning steps.

[0056] Figure 4 shows the results of evaluating 1000 random scenarios under the same conditions as above. For comparison, we compare the ideal case (ES) where an action that maximizes the objective is selected through exhaustive search without violating any constraints, the case where a policy that maximizes only the objective is controlled by reinforcement learning (RL) as in the existing method, and the cost C ｖ The results of the case where (t) is directly included in the objective function (constraint-added RL) are shown. The reward for constraint-added RL is as follows, and w ３ = 1 / 3.

[0057] As evaluation indicators, we calculated the "percentage of scenarios with constraint violations," "scenario average of P(t)," "average UE throughput Q(t)," and "scenario average of standard deviation of UE throughput," regarding the scenario with constraint violations as one or more UEs violating constraints.

[0058] Although the proposed method according to this embodiment has a lower power reduction rate than RL, it is able to reduce constraint violation scenarios to a certain extent, and obtains results close to the ideal state of ES. Furthermore, while constraint-added RL can significantly reduce constraint violations, the amount of power reduction is small, whereas the proposed method can efficiently explore the trade-off between power reduction and compliance with constraints.

[0059] (Hardware Configuration Example) The control device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on a cloud.

[0060] That is, the control device 100 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the control device 100. The program can be recorded on a computer-readable recording medium (such as a portable memory) and can be saved or distributed. The program can also be provided via a network such as the Internet or email.

[0061] Fig. 5 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 5 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected by a bus BS. The computer may further include a GPU.

[0062] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0063] The memory device 1003 reads and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the control device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0064] (Effects of the technology related to the embodiment) As described above, the technology described in this embodiment makes it possible to control the environment in which network services are provided while achieving reduced power consumption and satisfying communication quality constraints.

[0065] More specifically, a control method such as deep reinforcement learning is first used to output an action that maximizes power reduction and average quality, and then a two-stage search is performed using deep reinforcement learning to minimize the cost and constraint violations related to the similarity with the action output in the previous stage, thereby narrowing down solutions that are effective in reducing power consumption and minimizing constraint violations.This control method makes it possible to efficiently explore the trade-off between power reduction and compliance with constraints, making it possible to satisfy communication quality constraints while achieving a certain level of power consumption reduction.

[0066] The following additional notes are provided regarding the above-described embodiments.

[0067] <Additional Notes> (Additional Item 1) A control device for controlling an environment including a plurality of control objects that provide network services, comprising: a memory; and at least one processor connected to the memory, wherein the processor calculates a control action based on a control action plan that maximizes a power consumption reduction amount and average communication quality and observation values ​​from the environment, so as to increase a similarity between the control action plan and the control action plan and reduce a constraint violation of a constraint that communication quality achieves a target value. (Additional Item 2) The control device according to Additional Item 1, further comprising: a goal optimization control unit that calculates the control action plan based on the observation values ​​from the environment. (Additional Item 3) The control device according to Additional Item 1, wherein the processor calculates the control action so as to minimize a cost represented by a first cost for the similarity and a second cost representing the constraint violation. (Additional Item 4) The control device according to Additional Item 3, wherein the processor uses, as a function representing the first cost, a function that is convex with respect to the number of control objects that have changed state from an active state to a sleep state or from a sleep state to an active state among the plurality of control objects. (Supplementary Item 5) The control device according to Supplementary Item 3, wherein the processor uses, as the function representing the second cost, a function with a steep slope when the number of terminals violating the constraint changes from 0 to 1. (Supplementary Item 6) A control action calculation method executed by a control device for controlling an environment including a plurality of control objects that provide network services, the control action calculation method comprising: calculating a control action based on a control action plan that maximizes a power consumption reduction amount and average communication quality and observed values ​​from the environment, so as to increase the similarity between the control action plan and the proposed control action, and to reduce violations of the constraint that communication quality achieves a target value. (Supplementary Item 7) A non-transitory storage medium storing a program for causing a computer to function as each unit in the control device according to any one of Supplementary Item 1 to 5.

[0068] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0069] REFERENCE SIGNS LIST 100 Control device 110 Objective optimization control unit 120 Constraint optimization control unit 200 Environment 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A control device for controlling an environment having multiple control targets that provide network services, comprising a constraint optimization control unit that calculates control actions based on control action plans that maximize power consumption reduction and average communication quality and observed values ​​from the environment, so that the similarity between the control action plans and observed values ​​is increased and violations of the constraint that communication quality achieves a target value are reduced.

2. The control device according to claim 1, further comprising a goal optimization control unit that calculates the control action plan based on observed values ​​from the environment.

3. The control device according to claim 1, wherein the constraint optimization control unit calculates the control action so as to minimize a cost represented by a first cost relating to the similarity and a second cost representing the constraint violation.

4. The control device described in claim 3, wherein the constraint optimization control unit uses, as a function representing the first cost, a function that is a convex function with respect to the number of control objects that have changed state from an active state to a sleep state or from a sleep state to an active state among the plurality of control objects.

5. The control device according to claim 3, wherein the constraint optimization control unit uses a function that has a steep slope when the number of terminals violating the constraint changes from 0 to 1 as a function representing the second cost.

6. A control action calculation method executed by a control device for controlling an environment having multiple control targets that provide network services, the control action calculation method calculating a control action based on a control action plan that maximizes the amount of power consumption reduction and average communication quality and observed values ​​from the environment, so as to increase the similarity between the control action plan and the control action plan, and reduce violations of the constraint that communication quality achieves a target value.

7. A program for causing a computer to function as each part of the control device according to any one of claims 1 to 5.

Images