Training device, training method, and program

The learning device uses multi-agent reinforcement learning to coordinate multiple controls in cellular networks by having agents observe and act sequentially, addressing the instability of simultaneous control execution and improving network performance and resource utilization.

WO2024241542A1PCT designated stage expired Publication Date: 2024-11-28NT T INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/019381
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing methods for controlling antenna adjustment, power control, and resource allocation in cellular networks struggle to coordinate multiple controls simultaneously, leading to instability and decreased quality of service due to conflicting objectives and environmental changes.

Method used

A learning device employing multi-agent reinforcement learning coordinates coverage control, power control, and resource allocation control by having each agent observe the state of all controls and act one by one, reducing environmental instability and improving coordination among agents.

Benefits of technology

This approach stabilizes learning, minimizes uncovered or redundantly covered areas, reduces power consumption, and enhances the achievement of quality requirements for each slice, while optimizing user throughput and resource usage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023019381_28112024_PF_FP_ABST
    Figure JP2023019381_28112024_PF_FP_ABST
Patent Text Reader

Abstract

This training device trains a plurality of control agents for performing a plurality of controls on an environment, the training device comprising: an information collection unit that collects observed values obtained from the environment, reward values obtained according to the state of the environment, and actions output from the plurality of control agents; and a training unit that trains the plurality of control agents on the basis of information obtained by the information collection unit, while causing the plurality of control agents to act sequentially one by one.
Need to check novelty before this filing date? Find Prior Art

Description

Learning device, learning method, and program

[0001] The present invention relates to the control of radio sections for data transmission in cellular networks.

[0002] In recent cellular networks, the rapid increase in traffic demand has led to constant demand for improved performance. To meet these demands, various control mechanisms are simultaneously implemented in the wireless sections.

[0003] For example, by controlling the coverage area by adjusting the antenna tilt of the base station, it is possible to improve throughput through load balancing. Also, by controlling the data transmission power from the base station, it is possible to achieve an optimization that balances the improvement of throughput and the reduction of power consumption, taking into account factors such as interference.

[0004] Furthermore, by controlling the allocation of wireless resources to each slice in network slicing technology, it is expected that the required quality set for each slice can be achieved and wireless resource utilization can be improved.

[0005] Vannella, Filippo, et al. "Remote electrical tilt optimization via safe reinforcement learning." 2021 IEEE Wireless Communications and Networking Conference (WCNC). IEEE, 2021.C. Wang, D. Deng, L. Xu, W. Wang and F. Gao, "Joint Interference Alignment and Power Control for Dense Networks via Deep Reinforcement Learning," in IEEE Wireless Communications Letters, vol. 10, no. 5, pp. 966-970, May 2021, doi: 10.1109 / LWC.2021.3052079.Y. Abiko, T. Saito, D. Ikeda, K. Ohta, T. Mizuno and H. Mineno, "Flexible Resource Block Allocation to Multiple Slices for Radio Access Network Slicing Using Deep Reinforcement Learning," in IEEE Access, vol. 8, pp. 68183-68198, 2020, doi: 10.1109 / ACCESS.2020.2986050

[0006] Because user demand changes constantly, wireless section control requires a method that can adapt to environmental changes. Existing methods have proposed standalone controls that can adapt to environmental changes, such as a method for optimizing antenna adjustment (Non-Patent Document 1), a power control method (Non-Patent Document 2), and a method for optimizing resource allocation using network slicing (Non-Patent Document 3). However, no consideration has been given to how to coordinate these methods when they are used simultaneously. In particular, there is a problem in that it is difficult to properly learn multiple controls.

[0007] The present invention has been made in view of the above points, and has an object to provide a technology for appropriately training a plurality of control agents that perform a plurality of controls on an environment.

[0008] According to the disclosed technology, there is provided a learning device that learns a plurality of control agents to perform a plurality of controls on an environment, the learning device comprising: an information collection unit that collects observation values ​​obtained from the environment, reward values ​​obtained according to the state of the environment, and actions output from the plurality of control agents; and a learning unit that makes the plurality of control agents act one by one in turn, and learns the plurality of control agents based on the information obtained by the information collection unit.

[0009] The disclosed technology provides a technology for appropriately training multiple control agents that perform multiple controls over an environment.

[0010] It is a diagram showing an example of the overall configuration of a communication system. It is a diagram showing an image of multi-agent reinforcement learning. It is a diagram showing an example of the configuration of a control collaboration device 100. It is a diagram showing an example of the configuration of a learning device 400. It is a flowchart showing the operation of the learning device 400. It is a diagram showing an example of the hardware configuration of the device.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0012] Below, we will explain an example of data transmission in a cellular network in which control of the coverage area by adjusting antenna tilt, control of data transmission power from a base station, and control of resource allocation to multiple network slices are all performed together.

[0013] Note that "control of a coverage area by adjusting antenna tilt, etc.", "control of data transmission power from a base station," and "control of resource allocation to multiple network slices" are examples of multiple controls in a wireless section. The technology according to the present invention is applicable to multiple controls not limited to the above three controls.

[0014] (Regarding Issues) First, issues relating to the technology according to the present embodiment will be described in detail.

[0015] When multiple controls are attempted to be executed simultaneously in a wireless section, if they are executed independently, they may conflict with each other, hindering the achievement of their respective objectives, or failing to find the most optimal state.

[0016] For example, in the slice control described in Non-Patent Document 3, closed control is implemented within one base station, and when combined with control that affects base stations, such as coverage control, the users connected to the base station will change due to the coverage control. As a result, the state of the control target that serves as the input will not be stable, making learning difficult and possibly reducing the rate at which the slice achieves the required quality.

[0017] Therefore, when multiple controls that affect each other's state of the controlled objects are executed simultaneously, the environment after the control becomes unstable, making it difficult to learn simultaneously.

[0018] (Outline of the embodiment) In the present embodiment, a control collaboration device 100 (to be described later) performs multi-agent reinforcement learning by using coverage control, power control, and resource allocation control for network slices as agents.

[0019] The control collaboration device 100 coordinates three types of control, namely, coverage control, power control, and resource allocation control, and simultaneously performs these three types of control. As described above, these controls affect each other. For example, coverage control changes the users connected to the base station, which also affects the execution of power control and resource allocation control for network slices. Therefore, simply learning while simultaneously operating these controls makes learning difficult due to the instability of environmental changes caused by the influence of each control.

[0020] Therefore, in this embodiment, each agent observes the state of all controls, thereby sharing the state, and by having each agent take turns acting during learning, the instability of environmental changes is reduced, and by learning the effects of the controls of other agents, cooperation between agents is promoted.

[0021] (System Configuration Example) Fig. 1 shows an example of the overall configuration of a communication system according to this embodiment. As shown in Fig. 1, the communication system according to this embodiment has a configuration in which a plurality of base stations 10 are connected to a network 200. Each base station 10 has zero or more UEs (User Equipment) 20 under its control. A UE may also be called a terminal. A UE may also be called a user.

[0022] The network 200 is, for example, a core network in a mobile communication system. Each base station 10 wirelessly communicates with a UE 20. Since each base station 10 forms a cell as a wireless communication area, the network between the base station 10 and the UE 20 may be called a cellular network.

[0023] A control collaboration device 100 is connected to the network 200. The control collaboration device 100 is capable of acquiring information about each base station 10 via the network 200 and controlling each base station 10 via the network 200. In the framework of multi-agent reinforcement learning, the cellular network (including the base stations 10) that is the object of control may be called the "environment."

[0024] In the description of the operation of the control collaboration device 100, the environment having the base station 10 and the UE 20 as shown in FIG. 1 may be an actual environment or a system model on a computer.

[0025] (Regarding Multi-Agent Reinforcement Learning) Fig. 2 shows an image of multi-agent reinforcement learning in this embodiment. As shown in Fig. 2, there are agents A, B, and C. For example, agent A performs an action on the environment to "control the coverage area by adjusting antenna tilt, etc.", agent B performs an action on the environment to "control data transmission power from a base station," and agent C performs an action on the environment to "control resource allocation to multiple network slices."

[0026] Agent A, Agent B, and Agent C each take turns executing an action. Each agent observes (obtains) the state of the environment after executing the action. Each agent is also given a reward according to that state.

[0027] Each agent repeatedly executes its actions, resulting in time-series data of the state, action, and reward. Based on this time-series data, a policy for all agents is determined (learned) that maximizes the objective function (e.g., expected profit). Each agent can take optimal actions by using the learned policy.

[0028] The implementation method of each agent is not limited to a specific method, but it can be implemented as a neural network, for example. If each agent is a neural network, the learned policy described above corresponds to the optimal parameters of the neural network. Note that the agent may also be called a "control agent" or a "control agent model."

[0029] (Configuration Example of Control Collaboration Device 100) Fig. 3 shows a configuration example of the control collaboration device 100. Fig. 3 also shows an environment 300 that is the control target of the control collaboration device 100.

[0030] As shown in FIG. 3 , the control collaboration device 100 includes an environment observation unit 110 , a reward calculation unit 120 , a control recording unit 130 , an information aggregation unit 140 , a learning unit 150 , and a test unit 160 .

[0031] (Operation Example 1) In a cellular network having a plurality of base stations 10, the control collaboration device 100 is a device for simultaneously and cooperatively executing the following controls: control of changing a coverage area by changing antenna tilt, etc.; control of the total power for the base stations 10 to transmit data to UEs 20; and control of allocating multiple divided radio resources held by the base stations 10 to multiple network slices prepared by network slicing technology. The operation of each unit is as follows.

[0032] The environment observation unit 110 collects, at regular intervals from the environment 300 that is the object of control, the number of UEs per network slice connected to each base station 20, the average throughput per network slice connected to each base station 20, and indicators of the controlled objects of each control, such as the antenna tilt angle, the transmission power value of the base station 20, and the radio resource allocation value.

[0033] Each of the above values ​​may be called a state. These states change depending on the behavior output from the controlling agent model.

[0034] The reward calculation unit 120 calculates a reward value by setting a priority for each objective based on the operation objectives, such as improving user throughput, improving the rate of achievement of requests set in each network slice, improving power reduction rate, improving resource utilization rate, and reducing the size of areas without coverage or the size of overlapping areas due to antenna tilt settings.

[0035] The control recording unit 130 records, at regular time intervals, the actions of each control output from a plurality of control agent models prepared in the same number as the number of controls.

[0036] The information collection unit 140 aggregates (collects) the information obtained by the environment observation unit 110 , the information obtained by the remuneration calculation unit 120 , and the information obtained by the control recording unit 130 .

[0037] The learning unit 150 learns a control agent model based on the information collected by the information collecting unit 140, with each control as a control agent, but instead of making a plurality of controls act simultaneously, it makes them act one by one in order and learns all the control agent models simultaneously.

[0038] The test unit 160 has a learned control agent model that receives input of information such as user usage status collected from the environment 300 at a certain point in time by the environment observation unit 110 and outputs the behavior of each control.

[0039] (Operation Example 2) As described above, the multiple controls performed by the control collaboration device 100 are not limited to the three of "control of the coverage area by adjusting the antenna tilt, etc.", "control of data transmission power from the base station," and "control of resource allocation to multiple network slices." An operation example when not limited to a specific control is as follows.

[0040] In the second operational example, the control collaboration device 100 is a device for simultaneously executing collaboration control of a plurality of wireless sections in a cellular network having a plurality of base stations 10 .

[0041] The environment observation unit 110 collects information such as user usage status from the environment 300 to be controlled at regular time intervals.

[0042] The reward calculation unit 120 calculates a reward value set in accordance with a predetermined operational goal based on observed values ​​from the environment 300, which is the control target, at regular time intervals.

[0043] The control recording unit 130 records, at regular time intervals, the actions of each control output from a plurality of control agent models prepared in accordance with the number of controls. The information collecting unit 140 aggregates (collects) the information obtained by the environment observing unit 110, the information obtained by the reward calculating unit 120, and the information obtained by the control recording unit 130.

[0044] The learning unit 150 learns a control agent model with each control as a control agent based on the information collected by the information collecting unit 140. During learning, the learning unit 150 does not make multiple agents act simultaneously, but makes each agent act one by one in order, thereby learning all the control agent models simultaneously.

[0045] The test unit 160 has a learned control agent model that receives input of information such as the user's usage status at a certain point in time observed from the environment 300 by the environment observation unit 110, and outputs the behavior of each control.

[0046] (Details of Operation of Control Collaboration Device 100) The following describes in more detail an example of operation of the control collaboration device 100. Here, it is assumed that OFDMA is used for downlink in a cellular network.

[0047] In downlink OFDMA, the set of base stations is B={1, 2, ..., B}, and the set of UEs is N={1, 2, ..., N}. In coverage control, the area covered by each base station 10 is set, and the connected users change accordingly. In power control, the total data transmission power of each base station is set as P T As, 0 to P max In resource allocation control for network slicing, each base station allocates a resource block (RB) set R={1, 2, ..., R total} and this R total The number of RBs to be allocated to each slice is determined.

[0048] Here, the allocation indicator α is set to 1 when a base station b allocates RB r (resource block r) to user n, and 0 when it does not allocate RB r. b,r,n This value is determined by coverage control. The SINR of user n when base station b allocates resource block r to user n is then expressed as ζ b,r,n Then, the following formula holds:

[0049] where p b,rrepresents the power allocated to RB r by base station b, and R total The total power allocated to each RB is P T Also, g b,n Let β be the channel gain between base station b and user n, and N0 be the noise. In this case, the capacity of user n is as follows:

[0050] Here, there are three types of slices, which are eMBB type, URLLC type, and best effort (BE) type. The number of RBs allocated by base station b to these three types of slices is respectively R b, eMBB , R b, URLLC , R b, BE These values ​​are set for each base station, and the sum of these is R total Here, the requirement for eMBB type slices is that the capacity is C eMBB Furthermore, the URLLC type slice request is such that the transmission delay d, defined as the requested packet size divided by the capacity C, is equal to or greater than d URLLC Let us assume that:

[0051] Here, we aim to use three types of control to minimize areas not covered by base stations and areas with overlapping coverage, and to reduce power consumption, while also increasing the proportion of users that meet the quality requirements of eMBB slices and URLLC slices as much as possible, and for the BE type, to control so that the average throughput of all users is as high as possible.

[0052] The learning unit 150 in the control collaboration device 100 sets coverage control, transmission power control, and slice resource allocation control as agents, and performs learning within a multi-agent reinforcement learning framework, and the testing unit 160 executes control using multiple trained agents.

[0053] Here, the action vectors of the agents for coverage control, transmission power control, and slice resource allocation control are respectively denoted as A c = (a c,1 , a c,2 , …, ac,B ), A p = (a p,1 , a p,2 , …, a p,B ), A s =(a s,1 , a s,2 , …, a s,B ) a c, b represents the change in the antenna tilt of base station b, or a vector with elements equal to the number of antennas if there are multiple antennas, and this change increases or decreases the coverage area. p, b represents the amount of change in the transmission power of base station b.

[0054] a s, b is a s, b =(a s,b,eMBB , a s,b,URLLC , a s,b,BE ) and represents the vector of the amount of change in the number of RBs for each slice of base station b.

[0055] Although the amount of change is used in each of the above-described actions, the value itself, such as the angle of antenna tilt or transmission power, may also be used instead of the amount of change.

[0056] The state observed by the environment observation unit 110 corresponds to the state observed by each agent. Here, the state observed by each agent is a vector common to all agents, which includes the number of users belonging to each slice at each base station, the average throughput of users belonging to each slice at each base station, and the current value of each control (e.g., the antenna tilt angle at each base station, the transmission power at each base station, and the number of RBs allocated to each slice at each base station).

[0057] The reward function used by the reward calculation unit 120 is also common to each agent and is set, for example, as follows:

[0058] R total =w slice R slice +w coverage R coverage +w power R power R slice =w embb R embb +w urllc Rurllc +w BE R BE where w slice , w coverage , w power , w embb , w urllc , w BE is the predetermined priority of each reward. embb , R urllc For example, let us set the percentage of users who have achieved the eMBB slice and URLLC slice requirements, respectively, and R BE is set to a value proportional to the average throughput of users belonging to the BE slice. coverage For example, R is set to a value that decreases as the size of the area not covered by the base station or the size of the overlapping covered area increases. power For example, the smaller the total power of all base stations, the larger the value is set.

[0059] The learning unit 150 uses the above settings to perform learning using a multi-agent reinforcement learning algorithm. In this case, when exploratory actions are performed to acquire learning data (e.g., time-series data of states, actions, and rewards), the agents are made to act in sequence, rather than all at the same time. For example, when an agent for coverage control performs some action, action A for power control and resource allocation control for slices is performed. p , A s The change amount of is set to 0 so that a vector having an element of 0 is selected.

[0060] The test unit 160 uses each of the trained agents trained in the above-described manner to simultaneously perform multiple controls in the execution phase.

[0061] (Other Configuration Examples) The control collaboration device 100 may be called a learning device. Furthermore, the control collaboration device 100 may not include the test unit 160. For example, the control agent model learned by the control collaboration device 100 may be sent to another device, and a test unit in the other device may execute control using the control agent model.

[0062] The learning device 400 in this embodiment may also have the configuration shown in Fig. 4. The learning device 400 is a device that learns a plurality of control agents for performing a plurality of controls on the environment.

[0063] 4, the learning device 400 includes an information collection unit 410 and a learning unit 420. The learning device 400 may also include a testing unit.

[0064] The information collection unit 410 collects observed values ​​obtained from the environment, reward values ​​obtained according to the state of the environment, and actions output from a plurality of control agents.

[0065] The learning unit 420 causes the control agents to act one by one in turn, and learns the control agents based on the information obtained by the information collecting unit 410 .

[0066] 5 is a flowchart showing the operation of the learning device 400. In S101, the learning device 400 collects observation values ​​obtained from the environment, reward values ​​obtained according to the state of the environment, and actions output from the multiple control agents.

[0067] In S102, the learning device 400 causes the control agents to act one by one in turn, and learns the control agents based on the information obtained in S101.

[0068] (Hardware Configuration Example) Any of the devices described in this embodiment (the control collaboration device 100 and the learning device 400) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0069] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0070] Fig. 6 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 6 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected by a bus BS. The computer may further include a GPU.

[0071] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0072] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0073] (Summary, Effects, etc. of the Embodiment) As described above, in the present embodiment, all control states are observed and shared by each agent, and cooperation between agents (control) is achieved by each agent taking turns acting during learning. The cooperation of control stabilizes learning without interfering with each other's control objectives, and it is possible to realize control that minimizes areas not covered by base stations and areas with overlapping coverage, reduces power consumption, and improves the rate at which quality requirements for each slice are achieved, in a short learning time.

[0074] The following additional notes are provided regarding the above-described embodiments.

[0075] <Additional Notes> (Additional Item 1) A learning device that learns a plurality of control agents for performing a plurality of controls on an environment, comprising: a memory; and at least one processor connected to the memory, wherein the processor collects observation values ​​obtained from the environment, reward values ​​obtained according to the state of the environment, and actions output from the plurality of control agents, and learns the plurality of control agents based on the information obtained by the information collecting unit while causing the plurality of control agents to act one by one in sequence. (Additional Item 2) The learning device according to Additional Item 1, wherein the environment is a cellular network having a plurality of base stations, and the plurality of control agents include at least one of a control agent that performs coverage control, a control agent that performs transmission power control, and a control agent that performs slice-oriented resource allocation control. (Additional Item 3) The learning device according to Additional Item 1 or 2, wherein the observation values ​​and the reward values ​​are common to the plurality of control agents. (Supplementary Item 4) The learning device according to any one of Supplementary Items 1 to 3, wherein, when causing one control agent to select an action, the processor causes other control agents to select actions that result in a change of 0. (Supplementary Item 5) A learning method executed by a learning device that trains multiple control agents to perform multiple controls on an environment, the learning method comprising: an information gathering step of collecting observation values ​​obtained from the environment, reward values ​​obtained according to the state of the environment, and actions output from the multiple control agents; and a learning step of learning the multiple control agents based on the information obtained in the information gathering step while causing the multiple control agents to act one by one in turn. (Supplementary Item 6) A non-transitory storage medium storing a program for causing a computer to function as each unit in the learning device according to any one of Supplementary Items 1 to 4.

[0076] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0077] 10 Base station 20 UE 100 Control collaboration device 110 Environment observation unit 120 Reward calculation unit 130 Control recording unit 140 Information aggregation unit 150 Learning unit 160 Test unit 200 Network 300 Environment 400 Learning device 410 Information collection unit 420 Learning unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A learning device that learns a plurality of control agents to perform a plurality of controls on an environment, comprising: an information collection unit that collects observation values ​​obtained from the environment, reward values ​​obtained according to the state of the environment, and actions output from the plurality of control agents; and a learning unit that learns the plurality of control agents based on the information obtained by the information collection unit while making the plurality of control agents act one by one in sequence.

2. The learning device of claim 1, wherein the environment is a cellular network having a plurality of base stations, and the plurality of control agents include at least one of a control agent that performs coverage control, a control agent that performs transmission power control, and a control agent that performs resource allocation control for slices.

3. The learning device according to claim 1, wherein the observation value and the reward value are common to the multiple control agents.

4. The learning device according to claim 1, wherein, when the learning section causes one control agent to select an action, the learning section causes other control agents to select actions that result in a change of zero.

5. A learning method executed by a learning device that learns a plurality of control agents to perform a plurality of controls on an environment, comprising: an information collection step of collecting observation values ​​obtained from the environment, reward values ​​obtained according to the state of the environment, and actions output from the plurality of control agents; and a learning step of learning the plurality of control agents based on the information obtained by the information collection step while making the plurality of control agents act one by one in sequence.

6. A program for causing a computer to function as each unit in the learning device according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Parameter setting device, computer program, and parameter setting method

    JP2013026980A

  • Parameter setting device, computer program, and parameter setting method

    JP2013106202A

  • Reinforcement learning device

    JP2019101979A

  • Radio access network control with deep reinforcement learning

    US20230095706A1