Aggregation device, communication system, processing method, and program

Abstract

To provide a technique capable of optimizing line utilization in accordance with a radio environment even when the radio environment is changed.SOLUTION: The aggregation device includes a detection unit configured to determine whether or not an overall reward in the reinforcement learning satisfies a predetermined condition, or whether or not a path loss between a radio base station and a radio terminal satisfies a predetermined condition, and a connection relation change unit configured to change a connection relation between the radio base station and the radio terminal when the detection unit determines that the predetermined condition is satisfied.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to packet scheduling in wireless communication systems. [Background technology]

[0002] Currently, wireless communication systems have evolved to become heterogeneous networks based on multi-band, multi-access systems. In cellular communications, the fifth generation mobile communications (5G) has been put into practical use, and a wide range of frequencies, from sub-1 GHz to millimeter wave bands, are being used, and we are entering a world where cells of various sizes, from small cells to macrocells, are provided in an overlapping manner.

[0003] Wireless LAN, another typical wireless access system, also uses radio frequencies in the 2.4 / 5 / 6 / 60 GHz bands. Wireless devices such as smartphones generally have wireless interfaces (I / F) that support both cellular and wireless LAN access, and each I / F can support multiple bands. It is becoming common for wireless devices to select a wireless base station to connect to from multiple frequencies and multiple access methods and communicate.

[0004] In such an environment, there is a method for optimizing the use of multiple lines and maximizing performance in terms of large capacity and high reliability. For example, for high reliability, there is a method described in Patent Document 1 that uses reinforcement learning (RL, e.g., Q-learning, DQN: Deep Q-Network) to optimize the lines used to send packets more reliably. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-117803 Summary of the Invention [Problem to be solved by the invention]

[0006] The technology disclosed in Patent Document 1 makes it possible for each wireless device to autonomously select the optimal I / F to use for packet transmission. However, it is not possible to reassess the connection relationship between the wireless base station and the wireless terminal when there is a change in the wireless environment, and therefore it is not always possible to maximize performance.

[0007] The present invention has been made in consideration of the above points, and aims to provide a technology that enables optimization of line usage in accordance with the wireless environment, even when there is a change in the wireless environment. [Means for solving the problem]

[0008] According to the disclosed technology, there is provided an aggregation device that accommodates a plurality of radio base stations that perform scheduling using reinforcement learning, the aggregation device comprising: a detection unit that determines whether or not an overall reward in the reinforcement learning satisfies a predetermined condition, or whether or not a path loss between a wireless base station and a wireless terminal satisfies a predetermined condition; a connection relationship changing unit that changes a connection relationship between a wireless base station and a wireless terminal when the detection unit determines that the predetermined condition is satisfied; An aggregation device is provided, comprising: [Effects of the Invention]

[0009] According to the disclosed technology, it is possible to improve the speed and accuracy of convergence in learning in a communication device that performs scheduling using reinforcement learning. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of the configuration of a communication system. [Figure 2] FIG. 1 is a diagram illustrating the configuration of a wireless base station (or a wireless terminal). [Figure 3] FIG. 1 is a diagram illustrating the configuration of a wireless base station (or a wireless terminal). [Figure 4]10 is a flowchart showing an outline of an operation. [Figure 5] FIG. 2 is a configuration diagram of an aggregation device. [Figure 6] FIG. 10 is a sequence diagram for explaining operations related to the aggregation device. [Figure 7] FIG. 10 is a diagram illustrating an example of calculation of total remuneration. [Figure 8] 10 is a flowchart of an operation including review of connection relationships. [Figure 9] 10 is a flowchart (example 1) showing detailed operations for reviewing connection relationships. [Figure 10] FIG. 10 is a diagram for explaining a specific example of reviewing connection relationships. [Figure 11] 10 is a flowchart (example 2) showing detailed operations for reviewing connection relationships. [Figure 12] FIG. 1 is a diagram for explaining a system model. [Figure 13] FIG. 1 is a diagram illustrating reinforcement learning. [Figure 14] FIG. 1 illustrates Algorithm 1. [Figure 15] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0012] (System configuration example) Fig. 1 shows an example of the configuration of a communication system according to the present embodiment. As shown in Fig. 1, the communication system includes a plurality of radio base stations 100, a plurality of radio terminals 200, and an aggregation device 300. In the example of Fig. 1, the aggregation device 300 is connected to the Internet.

[0013] In addition, in this embodiment, dual connectivity or the like is used to allow one wireless terminal to use a plurality of wireless base stations in an integrated manner, and traffic management of the plurality of wireless base stations is performed.

[0014] 1, the radio base station 100 is connected to the aggregation device 300, but the radio terminal 200 may be connected to the aggregation device 300. Alternatively, both the radio base station 100 and the radio terminal 200 may be connected to the aggregation device 300.

[0015] In this embodiment, a wireless base station 100 having a plurality of wireless interfaces determines, for packets to be transmitted to a device (wireless terminal), the wireless interface through which the packets will be transmitted and the number of packets to be transmitted through the wireless interface, using a reinforcement learning technique to be described later. Note that the determination of the wireless interface and the determination of the number of packets may be called scheduling.

[0016] The technique according to this embodiment can also be applied to wireless terminal 200. The wireless base station and wireless terminal may be collectively referred to as a communication device. The operation of aggregation device 300 will be described later.

[0017] In addition, in the specific examples described below, two types of wireless interfaces, Sub-6 GHz and mmWave, are described, but the wireless interface is not limited to these. Furthermore, the "wireless interface" may be interpreted as a "frequency." In other words, in this embodiment, in a form in which multiple frequencies are aggregated and used, frequency selection and packet number determination can be achieved by a reinforcement learning technique described below.

[0018] 2 shows an example of the configuration of the radio base station 100. The radio terminal 200 may also have the same configuration as that shown in FIG.

[0019] As shown in FIG. 2, the wireless base station 100 includes a communication I / F unit 110, a control unit 120, a wireless communication unit 130, and an antenna 101.

[0020] The wireless communication unit 130 includes a scheduler unit 140, a receiving unit 131, a wireless communication signal generation unit 132, and an RF unit 135. The scheduler unit 140 includes a reinforcement learning unit 150, a terminal association control unit 141, a communication quality measurement unit 142, a total wireless resource allocation amount calculation unit 143, and an individual wireless resource allocation amount calculation unit 144. The number of "individual wireless resource allocation amount calculation units 144, receiving units 131, wireless communication signal generation units 132, RF units 135, and antennas 101" is equal to the number of wireless interfaces. However, any one of the "individual wireless resource allocation amount calculation units 144, receiving units 131, wireless communication signal generation units 132, RF units 135, and antennas 101" may be shared by multiple interfaces. Furthermore, the "individual wireless resource allocation amount calculation units 144, receiving units 131, wireless communication signal generation units 132, RF units 135, and antennas 101" may be referred to as "wireless interfaces."

[0021] The reinforcement learning unit 150 includes a Q table management unit 151, a state calculation unit 152, a reward calculation unit 153, and a risk evaluation unit 154. The operation of each unit is as follows.

[0022] The communication I / F unit 110 communicates with the aggregation device 300. The control unit 120 includes, for example, a CPU and a memory, and controls the entire device. The wireless communication unit 130 executes operations related to wireless communication.

[0023] The scheduler unit 140 performs packet scheduling and the like. The receiver unit 131 receives signals from other communication devices (e.g., feedback from a wireless terminal) via an antenna and RF unit. The wireless communication signal generator unit 132 generates signals to be transmitted wirelessly from the data of the packets to be transmitted. The RF unit 135 performs processing such as placing the signals on carrier waves. The scheduler unit 140 can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0024] The terminal association control unit 141 controls the association destination (which may also be referred to as the connection destination) of the wireless terminal. The processing operation of the terminal association control unit 141 will be described later. The communication quality measurement unit 142 measures the communication quality between the wireless base station 100 and the wireless terminal 200. In this embodiment, the communication quality measurement unit 142 measures the path loss as the communication quality. The communication quality measurement unit 142 also measures the packet loss rate based on, for example, the number of transmitted packets and feedback (e.g., ACK / NACK) from the communication partner. Note that this embodiment also supports a situation where instantaneous CSI feedback (ACK / NACK, etc.) cannot be obtained from each device and sporadic CSI feedback is obtained, and it is possible to acquire communication quality statistics (such as an average value across all devices) from the sporadic CSI feedback.

[0025] The total radio resource allocation amount calculation unit 143 determines the amount to be allocated to each radio interface with respect to the total number of packets to be transmitted, based on the behavior determined for each frame by the reinforcement learning unit 150. Furthermore, the individual radio resource allocation amount calculation unit 144 determines the amount of radio resources corresponding to the number of packets to be transmitted in the relevant radio interface (the radio interface connected to the individual radio resource allocation amount calculation unit 144), based on the behavior determined for each frame by the reinforcement learning unit 150.

[0026] The wireless base station 100 (or the wireless terminal 200) can also be represented by the configuration shown in Fig. 3. As shown in Fig. 3, the wireless base station 100 has a reinforcement learning unit 10, a transmitter 20, and a receiver 30. The reinforcement learning unit 10 performs the same processing as the reinforcement learning unit 150. The transmitter 20 performs processing related to transmission (e.g., calculation of transmission resource allocation, packet transmission), and the receiver 30 performs processing related to reception (e.g., feedback reception, communication quality calculation).

[0027] (About the reinforcement learning unit 150) In this embodiment, a configuration is adopted in which the radio base station 100 (or the radio terminal 200) aggregates a plurality of radio interfaces (or a plurality of frequencies).

[0028] The scheduler unit 140, which allocates wireless resources to packets transmitted from each wireless interface, is equipped with a reinforcement learning unit 150, which applies reinforcement learning (1) to autonomously learn and perform optimal connections to achieve the desired communication quality. Furthermore, a risk-averse learning method based on parallel updating of multiple Q tables (including single Q tables) (THL Dinh, M. Kaneko, K. Kawamura, T. Moriyama and Y. Takatori, "Improving Reliability by Risk-Averse Reinforcement Learning over Sub6GHz / mmWave Integrated Networks," ICC 2022 - IEEE International Conference on Communications, 2022, pp. 3178-3183, doi: 10.1109 / ICC45855.2022.9839175) (2) is used to enable behavioral selection that prioritizes communication reliability.

[0029] Regarding the application of reinforcement learning (1) above, in this embodiment, the state s(t) is the satisfaction level of each wireless terminal based on the packet loss rate (detected from ACK feedback) information of each wireless terminal on each wireless interface, and the action a(t) is the combination of wireless interfaces used for each device (wireless terminal if the source is a wireless base station) and packet scheduling (the number of packets transmitted on each wireless interface). In this embodiment, the Risk-Averse Average Q-learning method (Y. Gao, KY Lui, P. Hernandez-Leal, "Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets," RL4RealLife Workshop in International Conference on Machine Learning (ICML), 2021) is utilized to learn the optimal action a(t) for each device from the state s(t).

[0030] In this embodiment, it is assumed that uRLLC will be used, for example. In that case, it is considered that instantaneous CSI feedback cannot be used to maintain low latency. In this embodiment, the radio interface selection and packet scheduling method are designed to enable good risk-averse learning even when the instantaneous channel state is unknown.

[0031] Regarding the risk-averse learning method in (2) above, a term that quickly responds to the variance (risk) of cumulative reward sumr is included in the equations (Equations (11) and (12) described below) that show the concept of the evaluation function that responds to risk (magnitude of variance) in risk-averse learning, thereby reflecting the reduction in reward for high-risk behavior. The term that responds to the variance of cumulative reward sumr and reflects it in the evaluation is the second term (the term with Var) in Equation (12) described below (the equation obtained by Taylor expansion of Equation (11)).

[0032] As will be explained in detail later, in this embodiment, the instantaneous reward reflects the average packet reception success rate across all devices and the penalty due to a risk state (e.g., a state in which QoS targets such as reliability and delay are not achieved).

[0033] In this embodiment, the aggregation device 300 calculates a total reward as the reward based on feedback from the plurality of radio base stations 100 (or the plurality of radio terminals 200), and transmits the calculated total reward to the plurality of radio base stations 100 (or the plurality of radio terminals 200). In this embodiment, the aggregation device 300 reviews the connection relationship between the radio base stations 100 and the radio terminals 200 in response to changes in the radio environment.

[0034] In the reinforcement learning unit 150 shown in FIG. 2, the Q table management unit 151 holds, initializes, updates, etc. the Q table. The state calculation unit 152 calculates the state s(t). The reward calculation unit 153 transmits feedback information to the aggregation device 300 and receives a total reward from the aggregation device 300. Note that the reward calculation unit 153 can also calculate the reward r for s(t) and a(t) by itself. The risk assessment unit 154 calculates an evaluation function based on the Q table and selects an action. Note that the calculation of the evaluation function may be performed by the reward calculation unit 153.

[0035] Here, an outline of the operation of the radio base station 100 related to reinforcement learning will be described with reference to the flowchart of FIG.

[0036] In S101, the state calculation unit 152 acquires the satisfaction level of each wireless terminal based on packet loss rate information (detected from ACK feedback) of each wireless terminal in each wireless interface, and calculates the state s(t).

[0037] In S102, the risk assessment unit 154 determines an action a based on the multiple Q tables (or a single Q table) managed in the Q table management unit 151 using the ε-greedy method.

[0038] In S103, the reinforcement learning unit 150 notifies the total radio resource allocation amount calculation unit 142, the individual radio resource allocation amount calculation unit 143, etc. of the determined action a, and the radio base station 100 then executes the action a.

[0039] In S104 , the communication quality measurement unit 142 acquires packet loss information, and passes the packet loss information to the reward calculation unit 153 in the reinforcement learning unit 150 .

[0040] In S105, the reward calculation unit 153 transmits feedback information to the aggregation device 300 and acquires the total reward from the aggregation device 300. In S106, the Q table management unit 151 updates the multiple Q tables (or the single Q table).

[0041] (Operations Related to Aggregation Device 300) In the following, as an example, a case will be described in which a radio base station 100 is connected to an aggregation device 300 as shown in FIG. 1. However, even in the case in which a radio terminal 200 is connected to an aggregation device 300, the operations described below (operations using an aggregation device 300) can be applied by replacing the radio base station 100 below with the radio terminal 200.

[0042] In this embodiment, as shown in Fig. 1, an aggregation device 300 that can communicate with the radio base stations 100 is arranged. Each radio base station 100 performs scheduling for packet transmission to multiple radio interfaces using the above-mentioned reinforcement learning.

[0043] Each radio base station 100 transmits feedback information obtained through reinforcement learning to the aggregation device 300. The aggregation device 300 calculates an overall reward based on the feedback information received from each radio base station 100, and transmits the calculated overall reward to each radio base station 100.

[0044] Each wireless base station 100 updates the multiple Q table (or the single Q table) based on the total reward received from the aggregation device 300, and selects an action by referring to the updated multiple Q table (or the single Q table).

[0045] Furthermore, the aggregation device 300 reviews the connection relationship between the wireless base station 100 and the wireless terminal 200 in response to changes in the wireless environment.

[0046] (Configuration example of aggregation device 300) 5 shows an example of the configuration of the aggregation device 300. As shown in FIG. 5, the aggregation device 300 includes a communication I / F unit 310, an information collection and distribution unit 320, a reward calculation unit 330, a detection unit 340, and a connection relationship change unit 350.

[0047] The communication I / F unit 310 performs data communication with each wireless base station 100. The communication method between the aggregation device 300 and the wireless base station 100 may be wireless or wired.

[0048] The information collection and distribution unit 320 collects feedback information from each radio base station 100 via the communication I / F unit 310. The information collection and distribution unit 320 also distributes a total reward to each radio base station 100 via the communication I / F unit 310. The reward calculation unit 330 calculates a total reward based on the feedback information collected by the information collection unit 320.

[0049] The detector 340 detects changes in the radio environment in the area of ​​the multiple radio base stations 100 accommodated by the aggregation device 300. Specifically, the detector 340 determines that a change in the radio environment has occurred when it detects a change in the overall reward or path loss that is equal to or exceeds a predetermined specified value. The specified value may also be called a reference value, threshold, or the like.

[0050] The connection relation changing unit 350 reselects the destination (connection destination) of each IF of each wireless terminal 200, and notifies the wireless base station 100 of the reselection result.

[0051] (Example of system operation) Next, with reference to the sequence diagram of Fig. 6, an operation when using aggregation device 300 in the communication system according to this embodiment will be described. The operation shown in Fig. 6 focuses on the operation of performing reinforcement learning using the overall reward. The operation of reviewing the connection relationships will be described after this explanation.

[0052] The sequence shown in Fig. 6 is executed at predetermined time intervals (for example, every frame or at the execution period of reinforcement learning). Although there are actually multiple radio base stations 100, Fig. 6 shows only one radio base station 100. The operation shown in Fig. 6 is executed for each radio base station 100.

[0053] 6 shows, as an example, the operation in which the radio base station 100 communicates with the aggregation device 300, but the radio base station 100 in Fig. 6 may be replaced with the radio terminal 200. In other words, the operation in which the radio terminal 200 communicates with the aggregation device 300 is also similar to the operation shown in Fig. 6.

[0054] <s201> In S201, each radio base station 100 performs scheduling using a reinforcement learning technique. That is, each radio base station 100 selects users for each radio interface and determines the number of packets to be transmitted for each radio interface.

[0055] In this scheduling (action selection) based on the reinforcement learning method, multiple Q-tables (or a single Q-table) updated based on the total reward are used.

[0056] <s202> In S202, each wireless base station 100 calculates the risk state, the number of successfully received packets, and the number of transmitted packets based on feedback (ACK / NACK, etc.) from the communication counterpart (here, the wireless terminal 100), and transmits these as feedback information to the aggregation device 300. In addition, each wireless base station 100 also transmits the above feedback information to the aggregation device 300, including the path loss between each I / F of each wireless terminal 100.

[0057] Here, the risk state, the number of successfully received packets, and the number of transmitted packets are respectively represented as follows: Note that the example here assumes a system model (a model using Sub-6 GHz and mmWave) described later.

[0058] Risk status indicator:u k ν (t) Number of successfully received packets: Ω k ν (t) Number of packets sent: k ν (t) k represents a terminal (device), ν represents a wireless interface (Sub or mW), and t represents the target frame. k ν (t) is determined by equation (14) described later. In equation (14), ρ is the packet loss rate, and ρ max is the required packet loss rate.

[0059] Each wireless base station 100 transmits the above information for each wireless terminal and for each wireless interface as feedback information to the aggregation device 300. The information collection unit 320 of the aggregation device 300 acquires the feedback information transmitted from each wireless base station 100.

[0060] <s203> In S203, the reward calculation unit 330 of the aggregation device 300 calculates an overall reward, which is a reward for all the wireless base stations 100, using the feedback information collected from each wireless base station 100. In addition, the detection unit 340 determines whether a change in the wireless environment has occurred. Here, it is assumed that no change in the wireless environment has occurred.

[0061] The total reward is calculated, for example, using the formula shown in Figure 7. This formula also assumes a system model (a model using Sub-6 GHz and mmWave) described later. In the formula in Figure 7, b represents the wireless base station.

[0062] As shown in FIG. 7, the total reward r is the average over time t of the sum of the "average packet reception success rate of all devices and the penalty due to the risk state at each wireless IF" for the wireless base station.

[0063] In the example of Figure 7, the calculation of the average packet reception success rate is divided into cases depending on the behavior, a. As will be described later, in this system model example, a is one of the values ​​0, 1, or 2. If a is other than 2, the value shown in A in Figure 7 is used, and if a is 2, the larger of B and C is used.

[0064] <s204> In S204, the information collection and distribution unit 320 of the aggregation device 300 distributes the total reward calculated in S203 to each wireless base station 100.

[0065] <s205> In S205, each radio base station 100 updates the multiple Q table (or the single Q table) using the total reward received from the aggregation device 300 by the reinforcement learning method of the present embodiment.

[0066] <s206> In S206, each wireless base station 100 transitions to the new state and reflects it in the processing.

[0067] By the above-described processing, a system is realized that optimizes the number of packets transmitted over a line using multi-agent reinforcement learning.

[0068] (Operations including review of connection relationships) Next, an operation including review of the connection relationships will be described. In this embodiment, in a system that optimizes the number of packets transmitted through a line using multi-agent reinforcement learning, the aggregation device 300 changes the connection relationships between the wireless base stations 100 and the wireless terminals 200 to optimal ones before optimization by reinforcement learning in each wireless base station 100.

[0069] Optimization of the connection relationship between the radio base station 100 and the radio terminal 200 is executed at system startup. Furthermore, optimization of the connection relationship between the radio base station 100 and the radio terminal 200 is triggered by a change in the overall reward calculated by the aggregation device 300 or a change in the path loss between the I / F of each radio terminal 100 and the radio base station 200 exceeding a predetermined value after optimization by reinforcement learning. Through such control, the connection relationship between the radio terminal 200 and the radio base station 100 is autonomously optimized in accordance with environmental changes.

[0070] The path loss may be calculated, for example, by receiving a signal with known transmission power from the radio base station 200 at the radio terminal 200 and measuring the received power of that signal, or by receiving a signal with known transmission power from the radio terminal 200 at the radio base station 100 and measuring the received power of that signal.

[0071] The flow of operations related to reviewing the connection relationships will be described with reference to the flowchart of FIG.

[0072] In S301, the aggregation device 300 acquires the path loss between the radio base station 100 and each I / F of each radio terminal 200 from each radio base station 100, and calculates the overall reward as described above.

[0073] In S302, the detection unit 340 of the aggregation device 300 determines whether there is a change in the total reward or the path loss that is equal to or greater than a predetermined threshold. The change here refers to, for example, a change from the value (total reward or path loss) at the time of the previous connection relationship change. The "path loss" used for the determination may be the path loss with the largest amount of change among all "pairs of I / Fs of the radio base station 100 and the radio terminal 200" that are the objects of control, or the average path loss of all "pairs of I / Fs of the radio base station 100 and the radio terminal 200" that are the objects of control, or a path loss other than these.

[0074] If the determination in S302 is Yes, the process proceeds to S303. In S303, the connection relationship change unit 350 of the aggregation device 300 reselects the connection destination of each I / F of each wireless terminal 200. The reselection method is not limited to a specific method, but here, the reselection is performed using a method utilizing graph theory, as will be described later.

[0075] The connection relation changing unit 350 notifies each radio base station 100 of the result of the reselection of the connection destination. In each radio base station 100, the terminal association control unit 141 controls the connection destination for the radio terminal 200 based on the result of the reselection. As a result of the control, the radio base station 100 and each IF of each radio terminal 200 are connected according to the result of the reselection in S302.

[0076] In S304, each radio base station 100 initializes the state of reinforcement learning. If reinforcement learning is also performed in each radio terminal 200, each radio terminal 200 also initializes the state of reinforcement learning. In S305, each radio base station 100 optimizes transmission packets at each I / F using Q-learning.

[0077] (Details of the operation regarding the review of connection relationships) Next, the process of reviewing the connection relationships executed by the connection relationship change unit 350 in S303 of Fig. 8 will be described in detail with reference to the flowchart of Fig. 9. Here, optimization applying graph theory is performed as an example. In the explanation of the flow, a specific example will be described with reference to Fig. 10 as appropriate. In Fig. 10, AP represents a wireless base station, UE represents a wireless terminal, and IF represents an I / F number. Furthermore, in the following process, it is assumed that I / Fs with the same I / F number of different UEs communicate using the same channel. It is also assumed that the initial value of the solution set is an empty set.

[0078] In S401, the connection relation changing unit 350 sets a vertex for each "combination of a wireless base station and an I / F of a wireless terminal." In the example of Fig. 10, for example, "AP2, UE1, IF1" (the combination of AP2 and IF1 of UE1) is one vertex.

[0079] In S402, the connection relation changing unit 350 connects two vertices that are different wireless base stations with the same wireless terminal and the same I / F with an edge. Connecting with an edge means that there is a possibility of interference. In Fig. 10, for example, the vertex "AP2, UE1, IF1" and the vertex "AP1, UE1, IF1" are connected with an edge.

[0080] In S403, the connection relation changing unit 350 sets the "path loss between the wireless base station and the I / F of the wireless terminal" of each vertex as the weight of that vertex. In Fig. 10, for example, 10 is set for the vertex "AP2, UE1, IF1".

[0081] In S404, the connection relationship change unit 350 selects the vertex with the lowest weight from among the vertices not set in the solution set based on the set of "vertices and edges" (which may be called a graph) set in S403, sets this vertex to the solution set, and deletes the vertices connected to this vertex by edges. In the example of FIG. 10, the weight of the vertex "AP2, UE1, IF1" is the lowest, so this vertex is set to the solution set. In addition, the vertex "AP1, UE1, IF1" that is connected to this vertex by an edge is deleted from the graph. The vertices set in the solution set are also deleted from the graph.

[0082] If each I / F of each UE can connect to multiple (Bk) APs at the same time, the Bk vertices with the lowest weights (Bk vertices of the same I / F of the same UE) are selected, the selected Bk vertices are set as the solution set, and vertices connected to these vertices by edges are deleted from the graph. Note that the "weight of Bk vertices" is, for example, the average of the "weights of Bk" vertices.

[0083] In the example of FIG. 9, the connection relation changing unit 350 determines the connection destination of each I / F by repeatedly performing the process of S404 for the vertices present in the graph until the graph becomes empty.

[0084] For example, in the example shown in FIG. 10, the final solution set is {"AP2, UE1, IF1", "AP1, UE2, IF2", "AP1, UE1, IF2", and "AP2, UE2, IF1"}.

[0085] By the above process, it is possible to select a wireless base station that minimizes the path loss for each I / F in each wireless terminal.

[0086] Another example of the flow is shown in Fig. 11. The flow in Fig. 11 is obtained by adding S405 to the flow in Fig. 9.

[0087] 11, once the process of S404 has been performed once (or multiple times), the process proceeds to S405. In S405, the connection relationship change unit 350 recalculates the weight of each vertex. For example, the weight of a vertex having the same AP and IF number (channel) as the AP of the vertex selected in S404 is increased by a predetermined amount or ratio.

[0088] For example, in the example of FIG. 10, suppose that "AP2, UE1, IF1" is initially selected, and then the process proceeds to S405. In this case, the connection relation changing unit 350 increases the weight (path loss) of "AP2, UE2, IF1." That is, in the example of FIG. 10, the set value "22" is increased. By increasing the weight (path loss) of "AP2, UE2, IF1," it is possible to make it more difficult for "AP2, UE2, IF1" to be selected. Thereafter, the process proceeds to S403, and in S404, the changed weight is set for each vertex.

[0089] By this processing, as the number of connections increases, the priority of vertices with the same AP and IF numbers as the selected vertex can be lowered, making it possible to avoid UEs being connected to a single AP intensively.

[0090] 8 to 11, the combination of each I / F of the wireless terminal 200 with the wireless base station 100 is reviewed, and then reinforcement learning is executed again to optimize the packet transmission amount for each I / F. Indicators for this optimization include, for example, maximum capacity, maximum throughput, and high reliability (low delay, low packet loss).

[0091] The operation of the radio base station 100 in this embodiment (particularly the operation of the reinforcement learning unit 150) will be described in more detail below using an example in which a specific radio interface is used. In order to make the explanation of the reinforcement learning process in this system model easier to understand, the following describes an example of operation in which a reward is calculated by a single radio base station 100.

[0092] (System Model) In this embodiment, as shown in Fig. 12, downlink (DL) transmission in a wireless network configured with multiple APs accommodating multiple devices (UE) will be described as an example. Each AP is assumed to have a Sub-6 GHz and mmWave (millimeter wave) interface. Each AP corresponds to a wireless base station 100. A device corresponds to a wireless terminal 200. In the following description, the wireless base station 100 will be described as performing the reinforcement learning operation according to this embodiment, but the wireless terminal 200 can also perform the same operation.

[0093] As shown in Figure 12, AP b transmits a desired packet to a set of devices K. Also, the set of devices K receives DL interference from all other APs b' ≠ b.

[0094] At the beginning of each scheduling frame t, AP b assigns L k Let there be (t) packets, where each packet l∈L k (t) is d bits in size and is sent to device k∈K.

[0095] AP b transmits these packets over N subchannels on the sub-6GHz interface and M beams on the mmWave interface. Each sub-6GHz subchannel or mmWave beam can be assigned to a unique device in each scheduling time frame. Multiple devices can be supported in each frame via different subchannels in sub-6GHz and different beams in mmWave.

[0096] In the sub-6GHz band, the signal-to-interference-plus-noise ratio (SINR) from AP b to device k on subchannel n is:

[0097]

number

[0098] For the mmWave interface, analog beamforming is assumed, and the transmission beam width and beam direction on beam m from AP b to device k are θ bkm and β bkm and is adjusted according to the target device k and time frame t in each beam m.

[0099] For simplicity and without loss of generality, let the receive beam gain G at device k be k Rx is assumed to be fixed. To maximize the rate obtained, θ bkm is set to the narrowest beam width, and β bkm is given by the line-of-sight (LoS) direction from AP b to device k. Therefore, the SINR of beam m at device k served by AP b is given as follows:

[0100]

number

[0101]

number

[0102]

number

[0103] Therefore, the achievable rate for device k accommodated by AP b is:

[0104]

number

[0105] Let l be the number of packets sent to device k in frame t on interface ν. k ν (t)∈{0,…,L k (t)}. L k (t) is the total number of queued packets at frame t, so l k sub (t)+l k mW (t)≦L k (t). The number of packets successfully received by device k on each interface is Ω. k ν (t) can be calculated by AP b based on the ACK feedback of device k as follows:

[0106]

number

[0107]

number

[0108]

number

[0109] where r bk ν Since (t) is unknown in AP, l k,max ν is unknown in the AP. k ν (t)≦l k,max ν If (t), that is, the number of transmitted packets in the assigned subchannel or beam of device k is smaller than the number of packets that device k can receive, we assume that all these packets are successfully received and their ACKs are fed back to the AP. However, l k ν (t) ≥ l k,max ν If (t), then l k ν (t)-l k,max ν (t) The packet is in NACK state.

[0110] Based on the above, the packet loss occurrence up to frame t is averaged across both interfaces and defined as the PLR ​​(packet loss rate) of device k at frame t, as follows:

[0111]

number

[0112]

number

[0113]

number

[0114] (About Markov Decision Processes (MDPs)) The goal here is to achieve the individual PLR constraints (here ρ max The goal is to maximize the long-term PSR averaged over all devices while satisfying the condition ( ). This problem can be modeled as an MDP characterized by a state space, an action space, transition probabilities, and a reward function, as shown in Figure 13. In Figure 13, the state s t is the PLR ​​satisfaction level (and ACK feedback status) for all devices. t is the interface selection and packet scheduling for all devices. In this embodiment, the reward r(t) is obtained based on the state s(t) and the action a(t), and the action is optimized by maximizing the objective function.

[0115] Each AP (wireless base station) is an agent that makes interface selection and packet scheduling decisions. At each frame t, the AP determines its current state s t Know the state t consists of the current PLR satisfaction levels of devices associated with the AP and their feedback states in the previous frame t-1. t Based on this, AP will take action t That is, the AP determines the number of packets in each interface of each device in the current frame t and obtains the immediate reward r from the environment. t Get the new state s t+1 Transition to.

[0116] Since the information such as the instantaneous CSI and interface statistics is unknown, the AP determines the transition probability P(s t+1 |s t ,a t In this embodiment, this problem is solved by using a framework of RL (reinforcement learning).

[0117] (Risk-Averse Reinforcement Learning) To best satisfy stringent reliability requirements, this embodiment utilizes a Risk-Sensitive Reinforcement Learning (RSRL) approach called Risk-Averse Average Q-learning (RAQL) described above. Compared to traditional RL methods that aim to maximize expected returns, such as QL, RSRL introduces the concept of risk, which is linked to the variance of rewards. RAQL achieves further variance reduction, thereby reducing risk.

[0118] Instead of taking the expected reward as the objective function as in traditional RL, we use the expected utility of the reward as the objective function:

[0119]

number

[0120]

number

[0121] The symbols in the above formulas (11) and (12) have the following meanings:

[0122] J π : The average utility function (immediate reward r) under policy π in a Markov decision process t (discount sum) Π: Policy E π,h : Policy π, expected value under the state h of the wireless channel (propagation path, etc.) r t : Immediate reward value in process t β: parameter Var[]: Variance of [] Order of O():() As will be described later, in this embodiment, multiple Q-tables are simultaneously trained by using Equation (22) as an update rule. The sample variances of these Q-tables are then used as an approximation of the true variance. From these variances, a risk-averse ^Q-table is calculated and used for behavior selection.

[0123] (RAQL-based interface selection and packet scheduling method) Next, the algorithm based on RAQL executed by the AP (wireless base station 100) in this embodiment will be described in detail. The state space and the action space are defined as follows.

[0124] State: s(t) is the current QoS satisfaction level of the PLR ​​for all devices k∈K at frame t and the most recent ACK feedback for packets sent at frame t−1, as shown in Equations (13) and (14) below. ACK feedback may not be included in s(t).

[0125]

number

[0126]

number

[0127] Action: a(t) denotes the interface selection for each device to transmit its packets. To avoid the explosion of the action space size and make the proposed method scalable, we divide the interface selection task and packet scheduling task into three actions a(t) for device k, as explained below. k The AP does not have knowledge of the instantaneous CSI, but it is reasonable to assume that the long-term CSI, such as the average path loss or the average SINR, is known due to sporadic feedback.

[0128] Therefore, each AP can assign subchannels and beams based on the average CSI of each device. In this case, all subchannels are equivalent for each device, so the AP can randomly select each subchannel to be assigned to each device. The scheduling task of each AP is then equivalent to determining the number of packets to be transmitted per subchannel for each device. The frame length T s The maximum number of packets transmitted from AP b and successfully received by device k during the period can be estimated as shown in equation (15) below.

[0129]

number

[0130] a k (t)=0: Only the Sub-6GHz interface is used, and the number of transmitted packets is

[0131]

number

[0132] a k (t)=1: Only the mmWave interface is used, and the number of transmitted packets is

[0133]

number

[0134] a k (t)=2: Both the Sub-6GHz interface and the mmWave interface are used, but the mmWave interface is prioritized to maximize the number of packets transmitted by taking advantage of its high data rate.

[0135]

number

[0136]

number

[0137]

number

[0138]

number

[0139] The meaning of each symbol in formula (21) is as follows:

[0140] r(s(t),a(t)): immediate reward value in process t Ω k sub (τ): Number of packets successfully sent via Sub6GHZ I / F Ω k mW (τ): Number of packets successfully transmitted via the millimeter wave interface l k sub (τ): Number of packets sent via Sub6GHZ I / F l k mW (τ): Number of packets sent over the millimeter wave interface u k sub (t): Packet loss rate ρ at Sub6GH I / F is required quality ρ max Variables that change depending on whether u k mW (t): Packet loss rate ρ at millimeter wave I / F is the required quality ρ max Variables that change depending on whether As is clear from equation (14), u k ν If (t)=0, i.e., device k is in a risk state that does not satisfy the PLR ​​in equation (14), the reward is penalized.

[0141] When the aggregation device 300 is used, as already explained, the aggregation device 300 calculates the total reward shown in FIG. 7 based on feedback information from each radio base station 100 and distributes it to each radio base station 100.

[0142] The RAQL-based interface selection and packet scheduling method in this embodiment is executed by Algorithm 1 shown in Fig. 14. That is, the radio base station 100 executes this algorithm by, for example, running a program on the CPU. The meanings of the symbols are as follows:

[0143] ε: Search rate λ: Attenuation rate Number of I:Q tables λ p : Risk control parameters Q:Q Table V:Q table update count α: learning rate In Algorithm 1, the AP first initializes I Q-tables along with table V, which counts the number of selections of each action a under state s. The corresponding learning rate α is also initialized to 0, and the algorithm starts from a random state (lines 1-2).

[0144] At each frame t, a Q-table is randomly selected, and the Q-table is used to calculate the risk aversion ^Q-table by Equation (24) described below (lines 3-5). Unlike conventional QL, in RAQL, the Q-function is updated by Equation (22) below.

[0145]

number

[0146]

number

[0147]

number

[0148] Next, given the current state and search rate ε, an action a(t) is selected using the ε-greedy strategy. The AP transmits a packet based on the selected action and receives an immediate reward (Equation (21)) (lines 6-9). The environment then transitions to a new state (lines 10-16). This process is repeated until the maximum number of frames T is reached.

[0149] When the aggregation device 300 is used, the total reward in Fig. 7 is used as the reward in algorithm 1 in Fig. 14. Note that both the total reward and the reward for the radio base station 100 alone calculated by equation (21) may be used as the reward.

[0150] (Example of hardware configuration) The aggregation device 300, the radio base station 100, and the radio terminal 200 can all be realized by, for example, causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud. Hereinafter, the aggregation device 300, the radio base station 100, and the radio terminal 200 will be collectively referred to as devices.

[0151] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0152] Fig. 15 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 15 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are all connected to each other via a bus B. Note that the communication device may not be provided with the display device 1006.

[0153] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0154] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0155] (Effects of the embodiment) The technology according to this embodiment makes it possible to review the connection relationships between wireless base stations and wireless terminals when there is a change in the wireless environment. As a result, it becomes possible to optimize line usage (improvement of communication capacity, reliability, etc.) in line with the environment. In addition, graph theory, for example, can be used in reviewing the connection relationships. By using graph theory, it is possible to organize the relationships between interfaces (overlapping use, degree of interference), and proceed with the process of reviewing the connection relationships.

[0156] <Additional Notes> This specification discloses the following aggregation device, communication system, processing method, and storage medium. (Additional note 1) An aggregation device that accommodates a plurality of radio base stations that perform scheduling using reinforcement learning, a detection unit that determines whether or not an overall reward in the reinforcement learning satisfies a predetermined condition, or whether or not a path loss between a wireless base station and a wireless terminal satisfies a predetermined condition; a connection relationship changing unit that changes a connection relationship between a wireless base station and a wireless terminal when the detection unit determines that the predetermined condition is satisfied; An aggregation device comprising: (Additional note 2) The predetermined condition is that the change in the total reward or the change in the path loss is equal to or greater than a specified value. Item 1. The aggregation device according to item 1. (Additional note 3) The connection relation changing unit selects a wireless base station that minimizes path loss for each wireless interface of each wireless terminal. Item 1. The aggregation device according to item 1. (Additional note 4) The connection relationship change unit performs the selection using a graph consisting of a vertex set for each combination of a wireless base station, a wireless terminal, and a wireless interface of the wireless terminal, and an edge connecting two vertices having different wireless base stations with the same wireless terminal and the same wireless interface. Item 3. The aggregation device according to item 3. (Additional note 5) an information collection unit that collects feedback information from the plurality of radio base stations; a reward calculation unit that calculates the total reward for the plurality of radio base stations using the feedback information; Item 1. The aggregation device according to item 1, further comprising: (Additional note 6) A communication system including the aggregation device according to any one of appended claims 1 to 5 and the plurality of wireless base stations. (Additional note 7) A processing method executed by an aggregation device accommodating a plurality of radio base stations that execute scheduling using reinforcement learning, the method comprising: a detection step of determining whether or not an overall reward in the reinforcement learning satisfies a predetermined condition, or whether or not a path loss between a wireless base station and a wireless terminal satisfies a predetermined condition; a connection relationship changing step of changing a connection relationship between the wireless base station and the wireless terminal when it is determined in the detection step that the predetermined condition is satisfied; A processing method comprising: (Additional note 8) A non-transitory storage medium storing a program for causing a computer to function as each unit in the aggregation device described in any one of appendixes 1 to 5.

[0157] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0158] 100 wireless base stations 101 Antenna 110 Communication I / F section 120 control section 130 Radio Communication Department 131 Receiving unit 132 Wireless communication signal generator 135 RF section 140 Scheduler 141 Communication Quality Measurement Unit 141 Terminal Destination Control Unit 142 Communication Quality Measurement Unit 143 Total radio resource allocation calculation unit 144 Individual radio resource allocation calculation unit 150 Reinforcement Learning Department 151 Q Table Management Department 152 State calculation unit 153 Remuneration Calculation Department 154 Risk Assessment Department 200 Wireless Terminals 300 Aggregation Device 310 Communication I / F section 320 Information Collection and Distribution Department 330 Remuneration Calculation Department 340 Detection unit 350 Connection relationship change section 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output Device

Claims

1. An aggregation device that accommodates a plurality of radio base stations that perform scheduling using reinforcement learning, a detection unit that determines whether or not an overall reward in the reinforcement learning satisfies a predetermined condition, or whether or not a path loss between a wireless base station and a wireless terminal satisfies a predetermined condition; a connection relationship changing unit that changes a connection relationship between the wireless base station and the wireless terminal when the detection unit determines that the predetermined condition is satisfied; An aggregation device comprising:

2. The predetermined condition is that the change in the total reward or the change in the path loss is equal to or greater than a specified value. The aggregation device of claim 1 .

3. The connection relation changing unit selects a wireless base station that minimizes path loss for each wireless interface of each wireless terminal. The aggregation device of claim 1 .

4. The connection relationship change unit performs the selection using a graph consisting of a vertex set for each combination of a wireless base station, a wireless terminal, and a wireless interface of the wireless terminal, and an edge connecting two vertices having different wireless base stations with the same wireless terminal and the same wireless interface. The aggregation device according to claim 3 .

5. an information collection unit that collects feedback information from the plurality of radio base stations; a reward calculation unit that calculates the total reward for the plurality of radio base stations using the feedback information; The aggregation device of claim 1 further comprising:

6. A communication system comprising the aggregation device according to claim 1 and the plurality of wireless base stations.

7. A processing method executed by an aggregation device accommodating a plurality of radio base stations that execute scheduling using reinforcement learning, the method comprising: a detection step of determining whether or not an overall reward in the reinforcement learning satisfies a predetermined condition, or whether or not a path loss between a wireless base station and a wireless terminal satisfies a predetermined condition; a connection relationship changing step of changing a connection relationship between the wireless base station and the wireless terminal when it is determined in the detection step that the predetermined condition is satisfied; A processing method comprising:

8. A program for causing a computer to function as each unit in the aggregation device according to any one of claims 1 to 5.

Images