Wireless communication method, wireless terminal, and program for wireless terminal

Abstract

To relate to a radio communication method for obtaining high communication efficiency in a network including a plurality of radio terminals and a plurality of radio base stations, or the like.SOLUTION: A radio communication method includes: detecting a state Sk(t) of connection between a plurality of radio base stations and a radio terminal; (step 110) randomly determining an action to be adopted in the state Sk(t) in a probability ε (Step 108); (Step 112) inputting the state Sk(t) to a DQN in other cases; (Step 114) adopting an optimal action ak(t); (Step 122) updating a parameter of the DQN on the basis of a reward Γk(t) (Step 118) for the action ak(t); (Step 104) attenuating ε for each learning in a situation in which, a communication environment does not change; and (Step 106) resetting the attenuated ε to an initial value in a situation in which, the communication environment changes (Step 102).SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present disclosure relates to a wireless communication method, a wireless terminal, and a program for a wireless terminal, and in particular to a wireless communication method, a wireless terminal, and a program for a wireless terminal that are suitable for achieving high communication efficiency in a network including multiple wireless terminals and multiple wireless base stations, etc. [Background technology]

[0002] With the development of wireless communication systems, heterogeneous networks using multi-band, multi-access systems are reaching the stage of practical application. In cellular communications, the fifth generation mobile communications (5G) has been put into practical use. 5G communication systems use a wide range of frequencies, from sub-1 GHz bands to millimeter wave bands. In addition, an environment is being developed in which cells of various sizes, from small cells to macro cells, are provided in an overlapping manner.

[0003] Another typical wireless access system, wireless LAN, uses radio frequencies in the 2.4 / 5 / 60 GHz bands, and the use of the 6 GHz band is also being considered. Wireless terminals such as smartphones are generally provided with interfaces (IFs) for accessing both cellular and wireless LAN. Each IF is configured to support multiple frequency bands.

[0004] Wireless terminals generally select a wireless base station or access point (AP) to connect to from multiple frequency bands and access methods to establish communication. With functions such as dual connectivity, one wireless terminal may integrate and use multiple wireless base stations. In such a heterogeneous environment, it is important to control and optimize, across the entire system, which IF a wireless terminal uses and which base station it selects, in order to effectively use system resources.

[0005] Regarding technology for effectively utilizing system resources in a heterogeneous environment, for example, Non-Patent Document 1 and Non-Patent Document 2 below disclose a method for optimizing connections between wireless terminals and APs by reinforcement learning. More specifically, Non-Patent Document 2 discloses a technology for optimizing connections between wireless terminals using multiple IFs and multiple APs by using a method of Deep Q-Learning or Double Deep Q-Learning.

[0006] In addition, the following non-patent document 3 discloses Deep Q-Network (DQN), a method for handling high-dimensional inputs using reinforcement learning, and the following non-patent document 4 discloses Double Deep Q-Network (DDQN). [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] “Reinforcement Learning-aided Distributed User-to-Access Points Association in Interfering Networks”, Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Hirantha Abeysekera, Yasushi Takatori, IEEE Globecom, Waikoloa, Hawaii, USA, Dec. 2019 [Non-Patent Document 2] "Deep Reinforcement Learning-based User Association in Sub6GHz / mmWave Integrated Networks", Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Kenichi Kawamura, Takatsune Moriyama, Hirantha Abeysekera and Yasushi Takatori, in IEEE CCNC, Jan. 2021 [Non-Patent Document 3] "Human-level control through deep reinforcement learning," Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg & Demis Hassabis, Nature. 2015 Feb 26; Vol.518(7540): P529-533. doi: 10.1038 / nature14236 [Non-Patent Document 4] "Deep Reinforcement Learning with Double Q-Learning," H. Van Hasselt, A. Guez, and D. Silver, in Thirtieth AAAI conference on artificial intelligence, 2016 [Summary of the Invention] [Problems to be Solved by the Invention]

[0008] In deep reinforcement learning of DQN and DDQN as disclosed in the above-mentioned non-patent literature, learning is performed using the ε-greedy method. According to the ε-greedy method, the next action is randomly selected with a probability of ε for each learning iteration. In addition, the action that is determined to be optimal based on the learning result at that time is selected as the next action with a probability of 1-ε.

[0009] According to the above-mentioned ε-greedy method, the current optimal action is selected with a probability of 1-ε, and a new "optimum" is searched for by random actions with a probability of ε. Therefore, this method allows learning to proceed toward the true optimal solution without getting stuck on a local optimal solution found by chance.

[0010] In the ε-greedy method used in DQN and DDQN, the probability ε is further decayed each time learning is repeated. With this process, as learning progresses, the probability of selecting a random action decreases and the probability of selecting an optimal action increases. As a result, unnecessary searches at the advanced learning stage are avoided, and the time required to converge to the optimal solution is shortened.

[0011] In a heterogeneous multi-band / multi-access environment, mobile terminals such as smartphones and tablets may be used as wireless terminals. The communication environment surrounding the mobile terminal changes as the mobile terminal moves. In addition, when there are many wireless terminals in the same area, the change in the request from each wireless terminal affects the load of the wireless terminals and wireless base stations belonging to that area. As a result, the communication environment surrounding each wireless terminal changes significantly depending on the state of other wireless terminals. For these reasons, the optimal solution regarding which wireless base station and which IF a wireless terminal placed in a heterogeneous environment should connect to may not be a stable one, but a dynamic one that changes drastically.

[0012] Conventional learning methods that attenuate ε as learning progresses are effective in quickly obtaining an optimal solution in a stable communication environment. However, in a rapidly changing and dynamic communication environment, if ε decays, it becomes impossible to efficiently search for an optimal solution by following the changes in the environment. In this respect, conventional learning methods using DQN, DDQN, etc. have the problem that they cannot always use system resources effectively in a dynamic communication environment.

[0013] The present disclosure has been made with attention to the above-mentioned problems, and aims to provide a wireless communication method, a wireless terminal, and a program for a wireless terminal that always make effective use of system resources in a network including multiple wireless terminals and multiple base stations, regardless of changes in the communication environment. [Means for solving the problem]

[0014] In order to achieve the above object, a first aspect of the present disclosure is a wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, the method comprising: detecting a state of connection between the plurality of wireless base stations and the wireless terminal; randomly determining with probability ε a behavior to be adopted by said wireless terminal under detected conditions; determining, with probability (1-ε), an action to be taken by said wireless terminal under the sensed conditions in accordance with a result of a value function; transmitting a request to said plurality of radio base stations corresponding to an action to be taken; obtaining responses from the plurality of radio base stations to the request; calculating a reward for the behavior based on the response; updating a value for adopting the behavior under the state based on the reward; learning parameters of the value function based on the updated values; detecting a change in a communication environment surrounding the wireless terminal; performing a time-dependent attenuation process on the ε under a condition where a change in the communication environment is not detected; initializing the ε when a change in the communication environment is detected; It is preferable that the present invention includes the following:

[0015] A second aspect of the present disclosure is a wireless terminal having a function of connecting to a plurality of wireless base stations, Equipped with a CPU, The CPU, A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal; a process of randomly determining with probability ε an action to be taken under the detected condition; determining, with probability (1-ε), an action to be taken under the detected condition according to the result of the value function; transmitting a request corresponding to an action to be taken to said plurality of wireless base stations; obtaining responses from the plurality of wireless base stations to the request; calculating a reward for the behavior based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; A process of detecting a change in a communication environment surrounding the wireless terminal; a process of performing a time-dependent attenuation process on the ε under a situation where a change in the communication environment is not detected; A process of initializing the ε when a change in the communication environment is detected; It is advisable to carry out the following:

[0016] A third aspect of the present disclosure is a program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations, comprising: The CPU of the wireless terminal has A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal; a process of randomly determining with probability ε an action to be taken under the detected condition; determining, with probability (1-ε), an action to be taken under the detected condition according to the result of the value function; transmitting a request corresponding to an action to be taken to said plurality of wireless base stations; obtaining responses from the plurality of wireless base stations to the request; calculating a reward for the behavior based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; A process of detecting a change in a communication environment surrounding the wireless terminal; a process of performing a time-dependent attenuation process on the ε under a situation where a change in the communication environment is not detected; A process of initializing the ε when a change in the communication environment is detected; It is preferable that the program include a program for executing the above. Effect of the Invention

[0017] According to the present disclosure, in a network including a plurality of wireless terminals and a plurality of base stations, etc., it is possible to continue updating the connection combination to an optimal state regardless of changes in the communication environment. Therefore, according to the present disclosure, it is possible to always effectively utilize system resources in a heterogeneous environment. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram for explaining an overall configuration of a wireless communication system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram for explaining an example of a connection state between a wireless terminal and a wireless base station in the first embodiment of the present disclosure. [Diagram 3] FIG. 2 is a diagram illustrating a state in which a wireless terminal moves in a heterogeneous environment in the first embodiment of the present disclosure. [Figure 4]FIG. 2 is a block diagram for explaining a functional configuration of a radio base station used in the first embodiment of the present disclosure. [Diagram 5] FIG. 2 is a block diagram for explaining a functional configuration of a wireless terminal used in the first embodiment of the present disclosure. [Figure 6] 2 is a diagram for explaining a hardware configuration of a wireless terminal used in the first embodiment of the present disclosure. FIG. [Figure 7] FIG. 1 is a diagram for explaining an overview of DQN used in the first embodiment of the present disclosure. [Figure 8] FIG. 2 is a diagram for explaining the content of a process related to updating a DQN used in the first embodiment of the present disclosure. [Figure 9] 4 is a flowchart for explaining features of a wireless terminal according to the first embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Embodiment 1 [Overall configuration of the first embodiment] Fig. 1 is a diagram for explaining an overall configuration of a wireless communication system 10 according to a first embodiment of the present disclosure. As shown in Fig. 1, the wireless communication system 10 includes a wired communication network 12. A plurality of wireless base stations 16 are connected to the network 12 via a switch 14. Each wireless base station 16 can communicate with another wireless base station 16 via the network 12 and via a wireless path.

[0020] 1 includes a plurality of wireless terminals 18. In this embodiment, the wireless terminals 18 include mobile terminals such as smartphones or tablet terminals.

[0021] Each of the wireless terminals 18 includes a plurality of sets of applications and IFs for wireless communication. For example, the wireless terminals 18 include an application and IF for communication at 1 GHz or less, an application and IF for communication in the millimeter wave band, and an application and IF for communication in the 2.4 / 5 / 60 GHz bands used in wireless LAN.

[0022] The above-mentioned wireless base station 16 also has corresponding IFs, etc. Each wireless terminal 18 can wirelessly communicate with one or more wireless base stations 16 via the above-mentioned applications and IFs. This allows the wireless communication system 10 of the present embodiment to function as a heterogeneous network using multi-band multi-access.

[0023] In addition, the wireless communication system 10 of the present embodiment is compatible with wireless LAN access points (APs) in addition to wireless base stations for cellular communication installed by telecommunications carriers. For the sake of convenience, the following description will be given assuming that the "wireless base station 16" encompasses both.

[0024] [Features of the first embodiment] Next, features of the wireless terminal 18 in this embodiment will be described with reference to Figures 2 to 9. In the following description, it is assumed that the wireless base station 16 and the wireless terminal 18 are provided with an interface IF1 corresponding to communication in a first frequency band and an interface IF2 corresponding to communication in a second frequency band. IF1 is an interface corresponding to a frequency band of, for example, 1 GHz or less, and IF2 is an interface corresponding to frequencies in the millimeter wave band, for example.

[0025] In the following description, when it is necessary to specify a specific wireless terminal 18 among a plurality of wireless terminals, the symbol k will be used to indicate "wireless terminal 18." k Furthermore, when it is necessary to distinguish between the multiple wireless base stations 16, the reference numerals with suffixes, such as 16-1 and 16-2, will be used.

[0026] FIG. 2 shows a wireless terminal 18 k The figure shows that the wireless terminal 18 belongs to two wireless base stations 16-1 and 16-2. k The wireless terminal 18 measures the received signal strength indicator (RSSI) that indicates the channel quality in the communication environment surrounding the wireless terminal 18. Then, the wireless terminal 18 selects Bmax wireless base stations 16 as communication targets in descending order of the strength indicated by the RSSI. k is connected to wireless base stations 16-1 and 16-2.

[0027] Wireless terminal 18 k As described above, the wireless base station 16 has two interfaces, IF1 and IF2, corresponding to different bands. If Bmax wireless base stations 16 are the targets of communication, the wireless terminal 18 k The size of the state space is the number of possible combinations between the two IFs of the wireless terminal 18 and the two IFs of the Bmax wireless base stations 16. k However, a state is created in which the terminal belongs to the first wireless base station 16-1 at IF1 and belongs to the second wireless base station 16-2 at IF2.

[0028] FIG. 3 shows the results of a multi-band, multi-access heterogeneous environment in which a wireless terminal 18 k First, the wireless terminal 18 is moving. k In this environment, the wireless terminal 18 is located at the leftmost position in the figure. k However, if the state of belonging to the wireless base stations 16-1 and 16-2 is optimal, the wireless terminal 18 k It is desirable for a connection request from to be issued so that this state is realized.

[0029] However, the wireless terminal 18 k However, if the wireless terminal 18 moves from the leftmost position in the figure to the center of the figure, or even to the right side of the figure, the optimal connection combination will change from the original combination.k This also occurs due to the movement of other wireless terminals 18 (not shown) that belong to the same environment, changes in the requested data rates, etc. k When the communication environment changes, it is desirable to issue a connection request with a combination that is optimal for the changed environment.

[0030] In this embodiment, the wireless terminal 18 determines the wireless base station 16 and IF to which it desires to connect by using a Deep Q-Network (DQN) or a Double Deep Q-Network (DDQN). For this reason, it is desirable that the DQN or DDQN used here quickly converges to an optimal state for a stable communication environment, and quickly follows environmental changes in a dynamic communication environment.

[0031] 4 is a block diagram for functionally explaining an example of wireless base station 16. Wireless base station 16 includes an antenna unit 20. Antenna unit 20 is used by wireless base station 16 to transmit and receive wireless signals to and from other wireless base stations 16 and surrounding wireless terminals 18.

[0032] The antenna unit 20 is connected to the wireless communication unit 22. The wireless communication unit 22 includes an RF (Radio Frequency) unit 24 that processes high-frequency signals, and a MAC (Media Access Control) function unit 26 that has a function of media access control in wireless communication. The wireless communication unit 22 is configured to be able to realize both a function corresponding to a plurality of frequency bands, specifically, a function as an IF1 corresponding to 1 GHz or less, and a function as an IF2 corresponding to millimeter wave band frequencies.

[0033] The wireless base station 16 also includes a quality measurement unit 28. The quality measurement unit 28 includes a wireless environment measurement unit 30 and a traffic measurement unit 32. The wireless environment measurement unit 30 acquires information related to the wireless communication environment surrounding the wireless base station 16. The traffic measurement unit 32 acquires information related to the traffic between the wireless base station 16 and the wireless terminal 18. The quality measurement unit 28 calculates information on the communication quality between the wireless base station 16 and the wireless terminal 18 based on the information. The communication quality information includes, for example, the received power from the wireless terminal 18, the required data rate of the wireless terminal 18, the actual transmission rate, and the band usage rate in the wireless base station 16.

[0034] The wireless base station 16 includes a communication IF unit 34. The communication IF unit 34 is used for wired communication via the network 12. The wireless base station 16 can acquire communication quality information acquired by other wireless base stations 16 through wired communication via the communication IF unit 34 or through wireless communication via the wireless communication unit 22 and the antenna unit 20. The communication quality information acquired from other wireless base stations 16 is managed in a unified manner in the quality measurement unit 28 together with communication quality information acquired by the wireless base station 16 itself.

[0035] The wireless base station 16 includes an information notification unit 36. The information notification unit 36 ​​can read out communication quality information aggregated in the quality measurement unit 28. The information notification unit 36 ​​can provide the read communication quality information to the communication IF unit 34 and the wireless communication unit 22. This allows the wireless base station 16 to provide its own communication quality information to other wireless base stations 16 via a wired or wireless path.

[0036] The wireless base station 16 receives an association request from a wireless terminal 18 located in the vicinity via the wireless communication unit 22 and the antenna unit 20. The association request is processed by a request information evaluation unit 38. The request information evaluation unit 38 determines whether or not association is possible based on the bandwidth and data rate requested by the wireless terminal 18, as well as its own load state, etc.

[0037] The result of the determination by the request information evaluation unit 38 is sent to a request response notification unit 40 and an association information recording unit 42. The request response notification unit 40 provides the result of the determination as to whether association is possible or not to the wireless terminal 18 that issued the association request, via the wireless communication unit 22 and the antenna unit 20. The association information recording unit 42 updates and records information regarding which wireless terminal 18 is associated with the wireless base station 16 based on the result of the determination.

[0038] 5 is a block diagram for functionally explaining an example of wireless terminal 18. Wireless terminal 18 includes an antenna unit 44 and a wireless communication unit 46. Wireless communication unit 46 includes an RF unit 24 and a MAC function unit 50. These functions are substantially the same as the functions of the corresponding blocks included in wireless base station 16, and therefore description thereof will be omitted here.

[0039] The wireless terminal 18 includes a control information calculation unit 52. The control information calculation unit 52 acquires communication quality information from the wireless base station 16 to which the wireless terminal 18 belongs via the wireless communication unit 46 and the antenna unit 44. Then, based on the communication quality information, the control information calculation unit 52 determines, by the DQN or DDQN method, the optimal action a to which wireless base station 16 each of the IFs for wireless communication should be connected. k The process determines whether the target object is a target object or not, and performs learning for that purpose.

[0040] The control information calculation unit 52 includes a state calculation unit 54. The state calculation unit 54 calculates a state S to be provided to the input layer of the DQN or DDQN based on the communication quality information. k (t) is calculated.

[0041] The control information calculation unit 52 also includes a reward calculation unit 56. The reward calculation unit 56 calculates the reward for the adopted action a based on the communication quality information. k Reward Γ obtained by (t) k Calculate (t).

[0042] The control information calculation unit 52 further includes a DQN / DDQN update unit 58. The DQN / DDQN update unit 58 updates a parameter θ of the DQN or DDQN used in reinforcement learning. More specifically, the weighting coefficient w and the like that determine the characteristics of the DQN or DDQN as a value function are updated.

[0043] The DQN or DDQN updated by the DQN / DDQN update unit 58 is used in the attribute calculation unit 60. That is, the attribute calculation unit 60 determines the action a to be taken this time according to the updated DQN or DDQN. k (t), that is, which IF should be assigned to which wireless base station 16 is calculated.

[0044] DQN and DDQN input a state S k Given (t), the possible actions that the wireless terminal 18 can take are k For each of (t), the value of adoption is displayed in the output layer. Therefore, if the action with the largest value is selected, the action considered optimal at the current learning stage can be selected. However, the next action a k If (t) is always selected based on the above values, it is possible that we may become trapped in a local solution chosen by chance, and the truly optimal action may never be selected.

[0045] In order to avoid such a situation, the destination calculation unit 60 performs the action a according to the ε-greedy method. k For this reason, the destination calculation unit 60 randomly calculates the next action a with a probability ε regardless of whether it is DQN or DDQN. k (t) is selected. Also, with probability 1-ε, the action a that DQN or DDQN judges to be optimal is k (t) is selected.

[0046] Furthermore, the destination calculation unit 60 performs a decay process on ε every time learning is repeated. The possibility of falling into a local solution selected by chance is large in the early stages of learning and decreases as learning progresses. For this reason, when the behavior a is randomly selected, kThe need to select (t) decreases as learning progresses. When ε is decayed, the action a k It is possible to reduce the frequency with which (t) is randomly selected. Therefore, according to the destination calculation unit 60 of this embodiment, learning can be efficiently carried out and the parameter θ can be converged to an optimal value in a short period of time.

[0047] However, in the wireless communication system 10 of the present embodiment, the communication environment surrounding the wireless terminal 18 may change significantly due to the movement of the wireless terminal 18 or due to changes in the bandwidth or data rate required by other wireless terminals 18 located in the vicinity. If the communication environment changes, the parameter θ, which was the optimal value before the change, may no longer be the optimal value.

[0048] In this situation, i.e., when a new optimum needs to be found, the next action a k It is desirable for (t) to be chosen randomly as frequently as possible, and such a requirement cannot be met if ε decays as the learning process progresses.

[0049] In order to meet the above requirements, the wireless terminal 18 of this embodiment includes an environmental change detection unit 62 and an ε initialization determination unit 64. The environmental change detection unit 62 detects a change in the communication environment surrounding the wireless terminal 18. For example, when the combination of wireless base stations 16 that the wireless terminal 18 can access changes, or when the combination of IFs available on the wireless base station 16 side changes, the environmental change detection unit 62 determines that the communication environment surrounding the wireless terminal 18 has changed.

[0050] The ε initialization determination unit 64 determines that ε needs to be initialized when a change in the communication environment surrounding the wireless terminal 18 is detected. The determination of the ε initialization determination unit 64 is provided to the belonging calculation unit 60. Then, upon receiving the determination to initialize ε, the belonging calculation unit 60 initializes the attenuated ε to its original value. As a result, the next action a kThe probability ε that (t) is randomly determined increases, and the frequency of searching for a new optimum increases. Therefore, according to the belonging calculation unit 60 of this embodiment, in an environment in which the communication environment surrounding the wireless terminal 18 changes, the parameters of DQN and DDQN can be appropriately adapted to the change.

[0051] The wireless terminal 18 includes a memory unit 66. The memory unit 66 stores the following information: k Store as (t). 1. State S provided to the input layer of DQN or DDQN in this learning k (t), 2. That state S k Action taken in response to (t) k (t), 3. The action a k Reward Γ obtained by (t) k (t), 4. The action a k State S after transition due to (t) k (t+1).

[0052] The wireless terminal 18 also includes an attribute request unit 68. The attribute request unit 68 receives the action a from the attribute calculation unit 60. k Receive (t) and take action a k The wireless terminal 18 generates a request corresponding to (t) in the wireless base station 16-1 via IF1, or generates a request to connect the wireless terminal 18 to the wireless base station 16-2 via IF2. The generated request is transmitted to one or more wireless base stations 16 via the wireless communication unit 46 and the antenna unit 20.

[0053] The wireless terminal 18 includes an associated destination recording unit 70. The information received from the wireless base station 16 includes a response to the request as to whether it is OK or not. For example, the information includes a response to allow a new connection, or a response to allow an update of a connection. The associated destination recording unit 70 updates and records information about the associated wireless base station 16 based on these responses.

[0054] 6 is a diagram for explaining the hardware configuration of the wireless terminal 18. Here, a case will be explained where the wireless terminal 18 is a smartphone. However, the wireless terminal 18 is not limited to a smartphone, and may be a tablet terminal or a general-purpose computer system.

[0055] The wireless terminal 18 includes a CPU (Central Processing Unit) 72. The CPU 72 is connected to various hardware elements included in the wireless terminal 18 via a bus wiring 74. To the bus wiring 74, various memory devices such as a ROM (Read Only Memory) 76, a RAM (Random Access Memory) 78, and a storage 80 are connected.

[0056] The storage 80 includes a non-volatile memory such as a flash ROM. Programs and data required for executing various applications are stored in the storage 80. The memory unit 66 and the destination recording unit 70 described above are realized by the storage 80. The CPU 72 also performs processing according to the programs stored in the storage 80, thereby realizing functions of the control information calculation unit 52, the environmental change detection unit 62, the ε initialization determination unit 64, and the like shown in FIG. 5.

[0057] The wireless terminal 18 includes a communication interface 82. The communication interface 82 corresponds to the antenna unit 44 and the wireless communication unit 46. The wireless terminal 18 further includes an operation unit 84 and a display unit 86. The operation unit 84 includes a touch panel integrated with the display unit 86, in addition to various mechanical switch mechanisms and electronic sensor mechanisms.

[0058] [Overview of DQN] FIG. 7 is a diagram for explaining an overview of DQN. Hereinafter, with reference to FIG. 7, the wireless terminal 18 kThe outline of DQN, which is an example of deep reinforcement learning used by the present embodiment, will be described. The DQN used in this embodiment has an input layer, a hidden layer, and an output layer. In addition, the DQN has a parameter θ i t The characteristic of DQN as a value function is the parameter θ i t is determined by.

[0059] In Fig. 7, the input layer of DQN consists of M nodes S 1 ~S M These nodes include wireless terminals 18 k Status S k The elements that make up the state S are input. k The elements include, for example, the following elements: 1. Bandwidth utilization rate φb of wireless base station 16-b, 2. Wireless terminal 18 k The required data rate R of the interface f kf , 3. Wireless terminal 18 k and the wireless base station 16-b bkf (1 if it belongs, 0 if not) 4. Wireless terminal 18 k A request for attribution to the wireless base station 16-b of the interface f of bkf (1 if requested, 0 if not), 5. Wireless terminal 18 k The transmission rate r established between the interface f and the wireless base station 16-b bkf .

[0060] The number of hidden layers and the number of nodes are arbitrary. k But in state S k Actions that can be taken under a i The number of nodes is equal to the total number of actions in the i Some examples are shown below. ·Only IF1 is assigned to radio base station 16-1. · Both IF1 and IF2 are associated with radio base station 16-1. ·Only IF2 is assigned to radio base station 16-1. · IF1 is associated with the wireless base station 16-1, and IF2 is associated with the wireless base station 16-2. · · ·

[0061] Each node in the output layer is connected to a wireless terminal 18. k Possible action a i Q value for each of (i=1~N) = Q(S k ,a i ;θ i t ) is expressed. The Q value is expressed by the parameter θ i t Action a calculated by DQN under i This is the value of adopting it.

[0062] In the method using DQN, the largest Q value is selected from the Q values ​​that appear in the output layer, and the action a that produces that Q value is i Let a be the optimal action at the current time t. k According to DQN, in this way, at each learning stage, the optimal action at that time t is determined as a k (t) can be determined.

[0063] FIG. 8 shows the wireless terminal 18 k The control information calculation unit 52 calculates the parameters θ of DQN. i FIG. 8 is a diagram for explaining the flow of updating the action a k For convenience, (t) is shown after DQN / DDQN, but action a k As mentioned above, (t) is determined by DQN or DDQN only when the probability is 1-ε. And, under the probability ε, the action a k (t) is determined randomly. Parameter θ i The experience gained in both cases is k (t) is reflected.

[0064] Wireless terminal 18 k So, action a kAfter (t) is decided, the action a k Reward Γ for (t) k (t) is calculated. Reward Γ k (t) is calculated, for example, by the following formula (1).

[0065]

number

[0066] In the above formula (1), the first term on the right side is a point-adding item, and the second term is a point-deducting item. 1k , ω 2k is a coefficient that determines the allocation ratio for them. Also, ν is an identifier for distinguishing between the sub-1 GHz term that IF1 should handle and the millimeter wave (mW) term that IF2 should handle.

[0067] Furthermore, in the above formula (1), c in the first term on the right side 1k ν is the required data rate R kf A transmission rate r that satisfies bkf This means the additional points when the following is obtained. The value is calculated using the following formula (2).

[0068]

number

[0069] The above formula (2) is expressed as follows: bkf Points according to bkf / W ν This means that the following is given. However, the W in the last term ν is a factor used to offset and normalize the difference between the (sub) and (mW) frequency bands. 1. Wireless terminal 18 k The interface f of the bkf (t)=1). 2. Wireless terminal 18 k The wireless base station 16-b recognizes the ownership of the interface f (xbkf = 1), and the required data rate R kf Transmission rate exceeding r bkf The result is that (r bkf (t)>R kf ).

[0070] In the above formula (1), c in the second term on the right side 2k ν refers to the points deducted in the following two situations: 1. A request for attribution is made (a bkf (t)=1), the attribution was allowed, but (x bkf (t)=1), the required data rate R kf A transmission rate r that satisfies bkf If not obtained (r bkf (t) <R kf ). 2. A request for attribution was made (a bkf (t)=1), when the attribution is dropped from the wireless base station 16 (x bkf ν (t)=0).

[0071] Deduction in case of (sub)c 2k sub For example, it can be expressed as the following formula (3). Also, the deduction point c for (mW) 2k mW can be expressed, for example, as in the following equations (4) and (5).

[0072]

number

[0073]

number

[0074]

number

[0075] Thus, Action a kThe required data rate R as a result of (t) kf A transmission rate r that satisfies bkf If (t) is obtained, the reward Γ k On the other hand, if the transmission rate r bkf If (t) is not obtained, reward Γ k (t) becomes a small value.

[0076] Reward Γ k Once (t) is calculated, the current experience e k (t) is stored in memory (step 88). k (t), selected action a k (t), the reward obtained Γ k (t), and the state after the transition S k (t+1) is experience e k It is stored in memory as (t).

[0077] If there is a time correlation in the sample sequence, learning may not proceed properly. To solve this problem, we set the parameter θ i When updating, the experience stored in memory k The memory samples are randomly taken from and processed in batches (step 90).

[0078] A randomly taken memory sample, i.e. (S k , a k , Γ k , S k Experiences that include k Using this, the parameters θ of DQN / DDQN i (Step 92). However, S k ´ is action a k Specifically, the process of step 92 proceeds as follows.

[0079] action a k Reward for Γ k Once determined, the reward Γ k Based on the state S k Acting under a kThe "new Q value" for selecting S k ,a k The new Q value is calculated, for example, as shown in the following formula (6).

[0080] Q(S k ,a k )←Γ k +γ*maxQ(S k ´,a k ´) ···(6) where γ is the time discount rate.

[0081] The right-hand side of the above equation (6), Γ k +γ*maxQ(S k ´,a k ´) is (S k ,a k ) is the value that we want the "trained DQN" to calculate as the Q value for each combination. Therefore, the DQN parameter θ i is (S k ,a k ) input, the value is updated so that it appears in the output layer.

[0082] The following equation (7) represents the DQN parameter θ i The loss function L used to update k Shows.

[0083]

number

[0084] However, in the above equation (7), the first term on the right side, Q(S k ,a k ;θ i ) is the parameter θ i The main network with (S k ,a k ) is the Q value calculated for the combination of k ´,a k ´;θ i ― ) is the parameter θ i― If the target network has (S k ´,a k ´) is the Q value calculated for the combination.

[0085] When updating DQN, if the value of the network to be updated is calculated using the network being updated, learning may not be stable. For this reason, when updating DQN, in addition to the main network to be updated, a target network for calculating the update value is usually used. In this embodiment, the second term of equation (7), which corresponds to the new Q value, is calculated using the target network according to the above-mentioned usual method.

[0086] In step 92, the loss function L shown in the above equation (7) is used. k The DQN parameters θ are set so that i , i.e., the main network parameters θ i As a result, the DQN is updated to state S k For input, action a k The Q value of the model is trained to express the value on the right side of equation (6) above.

[0087] The target network is identical to the past main network except immediately after the update. The target network is then updated at a predetermined period (step 94). That is, when the time for updating arrives at the predetermined period, the parameter θ i ― is the parameter θ of the main network. i This completes the update of DQN.

[0088] In addition, the following equation (8) represents the loss function L used in step 92 when the DDQN method is used in this embodiment. k The principle of the update is the same as that of DQN, so we will not explain the details here.

[0089]

number

[0090] [Processing by the wireless terminal's CPU] FIG. 9 is a flowchart for explaining the flow of processing executed by the CPU 72 of the wireless terminal 18 to realize the above functions.

[0091] 9, first, the communication environment surrounding the wireless terminal 18 is confirmed (step 100). Specifically, accessible wireless base stations 16 are confirmed for each interface.

[0092] Next, it is determined whether or not a change has been detected in the communication environment (step 102).

[0093] As a result, if it is determined that no change has been observed, a process of attenuating the ε coefficient is carried out (step 104).

[0094] On the other hand, if a change in the communication environment is detected, the ε coefficient is reset to its initial value (step 106), causing the attenuated ε to return to its original value.

[0095] Next, a random number is generated that falls within the range of 0 to 1. Furthermore, it is determined whether the random number is equal to or smaller than ε (step 108).

[0096] If the random number ≦ ε is found to be true, the current state S k For (t), the next action is a k (t) is determined randomly (step 110).

[0097] On the other hand, if it is not determined in step 108 that the random number ≦ ε holds, the input layer of the DQN is assigned a state S k (t) is input (step 112). Then, the Q values ​​appearing in the output layer of the DQN are compared, and the action a with the largest Q value is selected. i (t) is the optimal action at the current time a k (t) is selected (step 114).

[0098] Next, the action a selected by the above process k Based on (t), an association request is sent to surrounding wireless base stations 16 (step 116).

[0099] When it receives a response to its request, it takes the selected action based on the information contained in the response. k Reward Γ for (t) k (t) is calculated (step 118).

[0100] Next, the experience e acquired in this routine is stored in the RAM 78 constituting the memory unit 66. k (t), i.e., (S k (t), a k (t), Γ k (t), S k (t+1)) is stored (step 120).

[0101] Thereafter, the DQN is updated by batch processing (step 122), and the current routine is terminated.

[0102] As described above, according to the wireless communication system 10 of the present embodiment, in a heterogeneous environment including a plurality of wireless terminals 18 and a plurality of wireless base stations 16, it is possible to manage the connection state using a neural network such as DQN or DDQN. Then, by using the ε-greedy method, it is possible to converge the learning of the network early. Furthermore, by initializing ε when the communication environment changes, it is possible to appropriately make the characteristics of the network as a value function follow the situation after the change. Therefore, according to the configuration of the present embodiment, it is possible to always keep using the system resources related to wireless communication effectively.

[0103] Incidentally, in the above-described first embodiment, the wireless terminal 18 and the wireless base station 16 use two IFs for wireless communication, but the present disclosure is not limited to this. For example, the number of IFs for wireless communication may be three or more. Furthermore, the number of IFs for wireless communication may be a mixture of IFs for licensed bands and IFs for unlicensed bands used for wireless LANs and the like. Alternatively, the number of IFs for wireless communication may be only one, and deep reinforcement learning such as DQN may be used to determine which wireless base station 16 the wireless terminal 18 should optimally request a connection to.

[0104] In the above-mentioned first embodiment, DQN or DDQN is used as a value function for determining an action. However, these are merely examples, and the present disclosure is not limited to these. Any function that can determine the value of each action that can be taken under a specific state can be used as a value function in the present disclosure. [Explanation of symbols]

[0105] 10. Wireless communication systems 16, 16-1, 16-2, 16-3 Radio base station 18, 18 k Wireless terminal 52 Control information calculation unit 60 Attribution Calculation Department 62 Environmental change detection unit 64 ε judgment part Gamma k (t) Remuneration a k (t) action e k (t) Experience S k (t) Status S k (t+1) State after transition R kf Requested Data Rate r bkf Transmission rate

Claims

1. A wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, comprising: detecting a state of connection between the plurality of wireless base stations and the wireless terminal; randomly determining with probability ε a behavior to be adopted by said wireless terminal under detected conditions; determining, with probability (1-ε), an action to be taken by said wireless terminal under the sensed conditions in accordance with a result of a value function; transmitting a request to said plurality of radio base stations corresponding to an action to be taken; obtaining responses from the plurality of radio base stations to the request; calculating a reward for the behavior based on the response; updating a value for adopting the behavior under the state based on the reward; learning parameters of the value function based on the updated values; detecting a change in a communication environment surrounding the wireless terminal; performing a time-dependent attenuation process on the ε under a condition where a change in the communication environment is not detected; initializing the ε when a change in the communication environment is detected; Including, A wireless communication method in which the value function, when a state is given to an input layer, expresses in an output layer the value of adoption for each action that the wireless terminal can take.

2. At least some of the plurality of wireless base stations have a plurality of wireless communication interfaces having different specifications, the wireless terminal has the plurality of interfaces for wireless communication, the state includes information on which wireless base station the wireless terminal is connected to for each interface; The wireless communication method according to claim 1 , wherein the action includes information regarding which interface of the wireless terminal is to be connected to which wireless base station.

3. The wireless communication method according to claim 1 or 2, wherein the value function is a Deep Q-Network.

4. The wireless communication method according to claim 1 or 2, wherein the value function is a Double Deep Q-Network.

5. A wireless terminal having a function of connecting to a plurality of wireless base stations, A CPU is provided, The CPU, A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal; a process of randomly determining with probability ε an action to be taken under the detected condition; determining, with probability (1-ε), the action to be taken under the detected condition according to the result of the value function; transmitting a request corresponding to an action to be taken to said plurality of wireless base stations; obtaining responses from the plurality of wireless base stations to the request; calculating a reward for the behavior based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; A process of detecting a change in a communication environment surrounding the wireless terminal; a process of performing a time-dependent attenuation process on the ε under a situation where a change in the communication environment is not detected; A process of initializing the ε when a change in the communication environment is detected; Run The value function is a wireless terminal that, when a state is given to an input layer, displays in an output layer the value of adoption for each action that the wireless terminal can take.

6. A wireless communication device having a plurality of wireless communication interfaces each having different specifications; The state includes information on which wireless base station the wireless terminal is connected to for each interface, The wireless terminal according to claim 5 , wherein the action includes information regarding which interface of the wireless terminal is to be connected to which wireless base station.

7. A program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations, A CPU provided in the wireless terminal, A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal; a process of randomly determining with probability ε an action to be taken under the detected condition; determining, with probability (1-ε), the action to be taken under the detected condition according to the result of the value function; transmitting a request corresponding to an action to be taken to said plurality of wireless base stations; obtaining responses from the plurality of wireless base stations to the request; calculating a reward for the behavior based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; A process of detecting a change in a communication environment surrounding the wireless terminal; a process of performing a time-dependent attenuation process on the ε under a situation where a change in the communication environment is not detected; A process of initializing the ε when a change in the communication environment is detected; A program for executing A program for a wireless terminal, wherein the value function, when a state is given to an input layer, displays in an output layer the value of adoption for each action that the wireless terminal can take.

8. the wireless terminal has a plurality of wireless communication interfaces having different specifications, The state includes information on which wireless base station the wireless terminal is connected to for each interface, 8. The program for a wireless terminal according to claim 7, wherein the action includes information as to which interface of the wireless terminal is to be connected to which wireless base station.

Images